Skip to content

Software Quality

Agent experience

Agent experience (AX) names the friction between a codebase and an autonomous coding agent working in it. It covers how readily the agent can understand the code, change it, and act on the system around it. It is what developer experience (DX) names for a human, with the developer replaced by a machine. The properties that reduce that friction are the maintainability properties: types, locality, fast deterministic tests, clear errors, and recorded intent.

Developer experience is measured by self-report survey and sits at folklore tier as causal evidence. Agent experience is mechanically measurable: task success rate, tokens spent, and retries are numbers, and they move with the state of the codebase, so a change to the code can be tried against an agent the way a performance change is benchmarked. The soft claims of DX ("types help", "locality helps") become testable when the developer is a machine.

Comprehension: reading and changing the code

An agent's cognitive load is a literal budget. Its context window is finite, so code where understanding one change means loading a dozen files costs tokens and raises the error rate. Locality, explicit types, and named contracts cut that cost. An agent also has no memory across sessions, so the recorded why (docstrings, ADRs) matters more than it does for a person who builds a mental model of the system over months.

Its feedback loop is the act-observe-correct cycle: run the tests, read the typechecker, read the error. This is the dominant comprehension lever, more so than for a human, because an agent works by closing loops and has little intuition of its own. A flaky test or a cryptic error degrades an agent harder than it degrades a person, who can discount a signal they know to be noise.

Interface: acting on the system

The agent also acts, and the interface it acts through is its own source of friction. A scriptable command-line tool or a clean typed API is usually enough, and agents drive them well. The Model Context Protocol earns its place where text in and text out cannot express the job: browser interaction is the clear case, where the agent must drive a page and read the rendered result, as the Playwright MCP server does.

Response noise is a cost on the interface, and it feeds back into comprehension. A tool that answers a three-value question with tens of kilobytes of output spends the context budget on noise.

What measurement shows

Early measurement already corrects folklore. Model providers recommend a repository-overview file, the AGENTS.md convention. Such files do not generally raise task success and raise inference cost by more than 20% on average: the actionable instructions in them are followed, but the narrative overview is not useful (Gloaguen et al. 2026)1. A context file earns its tokens only when it is corrective, telling the agent something that overrides a default it would otherwise reach. An overview restates what the model already infers from the code, so it pays tokens for information the agent had; an instruction that countermands a wrong prior (the real build command, a pattern to avoid) is information it lacked. A doc an agent writes for itself without curation echoes the model's own knowledge and adds noise.

What does separate success from failure is the comprehension and feedback work itself. Agents that gather the right context before editing and invest in validation succeed more often, and the structure of a trajectory (read, patch, validate) predicts the outcome better than its length (Mehtiyev and Assunção 2026)2. The model is the primary driver of the result, so codebase effects are second-order. The failures that remain are dominated by flawed reasoning and stalled exploration rather than syntax (Liu et al. 2025)3: the expensive part is comprehension, not generation.

The instrument needs its own scrutiny. A re-examination of SWE-bench found 32.67% of "resolved" instances had the solution disclosed in the issue text and a further 31.08% passed on weak tests; with both removed, one agent's resolution rate fell from 12.47% to 3.97% (Aleithan et al. 2024)4. The benchmark is still being calibrated, so a trend measured on one codebase is better evidence than an absolute leaderboard figure.

The feedback loop is an oracle in the loop

Giving an agent fast, deterministic tests and a clear compiler puts a sound oracle in its loop, the same cure bugs in AI-generated code prescribes for a generator that would otherwise ship plausible-but-wrong output. Agent experience is the supply side of that cure: a codebase whose oracles are fast, deterministic, and reachable is one an agent can iterate inside. The methods that make code maintainable and the methods that make an agent effective in it are the same methods, now with a meter attached.

Referenced by

References

  1. Gloaguen, Thibaud, Niels Mündler, Mark Müller, Veselin Raychev, and Martin Vechev. 2026. Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? https://doi.org/10.48550/arXiv.2602.11988. ↩

  2. Mehtiyev, Tural, and Wesley Assunção. 2026. Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure. https://doi.org/10.48550/arXiv.2604.02547. ↩

  3. Liu, Simiao, Fang Liu, Liehao Li, et al. 2025. An Empirical Study on Failures in Automated Issue Solving. https://doi.org/10.48550/arXiv.2509.13941. ↩

  4. Aleithan, Reem, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. 2024. SWE-Bench+: Enhanced Coding Benchmark for LLMs. https://doi.org/10.48550/arXiv.2410.06992. ↩