Paul Hammant's Blog: Coding Agents Guess Which Tests to Run. They Should Look It Up.

I’ve been updating trunkbaseddevelopment.com for coding agents, with Claude doing much of the typing. One section is now about AI coding agents committing straight to the trunk. That’s what you get if you remove the pull-request requirement and let a diaspora of agents in worktrees, VMs and containers race each other to HEAD. Those agents need fast local builds to win that race. Fast local builds on a big codebase need Test Impact Analysis (TIA). So I asked Claude a simple question: when you’re working in a repo with a too-large test suite, how do you pick which tests to run?

What a coding agent actually does

The answer was candid. Unless the repo provides it, there’s no coverage map and no record of which tests touched which lines last time. There’s no familiarity carried over from yesterday’s session either, unless someone wrote it into a CLAUDE.md or a memory file. What it does instead is reason from the diff:

1. By naming conventions: Foo.java to FooTest.java, foo.py to test_foo.py, a __tests__ folder next door.
2. By who uses what changed: grep for the changed symbols and take the callers, importers and subclasses, one or two hops out, and their tests.
3. By module scope: go test ./pkg/thing/..., mvn -pl module, a single pytest directory.
4. By graph-aware tooling where it exists: bazel query 'rdeps(...)', nx affected, jest --findRelatedTests. It trusts these most, because they are real dependency data and not its own reading of the code.
5. By widening when unsure: a change to a base class, config or serialization means running the bigger subset, or everything.

That’s TIA by inference. It’s the same thing a decent senior developer does in their head, and to be fair it’s quite good. Where it goes wrong is predictable: coupling that grep can’t see. Reflection, dependency injection wiring, annotation- and config-driven behaviour, string-keyed lookups, serialized formats, SQL, anything across a process boundary. A coverage-derived map catches those because it records what actually ran. The agent is inferring from text.

Claude also admitted to a behavioural risk. Like a hurried human, an agent under pressure to finish drifts towards running too few tests and declaring victory. The protection is the same as for the hurried human: the full suite runs in CI after the commit lands, and the agent says plainly which subset it ran.

Its own conclusion was the one I’d hoped for: given a real TIA map, it would rather query that than guess. TIA turns test selection from the agent’s judgement into data it can look up. Agents committing at volume may be the strongest argument for TIA yet.

Well of course, the pace of development of agentic coding means these gripes will be out of date soon, but it’s worth captureing the here and now.

Recap: what TIA is

I wrote The Rise of Test Impact Analysis for Martin Fowler’s site, and before that Reducing Test Times by Only Running Impacted Tests here. Briefly: run the suite with coverage instrumentation, record which production source files each test exercises, and store that as a map. For a given change, look up the tests that touch the changed files and run only those before the commit/push. The full suite still runs in CI afterwards. The subset is usually a small fraction of the whole, so the pre-push build gets fast again without anyone deleting tests or skipping the build.

DAG build systems like Bazel, Buck and Nx get much of the same benefit from their dependency graphs. I went through that in Google-style DAG build systems. TIA is the route for everyone using Maven, Gradle, MSBuild, npm and the rest.

The catch: maps are costly

TIA maps aren’t free:

• Building one is slow: You run the whole suite instrumented, ideally per test, which is slower than a normal run.
• They go stale, quickly:: Every commit makes the map slightly out of date. Most teams settle for rebuilding it nightly, or every N commits.
• Costly plumbing: It has to be stored, keyed to commits, and fetched by whoever (or whatever) wants it.
• Coverage is noisy: The same test can cover slightly different lines from run to run. This is especially true in legacy codebases.

The staleness tradeoff is reasonable, though. A stale map mostly misses brand-new tests and newly introduced coupling, and the full CI run after the commit catches those soon after. A map that is a day old and right 98% of the time beats a guess, and CI is the backstop for the other 2%, unless your CI is using the same TIA map for an elapsed time or “cloud minutes” reduction too.

Where TIA pays, and where it doesn’t

How much TIA saves depends on the shape of the codebase.

Good fit: Business applications are wide and shallow.

Hundreds of features, each test exercising a thin slice through them, and most changes touching one slice. Change the discount rules and you have a few dozen impacted tests out of thousands. The map pays for itself many times over. This is most of the enterprise software in the world, and most of what coding agents are being pointed at.

Bad fit: Programming-language implementations are narrow and deep.

Compilers, interpreters and VMs put nearly every test through the same core: lexer, parser, type checker, code generator, runtime. Change anything in that core and the map says “run everything”. That’s correct, but useless. TIA only helps there for changes at the edges, like a single standard library function or one diagnostic message. The tests are often data rather than code, too: source files in the language, with golden outputs. The useful question is “which language features does this input use?”, not “which test methods hit this line?”.

Language repos have their own techniques instead:

• Tiered suites: a small, fast smoke/conformance set before the push, with the big corpus left to CI.
• Tagging tests by language feature: touch closure codegen, run the closure tests. That’s TIA done by hand from domain knowledge rather than from coverage. Coding agents happen to be decent at it.
• Parallelism and sharding: when selection can’t cut the work, spread it out.

Not yet an observed habit

On trunkbaseddevelopment.com I have a page of observed habits of teams practicing Trunk-Based Development. Running the build locally before the push is one of them, and has been since the late 90’s. TIA isn’t one yet. It’s been possible for over a decade, and the people who adopted it rate it, but I can’t honestly say it’s something you see in most trunk-based teams making business solutions.

It should be. Agents are now doing much of the committing, many of them at once, all chasing the same HEAD. The local build is what keeps the trunk green, and the speed of the local build is what decides who wins the race to commit. An agent that guesses which tests to run is doing what we always did. An agent that can look it up is doing what we always should have.