The Tests Nobody Deletes

The Afternoon the Test Backlog Vanished

You know the ticket. “Add tests for the billing module,” two years old. One afternoon you hand it to Claude. Twenty minutes later there are forty tests, all green, and two of them hit an edge case you had forgotten existed. The relief is real. For a lot of teams this was the first time testing felt cheap, and that was good news.

Now picture the same repository a few months later. The suite is several times its old size and nobody has read most of it. CI still goes green, but green means less than it did when every test had an author who knew why it was there.

Tests Are Code, and Code Has Weight

A test that asserts behaviour is bound to that behaviour. Change the behaviour and the test has to change with it. John Ousterhout has a name for the general problem: change amplification, where a simple change forces edits in many different places. He meant code in general. We apply it to tests, and every test generated in volume adds one more place a simple change has to touch.

Suites outgrowing their code is older than agents. Agents speed it up: by mid-2025, more than half of agent pull requests touched at least one test file (sources).

The best objection comes from Simon Willison. When an agent can fix a broken test in a few minutes, he argues, the old reasons for skipping tests no longer hold. He is right about the rewriting. Deciding whether the rewritten test still checks the right behaviour is the expensive part, and that decision stays with a human.

Most test failures that aren’t flaky still earn their keep, because they usually point at real faults. The cost we mean grows with unreviewed volume, and in our client work it shows up as slower suites. Public data measures only the volume, and that grew sharply within a year (sources).

We argued elsewhere that code is turning into a throwaway artifact. Tests are the part nobody throws away.

Measuring What Nobody Tests

In June 2026 two researchers, Partha Protim Paul and Reid Holmes, asked a plain question. Of everything a library’s documentation promises, how much does no test check at all? Their tool reads the promised behaviours out of each method’s documentation and looks for a test that validates each one. They ran it on ten open-source Java libraries.

The answer: in the suites the libraries’ own developers wrote, at least one in six documented behaviours has no test checking it. Some projects do much better than others. Even the best one has a gap, with line coverage and mutation score close to perfect.

Then comes the comparison that matters to anyone using agents. On four of the libraries the researchers also measured suites written by two automatic test generators. Those suites left more behaviour untested than the developers’ suites did. By the usual metric they still looked healthy, with most of their methods at high line coverage.

The study is a preprint and not yet peer reviewed, and its gap figures are lower bounds. The numbers and a chart are in the appendix.

This is no proof that agents write bad tests. It shows that generated volume carries a larger behavioural gap than its coverage suggests, and a dashboard can’t tell the difference.

Why Coverage and Mutation Scores Miss It

Line coverage proves that a line ran. It says nothing about whether anyone checked the result. In the developer-written suites Paul and Holmes studied, more than a third of the methods with full line coverage still had at least one untested behaviour.

Mutation testing is the stronger instrument, and it still misses a lot. Even where a suite caught every artificial bug the tool planted in a method, that method often still had untested behaviour. The generated suites did worse here too. A high kill score hints at good behavioural coverage, but only weakly (numbers).

The same blindness hits the usual fix for a bloated suite: shrink it by coverage. In one study, dropping every test that looked redundant by coverage removed a large part of each suite and measurably weakened it. A second study looked at real builds. The tests marked redundant still caught real failures, and the slimmed suites missed them. Coverage is the wrong ruler for deleting, too (studies).

How Meta Filters Generated Tests

Meta’s TestGen-LLM doesn’t trust its own output. Every generated test passes three filters in order. It has to build. It has to pass reliably on repeated runs, or it gets thrown out as flaky. And it has to cover lines the existing suite didn’t. Whatever survives goes to a human reviewer.

The filters are strict. For three test classes in four, not a single generated test made it through all three filters (details). The paper used older models, and its last filter checks only line coverage, a weak gate as shown above. The structure still transfers.

A separate benchmark points the same way. One model wrote far more tests than another, reached barely more coverage and added less new ground, so the extra tests bought little (details).

The gate is what Meta built, and it is the part worth copying. The generator is the easy bit.

What Teams Can Change Now

All four steps run on what you already have (sources).

1. Gate generated tests. Copy Meta’s three filters as the floor: builds, passes repeated runs, adds coverage. Then add one of your own: every new test names the behaviour it checks, and a reviewer confirms it.
2. Measure the behavioural gap on the diff. BFinder, the tool from the study, is a research prototype, but the method transfers. For each method a change touches, list the behaviours its documentation promises and mark which ones a test asserts. That is the review question coverage can’t answer.
3. Run predictive selection before buying CI capacity. Facebook trained a model to pick the tests a change actually needs. It ran a fraction of the tests, still caught nearly every faulty change and halved the machine cost of testing. That is Facebook’s own report, but the principle scales down.
4. Delete on a schedule. Teams rarely delete tests, and when they do, it is mostly because the production code went away. Google and Microsoft have tools that remove dead code or skip test runs; we found nobody who deletes passing tests of live code on a schedule. Delete by behaviour, since coverage redundancy is the wrong criterion, as shown above. A test that checks no behaviour anyone can name is a candidate.

One deliberate deletion habit beats three dashboards.

Our Open Bet: Start From the Goals

BFinder derives behaviours bottom-up from method documentation, where code gets written. Impact mapping works top-down, from a business goal to the actors who influence it and the impacts needed from them. That is where a company decides what matters. An inventory built from the top would let a team rank behaviours by business weight and ask which of them any test covers.

Others have come close. Gojko Adzic, David Evans and Tom Roden call the actor-impact level of an impact map a good starting point for talking about capabilities in exploratory testing. A 2026 approach called RTM deletes redundant tests while keeping every requirement covered. We found no published work that uses an impact map as the inventory for measuring test adequacy or deciding deletion (sources).

We are trying this in client engagements now. We build the impact map with the people who own the goal and attach each test in a changed area to an impact. Impacts with no test are gaps, ranked by business weight. Tests with no impact go on the deletion list. We want to learn two things: whether the map is cheap enough to keep current, and whether ranking by business weight changes which gaps a team closes first. If keeping the map in sync costs more review time than it saves, or if the heavy impacts turn out to be the ones already well tested, we drop the idea.

We have nothing to report yet. When we do, it gets its own article, with measurements.

Why We Track This

We build with agents in client systems, so the growing suite is ours too. Test counts are easy to report and say very little. In a 100-day engagement we report covered behaviours and deleted tests, so the team that owns the code after we leave inherits a suite it can read. Both habits belong in our dev-org modernization work. The honest claim is narrow: we measure this and we delete, and we haven’t solved it.

Appendix: Numbers and Sources

Tests as code

• Change amplification. John Ousterhout, A Philosophy of Software Design, 2nd edition, chapter 2: “a seemingly simple change requires code modifications in many different places”. He wrote it about code in general; applying it to tests is our extension.
• Suite growth before agents. Bailey and Cadar (ICST 2025) found suites outgrowing code in most of nine mature C/C++ projects.
• Agent pull requests. Haque, Ingale and Csallner (MSR ‘26) report that the share of agent pull requests touching at least one test file rose from 31% to 52% between January and July 2025.
• The objection. Simon Willison, “First run the tests”, part of his Agentic Engineering Patterns: the old reasons for skipping tests “no longer hold when an agent can knock them into shape in just a few minutes.”
• Failures that find faults. Labuschagne, Inozemtseva and Holmes (ESEC/FSE 2017) found that 74.1% of deterministic test failures across 61 Travis projects pointed at real faults.
• CI volume. CircleCI’s vendor report “The 2026 State of Software Delivery” counts 59% more daily workflow runs year on year, while the median run got shorter.

The behavioural-gap study

Partha Protim Paul and Reid Holmes, “Beyond Coverage and Kill Scores: Empirically Measuring Test Suite Behavioural Gaps”, preprint, June 2026 (arXiv:2606.10417, not yet peer reviewed). Their proof-of-concept tool, BFinder, extracts the behaviours a method’s documentation promises, maps them to existing tests and flags every behaviour no test validates.

• Data set. Ten open-source Java libraries from Defects4J, among them commons-lang, gson and jsoup: 8,922 documented methods, 20,729 extracted behaviours, 93.1% extraction precision.
• Developer-written suites. At least 17.5% of detected behaviours are validated by no test. It is a lower bound, counted over the behaviour of documented methods only. Per project the gap runs from 6.6% (commons-cli) to 29.3% (jfreechart). commons-cli has 98.2% line coverage and a 93% mutation kill score, and it still has a gap.
• Generated suites. Four of the projects (commons-cli, -codec, -compress and -jxpath) also have suites from two test generators, EvoSuite and ASTER. Those suites leave at least 20.6% (EvoSuite) and at least 27.1% (ASTER) of detected behaviours untested. The developer-written suites on the same four projects land between 6.6% and 20.2%. Structurally, the generated suites look fine: 89.7% of EvoSuite’s analysed methods and 81.2% of ASTER’s sit at 91 to 100% line coverage.

Coverage and mutation score

• Line coverage. In the developer-written suites Paul and Holmes studied, 38.2% of methods with 100% line coverage still had at least one untested behaviour.
• Mutation score. Among methods where the developer suites killed every PIT mutant, 29.6% still contained an untested behaviour. For the generated suites the figure is 34% (EvoSuite) and 38% (ASTER). The correlation between kill score and behavioural coverage is ρ = 0.377.
• Minimising by coverage. Noemmer and Haas (SWQD 2020): statement-coverage-based minimisation on seven open-source Java projects removed 50 to 93% of tests and cost 3.5% to 21% of relative mutation score, about 12.5% on average.
• Real builds. Shi, Gyori, Mahmood, Zhao and Marinov (ISSTA 2018): tests that coverage- or mutant-based reduction marked redundant still caught real failures, and failed-build detection dropped by 9.5% to 52.2% across 1,478 failed builds in 32 projects.

The Meta gate

• Filter chain. Meta’s TestGen-LLM (Alshahwan et al., arXiv:2402.09171, FSE 2024). A candidate first has to build. Then it has to pass on its first run and on each of five runs; a single failure gets it thrown out as flaky. Finally it has to raise line coverage, and whatever survives goes to a human reviewer.
• Results. Across 86 test classes for Instagram Reels and Stories, 75% of classes got at least one generated test that built, 57% one that passed reliably, and only 25% one that also increased line coverage. The paper used Meta-internal models from 2023, and the last filter checks line coverage only.
• Unique contribution. In the MultiFileTest benchmark (Wang et al., arXiv:2502.06556), every tested model showed low unique contribution, meaning coverage that no other generated test already provided. GPT-o1 wrote about 2.9 times as many tests as GPT-5-mini, reached slightly higher coverage and had less than half the unique contribution. That holds for this pair on a Python benchmark only.

Selection and deletion

• Predictive test selection. At Facebook, predictive test selection ran fewer than a third of the tests that build-dependency-based selection would have run, still reported over 99.9% of faulty changes and halved the machine cost of testing (Machalica et al., arXiv:1810.05286, ICSE-SEIP 2019). That is Facebook’s own report.
• How rarely tests get deleted. Bhatta, Kendemah and Jha (MSR 2025) found that 0.47% of commits in seven Java projects delete tests, and in the six projects that also contain production code, 91.4% of deleted tests went because their production code went.
• Existing tools. Google’s Sensenmann removes dead code and Microsoft’s THEO (Herzig et al., ICSE 2015) skips runs.
• Coverage redundancy as a criterion. Shi et al. measured what it costs; see Coverage and mutation score.

Prior work on the open bet

• Impact maps in testing. Gojko Adzic, David Evans and Tom Roden call the actor-impact level of an impact map “a good starting point for discussing capabilities” in exploratory testing (Fifty Quick Ideas to Improve Your Tests).
• Requirement-preserving reduction. RTM (Pan, Niu, Briand and Hu, TOSEM 2026) deletes redundant tests while keeping full requirement coverage.