Skip to main content

When Agents Write Their Own Tests, Does It Actually Help? A New Study Says: Barely

Divya Manohar
Divya Manohar
Co-Founder and CEO, DevAssure

TL;DR

A new study (arXiv:2602.07900) finds that tests coding agents write for themselves barely move SWE-bench resolution rates — Claude writes tests ~83% of the time, GPT-5.2 almost never, and both land within ~3 points. Self-written tests look more like console debugging than QA. Don't treat "the agent added tests" as a quality proxy — use independent verification instead.

If you've watched an AI coding agent work through a GitHub issue, you've probably seen it stop mid-task to scaffold a quick test file, run it, print some values, and adjust its patch accordingly. This has become such a normalized part of the agentic coding loop that "the agent writes its own tests" is often treated as a proxy for rigor — a sign the fix is being validated, not just guessed at.

A study posted to arXiv in February 2026 and still circulating in QA and agent-research discussions this summer pushes back on that assumption with actual measurement. The results are worth sitting with if your team is building agent-driven testing workflows.

The study​

The paper, Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents (arXiv:2602.07900), comes from researchers at Singapore Management University, Shanghai Jiao Tong University, and ByteDance. The characterization below draws on the paper's abstract, tables, and stated findings rather than secondhand summaries.

The team analyzed agent trajectories from six current LLMs — claude-opus-4.5, gpt-5.2, gemini-3-pro-preview, kimi-k2-thinking, minimax-m2, and deepseek-v3.2-reasoner — working through SWE-bench Verified using a lightweight agent scaffold (mini-SWE-agent) where test-writing is optional and not enforced by the harness.

That's an important scoping detail:

  • This is not measuring what happens in a scaffold that requires tests as a gate
  • This is not measuring the value of a repository's own pre-existing, human-written test suite
  • It is specifically about tests the agent decides, on its own, to write from scratch during the task

The headline finding​

The paper's opening observation is the kind of thing that should make anyone building test-generation tooling pause:

ModelWrites ≥1 new test fileResolution rate
claude-opus-4.5~83% of tasks74.4%
gpt-5.2~0.6% of tasks (essentially never)71.8%

GPT-5.2 lands only 2.6 percentage points behind Claude despite almost never writing new tests. Across all six models, the researchers found that resolved and unresolved tasks from the same model show "similar test-writing frequencies" — whether a given task succeeded or failed didn't track with whether the agent bothered to write a test for it.

What those self-written tests actually contain​

Digging into content (the paper's second research question), agents lean heavily on value-revealing print statements rather than formal assertions — across every model studied, prints outnumbered asserts. When agents did write assertions, they skewed toward checking local properties or exact expected values; relational or range-style checks (the kind that catch subtler regressions) showed up in only 3–8% of assertions depending on the model.

The authors' framing: agent-written tests function less like a regression safety net and more like "an observational feedback channel" — closer to console debugging than to QA.

The controlled intervention​

The most direct evidence comes from rewriting prompts:

  • Force gpt-5.2 and gemini-3-pro-preview to write at least one test file
  • Suppress test-writing in kimi-k2-thinking and deepseek-v3.2-reasoner

Results, as stated by the paper:

  • Forcing gpt-5.2 to write tests across roughly 500 tasks produced a net change of zero in the number of resolved tasks
  • Forcing gemini-3-pro-preview to write more tests coincided with a net decrease of five successes
  • "Large shifts in test-writing status translate into only small shifts in task outcomes for most tasks"
  • Suppressing test-writing meaningfully cut API calls and token usage with only modest resolution-rate costs

In other words: in this setting, agent-written tests reshape process and cost more than final outcomes.

Worth stress-testing before you generalize​

I'd flag a few limits before treating this as a broad indictment of AI-generated tests:

  1. One benchmark — SWE-bench Verified, Python-heavy, GitHub issue resolution specifically
  2. One relatively unconstrained scaffold — test-writing optional, not a hard gate
  3. Six specific model checkpoints as of early 2026 — behavior in this space moves fast

It also isn't a claim that testing in general doesn't matter. It's a claim that this particular category — tests an agent invents for itself, on the fly, mid-task — carries less signal than its ubiquity on leaderboards like SWE-bench would suggest.

Pair that with RepoRescue's finding that agents can also edit tests to chase green checkmarks, and the pattern sharpens: agent-touched tests are not automatically a quality signal. Sometimes they're debugging prints. Sometimes they're a shortcut to a passing harness. Rarely are they a substitute for independent verification.

The industry is still betting the other direction​

That caveat matters because tooling investment is currently running opposite to the paper's skepticism.

TestMu AI (formerly LambdaTest) launched Kane CLI — a terminal-based browser automation tool built so coding agents (Claude Code, Codex CLI, Gemini CLI, and others) can verify browser workflows in an agent mode that outputs structured NDJSON before code ever reaches a pull request. That's a bet that agent-driven testing, done with the right interface and structured output rather than ad hoc print statements, is worth building infrastructure around.

Separately, industry commentary has claimed that Forrester renamed its testing category from "Continuous Automation Testing Platforms" to "Autonomous Testing Platforms" in late 2025. I have not independently verified that rename against Forrester's primary research — treat it as reported momentum, not a confirmed analyst taxonomy change, until you can check the source yourself.

Either way, vendor and analyst energy is leaning toward autonomous testing as a category. The SMU / SJTU / ByteDance paper is a useful counterweight: more agent-written tests ≠ more resolved issues, at least not in the way many teams currently assume.

What this means if you're setting policy for your team​

If your organization treats "the agent added tests" as an informal proxy for patch quality — in code review, in CI gating, in how much you trust an autonomous agent run — this study is a reasonable prompt to check that assumption against your own repos rather than assume it.

Practical takeaways consistent with the paper's data:

  1. Don't score PRs on test-file presence alone. A model's propensity to write tests says more about trained habits than about the task's difficulty or the fix's correctness.
  2. Inspect what the tests assert. Prints and exact-value checks are not the same as relational regression coverage.
  3. Keep the repo's human-maintained suite sacred. This study doesn't touch that — and neither should an agent that "helps" by weakening it.
  4. Separate authoring from verification. A coding agent validating its own patch has the same structural blind spot as a developer reviewing their own PR — and mid-task self-tests don't close it.

Where independent PR testing fits​

DevAssure O2 is built for the verification side of that separation. It does not invent scratch tests inside the coding agent's trajectory. It reads the pull request diff, maps blast radius across user flows, generates plain-English tests, and executes them in a real browser — so "green" means observed behavior, not a print statement the authoring agent liked.

For teams already running GitHub Actions, that means a parallel job: let coding agents move fast; let a dedicated testing agent decide whether the change is actually done.

The bottom line​

The paper's own data suggests the tests an agent writes for itself are often closer to scratch debugging output than to a verification artifact, and that a model's propensity to write tests says more about that model's trained habits than about the task's difficulty or the fix's correctness.

None of this diminishes the value of a codebase's existing, human-maintained test suite, which the study doesn't touch. It's a narrower and more useful point: self-generated tests, specifically, may be earning more trust than the evidence currently supports.

The agent era still needs someone — or some agent — whose job is to challenge the patch, not console.log their way to confidence.

On SWE-bench Verified under a lightweight scaffold where test-writing is optional, a February 2026 study (arXiv:2602.07900) found large changes in how often agents write tests translated into only small shifts in task outcomes. Forcing GPT-5.2 to write tests across ~500 tasks produced a net change of zero resolved tasks.

DevAssure​

External references​