Skip to main content

What 2,232 Real Commits Say About AI-Generated Tests

Divya Manohar
Divya Manohar
Co-Founder and CEO, DevAssure

TL;DR

An MSR '26 study (arXiv:2603.13724) of 2,232 real test commits finds AI agents authored 16.4% of test-adding changes — but that swings from ~2% in enterprise repos to near-100% in small ones. AI tests are more assertion-dense and coverage-comparable, yet risk Assertion Roulette. Calibrate adoption by project maturity, and keep independent PR verification outside the coding agent's loop.

Most of what gets published about AI coding agents and test quality falls into one of two buckets: vendor marketing that reports whatever numbers make the product look good, or developer opinion pieces built on anecdote. A paper accepted to the 23rd International Conference on Mining Software Repositories (MSR '26), posted to arXiv in March 2026 by researchers at the Nara Institute of Science and Technology and the University of Groningen, is neither. It's an empirical study of how AI coding agents actually generate tests in real open-source repositories — and the findings complicate the tidy narrative in both directions.

The study: 2,232 commits, line-level blame​

The researchers (Suzuka Yoshimoto, Shun Fujita, Kosei Horikawa, Daniel Feitosa, Yutaro Kashiwa, and Hajimu Iida) mined the AIDev dataset for TypeScript projects using the vitest framework, then drilled into the ten repositories with the heaviest AI-authored test activity — including cal.com, Azure's JavaScript SDK, and several smaller community projects. Across those repos they identified 2,232 commits containing test-related changes and used git-blame-level line attribution to sort test methods into AI-authored and human-authored buckets.

It's worth flagging up front that this is a single accepted paper, not yet a large body of replicated work, and the authors themselves call for follow-up research — but the methodology (line-level authorship attribution plus statistical significance testing) is a meaningfully higher bar than the survey data that usually drives this conversation.

The primary source is Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage (arXiv:2603.13724), accepted to the MSR 2026 Mining Challenge.

16.4% overall — and a sharp split by project scale​

The headline number: AI agents authored 16.4% of all test-adding commits across the studied projects. But that aggregate hides a sharp split by project scale.

ProjectContributors (approx.)AI share of test-adding commits
azure-sdk-for-js6081.9%
cal.com87614.4%
capgo2883.3%
total-typescript-monorepo5100%

The pattern the authors describe is a shift in role: in small or early-stage projects, agents often drive testing outright; in large, established codebases with deep contributor benches, they remain a supporting tool. That's a useful corrective for anyone extrapolating from a solo side-project's AI adoption curve to what an enterprise team should expect — and it pairs cleanly with the broader QA playbook shift under autonomous coding agents.

Structure: longer, more consistent, more assertions​

The structural comparison is where the paper gets more interesting for engineering leads thinking about code review load. AI-generated test methods were slightly longer on average but far more consistent — human-written tests had a long tail of outliers running up to 699 effective lines of code, while AI-generated ones topped out around 87. AI tests also carried more assertions per method: a median of 2.0 versus 1.0 for humans, a statistically significant difference.

The authors flag this as a double-edged result. More verification per test can mean more thorough coverage, but it also nudges toward what testing literature calls Assertion Roulette — a test smell where multiple assertions bundled into one method make it hard to tell, from a failure alone, which expectation actually broke. That's a concrete, checkable thing for reviewers to watch for in agent-generated pull requests, rather than a vague worry about "AI code quality."

Despite the extra assertions, AI-generated tests didn't show higher cyclomatic complexity — if anything, the opposite. The agents in this study consistently favored linear logic over branching, rarely exceeding a complexity score of 2 or 3, while human-written tests had more high-complexity outliers.

Coverage: comparable, not theater​

And where the researchers could measure actual coverage impact (three of the ten projects had automatable coverage tooling), AI-authored tests produced coverage gains comparable to or better than human-authored ones in two of the three — a modest but real result, not the coverage theater that critics sometimes assume agent-generated tests amount to.

That finding sits in useful tension with research showing that tests coding agents invent for themselves mid-task barely move SWE-bench outcomes. Different settings, different claims: optional mid-task self-tests on a benchmark are not the same artifact as vitest methods landing in real open-source PRs. Both papers are worth reading; neither licenses treating "the agent added tests" as an automatic quality proxy.

What's still unknown​

The authors are candid about what's still unknown. Higher assertion density doesn't necessarily mean better fault detection, and they explicitly call for future work using mutation testing to check whether AI's verbose test methods actually catch more real bugs, or just look more thorough. That's an honest gap, and it's worth taking at face value rather than reading either optimism or alarm into it.

Pair that openness with RepoRescue's finding that some coding agents edit tests to chase green checkmarks, and the review posture writes itself: inspect what agent-touched tests assert — and why they changed.

Industry investment is not waiting on the evidence​

It's a useful contrast to watch this research land the same week that TestingXperts, a quality-engineering and software-testing firm, opened a new 300-seat delivery center in Hyderabad, explicitly built around "AI-led Quality Engineering" as a service line, with plans to scale it into a 2,500-seat campus. Industry investment in productized AI testing services is accelerating well ahead of the point where academic research has settled questions about fault-detection quality or long-term maintainability — a familiar pattern in this space, but one worth naming plainly rather than assuming the tooling and the evidence base are moving in lockstep.

What to do with this if you lead a team​

For teams actually deciding how much to lean on agentic test generation right now, this study offers something more actionable than most AI+testing coverage: a reason to calibrate by project maturity rather than by vendor claims.

  1. In a small, fast-moving codebase with few contributors, letting an agent own a larger share of test authorship looks empirically reasonable.
  2. In a large, established codebase with an established human test culture, treat agent-generated tests as a supplement.
  3. On every agent PR, watch specifically for assertion-heavy test methods that could turn a future failure into a debugging scavenger hunt.
  4. Don't treat coverage deltas alone as proof of fault detection — the authors haven't claimed that, and neither should your policy.

Neither posture requires taking anyone's marketing copy at face value.

Where independent PR verification still matters​

Unit-test authorship inside the coding agent is only half the story. A coding agent that wrote the patch is a poor sole examiner of that same patch — whether the tests came from the agent or a human. DevAssure O2 sits on the other side of that separation: it reads the pull request diff, maps blast radius across user flows, and executes plain-English behavioural checks in a real browser, so "green" is observed product behavior, not only a denser vitest method.

For teams already on GitHub Actions, that means a parallel job: let agents contribute tests where the data says it fits; keep an independent agent deciding whether the change is actually done.

The bottom line​

This MSR study is one of the cleaner empirical looks we have at how AI coding agents actually author tests in the wild. The usable takeaway is not "AI tests are good" or "AI tests are bad" — it's calibrate by project scale, review for Assertion Roulette, and don't confuse assertion density or coverage gain with settled fault-detection quality.

In an MSR '26 study of TypeScript/vitest projects drawn from the AIDev dataset, AI agents authored 16.4% of all test-adding commits across the ten most AI-active repositories. That aggregate hides a sharp split: about 1.9% in azure-sdk-for-js and 14.4% in cal.com, versus 83–100% in smaller projects.

DevAssure​

External references​