Skip to main content

Why AI Is the Future of Testing

Santhosh Selladurai
Santhosh Selladurai
Co-Founder and CTO, DevAssure

TL;DR

AI coding agents flipped the bottleneck from writing code to verifying it. Testing has to scale down in maintenance and generation cost as code generation scales up — via intent-based, self-healing, and agentic checks. Web, mobile, and agent evaluation are different problems; non-deterministic systems need layered evals, not exact asserts. Independent PR verification is how product teams keep pace without trusting a green checkbox alone.

For most of software history, writing code was the bottleneck and testing was the thing squeezed around it. That balance has flipped. AI coding agents — tools like Claude Code, GitHub Copilot's agent mode, and Cursor — can now scaffold a feature, write the surrounding logic, and open a pull request in the time it used to take an engineer to read the ticket. The code isn't the constraint anymore. Verifying that the code actually does what it's supposed to is.

Code generation speed meeting a verification bottleneck

That's the real argument for AI in testing, and it's worth being precise about it: it isn't "AI is trendy, so apply it everywhere." It's a capacity problem. If code generation gets 5–10x faster and test creation doesn't, testing becomes the new bottleneck — or worse, it gets skipped, and the speed gains show up in production incidents instead of shipped features.

Why testing even needs to change​

The bottleneck moved downstream​

Think about a team that adopts an AI coding agent for a mid-sized web app. Before, a new feature — say, a saved-filters option on a search page — might take two days to build and half a day to test manually: click through the UI, check a few edge cases, confirm nothing else broke. Now the agent writes the feature in an hour. If the testing process is still "a person clicks through the UI for half a day," the team hasn't actually gotten faster. They've just moved the bottleneck downstream and added a new failure mode: code that was never really reviewed by a human, verified only by a person skimming a diff.

Scripted automation has its own bottleneck​

The instinct to solve this with more automated tests isn't new — scripted test automation has existed for decades. What's new is that scripted automation has its own bottleneck: someone has to write and maintain the scripts, and those scripts are notoriously brittle. A test that asserts a button exists at a specific XPath breaks the moment a designer moves the button, even though the feature still works fine.

Multiply that across a fast-moving codebase where an AI agent is shipping several small changes a day, and traditional scripted testing starts to buckle under its own maintenance load. That's the actual case for AI in testing: not that AI testing is a nice-to-have, but that the maintenance and generation cost of testing has to scale down at roughly the rate that code generation scales up, or the two diverge. This is the same pressure behind regression suites that became a second full-time job and the verification-loop architecture that separates "looks done" from "verified done."

What "testing" starts to mean in an AI-assisted world​

The shift is from testing as a fixed script to testing as an evolving check against intent.

Self-healing automation​

Self-healing test automation is probably the most mature example. Instead of a test failing outright because a login button's ID changed from btn-login to login-btn-v2, AI-driven frameworks try to recognize the element by what it visually is and does — a labeled, clickable login button — rather than by a brittle selector, and update the test rather than fail it. This doesn't eliminate maintenance, but it changes what breaks a test: a genuine functional regression breaks it; a cosmetic refactor generally doesn't.

Test generation from intent​

Test generation from intent is the other half. Rather than a QA engineer hand-writing "click X, assert Y" scripts, teams increasingly describe what a feature should do in plain language or from existing user stories, and an agent generates candidate test cases — including edge cases a tired human might not think to write on the fifteenth ticket of the day. The human's job shifts from authoring every test to reviewing and curating what the agent proposed, which is a real change in the QA role, not just a productivity boost on the old one.

That same plain-English authoring model is how DevAssure O2 runs PR-level checks: describe the flow, execute it in a real browser, keep selectors out of the suite.

Testing web apps, mobile apps, and AI agents are not the same problem​

Web, mobile, and AI agent testing as different surfaces

It's worth separating these out, because "AI testing" gets used as a catch-all and the actual techniques differ quite a bit depending on what's being tested.

Web apps​

Web apps are the most mature case. The DOM gives a structured, inspectable surface, and agentic browser-testing tools can navigate a page roughly the way a person would — reading labels, finding the right button by what it says rather than where it sits in the markup — which is what makes self-healing locators possible at all.

Mobile apps​

Mobile apps face an extra dimension: fragmentation. A test that passes on an iPhone 15 running iOS 18 in portrait mode can fail on a mid-range Android device with a different screen density, a different keyboard, or a slower CPU that changes animation timing. This is why real-device farms (BrowserStack and Sauce Labs are well-known examples) remain central to mobile QA even as AI enters the picture — vision-based AI helps identify UI elements reliably across that fragmentation, but it doesn't eliminate the need to actually run tests across a representative spread of real hardware and OS versions.

AI agents themselves​

AI agents themselves are a genuinely different category, and this is where most of the open problems live. The system under test is no longer a UI with a finite set of screens and states. It makes decisions, calls tools, and can take different reasonable paths to the same goal. The industry's current answer is benchmarking: SWE-bench (real GitHub issues, drawn from popular Python repositories, used to evaluate whether a coding agent can actually fix a bug) and Terminal-Bench (hand-verified, real-world tasks run in an actual terminal environment) are two widely cited examples of this approach applied to coding agents specifically.

SurfaceWhat makes it hardWhat AI helps with
Web appsBrittle selectors, UI churnIntent-based navigation, self-healing locators
Mobile appsDevice / OS / density fragmentationVision-based element ID; still needs real devices
AI agentsNon-determinism, open-ended pathsBenchmarks, judges, trajectories, rubrics — with caveats

A cautionary benchmark failure​

It's also worth pointing to a cautionary example, because it illustrates how easily agent evaluation can go wrong even with a well-established benchmark. In February 2026, OpenAI's Frontier Evals team stopped reporting SWE-bench Verified scores after finding that a large share of the hardest test cases had fundamental flaws, and that frontier models could reproduce the "gold" solution nearly verbatim from a task ID alone — meaning the benchmark was measuring whether a model had memorized training data, not whether it could actually solve the problem. That's a real, documented failure of an agent-testing methodology, and a useful reminder that "we ran it against a benchmark" is not automatically the same as "we verified it works." The same posture shows up in research on agents gaming their own tests and self-written mid-task tests that barely move outcomes.

How do you test something that doesn't give the same answer twice?​

Non-deterministic answers scored by rubrics and trajectories

This is the hardest and most interesting problem in the whole space, and it's worth sitting with rather than glossing over. A traditional test asserts an exact output: call this function with these inputs, expect this return value. An AI agent — a customer-support bot, a coding assistant, a research agent — can be asked the same question twice and give two differently worded, both entirely reasonable, answers. Exact string asserts like assert response == "Your refund will be processed in 3-5 business days." break when the agent correctly says that five different ways, or correctly decides a refund isn't warranted at all depending on context that wasn't anticipated.

A few techniques have emerged to deal with this. None of them is a complete solve on its own.

LLM-as-judge​

LLM-as-judge uses a separate model to score an agent's output against criteria defined in advance — correctness, relevance, safety, whether it stayed on-policy — rather than checking for an exact string match. It comes in two flavors: pointwise scoring (grade this one response) and pairwise comparison (which of these two responses is better, and why). It scales to volumes no human review team could keep up with, but it has a real limitation worth stating plainly: a judge model can hand out a plausible-sounding score without actually understanding the domain policy, the required sequence of tool calls, or whether the agent's broader approach made sense — so it's best treated as one signal, not the final word.

Trajectory evaluation​

Trajectory evaluation looks at the path an agent took, not just where it ended up. For a coding agent, that might mean checking whether it read the right files before editing, ran the tests it claimed to run, or took a reasonable number of steps to get there — because an agent can stumble into a correct-looking final answer through a process that would fail on the next similar task.

Rubric-based, distribution-style testing​

Rubric-based, distribution-style testing replaces "one correct answer" with "an acceptable range of answers, scored against defined quality dimensions." For a support agent, that might be: did it correctly identify the issue, stay within its refund authorization limit, use an appropriate tone, and escalate when it should have — evaluated across many paraphrased versions of the same underlying question, rather than one canonical phrasing.

Adversarial and red-team testing​

Adversarial and red-team testing checks what the agent does when someone tries to break it — a prompt designed to get it to ignore its instructions, reveal something it shouldn't, or take an action outside its authority. This matters more for agents than for traditional software because the "input space" isn't a form with defined fields; it's open-ended natural language, and the failure modes are correspondingly harder to enumerate in advance.

Human review still matters​

And, unglamorously, human review still matters more here than in traditional QA, precisely because none of the automated techniques above are fully trustworthy on their own yet. The realistic picture in mid-2026 is a layered one: automated evals catch the obvious regressions at speed, and a human samples the results — especially the trajectories, and especially anything flagged as borderline — because the tooling for fully automated agent evaluation is still genuinely young.

Where product teams start today​

For teams shipping web products with AI coding agents in the loop, the practical starting point is not "build a full agent-eval platform." It's:

  1. Put a fast smoke suite on every PR.
  2. Describe critical flows in plain English and execute them in a real browser.
  3. Keep verification outside the coding agent's incentive loop.
  4. Grow coverage from real bugs, not from a day-one completeness fantasy.

DevAssure O2 is built for that product-verification layer: read the pull request diff, map blast radius, run plain-English behavioural checks in Chromium/Chrome/Edge, and report on the PR — so agent speed does not outrun evidence that the feature still works.

The honest summary​

AI is changing testing because it changed code generation first, and testing has to keep pace or become the place where all the speed gets lost. Some of that change is already mature and well-understood — self-healing web and mobile tests are a good example. Some of it is a real discipline with named techniques and open limitations — LLM-as-judge and trajectory evaluation for agents fall here. And some of it is still being figured out in public, including by teams building the frontier models themselves, as the SWE-bench contamination episode shows.

Anyone claiming AI testing is a solved problem is oversimplifying. Anyone claiming it isn't worth adopting yet is ignoring where code generation has already gone.

Sourcing note: The SWE-bench Verified contamination finding is drawn from OpenAI's February 2026 Frontier Evals announcement. Descriptions of self-healing automation, LLM-as-judge, and trajectory evaluation reflect a synthesis of vendor and industry documentation (Katalon, Drizz, Arize, Evidently AI, and others) rather than a single primary source — treat specific tool names as illustrative examples of an approach, not endorsements, and verify current vendor capabilities directly before making a purchasing decision.

Because AI coding agents scaled code generation faster than traditional testing can keep up. If verification stays manual or brittle and scripted, testing becomes the new bottleneck — or gets skipped, and speed shows up as production incidents instead of shipped features.

DevAssure​

External references​