Skip to main content

Regression Testing Was Never Supposed to Be This Much Work

Divya Manohar
Divya Manohar
Co-Founder and CEO, DevAssure

TL;DR

Regression testing only needs to answer: did this change break something that used to work? Script suites freeze mechanics until flakes drown the signal. O2 keeps intent in plain English, scopes runs to PR blast radius, and adapts to UI drift — so QA capacity goes back to testing, not nursing locators.

Regression testing exists to answer one question cheaply and repeatedly: did this change break something that used to work? That's the whole job.

Somewhere along the way, most teams ended up spending more energy keeping the regression suite alive than they spend getting the answer it was supposed to give them. This post looks at why that happened, and at a different approach — DevAssure's O2 agent, which runs regression tests written in plain English instead of Selenium or Playwright code — and what changes when you let an agent decide what to test and how, rather than encoding every decision into a script.

The traditional regression suite becomes its own liability​

A script-based regression suite is a pile of decisions frozen in code: this element has this selector, this workflow has these exact steps in this exact order, this field expects this exact string. Every one of those decisions is correct on the day it's written and starts decaying the moment the UI changes. A button gets a new class name, a form gets reordered, a modal replaces an inline panel — none of that is a real regression, but all of it breaks a locator-based test.

The result is the pattern most engineering teams will recognize immediately: tests that fail intermittently for reasons that have nothing to do with the code under test — flaky tests — and a corresponding drift toward not trusting red builds.

Google's engineering org documented this at scale. In analysis presented by John Micco and colleagues (and discussed widely alongside Google's Testing Blog coverage of flaky tests), roughly 84% of pass-to-fail transitions in a large sample were attributable to flakiness rather than actual product breakages, with about 16% of tests showing some flakiness. Whatever the precise methodology details, the direction is not controversial: once a suite is flaky enough, engineers stop reading failures as signal. They re-run the job, or they comment out the offending test "temporarily," and temporarily becomes permanent because nobody has time to go back and fix a test that isn't blocking anyone urgently.

The regression suite is still there. It's just no longer doing the one job it exists to do. That's the same structural story as the quiet death of the test script — and why common regression mistakes keep recurring even on strong teams.

None of this is a knock on the engineers maintaining these suites — it's a structural problem. Script-based automation asks a human to encode both the intent of a test ("a user should be able to check out") and the mechanics of a browser interaction (find this selector, wait for this element, click here, type there). When the mechanics break, they break loudly and often, and fixing them consumes exactly the time that was supposed to go toward writing new coverage.

Separating intent from mechanics​

The premise behind DevAssure's O2 is to let engineers and QA teams describe intent and let an agent handle mechanics. A test case is a YAML file with a summary and a list of steps written the way you'd explain the test to a new hire:

summary: Login and verify dashboard
steps:
  - Open the application url
  - Log in with admin credentials from test data
  - Verify dashboard loads and shows admin menu
  - Log out
priority: P0
tags: [smoke, login]

There's no selector, no explicit wait, no page object. The agent reads "log in with admin credentials from test data," looks at the actual rendered page, finds the fields that correspond to that intent, and does it — the same way a human tester would if you handed them that instruction and a browser.

That's the shift underneath everything else in this post: the test file describes what should happen, not how to make the browser do it.

Letting the agent decide what to run, and how​

One consequence of that shift is that the agent can reason about what actually needs testing rather than mechanically re-running everything.

DevAssure O2's change-aware mode reads a pull request diff, maps the affected user journeys and components — what DevAssure describes as tracing blast radius — checks what's already covered, and generates or runs tests scoped to what the change could plausibly have broken. Instead of a 45-minute full regression pass on every commit regardless of what changed, the suite that runs is the suite that's relevant to the diff in front of it. If a change touches the checkout flow, O2 tests checkout; it doesn't also re-verify the settings page nobody touched.

That is the same idea behind automatically testing every pull request: regression signal should track risk, not ritual.

The "how" side: self-healing execution​

When a locator drifts or a flow gets reordered, a script-based test throws an error and stops. An agent that's reasoning about the page contextually can recognize that the field it's looking for moved, or that a confirmation step was added, and adjust its path to complete the same intent — what DevAssure calls self-healing execution.

This is also, functionally, how the agent copes with UI changes and data changes without a human going back to rewrite anything: it isn't matching against a fixed map of the page, it's re-reading the page each time and matching against the described intent.

Caveat worth keeping: self-healing that silently rewrites assertions to match buggy behavior is a real failure mode elsewhere in agentic QA. Prefer systems where intent stays human-readable, results land on the PR for review, and "heal" means completing the stated journey — not quietly lowering the bar. For more on that tension, see why your coding agent can't be your testing agent and agents gaming their own tests.

Data that doesn't go stale​

Hardcoded test data is its own maintenance category — accounts that get locked out from reuse, seeded records that drift out of sync with the schema, birthdates that were valid on the day someone wrote the fixture and increasingly aren't.

The library tooling available to O2 test cases includes built-in generators (referenced in test steps and config as faker tools) for values like first names, last names, full names, emails, and phone numbers, so a test can request "a random email" or "a random full name" and get a fresh, collision-free value at execution time instead of a static fixture someone has to keep clean. Combined with persona definitions (an admin persona, a deactivated_user persona, and so on) and environment-scoped test data, a test can describe the kind of user and data it needs in plain language, and the specifics get filled in at runtime.

The same plain-language mechanism is what makes something like date-of-birth format validation tractable without a dedicated code snippet. Rather than writing a regex and a parser to confirm a DOB field enforces MM/DD/YYYY, rejects a user under a minimum age, or reformats correctly across locales, you write the constraint as a step — verify the field only accepts dates in the expected format, verify a user under the required age can't proceed — and the agent evaluates it against what's actually rendered on the page. The validation logic doesn't disappear; it just isn't something a test author has to hand-code and then keep synchronized with the application every time the format requirement changes.

Complex workflows without a mountain of automation code​

Traditional automation frameworks accumulate abstraction layers to stay maintainable: page objects, shared step libraries, custom wait helpers, retry wrappers. Each layer is a reasonable response to a real problem, and each layer is also more code that has to be understood, tested, and kept current.

O2's equivalent of that reuse is closer to composition than abstraction. An action like login_as_admin — a named, reusable sequence of plain-English steps — can be referenced by name inside any test case, and hooks can define setup and teardown that runs once around a whole session rather than being copy-pasted into every test. A multi-step workflow spanning login, a multi-page form, conditional branches like an MFA prompt, and a verification step reads as a short list of natural-language instructions rather than a script file with its own control flow, error handling, and helper functions to maintain.

That doesn't eliminate good regression design — it removes the tax of expressing that design in a second programming language on top of the product.

Using the regression suite for what it's actually for​

The point of all of this isn't novelty for its own sake — it's getting back to the original job of a regression suite: fast, trustworthy signal on whether a change broke something.

Teams that are deep in script-based automation tend to spend a real share of their QA capacity on a second job they didn't sign up for: nursing the suite itself — chasing down why a test failed when the product is fine, updating locators after a redesign, deciding whether a red build is real or "just flaky again." Every hour in that second job is an hour not spent on the first one: deciding what should be tested next, or looking at a genuine regression the suite actually caught.

Scoping execution to blast radius, letting the agent adapt to UI drift instead of failing on it, generating fresh data instead of maintaining fixtures, and describing complex workflows in the same language you'd use to explain them to a colleague — none of these individually eliminates the need for good test design and judgment about what matters to cover. What they remove is the tax of keeping the mechanics working, so that the time a team has for testing goes back to testing.

For teams already on GitHub Actions, that looks like a parallel job: keep a thin golden-path suite if compliance demands it, and let O2 own change-scoped regression in plain English on every PR.

The bottom line​

Regression testing was never supposed to be a career in selector archaeology. It was supposed to be a cheap, repeated answer to a single question. Separate intent from mechanics, scope runs to what changed, and the suite can do that job again — without becoming the product you're actually maintaining.

To answer one question cheaply and repeatedly: did this change break something that used to work? When teams spend more time keeping the suite alive than reading that signal, the suite has become a liability rather than a tool.

DevAssure​

External references​