Test Automation: A Practical Guide to Getting Started

Manual QA doesn't scale with a growing codebase. Every new feature adds another set of paths someone has to click through by hand, and every release means re-checking that nothing old broke. Test automation exists to break that tradeoff — but only if it's built the right way. Here's a practical, in-depth look at what test automation actually involves, where teams typically go wrong, how to think about tooling choices, and what's changed recently with AI-driven approaches.

What is test automation?​

Test automation is the practice of using software to execute tests, compare actual outcomes against expected results, and report the difference — without a human manually performing each step. It spans several layers of the stack, and understanding the differences matters because each layer catches different kinds of bugs and has a different cost profile.

Unit tests check individual functions or components in isolation, usually with dependencies mocked or stubbed out. They're fast — often running in milliseconds — and cheap to write and maintain. They're typically owned by the engineers writing the code, live alongside it in the same repository, and run on every commit.

Integration tests verify that multiple components or services work together correctly — an API endpoint that reads from a real database, a service that calls another internal service, a frontend component that fetches from a live API. They catch the class of bug that unit tests structurally can't: two pieces that each work fine alone but don't work together.

End-to-end (E2E) tests simulate real user behavior through the full application, usually via a real or headless browser: logging in, navigating between pages, submitting forms, checking that the right content renders after an action. E2E tests are the most realistic layer — they exercise the app the way an actual user would — but they're also the slowest to run and the most expensive to write and maintain.

Most mature testing strategies use a mix of all three, often visualized as a pyramid: many fast, cheap unit tests at the base, a moderate number of integration tests in the middle, and a smaller number of slower, higher-value E2E tests at the top. The shape of the pyramid matters — teams that invert it, relying heavily on E2E tests for coverage that unit tests could provide more cheaply, tend to end up with slow, flaky, expensive test suites. E2E tests are where most automation pain actually shows up, and where the rest of this guide is focused.

Why teams automate testing​

The core case for automation is straightforward: repetitive manual testing doesn't scale, and it's a poor use of a skilled tester's time. A few specific reasons teams invest in it:

Consistency. A human tester might skip a step when rushed, run steps out of order, or interpret a vague requirement differently on different days. A script runs the same steps, the same way, every single time.

Speed. An automated suite can run a full regression pass in minutes, in parallel across many browsers or environments — something no manual process can realistically match once an application reaches even moderate size.

Coverage that keeps pace with the codebase. As an application grows, the number of paths worth testing grows with it. Automation is the only way test coverage can keep pace without proportionally growing the size of the QA team.

Faster feedback loops. When tests run automatically on every pull request, a developer finds out about a regression in minutes, while the change is still fresh in their head — rather than days later during a manual QA pass, when the context has to be rebuilt from scratch.

Freeing people for judgment work. Automating the repetitive, well-defined checks frees testers to focus on exploratory testing, edge cases, accessibility, and usability — the parts of QA that genuinely benefit from human judgment and can't be easily scripted.

A record over time. An automated suite is also a living record of what the application is supposed to do. New team members can read the test suite to understand expected behavior, and a failing test after a change is an immediate signal that either the change or the test's assumption needs attention.

Where traditional automation gets hard​

If test automation were easy, every team would already have full coverage. In practice, most teams hit the same recurring set of problems, and it's worth naming them explicitly because they shape which tooling approach makes sense.

Brittle selectors. Traditional E2E frameworks (Selenium, Cypress, Playwright) locate elements on a page using CSS selectors or XPath expressions. Every UI change — a renamed CSS class, a restructured component tree, a redesigned button — risks breaking tests that have nothing conceptually to do with the actual change being made. This is probably the single most common source of wasted engineering time in E2E automation: fixing tests that broke for reasons unrelated to real regressions.

High authoring cost. Writing E2E tests in code requires engineering time and familiarity with the testing framework's API. That means QA-authored coverage is often bottlenecked on developer availability, and the backlog of "tests we should write" tends to grow faster than the backlog of tests that actually get written.

Maintenance burden compounds over time. A test suite isn't a one-time investment — it needs ongoing upkeep as the application changes. Flaky or frequently-broken tests erode the team's trust in the suite as a whole. Once trust erodes, failures start getting ignored or re-run until they pass rather than investigated, which quietly defeats the entire purpose of having the suite.

Slow onboarding for non-engineers. The people who often understand the product and user flows best — QA analysts, product managers — usually aren't the ones who can comfortably write and debug a Selenium or Playwright test in code. That knowledge gap means test coverage often reflects what engineers had time to write, not necessarily what matters most from a product or user perspective.

Environment and data setup. E2E tests typically need a running application, seeded test data, and often authenticated sessions — all of which need to be provisioned reliably in CI, which is its own layer of infrastructure work separate from the tests themselves.

Flakiness from timing. Because E2E tests interact with a real (or real-ish) browser and network, they're prone to timing issues — a test that checks for an element before it's finished rendering, or a network call that's slower than usual in CI. This is a different failure mode than brittleness, but it produces the same outcome: tests that fail for reasons unrelated to the code under test.

What's changing: AI-driven test automation​

A newer category of tools uses AI agents to execute tests described in plain language rather than code, interpreting steps like "log in with admin credentials" or "verify the dashboard shows the welcome message" and carrying them out in a real browser by reasoning about the page's content and layout rather than matching a hardcoded selector. This shifts several of the traditional tradeoffs:

• Tests tend to be more resilient to UI changes, because the agent is interpreting intent ("click the login button") rather than depending on a specific CSS class or DOM structure staying exactly the same.
• Non-engineers can author and read tests, since steps are written in natural language rather than framework-specific code.
• Test cases can live as simple, readable YAML or CSV files rather than a codebase that only engineers can maintain, which lowers the barrier for a whole team to contribute coverage.

It's worth being precise about what this does and doesn't solve: it primarily addresses the brittleness and authoring-cost problems above. Environment setup, test data management, and the discipline of deciding what's worth testing in the first place are still work the team has to do regardless of tooling.

DevAssure O2 and the @devassure/cli package are built around this natural-language approach. A test case is just a YAML file with a summary and a list of plain-English steps:

summary: Add item to cart and complete checkout
steps:
  - Open the application url
  - Search for a product and add it to the cart
  - Proceed to checkout and fill in shipping details
  - Complete the purchase and verify the confirmation page
priority: P1
tags:
  - checkout
  - regression

DevAssure's AI agent interprets and executes these steps in a browser, so the test doesn't need a hardcoded selector for every button or field. A few things worth knowing about how it fits into a real workflow:

Organizing tests. Tests can be tagged and prioritized (priority: P0, P1, etc.), then run selectively — for example, running only smoke-tagged, P0 tests on every commit, and a broader set nightly:

devassure run --tag=regression --priority=P0,P1

Planning against a diff. Rather than always running the entire suite, devassure plan and devassure test can scope test selection to what actually changed between a branch and a base (--head, --base, or --commit), which keeps feedback fast as a suite grows.

CI/CD integration. DevAssure can be wired into GitHub Actions, GitLab CI, or CircleCI, typically added as a step after your build or preview-deploy step, so it's testing the actual deployed artifact:

devassure add-token "$DEVASSURE_TOKEN"
devassure test --url=$PREVIEW_URL --archive=./test-reports
devassure summary --last --json

Test data and personas. Config files (test_data.yaml, personas.yaml) separate environment-specific data (URLs, credentials) from the test steps themselves, so the same test can run against local, staging, or production-like environments by switching --environment.

Reporting. After a run, devassure open-report --last opens a full HTML report in your browser, and devassure summary --last --json gives a machine-readable summary for CI systems to parse.

Choosing a tooling approach​

There's no single right answer here — the right choice depends on your team's makeup and existing investment. A few honest tradeoffs to weigh:

If you already have a mature Selenium, Cypress, or Playwright suite with engineers actively maintaining it, ripping it out is rarely worth the disruption — the fixed cost is sunk, and the team already has the expertise. AI-driven tools tend to make the most sense when you're starting close to zero, when your existing suite has become expensive to maintain, or when you specifically want non-engineers to be able to contribute test coverage.

Code-based frameworks generally give you finer-grained control — direct access to network mocking, custom assertions, and tight integration with your codebase's testing utilities. Natural-language, AI-driven tools trade some of that control for lower authoring cost and better resilience to UI churn. Which tradeoff matters more depends on how frequently your UI changes relative to how often you need very specific, low-level test control.

Many teams run both: keep unit and some integration tests in code, and use an agent for the outer E2E / acceptance layer where selector churn hurts most — similar to the split in acceptance TDD with O2.

Getting started with test automation​

If you're introducing automation to a codebase or team that doesn't have much yet, a practical sequence that avoids the common early mistakes:

1. Start with smoke tests. Cover login and your one or two most critical workflows first, and get them running reliably on every build before expanding further. See our companion guide on smoke testing for what a minimal, high-value smoke suite looks like in practice.
2. Automate the most repeated manual tests first. If your QA team runs the same checklist before every release, that checklist is your highest-leverage automation target — it's already proven to be worth doing repeatedly.
3. Wire it into CI early, even with a small suite. A five-test suite that runs automatically on every pull request delivers more real value than a two-hundred-test suite that only gets run manually once a month.
4. Grow coverage incrementally, driven by real bugs. Add tests as you find bugs or as new critical paths emerge in the product, rather than trying to reach comprehensive coverage on day one — that approach usually stalls before it produces anything usable.
5. Watch the signal-to-noise ratio. If a suite is generating more false failures than real ones, treat that as an urgent problem to fix before expanding the suite further — a suite nobody trusts is worse than no suite at all, because it creates a false sense of safety.
6. Assign ownership. Test suites that don't have a clear owner tend to degrade quietly until someone notices they've stopped catching anything real. Decide early who's responsible for keeping the suite healthy.

The bottom line​

Test automation isn't about replacing manual QA — it's about removing the repetitive, well-defined parts of testing so people can focus on the parts that actually require judgment: exploratory testing, usability, and the edge cases a script wouldn't think to check. Where automation has historically gone wrong has usually been brittle, code-heavy tooling that costs more to maintain than it saves. Natural-language, AI-driven tools like DevAssure are a newer answer to that specific problem, lowering both the authoring cost and the maintenance burden. But the fundamentals hold regardless of which tooling you choose: start small, automate the highest-leverage checks first, run them on every build, and keep a close eye on whether the suite is still earning the team's trust.

Links​

DevAssure​

• DevAssure O2: https://www.devassure.io/o2-testing-agent
• Smoke test: what it is and how to automate it: https://www.devassure.io/blog/smoke-test-what-it-is/
• Regression testing was never supposed to be this much work: https://www.devassure.io/blog/regression-testing-never-supposed-to-be-this-much-work/
• Test automation with AI agents: https://www.devassure.io/blog/test-automation-with-ai-agents/
• TDD with DevAssure O2: https://www.devassure.io/blog/tdd-with-devassure-o2/
• GitHub Actions: https://www.devassure.io/blog/github-actions/
• O2 on GitHub Marketplace: https://www.devassure.io/blog/devassure-o2-agent-github-marketplace/
• CLI get started: https://www.devassure.io/docs/DevAssure/Invisible%20Agent/CLI/overview/
• @devassure/cli on npm: https://www.npmjs.com/package/@devassure/cli
• DevAssure: https://www.devassure.io