Skip to main content

Smoke Test: What It Is, Why It Matters, and How to Automate It

Divya Manohar
Divya Manohar
Co-Founder and CEO, DevAssure

TL;DR

A smoke test is a short, shallow check that a new build is usable — load, login, core paths — so you fail in minutes, not after a full regression. Keep it tagged, run it first in CI, and automate it with plain-English cases in DevAssure O2 instead of brittle selectors.

If a build breaks in an obvious way — the login page won't load, the checkout flow throws a 500, the dashboard is a blank white screen — you want to know in minutes, not after a full regression suite finishes hours later. That's the job of a smoke test.

What is a smoke test?​

A smoke test is a quick, shallow check run against a new build to confirm the application starts up and its most critical paths work at all. The term comes from hardware testing: you power on a circuit and check whether it literally starts smoking. In software, "smoke" means the same thing figuratively — you're looking for the build to fail loudly and immediately if something fundamental is broken, before anyone invests time in deeper testing.

Smoke tests are not exhaustive. They don't check edge cases, error states, or every permutation of a feature. A typical smoke suite for a web app might cover:

  • The app loads without errors
  • A user can log in
  • The main navigation or dashboard renders
  • One or two core workflows complete end to end (e.g., adding an item to a cart, submitting a form)

If any of these fail, the build is considered broken and the team stops — there's no point running a full regression suite against a build that can't even log a user in.

Smoke testing vs. regression testing vs. sanity testing​

These three terms get confused often enough that it's worth being precise:

Smoke testingSanity testingRegression testing
BreadthBroadNarrowBroad
DepthShallowDeeper in one areaDeep
WhenEvery new build / PRAfter a specific fix or small changeBefore release or on a nightly schedule
GoalIs this build usable at all?Does this fix work as expected?Did we break existing behavior?

Smoke testing is broad but shallow — a small number of checks across the whole application, run on every new build, meant to catch build-breaking issues fast.

Sanity testing is narrow but deeper — after a specific bug fix or small change, you verify that the fixed area works correctly, without re-testing everything else.

Regression testing is broad and deep — a comprehensive suite that re-verifies existing functionality still works after changes, typically run less frequently than smoke tests because it takes longer.

A common pattern: smoke tests run on every commit or pull request, sanity tests run after specific fixes, and full regression suites run before a release or on a nightly schedule.

Why smoke testing matters​

The value of a smoke test is speed of feedback. Catching a broken build in 5 minutes at the top of a CI pipeline is far cheaper than discovering it after a QA engineer has spent an hour manually clicking through a feature that was never reachable in the first place. A few concrete benefits:

Fast failure. Smoke tests are designed to be quick — often just a handful of checks — so they can run on every build without slowing down the pipeline.

Protects downstream testing. If your CI pipeline runs smoke tests before triggering a longer regression suite, you avoid burning compute and engineer attention on a build that was never viable to begin with.

Low maintenance overhead. Because smoke tests only touch the most stable, most critical paths in an application, they tend to break less often than deep, detail-oriented regression tests — which means less time spent fixing flaky tests.

A shared safety net. A well-defined smoke suite gives the whole team — engineers, QA, product — a shared, fast signal for "is this build even usable," independent of who is actively testing at that moment.

What makes a good smoke test suite​

A smoke suite that tries to do too much stops being a smoke suite. Some practical guidelines:

  1. Keep it short. If your smoke suite takes as long as your regression suite, it's not doing its job. Aim for minutes, not tens of minutes.
  2. Cover critical paths only. Login, core navigation, and the one or two workflows your business can't function without (checkout, search, submitting a support ticket, whatever is central to your product).
  3. Run it on every build. Smoke tests are only valuable if they run consistently — ideally as the first stage of CI, before more expensive test suites kick off.
  4. Tag it clearly. If you maintain a larger test suite, tag your smoke tests explicitly (e.g., a smoke tag) so they can be run in isolation from full regression runs.
  5. Keep it stable. A flaky smoke test that fails intermittently for reasons unrelated to the build erodes trust in the whole gate. Prioritize reliability over coverage for this particular suite.

Automating smoke tests with DevAssure​

Writing and maintaining smoke tests by hand — especially UI-level ones — is exactly the kind of repetitive work that's worth automating. DevAssure O2 lets you describe smoke tests in plain-English steps inside a YAML file, and an AI agent executes them in a real browser, so you're not maintaining brittle CSS or XPath selectors as your UI changes.

A minimal smoke test in DevAssure might look like:

summary: Login and verify dashboard loads
steps:
  - Open the application url
  - Log in with admin credentials from test data
  - Verify dashboard loads and shows admin menu
priority: P0
tags:
  - smoke

Because DevAssure supports tagging, you can group all of your smoke-level checks under a smoke tag and run just that subset on every commit:

devassure run --tag=smoke --priority=P0

That gives you the fast-feedback loop a smoke suite is supposed to provide, without hand-maintaining selectors or a separate automation framework. Wire the same command into GitHub Actions, GitLab CI, or CircleCI as the first gate in your pipeline — for example with @devassure/cli and a token:

devassure add-token "$DEVASSURE_TOKEN"
devassure run --tag=smoke --priority=P0 --archive=./test-reports
devassure summary --last --json

A broken build fails fast before anything more expensive runs. For PR-scoped blast-radius testing beyond a fixed smoke tag, see how O2 plans from a git diff and the GitHub Marketplace action.

The bottom line​

Smoke testing isn't about catching every bug — it's about catching the build-breaking ones immediately, so your team doesn't waste time testing something that was never usable to begin with. Keep the suite small, run it on every build, and if you're doing it by hand today, it's one of the easiest places to start with test automation.

A smoke test is a quick, shallow check against a new build to confirm the app starts and its most critical paths work at all — for example load, login, main navigation, and one or two core workflows. It is meant to fail loudly and fast if something fundamental is broken, before deeper testing begins.

DevAssure​