Skip to main content

Building an in-house test execution grid with Playwright and GitHub Actions

Divya Manohar
Divya Manohar
Co-Founder and CEO, DevAssure

Every team that gets serious about end-to-end testing eventually builds the same thing: a pipeline that takes a pull request, spins up browsers somewhere, runs the suite, and reports back. In 2026 the default stack for this is Playwright on GitHub Actions - no Selenium Grid hub/node topology to babysit, because Playwright's own worker model plus CI-level sharding is the grid.

This guide builds that setup end to end:

Local project to CI workflow, sharded parallel execution, self-hosted runners, then scheduled regression jobs

Then we do the part most tutorials skip: what happens at 1,000+ tests, what the infrastructure actually costs in dollars and engineer-hours, and why flakiness is an environment problem before it's a test problem.

Part 1: The setup​

Step 1 - Scaffold the project​

npm init playwright@latest

The installer asks for TypeScript vs JavaScript, a test directory, and whether to add a GitHub Actions workflow - say yes to that last one; we'll replace it, but it's a useful skeleton. Then install browsers with system dependencies:

npx playwright install --with-deps

--with-deps matters on Linux: it pulls the OS-level libraries (fonts, codecs, X11 bits) that browsers need. Forgetting it is the single most common "works on my Mac, dies in CI" failure. 😂

Step 2 - A CI-ready playwright.config.ts​

The default config is tuned for local development. CI needs different behavior:

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,        // fail CI if someone left test.only
  retries: process.env.CI ? 2 : 0,     // retry in CI, never locally
  workers: process.env.CI ? '100%' : undefined,
  reporter: process.env.CI
    ? [['blob'], ['github']]           // blob = mergeable shard reports
    : [['html']],
  use: {
    baseURL: process.env.BASE_URL ?? 'http://localhost:3000',
    trace: 'on-first-retry',           // full trace only when a retry happens
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
    { name: 'firefox',  use: { ...devices['Desktop Firefox'] } },
    { name: 'webkit',   use: { ...devices['Desktop Safari'] } },
  ],
});

Three decisions here that pay off later:

  • retries: 2 in CI only. Retries are a flakiness tourniquet, not a cure - but without them, one flaky test blocks every merge. Locally you want zero retries so flakiness stays visible.
  • trace: 'on-first-retry'. Traces are Playwright's best debugging artifact and its heaviest one. Capturing on every run at scale generates gigabytes per day; on-first-retry captures exactly the runs you'll investigate.
  • The blob reporter. This is what makes sharding work - each shard emits a mergeable blob, and a final job stitches them into one HTML report.

Step 3 - The basic workflow​

.github/workflows/e2e.yml, single job, runs the whole suite on every PR:

name: E2E tests
on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 14

Set timeout-minutes explicitly. The default job timeout is 6 hours, and a hung browser process will happily bill you for all of them.

This works until roughly 50–100 tests. Then a full run crosses the 10-minute mark, developers start merging before green, and you need parallelism.

Step 4 - Sharding: the part that makes it a grid​

Playwright splits a suite across machines natively with --shard. Combined with a GitHub Actions matrix, each shard becomes its own runner VM:

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --shard=${{ matrix.shard }}/4
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report
          retention-days: 1

  merge-reports:
    if: ${{ !cancelled() }}
    needs: [test]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - uses: actions/download-artifact@v4
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true
      - run: npx playwright merge-reports --reporter html ./all-blob-reports
      - uses: actions/upload-artifact@v4
        with:
          name: html-report
          path: playwright-report
          retention-days: 14

fail-fast: false is essential - the default cancels all shards when one fails, which destroys exactly the signal you sharded for.

Note what sharding does to billing: four shards each spend ~2 minutes on checkout, npm ci, and browser install before running a single test. That per-shard tax is why "just add more shards" has diminishing - and eventually negative - returns.

Optimization: skip browser installs with the official container. Run jobs in Microsoft's Playwright image and browsers are pre-baked:

    container:
      image: mcr.microsoft.com/playwright:v1.62.0-noble

Step 5 - Scheduled regression runs​

PR-triggered runs catch what the PR broke.

Scheduled runs catch what everything else broke - dependency drift, backend deploys, expiring certs.

This is the same job regression suites were supposed to do, before the suite itself became the work:

on:
  schedule:
    - cron: '30 3 * * *'   # 03:30 UTC = 09:00 IST nightly
  workflow_dispatch:        # manual trigger for on-demand full runs

Step 6 - Self-hosted runners: when GitHub's machines stop being enough​

Teams move to self-hosted runners for three reasons: the app under test lives on a private network; the suite needs more CPU than hosted runners offer at acceptable cost; or the minutes bill got scary. Registration is genuinely quick:

mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-x64.tar.gz -L \
  https://github.com/actions/runner/releases/latest/download/actions-runner-linux-x64-<version>.tar.gz
tar xzf actions-runner-linux-x64.tar.gz
./config.sh --url https://github.com/<org>/<repo> --token <REG_TOKEN> --labels e2e
sudo ./svc.sh install && sudo ./svc.sh start

Then target it: runs-on: [self-hosted, e2e].

That's the demo. Production is a different animal:

Ephemeral runners are non-negotiable for browser tests. A persistent runner accumulates state - browser profiles, leftover processes, disk-filling videos - and state is flakiness. Register with --ephemeral so each runner takes one job and dies, which means you need something respawning them.

That something is an autoscaling layer. The standard answer is ARC (Actions Runner Controller) on Kubernetes, or homegrown EC2 auto-scaling groups with lifecycle hooks. Either way you've just adopted a distributed system: runner images to rebuild when Chrome updates, scale-up latency to tune, orphaned instances to reap, registration tokens to rotate.

Security scope creep. Self-hosted runners execute arbitrary workflow code on your network. GitHub's own guidance is to never use them on public repos, and on private ones you still want them isolated, unprivileged, and short-lived.

At this point, notice what happened: you set out to run tests and now you operate a fleet.

Part 2: The same pipeline everywhere else (briefly)​

The architecture above is portable; only the YAML dialect changes.

Jenkins - maximum control, maximum ownership. You run the controller, the agents, the plugin tree, and the upgrades. Browser-test agents are the same Docker/ephemeral story as above, minus GitHub's managed control plane. Still common where compliance requires everything on-prem.

CircleCI - managed runners with first-class test splitting (circleci tests split --split-by=timings balances shards by historical duration, which is genuinely better than Playwright's file-count split). You trade GitHub-native integration for it.

GitLab CI - same shape via parallel: and matrix: keywords; GitLab Runners are the self-hosted equivalent. If your code is in GitLab, everything here translates almost line for line.

Buildkite - hybrid model: their control plane, your compute, by design. Popular with teams that were going to run self-hosted anyway and want better orchestration than raw ARC.

Different logos, same physics: browsers need machines, machines need images, images need updating, and parallelism multiplies all three. That is the same hidden cost of building test automation that shows up whether the framework is Playwright or Selenium.

Part 3: The bill arrives​

Everything above works. Teams run it in production every day. Here's what the guide-shaped tutorials don't model: the curve as the suite grows.

The math at thousands of tests​

A browser test averages 30–90 seconds of wall time. Call it 45s. The arithmetic is unforgiving:

Suite sizeTest-minutes per full runWall time @ 10 shardsWall time @ 20 shards
200 tests150~17 min~10 min
1,000 tests750~77 min~40 min
5,000 tests3,750~6.3 hrs~3.2 hrs

(Wall time includes ~2 min setup tax per shard. These are estimates for planning, not benchmarks - your per-test average is the number to measure first.)

Billable minutes don't shard. Wall-clock drops. The invoice doesn't.

  • 20 shards finishing in 40 minutes still bill ~800 machine-minutes.
  • GitHub-hosted Linux rate ($0.006/min), a 1,000-test full run is about $4.80. Trivial, until you multiply.
  • A 15-engineer team at approx 100 full runs a day (30 PR runs + nightlies across three environments) is ~$480/day, ~$10K+/month in raw minutes.
  • Traces and videos add artifact storage at $0.008/GB/day. A failure-heavy week generates a lot of gigabytes.

The usual fix — subset on PRs, full suite nightly — is rational. It is also how bugs start shipping between nightlies.

Concurrency is the wall you hit before cost.

  • Free plan: 20 concurrent jobs. Paid tiers go higher; check the current cap.
  • At 20 shards per run, two simultaneous PRs saturate a Free-tier org. A third queues.
  • That's the 5 PM effect: 12 minutes at 11 AM becomes 50 minutes when everyone merges before standup or EOD.
  • Developers respond by trusting CI less and merging on red more.

The human cost​

This is the line item that kills the ROI, and it's the one nobody budgets. An in-house grid at scale generates a permanent stream of work:

  • Browser churn. Chrome ships a new stable roughly every four weeks. Each Playwright release pins new browser builds; each upgrade means bumping the npm package, the Docker image tag, and the runner image together, then triaging whatever visual or timing differences the new build introduces.
  • Image and runner drift. ubuntu-latest migrates under you (22.04 → 24.04 broke plenty of pipelines). Self-hosted images need the same treatment on your schedule, forever.
  • Actions maintenance. Version bumps for every action in the workflow, breaking changes in artifact handling (the v3 → v4 artifact migration was not painless), token and secret rotation.
  • The nightly babysitter. Someone reads the failure report every morning and decides: real bug, flaky test, or environment hiccup? At 1,000+ tests even a 1% flake rate means ~10 red tests every night that a human must adjudicate.

In my experience - and this is an estimate from having built these systems, not a study I can cite - a mid-size engineering org spends somewhere between a quarter and half of a full-time engineer on grid upkeep once the suite passes 1,000 tests. At loaded cost, that dwarfs the compute line. It's also, universally, work nobody wants: the engineer maintaining the test grid is not shipping product, and they know it.

This is the same maintenance trap as hand-written Playwright tests themselves: the suite starts as coverage and becomes a second product.

The flakiness tax​

Here's the mechanism that makes environment maintenance and flaky tests the same problem: a test that is deterministic in a clean environment becomes probabilistic in a drifting one.

  • Hosted runners are shared multi-tenant VMs with variable CPU - the same animation that finishes in 300ms at 11 AM takes 900ms under noisy neighbors, and your 500ms assumption flakes.
  • A persistent self-hosted runner leaks state: a previous job's browser process holding a port, a full disk from unpruned videos, a stale auth cookie in a profile that shouldn't exist.
  • A browser version mismatch between the npm package and the runner image changes font rendering by two pixels and every screenshot assertion goes red.
  • Parallel shards hit the same staging backend and trip each other's data - the test didn't change, the neighbor did.

Teams respond with retries, then with generous timeouts, then with quarantine lists - each step slowing the suite and eroding the one thing a test suite exists to provide: a signal you trust. The endgame everyone recognizes: CI goes red, and the first hypothesis isn't "we broke something," it's "the grid is being the grid." Once that's the default read, the suite has stopped doing its job regardless of what it costs.

Part 4: What we built instead​

Everything above is why we built O2 Cloud Agent the way we did. Not because the DIY grid doesn't work - it demonstrably does, and if you have the platform team and the compliance requirement to own it, this guide is a fine blueprint. But the failure mode is always the same: the grid starts as a week of setup and becomes a permanent engineering sub-project whose output is maintaining the ability to test rather than testing.

O2 Cloud Agent deletes the sub-project:

  • Tests are plain-English steps, not Playwright code - so there's no framework version to chase, no selectors to repair when the DOM shifts.
  • Execution happens in real browsers on infrastructure we run. No runner images, no browser pinning, no ARC cluster. The environment is clean every run, which removes the entire class of drift-induced flakiness above.
  • Parallelism is a number field, not a matrix strategy plus a concurrency plan. More agents, more parallel runs, isolated sessions.
  • Scheduling is built in - nightly, post-deploy, or triggered from the pipeline you already have - with every run scored and stored, so the morning triage is reading results, not diagnosing infrastructure.
  • Pricing is flat: $25 per agent per month, unlimited executions. No per-minute meter, which means no one ever again decides not to run the suite because of what it costs.

The honest comparison isn't compute vs. compute - self-hosted VMs will win that line on a spreadsheet. It's the total: compute, plus the fraction of an engineer the grid consumes, plus the flakiness tax on everyone's trust in CI. That total is the tax nobody signed up for.

A single-job workflow is enough until roughly 50–100 tests. Once a full run crosses about 10 minutes, developers start merging before green and you need Playwright --shard plus a GitHub Actions matrix so each shard is its own runner VM.

External references​