CircleCI Orb

Overview

DevAssure O2 is an AI-powered end-to-end (E2E) testing agent that integrates directly into CircleCI pipelines via a reusable orb. It reads code diffs, automatically generates targeted UI tests, executes them on real browsers, and reports results back to the pipeline — with no test scripts to write or maintain.

Core workflow: reads code diff → maps blast radius → generates tests → executes → reports

The orb wraps the DevAssure CLI (@devassure/cli) and works on branch pipelines, pull request pipelines, and any other CircleCI job trigger.

Orb registry: devassure/devassure

Registry URL: https://circleci.com/developer/orbs/orb/devassure/devassure

How It Works

When a CircleCI pipeline runs:

The orb installs @devassure/cli and validates the Node.js version.
The DevAssure O2 agent reads the code diff from the branch or commit.
It identifies the impacted areas (blast radius) of the changes.
It generates comprehensive end-to-end UI tests in natural language to validate those changes.
It executes the generated tests in real browsers.
It surfaces bugs and test results in the pipeline, with downloadable reports and optional test result artifacts.

Prerequisites

Runners: Unix-based CircleCI executors only (Linux Docker or machine). Windows executors are not supported.
Node.js: Version 20 or above. The orb validates this automatically.
Git: A proper checkout with sufficient history for branch diffs to resolve. Use fetch-depth: 0 or equivalent when testing across branches.
DevAssure API Token: Required for authentication. See Authentication below.
Uncertified Orbs: The DevAssure orb is a community orb. To use it, enable Allow Uncertified Orbs in your organization's Settings → Security tab.

Authentication

Creating a DevAssure API Token

Log in at https://app.devassure.io or sign up for free at https://app.devassure.io/sign_up.
Navigate to Account → Settings → Token.
Generate a new API token.
Store it as a CircleCI environment variable or context variable named DEVASSURE_TOKEN.

Configuring the Token in CircleCI

Recommended approach — store the token in a CircleCI Context and reference it in your workflow:

workflows:
  e2e:
    jobs:
      - devassure/run:
          context: devassure-context   # Context containing DEVASSURE_TOKEN

Alternatively, store it as a Project Environment Variable in Project Settings → Environment Variables. The orb resolves DEVASSURE_TOKEN automatically from the environment.

Security note: Never hardcode the token in .circleci/config.yml. Always use a CircleCI Context or project-level environment variable.

Installation & Basic Setup

Step 1: Enable Uncertified Orbs

In your CircleCI organization, go to Settings → Security and enable Allow Uncertified Orbs.

Step 2: Import the Orb

Add the orb to your .circleci/config.yml:

version: 2.1

orbs:
  devassure: devassure/devassure@1

Pin to a specific major version (e.g., @1) for stability. Avoid @volatile.

Step 3: Add the Job to a Workflow

version: 2.1

orbs:
  devassure: devassure/devassure@1

workflows:
  e2e:
    jobs:
      - devassure/run:
          context: devassure-context

Commands & Jobs

The orb exposes a primary job — devassure/run — which accepts a command parameter to select the DevAssure operation. The default command is test.

Available Commands

Command	Description
`setup`	Installs and validates the DevAssure CLI
`test`	Generates and runs E2E tests from a branch or commit diff
`run`	Executes existing tests with optional filters
`summary`	Retrieves a test session summary
`archive` / `archive-report`	Archives the test report for a session

Usage Examples

Default — Test on Every Push

version: 2.1

orbs:
  devassure: devassure/devassure@1

workflows:
  e2e:
    jobs:
      - devassure/run:
          context: devassure-context

`test` — Generate and Run Tests from a Diff

version: 2.1

orbs:
  devassure: devassure/devassure@1

workflows:
  e2e:
    jobs:
      - devassure/run:
          context: devassure-context
          command: test
          minimum_score: 80
          workers: 2
          path: e2e/tests
          source: << pipeline.git.branch >>
          target: main
          commit_id: << pipeline.git.revision >>
          url: https://example.com
          environment: staging
          headless: "true"

`run` — Execute Existing Tests with Filters

version: 2.1

orbs:
  devassure: devassure/devassure@1

workflows:
  e2e:
    jobs:
      - devassure/run:
          context: devassure-context
          command: run
          workers: 2
          path: e2e
          filter: smoke
          query: "login flow"
          tag: nightly
          priority: high
          folder: reports
          url: https://example.com
          environment: staging
          headless: "false"

`summary` — Retrieve Session Summary

- devassure/run:
    context: devassure-context
    command: summary
    session_id: sess_123

If session_id is not provided, the orb runs devassure summary --last.

`archive` — Archive a Test Report

- devassure/run:
    context: devassure-context
    command: archive
    session_id: sess_123

If session_id is not provided, the orb runs devassure archive-report --last.

Pass Token Directly via Parameter

- devassure/run:
    token: $DEVASSURE_TOKEN
    command: setup

Prefer environment variables or contexts over inline token parameters. This approach is acceptable only when your policy explicitly allows it.

Parameters Reference

All parameters are optional.

Parameter	Default	Description
`command`	`test`	DevAssure command: `setup`, `test`, `run`, `summary`, `archive`, or `archive-report`
`token`	(empty)	DevAssure API token. Falls back to `DEVASSURE_TOKEN` environment variable if empty
`path`	(empty)	Relative path to run from (useful when tests live in a subdirectory)
`source`	(empty)	Source branch for `test` command scope and branch checkout
`target`	(empty)	Target branch for `test` command baseline comparison
`commit_id`	(empty)	Commit SHA for `test` command scope
`filter`	(empty)	Filter expression for narrowing `run` command execution
`query`	(empty)	Query string for selecting tests in `run`
`tag`	(empty)	Tag-based selector for `run`
`priority`	(empty)	Priority-based selector for `run`
`folder`	(empty)	Folder selector for `run`
`url`	(empty)	Application URL under test (used by `test` and `run`)
`headless`	`true`	Headless browser mode for `test` and `run` (`"true"` or `"false"`)
`session_id`	(empty)	Session ID for `summary` or `archive` (defaults to latest session)
`archive`	`true`	Set to `false` to skip `archive-report --last` and artifact archiving for `test`/`run`
`minimum_score`	`75`	Minimum passing score threshold for `test`/`run`. Set to `0` to disable
`workers`	(empty)	Number of parallel workers (integer > 0)
`environment`	(empty)	Environment name passed to `test`/`run` (e.g., `staging`, `qa`, `production`)
`verbose`	(empty)	Enable verbose logging (`true` / `false`)
`debug`	(empty)	Enable debug logging (`true` / `false`)

Command Parameter Mapping

The orb forwards supported parameters to the CLI using --<arg-name>="<arg-value>" format.

Command	Forwarded Parameters
`setup`	(none) — optional `verbose` / `debug` flags
`test`	`path`, `source`, `target`, `commit_id`, `url`, `workers`, `environment`, `headless`
`run`	`path`, `filter`, `query`, `tag`, `priority`, `folder`, `url`, `workers`, `environment`, `headless`
`summary`	`session_id` (when set), otherwise `--last`
`archive` / `archive-report`	`session_id` (when set), otherwise `--last`

Note: headless always defaults to true and is always passed for test and run. session_id and --last are mutually exclusive — the orb ensures only one is passed.

Score Validation

For test and run commands, the orb automatically runs devassure summary --last after execution and validates the score against minimum_score.

Score validation behavior:

If minimum_score is non-numeric or <= 0 → validation is skipped.
If the score is missing or N/A → the job fails.
If the score is below minimum_score → the job fails.
If the score meets or exceeds minimum_score → the job passes.

To disable score validation, set minimum_score: 0.

Report Archiving & Artifacts

For test and run commands, report archiving is enabled by default (archive: true). The orb archives the report and optionally stores it as a CircleCI artifact for download.

Archived reports can be opened using:

DevAssure CLI (@devassure/cli)
DevAssure VSCode Extension

To skip archiving, set archive: false.

Using CircleCI Pipeline Variables

CircleCI provides built-in pipeline values that map naturally to DevAssure parameters:

DevAssure Parameter	CircleCI Pipeline Value
`source`	`<< pipeline.git.branch >>`
`commit_id`	`<< pipeline.git.revision >>`
`target`	Hardcode your base branch (e.g., `main`)

Example:

- devassure/run:
    context: devassure-context
    command: test
    source: << pipeline.git.branch >>
    target: main
    commit_id: << pipeline.git.revision >>
    url: https://example.com
    environment: staging

Runner Sizing Recommendations

Workers / Concurrency	Recommended Spec
Baseline (default)	4 vCPU / 16 GB RAM
Higher concurrency	Scale vCPU and RAM proportionally to avoid contention

Increasing workers without scaling the executor will cause performance degradation and potential test failures due to resource contention. Use resource_class: large or higher in your executor configuration when running parallel browser sessions.

Security & Best Practices

Store DEVASSURE_TOKEN in a CircleCI Context with security groups restricted to approved users — never hardcode it in config files.
Pin the orb to a specific major version (e.g., @1). Avoid @volatile in production pipelines.
Keep executor resources aligned with workers count (see Runner Sizing Recommendations).
Use security groups on CircleCI Contexts to limit which jobs and users can access the token.

Local Execution

The DevAssure O2 agent can also be invoked locally using the DevAssure CLI npm package:

npm install -g @devassure/cli
devassure test --url https://your-app.com

See the DevAssure CLI documentation for full local usage details.

FAQ

How does `minimum_score` work?

minimum_score is the lowest test score that still passes the job. The default is 75. If the score falls below this value, the CircleCI job fails; if it meets or exceeds the threshold, the job passes. Set minimum_score: 0 to disable score validation entirely.

How are credits consumed?

Credits depend on browser interactions and test complexity. Per-run usage is visible in the DevAssure portal at app.devassure.io/usage.

How do I view the complete test report?

Download the job artifacts from the CircleCI pipeline run and open the archived report using the DevAssure CLI or the DevAssure VSCode extension.

Can I run the agent locally?

Yes. Install @devassure/cli via npm and run devassure commands directly from your local machine. This is useful for testing before pushing to CI.

Can I use `session_id` with the `summary` command?

Yes. Pass a specific session_id to retrieve that session's summary. If omitted, the orb defaults to devassure summary --last (most recent session). The --last flag and session_id are mutually exclusive.

Why is "Allow Uncertified Orbs" required?

The DevAssure orb is a community-published orb. CircleCI requires organizations to explicitly opt in to using orbs not certified by CircleCI. This setting is at Organization Settings → Security → Allow Uncertified Orbs.

Overview​

How It Works​

Prerequisites​

Authentication​

Creating a DevAssure API Token​

Configuring the Token in CircleCI​

Installation & Basic Setup​

Step 1: Enable Uncertified Orbs​

Step 2: Import the Orb​

Step 3: Add the Job to a Workflow​

Commands & Jobs​

Available Commands​

Usage Examples​

Default — Test on Every Push​

test — Generate and Run Tests from a Diff​

run — Execute Existing Tests with Filters​

summary — Retrieve Session Summary​

archive — Archive a Test Report​

Pass Token Directly via Parameter​

Parameters Reference​

Command Parameter Mapping​

Score Validation​

Report Archiving & Artifacts​

Using CircleCI Pipeline Variables​

Runner Sizing Recommendations​

Security & Best Practices​

Local Execution​

FAQ​

How does minimum_score work?​

How are credits consumed?​

How do I view the complete test report?​

Can I run the agent locally?​

Can I use session_id with the summary command?​

Why is "Allow Uncertified Orbs" required?​

Related Resources​