End-to-end tests that read like a QA checklist.

What the agent does with that spec#

Screenshots from the run below, against the small demo app in tests/fixtures/app.

1. 

   1Sign in once

   agent-browser labels every control. The agent types the username and fills the password with fill_secret, so it never sees it. The session is saved and the next run starts signed in.

2. 

   2Reach the goal

   For goal('add a todo …') the agent fills the input and presses Add. qaspec checks that POST /api/todos returned 201 and that no console error fired during the step.

3. 

   3Catch what looks fine

   This page renders without visible problems, but its script throws and its API returns 500. A person clicking through would miss it. expect.console.noErrors() fails the step.

The run, and its report#

Real output, one browser session for three suites.

qaspec, regular E2E and agentic E2E#

Regular E2E is a scripted Playwright or Cypress suite. Agentic E2E is e2e by TesterArmy, where an agent acts and asserts but never sees the console or network. The rows where qaspec loses are in the table too: judged checks always call a model, and native mobile apps are out of scope.

Notes under some marks explain them; on a phone they are in the detailed tables below. Based on the e2e 0.16 docs.

In detail#

The same comparison with the details, plus agent-browser on its own. qaspec sits between scripted tests and an agent you drive by hand. It borrows the agent loop from e2e by TesterArmy, gives the model the console and network that e2e hides, and leaves the browser to agent-browser.

Writing tests

What a check can see

Running them

Based on e2e 0.16 and agent-browser 0.27 docs, and on porting a real two-app suite. Scripted E2E means Playwright or Cypress. Corrections welcome as issues.

New since 0.1.0

On main, not in a release yet. Build from source to try it.

Replay cache

A goal that passed is recorded as role and name locators. The next run replays it without the model, and the agent takes over only where the page changed. In the demo, a second run went from 8 model calls to 4, and from 24 to 18 seconds. --cache strict fails on a missing or stale recording, for CI.

Parallel workers

--jobs N splits suites across N browsers, keeping each sign-in, dependency and capture in one worker. A memory guard lowers N to what fits at about 1.5 GB per browser. One browser stays the default.

Mobile web

device: 'iPhone 14' on a suite, project or the whole run emulates the phone's size, pixel ratio and user agent, and tells the agent to look for menus and scroll. Suites with a device run last, because agent-browser cannot undo the mobile user agent without relaunching.

Install#

The only runtime dependency is agent-browser. Prebuilt binaries for Linux and macOS are on the release page.

$ npm i -g agent-browser && agent-browser install
$ cargo install --git https://github.com/jaivial/qaspec --tag v0.1.0
$ qaspec init     # qaspec.toml and specs/home.qa.ts
$ qaspec check    # specs, config and secrets, no browser
$ qaspec run

Projects, environments and credentials#

Base URLs, users and secrets live in qaspec.toml. Specs only name them. The examples/shop project is a store and its back office:

import { suite, step, goal, expect, capture } from 'qaspec';

// One journey, several checks in order, one browser, signed in once.
suite('buyer checks out a product', { project: 'store', as: 'buyer', start: '/' }, () => {

  step('find the product', () => {
    goal('search for "${params.product}" and open its product page');
    expect('the product page of "${params.product}" shows a price and an Add to cart button');
    expect.network('GET /api/products/**').status(200);
    expect.console.noErrors();
  });

  step('add it to the cart', () => {
    goal('add one unit to the cart');
    expect('the cart badge shows 1 item');
    expect.network('POST /api/cart/**').status(201);
    expect.storage.local('cartId').exists();
  });

  step('pay', () => {
    goal('go to checkout and pay with the saved card');
    expect('an order confirmation with an order number is shown');
    expect.network.noFailures();
    capture.url('orderUrl');
    capture('orderNumber', 'the order number on the confirmation page');
  });

  // Independent of the previous state thanks to `start`; runs even if checkout failed.
  step('order history lists it', { start: '/account/orders', needs: ['pay'] }, () => {
    expect.visible('${orderNumber}');
  });
});

Switch environment

--env staging or QASPEC_ENV

Override a value

--set params.product=…, --set projects.store.base_url=…

A project is down

Its health check fails and its suites, plus the projects that depend on it, are reported as blocked rather than failed.

Journey across apps

step('…', { project: 'store' }) switches to that app's tab, which keeps its page and session.

Pass data on

capture('orderNumber', …), then ${orderNumber} in later steps and files.

Two users in one app

Suites are grouped by user. Switching saves one sign-in and restores the other, in the same browser.

Expectations#

Exact checks run first and cost nothing. The model judges only if they pass.

The full reference is in the README. Exit codes: 0 passed, 1 failed, 2 blocked, 3 configuration error.