Tester Army e2e as a merge gate: good checkout exits 0, a missing button exits 1

tester-army/e2e is a testing framework for web and mobile apps. We put one Chromium checkout spec behind a fail-closed gate and ran it against two fixtures on October 14, 2026. The good app passed with exit 0, and a copy with the Confirm order button removed was refused with exit 1. The verdicts are right, and the gate's own test counters did not read the runner's report format.

The catch

Each gate report carries an e2eSummary block, and it is wrong in both runs. The gate looks for a top-level summary key in the runner's report.json. e2e 0.20.0 writes schemaVersion: "report-1" with results nested under run, so the key is missing and the gate falls back to defaults. The passing report shows 0 passed, 0 failed, 0 skipped even though the checkout step passed. The refused report shows 1 failed, and that 1 is the exit-code fallback. outcome and exitCode follow the runner's exit status, so the merge decision holds. If you want test counts from reports/gate-report.json, fix the parser first.

Coverage is one spec. It signs in, adds an item to the cart, and confirms the order. A change that breaks a screen the spec never visits will pass the gate.

What we ran (keyless)

bun install
bun run browsers
export E2E_TELEMETRY_DISABLED=1
bun run gate --app fixtures/app
bun run gate --app fixtures/app-broken

From reports/gate-report.json and reports/gate-report-broken.json, both run on October 14, 2026:

The broken fixture is the kind of change an agent makes while tidying markup. The Confirm order button is replaced by an HTML comment and its click handler is gone. The spec clicks Confirm order and expects the status to read "Order confirmed", so the runner's nested report in the refused case records status: "failed" and exitCode: 1, and the gate maps that to exit 1.

The lab's RESULTS.md records 6 tests with 15 expects passing, covering the pass, fail and infrastructure exit codes and the report schema. The upstream CI cohort at the pin ran green across lint, typecheck, docs, the Node 22, 24 and 26 test matrix, and the pnpm 11 and 12 install smoke.

How it is wired

After an agent edits a UI, it runs bun run gate --app <path>. The gate serves that directory with Bun.serve on 127.0.0.1, points e2e at it through E2E_APP_URL, and writes reports/gate-report.json. Exit 0 allows the merge. Exit 1 means a user flow broke. Exit 2 means the infrastructure failed, such as a missing Chromium or a server that would not start, and the gate treats it as a refusal. The spec uses locators only, so the default path needs no model keys.

Minilab (open source)

Upstream is tester-army/e2e (10,482 stars, Apache-2.0, TypeScript, as of October 14, 2026), pinned at 292cd059. Clone the lab with submodules and run both gate commands above. Then run PLAYWRIGHT_BROWSERS_PATH="$(mktemp -d)" bun run gate --app fixtures/app to see the exit 2 path.

bun install && bun run browsers && bun run gate --app fixtures/app