strands-decider as a merge gate: 12 tests green, and the keyless path uses a fake model

strands-labs/strands-decider is a small decision model that picks between options or rates on a scale, with a calibrated confidence on every answer. We wrapped it as decider-gate, a CLI that reads a PR summary and returns allow, deny or escalate as an exit code the agent has to honor. Our keyless run on October 14, 2026 passed 12 tests and drove the deny and escalate paths, and those paths ran against a deterministic fake decider, so the model itself appears in one optional CI job.

The catch

The required gate never loads the model. Keyless CI sets HF_HUB_OFFLINE=1 and swaps in FakeDecider, a stub that returns whatever noul, score and confidence you pass it. The 0.91 and 0.42 confidences below are fixed inputs to that stub. They prove the mapping from answers to exit codes and tell you nothing about how well the 2B checkpoint judges a real diff.

The real model, StrandsAgents/strands-decider-2B-qwen3.5-v1-2610, runs in a separate live (optional model) job. It loads the checkpoint on CPU, feeds it the unsafe fixture, and asserts one thing, that the decision is anything other than allow. That job reported success in 54 s. It is also marked continue-on-error, so it can fail without blocking a merge. One fixture and one assertion make a smoke test. Before you let this gate merge on its own, run the live slice on your own PR history and look at where the confidences land.

The gate only sees what pr.json tells it. The skill builds that file from gh pr view and writes ci_status as unknown, so the model has no idea whether CI passed unless you fill it in.

What we ran (keyless)

uv sync --locked --group dev
HF_HUB_OFFLINE=1 HF_HUB_DISABLE_TELEMETRY=1 uv run pytest -q
uv run decider-gate --pr-json tests/fixtures/unsafe-pr.json --fake deny
uv run decider-gate --pr-json tests/fixtures/safe-pr.json --fake escalate --threshold 0.8

From reports/gate.json, run on October 14, 2026, upstream pinned at 8b6faa55:

The unsafe fixture is the PR you want a gate to catch. It rewrites two GitHub Actions workflows, deletes 220 lines including tests/test_gate.py, and arrives with failing CI. The safe fixture is a one-line README typo fix. Escalating the typo fix is deliberate in this run, because the stub reports 0.42 confidence and anything under 0.8 goes to a human.

The exit codes are the contract. 0 is allow, 1 is deny, 2 means the model failed to load and the gate fails closed, and 3 is escalate.

How it is wired

The gate sits between the agent's open PR and gh pr merge. The agent builds pr.json, runs decider-gate --threshold 0.8, merges only on exit 0, and stops to ask a reviewer on exit 3. If the checkpoint cannot load, the gate returns a deny with confidence 0, so a missing model blocks the merge.

Minilab (open source)

Upstream is strands-labs/strands-decider (633 stars, Apache-2.0, Python, as of October 14, 2026). Clone the lab with submodules and run the pytest line above. To put the real model behind the gate, install the gate group with uv sync --locked --group dev --group gate and run uv run pytest -q -m live tests/test_live.py with HF_HUB_OFFLINE=0.

uv sync --locked --group dev && HF_HUB_OFFLINE=1 uv run pytest -q