Open Code Review as a merge gate: 8 of 8 tests green, LLM review untested

alibaba/open-code-review is a code review tool that pairs deterministic pipelines with an LLM agent and leaves line-level comments. We pinned v1.13.0 behind a gate that fails a PR on any finding at or above a chosen severity, and our keyless run on October 14, 2026 passed 8 of 8 tests with all 8 lab CI jobs green. Without a provider key the gate never asks OCR for findings, so the keyless verdict comes from three scanners we wrote and OCR serves as a preflight.

The catch

In keyless mode the gate calls ocr delegate preview --format json, checks that the output parses as JSON, and moves on. The findings come from three regular expressions in src/keyless.ts. One matches a secret literal assigned to a name like API_KEY, SECRET, TOKEN or PASSWORD. One matches an sk-live, sk-test or sk-proj key. One matches a SELECT, INSERT, UPDATE or DELETE built with string +. Null dereferences, XSS and thread-safety bugs, the rules OCR advertises, pass the keyless gate untouched.

OCR's own review runs only when a provider key such as GEMINI_API_KEY or ANTHROPIC_API_KEY is present. The gate then calls ocr review --format json --audience agent and normalizes the comments. Our CI has no provider secrets, so the live-review job no-ops and the report records it as pass_noop_without_keys. We have no measured LLM review in this lab, and the cost and latency of that path are yours to find out.

What we ran (keyless)

bun install --frozen-lockfile
bun run ensure:ocr
bun test
bun run review --base origin/main --head HEAD --fail-on high --keyless

From reports/gate.json, run on October 14, 2026:

The review tests build a throwaway git repo for each case. The bad fixture adds a hard-coded sk-live key on line 1 of app.ts and a SQL query built by concatenation on line 3. The gate reports hardcoded-secret on line 1 and sql-string-concat on line 3 and exits 1. A clean diff exits 0 with zero findings. Pointing OCR_BINARY at a path that does not exist exits 2, so a broken install blocks the merge.

The lab name refers to Scion, the source control system Anthropic describes in its post on redesigning source control. Scion is internal and not for general use, so this gate runs on plain git.

How it is wired

The gate runs on the agent's branch before merge. It takes git diff --unified=0 base...head, picks the keyless or LLM path, and writes review.json and review.md with file, line, severity and rule for each finding. Exit 0 lets the merge through. Exit 1 sends the agent back to fix the named lines. Exit 2 means the review could not run. The ocr-merge-gate skill under .claude/skills/ tells the agent to get exit 0 before it merges.

Minilab (open source)

Upstream is alibaba/open-code-review (47,166 stars, Apache-2.0, Go, as of October 14, 2026), pinned at 1672852136fc (v1.13.0). Clone the lab with submodules and run bun test. With a provider key set, drop --keyless from the review command to measure OCR's own findings on your diffs.

bun install --frozen-lockfile && bun run ensure:ocr && bun test