AI-Infra-Guard flagged our fake model HIGH at 100. It flagged real Gemini HIGH at 70.

Tencent's AI-Infra-Guard ships an api-checker that black-box audits an OpenAI-compatible relay, to check that the model behind it is the one on the label. We pinned v4.6.5 and pointed it at a deterministic fake model on 127.0.0.1 with no API keys. The quick profile scored the fake MEDIUM at 50 and the full profile scored it HIGH at 100. The catch showed up in a separate live run, where Google's own Gemini endpoint also came back HIGH, at 70.

The catch

That live baseline (reports/live-gemini-baseline.json) ran the quick profile against Gemini's OpenAI-compatible endpoint with gemini-2.0-flash. The models probe returned HTTP 200 with 63 models, and gemini-2.0-flash was not in the list, which scored a MEDIUM finding worth 20. The liveness, identity and fingerprint probes then all failed with the same parse error, 'list' object has no attribute 'get', and liveness alone scored a HIGH finding worth 50. The verdict was HIGH and the CLI exited 1.

So a HIGH verdict can come from a model name the relay does not list, or a response shape the probe cannot parse, as easily as from a swapped model. Open the findings and the per-probe errors before you drop a relay. Also expect the audit to spend tokens on the relay you test, since the HTTP summary step makes one extra call to the model. Use a spend-capped key.

What we ran (keyless)

git clone --recurse-submodules https://github.com/SkilledDragonCorp/minilab-aig-relay-audit.git
cd minilab-aig-relay-audit
uv sync --locked --group dev
cd upstream
python3 -m venv services/api_checker/.venv
services/api_checker/.venv/bin/pip install -r services/api_checker/requirements.txt
go build -o ai-infra-guard ./cmd/cli/main.go
export AIG_API_CHECKER_PYTHON="$PWD/services/api_checker/.venv/bin/python"
cd ..
uv run pytest -q

From reports/gate.json, run on October 14, 2026:

The lab suite passed 11 tests in 57.89 s, including one that checks the relay API key never lands in the written report. The gate also records the upstream api_checker suite at 115 passed with 155 subtests. In CI, six jobs succeeded and the live-gemini job was skipped.

The fourth row is a safety control. Without AIG_API_CHECKER_ALLOW_HTTP and AIG_API_CHECKER_ALLOW_PRIVATE_TARGETS, the HTTP server answers a plain-HTTP loopback target with a 400, so the audit server will not probe local or private addresses unless you opt in.

How it is wired

The audit runs before the relay goes into an agent's config. relay-audit run sends probes to the relay's /v1/models and /v1/chat/completions, either through the Python run_relay_audit path with profiles quick, standard and full, or through api-checker serve over SSE. It exits 0 only when the verdict is LOW with no findings, 1 on any finding, and 2 on a tool error, so you can gate a deploy on the exit code.

Minilab (open source)

Upstream is Tencent/AI-Infra-Guard (6,868 stars, Apache-2.0, Python, as of October 14, 2026). Re-run uv run pytest -q for the keyless gate. To audit a relay you own, set a spend-capped RELAY_API_KEY and run uv run relay-audit run --base-url "$RELAY_BASE_URL" --model "$RELAY_MODEL" --profile full --out reports/relay-audit.json.

uv sync --locked --group dev && uv run pytest -q