Eval harnesses rot without keys. Here's the keyless ADK gate.

We re-ran SkilledDragonCorp/minilab-google-adk-eval on 2026-10-13 with google-adk[eval]==2.11.0 on the ScriptedLlm path, every GOOGLE_* variable unset, and got 1 ADK eval case passed, 4 of 4 unit tests green, ruff and mypy clean, and the whole gate in 14.247 seconds (run on October 13, 2026). The catch is that ADK's judge metrics need a live Gemini key, so a harness that depends on them by default only goes green on whichever laptop has the key exported. The keyless path proves a narrower thing on purpose, and you can reproduce it without credentials.

The catch

ADK ships judge metrics (final_response_match_v2, hallucinations_v1, safety_v1) that call a live model, and they need GOOGLE_API_KEY or a Vertex project to run. A harness that depends on those by default depends on a wallet and on whoever remembers to export the variable, and it rots the first week nobody pays attention.

So the keyless path checks less, deliberately. On a deterministic fixture, the agent calls the right tool with the right arguments (tool_trajectory_avg_score, threshold 1.0) and its final text matches the reference closely enough (response_match_score, threshold 0.85). That tells you the eval plumbing works on a clean clone, and it says nothing about how your production agent behaves. Live judges stay opt-in behind eval_config_live.json, and the CLI picks that config only when real credentials are present.

What we ran (keyless)

uv sync --extra dev && uv run adk-lab eval

adk-lab eval wraps the official adk eval … --print_detailed_results call, parses the summary, and then writes the keyless gate extras into the same scorecard: unit pytest (deselecting the @pytest.mark.e2e test, which would recurse into run_eval), ruff, mypy, the eval-case count from evalset.json, and the number of SKILL.md files under .claude/skills.

From reports/eval.json (run on October 13, 2026):

Two notes on the numbers. The gate duration is wall-clock on a laptop with warm uv caches, so expect it to move between machines. It stays in the seconds, which is what keeps it inside the pre-push loop. And inference_call_count_v1 reads 0.0 on this path because ScriptedLlm replays recorded LlmResponse turns instead of calling a model. That is the point of the path, and also why it says nothing about model quality.

How it is wired

Fixtures feed an ADK agent scaffold. The evaluator scores trajectory and response match, then writes eval.json. Live Gemini judges sit on a separate opt-in path that only runs when you choose to spend a key. Keep the keyless scorecard in CI, turn on the live judges when you mean to, and keep their results out of the merge gate unless you are prepared to pay for every PR.

The informational counters (tool_call_count_v1, inference_call_count_v1, token_usage_v1, invocation_duration_v1) are recorded but never fail the run. The deterministic metrics gate the run; the counters are there to trend over time.

Two skills ship under .claude/skills/. adk-agent-scaffold is a researcher LlmAgent with one pure-Python FunctionTool and a ScriptedLlm(BaseLlm), so the agent runs with no network. adk-eval-author explains how to author evalset.json and eval_config.json, which metrics are keyless, and how the scorecard gets written. Both are layout-checked by scripts/check-skills-layout.sh (frontmatter name must equal the folder name; no .cursor/skills or .agents/skills duplicates) and counted in the report (scaffold_count: 2).

If you extend the lab, add cases to src/adk_lab/evalset.json, extend the scripted turns to match, and keep the default config free of live metrics. The next uv run adk-lab eval will count the new cases and refresh generated_at on its own.

Minilab (open source)

Docs: adk.dev/evaluate · upstream google/adk-python at pin 2.11.0

Re-run the gate before you trust the numbers on your own agent. The ScriptedLlm path replays recorded turns, so it is deterministic; a live agent is not, and its scores will move between runs.

uv sync --extra dev && uv run adk-lab eval