review: the live producer (live A/B phase 2)

[jwbron]

Phase 2 of the live A/B eval plan (#232), complete, stacked on the Phase 1 corpus format (#233): everything needed to run the real model sub-agents over a live-enabled corpus case.

2a: eval/agent-extract.ts

Parses review.md's ## agent: sections into {name, description, model, prompt}. String in, no filesystem access, because the baseline arm of an A/B extracts the merge-base review.md via git show. Parsing is strict: missing/unterminated frontmatter, a name mismatch, a missing model, or an empty body throws with every problem listed, since a silently dropped agent would skew an eval arm without failing it. An integration test extracts the real review.md (21 agents) and pins the default roster's reference to the production staging root so a rename fails a test instead of silently staging nothing.

2b: eval/live-stage.ts

Materializes the production /tmp/gh-aw/review/ layout for one case, per review.md Step 1's staging contract: pr-context.json (synthetic identity fields), full.diff/pr.diff/full-stripped.diff, files.json + review-files.json with the hasPatch cross-check, provenance.json, routing.json from the deterministic router, out/, and the post-change checkout from the case's tree. rewriteAgentPrompt swaps the production staging root across a prompt.

2c: eval/live-producer.ts + eval/live-runner.ts

produceLive dispatches the live roster (default finders plus the router's lensesToSpawn; no pattern-triage or thread-reconciler in eval) behind an injected LiveAgentRunner seam, the same pattern as judge.ts's injected model, so all its logic is stub-tested with zero model calls. It maps the three sub-agent output contracts into exactly the shapes the deterministic runner consumes (RecordedFinding[] for produceFindings, CaseVerification[] for the validation replay): label-shape findings get the code-assigned lens (correctness, or conventions for the skill-auditor so labelForFinding reproduces the best-practice label variants) and the production 0.7 confidence default; lens findings validate against the schema as-is; the claim-validator's {claims: [...]} output becomes verifications. It stages claims.json, resolves {{#runtime-import}} directives from the case tree (a case opts into a skills index by carrying the file), retries once on malformed output with the rejection fed back, keeps partial results when an agent fails twice, prefixes colliding finding ids, and accounts per-agent cost/turns/wall-clock.

live-runner.ts is the only module touching a real model runtime: one Agent SDK query() per dispatch, Read/Grep/Glob only, cwd pinned to the staged checkout, the agent's pinned model, hard turn and wall-clock caps, plus the CLI smoke entry: pnpm dlx tsx workflows/review/eval/live-runner.ts --case <id> (requires ANTHROPIC_API_KEY). Adds @anthropic-ai/claude-agent-sdk as a dev dependency.

Documented deviations from production (module doc): no pattern-triage pass, runtime imports resolve from the case tree or a fixed note, and the investigation-cap CLI is not staged (its own denied-budget fallback applies).

Test plan:

• pnpm run test --run: 680 tests green (21 new across the three test files: extraction fixtures + real-review.md integration, staging via memfs, producer via a scripted stub runner covering happy path, retry, double-failure, empty-findings skip, id collisions, missing-agent error, and runtime-import resolution).
• pnpm run typecheck and eslint clean.
• The CLI smoke run against a live case needs ANTHROPIC_API_KEY, which this environment does not have; it is the first thing to run when testing this PR (any of the 13 live case ids, e.g. --case incident-money-rounding). Expected cost: roughly $0.50 to $2 per case with the default roster.

Next steps (human)

1. Key-bearing validation DONE, superseded by the phase 4 acceptance runs, which exercised this producer end to end with a real key over 7 live cases x 2 arms x 2 PRs (runs 29052332224 and 29052334404); measured ~$0.55-0.72 per case per arm, inside the plan's expected envelope. The CLI smoke remains available for local iteration.
2. Sanity-check the SDK runner's settings (live-runner.ts): permissionMode: "bypassPermissions" with tools restricted to Read/Grep/Glob and cwd pinned to the staged checkout. Confirm you are comfortable with that posture for CI.
3. Review the documented deviations from production in live-producer.ts's module doc (no pattern-triage, runtime-import resolution, investigation-cap absence); each is a measurement caveat for the A/B.
4. New dev dependency: @anthropic-ai/claude-agent-sdk. Confirm it clears whatever dependency review applies to this repo.

[changeset-bot]

🦋 Changeset detected

Latest commit: 25133b4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
review	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR