review: make the live eval a measurement tool (repeat aggregation, powered runs, drift watch)

.changeset/review-dispatch-tax.md

@@ -0,0 +1,5 @@
+---
+"review": patch
+---
+
+Dispatch-tax trim: dedupe the specialist-lens discipline snippets. The v1.4.0 re-run's cost premium vs the v1.3.1 baseline sat at dispatch time (uncached input +51-65%, cache writes +18-34% per run), partly because the shared discipline blocks (bounded investigation, untrusted input, lens-owned skills with quote-the-rule, the schema-rules trailer, the tri-state hunt contract) were stamped verbatim into every one of the eleven specialist-lens definitions and paid on every dispatch. The shared text now lives once, in a marker-delimited "REVIEW DISCIPLINES" section of review.md's main body; Step 1 stages it to `/tmp/gh-aw/review/disciplines.md` with one whole-line-anchored `sed` extraction of the engine-provided rendered prompt (`$GH_AW_PROMPT`), verified by grep with a byte-for-byte heredoc fallback, and each lens carries only a pointer plus its own domain notes (its investigation examples, review rules, hunts, and output JSON, all unchanged). Behavior-neutral by construction: the instruction content is unchanged, the three variants that had already drifted apart are unified on the fullest wording, and the label-shape reviewers (whose variants differ materially) keep their own copies. Pinned by `lib/disciplines.test.ts`: the section extracts cleanly under the exact sed range semantics, every lens points at the staged file and carries no residual copy, and the label-shape reviewers still carry theirs.

.changeset/review-eval-measurement.md

@@ -0,0 +1,5 @@
+---
+"review": patch
+---
+
+Make the live eval instrument a measurement tool, not just a tripwire (the tuning memo's rev-2 sizing items). `eval/aggregate.ts` pools N `live-ab-report.json` artifacts (local paths or `gh` run ids) into per-case pass rates per arm with 95% Wilson intervals plus pooled recall/verdict/noise rows, classifies misses true-miss vs found-but-dropped per gate bucket, warns on multi-sha pools, and, when every pooled report ran byte-identical arms, renders per-metric noise-floor bands as data. `live-ab.ts` gains `--repeats <n>` (the ~$29 targeted-repeats powered run: each arm runs every selected case n times in one dispatch, budget split per arm-run, report rendered through the aggregation core; the adversarial gate is decided by strict majority across repeats instead of the single-run best-of-three retry) and the A/B workflow exposes `cases`/`repeats`/`force_arms` dispatch inputs. A new scheduled workflow (`review-eval-drift.yml`, weekly) runs the full live corpus x3 repeats with both arms pinned to main's review.md and publishes the aggregated report (job summary, artifact, and a visibility PR committing the report under `.github/review-eval/drift/` so merges accumulate an in-repo time series): the memo's cumulative drift watch, doubling as a rolling noise-floor measurement. Live defect specs gain `altLocations` (alternate anchor sites with their own line windows, validated like the primary): a defect that spans files has more than one correct anchor, and `incident-sql-missing-index` (8/16 in the 07-09 wave) turns out to have been measuring anchor-site preference, not recall; replaying all 28 recorded arm-runs, the reviewer found the missing index every time (26 posted, 2 provenance-dropped mis-anchors), so the case now accepts the hot-query anchor and its residual misses classify as the anchor-snap defect class. `eval/match-arbiter.ts` implements the matcher's fallback seam on the pinned Haiku snapshot (`claude-haiku-4-5-20251001`): unmatched specs only, same-file candidates only, capped per case, every claim recorded `via: "fallback"` for audit, prompt biased to refuse, API failures degrade to non-matches; on by default with `--no-match-arbiter` to opt out. Multi-repeat runs checkpoint their artifact after every repeat, so a crash or cancellation cannot forfeit completed repeats. The measured noise floor (run 29069228968: 6 identical arm-samples, full corpus x3, $58.71) renders as data in every single-run report footer: must-catch recall 54-86%, verdict agreement 75-100%, noise 50-60%, judge quality 0.82-0.86; the drift run's default budget is $85 to cover the 14-case corpus without budget skips. `eval/README.md` is the operator guide: the three tiers, every CLI flag and CI entry point, powered-run recipes, corpus-growth rules, report-reading guidance with the measured noise floor, costs, model pins, and the harness's historical limits. Reports stamp their ruler (matcher configuration + corpus content hash) and the aggregate warns when a pool mixes rulers or when identical-arm samples scored unequal case sets (budget skips), which would fold case-mix variance into the noise-floor bands; bands now report SD alongside min/max, and repeat budgets roll unspent headroom forward instead of stranding it in fixed slices.

.changeset/review-live-ab-ci.md

@@ -0,0 +1,5 @@
+---
+"review": patch
+---
+
+CI wiring for the live A/B (live A/B plan, phase 4): the `Review Eval A/B` workflow runs on every non-draft PR touching `workflows/review/**` (plus `workflow_dispatch`), executing the arm-to-arm live eval against the PR's merge-base and posting the delta report as a sticky PR comment, a job summary, and an artifact. Per-PR runs cover the smoke-tagged live subset; a `full-eval` label (or the dispatch input) lifts that to every live case, a `skip-live-eval` label opts out, the changeset release branch is excluded, secretless runs (forks) skip green, and a new push cancels a superseded run. The job fails only when the runner's adversarial hard gate fails on the candidate arm. The runner gains `--smoke-only` and writes a markdown sibling of the JSON report for the comment step.

.changeset/review-live-ab-runner.md

@@ -0,0 +1,5 @@
+---
+"review": minor
+---
+
+The live A/B runner (live A/B plan, phase 3). `eval/live-match.ts` maps a run's posted candidates onto a live case's labeled defect specs (deterministic path/window/mechanism rule first, then an injected hard-capped fallback arbiter whose matches are recorded for audit) and aggregates live metrics: must-catch recall, verdict agreement, clean false-flag, and noise. `eval/live-ab.ts` runs the model sub-agents from two review.md versions (the merge-base baseline via `git show`, the working-tree candidate) over the same live corpus, each arm under half a hard USD budget with sticky stop-and-report-skips semantics, replays each live result through the deterministic pipeline (`runner.ts` gains a `validation` override on `RunOptions` so a live validator's output replaces the recorded block), diffs spec-level regressions over the cases both arms scored, judges both arms' comments with the pinned judge (quality aggregates only; extracted to `eval/judge-live-model.ts`, now shared with `live-judge.ts`), and emits a JSON report plus a markdown table (also appended to GITHUB_STEP_SUMMARY). Report-only except the playbook's standing rule: the candidate arm failing any adversarial-injection case exits non-zero.

.changeset/review-live-corpus-format.md

@@ -0,0 +1,5 @@
+---
+"review": minor
+---
+
+Live-enabled corpus cases (live A/B eval plan, phase 1). The eval corpus gains an opt-in `live` block carrying what a REAL model run needs and the deterministic replay does not: the PR context (`prContext`, with the description treated as untrusted author text), a post-change file tree on disk next to the case (`<id>/case.json` + `<id>/tree/`, coexisting with the flat `<id>.json` layout), and labeled defect specs (`mustCatchSpecs` / `mustNotFlagSpecs`: path, line window, mechanism keyword alternates) so a live run's model-chosen finding ids can be matched to ground truth. A live case must carry the `live` tag, a cleanly-parseable diff, and every non-removed changed file in its tree; the loader enforces all of it and `loadLiveCorpus()` returns the subset. Ten cases are converted with hand-authored real diffs and trees (five smoke incidents, both clean cases, one adversarial injection whose payload lives in the diff, one golden holdout, one synthetic mutation); their recorded line anchors now point at the authored defect lines, and the deterministic suite runs them unchanged, provenance gate included.

.changeset/review-live-producer-staging.md

@@ -0,0 +1,5 @@
+---
+"review": minor
+---
+
+The live producer (live A/B plan, phase 2): everything needed to run the REAL model sub-agents over a live-enabled corpus case. `eval/agent-extract.ts` parses `review.md`'s `## agent:` sections into name/description/model/prompt as pure data (string in, no fs), so an A/B can extract the baseline arm from a `git show` of the merge-base; parsing is strict and lists every malformed section at once, and an integration test runs it over the real `review.md`. `eval/live-stage.ts` materializes the production `/tmp/gh-aw/review/` staging layout for a case (pr-context.json, full/pr/full-stripped diffs, files.json with hasPatch, review-files.json, provenance.json, routing.json, out/, and the post-change checkout) and rewrites extracted prompts to point at it. `eval/live-producer.ts` dispatches the default finders plus the router's lenses through an injected runner seam, maps all three output contracts (label-shape reviewers, structured-schema lenses, the claim-validator's three-state verdicts) into the exact `RecordedFinding`/`CaseVerification` shapes the deterministic runner consumes, stages `claims.json`, resolves `{{#runtime-import}}` directives from the case tree, retries once on malformed output, keeps partial results on agent failure, and accounts per-agent cost. `eval/live-runner.ts` is the one SDK-backed runner (Claude Agent SDK; Read/Grep/Glob only, cwd pinned to the staged checkout, hard turn and wall-clock caps) plus a CLI smoke entry (`pnpm dlx tsx workflows/review/eval/live-runner.ts --case <id>`, requires ANTHROPIC_API_KEY).

.changeset/review-out-artifact-upload.md

@@ -0,0 +1,5 @@
+---
+"review": patch
+---
+
+Fix the out/ run-artifact upload, which failed on every run under gh-aw v0.81.6. The orchestrator passed the absolute path `/tmp/gh-aw/review/out/` exactly as instructed, but gh-aw's upload_artifact tool stages an uploaded directory under its basename and records only the staging-relative name (`out`), and the safe_outputs job then filters the staged files (`out/<agent>.json`) against `allowed-paths` with a fully anchored matcher; the absolute pattern `/tmp/gh-aw/review/out/**` therefore matched nothing, every run annotated `ERR_VALIDATION: upload_artifact: no files matched the selection criteria`, and no sub-agent outputs or `pre-existing.json` reached the run artifacts, blocking post-hoc claim auditing, the live counters, and the A/B runner, which all read them. `allowed-paths` now lists the staging-relative `out/**` (plus the absolute form, since the filter is an OR, in case a future gh-aw matches original paths). Step 9 also copies `claims.json` into `out/` before uploading, so the artifact carries the validator's input alongside its verdicts (`claim-validator.json`) and the provenance gate's set-asides (`pre-existing.json`).

.changeset/review-out-of-lane.md

@@ -0,0 +1,5 @@
+---
+"review": minor
+---
+
+Out-of-lane observations are handed off, never dropped. Observed on the review-v1.4.0 re-run lifecycle: the skill-auditor found a real eventual-consistency bug (dedup reads via a Query immediately after PutMulti) and discarded it, verbatim "that's a correctness concern, not a quotable skill-rule violation, so I'll leave it"; correct under quote-the-rule, but the observation died and the defect shipped unflagged. The skill-auditor and every specialist lens may now return `out_of_lane_observations[]` (path, optional line, the concern stated concretely, a required concrete `failure_scenario`, optional `suggested_lane`) for real concerns their own mandate does not let them report. The orchestrator converts each one into a candidate comment with the code-assigned label `question (non-blocking)` (a handoff is not a vetted finding and can never block on its own; the claim-validator never upgrades severity) and routes it through the identical provenance gate, scope filter, claims.json, and validation path as every other candidate. The shape is validated by the new `validateOutOfLaneObservation` in `lib/finding-schema.ts` (a sibling type, so `FINDING_SCHEMA_VERSION` stays 2: no serialized finding is invalidated).

.changeset/review-rereview-accountability.md

@@ -0,0 +1,5 @@
+---
+"review": minor
+---
+
+Re-review accountability, code-rendered. Observed on the review-v1.4.0 re-run lifecycle (Khan/webapp#40730): run 2 resolved the fixed threads and said nothing about the three blocking threads it left open, under a bare "Changes requested" body; run 3 approved with an empty body while resolving 11 threads. The verdict body now accounts for every prior thread, rendered deterministically from the `thread-reconciler`'s keep/resolve lists by the new `lib/rereview.ts` (never composed by the model): each still-unaddressed prior thread is enumerated as a link to its earlier comment ("as of \<sha\>", blocking first, with a verbatim excerpt of the earlier comment's first line), with the resolved count stated; an approval that resolved the last open threads says every prior thread is resolved. `threads.json` staging gains a `url` field (the thread's first comment `html_url`) to power the links, `renderReviewBody` gains a `rereviewSection` slot, and the redundant-approval skip treats a non-empty section as content. Fail-open: a missing or unparseable staging input renders an empty section, leaving the body exactly as before.

.changeset/review-rereview-live-corpus.md

@@ -0,0 +1,5 @@
+---
+"review": minor
+---
+
+Re-review coverage for the eval corpus: a live-enabled case may now carry a `live.rereview` block that makes it an open-PR snapshot instead of a first review, staging the prior review's unresolved threads (author replies included), a stamped prior review derived from `priorDiff` (the same hidden-comment mechanics production reads), and the mode dial's depth plan. The live producer dispatches the `thread-reconciler` on such cases at every depth and sizes the finder roster by the plan (`scoped` keeps the roster over the scoped diff, `flip-gated` keeps the correctness pass, `fast` reconciles only); `eval/rereview-match.ts` scores thread-resolution accuracy per ground truth, the flip-gate input, and duplicate comments on kept threads, and the A/B runner prices a mode via `--re-review-mode` on the candidate arm. The deterministic replay learns the same rules: `computeVerdict` gains `keptBlockingCount` (the flip rule as code: an approval is floored at REQUEST_CHANGES while a prior blocking thread stays open) and re-review cases render the accountability section into the planned body. Ships with the `golden-retention-lifecycle-1/2/3` chain, a sanitized port of a measured seeded live trial: planted bugs, a bad partial fix with added bugs (kept threads plus fresh defects), then the full fix that must flip to APPROVE. A mode sweep (`eval/rereview-sweep.ts`) prices every dial setting in one command: it runs the working tree's reviewer over the rereview cases at each mode and reports recall, thread resolution, flip-gate correctness, duplicates, and dollars per mode, with the executed depth per case (the tripwire may override the dial). `golden-retention-fix-push` is the under-threshold pricing case: a one-hunk fix push (unreviewed share 0.2) whose fix resolves the prior blocking thread but plants a fresh defect inside the new hunk, so scoped and flip-gated must catch it and fast definitionally cannot. The sweep is dispatchable in CI (`review-rereview-sweep.yml`, manual only since it spends model dollars); the table lands in the job summary and the JSON report in the run artifact. Every model-spending eval now has the same manual trigger surface: PR labels (`rereview-sweep`, `live-judge`, `full-eval` now firing on the labeling event itself) plus `workflow_dispatch`; and the A/B short-circuits before spending when `review.md` is byte-identical in both arms, posting a "no reviewable delta" verdict and running nothing (`--force-arms` keeps deliberate wobble controls possible).

.changeset/review-rereview-mode-dial.md

@@ -0,0 +1,5 @@
+---
+"review": minor
+---
+
+The re-review mode dial, the runs-per-PR cost lever: a per-repo `re-review` line in `.github/aw/review/ROUTING` (`full` | `scoped` | `flip-gated` | `fast`, default `full`) decides how much of the roster a repeat review of the same PR runs. `scoped` runs the whole roster staged only the hunks new since the last fully-reviewed fingerprint; `flip-gated` runs reconciliation plus the correctness pass, whose validated blocking findings veto a REQUEST_CHANGES→APPROVE flip; `fast` is reconcile-only. Three deterministic guards (`lib/rereview-mode.ts`): the anchoring full review is taken at ready-for-review (never on a draft skeleton); the flip gate; and a divergence tripwire that stamps a content-hashed hunk signature into the review body as a hidden comment (surviving cache eviction, dismiss-stale-approvals, and COMMENTED-only histories) and re-arms full review when the unreviewed share reaches 0.4. Adversarial lifecycle cases (rewrite-after-approval, sparse-PR-then-payload) live in `eval/lifecycle/` and are replayed deterministically; run cost now aggregates per executed depth (`costByRereviewDepth`) so the live A/B can price `scoped` against `full` on recall and dollars. Ships default-`full`: no consumer's behavior changes until its repo adds a mode line. Motivated by measured lifecycle costs on the seeded trial (v1.3.1: $7.19/$8.77/$9.14; v1.4.0: $8.31/$9.87/$11.29 per push), where a fresh seeded defect was caught on a re-review push that a reconcile-only path would have missed; the case for `scoped` as the recommended reduced depth. Supersedes the consumer-side stopgap in Khan/frontend#13420.

.changeset/review-trial-skill.md

@@ -0,0 +1,4 @@
+---
+---
+
+Docs/tooling only: add the `review-trial` Claude Code skill (`.claude/skills/review-trial/SKILL.md`), packaging the Khan/webapp#40678 seeded-defect live-trial choreography (live A/B plan, phase 5): isolated per-arm PRs with distinct workflow names, run collection, defect-table scoring, corpus export with the private-to-public sanitization gate, and cleanup. Seeds and ground truth stay human-authored; costs are projected and confirmed before any PR is created.