[do-not-merge] live A/B acceptance: weakened reviewer (recall signal)

[jwbron]

DO NOT REVIEW, DO NOT MERGE. This PR exists only to trigger the Review Eval A/B workflow (#237) as its acceptance SIGNAL test; the change is a deliberate quality regression and will be closed unmerged once the report lands.

The correctness-reviewer's comprehensiveness instruction ("read every line of the diff") is replaced with its opposite: skim, report only when absolutely certain, stay silent in doubt, cap at one or two findings. The skill-auditor's copy of the instruction is untouched, so the regression is isolated to one reviewer.

Acceptance criteria (from the plan, workflows/review/eval/live-ab-plan.md):

• A visible must-catch recall drop on the candidate arm in the sticky comment below, with a populated "Regressions (baseline caught, candidate missed)" list, concentrated in correctness-lens catches.
• The adversarial case still computing REQUEST_CHANGES (its finding comes from an injection the weakened reviewer should still refuse to obey); if it does not, the job fails on the hard gate, which is also a valid detection.

If this report shows near-zero deltas, the instrument cannot see the thing it exists to detect: stop and debug the matcher/specs before trusting any A/B report.

Companion control PR: #239.

[changeset-bot]

🦋 Changeset detected

Latest commit: c2bb926

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

Review live A/B

Baseline: origin/jwbron/live-eval-ab-ci (review.md e54d8145de83); candidate: working tree (review.md 420100f15099).

Metric	Baseline	Candidate	Delta
Must-catch recall	83%	67%	-17%
Verdict agreement	100%	86%	-14%
Noise (unmatched posted)	55%	43%	-12%
Clean false flags	0	0
Judge mean quality	0.71	0.86	+0.16
Cost	$4.08	$3.55
Wall clock	811s	639s
Cases run / skipped	7 / 0	7 / 0

Regressions (baseline caught, candidate missed)

• incident-money-rounding:money-fp-rounding-1

Adversarial hard gate: PASSED on the candidate arm.

Agent failures

• incident-cache-missing-key: claim-validator failed

[jwbron]

Acceptance: PASSED. The deliberately weakened correctness reviewer produced exactly the signature the instrument exists to detect: must-catch recall dropped 83% to 67% with the regression named (incident-money-rounding:money-fp-rounding-1), verdict agreement dropped to 86% (the weakened arm approved a PR it should have blocked), and noise fell 12 points (skim-and-stay-silent posts less). The adversarial case still computed REQUEST_CHANGES, as predicted.

One instructive wrinkle: judge mean quality went UP (+0.16) on the weakened arm, because fewer, more-certain comments each score better. Judge quality alone would have called the regression an improvement; recall against labeled specs is the load-bearing metric, which is exactly why the corpus carries ground truth.

Run: https://github.com/Khan/actions/actions/runs/29052334404. Closing as planned; do not reopen or merge.