From 8405552c2d7961d82055a97af8e69a28b682bcbb Mon Sep 17 00:00:00 2001 From: wenshao Date: Thu, 30 Jul 2026 15:23:09 +0800 Subject: [PATCH] fix(verify-pr): budget evidence capture in scope selection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-merge measurement of #8016, on the first two live runs after it landed (#7975 and #8066): the browser installed successfully — "Install evidence browser: success" in both — and both reports carried **zero images**. #7975 has 31 tables, #8066 has 407 assertions. Neither has a witness. The cause is the third instance of one structural bug, and it is the same one twice already fixed: #7917 the /verify recommendation sat inside a section headed "local invocation ONLY" -> 1/16 reach #8016 captures were "Optionally ... when text cannot carry the oracle" -> 0/14 now captures are a required deliverable in the ARTIFACT CONTRACT, but the plan the agent executes is the Scope-selection budget list, and that list had zero mentions of evidence, png, capture or screenshot An agent that budgets by that list never allocates time for a capture, however firmly a later section says to produce one. So captures are now budget item 4 alongside the A/B, the harnesses and the gates, with the time reserved (~5 min), the gate named (QWEN_VERIFY_CHROMIUM=1), the count bounded (normally two, at most a handful), and the measured failure stated so the rule carries its own reason. The report structure gets the matching half: the A/B table item now says to reference the capture of those cells by filename. A produced image with nowhere to go is as dead as an unproduced one. Mutation-verified 4/4: dropping the budget item, the time allowance, the count bound, or the report-side reference each turns the test red. Not changed, and worth stating rather than implying: the seven techniques from #8010 do not appear in either report. I do not read that as failure — every one of them is conditional ("when one fix bundles two changes", "when the oracle is an instrument", "when a PR adds a writer to a shared store") and neither PR met those conditions. #8010 is not yet falsifiable from these runs. What both reports DO show is older skill rules working: a naive-matcher differential in #8066, and #7975's realpath check finding node_modules/@qwen-code/qwen-code-core resolving into the HEAD tree and then reasoning about whether that contaminates the control. 107/107 tests; prettier and eslint clean. --- .qwen/skills/verify-pr/SKILL.md | 9 +++++++ scripts/tests/qwen-triage-workflow.test.js | 30 ++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/.qwen/skills/verify-pr/SKILL.md b/.qwen/skills/verify-pr/SKILL.md index fc1bad504f..2b75bfe107 100644 --- a/.qwen/skills/verify-pr/SKILL.md +++ b/.qwen/skills/verify-pr/SKILL.md @@ -104,6 +104,13 @@ secondary claims. Budget by value: 1. **A/B load-bearing proof of the central claim** (always, ~half the budget). 2. **One or two wire-oracle harnesses** on the changed surface. 3. **Targeted gates**: tests/typecheck of the affected workspace(s) only. +4. **Capture the A/B and the matrix as they print** (~5 minutes, whenever + `QWEN_VERIFY_CHROMIUM=1`). This is a budget line, not an afterthought: + two live runs with the browser installed and working produced **zero** + images, because the instruction lived in the artifact contract while the + plan the agent follows is this list. Decide here how many captures the + round needs — normally two, at most a handful — and reserve the time. + See the artifact contract for the mechanics and the naming rule. Everything else is explicitly out of scope — and is **listed as not covered** in the report. Never let breadth eat the A/B: one proven load-bearing claim @@ -572,6 +579,8 @@ central claim from being tested — say why. them. Cite the tables below by name instead of restating their numbers in prose: a number written twice is a number that can disagree with itself. 3. **Central claim + A/B table** (cells, oracles, head vs control counts). + Reference the capture of those cells here by its filename — a table with + no witness beside it is the shape every report has had so far. 4. **Corrections**, when an earlier review round or bot comment described the code inaccurately (a wrong ARIA role, a wrong mechanism, a misattributed cause). State the correct fact with its evidence and label diff --git a/scripts/tests/qwen-triage-workflow.test.js b/scripts/tests/qwen-triage-workflow.test.js index ed0e55cedb..f781838125 100644 --- a/scripts/tests/qwen-triage-workflow.test.js +++ b/scripts/tests/qwen-triage-workflow.test.js @@ -3382,6 +3382,36 @@ describe('qwen-triage verify round-3 hardening', () => { expect(soft).toBeGreaterThanOrEqual(Math.floor(agentMinutes * 0.8)); }); + // Third instance of one structural bug: an instruction placed outside the + // flow the agent actually follows. #7917 buried the /verify recommendation + // in a "local invocation ONLY" section; #8016 marked captures "Optionally"; + // and captures still came out ZERO on two live runs (#7975, #8066) where + // the browser installed fine — because the plan the agent executes is the + // Scope-selection budget list, and that list had no capture line at all. + it('budgets evidence capture in scope selection, not only in the contract', () => { + const scope = verifySkill.slice( + verifySkill.indexOf('## Scope selection'), + verifySkill.indexOf('## Method'), + ); + // A numbered budget item alongside the A/B, harnesses and gates. + expect(scope).toMatch(/^4\. \*\*Capture/m); + expect(scope).toContain('QWEN_VERIFY_CHROMIUM=1'); + // Time reserved, and the failure that motivated it named — a rule + // stated without its failure reads as advice and gets skipped. + expect(scope).toContain('~5 minutes'); + expect(scope).toContain('produced **zero**'); + // Bounded: the cap exists so "budget it" does not become eight images. + expect(scope).toMatch(/normally two, at most a handful/); + + // And the report has somewhere to put it, or a produced capture has no + // referent and the naming rule means nothing. + const structure = verifySkill.slice( + verifySkill.indexOf('### report.md structure'), + verifySkill.indexOf('## Hard rules'), + ); + expect(structure).toContain('Reference the capture of those cells here'); + }); + // Cleanups must never descend through a PR-writable parent, and an // outward-resolving hooks entry must be removed rather than reported. it('survives symlink escapes in the workspace cleanup', () => {