fix(verify-pr): budget evidence capture in scope selection - #8104
Merged
Conversation
Post-merge measurement of #8016, on the first two live runs after it landed (#7975 and #8066): the browser installed successfully — "Install evidence browser: success" in both — and both reports carried **zero images**. #7975 has 31 tables, #8066 has 407 assertions. Neither has a witness. The cause is the third instance of one structural bug, and it is the same one twice already fixed: #7917 the /verify recommendation sat inside a section headed "local invocation ONLY" -> 1/16 reach #8016 captures were "Optionally ... when text cannot carry the oracle" -> 0/14 now captures are a required deliverable in the ARTIFACT CONTRACT, but the plan the agent executes is the Scope-selection budget list, and that list had zero mentions of evidence, png, capture or screenshot An agent that budgets by that list never allocates time for a capture, however firmly a later section says to produce one. So captures are now budget item 4 alongside the A/B, the harnesses and the gates, with the time reserved (~5 min), the gate named (QWEN_VERIFY_CHROMIUM=1), the count bounded (normally two, at most a handful), and the measured failure stated so the rule carries its own reason. The report structure gets the matching half: the A/B table item now says to reference the capture of those cells by filename. A produced image with nowhere to go is as dead as an unproduced one. Mutation-verified 4/4: dropping the budget item, the time allowance, the count bound, or the report-side reference each turns the test red. Not changed, and worth stating rather than implying: the seven techniques from #8010 do not appear in either report. I do not read that as failure — every one of them is conditional ("when one fix bundles two changes", "when the oracle is an instrument", "when a PR adds a writer to a shared store") and neither PR met those conditions. #8010 is not yet falsifiable from these runs. What both reports DO show is older skill rules working: a naive-matcher differential in #8066, and #7975's realpath check finding node_modules/@qwen-code/qwen-code-core resolving into the HEAD tree and then reasoning about whether that contaminates the control. 107/107 tests; prettier and eslint clean.
Collaborator
|
✅ Qwen Triage finished — CI landed green on ✅ Qwen Triage 已完成 —— |
Collaborator
|
Thanks for the PR! Template looks good ✓ Problem: observed and well-evidenced. Two live Direction: aligned. This is the verify-pr skill's own infrastructure — making the evidence capture instruction land where the agent actually plans against. No product-direction concerns. Size: not applicable — no core paths touched. Two files, +39/−0. Approach: the scope feels right. The fix puts captures where the agent budgets its work (Scope selection item 4) rather than adding more prose to a section the agent reads but doesn't plan from. The report-side reference (item 3 in report.md structure) is the matching half — a produced image with nowhere to go is indeed dead. One test covers both sides with mutation verification. Nothing to cut. Risk: no elevated risk signals. Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 问题：已观测且有充分证据。两次真实 方向：对齐。这是 verify-pr skill 自身的基础设施——让截图指令落在 agent 真正据以规划的位置。无产品方向顾虑。 规模：不适用——未触及核心路径。两个文件，+39/−0。 方案：范围合理。修复把截图放在 agent 分配预算的地方（Scope selection 第 4 项），而不是在 agent 会读但不会据以规划的章节加更多文字。报告侧的引用（report.md structure 第 3 项）是配套的半边——产出了却无处引用的图片确实是死的。一条测试以变异验证覆盖了两侧。无可删减。 风险：无升级风险信号。 进入代码审查 🔍 — Qwen Code · qwen3.8-max-preview Reviewed at |
Collaborator
Code reviewIndependent proposal: given that capture instructions exist elsewhere in the skill but the agent doesn't follow them because they're absent from the budget list, I would add a numbered item to Scope selection with a time estimate, gate condition, count bound, and the failure that motivated it — plus a report-side reference so captures have a referent. One test asserting both sides. Comparison with the diff: the PR does exactly this, and does it well. No simpler path missed. The budget item (SKILL.md, Scope selection) carries the four things the PR description identifies: time reserved (~5 minutes), gate named ( The report-structure addition (item 3) is the matching half — "Reference the capture of those cells here by its filename" gives a produced image somewhere to land. The test slices the skill text between the right section boundaries and asserts the budget item's position ( No correctness issues, no security concerns, no convention violations. Nothing to flag. TestingThis is a skill-text and test change with no user-visible runtime behavior — real-scenario testing is N/A. CI status for the reviewed commit (fetched once, not polled): Final CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行（取最新一次运行），省略 skipped，失败项排在最前。 The ubuntu test suite is still running. Windows/macOS and integration tests were skipped (typical for this repo's CI configuration on same-repo branches). No failures so far. The finalize workflow will update this table when CI settles. Not verified: the actual effect on 中文说明代码审查独立方案： 鉴于截图指令已存在于 skill 的其他位置但 agent 不遵循（因为预算清单中没有），我会向 Scope selection 添加一个编号条目，附带时间估计、门槛条件、数量上限和触发它的失败——外加报告侧的引用，让截图有指代对象。一条测试断言两侧。 与 diff 的比较： PR 正是这样做的，且做得好。没有遗漏更简路径。 预算条目（SKILL.md，Scope selection）携带了 PR 描述中识别的四样东西：预留时间（约 5 分钟）、点明门槛（ 报告结构的补充（第 3 项）是配套的半边——"按文件名引用该组单元格的截图"让产出的图片有落脚处。 测试在正确的章节边界之间切片 skill 文本，断言预算条目的位置（ 无正确性问题、无安全隐患、无规范违反。无需标记。 测试这是 skill 文本和测试的改动，无用户可见的运行时行为——真实场景测试不适用。 已审查提交的 CI 状态（获取一次，未轮询）：ubuntu 测试套件仍在运行，Windows/macOS 和集成测试已跳过。目前无失败。finalize 工作流会在 CI 结束后更新此表。 未验证：对 — Qwen Code · qwen3.8-max-preview Reviewed at |
Collaborator
|
Confidence: 5/5 — clean across every stage; would merge without hesitation. This is the third fix for the same structural bug — an instruction placed where the agent reads it but doesn't plan from — and the diagnosis is backed by two live runs with measurement tables. The fix is exactly right: put the capture where the budget is, give it the four things a rule needs to survive contact with a real run, and add the report-side reference so a produced image has a referent. Nine lines of skill text, thirty lines of test, nothing extraneous. My independent proposal matched the diff almost exactly. No simpler path exists — the budget list is where the agent plans, and the report structure is where the output lands. Both needed the change. The test pins position (numbered item 4 in Scope selection), properties (gate, time, count bound, failure history), and the report-side reference. Mutation-verified 4/4 per the PR description. If someone later moves the capture instruction back to a section the agent ignores, the test catches it. The PR is honest about what it cannot yet prove: the actual effect on Approval deferred until CI lands green on 中文说明置信度：5/5 —— 每个阶段都干净；毫不犹豫会合入。 这是同一个结构性错误的第三次修复——指令放在 agent 会读但不会据以规划的位置——诊断有两次真实运行的测量表格支撑。修复方向完全正确：把截图放在预算所在处，给它一条规则在真实运行中存活所需的四样东西，并补上报告侧的引用让产出的图片有指代对象。九行 skill 文本，三十行测试，无多余内容。 我的独立方案与 diff 几乎完全一致。不存在更简路径——预算清单是 agent 规划的地方，报告结构是产出落地的地方。两者都需要改动。 测试固定了位置（Scope selection 中的编号第 4 项）、属性（门槛、时间、数量上限、失败历史）和报告侧引用。按 PR 描述做了 4/4 变异验证。如果将来有人把截图指令移回 agent 忽略的章节，测试会捕获。 PR 对尚无法证明的部分是诚实的：对 审批延迟至 CI 在该提交上全绿。 — Qwen Code · qwen3.8-max-preview Reviewed at |
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
approved these changes
Jul 30, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Makes evidence capture a line in the budget the agent actually plans against, because #8016 shipped the capability and the first two live runs still produced zero images.
The measurement
#8016 merged, then two
/verifyruns landed on it:Install evidence browser: successInstall evidence browser: successThe browser installed.
QWEN_VERIFY_CHROMIUM=1was set. Nothing was captured.The cause — third instance of one structural bug
This is the same defect twice already fixed in this lane:
The artifact contract is a checklist of outputs. The plan the agent executes is the Scope-selection budget list:
grep -ciE 'evidence|png|capture|screenshot'over that whole section: 0. An agent budgeting by that list never allocates time for a capture, however firmly a later section says to produce one.The fix
Captures become budget item 4, with the four things a rule needs to survive contact with a real run:
QWEN_VERIFY_CHROMIUM=1And the report structure gets the matching half: the A/B table item now says to reference the capture of those cells by filename. A produced image with nowhere to go is as dead as an unproduced one.
Why it's needed
The capture is the provenance artifact — a table is the agent's claim, a capture of the run is the witness. #7975 has 31 tables and no witness; #8066 has 407 assertions and no witness. That is exactly the gap between this lane's output and the maintainer-written rounds it is modelled on.
Reviewer Test Plan
How to verify
One test asserts the budget item exists as a numbered item in Scope selection (
/^4\. \*\*Capture/m— position is the whole point, since the instruction already existed elsewhere), plus the gate, the time allowance, the count bound, and the report-side reference.Mutation-verified 4/4:
~5 minutesnormally two, at most a handful107/107 tests; prettier and eslint clean.
What the same measurement says about #8010 — reported, not spun
The seven techniques from #8010 do not appear in either report. I do not read that as failure, and the distinction matters:
Not independently exercisedin its Not covered list.What both reports do show is older skill rules working, which is worth recording because it is the control for this whole exercise:
node_modules/@qwen-code/qwen-code-coreresolving into the HEAD tree, then reasoned explicitly about whether that contaminates the base control (it concluded not, because the base paths call no new core API). That is the A/B section'sUnlike captures — which are unconditional whenever a harness ran — those are the conditional rules behaving as designed. The acceptance check for #8010 is a future round on a PR that does bundle two fixes or add a writer to a shared store.
Tested on
Skill text and its test. The acceptance check is the next
/verifyrun: images per report going from 0 to a small purposeful number, and the A/B table carrying a filename beside it.Risk & Scope
evidence/is empty), not more prose.Linked Issues
Follow-up to #8016, whose acceptance criterion this addresses. Same structural defect as #7917. No issues closed.
中文说明
本 PR 做了什么
把证据截图变成agent 真正据以规划的那份预算中的一项——因为 #8016 已经把能力做好了，而合入后最初两次真实运行仍然产出 0 张图。
测量
#8016 合入后，两次
/verify落地：Install evidence browser: successInstall evidence browser: success浏览器装上了。
QWEN_VERIFY_CHROMIUM=1也设了。什么都没截。病因——同一个结构性错误的第三次
这与本车道已经修过两次的缺陷完全同型：
artifact contract 是一份产出物清单。而 agent 真正执行的计划是 Scope selection 的预算表：
对该小节全文执行
grep -ciE 'evidence|png|capture|screenshot'：0。按这份清单规划预算的 agent 永远不会为截图分配时间，无论后面的章节把话说得多硬。修法
截图成为预算第 4 项，并带上一条规则要在真实运行中存活所需的四样东西：预留时间（约 5 分钟）、点明门槛（
QWEN_VERIFY_CHROMIUM=1时）、限定数量（通常两张，最多数张，以免"给它预算"变成八张）、写出它自身的失败（两次浏览器可用的真实运行产出 0 张——不附带理由的规则读起来就是建议）。报告结构补上对应的另一半：A/B 表格那一项现在要求按文件名引用该组单元格的截图。产出了却无处引用的图片，和没产出一样是死的。
为什么需要
截图就是溯源凭证——表格是 agent 的主张，运行截图是见证。#7975 有 31 张表格、零见证；#8066 有 407 条断言、零见证。这正是本车道产出与它所对标的维护者手写轮次之间的差距。
评审验证方案
一条测试断言该预算项作为 Scope selection 中的编号条目存在（
/^4\. \*\*Capture/m——位置就是全部要点，因为该指令在别处早已存在），外加门槛、时间预留、数量上限、以及报告侧的引用。4/4 变异验证：删除预算项、删除
~5 minutes、删除数量上限、删除报告侧引用，各使测试变红。107/107 测试；prettier 与 eslint 干净。
同一次测量对 #8010 的说明——如实报告，不做美化
#8010 的七项技术在两份报告中均未出现。我不认为这是失败，而这个区分很重要：
Not independently exercised。而两份报告确实显示了更早的 skill 规则在生效，这一点值得记录，因为它是整件事的对照组：
node_modules/@qwen-code/qwen-code-core解析到了 HEAD 树，随后明确论证这是否污染 base 对照（结论是不污染，因为 base 路径不调用任何新的 core API）。这是 A/B 小节那条与截图不同（只要跑过 harness 就无条件适用），上述是条件性规则按设计行事。#8010 的验收标准是未来某次针对确实捆绑了两处修复、或确实向共享存储新增写入者的 PR 的验证轮次。
测试平台
macOS ✅；Windows N/A；Linux N/A——skill 文本及其测试。验收标准是下一次
/verify运行：每份报告的图片数从 0 变成一个少而有目的的数值，且 A/B 表格旁带有文件名。风险与范围
evidence/为空时由发布方 warning），而不是再写更多散文。关联 Issue
#8016 的后续，处理的正是其验收标准。与 #7917 同型的结构性缺陷。不关闭任何 issue。