feat(triage): surface the sandboxed lanes on the CI path - #7917
Merged
Conversation
Measured on 2026-07-28 across 22 open PRs: of the 16 whose AUTHOR had write access and that already carried a triage comment, exactly 1 mentioned `/verify`. The lane it recommends has produced real evidence on four PRs (#7829 2565 assertions, #7821 1347, #7830 27, #7881 17), so the gap is not that the lane is useless — it is that almost nobody is told it exists. The instruction was there the whole time. It was a conditional clause inside a section headed "2c. Real-Scenario Testing — local invocation ONLY" whose first words are "Never in unattended CI." An agent running in CI reasonably skips that section, and the assembly order at the end of Stage 2 only allotted 2c a slot "when one was driven locally" — so on the CI path there was no place for the recommendation to go even if it had been read. Split the CI-path half out into its own section, 2b-bis, sited immediately after the CI-evidence step it follows from: 2b tells you the suite is green, and cannot tell you the suite pins the change. It is a required element of the Stage 2 comment when the central claim is behavioural, with two explicit skip conditions — nothing behavioural to settle, or the author lacks write (both lanes execute the author's code, so recommending them on an external contributor's PR is a guaranteed denial). It must name the specific unsubstantiated claim, because a bare "you could run /verify" is noise and noise is why the line got skipped. Stage 1e carried the same dead pointer, and worse: on the high-risk paths — the strongest triage-time signal in the skill, 10 of 31 reverted PRs against 5 of 60 controls, p = 0.006 — it recommended tmux alone and never named /verify. The PRs most likely to be reverted were the ones never offered the lane that proves a change is load-bearing. 1e now points at 2b-bis and names both lanes. The fix is positional, so the tests are positional: asserting that the file mentions `/verify` would have passed throughout the entire period the recommendation was dead. Mutation-verified 4/4 — moving 2b-bis back below the local-only heading, dropping it from the assembly order, dropping the author-write carve-out, and reverting 1e to the tmux-only pointer each turn a test red. Prose assertions are whitespace-normalised and shown to survive a maximal re-wrap, since prettier reflows this file.
Collaborator
|
|
Collaborator
|
Thanks for the PR! Template looks good ✓ Problem: observed deficiency with hard data — 1/16 eligible PRs mentioned Direction: aligned — this is internal tooling that makes the triage bot surface lanes that already exist and produce real evidence. No public contract, auth, sandbox, or telemetry surface touched. CHANGELOG: no direct reference, but the area is relevant (triage quality directly affects merge safety). Size: not applicable — no core paths touched. 2 files, +112/−17 (skill text + its test). Approach: the scope feels right. The fix is positional (move the recommendation above the local-only heading into its own Risk: no elevated risk signals — neither changed file matches the high-risk path patterns. Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 问题：已观测到的缺陷，有硬数据支撑——16 个符合条件的 PR 中只有 1 个提到了 方向：对齐——这是内部工具改进，让 triage bot 暴露已经存在且产出真实证据的车道。不涉及公共接口、认证、沙箱或遥测。CHANGELOG：无直接引用，但该领域相关（triage 质量直接影响合入安全）。 规模：不适用——未触及核心路径。2 个文件，+112/−17（skill 文本 + 测试）。 方案：范围合理。修复是位置性的（将推荐移到 local-only 标题之上，独立为 风险：无升级风险信号——两个变更文件均不匹配高风险路径模式。 进入代码审查 🔍 — Qwen Code · qwen3.8-max-preview Reviewed at |
Collaborator
Code ReviewIndependent proposal: given the diagnosis (recommendation buried in a "local invocation ONLY" section that CI agents skip), I would extract the CI-relevant instruction into its own section positioned before the local-only heading, make it a required Stage 2 element for behavioral claims, add explicit skip conditions, fix the 1e dead pointer, and write positional tests. The PR does exactly this. Findings: no blockers. The diff is clean and focused. The old 2c text mixed CI-path and local-only instructions in a single block — the new 2b-bis section cleanly separates them, and the 2c rewrite is a two-sentence redirect. The author-write carve-out is preserved verbatim. The assembly order update slots The 1e fix is correct: the old text said "recommend E2E verification in tmux (Stage 2c)" — a dead pointer into the section CI agents skip. The new text names both lanes and points at 2b-bis. The "(see Stage 2c)" reference in the author-lacks-write fallback is also removed. Tests follow the existing pattern (read No AGENTS.md violations. No over-abstraction, no duplication, no scope creep. CI Test EvidenceFinal CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行（取最新一次运行），省略 skipped，失败项排在最前。 The ubuntu unit test suite is still running. macOS/Windows tests and integration tests are skipped (expected for a skill-text + test-only change). No failures so far. The finalize workflow will update this table when CI settles. 2b-bis: skip — this is a skill-text change (instruction positioning and its positional tests), not a product behavioral claim. The tests substantiate the positional fix; the real validation is the post-merge measurement the author describes (1/16 baseline). 中文说明代码审查独立方案： 根据诊断（推荐藏在 CI agent 会跳过的 "local invocation ONLY" 小节中），我会将 CI 相关指令提取到独立小节、置于 local-only 标题之前，设为行为性主张 PR 的 Stage 2 必需元素，添加明确的跳过条件，修复 1e 的死指针，并编写位置性测试。本 PR 的做法与此完全一致。 发现： 无阻塞问题。 diff 干净且聚焦。旧的 2c 文本在同一个块中混合了 CI 路径和 local-only 指令——新的 2b-bis 小节将它们干净地分离，2c 的重写是两句话的重定向。作者写权限豁免原样保留。组装顺序更新将 1e 修复正确：旧文本写 "recommend E2E verification in tmux (Stage 2c)"——一个指向 CI agent 会跳过的小节的死指针。新文本同时点名两条车道并指向 2b-bis。作者无写权限回退中的 "(see Stage 2c)" 引用也已移除。 测试遵循既有模式（从文件读取 无 AGENTS.md 违规。无过度抽象、无重复、无范围蔓延。 CI 测试证据ubuntu 单元测试套件仍在运行中。macOS/Windows 测试和集成测试已跳过（skill 文本 + 纯测试变更的预期行为）。目前无失败。finalize 工作流将在 CI 结束后更新此表。 2b-bis： 跳过——这是 skill 文本变更（指令定位及其位置性测试），不是产品行为性主张。测试证实了位置性修复；真正的验证是作者描述的合入后测量（1/16 基线）。 — Qwen Code · qwen3.8-max-preview Reviewed at |
Collaborator
|
Confidence: 5/5 — clean across every stage; would merge without hesitation. The diagnosis is data-backed (1/16 measurement), the root cause is clearly identified (positional — CI agents skip a section headed "local invocation ONLY"), and the fix is the minimal positional correction: extract the CI-relevant instruction into its own 2b-bis section, fix the 1e dead pointer, update the assembly order. The tests are positional to match, whitespace-normalized against prettier reflows, and mutation-verified 4/4. The author-write carve-out is preserved verbatim. No scope creep — the 1e fix is the same defect class, caught while investigating mutation 3's contradictory landing check. The PR is honest about what the tests do and don't prove: they pin the instruction's presence and position, not the agent's resulting behavior. The real validation is the post-merge measurement against the 1/16 baseline. That's the right framing for a skill-text change. Approval deferred until CI lands green on 中文说明置信度：5/5 —— 每个阶段都干净，毫不犹豫可以合入。 诊断有数据支撑（1/16 测量），根因明确（位置性——CI agent 跳过标题为 "local invocation ONLY" 的小节），修复是最小的位置性纠正：将 CI 相关指令提取到独立的 2b-bis 小节，修复 1e 的死指针，更新组装顺序。测试与之匹配——位置性断言、空白归一化抗 prettier 重排、4/4 变异验证。作者写权限豁免原样保留。无范围蔓延——1e 修复是同一缺陷类，在调查变异 3 的矛盾落地检查时发现。 PR 诚实地说明了测试证明和不证明什么：它们钉住指令的存在和位置，而非 agent 由此产生的行为。真正的验证是合入后对 1/16 基线的测量。对于 skill 文本变更，这是正确的框架。 审批延迟至 CI 在 — Qwen Code · qwen3.8-max-preview Reviewed at |
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
中文说明
— qwen3.7-max via Qwen Code /review
doudouOUC
reviewed
Jul 28, 2026
doudouOUC
left a comment
doudouOUC
left a comment
Collaborator
There was a problem hiding this comment.
中文说明
— qwen3.7-max via Qwen Code /review
Collaborator
Author
|
@qwen-code /takeover |
Collaborator
Author
|
@qwen-code /takeover |
qwen-code-dev-bot
added
the
autofix/takeover
Collaborator
|
🤝 Takeover engaged: the autofix loop now manages this PR — it will address new review feedback and resolve base conflicts until the label is removed or the round cap is reached. Remove the 中文说明🤝 已接管：autofix 循环现在管理此 PR —— 将持续处理新的评审反馈与 base 冲突，直到移除标签或达到轮次上限。移除 |
Collaborator
|
🔄 Takeover re-armed: the round counter starts a fresh window (previous rounds no longer count toward the cap); management continues. 中文说明🔄 已重新武装：轮次计数开启新窗口（此前轮次不再计入上限），托管继续。 |
qwen-code-ci-bot
approved these changes
Jul 28, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
No issues found. LGTM! ✅
中文说明
未发现问题。LGTM！✅
— qwen3.7-max via Qwen Code /review
Collaborator
Collaborator
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下： Autofix review round: no code change warrantedThis round's feedback contains no code-review findings to address. Both reviews The cited CI failure is unrelated to this PR, and no in-scope change could
Local verification of what this PR actually changes:
Recommended next step: re-run the 中文说明Autofix 审查轮次：无需修改代码本轮反馈中没有需要处理的代码审查意见。两条 review 都是内容层面的批准（"Reviewed."），仅因为 CI 门禁红灯（ 所引用的 CI 失败与本 PR 无关，且没有任何范围内的改动能够影响它。证据如下：
针对本 PR 实际改动内容的本地验证：
建议的下一步：重新运行 Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧠 Handled by Qwen Code · model/模型 |
doudouOUC
approved these changes
Jul 28, 2026
doudouOUC
left a comment
doudouOUC
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed at HEAD 05637d7. Approve.
Full review performed with independent verification rather than re-reading the PR's own claims:
- Positional tests re-executed independently: I extracted the HEAD copies of
pr-workflow.mdand re-ran every assertion of the two new tests outside vitest — all 16 pass. Additionally, everyindexOfanchor the tests slice on (local invocation ONLY,Post a single Stage 2 comment,### Stage 3, the 2b-bis heading, the 1e/terminal-gate markers) occurs exactly once in the file at HEAD, so none of the slices can silently bind to the wrong region. - No content loss in the move: the old 2c preamble's trigger table, the AUTHOR-write requirement, and the external-contributor carve-out (disposable container / reproduce by hand) are all preserved in 2b-bis; 2c now delegates its CI-path signal to 2b-bis instead of carrying a dead conditional.
- The tests are wired into CI:
scripts/tests/qwen-triage-workflow.test.jsruns viatest:ci→test:scripts, andTest (ubuntu-latest, Node 22.x)is green on HEAD — so the mutation-verified guards are live, not decorative. - No stale pointers: remaining
Stage 2creferences inSKILL.mdandpr-workflow.mdall describe the still-local-only tmux lane, which is unchanged; Stage 1e's deadStage 2cpointer is gone and the section-scoped test pins that. - Prior blockers cleared: the
web-shell E2E Smokefailure that withheld the deferred approval was onef89b50; it passes on05637d7. No unresolved review threads.
The fix is exactly as scoped: positional, with positional tests that would have been red for the entire period the recommendation was dead. Nothing further to request.
中文总结
在 HEAD 05637d7 上完成全面审查，批准。
- 独立重跑了两个新增位置断言测试的全部 16 条断言（在 vitest 之外直接对 HEAD 文件内容执行），全部通过；且测试所依赖的每个
indexOf锚点字符串在文件中均只出现一次，切片边界无歧义。 - 原 2c 前言中的触发器表格、AUTHOR 写权限要求、外部贡献者兜底方案在 2b-bis 中完整保留，无信息丢失。
- 测试通过
test:ci→test:scripts实际在 CI 中运行，HEAD 上 Test job 已绿。 SKILL.md与pr-workflow.md中残留的Stage 2c引用均指向仍然存在的本地 tmux 通道，无失效指针。- 此前导致降级的
web-shell E2E Smoke失败发生在旧提交ef89b50，当前 HEAD 已通过；无未解决 review thread。
Collaborator
|
Released in v0.21.1. |
This was referenced Jul 29, 2026
doudouOUC
pushed a commit
to doudouOUC/qwen-code
that referenced
this pull request
Jul 29, 2026
…ger (QwenLM#7965) Post-merge measurement of QwenLM#7917, one day in: 9 eligible PRs, two considered-and-declined mentions (both correct calls), zero positive recommendations. The one clear behavioural candidate — QwenLM#7947, bounded reads of large text files — wrote "Not verified: Windows and Linux manual runs (author tested on macOS only)" in its own Stage 2 comment and never named a lane. That is the failure shape worth fixing: the judgement-based rule ("when neither static review nor 2b substantiates it") failed exactly where the comment had already written the gap down in so many words. The model judged that pending CI would cover it; a green suite proves the tests pass, not that the untested behaviour holds. So the trigger is now textual, not judgemental: before posting, grep your own draft. A sentence of the shape "not verified", "author tested on one platform only", or "author's claim, not independently re-run" IS the trigger — the 2b-bis line is that same sentence with the remedy attached, and omitting it means telling the maintainer what is missing while withholding the one command that would supply it. Pending CI does not lift the trigger. The two legitimate skip cases (nothing behavioural to settle; author lacks write) are unchanged, and the rule is ordered before them so they read as outs from the requirement, not the requirement as an out from them. The trigger phrases are verbatim from real comments: "not verified" and "author tested on macOS only" from QwenLM#7947, "author's claim, not independently re-run" from QwenLM#7951. Mutation-verified 3/3: dropping the trigger paragraph, moving it after the skip cases, and dropping the pending-CI sentence each turn the test red. n=1 is thin evidence for a behavioural rule change — but this rule is text-matching, not probability-weighing, so it cannot overfit to the sample that motivated it. Co-authored-by: wenshao <wenshao@example.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Makes the triage bot actually tell a maintainer when
@qwen-code /verify(or/tmux) would settle a claim it cannot settle itself.Measured on 2026-07-28 across 22 open PRs: of the 16 whose AUTHOR had write access and that already carried a triage comment, exactly 1 mentioned
/verify.The lane is not the problem — it has produced real, counted evidence on four PRs (#7829 2565 assertions, #7821 1347, #7830 27, #7881 17), three of them triggered by people other than its author. The problem is that almost nobody is told it exists.
Why it was 1/16
The instruction was there the whole time, as a conditional clause inside this section:
An agent running in CI reads a heading that says local invocation ONLY and an opening sentence that says Never in unattended CI, and skips the section. And even if it had read it, the Stage 2 assembly order allotted 2c a slot only "when one was driven locally" — on the CI path there was nowhere for the line to go.
The change
2b-bis, sited immediately after the CI-evidence step it follows from. 2b tells you the suite is green; it cannot tell you the suite pins the change. That is the gap, and naming the lane that closes it is now a required element of the Stage 2 comment for behavioural claims./verify" is noise, and noise is why the line got skipped.The worse half: high-risk paths
Stage 1e is the strongest triage-time signal in the skill — 10 of 31 reverted PRs touched those paths against 5 of 60 controls, p = 0.006. It carried the same dead pointer and recommended tmux alone, never naming
/verify:So the PRs statistically most likely to be reverted were the ones never offered the lane that proves a change is load-bearing. 1e now points at 2b-bis and names both lanes, with a note that this is where a green-but-unpinning suite is most expensive.
Why it's needed
/verifyanswers a question CI structurally cannot: do the PR's tests fail without the PR's code? On #7829 the answer was visible only because the lane ran base production code against HEAD test files — 69/75, the six failures being exactly the new assertions. A maintainer cannot get that from a green check, and will not go looking for a lane nobody mentions.Reviewer Test Plan
How to verify
The fix is positional, so the tests are positional. Asserting that the file mentions
/verifywould have passed throughout the entire period the recommendation was dead — that is precisely the vacuous test this defect deserves and does not get. The tests assert that the recommendation section appears before thelocal invocation ONLYheading, that it is reachable on an unattended run, that it survives in the assembly order, and that 1e names/verifyrather than pointing atStage 2c.Mutation-verified 4/4:
expected 33648 to be less than 33427(2b-bis)from the Stage 2 assembly orderStage 2cpointerexpected … to contain '@qwen-code /verify'Worth recording, because it nearly produced a wrong conclusion: mutation 3's landing check printed
landed: Falsewhile the test went red — a contradiction. The phrase it greps for occurs twice in the file (Stage 1e's pre-existing copy at line 249, and the new section's). The mutation removed only the second, so a whole-file check said "not applied" while the section-scoped assertion correctly failed. The mutation had landed; the check was scoped wrong. Chasing that contradiction is what surfaced the Stage 1e defect above, which was not in the original scope of this PR.Prose assertions are whitespace-normalised and shown to survive a maximal re-wrap (the harness joins every prose line in the section and re-runs), because prettier reflows this file and a test that goes red on a re-wrap teaches people to ignore it.
81 tests, 79 pass; prettier and eslint clean.
Evidence (Before & After)
Before — the measurement that motivated this,
author-write? / triage-comment? / mentions /verify?across open PRs:After: the same PRs would carry a line naming the trigger and the claim it settles, e.g.
Tested on
Skill text and its tests; no platform-dependent behaviour.
Risk & Scope
/verifyfrom/triageis deliberately not part of this PR. That needs reliability the lane does not yet have (4 of 8 terminal reports were usable; fix(triage): retry a transient npm ci before blaming the PR for it #7884 fixes one of the four failures) and a trigger policy for cost, since/verifycosts ~20 min of a self-hosted runner plus a full model budget per run. Recommending is the part that works today and is unaffected by the author-write gate, since permission is only needed to execute.main(20 chunks × 200 ms exceeds vitest's 5 s default) are fixed by fix(triage): make the build-process guard diagnosable and zombie-aware #7858, not duplicated here.Linked Issues
Follow-up to #7710 and #7884. No issues closed.
中文说明
本 PR 做了什么
让 triage bot 真正告诉维护者：什么时候
@qwen-code /verify（或/tmux）能解决它自己解决不了的问题。2026-07-28 实测 22 个 open PR：其中「作者有写权限且已有 triage 评论」的 16 个里，只有 1 个提到了
/verify。问题不在这条车道本身——它已在 4 个 PR 上产出真实的、可计数的证据（#7829 2565 条断言、#7821 1347 条、#7830 27 条、#7881 17 条），其中 3 个是它的作者之外的人触发的。问题是几乎没人被告知它的存在。
为什么是 1/16
指令一直都在，但它是一个条件从句，藏在这样一个小节里：
一个在 CI 中运行的 agent，看到标题写着 local invocation ONLY、首句写着 Never in unattended CI，就会跳过整节。而且即便读了也无处安放——Stage 2 的组装顺序只在「when one was driven locally」时才给 2c 留位置，CI 路径下根本没有挂载点。
改动
2b-bis小节，紧接在它所承接的 CI 证据步骤之后。2b 告诉你测试套件是绿的，但无法告诉你这套测试钉住了本次改动。这就是缺口，而点明填补该缺口的车道，现在是行为性主张 PR 的 Stage 2 评论的必需元素。更严重的另一半：高危路径
Stage 1e 是 skill 中最强的 triage 期信号——31 个被回滚的 PR 中有 10 个触及这些路径，对照组 60 个中只有 5 个，p = 0.006。它带着同样的死指针，而且更糟：只推荐 tmux，从不提
/verify：也就是说，统计上最可能被回滚的那批 PR，恰恰从未被推荐过那条能证明改动是承重的车道。1e 现在指向 2b-bis 并同时点名两条车道，并注明这里正是「测试全绿但没钉住改动」代价最高的地方。
为什么需要
/verify回答了一个 CI 结构上无法回答的问题：这个 PR 的测试，在没有这个 PR 的代码时会失败吗？ 在 #7829 上，答案之所以可见，是因为该车道把 base 的生产代码与 HEAD 的测试文件组合起来跑——69/75，失败的 6 条恰好是新增的断言。维护者无法从一个绿色的 check 得到这个结论，也不会去寻找一条没人提过的车道。评审验证方案
这个修复是位置性的，所以测试也必须是位置性的。 断言「文件中提到了
/verify」在这条推荐彻底失效的整段时间里都会通过——那正是这个缺陷理应得到、而本 PR 不给它的那种空洞测试。测试断言的是：推荐小节出现在local invocation ONLY标题之前、在无人值守运行下可达、存活于组装顺序中，以及 1e 点名/verify而非指向Stage 2c。4/4 变异验证：把 2b-bis 移回 local-only 标题之下（即原始 bug）→
expected 33648 to be less than 33427；从组装顺序中删除(2b-bis)→ 变红；删除作者写权限豁免 → 变红；把 1e 还原为只提 tmux 的Stage 2c指针 →expected … to contain '@qwen-code /verify'。值得记录，因为它差点导致错误结论：变异 3 的落地检查打印了
landed: False，而测试却变红了——这是矛盾的。它所 grep 的短语在文件中出现两次（Stage 1e 既有的一处在第 249 行，以及新小节中的一处）。变异只删除了后者，于是全文范围的检查报告「未应用」，而按小节切片的断言正确地失败了。变异确实落地了，是那个检查的范围写错了。 追查这个矛盾，正是发现上述 Stage 1e 缺陷的原因——而它原本不在本 PR 的范围内。散文类断言做了空白归一化，并实测能在最大程度重排后存活（测试脚手架把该小节所有散文行合并成长行后重跑），因为 prettier 会重排此文件，而一个因重排就变红的测试只会教人忽略它。
81 条测试、79 条通过；prettier 与 eslint 干净。
风险与范围
/verify单次约耗费 20 分钟自建 runner 加一整份模型预算。推荐是今天就能奏效、且不受作者写权限限制的那部分，因为权限只在真正执行时才需要。main上超时的两条 proxy-watchdog 测试（20 个数据块 × 200 ms 超过 vitest 默认的 5 秒）由 fix(triage): make the build-process guard diagnosable and zombie-aware #7858 修复，此处不重复。关联 Issue
#7710 与 #7884 的后续。不关闭任何 issue。