feat(triage): make the not-verified sentence a mechanical 2b-bis trigger - #7965
Merged
Conversation
Post-merge measurement of #7917, one day in: 9 eligible PRs, two considered-and-declined mentions (both correct calls), zero positive recommendations. The one clear behavioural candidate — #7947, bounded reads of large text files — wrote "Not verified: Windows and Linux manual runs (author tested on macOS only)" in its own Stage 2 comment and never named a lane. That is the failure shape worth fixing: the judgement-based rule ("when neither static review nor 2b substantiates it") failed exactly where the comment had already written the gap down in so many words. The model judged that pending CI would cover it; a green suite proves the tests pass, not that the untested behaviour holds. So the trigger is now textual, not judgemental: before posting, grep your own draft. A sentence of the shape "not verified", "author tested on one platform only", or "author's claim, not independently re-run" IS the trigger — the 2b-bis line is that same sentence with the remedy attached, and omitting it means telling the maintainer what is missing while withholding the one command that would supply it. Pending CI does not lift the trigger. The two legitimate skip cases (nothing behavioural to settle; author lacks write) are unchanged, and the rule is ordered before them so they read as outs from the requirement, not the requirement as an out from them. The trigger phrases are verbatim from real comments: "not verified" and "author tested on macOS only" from #7947, "author's claim, not independently re-run" from #7951. Mutation-verified 3/3: dropping the trigger paragraph, moving it after the skip cases, and dropping the pending-CI sentence each turn the test red. n=1 is thin evidence for a behavioural rule change — but this rule is text-matching, not probability-weighing, so it cannot overfit to the sample that motivated it.
Collaborator
|
✅ Qwen Triage finished — CI landed green on ✅ Qwen Triage 已完成 —— |
Collaborator
|
Thanks for the PR! Template looks good ✓ Problem: observed and data-backed. The post-merge measurement of #7917 shows the 2b-bis lane went from "never read" to "considered but still not written" — and the one clear behavioural candidate in the window (#7947) wrote "author tested on macOS only" in its own Stage 2 comment yet never named a lane. That's a concrete miss with evidence, not a theoretical concern. Direction: aligned. This is a follow-up to #7917 (already merged), tightening the triage skill's instruction text. Internal process improvement, no product surface or public contract impact. Size: not applicable — no core paths touched. Approach: the scope feels right. Converting a judgement-based condition to a text-triggered rule is the minimal fix for a judgement failure — you can't reword a judgement rule to fix a judgement miss, but you can make the draft's own admission the trigger. 14 lines of prose + one test that pins presence, trigger phrases, pending-CI clause, and position relative to the skip cases. Nothing to cut. Risk: no elevated risk signals. Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 问题：已观测且有数据支撑。#7917 合入后的测量显示 2b-bis 车道从"从未被读到"进步到"被考虑但仍未被写出"——窗口内唯一清晰的行为性候选（#7947）在自己的 Stage 2 评论里写了 "author tested on macOS only"，却从未点名任何车道。这是一个有证据的具体漏判，不是理论性问题。 方向：对齐。这是 #7917（已合入）的后续，收紧 triage skill 的指令文本。内部流程改进，不涉及产品表面或公共契约。 规模：不适用——未触及核心路径。 方案：范围合理。把基于判断的条件转为文本触发规则，是修复判断失效的最小改动——判断规则的失效无法靠改写判断规则来修复，但可以让草稿自身的承认成为触发器。14 行散文 + 一个测试，钉住规则存在、触发短语、pending-CI 条款、以及相对于跳过条件的位置。没有可砍的部分。 风险：无升级风险信号。 进入代码审查 🔍 — Qwen Code · qwen3.8-max-preview Reviewed at |
Collaborator
|
Code review Independent proposal: given that the judgement-based 2b-bis condition fails exactly where the comment has already written the gap down, I would add a text-triggered rule — "if your draft contains 'not verified' / 'author tested on X only' / similar admissions, that sentence is the trigger" — placed before the skip cases so they read as exemptions, and pin it with a test that checks presence, trigger phrases, and position. The PR matches this exactly. The new paragraph sits between the "value is in naming what is unsubstantiated" prose and the "Skip it" section — correct placement, and the test asserts The test follows the established pattern in the file (extract section → normalize whitespace → assert on phrases and ordering), and the PR body documents 3/3 mutation verification with landing proofs. The skip cases are preserved and asserted. No correctness issues, no convention violations, nothing to flag. CI test evidence Final CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行（取最新一次运行），省略 skipped，失败项排在最前。 The ubuntu suite is still running. macOS and Windows tests are skipped (expected for a skill-text + test-only change). Will be updated by the finalize workflow once CI settles. Sandboxed lane: no behavioural claim to settle — this PR changes triage skill instruction text and its test, not product behaviour. Real-scenario testing: N/A — unattended CI run; no user-visible product behaviour. 中文说明代码审查 独立方案：鉴于基于判断的 2b-bis 条件恰恰在评论已写下缺口的地方失效，我会添加一条文本触发规则——"如果草稿包含 'not verified' / 'author tested on X only' / 类似承认，那个句子就是触发器"——放在跳过条件之前使其读作豁免，并用测试钉住存在性、触发短语和位置。 PR 与此完全一致。新段落位于 "value is in naming what is unsubstantiated" 和 "Skip it" 之间——位置正确，测试断言 CI 测试证据：ubuntu 套件仍在运行，macOS/Windows 跳过（skill 文本 + 测试改动，符合预期）。将由 finalize workflow 在 CI 完成后更新。 沙箱车道：无行为性主张可证——本 PR 修改的是 triage skill 指令文本及其测试，非产品行为。 真实场景测试：N/A——无人值守 CI 运行；无用户可见的产品行为。 — Qwen Code · qwen3.8-max-preview Reviewed at |
Collaborator
|
Confidence: 5/5 — clean across every stage; would merge without hesitation. This is exactly the kind of follow-up that makes a process improvement stick. #7917 added the 2b-bis lane recommendation as a judgement call, and the post-merge data showed it was being considered but still not written where it mattered most — on the PR whose own comment admitted the gap. The fix is mechanical: make the draft's own admission the trigger. You can't overfit a text-matching rule to the sample that motivated it, and the failure mode it adds (an unnecessary 2b-bis line on a trivial "not verified") is the cheap direction. The implementation is minimal — one paragraph of instruction text placed before the skip cases (position is load-bearing and the test pins it), plus a test that follows the established pattern and was mutation-verified 3/3. Nothing to cut, nothing missing. Approval deferred until CI lands green on 中文说明置信度：5/5 —— 每个阶段都干净；毫不犹豫地合入。 这正是让流程改进真正生效的那种后续。#7917 把 2b-bis 车道推荐作为判断条件加入，合入后的数据显示它被考虑了，却仍然没有在最需要的地方被写出——在那个自己的评论都承认了缺口的 PR 上。修复是机械的：让草稿自身的承认成为触发器。文本匹配规则不会对促成它的样本过拟合，而它新增的失败模式（在琐事 "not verified" 上多一行 2b-bis）是代价低的方向。 实现是最小的——一段指令文本放在跳过条件之前（位置是承重的，测试钉住了它），加上一个遵循已有模式、经 3/3 变异验证的测试。没有可砍的，也没有遗漏。 审批推迟至 CI 在 — Qwen Code · qwen3.8-max-preview Reviewed at |
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
approved these changes
Jul 29, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
qwen-code-ci-bot
approved these changes
Jul 29, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
No issues found. LGTM! ✅
中文说明
未发现问题。LGTM！✅
— qwen3.7-max via Qwen Code /review
doudouOUC
approved these changes
Jul 29, 2026
doudouOUC
left a comment
doudouOUC
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed at HEAD 409e563. Approve.
Verified independently against the HEAD tree, not the PR's own claims:
- All 9 assertions of the new test re-executed outside vitest against the HEAD
pr-workflow.md— all pass. Both slice anchors (#### 2b-bis.,#### 2c. Real-Scenario) occur exactly once in the file, so the section slice cannot bind to the wrong region, and the paragraph sits exactly where claimed: after the "noise" paragraph, before the skip cases (noise < trigger < skipconfirmed by index). - The rule composes cleanly with the existing outs. The two skip cases are byte-identical to what #7917 landed, the trigger paragraph precedes them, and the test pins both the position and the survival of both outs — a "not verified" sentence on a no-write-access PR still resolves through skip case 2, so the mechanical trigger cannot force a guaranteed-denial recommendation.
- The diff is purely additive (+50/-0) on top of #7917's merged blob (
9ba1488base confirmed); no cross-reference elsewhere in the skill orSKILL.mdrefers to the internal structure of 2b-bis, so nothing else needed updating. - The trigger phrases are real, not invented: "Not verified: Windows and Linux manual runs (author tested on macOS only)" appears verbatim in #7947's Stage 2 comment, and #7947 names no lane — the motivating miss is checkable and checks out.
- CI fully green on HEAD, including
Test (ubuntu-latest, Node 22.x)which runs this test file viatest:scripts; no unresolved review threads;MERGEABLE.
The design call is right: a judgement rule that failed precisely where the gap was already written down in prose is not fixable by more prose, and a text-match trigger has the asymmetric failure mode the PR names — a spurious line is ignorable, an unwritten one is not.
中文总结
在 HEAD 409e563 上完成全面审查，批准。
- 新测试的全部 9 条断言在 vitest 之外对 HEAD 文件独立重跑，全部通过；两个切片锚点在文件中均只出现一次，段落位置（"noise"段之后、跳过条件之前）经索引比较确认。
- 规则与既有豁免条件组合自洽：两个跳过条件与 #7917 合入时逐字一致，触发段落位于其前，测试同时钉住位置与两个豁免的存续——无写权限 PR 上的 "not verified" 句子仍走豁免二，机械触发不会导致必然被拒的推荐。
- diff 纯增量（+50/-0），基于 #7917 已合入的 blob（
9ba1488基线已确认）；技能内无其他位置引用 2b-bis 内部结构，无需联动修改。 - 触发短语取自真实评论可核验：#7947 的 Stage 2 评论逐字含有 "author tested on macOS only" 且全文未点名任何车道，动机成立。
- HEAD 上 CI 全绿（含运行本测试文件的 Test job）；无未解决 thread；MERGEABLE。
设计判断正确：判断型规则恰在缺口已被白纸黑字写下的地方失效，靠改写措辞无法修复；文本匹配触发的失败方向是廉价的——多余的一行可以被忽略，从未写出的一行无法被执行。
wenshao
pushed a commit
that referenced
this pull request
Jul 29, 2026
#7965 landed on main and edits the same 2b-bis anchor this branch does. Both sides are kept: main's mechanical text-trigger paragraph verbatim, and this branch's header change — with the sponsored lane the second case is no longer a skip but an adjustment ("in one case, and adjust it in another"). The merge also surfaced two places still describing the ephemeral-VM design this branch abandoned when external runs moved onto ECS. One was caught by a test that pinned the old wording ("ephemeral GitHub-hosted runner"); the other, in the Stage 1e high-risk-paths bullet, had no test and was found by grepping the branch for the stale term. Both now name what actually bounds an external run: the pinned head, the risk screen, and the workspace wipe. The test asserts the controls and rejects "ephemeral" outright, so this cannot drift back silently. 93/93 tests; prettier, eslint and actionlint clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Turns the 2b-bis sandboxed-lane recommendation from a judgement call into a text-triggered rule: if the Stage 2 draft itself contains a "not verified / author tested on one platform only" sentence, that sentence is the trigger.
The measurement that motivated it
#7917 merged 2026-07-28T14:28Z. Re-running its own acceptance criterion one day later, on every Stage 2 comment posted since the merge (write-access authors only):
The two mentions are both considered-and-declined — and both are correct calls (#7952 is a lockfile defect the diff itself disproves; #7951 is CI-workflow behaviour the product lanes cannot exercise). That is real progress over the baseline, where the section was simply never read.
But the one clear behavioural candidate in the window — #7947,
fix(serve): allow bounded reads of large text files— wrote this in its own Stage 2 comment:and never named a lane (grep count 0).
Why this failure shape gets a rule and not more prose
The judgement-based condition — "when neither static review nor 2b substantiates it" — failed exactly where the comment had already written the gap down in so many words. The model plausibly judged that the still-running CI would substantiate the claim; a green suite proves the tests pass, not that the untested behaviour holds. No rewording of a judgement rule fixes a judgement failure. A text rule does:
The trigger phrases are verbatim from real comments: "not verified" and "author tested on macOS only" from #7947, "author's claim, not independently re-run" from #7951.
The two legitimate skip cases (nothing behavioural to settle; author lacks write) are unchanged, and the rule is placed before them, so they read as outs from the requirement rather than the requirement as an out from them.
Why it's needed
The point of #7917 was that a maintainer cannot act on a line that was never written. The post-merge data shows the line is now considered but still not written where it matters most — on the PR whose own comment admits the gap. This closes that residual, mechanically.
Reviewer Test Plan
How to verify
The test pins the rule's presence, its trigger phrases, the pending-CI clause, and its position before the skip cases — position is load-bearing, because a rule placed after the outs reads as subordinate to them.
Mutation-verified 3/3, each with landing proof before the result was read:
to contain 'mechanical, not a judgement call'expected 2484 to be less than 1818to contain '"CI is still running" does not lift'89/89 tests pass (the two proxy-watchdog timeouts are gone from
mainsince #7858 merged); prettier and eslint clean. Prose assertions are whitespace-normalised, same as the sibling tests, so a prettier re-wrap cannot redden them.Evidence (Before & After)
Before — #7947's Stage 2 comment, the miss this rule exists for:
(no lane named anywhere in the comment)
After — the same draft self-triggers, and the required line is the admission with the remedy attached:
Tested on
Skill text and its tests; no platform-dependent behaviour.
Risk & Scope
/verifyremains out of scope — unchanged from feat(triage): surface the sandboxed lanes on the CI path #7917's reasoning.Linked Issues
Follow-up to #7917 (its post-merge measurement is the motivation). Related to #7710. No issues closed.
中文说明
本 PR 做了什么
把 2b-bis 的沙箱车道推荐从主观判断改为文本触发的规则：如果 Stage 2 评论草稿自己就写了"未验证 / 作者仅在单一平台测试"这类句子，那个句子本身就是触发器。
促成本次改动的测量
#7917 于 2026-07-28T14:28Z 合入。一天后，按它自己写下的验收标准，对合入后发布的全部 Stage 2 评论（仅限有写权限的作者）重跑同一测量：
那 2 次提及都是**「考虑后明确否决」**——且各自的判断是对的（#7952 是 diff 本身即可证伪的 lockfile 缺陷；#7951 是产品车道无法验证的 CI workflow 行为）。相比基线（该小节根本不被读到），这是真实的进步。
但窗口内唯一清晰的行为性候选——#7947
fix(serve): allow bounded reads of large text files——在自己的 Stage 2 评论里写着：却从头到尾没有点名任何车道（grep 计数为 0）。
为什么这种失败形态需要一条规则而不是更多措辞
基于判断的条件——"当静态评审与 2b 都无法证实时"——恰恰在评论已经把缺口白纸黑字写下来的地方失效了。 模型大概率判断"还在跑的 CI 会证实它"；而绿色的测试套件只证明测试通过，不证明未测试的行为成立。判断规则的失效无法靠改写判断规则来修复，文本规则可以：
触发短语全部逐字取自真实评论：#7947 的 "not verified" 与 "author tested on macOS only"，#7951 的 "author's claim, not independently re-run"。
两个正当的跳过条件（无行为性主张可证；作者无写权限）保持不变，且规则被放在它们之前——让跳过条件读作规则的豁免，而不是让规则读作跳过条件的豁免。
为什么需要
#7917 的立意是：维护者无法根据一行从未被写出的字采取行动。合入后的数据显示，这行字现在会被考虑，却仍然没有在最需要的地方被写出——在那个自己的评论都承认了缺口的 PR 上。本 PR 以机械规则关闭这个残余缺口。
评审验证方案
测试钉住规则的存在、触发短语、pending-CI 条款、以及它位于跳过条件之前的位置——位置是承重的：放在豁免之后的规则读起来就从属于豁免。
3/3 变异验证，每项在读取结果前先证明改动落地：删除触发段落（标记消失 →
to contain 'mechanical, not a judgement call'）；移到跳过条件之后（索引比较翻转 →expected 2484 to be less than 1818）；仅删除 pending-CI 句（标记消失 →to contain '"CI is still running" does not lift'）。89/89 测试通过（#7858 合入后
main上那两条 proxy-watchdog 超时已消失）；prettier 与 eslint 干净。散文断言与同类测试一样做了空白归一化，prettier 重排不会使其变红。测试平台
macOS ✅；Windows N/A；Linux N/A——skill 文本及其测试，无平台相关行为。
风险与范围
/verify依旧不在范围内——理由与 feat(triage): surface the sandboxed lanes on the CI path #7917 不变。关联 Issue
#7917 的后续（其合入后测量即本 PR 的动机）。与 #7710 相关。不关闭任何 issue。