From 409e563b02692d23639278ee98cfe9f7f6f4a2ab Mon Sep 17 00:00:00 2001 From: wenshao Date: Wed, 29 Jul 2026 08:07:28 +0800 Subject: [PATCH] feat(triage): make the not-verified sentence a mechanical 2b-bis trigger MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-merge measurement of #7917, one day in: 9 eligible PRs, two considered-and-declined mentions (both correct calls), zero positive recommendations. The one clear behavioural candidate — #7947, bounded reads of large text files — wrote "Not verified: Windows and Linux manual runs (author tested on macOS only)" in its own Stage 2 comment and never named a lane. That is the failure shape worth fixing: the judgement-based rule ("when neither static review nor 2b substantiates it") failed exactly where the comment had already written the gap down in so many words. The model judged that pending CI would cover it; a green suite proves the tests pass, not that the untested behaviour holds. So the trigger is now textual, not judgemental: before posting, grep your own draft. A sentence of the shape "not verified", "author tested on one platform only", or "author's claim, not independently re-run" IS the trigger — the 2b-bis line is that same sentence with the remedy attached, and omitting it means telling the maintainer what is missing while withholding the one command that would supply it. Pending CI does not lift the trigger. The two legitimate skip cases (nothing behavioural to settle; author lacks write) are unchanged, and the rule is ordered before them so they read as outs from the requirement, not the requirement as an out from them. The trigger phrases are verbatim from real comments: "not verified" and "author tested on macOS only" from #7947, "author's claim, not independently re-run" from #7951. Mutation-verified 3/3: dropping the trigger paragraph, moving it after the skip cases, and dropping the pending-CI sentence each turn the test red. n=1 is thin evidence for a behavioural rule change — but this rule is text-matching, not probability-weighing, so it cannot overfit to the sample that motivated it. --- .qwen/skills/triage/references/pr-workflow.md | 14 ++++++++ scripts/tests/qwen-triage-workflow.test.js | 36 +++++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/.qwen/skills/triage/references/pr-workflow.md b/.qwen/skills/triage/references/pr-workflow.md index 9ba1488da3c..f9c60483961 100644 --- a/.qwen/skills/triage/references/pr-workflow.md +++ b/.qwen/skills/triage/references/pr-workflow.md @@ -499,6 +499,20 @@ skipped. The value is in naming what is currently **unsubstantiated**; a reader who disagrees with that assessment can ignore the line, which is a better outcome than never seeing it. +**Text trigger — this rule is mechanical, not a judgement call.** Before +posting, grep your own draft. If your comment contains a sentence of the +shape "not verified …", "author tested on only", "author's +claim, not independently re-run", or any other admission that a behavioural +claim rests on the author's word or a single environment — **that sentence +is the trigger.** You have already written down the gap; the 2b-bis line is +the same sentence with the remedy attached, and omitting it means telling +the maintainer what is missing while withholding the one command that would +supply it. "CI is still running" does not lift the trigger: a green suite +proves the tests pass, not that the untested behaviour holds. (Real miss +this rule exists for: a serve-side bounded-read change whose own Stage 2 +comment said "author tested on macOS only" and never named a lane — the +gap was written down and the remedy was not.) + **Skip it — explicitly — in exactly two cases:** - **No behavioural claim to settle**: docs, types, pure refactor with an diff --git a/scripts/tests/qwen-triage-workflow.test.js b/scripts/tests/qwen-triage-workflow.test.js index 10f63b14698..061a21c8e82 100644 --- a/scripts/tests/qwen-triage-workflow.test.js +++ b/scripts/tests/qwen-triage-workflow.test.js @@ -553,6 +553,42 @@ describe('qwen-triage tmux workflow', () => { expect(order).toContain('not an enrichment'); }); + it('makes the not-verified sentence a mechanical 2b-bis trigger', () => { + // Post-merge measurement of #7917 (2026-07-29): 9 eligible PRs, two + // considered-and-declined mentions, zero positive recommendations — and + // the one clear behavioural candidate (#7947, bounded reads) wrote + // "author tested on macOS only" in its own Stage 2 comment and never + // named a lane. The judgement-based rule ("when 2b cannot settle it") + // failed exactly where the comment had already written the gap down. So + // the trigger is now textual: the draft's own admission is the trigger, + // and pending CI does not lift it. + const section = prSkill + .slice( + prSkill.indexOf('#### 2b-bis.'), + prSkill.indexOf('#### 2c. Real-Scenario'), + ) + .replace(/\s+/g, ' '); + expect(section).toContain('mechanical, not a judgement call'); + expect(section).toContain('grep your own draft'); + // The trigger phrases are the ones real comments actually emit — the + // first two are verbatim from #7947's and #7951's Stage 2 comments. + expect(section).toContain('not verified'); + expect(section).toContain('author tested on only'); + expect(section).toContain('not independently re-run'); + // Pending CI must not lift the trigger: #7947's likely out was "the + // ubuntu Test job is still in progress". + expect(section).toContain('"CI is still running" does not lift'); + // The mechanical rule must sit BEFORE the skip cases, or the skip cases + // read as outs from it rather than the other way around. + expect(section.indexOf('mechanical, not a judgement call')).toBeLessThan( + section.indexOf('Skip it — explicitly'), + ); + // And the skip cases themselves must survive — the trigger tightens the + // rule, it does not replace the two legitimate outs. + expect(section).toContain('No behavioural claim to settle'); + expect(section).toContain('The author lacks write access'); + }); + it('names /verify on the high-risk paths, not just tmux', () => { // 1e is the strongest triage-time signal in the skill (10 of 31 reverted // PRs touched these paths vs 5 of 60 controls, p = 0.006) — and it used