Repository navigation
🍒 [v1.207] fix(test): narrow 4 case grader regexes that caused nightly false positives - #5000
Merged
rockymadden merged 1 commit intoNov 19, 2026
Conversation
…itives (#4937) * fix(test): narrow 4 case grader regexes that caused false positives in the 2026-11-18 nightly artifact-safety-guard: regex matched sed/python on any file; anchor to caseplan.(json|case) only — editing .case.ts source is legitimate. golden-rebuild-cm-expense: regex crossed compound-command boundaries (;/&&), matching a `cp caseplan.case` in a later segment after a `sed -i` targeting .case.ts. Scope [^\n]* → [^;\n&]* so each alternation stays within its command segment. diagnose-faulted-instance-runtime: CLI verb is `instances` (plural); regex used `instance` (singular). Accept both with `instances?`. connector-wait-trigger: regex expected `maestro registry pull` but the CLI verb is `maestro case registry pull` — `case` is always in the path. Factor `case\s+` out of the alternation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): use lookahead for python/node Rule 14 regex in safety-guard The post-pattern caseplan.(json|case) anchor fails when caseplan.json is the first argument to open() — the write-mode match consumes past it. A lookahead checks for the filename anywhere in the command before matching the write pattern, so both `open("caseplan.json", "w")` and `Path("caseplan.case").write_text(...)` are caught. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): restore bare maestro registry pull/prepare in connector-wait regex The previous commit narrowed the regex to require `case` before `registry`, losing the `maestro registry pull/prepare` path that the corpus test and some agents use. Add both branches: `case registry (pull|prepare)` and bare `registry (pull|prepare)`. Fixes the test_connector_resolution_criteria_accept_either_loop CI failure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): make Rule 14 sed/perl guards quote-aware and restore artifact set [^;\n&]* ended the match at a ; or & inside a quoted sed/perl script, so multi-expression scripts and & back-references on caseplan.json passed as clean. The safety guard also lost registry-resolved.json. Consume quoted strings whole, stop only at an unquoted ; & |, and guard the same artifact set as the sibling redirection criterion. Add should-match/should-not-match fixtures loaded from the task YAMLs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * test(case): record Rule 14 citations from the new guard fixtures Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): share one Rule 14 guard across both case tasks and close quoted-path gap The quote-aware loop consumed a quoted path whole, so sed -i ... "P/caseplan.json" passed as clean. Let the match stop inside a quoted path, allow options before -i (sed -E -i, sed -e ... -i, perl -p -i), accept --in-place but not --quiet, require the artifact name to end the token (caseplan.case.ts is source), and add python heredoc and ruby -e branches. The golden task now carries the same pattern as the safety task; a test pins them equal and lists the known indirect-path misses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): require the Rule 14 write call itself to name the artifact The interpreter branches matched a read of caseplan.json plus any write on the same line, so a read-then-scratch-write one-liner failed the guard. Match only open(...artifact..., 'w'), Path(...artifact...).write_*, fs.write*(...artifact) and File.write(...artifact), drop the interpreter prefix, accept heredocs without '-', and bound each argument window so long commands stay linear. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): catch copy/replace writes and quoted "$UIP" in case guards Rule 14 guard: catch shutil.copy/move, os.replace/rename, fs.promises and fs.copyFile/cp when the destination names an artifact, and ruby -i. A copy of the plan to a backup stays allowed. Require a node single-target write to name the artifact in its first argument, so a nested read is not a hit. Diagnose guards: accept a quoted "$UIP". Runtime-built paths stay listed as documented misses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): catch Rule 14 writes through a variable or a nested path call The literal-path branches missed the common edit shape: assign the plan path to a variable, read it, write it back through the same variable. Main caught it; this head did not. Add a variable-path branch for python, node and sed -i $F, and accept os.path.join, keyword mode and Path(...).open('w'). Read-then-scratch-write stays allowed. Bound the sed/perl token loop so repeated unquoted tokens stay linear. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): restore main's recall for built paths and catch copies onto the plan After an artifact mention, a write whose target is not a string literal (a variable, an f-string, os.path.join, Path(f)) is a hit again, as on main; a write to a literal scratch path stays allowed. Allow two nesting levels inside open(...). Catch shutil/os/cp/mv/install when the artifact is the destination; a copy of the plan to a backup stays allowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): catch glob and find -exec in-place edits that can reach the plan With these two branches, no fixture edit that main's guard caught is missed at this head (54 edits, 0 main-only, 21 head-only). A glob of source files such as src/*.ts stays allowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): stop treating source finds and plan backups as Rule 14 writes find -name '*.ts' -exec sed -i matched because -name accepted any glob; it now needs an artifact name or *.json / *.case. A copy counted when the artifact appeared anywhere in its arguments; it now counts only when the final argument is the artifact outside a backup location, and $F only as the destination. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): exclude only backup directories from the copy-destination guard The exclusion tested the whole destination path, so a project under /tmp or a directory containing "orig" hid every cp onto the plan. Exclude only a .flow-sdk/, backup(s)/, bak/, orig/ or original/ component, and let a trailing redirect (2>&1, > /dev/null) follow the destination. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): treat every writable open() mode as a Rule 14 write The mode check accepted only a leading w, a or x, so open(..., 'r+') and 'wb+' on the plan passed; main caught both. Match any short mode string that contains w, a, x or +, for literal paths, variables and Path.open. 'r', 'rb' and 'rt' reads stay allowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): count a write through a variable only when it holds an artifact path The non-literal-target branch flagged any write after a plan read, so a read plus a scratch write through a variable (out='/tmp/s.txt') became a new false fail in the golden task. Drop that branch. The variable-path branch now accepts built assignments (f-string, join, concatenation, backtick, Path), Path(v) writes and copies onto v, so recall against main is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): bound the tracked variable and its write destination A write through tmp matched a tracked p because nothing bounded the name, and shutil.copy(p, p+'.bak') counted as a write onto p. Require the variable to stand alone, and to be the whole destination argument of a copy, move or writeFile. Widen the lookahead from 2000 to 8000 characters so a long script still links the assignment to its write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * test(case): shorten Rule 14 guard test names to 30 characters Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): match the grader's window and normalization; drop two nightly false fails A Python variable named cp read as the cp command in the shlex-normalized copy, and \b let test-caseplan.case read as caseplan.case; both failed read-only nightly runs. Refuse an assignment after the copy word and any of [\w.-] before an artifact name. The fixtures now go through a copy of coder-eval's _match_haystacks (first 2000 characters plus the normalized copy), so none can claim a hit the grader cannot see; the lookahead window returns to 2000 to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): catch open(str(p)) on a tracked path; mirror the grader's later windows open(str(p),'w') with p=Path('...caseplan.json') escaped both guards; main caught it. Accept str(), os.fspath() and Path() around a tracked variable inside open(). Drop the duplicated lookbehind in the shell-copy branch. The fixture matcher now also searches coder-eval's later windows (logical-line packing, 256-character overlap on one long line). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q * fix(test): pair the wrapper parentheses around a tracked variable in open() The optional closing paren was independent of the optional wrapper, so print(open(p), 'w') read as open(p, 'w'). Match either the bare variable or a complete str()/os.fspath()/Path() call, then the mode comma. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
github-actions
Bot
requested review from
a team,
StephenHanzlik,
Yiqi517,
abhiram-vad,
dmetzgar,
roberts-cliff,
song-zhao-25 and
tmatup
as code owners
November 19, 2026 15:30
2 tasks
dmetzgar
approved these changes
Nov 19, 2026
rockymadden
deleted the
fix-case-grader-regexes-nov18-CP-to-release-v1-207
branch
November 19, 2026 20:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The 2026-11-18 nightly (run
2026-11-18_04-23-44, commitb79a350a) dropped case from 98% to 91% (72/79). Four of the seven failures are grader false positives: the agent did the right thing, and the regex penalized it.sed -iandpython open(w)on any file, so ased -ion the.case.tssource failed. The.case.tsis the authored file (SKILL.md step 5), so a scripted edit of it is not an artifact write.[^\n]*, which crossed;and&&. Example:sed -i '...' X.case.ts; cp X/caseplan.case ....caseplan.json,caseplan.case,registry-resolved.json,bindings_v2.jsonandentry-points.json.sed,perlorruby -i, orawk -i inplace, on an artifact named directly or through a shell variable (F=…caseplan.json; sed -i … $F)open(…, <mode>), where the mode is any writable literal (w,a,x,r+,rb+,wb+and so on). The call can have up to two nested calls and can pass the mode as keywordmode=. A mode held in a variable is not read. Also:Path(…).write_*andPath(…).open('w'),fs.write*,fs.append*,fs.promises.*,File.writeos.path.join, a concatenation, a backtick string orPath(...). The write can beopen(v, <mode>)(also withstr(v),os.fspath(v)orPath(v)inside),v.write_text,Path(v).write_*,writeFileSync(v, or a copy or move whose whole destination isv. The variable must stand alone, sotmpdoes not matchp, and the assignment and the write must fall in the same grader window. Main caught these with "mentions the artifact + any write". A write through a variable that never held an artifact path, and a write top+'.bak', are not caught. A reused name (p='caseplan.json'; …; p='/tmp/s'; open(p,'w')) still counts as a write, because the regex cannot see the reassignment.shutil.copy/move,os.replace/rename,fs.rename/copyFile/cp, and shellcp/mv/installwhen the final argument is the artifact (or$Fholding it), optionally followed by a redirect, and no directory in that path is.flow-sdk/,backup(s)/,bak/,orig/ororiginal/s/a/b/; s/c/d/and&back-references do not end the command. Quoted paths, options before-i,--in-place, and Python heredocs are caught.open('/tmp/x','w')P/*.json,P/*) is caught, and so is afindwhose-nameis an artifact name or*.json/*.caseand that runs-exec sed -i. A source glob is not caught:src/*.ts, orfind -name '*.ts'._match_haystacksand_search_windows: in the safety task, main catches no edit that this head misses, this head catches 31 more, and false fails drop from 19 to 0. In the golden task, main catches no edit that this head misses, this head catches 54 more, and false fails drop from 6 to 0.cp, and the scratch filetest-caseplan.case.DOCUMENTED_MISSES: a separate script file, shell redirection, and a variable assigned and written more than one grader window apart. A copy onto a project that sits under a folder namedbackup/(or the other backup names) is also skipped, because the skip applies to any directory in the path. In the safety task, the redirection/helper-script criterion still catchespython fix.py,> caseplan.jsonandtee. The golden task has no such criterion; this PR does not change that.instances(plural), and the regex requiredinstance. Fix:instances?in the read and the write guard. Both now accept a quoted"$UIP".maestro registry pull. The SDK loop runsmaestro case registry pull. Fix: accept both.Verification
tests/tasks/uipath-maestro-case/_shared/test_rule14_guard_patterns.pyloads each pattern from its task YAML and scores it the way the grader does: coder-eval's search windows (the first 2000 characters, then line-packed windows) and their shlex-normalized forms, copied from_match_haystacksand_search_windows. Contents:pytest tests/tasks/uipath-maestro-case/ tests/scripts/test_tenant_catalog.py -q: 717 passed, 2 skipped, 11 subtests passed.Test plan
🤖 Generated with Claude Code
https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q
Automatically cherry-picked from #4937 to
release/v1.207by @rockymadden.