Skip to content

🍒 [v1.207] fix(test): narrow 4 case grader regexes that caused nightly false positives - #5000

Merged
rockymadden merged 1 commit into
release/v1.207from
fix-case-grader-regexes-nov18-CP-to-release-v1-207
Nov 19, 2026
Merged

rockymadden merged 1 commit into
release/v1.207from
fix-case-grader-regexes-nov18-CP-to-release-v1-207

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Summary

The 2026-11-18 nightly (run 2026-11-18_04-23-44, commit b79a350a) dropped case from 98% to 91% (72/79). Four of the seven failures are grader false positives: the agent did the right thing, and the regex penalized it.

  • Rule 14 guard (artifact-safety-guard and golden-rebuild-cm-expense). Both tasks now carry one identical pattern, and a test pins them equal.
    • On main, the safety guard matched sed -i and python open(w) on any file, so a sed -i on the .case.ts source failed. The .case.ts is the authored file (SKILL.md step 5), so a scripted edit of it is not an artifact write.
    • On main, the golden guard used [^\n]*, which crossed ; and &&. Example: sed -i '...' X.case.ts; cp X/caseplan.case ....
    • The artifacts are caseplan.json, caseplan.case, registry-resolved.json, bindings_v2.json and entry-points.json.
    • The guard catches:
      • an in-place edit: sed, perl or ruby -i, or awk -i inplace, on an artifact named directly or through a shell variable (F=…caseplan.json; sed -i … $F)
      • a Python, node or Ruby write whose target names the artifact: open(…, <mode>), where the mode is any writable literal (w, a, x, r+, rb+, wb+ and so on). The call can have up to two nested calls and can pass the mode as keyword mode=. A mode held in a variable is not read. Also: Path(…).write_* and Path(…).open('w'), fs.write*, fs.append*, fs.promises.*, File.write
      • a write through a variable that was assigned an artifact path. The assignment can be a literal, an f-string, os.path.join, a concatenation, a backtick string or Path(...). The write can be open(v, <mode>) (also with str(v), os.fspath(v) or Path(v) inside), v.write_text, Path(v).write_*, writeFileSync(v, or a copy or move whose whole destination is v. The variable must stand alone, so tmp does not match p, and the assignment and the write must fall in the same grader window. Main caught these with "mentions the artifact + any write". A write through a variable that never held an artifact path, and a write to p+'.bak', are not caught. A reused name (p='caseplan.json'; …; p='/tmp/s'; open(p,'w')) still counts as a write, because the regex cannot see the reassignment.
      • a copy or move whose destination is the artifact: shutil.copy/move, os.replace/rename, fs.rename/copyFile/cp, and shell cp/mv/install when the final argument is the artifact (or $F holding it), optionally followed by a redirect, and no directory in that path is .flow-sdk/, backup(s)/, bak/, orig/ or original/
    • Quoted sed/perl scripts are consumed whole, so s/a/b/; s/c/d/ and & back-references do not end the command. Quoted paths, options before -i, --in-place, and Python heredocs are caught.
    • These are not caught, on purpose:
      • a read of the plan followed by a write to a literal scratch path, such as open('/tmp/x','w')
      • a nested read inside a scratch write
      • a copy of the plan to a backup
    • An in-place edit on a glob that can reach an artifact (P/*.json, P/*) is caught, and so is a find whose -name is an artifact name or *.json/*.case and that runs -exec sed -i. A source glob is not caught: src/*.ts, or find -name '*.ts'.
    • Against main, over the fixtures (71 edit commands, 42 legitimate commands), scored with a copy of coder-eval's _match_haystacks and _search_windows: in the safety task, main catches no edit that this head misses, this head catches 31 more, and false fails drop from 19 to 0. In the golden task, main catches no edit that this head misses, this head catches 54 more, and false fails drop from 6 to 0.
    • Two read-only nightly runs that the earlier head false-failed now pass: a Python variable named cp, and the scratch file test-caseplan.case.
    • Known misses, listed in DOCUMENTED_MISSES: a separate script file, shell redirection, and a variable assigned and written more than one grader window apart. A copy onto a project that sits under a folder named backup/ (or the other backup names) is also skipped, because the skip applies to any directory in the path. In the safety task, the redirection/helper-script criterion still catches python fix.py, > caseplan.json and tee. The golden task has no such criterion; this PR does not change that.
  • diagnose-faulted-instance-runtime. The CLI verb is instances (plural), and the regex required instance. Fix: instances? in the read and the write guard. Both now accept a quoted "$UIP".
  • connector-wait-trigger. The regex expected maestro registry pull. The SDK loop runs maestro case registry pull. Fix: accept both.

Verification

  • tests/tasks/uipath-maestro-case/_shared/test_rule14_guard_patterns.py loads each pattern from its task YAML and scores it the way the grader does: coder-eval's search windows (the first 2000 characters, then line-packed windows) and their shlex-normalized forms, copied from _match_haystacks and _search_windows. Contents:
    • should-match fixtures: the review's true positives
    • should-not-match fixtures: the nightly false positive, reads, and read-then-scratch-write
    • the documented misses
    • a linear-time check on long commands
  • Falsifier: against each previous head, the fixtures fail 9, 23, 11, 14, 22, then 22 times.
  • pytest tests/tasks/uipath-maestro-case/ tests/scripts/test_tenant_catalog.py -q: 717 passed, 2 skipped, 11 subtests passed.

Test plan

  • CI passes
  • Next nightly recovers the 4 tasks. This needs a live run.

🤖 Generated with Claude Code

https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q


Automatically cherry-picked from #4937 to release/v1.207 by @rockymadden.

…itives (#4937)

* fix(test): narrow 4 case grader regexes that caused false positives in the 2026-11-18 nightly

artifact-safety-guard: regex matched sed/python on any file; anchor to
caseplan.(json|case) only — editing .case.ts source is legitimate.

golden-rebuild-cm-expense: regex crossed compound-command boundaries
(;/&&), matching a `cp caseplan.case` in a later segment after a `sed -i`
targeting .case.ts. Scope [^\n]* → [^;\n&]* so each alternation stays
within its command segment.

diagnose-faulted-instance-runtime: CLI verb is `instances` (plural);
regex used `instance` (singular). Accept both with `instances?`.

connector-wait-trigger: regex expected `maestro registry pull` but the
CLI verb is `maestro case registry pull` — `case` is always in the path.
Factor `case\s+` out of the alternation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): use lookahead for python/node Rule 14 regex in safety-guard

The post-pattern caseplan.(json|case) anchor fails when caseplan.json is
the first argument to open() — the write-mode match consumes past it. A
lookahead checks for the filename anywhere in the command before matching
the write pattern, so both `open("caseplan.json", "w")` and
`Path("caseplan.case").write_text(...)` are caught.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): restore bare maestro registry pull/prepare in connector-wait regex

The previous commit narrowed the regex to require `case` before `registry`,
losing the `maestro registry pull/prepare` path that the corpus test and
some agents use. Add both branches: `case registry (pull|prepare)` and
bare `registry (pull|prepare)`.

Fixes the test_connector_resolution_criteria_accept_either_loop CI failure.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): make Rule 14 sed/perl guards quote-aware and restore artifact set

[^;\n&]* ended the match at a ; or & inside a quoted sed/perl script, so
multi-expression scripts and & back-references on caseplan.json passed as
clean. The safety guard also lost registry-resolved.json. Consume quoted
strings whole, stop only at an unquoted ; & |, and guard the same artifact
set as the sibling redirection criterion. Add should-match/should-not-match
fixtures loaded from the task YAMLs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* test(case): record Rule 14 citations from the new guard fixtures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): share one Rule 14 guard across both case tasks and close quoted-path gap

The quote-aware loop consumed a quoted path whole, so sed -i ... "P/caseplan.json"
passed as clean. Let the match stop inside a quoted path, allow options before -i
(sed -E -i, sed -e ... -i, perl -p -i), accept --in-place but not --quiet, require
the artifact name to end the token (caseplan.case.ts is source), and add python
heredoc and ruby -e branches. The golden task now carries the same pattern as the
safety task; a test pins them equal and lists the known indirect-path misses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): require the Rule 14 write call itself to name the artifact

The interpreter branches matched a read of caseplan.json plus any write on the
same line, so a read-then-scratch-write one-liner failed the guard. Match only
open(...artifact..., 'w'), Path(...artifact...).write_*, fs.write*(...artifact)
and File.write(...artifact), drop the interpreter prefix, accept heredocs
without '-', and bound each argument window so long commands stay linear.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): catch copy/replace writes and quoted "$UIP" in case guards

Rule 14 guard: catch shutil.copy/move, os.replace/rename, fs.promises and
fs.copyFile/cp when the destination names an artifact, and ruby -i. A copy
of the plan to a backup stays allowed. Require a node single-target write
to name the artifact in its first argument, so a nested read is not a hit.
Diagnose guards: accept a quoted "$UIP". Runtime-built paths stay listed
as documented misses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): catch Rule 14 writes through a variable or a nested path call

The literal-path branches missed the common edit shape: assign the plan
path to a variable, read it, write it back through the same variable.
Main caught it; this head did not. Add a variable-path branch for python,
node and sed -i $F, and accept os.path.join, keyword mode and
Path(...).open('w'). Read-then-scratch-write stays allowed. Bound the
sed/perl token loop so repeated unquoted tokens stay linear.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): restore main's recall for built paths and catch copies onto the plan

After an artifact mention, a write whose target is not a string literal
(a variable, an f-string, os.path.join, Path(f)) is a hit again, as on
main; a write to a literal scratch path stays allowed. Allow two nesting
levels inside open(...). Catch shutil/os/cp/mv/install when the artifact
is the destination; a copy of the plan to a backup stays allowed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): catch glob and find -exec in-place edits that can reach the plan

With these two branches, no fixture edit that main's guard caught is
missed at this head (54 edits, 0 main-only, 21 head-only). A glob of
source files such as src/*.ts stays allowed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): stop treating source finds and plan backups as Rule 14 writes

find -name '*.ts' -exec sed -i matched because -name accepted any glob;
it now needs an artifact name or *.json / *.case. A copy counted when the
artifact appeared anywhere in its arguments; it now counts only when the
final argument is the artifact outside a backup location, and $F only as
the destination.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): exclude only backup directories from the copy-destination guard

The exclusion tested the whole destination path, so a project under /tmp
or a directory containing "orig" hid every cp onto the plan. Exclude only
a .flow-sdk/, backup(s)/, bak/, orig/ or original/ component, and let a
trailing redirect (2>&1, > /dev/null) follow the destination.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): treat every writable open() mode as a Rule 14 write

The mode check accepted only a leading w, a or x, so open(..., 'r+') and
'wb+' on the plan passed; main caught both. Match any short mode string
that contains w, a, x or +, for literal paths, variables and Path.open.
'r', 'rb' and 'rt' reads stay allowed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): count a write through a variable only when it holds an artifact path

The non-literal-target branch flagged any write after a plan read, so a
read plus a scratch write through a variable (out='/tmp/s.txt') became a
new false fail in the golden task. Drop that branch. The variable-path
branch now accepts built assignments (f-string, join, concatenation,
backtick, Path), Path(v) writes and copies onto v, so recall against main
is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): bound the tracked variable and its write destination

A write through tmp matched a tracked p because nothing bounded the name,
and shutil.copy(p, p+'.bak') counted as a write onto p. Require the
variable to stand alone, and to be the whole destination argument of a
copy, move or writeFile. Widen the lookahead from 2000 to 8000 characters
so a long script still links the assignment to its write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* test(case): shorten Rule 14 guard test names to 30 characters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): match the grader's window and normalization; drop two nightly false fails

A Python variable named cp read as the cp command in the shlex-normalized
copy, and \b let test-caseplan.case read as caseplan.case; both failed
read-only nightly runs. Refuse an assignment after the copy word and any
of [\w.-] before an artifact name. The fixtures now go through a copy of
coder-eval's _match_haystacks (first 2000 characters plus the normalized
copy), so none can claim a hit the grader cannot see; the lookahead
window returns to 2000 to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): catch open(str(p)) on a tracked path; mirror the grader's later windows

open(str(p),'w') with p=Path('...caseplan.json') escaped both guards;
main caught it. Accept str(), os.fspath() and Path() around a tracked
variable inside open(). Drop the duplicated lookbehind in the shell-copy
branch. The fixture matcher now also searches coder-eval's later windows
(logical-line packing, 256-character overlap on one long line).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

* fix(test): pair the wrapper parentheses around a tracked variable in open()

The optional closing paren was independent of the optional wrapper, so
print(open(p), 'w') read as open(p, 'w'). Match either the bare variable
or a complete str()/os.fspath()/Path() call, then the mode comma.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013tiDkBvodTad2Je4FVfz4q

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@rockymadden
rockymadden merged commit 3fa565f into release/v1.207 Nov 19, 2026
3 checks passed
@rockymadden
rockymadden deleted the fix-case-grader-regexes-nov18-CP-to-release-v1-207 branch November 19, 2026 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants