Skip to content

test(maestro-flow): copy_taxonomy expects Credit Note's own processor - #4650

Closed
cezara98t wants to merge 2 commits into
mainfrom
test/copy-taxonomy-processor-key
Closed

cezara98t wants to merge 2 commits into
mainfrom
test/copy-taxonomy-processor-key

Conversation

@cezara98t

@cezara98t cezara98t commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

copy_taxonomy fails on main since the CLI provisions processors: "Extract Credit Note has no processorKey" (claude-code, Opus 5.5, CLI 1.207.0-dev.9816, score 0.86). The agent copied the taxonomy correctly, and nodes schema set gave Credit Note a key of its own, which is right.

The criterion now requires Credit Note to end up with a processorKey that is not Extract Invoice's. copy_taxonomy.yaml is the same as in #4570, which carries the skill side of processor provisioning, so this fixes the task on main now and the two merge cleanly in either order.

On the failing run's artifact the new assertion passes. On edited copies of the fixture it fails a keyless Credit Note, Invoice's key, and Invoice's key as a read-only mirror. Split from #4635.

🤖 Generated with Claude Code

The CLI now provisions a processor on nodes schema set, so Extract Credit
Note getting a key of its own is correct. validate refuses a keyless node
and a second author, so only a copied read-only key is left to check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @cezara98t's task in 45s —— View job


Coder-eval task lint (advisory)

1 task YAML changed. Verdicts: 0 Critical, 0 High, 0 Medium, 1 Low, 0 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

  • ✅ Author confirms passing run: "Regraded the failing run's artifact with the new assertion: it passes, and the run's other four criteria had passed."

Per-task lint

tests/tasks/uipath-maestro-flow/extract-document/copy_taxonomy/copy_taxonomy.yaml: verdict Low

Issues:

  • [Low] Validate-only flow tests miss correctness: this is an integration task with no flow debug. The description gives the reason at L11-12 ("a CLI-authored Extract document node extracts nothing until the flow is saved once in Studio Web"), so the severity drops from Medium to Low. This issue was already there before this PR.
  • [Low] Meaningful coverage: L81 hard-codes Extract Invoice's key 3f6b2a9e-…f713, copied from _fixtures/BillingDocs/BillingDocs/BillingDocs.flow:226. If someone changes the fixture's key, this assertion keeps passing and stops catching a copied key. The new assertion also passes when Credit Note has no key at all. That gap is covered only because validate_flow.py --no-warnings (L97) rejects a keyless node, so L81 and L97 now depend on each other.

Suggested fixes:

  • Add a comment at L81 noting that the literal must match BillingDocs.flow:226. Alternatively, have check_extract_document.py read the source node's key from the fixture and assert the two keys differ.
  • Optional: add a second assertion on L77-85 that nodes[?id=='extractCreditNote1'] | [0].inputs.processorKey is non-null. Then "gets a processor of its own" (L7-8) is graded directly rather than through flow validate.

The diff itself looks right. I checked the fixture: the hard-coded key appears only once, on Extract Invoice, so Credit Note starts with no key. check_brownfield_preserved.py (L89) also guarantees Invoice keeps that key, so the assertion compares against the correct value. Shell approval was missing, so I couldn't run check-cli-verbs.py. The two CLI commands are run through run_command helpers, not command_executed, so that axis has nothing to check anyway.

Within-PR duplicates

  • No duplicate clusters detected. Only one task changed.

Conclusion

  • ⚠ 1 task has issues, max severity Low. Advisory only, not blocking merge.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @cezara98t's task in 45s —— View job


Summary

This PR relaxes one copy_taxonomy criterion. It used to fail any processorKey on Extract Credit Note. Now it fails only when that key is Extract Invoice's (3f6b2a9e-…f713). The CLI now gives a node its own processor on nodes schema set, so a key of its own is correct.

Change-by-Change Review

1. tests/tasks/uipath-maestro-flow/extract-document/copy_taxonomy/copy_taxonomy.yaml

OK

  • Literal checked: 3f6b2a9e-8c41-4d7a-9b25-61e0c4d8f713 matches _fixtures/BillingDocs/BillingDocs/BillingDocs.flow:226, which is Extract Invoice's inputs.processorKey. It is the only processorKey in the fixture, so the assertion targets exactly "copied the source's key".

  • Expression (line 81): '…' is a valid JMESPath raw-string literal, and && with == works inside a filter. The expression returns 0 in three cases:

    • Credit Note has no key.
    • Credit Note has a different key.
    • Credit Note is missing.

    The missing case is already caught by the copy check at weight 3.0 and the preservation check.

  • Overlap with validate: the PR body says flow validate already refuses a second node writing to the same processor. If so, a copied key fails both this check and the validate_flow.py --no-warnings criterion. The overlap is fine: this check names the exact mistake, which makes failures easier to diagnose.

  • Prose: the updated description and criterion text match the new behaviour and the repo's terse style. Weight (1.5), pass_threshold, tags and task_id are unchanged and valid.

What's Missing

Nothing identified. Nothing under skills/ changed, so CODEOWNERS and the registries don't apply.

Area Ratings

Area Status Notes
Frontmatter OK No skill files changed
E2E Tests OK Assertion is narrower and still pins the real mistake
Skill Body OK —
References & Assets OK —
Repo Hygiene OK The UUID is the fixture's own processor key, not a secret; change is limited to one task file

Issues for Manual Review

  • The new criterion relies on CLI 1.207.0-dev creating a processor on nodes schema set, and on flow validate refusing keyless nodes. Older CLIs aren't covered: a run there that leaves Credit Note without a key now passes this criterion. It should still fail validate.
  • I didn't run the task. The regrade of the failing run is the author's claim in the PR body.

Conclusion

Approve. The change is small and correct: the fixture's key and the expression both check out.

Credit Note must end up with a key of its own: schema set provisions one for a
node that has none. The file now matches #4570's, so the two merge cleanly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cezara98t cezara98t changed the title test(maestro-flow): copy_taxonomy fails only a copied processorKey test(maestro-flow): copy_taxonomy expects Credit Note's own processor Nov 12, 2026
@cezara98t

Copy link
Copy Markdown
Contributor Author

fixed in #4570

@cezara98t cezara98t closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant