Skip to content

fix(maestro-flow): keep Extract document out of an SDK-sourced project - #4635

Open
cezara98t wants to merge 6 commits into
mainfrom
fix/extract-document-eval-failures
Open

cezara98t wants to merge 6 commits into
mainfrom
fix/extract-document-eval-failures

Conversation

@cezara98t

@cezara98t cezara98t commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Run: claude-code, Opus 5.5, canonical skill, score 0.0. The agent read extract-document-node.md in full, then edited PermitDesk.flow in place and left .flow-sdk/PermitDesk.flow.ts stale. It even warned the user not to compile the source, because the next compile deletes the node. That is the damage the separate-project rule exists to prevent.

Why the rule lost:

  1. It sat as prose under "Adding the node", below the numbered loop (steps 0–7) the agent follows. Step 0, "Find the flow", never mentioned .flow-sdk/.
  2. SKILL.md says a project that already exists is edited through its .flow, with no exception. It covered Extract document only for a new flow ("drop --sdk-source").
  3. The rule gave no wording for the user, who asked for a step in their flow, so a separate project read as ignoring the request.

Changes:

  • Loop step 0: a project with a supplied .flow-sdk/<Name>.flow.ts stops here for a new node, which goes in a separate project. A cache-only .flow-sdk/ (bindings.json, connectors/, connectors-local/) and a flow that already holds the node are edited in place.
  • "Adding the node": the prose becomes steps. Leave the project untouched, flow init <NewName> without --sdk-source, then tell the user why.
  • SKILL.md: the Extract document sentence names the supplied-source case. It is folded into the existing line because the "Project layout" section is at its byte budget.

All three passages are inside blocks the studioweb flavor replaces (extract-find-flow, extract-sdk-mixing, sdk-project-layout). Studio Web output is unchanged, which is intended. There the .flow is authoritative and .flow-sdk/ is a per-turn draft, so the flavor deliberately edits in place.

Proof

sdk_project: not reproduced locally. It needs a coder-eval run. The evidence is the failing run's transcript: the agent read the rule, then edited in place. No passing-run claim yet, so a rerun of skill-flow-extract-document-sdk-project on the claude-code agent is needed before merge.

Checks run locally:

  • npm run skills:validate and skills:build: OK. The studioweb tree has none of the new text.
  • test_skill_size_budget.py and test_brownfield_turn_rules.py: 21 passed ("Project layout" at 8,497 of 8,500 bytes).
  • check-sensitive-content.py: OK.
  • The flavor contract test file reports 11 failures. All are npm-pack/publish transaction tests plus one uipath-planner test, all failing on a trim of undefined from npm-pack output. None touch uipath-maestro-flow, and they look tied to the local npm.

Not run here: the coder-eval tasks themselves (eval sandbox and tenant).

The copy_taxonomy criterion fix moved to #4650.

Follow-ups, not in this PR

  1. Keep sdk_project off the Studio Web agent. It fails there too (studio-web, gpt-5-6-luna, studioweb flavor), but correctly: the flavor tells the agent to edit the .flow in place. The repo has no way to exclude a task from a host, and the Studio Web runner picks tasks outside this repo. The tag vocabulary is closed, so this needs agreement with the runner owner. A candidate is a flat marker such as coding-agent-only in tests/README.md § Tag Taxonomy, honoured by the runner.
  2. Document processor provisioning. extract-document-node.md still says "Nothing in the CLI provisions one" and tells the user to save in Studio Web. uip 1.207 provisions a processor on nodes schema set. The tests on test/extract-document-processor already assume it does.

🤖 Generated with Claude Code

cezara98t and others added 2 commits November 12, 2026 11:35
…t; grade copy_taxonomy on the key it must not copy

sdk_project failed on the coding agent: the agent read the separate-project
rule and still edited PermitDesk.flow in place, leaving .flow-sdk/PermitDesk.flow.ts
stale so the user's next compile deletes the node. The rule sat as prose
below the numbered loop the agent follows, and SKILL.md's "existing project:
edit its .flow" line carried no exception for it. Put the .flow-sdk/ check
in loop step 0, turn the rule into steps (leave the project, init a new one
without --sdk-source, tell the user why), and name the case in SKILL.md.
All three passages sit in blocks the studioweb flavor replaces, so Studio
Web, where the .flow is authoritative, is unchanged.

copy_taxonomy asserted Extract Credit Note has no processorKey. A CLI that
provisions a processor on `nodes schema set` gives it a fresh key of its
own, which is correct; assert instead that it is not Extract Invoice's key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… not only a copied one

The inequality check passed any key that was not Extract Invoice's, so a
made-up GUID with no processor behind it graded as correct. A helper now
accepts no key, or a key an idpProcessor resource in the solution declares.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @cezara98t's task in 53s —— View job


Coder-eval task lint (advisory)

1 task YAML changed. Verdict counts: 0 Critical, 0 High, 0 Medium, 1 Low, 0 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

  • ✅ Accepted, with a caveat. The author shows the failing run's artifact (task.json: 4/5 criteria passed, and only the replaced processorKey criterion failed) now passing the new helper: "OK: extractCreditNote1 has its own processor". That is enough to infer the task passes as a whole. It is still an offline replay, not a fresh coder-eval run of skill-flow-extract-document-copy-taxonomy. Please consider adding a line after a rerun, for example: Ran skill-flow-extract-document-copy-taxonomy on claude-code and it passed. (The sdk_project rerun the PR body already calls for is about a skill change. Its YAML is unchanged, so this lint does not cover it.)

Per-task lint

tests/tasks/uipath-maestro-flow/extract-document/copy_taxonomy/copy_taxonomy.yaml: verdict Low

Issues:

  • [Low] Could pass for the wrong reason: the helper (copy_taxonomy_helper.py:29-38, used at yaml L80-86) accepts any fresh key that has a matching *.json under BillingDocs/resources/ with kind: idpProcessor. An agent that invents a GUID and hand-writes a stub resource file therefore passes. Hand-editing like that is unlikely: the prompt doesn't steer toward it, and the other four criteria are strong. Still, the "synthesised key fails" guarantee only holds against an agent that doesn't fake the resource too.

Suggested fixes:

  • Make the resource check stricter if you can do it cheaply. For example, require the fields the CLI writes on provisioning (resource.name or the processor-specific spec fields alongside kind/key), or check that the key also appears in bindings_v2.json / the solution's package resource. Either is harder to fake than a two-field stub.
  • Optional: add an advisory command_executed (pass_threshold: 0) on uip\s+maestro\s+flow\s+nodes\s+schema\s+set. It records whether the copy went through the CLI path the skill teaches, without gating on it.

Other axes are clean:

  • Self-report anti-pattern: none. Every criterion reads BillingDocs.flow or the sandbox resources.
  • Prompt over-specification: none. The prompt states only the goal (L28-29).
  • Meaningful coverage: strong. A shared copies checker, json_check on what must be kept, brownfield preservation, and validate --no-warnings.
  • Validate-only flow tests: this is an integration-tier task. The description (L14-15) explains why there is no flow debug ("extracts nothing until the flow is saved once in Studio Web"), which drops the finding to Low under the carve-out. It was already like this before this PR, so it is not scored against this change.
  • Near-duplicate: none. The siblings (add_line_items, rename_field, fix_rejected_taxonomy, …) test different operations.
  • CLI verb reachability: N/A. The task has no command_executed criteria.
  • Sandbox pins and run limits: none. run_limits: is top-level.

The helper itself checked out: the fixture has no idpProcessor resource, so a copied Invoice key fails the resource check even if its case is changed. That closes the case-sensitive == comparison at L47.

Within-PR duplicates

  • No duplicate clusters detected (only one task changed).

Conclusion

  • ⚠ 1 task has issues, max severity Low. Advisory only, not blocking merge.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @cezara98t's task in 1m 20s —— View job


Summary

This PR fixes two failing uipath-maestro-flow Extract document evals. For sdk_project, the skill change makes the ".flow-sdk/ means use a separate project" rule an explicit step: it's added to loop step 0, written as numbered steps under "Adding the node", and named in SKILL.md. For copy_taxonomy, a json_check that assumed no processor provisioning is replaced by a helper. The helper passes a node with no key or with its own provisioned key, and fails a node that copied Invoice's key or made one up.

Change-by-Change Review

1. skills/uipath-maestro-flow/references/extract-document-node.md (step 0 and "Adding the node")

OK

  • The rule now sits in the loop the agent actually follows (step 0, line 44) and links to the full procedure instead of repeating it. That counts as bridge content.
  • Steps 1–3 under "Adding the node" are prescriptive and leave nothing open:
    • "Leave that project as it is" matches the sdk_project criterion (preserved nodes, byte-identical .flow.ts).
    • The flow init <NewName> command without --sdk-source matches the shape in SKILL.md.
    • Step 3 gives the agent wording for the user.
  • Both edits are inside extract-find-flow and extract-sdk-mixing, which skill-flavors/studioweb/.../extract-document-node.md replaces. So Studio Web output is unchanged, which is intended.

2. skills/uipath-maestro-flow/SKILL.md:46

OK. A one-clause addition to an existing line, with a link. It fits the byte-budget argument in the PR description.

3. copy_taxonomy.yaml and copy_taxonomy_helper.py

OK, with one Low note

  • $REFERENCE_DIR is ../.., i.e. tests/tasks/uipath-maestro-flow, so extract-document/copy_taxonomy/copy_taxonomy_helper.py resolves.
  • The resource lookup matches the CLI's resource shape (resource.kind / resource.key). It matches kind case-insensitively and walks BillingDocs/resources recursively, like the checker on origin/test/extract-document-processor.
  • Comparing against Invoice's key is case-sensitive while the resource lookup lowercases. This is harmless: the fixture's Invoice key (3f6b2a9e…, BillingDocs.flow:226) has no idpProcessor resource, so a copy that differs only in case still fails at the resource check.
  • Low: _declared_processor_keys() duplicates the processor-resource lookup already on origin/test/extract-document-processor (_shared/check_extract_document.py, around line 385). When that branch merges, replace this helper with the shared checker so the two can't drift.

What's Missing

  • Medium: no passing-run claim for skill-flow-extract-document-sdk-project. The PR description says so itself. Per .claude/rules/test-writing.md, rerun it on the claude-code agent and add the result before merge. The skill fix is only proven by that rerun.
  • Medium (follow-up, already noted in the PR): the reference now contradicts the test. extract-document-node.md:185 says "Nothing in the CLI provisions one", and the :236 row tells the user to save in Studio Web. The copy_taxonomy grading now accepts CLI provisioning on nodes schema set. Track this so the reference gets updated once uip 1.207 behaviour is confirmed.
  • Nothing else identified: no CODEOWNERS or registry changes are needed (no skill folder was added or renamed), and the scope stays within the uipath-maestro-flow skill and its tests.

Area Ratings

Area Status Notes
Frontmatter OK Not touched
E2E Tests Issue sdk_project has no passing-run claim yet
Skill Body OK Rule moved into the numbered loop; flavor blocks respected
References & Assets OK Links resolve (#adding-the-node, document-pipeline.md, extract-document-taxonomy.md#starting-from-a-flow-in-studio-web)
Repo Hygiene OK No secrets or personal paths; the GUIDs are fixture data

Issues for Manual Review

  • Does a separate project answer the user's request? The user asked for the step in their flow. Step 3 explains why it isn't there, but says nothing about connecting the new project back to PermitDesk (for example, as a called flow). A domain owner should decide whether that belongs in this guidance.
  • Is processor provisioning stable? Confirm that uip 1.207 provisioning on nodes schema set is the intended, lasting behaviour. The helper's "no key also passes" branch keeps older CLIs green.
  • Studio Web host exclusion for sdk_project (follow-up 1) needs the runner owner to agree. Nothing in this repo can do it today.

Conclusion

Approve once the sdk_project rerun passes. The skill change targets the root cause shown in the failing transcript and keeps the studioweb flavor unchanged. The new copy_taxonomy criterion is stricter than the old one where it counts: it still rejects copied keys and also catches synthesised ones. The only blocker is the missing passing run. Fold the helper into the shared checker after test/extract-document-processor lands.

cezara98t and others added 3 commits November 12, 2026 12:14
…l extract with

A node extracts with the processor its processorKey names, and that
processor holds its own copy of the taxonomy. Checking only that the key is
not Invoice's, or that a resource declares it, passed a processor still
holding the old schema. The helper now also requires the provisioned
resource's prompt and output schema to equal the copied ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, drop the helper

Validate already refuses a keyless node and a second author, and schema set
and pack keep the resource in step with the node, so only a copied
read-only key is left to check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cezara98t cezara98t changed the title fix(maestro-flow): keep Extract document out of an SDK-sourced project; copy_taxonomy grades the processor it must not copy fix(maestro-flow): keep Extract document out of an SDK-sourced project Nov 12, 2026

@rockymadden rockymadden left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Commented · Fix the open blocker before merge.

What

Moves the Extract document SDK-source guard ahead of the editing loop and makes the separate-project instruction explicit.

Axes

Axis Blockers Non-blockers Top finding
Correctness & logic 1 0 Distinguish source ownership from SDK caches · skills/uipath-maestro-flow/references/extract-document-node.md:44
Code quality & simplicity 0 0 none
Types & contracts 0 0 none
Tests & validation 0 1 Validate the affected SDK-project task
Security 0 0 none
Architecture & design 0 0 none
Error handling & resilience 0 0 none
Interface, docs & compatibility 0 0 none

Overall findings

  • [non-blocker][Tests & validation] Validate the affected SDK-project task

The changed behavior targets sdk_project, but that task is absent from the passing34-task CI artifact and the body states there is no passing run for it. Run that task with a coding agent after the source guard is fixed, including a cache-only project case.

Validation

Read both changed files, project-layout/binding guidance, task context and matching Studio Web overrides. Head-matching smoke CI passed34tasks but did not include sdk_project. Studio Web replaces all three touched guidance blocks. No local tests, builds or live service writes performed.

tl;dr: 1 blocker, 1 non-blocker.

<!--skill-flavor:extract-find-flow:start-->
0. **Find the flow.** Everything here edits a `.flow` on disk. When the user describes a flow they are already building and none exists locally, they may be referring to one
that lives in Studio Web and has to be pulled down first — ask them for its URL or solution id, since nothing lists them ([Starting from a flow in Studio Web](extract-document-taxonomy.md#starting-from-a-flow-in-studio-web)).
that lives in Studio Web and has to be pulled down first — ask them for its URL or solution id, since nothing lists them ([Starting from a flow in Studio Web](extract-document-taxonomy.md#starting-from-a-flow-in-studio-web)). **Where the project folder holds `.flow-sdk/`, do not edit its `.flow`:** that flow is compiled from TypeScript source, so the node goes in a separate project ([Adding the node](#adding-the-node)), and every later step runs on that project's `.flow`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker][Correctness & logic] Distinguish source ownership from SDK caches

A .flow-sdk/ directory can hold only bindings and connector caches; SKILL.md calls it the SDK work directory, and registry prepare writes there. Its presence does not prove a maintained .flow.ts owns this Flow. This new step-0 guard also applies before edits to an existing extraction node, so a CLI-authored Flow with caches gets redirected to another project instead of receiving the requested edit. Check actual source ownership and cover a cache-only directory; do not treat every .flow-sdk/ as authoritative TypeScript.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6b0d86d. The redirect now keys on a supplied source, the same test SKILL.md and brownfield.md use for "A supplied source": a .flow-sdk/<Name>.flow.ts the task handed over, not one decompiled for an edit. It applies only to adding a new node.

  • A .flow-sdk/ holding only bindings.json, connectors/ or connectors-local/ is called out as the SDK's cache: edit the .flow as usual.
  • A flow that already holds an Extract document node is edited in place, so an edit to an existing node is no longer redirected.

Step 0, the "Adding the node" paragraph and the SKILL.md sentence all use the same test. The Studio Web flavor replaces all three, so its output is unchanged.

…urce

A `.flow-sdk/` directory alone does not mean TypeScript owns the flow: it
also holds the SDK's caches (`bindings.json`, `connectors/`,
`connectors-local/`), and a decompile for an edit writes a working
`<Name>.flow.ts` there too. The redirect now applies only to a project whose
`.flow-sdk/<Name>.flow.ts` the task supplied, and only to a new node. A
cache-only `.flow-sdk/` and a flow that already holds the node are edited
in place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants