Skip to content

test(uipath-maestro-flow): short prompts and effect graders for the billing discrepancy and dispute resolution tasks - #4903

Open
astridstaicu wants to merge 10 commits into
mainfrom
test/flow-billing-sw-ga-prompts
Open

astridstaicu wants to merge 10 commits into
mainfrom
test/flow-billing-sw-ga-prompts

Conversation

@astridstaicu

@astridstaicu astridstaicu commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

This changes the two existing path-to-ga Flow tasks in place, keeping the same task IDs. It adds no tests.

Task Change
multi_node/billing_discrepancy_detector Short prompt; graders check what the debug run did
multi_node/billing_dispute_resolution Short prompt; looser structure check; two effect runs; new invoice PDF fixture (MCS-2026-04903)

The prompts are what a Studio Web user would type: a business request, a short <!-- helper --> block, and one line saying the run is headless. They replace the step-by-step build specs. The outcome-graded tag is added to both.

What the graders check

The main criteria grade what the run did, read from the flow debug trace, rather than what the flow file contains.

Detector

  • flow validate passes.
  • One grader debug run, in which:
    • the BillingDisputeERP query, the BillingDisputeCRM query and their merge each completed;
    • the outputs, read by name, are totalOvercharge=1610, discrepancyCount=1, matchedInvoiceNumber=MCS-2026-04872, accountTier=Enterprise. The old check searched every value, so the invoice number also matched the flow's own input.
  • Advisory: the agent ran its own uip maestro flow debug, plus the existing bindings and structural checks.

Resolution

  • flow validate passes.

  • Structure:

    • trigger → IxP → ERP/CRM → analyst (SOP index attached) → FinancialPostingFunction → writer → end;
    • a rejection path that skips posting;
    • no HITL node and no send-email node.

    It no longer pins the reference's switch + decision layout, because "automatically approve manager review" can be built without them.

  • The SOP index's resource.json sidecar exists (unchanged check).

  • Overcharge run, a real billing error on the tenant: invoice MCS-2026-04903, line 2, 90 h billed at $290 against a contracted $235, with a matching invoice PDF added in this PR.

    • IxP and both lookups complete.
    • FinancialPostingFunction completes, and its live reply cites MCS-2026-04903.
    • The writer completes.
    • The outputs include the SOP's credit of 4950 and a text citing MCS-2026-04903 and that credit.
  • No-discrepancy run (MCS-2026-04872 line 4, billed exactly at contract): the run completes without running FinancialPostingFunction or the writer, and returns an explanation.

  • Advisory: the agent ran its own debug.

Why the "agent ran its own test" check is advisory: on Studio Web the agent may test with the RunProject proxy tool, which the eval testbench stubs, and its CLI debug can't attach the PDF.

Why the test data comes from the tenant: the tenant's live Billing Dispute SOP decides a rate dispute by comparing the ERP row's billed unitPrice with its contractedRate. The SOP and ERP fixture files in coder_eval_uipath describe different data. MCS-2026-04872 line 5 is billed at the contracted $185, so it is not an overcharge on the tenant.

Why there's no "record was created" check: FinancialPostingFunction writes nothing. Its provisioning source (coder_eval_uipath SolutionSources/BillingDispute/FinancialPostingFunction/Workflow.json) only echoes the invoice. The email is drafted, not sent.

No longer in the task: _shared/advisory_billing_dispute_resolution.py, the check for five fixed output names, because the prompt stops naming the outputs. The file and its tests are left in place.

Testing

  • pytest on the resolution grader tests, test_flow_check.py and test_same_ground_advisories.py: 315 passed.
  • Offline regression of the new structure check on real agent-built flows: it passes all 7 flows the old check passed (claude-code ×2, codex ×2, studio-web ×2, antigravity, all from November runs). It also passes the Studio Web short-prompt flow, which the old check rejects because there is no core.logic.switch.
  • An earlier Studio Web run of these prompts as separate tasks (adhoc-2026-11-18_09-28-17): detector 1.0. The resolution graders crashed on cross-folder imports, which are gone now that everything sits in the task's own folder.
  • check-task-driver.py and check-task-host-paths.py pass.
  • Live runs on every harness: pending, results to follow on this PR.

🤖 Generated with Claude Code

astridstaicu and others added 4 commits November 18, 2026 11:04
…repancy and dispute resolution flows

Adds two autopilot-sw-ga tasks that ask the path-to-ga BillingDiscrepancyDetector
and BillingDisputeResolution scenarios the way a Studio Web user would: a
business request, a short helper block, no build instructions.

Graders check what the run did, read from the debug trace:
- detector: the ERP and CRM queries and their merge completed; the four named
  outputs carry the live values; the agent ran its own debug.
- resolution: a valid overcharge is posted through FinancialPostingFunction and
  gets a draft citing the invoice and the returned credit; a line billed at
  contract is neither posted nor drafted. The structure check accepts any
  routing that auto-approves manager review instead of pinning switch+decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e posting's Orchestrator job

FinancialPostingFunction writes no record (the provisioning source is a
dependency-free echo), so the overcharge check now asserts its live reply cites
the disputed invoice. Each effects run leaves its debug window in .grading/,
and a new advisory probe lists FinancialPostingFunction jobs created in that
window: one Successful job for the overcharge, none for the no-discrepancy run.
Advisory because nothing has proven a flow debug's API-workflow call creates a
job, and the shared tenant posts the same invoice from other tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…isting billing tasks

Folds the Studio Web-style prompts and effect-based graders into
billing_discrepancy_detector and billing_dispute_resolution in place (same
task ids), and removes the separate *_sw_ga tasks: no new tests, the existing
two change.

- detector: the debug trace must show the ERP query, the CRM query and their
  merge completed; outputs are read by name (the invoice-number check no longer
  matches the flow's own input); the agent must run its own debug.
- resolution: the structure check accepts any routing that auto-approves
  manager review instead of pinning switch+decision; two debug runs check that
  a valid overcharge is posted (live reply cites the invoice) and drafted, and
  that a line billed at contract is neither; an advisory probe lists the
  posting's Orchestrator jobs. The fixed five-output-names advisory is no
  longer in the task, since the prompt stops naming them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Studio Web the agent may test with the RunProject proxy tool, which the eval
testbench stubs, and its CLI debug cannot attach the invoice file. A gating
check would fail Studio Web for an eval-environment gap; the grader's own debug
runs stay the gating effect checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@astridstaicu astridstaicu changed the title test(uipath-maestro-flow): Studio Web GA prompts for the billing discrepancy and dispute resolution flows test(uipath-maestro-flow): short prompts and effect graders for the billing discrepancy and dispute resolution tasks Nov 18, 2026
@astridstaicu

astridstaicu commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

🤖 I am AI Agent Claude Code Claude Opus 5.5 responding on behalf of Astrid Neacsu.

What this changes

Two existing Flow eval tasks, BillingDiscrepancyDetector and BillingDisputeResolution, now ask for the automation the way a Studio Web user would: a short business request instead of a step-by-step build spec. Their graders now check what the automation did when it ran, such as whether the posting happened or whether both lookups ran, instead of mostly checking what the flow file contains. No task is added or removed, and the task IDs stay the same, so run history lines up.

Why

The old prompts told the agent which node types to use, how to wire them, which variables to declare and how to test. That measures whether an agent can follow a recipe, not whether it can build what a user asks for. Because the graders were written against that recipe, a correct flow built differently could fail. Some checks could also pass without the automation doing its job.


BillingDiscrepancyDetector

The scenario (prompt)

Before After
Request A build spec: name the solution, typed inputs, "fan out from the trigger and rejoin at a merge, do not chain", "use the Data Service query activity, not the native Data Fabric nodes", "not complete until validate passes" The user's words: check a disputed invoice line against the contract, pull the account tier, query ERP and CRM in parallel, with the same formula and output names as before
Hints Throughout the prompt One line in a helper block that only some experiments show: the three numeric inputs are numbers
Testing Validate only; the grader debugs "Build and do a test run"

The business logic is identical: same inputs, same formula, same four outputs, same tenant data.

The assertions

Check Before After
Flow validates Required Required, unchanged
The lookups really ran Only that an entity query node and a merge node exist in the file The debug run shows the ERP query, the CRM query and the merge each completed
The answer is right 1610, 1, MCS-2026-04872 and Enterprise appear among all values in the run, including the flow's inputs. The invoice number is an input, so that one always matched Each output is read by name: totalOvercharge = 1610, discrepancyCount = 1, matchedInvoiceNumber = MCS-2026-04872, accountTier = Enterprise
The agent tested its build Not checked Advisory: the agent ran its own debug
Bindings, parallel structure Advisory Advisory, unchanged

BillingDisputeResolution

The scenario (prompt)

Before After
Request A 10-step pipeline: the exact IxP model ID, a "Normalize Data" script and its field paths, a switch on three named values, a decision node with trueLabel/falseLabel and the expression false, a "Log Credit Amount" script, five named output variables and what each end node maps The user's words: extract the invoice, look up CRM and ERP in parallel, have an agent use the SOP index to recommend a credit, rejection or manager review, auto-approve manager review, post through FinancialPostingFunction, have a second agent draft an email (not sent), and return the decision, action, credit, explanation and draft
Hints Throughout the prompt One helper block that only some experiments show: BillingDisputeCRM and BillingDisputeERP are Data Service entities, the shape of the dispute submission input, and that the invoice PDF is a file input
Testing Debug until Completed, at most 3 tries "Build and test the automation"

The resources are the same: the IxP model, both entities, the SOP index, FinancialPostingFunction and the tenant data. The agent now decides the routing and the variable names itself.

The assertions

Check Before After
Flow validates Required Required, unchanged
Structure Required the reference layout: a switch with at least 3 branches, then a decision node, then posting. A flow that sends "manager review" straight to posting failed Required, but layout-free: invoice extraction and both lookups come after the trigger, an analyst agent comes before posting, the SOP index is attached, a second agent comes after posting, and a rejection path skips posting. No human-approval node and no send-email node, as before
SOP grounding is live Required (resource.json sidecar) Required, unchanged
Valid overcharge One debug run with a dispute on "line 3, $300 vs $290". The tenant's real line 3 is a different item, so the case didn't match the data. It passed if the outputs contained the invoice number and any one of 11 words or numbers such as "credit", "valid", "contract" or "300" Debug run on a real billing error on the tenant: invoice MCS-2026-04903, line 2, 90 hours billed at $290 when the contract says $235, with a matching invoice PDF (added in this PR). In the trace: extraction and both lookups completed, FinancialPostingFunction completed and its live reply names the invoice, and the drafting agent completed. The outputs must include the credit the SOP gives, $4,950 (90 × $55), and a text that cites both the invoice number and that credit
No discrepancy Not tested New debug run on line 4, billed exactly at contract: the run completes, FinancialPostingFunction does not run, the drafting agent does not run, and an explanation comes back
The agent tested its build Not checked separately Advisory: the agent ran its own debug
Five fixed output names, binding keys, IxP file field Advisory No longer run, since the prompt no longer names the outputs. The file is left in place

Known limits

  • No "a record was created" check. FinancialPostingFunction writes nothing: its source in coder_eval_uipath only echoes the invoice. The email is drafted, never sent. The strongest available proof is that the posting step completed and replied for the right invoice.
  • No check on Orchestrator jobs either. An earlier version listed FinancialPostingFunction jobs after each run. It found none even on runs whose trace showed the posting completed, so it was removed.
  • The test data follows the tenant, not the repo fixtures. The tenant's live SOP decides a rate dispute by comparing the price billed on the ERP row with that row's contracted rate. The SOP and ERP fixture files in coder_eval_uipath describe different data, so they don't predict the right outcome here.
  • The "agent tested its build" check is advisory. On Studio Web the agent may use the RunProject tool, which the eval testbench stubs, and its command-line debug can't attach the PDF. The grader's own debug runs are the required checks.
  • The no-discrepancy run is new and required. It depends on the analyst agent choosing "reject" for a correctly billed line, so it is the check most likely to flip between runs.

How to check it

  • Offline: the new structure check was run on 7 real flows that agents built in November runs (claude-code, codex, studio-web, antigravity). The old check passed all 7, and so does the new one. It also passes the flow a Studio Web agent built from the short prompt. The old check rejects that flow because it has no switch node.

  • Live, first round (head 011fd1c6): the detector passed on all five harnesses. The resolution passed on codex and antigravity and failed on claude-code, delegate and Studio Web, for two reasons in the test, not in the agents:

    • The overcharge case then used line 5 of MCS-2026-04872, which is billed at the contracted $185. The agents on claude-code and delegate rejected the customer's $300 claim, which is correct by the tenant's SOP. The run now uses the real $290-vs-$235 error on MCS-2026-04903.
    • The Studio Web agent didn't know BillingDisputeCRM and BillingDisputeERP are Data Service entities and left placeholder lookups. The helper block now says so.
  • Live, second round (head b0be6560), one run of both tasks per harness:

    • codex 14321451
    • claude-code 14321452
    • studio-web 14321454
    • antigravity 14321455
    • delegate 14321458

    Today's main baseline: antigravity and delegate pass both tasks. Results will be posted here.

…un; drop the job probe

The tenant's live Billing Dispute SOP ("Incorrect rate") compares the billed
unitPrice with the ERP row's contractedRate. Line 5 of MCS-2026-04872 is billed
at the contracted $185, so rejecting the customer's $300 claim is correct, and
the overcharge run failed an agent that did. The overcharge run now disputes
MCS-2026-04903 line 2 (90 h billed at $290 vs a contracted $235) with a matching
invoice PDF, and requires the SOP's credit of 4950 in the outputs and the draft.

The helper block now says BillingDisputeCRM and BillingDisputeERP are Data
Service entities: without it the Studio Web agent found no such resources and
left placeholder nodes.

The Orchestrator job probe is removed: it found no FinancialPostingFunction job
even on runs whose trace showed the posting completed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@astridstaicu
astridstaicu marked this pull request as ready for review November 18, 2026 11:35
@astridstaicu
astridstaicu requested a review from a team as a code owner November 18, 2026 11:35
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @astridstaicu's task in 1m 45s —— View job


Summary

This PR rewrites the prompts for two path-to-ga Flow tasks (billing_discrepancy_detector, billing_dispute_resolution) as short Studio Web–style requests. It moves the main grading from the flow file to the flow debug trace (completed nodes, named outputs, the posting reply), loosens the resolution structure check so it no longer requires switch + decision, and adds a second debug case (no discrepancy) with a new invoice PDF fixture. It changes only test tasks and touches no skills.

Change-by-Change Review

1. billing_discrepancy_detector/check_billing_discrepancy_detector.py — OK

  • Reading the outputs by name with assert_named_equals fixes the old false pass, where MCS-2026-04872 matched the echoed input.
  • read_ids_by_entity ties each named entity to a node id that actually completed. This is stronger than the old "any query node exists" check.
  • The merge is still required, but now through assert_node_type_executed (it must run) rather than just exist in the source. That is consistent with "query in parallel".
  • Checked: assert_named_equals, assert_node_type_executed, completed_node_ids_of_type, _iter_flow_nodes and entity_reads/entity_name all exist in _shared/ with the signatures used here. ENTITY_QUERY_HINTS covers both the connector and the native read shapes, so dropping the prompt's "use query-entity-records, not native Data Fabric" line won't reject a correct native build.

2. billing_discrepancy_detector.yaml — Low

  • The prompt no longer states that the task is only done when flow validate passes (.claude/rules/test-writing.md § Prompts: "State the goal and the success bar"). It is still graded at weight 3.0. I understand the Studio Web realism goal, but a one-line bar such as "the flow must pass validation" costs little. The resolution prompt has the same gap.
  • The description says it "checks that the agent ran its own test run". That check is advisory (pass_threshold: 0.0), so the description should say so.

3. billing_dispute_resolution/check_structure_billing_dispute_resolution.py — OK

  • reach(..., blocked=api) is a clean way to express "a rejection path that skips posting".
  • Defining the analyst as "downstream of every read and upstream of the API workflow" correctly separates it from the writer, which is also downstream of the reads.
  • The need(len(ends) >= 2) check was dropped, but the new rejection-path check plus the "end after writer" check imply two terminal paths. They could share one end node, which is fine.

4. billing_dispute_resolution/check_e2e_billing_dispute_resolution.py — Low/Medium

  • Two ways of finding the flow (check_e2e…:637-642): _flow() takes the alphabetically first **/*.flow under the project dir. completed_node_ids_of_type / _iter_flow_nodes (used in the detector and in flow_check) resolve through **/project.uiproj. If the sandbox ever holds two .flow files (a leftover scratch flow, a second project in the solution), the ids from _flow() and those in the trace can disagree, and the check fails with a confusing message. Prefer resolving the flow the same way the shared helpers do, or assert there is exactly one .flow.
  • check_no_discrepancy uses all executions, not only completed ones (:786-788). This is intentional and correct: a posting that faulted still counts as "posted". A short comment saying so would stop a later edit from "fixing" it to completed=True.
  • The no-discrepancy case accepts any non-empty out string as "an explanation" (:799-801). A literal like "No resolution email was drafted." mapped on the reject end node satisfies it, even if the analyst's rationale never reaches an output. That is acceptable as a weight-3 effect check. If you want it tighter, also require the uipath.agent.autonomous analyst to have completed, so a flow that rejects on a script before the agent still fails.
  • Arithmetic checked: 90 × 290 = 26,100, (290 − 235) × 90 = 4,950, 300 × 14 − 2,590 = 1,610. _amount_spellings(4950) covers 4950, 4,950, 4950.00 and 4,950.00, so $4,950 matches.

5. billing_dispute_resolution.yaml — Low

  • The advisory _shared/advisory_billing_dispute_resolution.py was removed. The PR body describes it as "the check for five fixed output names", but its description also covered CONNECTION bindings carrying a real tenant key and the IxP fileRef being a plain =js: string (FC419). Both are still meaningful under the short prompt, so this drops some advisory coverage. Consider keeping the advisory without the output-names part.
  • task_timeout going 5400 → 7200 fits the two 1800 s grader debug runs.

6. test_billing_dispute_resolution_graders.py (new) — Low

  • _V1 = _HERE (line 17) is a leftover alias. Use _HERE directly.
  • It sits next to the existing test_structure_billing_dispute_resolution.py, which covers the same assert_topology. Consider merging the two test files.
  • I couldn't run pytest here: the sandbox blocked the command pending approval. I'm relying on the 315-passed claim in the PR body.

7. invoice-MCS-2026-04903.pdf (new binary) — Low

  • .github/code_review.md discourages binary files, but there is precedent (invoice-MCS-2026-04872.pdf) and IxP needs a real PDF, so I'm noting it rather than objecting. Test tasks aren't in the shipped tarball, so the sensitive-content gate doesn't apply. Still, make sure the fixture has no real-looking addresses or phone numbers. I couldn't extract its text here.

What's Missing

  • Live run results on every harness. The PR body says they are pending, and test-writing.md step 5 asks for a passing-run claim before merge.
  • _shared/advisory_billing_dispute_resolution.py and its tests in _shared/test_same_ground_advisories.py are now referenced by no task. Either delete them or note that they are being kept on purpose for reuse.

Area Ratings

Area Status Notes
Frontmatter OK No skill changes
E2E Tests OK Outcome grading is stronger; the advisory and fixture points above are minor
Skill Body OK N/A
References & Assets OK N/A
Repo Hygiene OK Scoped to two task folders; no secrets; binary PDF fixture follows precedent

Issues for Manual Review

  • Routing on the no-discrepancy case. The customer says the rate "looks too high", so the analyst may recommend escalate_to_manager instead of reject. Because the prompt auto-approves manager review, that path posts, and the no-discrepancy check fails. The check is right only if the tenant's live SOP clearly sends an at-contract line to rejection rather than review. Please confirm this against the live SOP, not the repo fixture.
  • What IxP extracts from the new PDF. Whether IxP returns line 2 as 90 h × $290 from invoice-MCS-2026-04903.pdf, and whether the tenant's ERP row for MCS-2026-04903 really has contractedRate = 235, can only be checked on the live tenant.
  • The posting reply. node_output_leaves(payload, posted) assumes the API-workflow node's output lands in globals["<id>.output"]. The unit test fakes that shape; please confirm it in one real trace.

Conclusion

Approve once the live harness runs are in. The move to grading what the run did is a real improvement: named outputs fix the input-echo false pass, and the posting reply is now tied to the invoice. Nothing blocks the merge. Main follow-ups: resolve the flow file the same way in both graders, keep the bindings/fileRef advisory coverage, and verify the no-discrepancy routing against the live SOP.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @astridstaicu's task in 1m 9s —— View job


Coder-eval task lint (advisory)

2 task YAMLs changed. Verdicts: 0 Critical, 0 High, 0 Medium, 2 Low, 0 OK. One High is raised separately under passing-run evidence.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

  • ✅ billing_discrepancy_detector: the PR body says "An earlier Studio Web run of these prompts as separate tasks (adhoc-2026-11-18_09-28-17): detector 1.0". A follow-up comment also reports that the detector passed on all five harnesses at 011fd1c6.
  • ❌ High for billing_dispute_resolution: the PR body has no passing run of the current task. It says the graders "crashed on cross-folder imports" and that live runs are "pending". The first-round passes on codex and antigravity that a comment mentions used the old overcharge case (MCS-2026-04872 line 5), which this PR has since replaced. When the second round (14321451–14321458) finishes, please edit the PR description to add a line like: Ran skill-flow-devcon-billing-dispute-resolution on <harness> (run <id>) and it passed.

Per-task lint

tests/tasks/uipath-maestro-flow/multi_node/billing_discrepancy_detector/billing_discrepancy_detector.yaml — verdict: Low

Issues:

  • [Low] Meaningful coverage: the grader runs one debug input set (line 5, 300 × 14 → 1610), so the "otherwise 0" branch of the formula (prompt L56–57) is never exercised. A flow that skips the clamp, or always returns discrepancyCount=1, still passes.
  • [Low] Could pass for the wrong reason: matchedInvoiceNumber=MCS-2026-04872 also passes when the flow echoes the invoiceNumber input unchanged. Reading the output by name is better than the old any-value search, but this one value still proves nothing about the ERP lookup.

Suggested fixes:

  • Add a second run_command that calls check_billing_discrepancy_detector.py with a line billed at or under contract (e.g. a no-overcharge argv mode), and assert totalOvercharge=0 and discrepancyCount=0.
  • Optionally, check that matchedInvoiceNumber comes from the ERP row: either source it from the query node's output in the trace, or debug once with a differently cased or padded invoiceNumber and compare against the canonical value.

tests/tasks/uipath-maestro-flow/multi_node/billing_dispute_resolution/billing_dispute_resolution.yaml — verdict: Low

Issues:

  • [Low] Meaningful coverage: the no-discrepancy run (L114–120, weight 3.0, required) passes on any non-empty string output (check_e2e_…py:261-263). It also depends on the analyst choosing reject over manager review for a line the customer calls "too high". Manager review is auto-approved and then posted, so the result can flip between runs (the PR comment says so too).
  • [Low] Could pass for the wrong reason: the no-discrepancy check only proves that something did not happen. A flow that dies right after the lookups, without completing, would pass if the run status is not also required to be Completed. Please confirm main() asserts a completed run for this mode as well as for the overcharge mode.

Suggested fixes:

  • In check_no_discrepancy, also require a text output that matches a rejection or no-discrepancy cue (e.g. /reject|no (pricing )?discrepanc|matches (the )?contract/i). A plain "non-empty" check is too weak.
  • If the decision proves unstable across replicates, consider pass_threshold < 1.0 or weight 2.0 on L114–120 until there is evidence that it is stable.

Not flagged: both prompts state business goals plus ground-truth anchors (formula, output names, resource names), so there is no Prompt over-specification. Both tasks are e2e with grader-driven flow debug runs, so Validate-only does not apply. There are no run-limit fields under agent: and no @uipath/cli in the sandbox. The siblings billing_invoice_lookup, billing_dispute_analyst and billing_resolution_writer each test one component; the resolution task composes them and the detector adds parallel ERP+CRM queries plus a computation, so there is no Near-duplicate. I could not run scripts/check-cli-verbs.py here because the command needs approval. Both command_executed patterns target uip maestro flow debug, which is a current verb.

Within-PR duplicates

  • No duplicate clusters. Both tasks are modified, not added.

Conclusion

  • ⚠ 2 tasks have issues, max severity Low (plus one High: no passing-run claim for billing_dispute_resolution). Advisory only, not blocking merge.

…harness round

- The email draft may come back as an object ({subject, body}); the overcharge
  check now reads text nested in outputs. Delegate's run returned a correct
  $4,950 draft this way and was failed.
- The no-discrepancy run now disputes MCS-2026-04872 line 5, whose ERP row
  carries contractedRate 185 = the billed 185. Line 4 has no contractedRate,
  so analysts reasonably chose manager review, which the prompt auto-approves.
- The writer may be an existing tenant agent (uipath.core.agent.*); the
  structure and effects checks count it, as a Studio Web build used one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@astridstaicu

astridstaicu commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

🤖 I am AI Agent Claude Code Claude Opus 5.5 responding on behalf of Astrid Neacsu.

Eval results

Round 3, head 522213a7: 9 of 10 pass

Harness Detector Resolution Run
codex pass pass adhoc-2026-11-18_12-43-33
claude-code pass pass adhoc-2026-11-18_12-42-07
antigravity pass pass adhoc-2026-11-18_12-42-36
delegate pass pass adhoc-2026-11-18_12-42-16
studio-web pass fail adhoc-2026-11-18_12-49-44

The Studio Web failure is in the eval's Studio Web testbench, not in this PR or the agent:

  • The agent added FinancialPostingFunction to the solution as a resource.
  • The testbench's export wrote that package resource with "files": [], so Orchestrator refused to provision it: No package file provided for package resource FinancialPostingFunction.api.FinancialPostingFunction.1 (error 4413).
  • The Node CLI writes the same resource with the package's download URL, and its debug provisions fine.

A fix in the Autopilot repo is in progress.

The earlier rounds, kept below, show how the test cases were corrected.

Rounds 1–2

BillingDiscrepancyDetector: passed on every harness in every round, as did today's main runs of the old prompt. The short prompt and the new effect checks don't make it worse anywhere.

BillingDisputeResolution: most failures in the first two rounds came from the test itself, not from the agents. Each one is fixed in this PR:

What failed Why it was the test's fault Fix
The overcharge run on claude-code and delegate The case disputed line 5 of MCS-2026-04872 at $300, but the tenant bills that line at the contracted $185. The agents rightly rejected it by the tenant's own SOP The overcharge run now uses a real billing error on the tenant: MCS-2026-04903 line 2, 90 hours at $290 against a contracted $235, with a matching invoice PDF. Round 2 posted the right $4,950 credit on claude-code, antigravity and delegate
The billed-at-contract run on antigravity and delegate Line 4's ERP row has no contracted rate, so "manager review" was a reasonable call, and the prompt auto-approves that Now uses line 5, whose ERP row shows $185 billed against $185 contracted
The overcharge run on delegate The flow returned the email as an object (subject, body), and the check read only plain text. The email was correct The check now reads text inside objects
Structure on Studio Web The flow drafted the email with an existing tenant agent instead of a new inline one. The request allows that An existing tenant agent now counts as the second agent

Two failures were not the test's fault:

  • Studio Web: at runtime the analyst agent couldn't find the "Billing Dispute SOP Index" (CONTEXT_GROUNDING_INDEX_NOT_FOUND), in both debug runs. This is a real build problem. My unproven guess is that the agent bound the index in the wrong folder, because the short prompt names the folder only for the extraction model. If it repeats in round 3, one helper line naming the index's folder should fix it.
  • claude-code, once: flow debug returned "fetch failed", a network error before the flow ran.

Two other findings:

  • The tenant's live SOP and ERP data differ from the fixture files in coder_eval_uipath. The test cases are now taken from the tenant's real data.
  • Graded against the tenant's SOP, the old prompt's runs that posted a $1,610 credit for line 5 were wrong, yet the old checks passed them.

…l run

All 27 path-to-ga tasks (Flow, BPMN, Case, HITL) now read like a Studio Web
user's request: a context paragraph, the business ask, a 1-3 line helper
block, "build and test". Build instructions (node types, wiring, CLI steps,
pinned paths) are gone. Gating criteria run the automation against the live
tenant and assert what it did: steps completed or skipped on the path, named
outputs, and real effects (Slack ts read back, Jira issue read back, Action
Center task created and completed, debug instance variables). Structural
checks accept any correct layout; the agent's own test run is advisory.

Fixes test-caused failures found in recent runs: graders pinned to CLI-only
paths, operations or metadata keys Studio Web doesn't write, the BPMN live
debug relying on the agent's own run, a sed false positive in the CM golden
guard, over-strict label and field-name checks.

Flow _shared prompt-corpus tests accept the user-style context sentence and a
named automation; the connector-pin rule is dropped (graders accept both
entity-read shapes).

Known environment blockers on alpha codereval, not fixed here: the Jira
connection returns 403 (flow and bpmn Jira tasks, customer escalation), the
Outlook connection fails its ping (case outcome_external), and no GitHubMcp
server exists (inline_mcp_server_tool stays skipped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
astridstaicu and others added 2 commits November 18, 2026 16:45
…Flow path-to-ga scenarios

The eight Flow scenarios now use Ioana's v3 prompts verbatim, apart from the
eval-tenant substitutions her notes give (folder Shared/uipath-maestro-flow/
BillingDispute, #coding-agent-testing instead of a direct message, a Task in
the CE Jira project). The prompts name no variables and no connections.

Graders no longer depend on names the agent picks: the new
_shared/flow_io_roles.py maps seeded values onto the flow's declared inputs
by meaning (object inputs from the paths the flow reads plus a synonym
fallback) and finds results among declared outputs by value, in the formats
the prompts ask for. Side effects stay the strongest checks: Slack messages
read back by ts, Jira issues read back by key, FinancialPostingFunction's live
reply. Each behaviour the prompts state gets a grader case (not-found invoice
and customer, a thin dispute description, a duplicate escalation). Expected
values follow the eval tenant's live ERP/CRM data and SOP, and v3's severity
rules. Structure, send-identity and CLI-command checks are advisory or gone;
the Data Service connection preflight is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- case edit/*: the inline `python3 -c "` criteria began with a space once the
  folded scalar joined the lines (IndentationError on every harness, main's
  daily too); the code now starts after the quote. Their pre_run also moved the
  fixture out of the Studio Web workspace after the bridge opened it, so the
  agent saw an empty /solution; the move is gone.
- hitl: completing the task sent the whole `tasks data get` record as --data
  (incident 170001 on resume); it now sends only the form fields. The app task
  is created in the app's folder, not the personal one; the poll now lists the
  folders the HITL node names, and tenant-wide by job key as a fallback.
- bpmn: "an empty Agentic Process project" let Studio Web agents create an
  Agent or a Flow; the prompts now say "Agentic Process (Maestro BPMN)".
- flow orchestrator: the match-word guess iterated a set, so the seeded match
  value depended on the hash seed; it is now deterministic and reads
  `[...].indexOf/includes(match)` membership lists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bai-uipath

Copy link
Copy Markdown
Contributor

Grading on what the run does is a real improvement over checking the flow file: outputs read by name, live effects, and any layout accepted. Some tradeoffs come with prompts that name nothing, and they're worth deciding on purpose before these tasks become the GA gate:

  • The grader has to guess the inputs. With no input or output names in the prompt, flow_io_roles.py decides which input means what from names, descriptions and the paths the flow reads. When it guesses wrong, a correct flow fails, and it looks like an agent failure. Real users never hit this, because they fill in their own inputs, so the eval adds a failure the product doesn't have. Options: let the prompt state what goes in and comes out (names and shape) and still leave out the build steps, or at least report an input the grader couldn't match as a grader error, not a task failure.
  • Judgment calls become pass/fail. Where the SOP leaves room (credit vs manager review vs reject), the expected answer depends on how the agent reads it, and the test data has to be picked around that reading. Those checks will flip between runs and between models for reasons that have nothing to do with skill quality. I'd make only the cases the SOP clearly decides required, and make the rest advisory or score them over several runs.
  • Missing facts, and nobody to ask. The prompt tells the agent no user is present, so the agent can only guess any fact the request leaves out, such as which folder a resource lives in. In Studio Web a user would answer that question. Either the prompt includes every fact the grader depends on, or the case belongs in the simulation tasks, where a simulated user can answer. Short prompts alone don't solve this part of the ambiguity concern.
  • The right answers live on the tenant. Expected values follow the tenant's live SOP and ERP rows, which already differ from the fixtures in coder_eval_uipath. A reseed or an SOP edit changes the right answer, and nothing in the repo shows it. Seeding from repo fixtures, or bringing the fixtures in line with the tenant, would keep the expected answers reviewable.
  • Run history stops being comparable. Same task IDs with new prompts means a pass-rate change will mix prompt difficulty with skill changes. Running the old prompt as a variant for a few nightlies would separate the two.

None of this argues against the direction. These are just the costs of naming nothing, and each has a cheap mitigation.

- flow escalation_jira_ticket: a Studio Web build created the ticket with the
  generic `uipath-atlassian-jira.insert-record` op, which completed; the grader
  only counted the curated create-issue op. Any completed Jira step now counts;
  the proof stays the CE key it returned, read back from Jira.
- flow billing_resolution_writer: the agent returned its draft as JSON text and
  a script parsed it; the grader now also reads the strings inside JSON-text
  agent outputs when checking the body came from the agent.
- bpmn live_debug_e2e: the one BPMN prompt still saying only "Agentic
  Process"; a Studio Web agent built an Agent plus an API workflow. It now says
  "Agentic Process (Maestro BPMN)" like the other five.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants