Repository navigation
test(uipath-maestro-flow): short prompts and effect graders for the billing discrepancy and dispute resolution tasks - #4903
astridstaicu wants to merge 10 commits into
Conversation
…repancy and dispute resolution flows Adds two autopilot-sw-ga tasks that ask the path-to-ga BillingDiscrepancyDetector and BillingDisputeResolution scenarios the way a Studio Web user would: a business request, a short helper block, no build instructions. Graders check what the run did, read from the debug trace: - detector: the ERP and CRM queries and their merge completed; the four named outputs carry the live values; the agent ran its own debug. - resolution: a valid overcharge is posted through FinancialPostingFunction and gets a draft citing the invoice and the returned credit; a line billed at contract is neither posted nor drafted. The structure check accepts any routing that auto-approves manager review instead of pinning switch+decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e posting's Orchestrator job FinancialPostingFunction writes no record (the provisioning source is a dependency-free echo), so the overcharge check now asserts its live reply cites the disputed invoice. Each effects run leaves its debug window in .grading/, and a new advisory probe lists FinancialPostingFunction jobs created in that window: one Successful job for the overcharge, none for the no-discrepancy run. Advisory because nothing has proven a flow debug's API-workflow call creates a job, and the shared tenant posts the same invoice from other tasks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…isting billing tasks Folds the Studio Web-style prompts and effect-based graders into billing_discrepancy_detector and billing_dispute_resolution in place (same task ids), and removes the separate *_sw_ga tasks: no new tests, the existing two change. - detector: the debug trace must show the ERP query, the CRM query and their merge completed; outputs are read by name (the invoice-number check no longer matches the flow's own input); the agent must run its own debug. - resolution: the structure check accepts any routing that auto-approves manager review instead of pinning switch+decision; two debug runs check that a valid overcharge is posted (live reply cites the invoice) and drafted, and that a line billed at contract is neither; an advisory probe lists the posting's Orchestrator jobs. The fixed five-output-names advisory is no longer in the task, since the prompt stops naming them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Studio Web the agent may test with the RunProject proxy tool, which the eval testbench stubs, and its CLI debug cannot attach the invoice file. A gating check would fail Studio Web for an eval-environment gap; the grader's own debug runs stay the gating effect checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What this changesTwo existing Flow eval tasks, BillingDiscrepancyDetector and BillingDisputeResolution, now ask for the automation the way a Studio Web user would: a short business request instead of a step-by-step build spec. Their graders now check what the automation did when it ran, such as whether the posting happened or whether both lookups ran, instead of mostly checking what the flow file contains. No task is added or removed, and the task IDs stay the same, so run history lines up. WhyThe old prompts told the agent which node types to use, how to wire them, which variables to declare and how to test. That measures whether an agent can follow a recipe, not whether it can build what a user asks for. Because the graders were written against that recipe, a correct flow built differently could fail. Some checks could also pass without the automation doing its job. BillingDiscrepancyDetectorThe scenario (prompt)
The business logic is identical: same inputs, same formula, same four outputs, same tenant data. The assertions
BillingDisputeResolutionThe scenario (prompt)
The resources are the same: the IxP model, both entities, the SOP index, FinancialPostingFunction and the tenant data. The agent now decides the routing and the variable names itself. The assertions
Known limits
How to check it
|
…un; drop the job probe
The tenant's live Billing Dispute SOP ("Incorrect rate") compares the billed
unitPrice with the ERP row's contractedRate. Line 5 of MCS-2026-04872 is billed
at the contracted $185, so rejecting the customer's $300 claim is correct, and
the overcharge run failed an agent that did. The overcharge run now disputes
MCS-2026-04903 line 2 (90 h billed at $290 vs a contracted $235) with a matching
invoice PDF, and requires the SOP's credit of 4950 in the outputs and the draft.
The helper block now says BillingDisputeCRM and BillingDisputeERP are Data
Service entities: without it the Studio Web agent found no such resources and
left placeholder nodes.
The Orchestrator job probe is removed: it found no FinancialPostingFunction job
even on runs whose trace showed the posting completed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Claude finished @astridstaicu's task in 1m 45s —— View job SummaryThis PR rewrites the prompts for two Change-by-Change Review1.
|
| Area | Status | Notes |
|---|---|---|
| Frontmatter | OK | No skill changes |
| E2E Tests | OK | Outcome grading is stronger; the advisory and fixture points above are minor |
| Skill Body | OK | N/A |
| References & Assets | OK | N/A |
| Repo Hygiene | OK | Scoped to two task folders; no secrets; binary PDF fixture follows precedent |
Issues for Manual Review
- Routing on the no-discrepancy case. The customer says the rate "looks too high", so the analyst may recommend
escalate_to_managerinstead ofreject. Because the prompt auto-approves manager review, that path posts, and the no-discrepancy check fails. The check is right only if the tenant's live SOP clearly sends an at-contract line to rejection rather than review. Please confirm this against the live SOP, not the repo fixture. - What IxP extracts from the new PDF. Whether IxP returns line 2 as 90 h × $290 from
invoice-MCS-2026-04903.pdf, and whether the tenant's ERP row for MCS-2026-04903 really hascontractedRate = 235, can only be checked on the live tenant. - The posting reply.
node_output_leaves(payload, posted)assumes the API-workflow node's output lands inglobals["<id>.output"]. The unit test fakes that shape; please confirm it in one real trace.
Conclusion
Approve once the live harness runs are in. The move to grading what the run did is a real improvement: named outputs fix the input-echo false pass, and the posting reply is now tied to the invoice. Nothing blocks the merge. Main follow-ups: resolve the flow file the same way in both graders, keep the bindings/fileRef advisory coverage, and verify the no-discrepancy routing against the live SOP.
|
Claude finished @astridstaicu's task in 1m 9s —— View job Coder-eval task lint (advisory)2 task YAMLs changed. Verdicts: 0 Critical, 0 High, 0 Medium, 2 Low, 0 OK. One High is raised separately under passing-run evidence. Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge. Evidence of passing run
Per-task lint
|
…harness round
- The email draft may come back as an object ({subject, body}); the overcharge
check now reads text nested in outputs. Delegate's run returned a correct
$4,950 draft this way and was failed.
- The no-discrepancy run now disputes MCS-2026-04872 line 5, whose ERP row
carries contractedRate 185 = the billed 185. Line 4 has no contractedRate,
so analysts reasonably chose manager review, which the prompt auto-approves.
- The writer may be an existing tenant agent (uipath.core.agent.*); the
structure and effects checks count it, as a Studio Web build used one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eval resultsRound 3, head
|
| Harness | Detector | Resolution | Run |
|---|---|---|---|
| codex | pass | pass | adhoc-2026-11-18_12-43-33 |
| claude-code | pass | pass | adhoc-2026-11-18_12-42-07 |
| antigravity | pass | pass | adhoc-2026-11-18_12-42-36 |
| delegate | pass | pass | adhoc-2026-11-18_12-42-16 |
| studio-web | pass | fail | adhoc-2026-11-18_12-49-44 |
The Studio Web failure is in the eval's Studio Web testbench, not in this PR or the agent:
- The agent added FinancialPostingFunction to the solution as a resource.
- The testbench's export wrote that package resource with
"files": [], so Orchestrator refused to provision it:No package file provided for package resource FinancialPostingFunction.api.FinancialPostingFunction.1(error 4413). - The Node CLI writes the same resource with the package's download URL, and its debug provisions fine.
A fix in the Autopilot repo is in progress.
The earlier rounds, kept below, show how the test cases were corrected.
Rounds 1–2
BillingDiscrepancyDetector: passed on every harness in every round, as did today's main runs of the old prompt. The short prompt and the new effect checks don't make it worse anywhere.
BillingDisputeResolution: most failures in the first two rounds came from the test itself, not from the agents. Each one is fixed in this PR:
| What failed | Why it was the test's fault | Fix |
|---|---|---|
| The overcharge run on claude-code and delegate | The case disputed line 5 of MCS-2026-04872 at $300, but the tenant bills that line at the contracted $185. The agents rightly rejected it by the tenant's own SOP | The overcharge run now uses a real billing error on the tenant: MCS-2026-04903 line 2, 90 hours at $290 against a contracted $235, with a matching invoice PDF. Round 2 posted the right $4,950 credit on claude-code, antigravity and delegate |
| The billed-at-contract run on antigravity and delegate | Line 4's ERP row has no contracted rate, so "manager review" was a reasonable call, and the prompt auto-approves that | Now uses line 5, whose ERP row shows $185 billed against $185 contracted |
| The overcharge run on delegate | The flow returned the email as an object (subject, body), and the check read only plain text. The email was correct |
The check now reads text inside objects |
| Structure on Studio Web | The flow drafted the email with an existing tenant agent instead of a new inline one. The request allows that | An existing tenant agent now counts as the second agent |
Two failures were not the test's fault:
- Studio Web: at runtime the analyst agent couldn't find the "Billing Dispute SOP Index" (
CONTEXT_GROUNDING_INDEX_NOT_FOUND), in both debug runs. This is a real build problem. My unproven guess is that the agent bound the index in the wrong folder, because the short prompt names the folder only for the extraction model. If it repeats in round 3, one helper line naming the index's folder should fix it. - claude-code, once:
flow debugreturned "fetch failed", a network error before the flow ran.
Two other findings:
- The tenant's live SOP and ERP data differ from the fixture files in
coder_eval_uipath. The test cases are now taken from the tenant's real data. - Graded against the tenant's SOP, the old prompt's runs that posted a $1,610 credit for line 5 were wrong, yet the old checks passed them.
…l run All 27 path-to-ga tasks (Flow, BPMN, Case, HITL) now read like a Studio Web user's request: a context paragraph, the business ask, a 1-3 line helper block, "build and test". Build instructions (node types, wiring, CLI steps, pinned paths) are gone. Gating criteria run the automation against the live tenant and assert what it did: steps completed or skipped on the path, named outputs, and real effects (Slack ts read back, Jira issue read back, Action Center task created and completed, debug instance variables). Structural checks accept any correct layout; the agent's own test run is advisory. Fixes test-caused failures found in recent runs: graders pinned to CLI-only paths, operations or metadata keys Studio Web doesn't write, the BPMN live debug relying on the agent's own run, a sed false positive in the CM golden guard, over-strict label and field-name checks. Flow _shared prompt-corpus tests accept the user-style context sentence and a named automation; the connector-pin rule is dropped (graders accept both entity-read shapes). Known environment blockers on alpha codereval, not fixed here: the Jira connection returns 403 (flow and bpmn Jira tasks, customer escalation), the Outlook connection fails its ping (case outcome_external), and no GitHubMcp server exists (inline_mcp_server_tool stays skipped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Flow path-to-ga scenarios The eight Flow scenarios now use Ioana's v3 prompts verbatim, apart from the eval-tenant substitutions her notes give (folder Shared/uipath-maestro-flow/ BillingDispute, #coding-agent-testing instead of a direct message, a Task in the CE Jira project). The prompts name no variables and no connections. Graders no longer depend on names the agent picks: the new _shared/flow_io_roles.py maps seeded values onto the flow's declared inputs by meaning (object inputs from the paths the flow reads plus a synonym fallback) and finds results among declared outputs by value, in the formats the prompts ask for. Side effects stay the strongest checks: Slack messages read back by ts, Jira issues read back by key, FinancialPostingFunction's live reply. Each behaviour the prompts state gets a grader case (not-found invoice and customer, a thin dispute description, a duplicate escalation). Expected values follow the eval tenant's live ERP/CRM data and SOP, and v3's severity rules. Structure, send-identity and CLI-command checks are advisory or gone; the Data Service connection preflight is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- case edit/*: the inline `python3 -c "` criteria began with a space once the folded scalar joined the lines (IndentationError on every harness, main's daily too); the code now starts after the quote. Their pre_run also moved the fixture out of the Studio Web workspace after the bridge opened it, so the agent saw an empty /solution; the move is gone. - hitl: completing the task sent the whole `tasks data get` record as --data (incident 170001 on resume); it now sends only the form fields. The app task is created in the app's folder, not the personal one; the poll now lists the folders the HITL node names, and tenant-wide by job key as a fallback. - bpmn: "an empty Agentic Process project" let Studio Web agents create an Agent or a Flow; the prompts now say "Agentic Process (Maestro BPMN)". - flow orchestrator: the match-word guess iterated a set, so the seeded match value depended on the hash seed; it is now deterministic and reads `[...].indexOf/includes(match)` membership lists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Grading on what the run does is a real improvement over checking the flow file: outputs read by name, live effects, and any layout accepted. Some tradeoffs come with prompts that name nothing, and they're worth deciding on purpose before these tasks become the GA gate:
None of this argues against the direction. These are just the costs of naming nothing, and each has a cheap mitigation. |
- flow escalation_jira_ticket: a Studio Web build created the ticket with the generic `uipath-atlassian-jira.insert-record` op, which completed; the grader only counted the curated create-issue op. Any completed Jira step now counts; the proof stays the CE key it returned, read back from Jira. - flow billing_resolution_writer: the agent returned its draft as JSON text and a script parsed it; the grader now also reads the strings inside JSON-text agent outputs when checking the body came from the agent. - bpmn live_debug_e2e: the one BPMN prompt still saying only "Agentic Process"; a Studio Web agent built an Agent plus an API workflow. It now says "Agentic Process (Maestro BPMN)" like the other five. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What
This changes the two existing
path-to-gaFlow tasks in place, keeping the same task IDs. It adds no tests.multi_node/billing_discrepancy_detectormulti_node/billing_dispute_resolutionThe prompts are what a Studio Web user would type: a business request, a short
<!-- helper -->block, and one line saying the run is headless. They replace the step-by-step build specs. Theoutcome-gradedtag is added to both.What the graders check
The main criteria grade what the run did, read from the
flow debugtrace, rather than what the flow file contains.Detector
flow validatepasses.totalOvercharge=1610,discrepancyCount=1,matchedInvoiceNumber=MCS-2026-04872,accountTier=Enterprise. The old check searched every value, so the invoice number also matched the flow's own input.uip maestro flow debug, plus the existing bindings and structural checks.Resolution
flow validatepasses.Structure:
It no longer pins the reference's switch + decision layout, because "automatically approve manager review" can be built without them.
The SOP index's
resource.jsonsidecar exists (unchanged check).Overcharge run, a real billing error on the tenant: invoice MCS-2026-04903, line 2, 90 h billed at $290 against a contracted $235, with a matching invoice PDF added in this PR.
MCS-2026-04903.MCS-2026-04903and that credit.No-discrepancy run (MCS-2026-04872 line 4, billed exactly at contract): the run completes without running FinancialPostingFunction or the writer, and returns an explanation.
Advisory: the agent ran its own debug.
Why the "agent ran its own test" check is advisory: on Studio Web the agent may test with the RunProject proxy tool, which the eval testbench stubs, and its CLI debug can't attach the PDF.
Why the test data comes from the tenant: the tenant's live Billing Dispute SOP decides a rate dispute by comparing the ERP row's billed
unitPricewith itscontractedRate. The SOP and ERP fixture files incoder_eval_uipathdescribe different data. MCS-2026-04872 line 5 is billed at the contracted $185, so it is not an overcharge on the tenant.Why there's no "record was created" check: FinancialPostingFunction writes nothing. Its provisioning source (
coder_eval_uipathSolutionSources/BillingDispute/FinancialPostingFunction/Workflow.json) only echoes the invoice. The email is drafted, not sent.No longer in the task:
_shared/advisory_billing_dispute_resolution.py, the check for five fixed output names, because the prompt stops naming the outputs. The file and its tests are left in place.Testing
pyteston the resolution grader tests,test_flow_check.pyandtest_same_ground_advisories.py: 315 passed.core.logic.switch.adhoc-2026-11-18_09-28-17): detector 1.0. The resolution graders crashed on cross-folder imports, which are gone now that everything sits in the task's own folder.check-task-driver.pyandcheck-task-host-paths.pypass.🤖 Generated with Claude Code