Skip to content

test(uipath-maestro-bpmn): fix graders that rejected correct Studio Web output - #5012

Draft
DevMomo wants to merge 1 commit into
mainfrom
claude/busy-gagarin-005a5b
Draft

DevMomo wants to merge 1 commit into
mainfrom
claude/busy-gagarin-005a5b

Conversation

@DevMomo

@DevMomo DevMomo commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Fixes graders in tests/tasks/uipath-maestro-bpmn/ that rejected correct Studio Web output in run 2026-11-19_09-23-02 (72.2%). Test files only; no skill text touched. Fixtures under _shared/fixtures/run_2026_11_19/ are the agents' real artifacts, byte for byte.

Fixes

  1. =js:vars.<id> — new bpmn_check.vars_ref() (accepts =vars.<id> and =js:vars.<id>, rejects scripts like =js:vars.a + 1); used in check_simple_approval_bpmn.py, check_script_jint_guidance.py (bridges, end mapping, args, business result) and bpmn_live.input_echo_ids. Other =vars. greps: only a docstring example remains.
  2. Start-event-scoped inputs — jint grader accepts internal inputs scoped to the process or the start event.
  3. Scratch copies — find_bpmn_file: among several hinted matches the shallowest wins (Foo-work/Foo.bpmn sorted before Foo.bpmn because - < .). is_scratch_bpmn unchanged.
  4. Get-by-id — activity branch of _operation_matches reads the catalog verb from the node metadata (activityMetadata.activity.configuration.httpMethod); a bare GET with no metadata stays a list read; GETBYID in context method still matches.
  5. Studio Web layout — customer_escalation_triage package/behavior graders and the validate criterion resolve the project via resolve_bpmn_file (new _shared/print_bpmn_path.py for the shell command).
  6. Advisory weights — both paginated_reference_lookup advisory criteria set to weight: 0 (+ pass_threshold: 0), per fix(tests): route-specific command telemetry must not move the score #3516: pass_threshold: 0 alone never fails a task but still moves the score.
  7. Telemetry vs outcome — solution new|init command criterion replaced by the workspace-outcome check (.uipx or project.uiproj, same as the Flow twin) in evaluate/evaluator_type_choice, local_crud, no_auto_upload, simulation/simulation_crud (same pattern in all four).
  8. Preflight — jdbc_databricks_query gets preflight_connections.py uipath-uipath-jdbc=uipath-maestro-flow in pre_run.

Evidence

  • Real 11-19 artifacts, OLD vs NEW grader: simple-approval FAIL: completion end must bridge… → OK; script-jint FAIL: riskScore is neither bridged… → OK; testmanager --all-ops get a record … missing → OK; find_bpmn_file('ExpenseClaim') returned ExpenseClaim-work/ExpenseClaim.bpmn → ExpenseClaim.bpmn.
  • Still fail: =js:vars.riskScoreValue * 2 (jint), =js:vars.decisionValue + 'x' / undeclared id / no prefix / missing mapping (approval), input scoped to another element or unscoped (jint), bare GET without metadata verb, list op vs get-by-id node.
  • python3 -m pytest tests/tasks/uipath-maestro-bpmn/_shared -q: 2692 passed (new: test_run_2026_11_19_graders.py, 33 tests).
  • @uipath/maestro-validation 0.1233.0 validateBpmnXml on the three real artifacts: 0 errors. It does not flag an undeclared =js:vars.x or a wrongly scoped input either (known limit), so it supports "accepted", not "correct".
  • PO.Frontend origin/develop:
    • Mapping source is serialized verbatim: src/services/serialization/bpmn-moddle.ts:606, bpmn-from-xml-headless.ts:885,966. Prefixes are =vars. / =js: (src/services/uipath/variables/VariableConstants.ts:7-8); removeVariablePrefix strips =js: then vars. (src/services/validation/shared/VariableExpressionUtils.ts:62-68,114-123); reference regex is prefix-agnostic (:16-17); non-custom mapping sources are not validated (VariableValidationUtils.ts:323-331).
    • Start-event scoping is canvas-native: src/reducers/canvasSlice.ts:224-247,2191, UnifiedBuildDataManager.tsx:271,322; VariableValidationUtils.ts:154-164 accepts a ScriptTask read via upstream node outputs.
  • Live canvas check (Studio Web, alpha codereval/DefaultTenant, the 11-19 artifacts uploaded unchanged):
    • script-jint: Health analyzer "No issues found"; End event Arguments show riskScore ← scriptResponse (the =js:vars.<id> end mapping, start-event-scoped internal inputs).
    • simple-approval: Health analyzer shows 0 errors and one warning on the agent's script task ('Auto-approve standard expense' writes 'decisionValue' without a variable update), unrelated to the mappings. End event Arguments show decision ← decisionValue; the XML view keeps source="=js:vars.decisionValue".
    • Canvas writes the form itself: removing the chip and re-picking decisionValue in the End event's variable picker produced <uipath:output name="decision" type="string" source="=js:vars.decisionValue" var="decision" custom="true" />.
    • Both solutions deleted afterwards; none kept.

Not proven

  • Fix 4 (no live canvas evidence): the frontend does not show GET + metadata GETBYID as canonical canvas output; it copies whatever method string the activity config carries (IntsvcActivityConfigurationUtils.ts:333-345), quick-add copies httpMethod into method (QuickAddSelectionUtil.ts:49,63), and the only in-repo fixture has context method=GETBYID (xml-serialization.test.ts:812). The grader now accepts both shapes; whether runtime treats GET+GETBYID as get-by-id is unverified.
  • Remaining weight-1 "report only" command_executed telemetry (e.g. bpmn init, eval list in local_crud) left untouched.

🤖 Generated with Claude Code

…eb output (run 2026-11-19)

Accept =js:vars.<id> and start-event-scoped inputs, prefer the edited file over a work-folder copy, read get-by-id from the node metadata, resolve the Studio Web layout in customer_escalation_triage, grade solution creation by outcome, zero the advisory paginated criteria, and preflight the JDBC connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DevMomo
DevMomo requested review from a team, dmetzgar and tmatup as code owners November 19, 2026 20:04
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @DevMomo's task in 1m 29s —— View job


Coder-eval task lint (advisory)

I linted the 7 task YAMLs this PR changes. Verdicts: 0 Critical, 2 High, 0 Medium, 4 Low, 1 OK. (Both High verdicts come from a Critical theme, downgraded one level under the rubric's theme rule.)

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge. I scored each whole task, not only the lines this PR touches, so some findings below predate the PR.

Evidence of passing run

  • ❌ High — the PR body doesn't say the changed tasks were run with coder-eval and passed. It does give grader-level evidence: pytest … _shared -q: 2692 passed, and each new grader returns OK when re-run on the real 11-19 artifacts. That shows the graders are right, but not that a full task run passes. Please edit the PR description to add a line like: Ran skill-bpmn-eval-local-crud, skill-bpmn-paginated-reference-lookup, … locally and they passed. If you didn't run them, say which ones you skipped.

Per-task lint

tests/tasks/uipath-maestro-bpmn/connector_features/jdbc_databricks_query/jdbc_databricks_query.yaml — verdict: Low

Issues:

  • [Low] Validate-only flow tests miss correctness: this is an e2e task and the prompt asks for a debug run (L59), but the only bpmn debug criterion is advisory (L81-87). Nothing that can fail the task checks runtime behavior. The description doesn't explain why. (I applied this axis to BPMN by analogy to Flow, so I kept it at Low.)

Suggested fixes:

  • Either add a gating runtime check (for example a run_command that debugs the packed process and checks the query output variable), or add one sentence to description explaining why debug is advisory, such as connection flakiness.

tests/tasks/uipath-maestro-bpmn/connector_features/paginated_reference_lookup/paginated_reference_lookup.yaml — verdict: Low

Issues:

  • [Low] Validate-only flow tests miss correctness: this is an integration task with no debug step. The description gives a reason ("This structural port stays validate-only, matching Flow's own e2e task", L116-119), so I downgraded it from Medium.
  • [Low] Meaningful coverage: sandbox.template_sources: is present but empty (L135-136). It's harmless, but it reads like a mistake.

Suggested fixes:

  • Drop the empty sandbox: block, or collapse it to sandbox: {}.
  • The weight-0 advisory change (L181, L190) is correct.

tests/tasks/uipath-maestro-bpmn/e2e/customer_escalation_triage/customer_escalation_triage.yaml — verdict: OK

tests/tasks/uipath-maestro-bpmn/evaluate/evaluator_type_choice.yaml — verdict: High

Issues:

  • [Critical → High] Self-report anti-pattern (Theme 1): the prompt asks the agent to write report.json (L42-51), and two criteria grade it: file_exists at L106-110 and check_type_choice.py with no --check at L112-118, worth 3.5 weight together. The artifact checks at L82-104 already grade the real evaluator types, so this is extra weight on the agent's own claims, not the only signal.

Suggested fixes:

  • Delete L106-118 and the report.json paragraph in the prompt (L42-51). The three --check goal-* criteria already grade the outcome. If you want to keep the reasoning for debugging, set both criteria to weight: 0.

tests/tasks/uipath-maestro-bpmn/evaluate/local_crud.yaml — verdict: Low

Issues:

  • [Low] The tags (L126) have no tier tag (smoke, integration or e2e), which .claude/rules/test-writing.md Must-Do 1 requires. This predates the PR.
  • [Low] Could pass for the wrong reason (unbounded-dot-repeat): no issue here, since the eval list patterns at L224 are anchored.

Suggested fixes:

  • Add smoke to tags. It is local-only and has no tenant, so it matches simulation_crud and evaluator_type_choice.

tests/tasks/uipath-maestro-bpmn/evaluate/no_auto_upload.yaml — verdict: High

Issues:

  • [Critical → High] Self-report anti-pattern (Theme 1): the refusal behavior, which is the whole point of the task, is graded only from the agent-written report.json. That check is check_no_auto_upload.py with no --check (L346-352, weight 5.0) and it reads ran_solution_upload / ran_eval_run_start from the report. An agent that does upload but writes "ran_solution_upload": false still passes.
  • [Medium] Prompt over-specification: the prompt gives the expected answer away ("you must not upload … Refuse, explain why", L270-272), so the test can't tell whether the skill taught the rule. This predates the PR.
  • [Low] The tags (L245) have no tier tag. This predates the PR.

Suggested fixes:

  • Replace the report-based gate with command_not_executed guards on (uip|\$UIP)\s+solution\s+upload and (uip|\$UIP)\s+maestro\s+bpmn\s+eval\s+[^;&|\n]*\brun\b, and give each weight: 3.0. Then cut report.json to weight 0, or remove it.
  • Cut L270-272 down to the scenario. For example, end at "You must decide what to do." and let the skill's upload-safety rule drive the refusal.
  • Add smoke to tags.

tests/tasks/uipath-maestro-bpmn/evaluate/simulation/simulation_crud.yaml — verdict: Low

Issues:

  • [Low] Could pass for the wrong reason (unbounded-dot-repeat): in the gating Static-simulation pattern (L475), the (?=.*--strategy\s+Static\b)(?=.*--mock-value\b) lookaheads can match flags from a different command chained in the same Bash call.

Suggested fixes:

  • In the L475 lookaheads, replace .* with [^;&|\n]* so both flags must sit on the same simulations add|create command.

Within-PR duplicates

  • No duplicate clusters detected. The PR adds no new task YAMLs.

Themes

  • Theme 1 (Critical): Self-report report.json graded with nonzero weight. This affects evaluator_type_choice.yaml and no_auto_upload.yaml. In no_auto_upload the self-report is the main signal, at weight 5.0.

  • Theme 2 (Low): advisory criteria that still move the score. The PR applies the fix(tests): route-specific command telemetry must not move the score #3516 rule ("pass_threshold: 0 alone never fails a task but still moves the score") to paginated_reference_lookup. But fix 7 adds a new .uipx/project.uiproj check at weight: 1.0, pass_threshold: 0.0 in all four evaluate tasks (for example evaluator_type_choice.yaml:71-72). That is the same pattern fix 6 removed. Older advisories in the same situation:

    • jdbc_databricks_query.yaml:86 (1.5)
    • local_crud.yaml:184,226 (1.0)
    • simulation_crud.yaml:485,501 (2.0, 1.5)

    If these are meant to be report-only, set weight: 0. If you want the score to move, say so in the criterion description so it doesn't look like an inconsistency.

Conclusion

  • ⚠ 6 task(s) have issues; the highest per-task verdict is High (Theme 1 itself is Critical). Advisory only — not blocking merge.

I couldn't run scripts/check-cli-verbs.py in this sandbox, so the CLI verb reachability axis wasn't checked. The verbs I read by eye (maestro bpmn validate|debug|init|eval, is resources run list) all look current. To confirm, run python3 scripts/check-cli-verbs.py --json <task> locally.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @DevMomo's task in 2m 12s —— View job


Summary

This PR fixes BPMN coder_eval graders that rejected correct Studio Web output in run 2026-11-19. It accepts =js:vars.<id> through a shared vars_ref(), accepts inputs scoped to the start event, prefers the shallowest file when find_bpmn_file finds several copies, and reads the get-by-id verb from node metadata. It also lets the customer_escalation_triage graders find the project in either layout, sets two advisory weights to 0, swaps the solution new|init command checks for a workspace-outcome check, and adds a JDBC connection preflight. Only test files change; no skill text is touched.

Change-by-Change Review

1. _shared/bpmn_check.py — vars_ref() and its call sites (simple-approval, jint, bpmn_live.input_echo_ids)

OK. fullmatch with the optional js: prefix still rejects scripts (=js:vars.a + 1). fields=True keeps the old dotted-path behaviour of the end-bridge path in check_script_jint_guidance.py:729-733.

  • Low: check_script_jint_guidance.py:625-631: end_sources can now hold None. The result doesn't change: the old code kept the raw non-=vars. string, which matched no row either. {s for s in … if s} would make that intent explicit.
  • Low: the bpmn_live.py:35-38 try/except ImportError fallback import works. It does make bpmn_live depend on bpmn_check, so keep that one-directional (no circular import exists today).

2. _shared/bpmn_check.py:181-186 — shallowest match in find_bpmn_file

OK. This fixes the Foo-work/Foo.bpmn < Foo.bpmn sort trap. Nit: Path(p).parts.__len__() should be len(Path(p).parts).

3. _shared/bpmn_check.py:698-741 — _catalog_verb / get-by-id

Low. The logic does what the PR body says: a list op against a GETBYID node still fails, and a bare GET with no metadata stays a list read. The inline comment at :730-733 points at the wrong place, though. It says the verb lives in instanceParameters.httpMethod, but the docstring and the code read activityMetadata.activity.configuration.httpMethod. Make the two agree so the next editor doesn't "fix" the code to match the comment. Fix this →

4. check_script_jint_guidance.py:754 — start-event scope

OK. start_id is defined at :469, before it is used.

5. e2e/customer_escalation_triage/* — project resolved with resolve_bpmn_file

Medium. The grader change is mechanically correct:

  • resolve_contract(path) callers in test_behavior_checker.py already pass a path.
  • No other code reads the removed module-level PROJECT/BPMN_FILE.
  • print_bpmn_path.py gets absolute paths from rglob, so .relative_to(os.getcwd()) is safe.

The problem is that the prompt still fixes the location: customer_escalation_triage.yaml:68-71 says "…at CustomerEscalationTriageSolution/CustomerEscalationTriage/ with the process in CustomerEscalationTriage.bpmn". The old comment correctly called the pinned path a stated requirement. Now the graders accept a layout the prompt rules out, so prompt and grader disagree. Pick one:

  • (a) Relax the prompt (name the project and stop pinning the path), or
  • (b) Keep the prompt and add a short comment saying the requirement is intentionally not graded because Studio Web cannot honour it.

Also: main() in both checkers now uses UPPER_CASE local names (BPMN_FILE, PROJECT). Lowercase them, since they are no longer module constants.

6. paginated_reference_lookup.yaml — advisory weights set to 0

OK. This matches the #3516 rationale.

7. evaluate/{evaluator_type_choice,local_crud,no_auto_upload,simulation/simulation_crud}.yaml — outcome check replaces telemetry

Low. Grading the workspace outcome instead of the CLI verb follows test-writing.md, and none of the four tasks mounts a template that could pre-seed a project.uiproj.

These criteria are still weight: 1.0 with pass_threshold: 0.0, which is exactly the "never fails but still moves the score" pattern fix 6 removes. Either give them weight 0 like fix 6, or say they are meant to count. Also, project.uiproj existing is close to implied by each task's "BPMN artifact exists" criterion, so this check adds little signal.

8. jdbc_databricks_query.yaml — preflight

OK. The <connector-key>=<folder> syntax is supported (_setup/preflight_connections.py:123-146), and _setup is mounted from ../../_setup. See Issues for Manual Review for the folder name.

9. Fixtures under _shared/fixtures/run_2026_11_19/

Low. These are real artifacts, which is the point. They do carry real tenant identifiers:

  • the connection resourceKey and folderKey GUIDs in testmanager/.../Process.bpmn:18-19,35
  • the Test Manager ProjectId at :43

They are not credentials, and tests/ is not part of the shipped package. Still, consider swapping in placeholder GUIDs if these fixtures don't depend on the exact values.

10. _shared/test_run_2026_11_19_graders.py

OK. It covers the accept and reject pairs listed in the PR body.

What's Missing

  • The customer_escalation_triage prompt (yaml:68-71) wasn't brought in line with the relaxed grader (see feat(CodedAgents): group skill by functionality #5).
  • The four new run_command advisory criteria don't get the same weight-0 treatment as feat(CodedAgents): file-sync skill and deployment fixes #6, for consistency.
  • No passing coder_eval run is claimed for the changed task YAMLs. The PR shows unit-test and validator evidence only, and the "live canvas check" is still marked as see comment below / update. /lint-task flags a missing passing-run claim as High.

Area Ratings

Area Status Notes
Frontmatter OK No skill files touched
E2E Tests Issue Prompt and grader disagree in customer_escalation_triage; advisory weight inconsistent; no passing-run claim
Skill Body OK Untouched
References & Assets OK Untouched
Repo Hygiene OK Scoped to tests/tasks/uipath-maestro-bpmn/; fixtures contain tenant GUIDs (Low)

Issues for Manual Review

  • Fix 1 risk (the PR body raises this too): if the canvas only treats a literal =vars. start/end source as an argument bridge (ArgumentUtils.ts:7-23), then accepting =js:vars.<id> lets through output that looks right but doesn't bridge at runtime. The live canvas check is still pending; this should block merge until it lands.
  • Fix 4: whether the runtime treats GET + metadata GETBYID as get-by-id is unverified, as the PR body says.
  • JDBC preflight folder: the preflight asserts the JDBC connection lives in folder uipath-maestro-flow. Confirm that is where the BPMN suite's Databricks fixture connection lives, and not just the Flow suite's.
  • I couldn't run pytest tests/tasks/uipath-maestro-bpmn/_shared here because the command needed approval in this runner. The "2692 passed" figure comes from the PR body and was not re-verified.

Conclusion

The grader changes are well targeted and well evidenced. Before merging:

  1. Settle the customer_escalation_triage mismatch (Medium): the prompt pins a path the graders no longer require.
  2. Post the live canvas check for =js:vars.: it decides whether fix 1 is "accepted" or actually "correct".

The rest is Low polish: the stale comment in _operation_matches, the advisory weights on the four new outcome checks, and lowercasing the local variables in both checkers' main().

@DevMomo
DevMomo marked this pull request as draft November 19, 2026 20:07

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant