Skip to content

fix(uipath-agents): count Studio Web's resources add where tasks required resources refresh - #5004

Open
astridstaicu wants to merge 2 commits into
mainfrom
fix/agents-sw-resources-add-criterion
Open

astridstaicu wants to merge 2 commits into
mainfrom
fix/agents-sw-resources-add-criterion

Conversation

@astridstaicu

Copy link
Copy Markdown
Collaborator

What

Eight low-code agent tasks gate on uip solution resources refresh. They now also accept uip solution resources add for the task's own resource kind, through the CLI or Studio Web's ManageSolutionResources host tool.

Task Kind accepted by add
external_rpa_tool, external_maestro_tool, external_apiworkflow_tool, external_agent_tool Process
context_index, conversational/with_context_grounding Index
is_connector_tool Connection
memory_space MemorySpace

refresh still counts, so terminal harnesses grade as before.

Why

#4868 added this rule to the studioweb uipath-agents skill: "Never run uip solution resources refresh in Studio Web; declare each tenant resource with uip solution resources add". Refresh always fails there, because Studio Web keeps no .uipx on disk. The tasks still required refresh, so they penalized agents that followed the skill.

Evidence from the Studio Web daily eval:

  • 2026-11-18 (2026-11-18_09-22-24): on external_rpa_tool the agent ran uip solution resources refresh. The command errored, and the criterion passed anyway.
  • 2026-11-19 (2026-11-19_09-23-02): all eight tasks failed only this criterion ("Matched 0/1 required commands"). Each agent had declared the resource with resources add, for example ManageSolutionResources {"action": "add", "kind": "Connection", ...} on is_connector_tool and {"action": "add", "kind": "MemorySpace", ...} on memory_space.

The patterns follow the #4867 lookup patterns. The \A(?!\{") guard keeps the CLI alternative off other tools' JSON, and the host-tool alternative needs action: add and the right kind in params.

Testing

  • tests/scripts/test_task_command_patterns.py:
    • New: test_declaring_the_resource_counts_refresh_or_add, built from the calls recorded on 11-19.
    • New: test_declaring_the_resource_needs_add_of_the_same_kind.
    • Existing: the quote and nested-edit guard tests now cover the new patterns too.
  • The old pattern does not match the recorded host-tool add call. The old criteria were also filtered to Bash, so the host tool could never count.
  • tests/scripts (1271 passed), the uipath-agents checker tests, and check-task-driver.py, check-task-host-paths.py and check-task-doc-citations.py pass. One unrelated local failure, test_stage_shared.py, fails the same way on main.

🤖 Generated with Claude Code

…ired resources refresh

Eight low-code agent tasks gate on `uip solution resources refresh`. The
studioweb skill (#4868) now tells Studio Web agents not to run it, because it
always fails there (no .uipx on disk), and to declare each resource with
`uip solution resources add` instead. On the 2026-11-19 Studio Web daily all
eight failed only this criterion, after the agents declared the resource with
`resources add` (CLI or the ManageSolutionResources host tool). The day
before, the agent ran refresh, it errored, and the criterion passed.

Each criterion now accepts refresh, or an add of the task's own resource kind
(Process, Index, Connection, MemorySpace), through the CLI or the host tool,
with the same guards as the #4867 lookup patterns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @astridstaicu's task in 1m 11s —— View job


Summary

The PR changes the "declare resource" criterion in 8 low-code uipath-agents tasks. Each task now accepts uip solution resources refresh, or resources add for its own resource kind, sent through the CLI or through Studio Web's ManageSolutionResources host tool. The change is correct and well-scoped, and it follows the lookup patterns from #4867 exactly.

Change-by-Change Review

1. The 8 task YAMLs (command_pattern + description)

OK

  • The regex has the same shape as the merged resources list patterns (for example external_apiworkflow_tool.yaml:59). Both use the \A(?!\{") guard, so quotes from Write/Task JSON are not counted. Both use [^{}]* lookaheads, so action/kind values nested inside an edit patch are not counted either.
  • Each kind mapping is right for its task. The external agent, RPA, Maestro and API workflow tasks map to Process, which agrees with project-lifecycle.md:290 and process.md. The other tasks map to Index, Connection and MemorySpace.
  • Removing tool_name: "Bash" is required, because the host tool is recorded as a ProxyTool call. The JSON guard keeps this safe.
  • refresh\b still counts, so terminal harnesses keep their current grading. The PR body already says that a failed refresh in Studio Web still gets credit, because command_executed doesn't check exit status. That limitation existed before and is out of scope here.

2. tests/scripts/test_task_command_patterns.py

Low. The new positive and negative tests use the commands recorded on 11-19, which is good. Some gaps:

  • The "existing guard tests cover the new patterns" claim is weaker than it reads. _host_tool_patterns() does pick the new patterns up. But the QUOTES strings (lines 296–308) only quote list/get commands, so the new refresh|add alternatives pass without being exercised. The same applies to NESTED (lines 318–325), which never puts action: "add" in an edit patch. Two additions would make those tests real for this PR:
    • a quoted uip solution resources refresh / resources add --kind Process string in QUOTES
    • "add" in the NESTED action tuple
      Fix this →
  • Results differ with and without DOTALL (Low, inherited from fix(uipath-agents): count Studio Web's ManageSolutionResources lookups in resource criteria #4867). The module docstring says every pattern must grade the same both ways. With a line continuation, such as uip solution resources add --source remote \ followed by --kind Process ... on the next line, add\b.*--kind\s+Process matches only under DOTALL. Under DOTALL, .* can also run past ; or a newline, so add --kind Index ...; uip solution resources list --kind Process would count as a Process add. Neither case is likely in these single-resource tasks, and I confirmed it by reading the regex, not by running it. If you want to tighten it, use add\b[^;&|\n]*--kind\s+Process in both this PR's patterns and the lookup patterns, and add a test for the line-continuation case.

What's Missing

  • Nothing blocking. The test gaps described in item 2 are the only additions I'd suggest.

Area Ratings

Area Status Notes
Frontmatter OK No skill files changed
E2E Tests OK Criteria now match the studioweb skill rule from #4868
Skill Body OK n/a
References & Assets OK n/a
Repo Hygiene OK Scoped to uipath-agents tasks. The cloudKey GUID in the test is a recorded test-tenant id, not a secret

Issues for Manual Review

  • I couldn't run pytest or ad-hoc Python here because the commands needed approval. The pass counts above come from the PR description, and the regex behaviour comes from reading the patterns.
  • Please confirm that Studio Web's ManageSolutionResources add takes kind: "Process" for a published agent (EmailDrafter), the same way the CLI does. No call was recorded for external_agent_tool.

Conclusion

Approve, with optional fixes. The change is correct, consistent with #4867, and fixes a real grading penalty. The suggestions above are Low-severity test-coverage improvements.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @astridstaicu's task in 54s —— View job


Coder-eval task lint (advisory)

8 task YAMLs changed. Verdicts: 0 Critical, 0 High, 0 Medium, 8 Low, 0 OK. One PR-level High: the PR body has no passing-run claim.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

  • ❌ High: the PR body doesn't say the changed tasks were run and passed with the new criterion. It cites the failing 2026-11-19 Studio Web run and lists unit-test and checker passes (tests/scripts, check-task-*.py), but no coder-eval run after the change. Please edit the PR description to add a line like: Ran skill-agent-external-rpa-tool (and the other 7) on the Studio Web harness and a terminal harness, and they passed. Running on both matters, because the change has to keep terminal grading working too.

Per-task lint

The scope is the changed criterion. The rest of each task is unchanged from main. I couldn't run scripts/check-cli-verbs.py in this job because the command needed approval. Going by the catalog verbs solution resources refresh|add|list|get, the patterns look reachable.

tests/tasks/uipath-agents/lowcode/external_rpa_tool/external_rpa_tool.yaml — verdict: Low (theme-captured; see Theme 1, 2)

tests/tasks/uipath-agents/lowcode/external_maestro_tool/external_maestro_tool.yaml — verdict: Low (theme-captured; see Theme 1, 2)

tests/tasks/uipath-agents/lowcode/external_apiworkflow_tool/external_apiworkflow_tool.yaml — verdict: Low (theme-captured; see Theme 1, 2)

tests/tasks/uipath-agents/lowcode/external_agent_tool/external_agent_tool.yaml — verdict: Low (theme-captured; see Theme 1, 2)

tests/tasks/uipath-agents/lowcode/context_index/context_index.yaml — verdict: Low (theme-captured; see Theme 1, 2)

tests/tasks/uipath-agents/lowcode/conversational/with_context_grounding/with_context_grounding.yaml — verdict: Low (theme-captured; see Theme 1, 2)

tests/tasks/uipath-agents/lowcode/is_connector_tool/is_connector_tool.yaml — verdict: Low (theme-captured; see Theme 1, 2)

tests/tasks/uipath-agents/lowcode/memory_space/memory_space.yaml — verdict: Low (theme-captured; see Theme 1, 2)

Within-PR duplicates

  • No duplicate clusters detected. The PR adds no new tasks; it changes one criterion in eight existing tasks.

Themes

  • Theme 1 — [Low] Could pass for the wrong reason (unbounded-dot-repeat). In the CLI branch, add\b.*--kind\s+<Kind> puts .* between the verb and the kind. A kind named in a later command in the same Bash call still counts. For example, on external_rpa_tool (line 103), uip solution resources add --kind Index … && uip solution resources list --kind Process matches. Fix: use add\b[^;&|\n]*--kind[\s=]+<Kind>\b. This keeps the match inside one command, and [\s=] also accepts --kind=Process, which the current pattern rejects.
  • Theme 2 — [Low] Meaningful coverage (looser terminal grading). The old criterion was refresh-only and Bash-only. Now a terminal agent that runs resources add and never refresh also passes, even though the terminal skill teaches refresh. The weight-5.0 check_*.py run_command criteria still check the resulting resource/binding shape, so the risk is small. If terminal runs should still be held to refresh, consider a separate advisory (pass_threshold: 0) criterion for refresh on the terminal variant. Otherwise, leave as is.

Notes for the author (no change needed):

  • Dropping tool_name: "Bash" is correct here: the host-tool call is recorded as a non-Bash ProxyTool call, so the old filter could never match it. The \A(?!\{") guard keeps the CLI branch off other tools' JSON, consistent with the fix(uipath-agents): count Studio Web's ManageSolutionResources lookups in resource criteria #4867 lookup patterns at line 59/66.
  • test_declaring_the_resource_needs_add_of_the_same_kind is a good negative-case set. Consider adding the cross-command case from Theme 1 as a negative test once the pattern is tightened.

Conclusion

  • ⚠ 8 tasks have issues; the highest severity is Low (shared Themes 1 and 2). The missing passing-run claim is a PR-level High. Advisory only, not blocking merge.

… guard tests cover refresh and add

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant