Skip to content

fix(uipath-agents): check the IS connection can see the value the user named - #4910

Merged
astridstaicu merged 2 commits into
mainfrom
fix/uipath-agents-is-describe-keep-catalog-activity
Nov 18, 2026
Merged

astridstaicu merged 2 commits into
mainfrom
fix/uipath-agents-is-describe-keep-catalog-activity

Conversation

@astridstaicu

@astridstaicu astridstaicu commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Why

After #4868, helpdesk-jira-triage still fails on Studio Web (eval run adhoc-2026-11-18_10-24-40, build 14320034) with "no Jira create-issue IS tool". The tenant has three Jira connections, and only one of them, is-sandboxes-test…sandbox-380 (ac8506ff…), can see project AB. The other two, including skills-evals-golden-jira, answer No project could be found with key 'AB' when the agent lists project/AB/issuetypes. The project is either missing from those sites or the connection's account can't browse it.

  • The nightly agent (2026-11-17_04-23-30) tried the listing on all three connections, found AB on ac8506ff, and built the tool on curated_create_issue.
  • The Studio Web agent tried two of them. Without AB's issue types, describe curated_create_issue returns no summary or description, so it switched to the raw issue object, which the grader rejects.

The skill told the agent to recommend the default enabled connection, and never said to check that the connection can see what the user named.

What

integration-service.md:

  • Step 3: when the task names something in the connected system (a Jira project, a Slack channel), the connection must be able to see it.
  • Step 6, "Fields that load after a selection": when listing a value the user named comes back not found, run the same listing on the other enabled connections and tell the user which ones can see it, so they choose. When no one can answer, use the one that can and say so. Use the chosen connection in the tool, its resourceKey, and the solution. If none can see it, tell the user and stop.

This is canonical text with no Studio Web override, so both trees change.

The task's setup has a related problem, and this PR does not fix it. The prompt says "using the existing JIRA Connection", but the eval's own skills-evals-golden-jira connection can't see project AB. Giving that connection access to AB, or naming the connection in the prompt, would make the task deterministic.

Verification

  • npm run skills:validate, skills:check-links and skills:build pass, and the new text is in the built studioweb tree.
  • The cause is taken from the transcripts of the Studio Web run above and the 2026-11-17_04-23-30 nightly. Both runs see the same three connections, and only ac8506ff lists AB.
  • Studio Web eval reruns on this branch (helpdesk-jira-triage and expense-fraud-review, twice each) are builds 14321438 and 14321439. expense-fraud-review failed in the same run on an escalation field that fix(uipath-agents): keep IS tools on the catalog activity; route long-document research to Deep RAG #4868 didn't touch, so it's included to see whether that failure was a one-off.

🤖 Generated with Claude Code

…oject

When several connections exist for a connector, the agent took the first one
even though its Jira site had no project AB, so the issue fields never loaded
and it fell back to the raw `issue` object. Step 3 now recommends the
connection that can see what the task names, and Step 6 says to retry the
listing on each enabled connection on a 404 and never fall back to the raw
object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @astridstaicu's task in 44s —— View job


Summary

Two additions to the low-code IS guide (integration-service.md). Step 3 now says to recommend the connection that can see the entity the user named. Step 6 now says that when the reference listing for a user-named value returns 404, the agent should try the other enabled connections, switch to the one that finds the value, and stop if none does.

Change-by-Change Review

1. Step 3: recommend the connection that can see the named entity (integration-service.md:272)

Low

The rule is correct and points to Step 6 instead of repeating it.

  • Unclear ordering. "Recommend the default enabled one" is still followed by "recommend the connection that can see it", and the check that tells them apart only happens in Step 6, after the user has already confirmed a connection in Step 3. To read in order, the sentence could say: "…recommend the default enabled one; if the task names something in the connected system, run the Step 6 listing on each enabled connection first and recommend the one that finds it."

2. Step 6: fall back to another connection on 404 (integration-service.md:315)

Medium

This fixes the failure described in the PR: the agent gives up after two connections and then switches to the raw issue object. Two gaps:

  • Other tools on the same connector are not covered. Line 359 (and line 196) say solutionProperties.resourceKey "is shared by all tools using the same connector." If the agent has already built another tool on this connector (for example, Jira search) with the first connection, switching only this tool leaves two different resourceKeys for one connector, which breaks that rule. Add: "…and in every other tool on this connector, since they share one resourceKey." Alternatively, run the check before any tool on the connector is built.
  • The user is not told about the switch. In Step 3 the user confirms a connection ("let the user confirm"). Step 6 now replaces it without saying so. One clause would fix this: "tell the user which connection you switched to and why."

Minor:

  • "Never fall back to the raw object" repeats line 284. Here it is a short reminder at the exact point where the agent got it wrong, so it is acceptable.
  • Only 404 is covered. A 401 or 403 on a broken connection can fail the same way. Consider "returns 404 or says the value does not exist."

3. Flavor overrides

OK

The skill-flavors/*/…/integration-service.md override replaces only is-solution-files-refresh, is-connection-definition-refresh and is-step8-refresh. Steps 3 and 6 are outside any marked block, so every flavor gets this change. This matches the PR description.

What's Missing

  • No coder_eval task covers a tenant with several connections where only one can see the target entity. Coverage currently depends on the queued Studio Web rerun of helpdesk-jira-triage. A mocked or fixture-based check (for example, extending _shared/check_is_jira_create_issue.py to assert that the chosen connection.id is the one that sees the project) would keep this from regressing.

Area Ratings

Area Status Notes
Frontmatter OK Not touched
E2E Tests Issue No regression task for picking among several connections (eval rerun pending)
Skill Body OK Not touched
References & Assets Issue Shared resourceKey across tools not handled when switching connection
Repo Hygiene OK Change limited to one skill; no secrets or personal paths; no new links

Issues for Manual Review

  • Confirm that uip is resources run list gives a recognisable 404 for a missing project. The agent needs to tell "not on this site" apart from transport or authentication errors.
  • Confirm that uip solution resources refresh picks up the switched connection from bindings_v2.json with no other step. The PR text says "when adding the connection to the solution" but doesn't name a command.

Conclusion

The fix matches the root cause found in the transcripts and is small. Approve after addressing the shared-resourceKey gap. Telling the user about the switch is also recommended. The other points are optional.

… named value

Replace the silent retry across connections with: list which enabled
connections can see the value and let the user choose; when no one can
answer, use the one that sees it and say so. Drop the repeated raw-object
rule (Step 4 already has it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@astridstaicu astridstaicu changed the title fix(uipath-agents): pick the IS connection that can see the user's project fix(uipath-agents): check the IS connection can see the value the user named Nov 18, 2026
@astridstaicu
astridstaicu merged commit 8386b5b into main Nov 18, 2026
50 checks passed
@astridstaicu
astridstaicu deleted the fix/uipath-agents-is-describe-keep-catalog-activity branch November 18, 2026 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant