Skip to content

feat(uipath-mcp-servers): add MCP server guardrails [PRODEV-1345] - #4930

Open
PopescuTudor wants to merge 1 commit into
mainfrom
feat/mcp-servers-guardrails
Open

PopescuTudor wants to merge 1 commit into
mainfrom
feat/mcp-servers-guardrails

Conversation

@PopescuTudor

@PopescuTudor PopescuTudor commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What

The uipath-mcp-servers skill now covers MCP server guardrails: uip agenthub mcp-guardrails list|get|create|update|delete --validator <validator> and uip agenthub guardrail-models list, from UiPath/cli#5954, #5955, #5956 and #5199 (CLI 1.208). PRODEV-1345.

  • SKILL.md: a Guardrails section, Critical Rule 12 (ask before choosing block), the description, When to Use and What NOT to Do. Checking a Server now says refresh-tools deletes the guardrails set on tools, and the server's own rules stay.
  • references/guardrails-guide.md (new): which validator for which request, settings and defaults per validator, entity names, log or block (block fails closed; a blocking llm-as-judge rule blocks every PAT caller), judge models, changing and removing a rule, rules the CLI does not manage.
  • references/troubleshooting-guide.md: the guardrail messages, and the server-read 403 hint from UiPath/cli#5969.

Evals

  • guardrail-create (smoke, mock): block US Social Security numbers and IBANs on a remote server. Graded from the call log: one mcp-guardrails create --validator pii-detection --action block with exactly USSocialSecurityNumber and InternationalBankingAccountNumber, the user's --reason, the server's folder; no mcp update.
  • guardrail-update (smoke, mock): add credit card numbers to a PII rule that catches emails, keeping the rest. --entity replaces the selection, so every executed update must pass both entities and change no action, reason or Email threshold; no delete and create, no mcp update.
  • Activation prompts uipath-mcp-servers-058 to -060.

Both evals ask for something the CLI's --help examples do not show (those block Email and CreditCardNumber, and switch prompt injection to log), so copying an example does not pass them.

The graders were checked against hand-written call logs, passing and failing, and against a simulated run through the shared mock. The tasks have not run under coder-eval yet: it is not available locally, so this PR's smoke job is their first run.

Before merge

  • The CLI verb catalog (assets/uip-catalog-snapshot.json) must list the new commands. The nightly refresh merges itself; the 11-17 snapshot predates them.
  • The Studio Web command snapshot (assets/uip-browser-commands-snapshot.json) must list them too. Its refresh waits on chore: refresh uip browser command surface (cli c0d082ee2) #4629; until then the Studio Web check fails on the guardrail mentions.

To ship with the commands, this merges before the 1.208 cut, Sunday 2026-11-29 06:00 UTC.

🤖 Generated with Claude Code

@PopescuTudor
PopescuTudor requested review from a team and cristipufu as code owners November 18, 2026 15:26
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @PopescuTudor's task in 1m 39s —— View job


Summary

This PR teaches uipath-mcp-servers the new uip agenthub mcp-guardrails list|get|create|update|delete and guardrail-models list commands. It adds Critical Rule 12 (ask before block), a new references/guardrails-guide.md, the guardrail error messages in the troubleshooting guide, two mocked smoke tasks and three activation prompts. The structure is sound, the graders are careful, and the anchors and links resolve. My two main findings: the guardrail facts are copied across SKILL.md, the guide and Rule 12, and no test exercises the "ask before block" path that Rule 12 exists for.

Change-by-Change Review

1. skills/uipath-mcp-servers/SKILL.md: frontmatter, When to Use, Folder Context

OK. The description is still well under 1024 characters, keeps its → redirects and has no status tag. The Folder Context table puts guardrail-models list in the "no folder flag" row and the mcp-guardrails verbs in the "folder required" row, which matches the guide. The refresh-tools bullet now says tool-level guardrails are deleted and server-level ones stay, a useful distinction.

2. SKILL.md:165-176, the Guardrails section, against Rule 12 and guardrails-guide.md

Medium (Reference Pointer Discipline / Duplication). This section links to guardrails-guide.md but repeats facts that the guide or Rule 12 already states:

  • The PAT warning appears three times. It is in Rule 12 (SKILL.md:60), the create bullet (SKILL.md:171) and the guide (guardrails-guide.md:21). "Ask when --action is unspecified" also appears three times: SKILL.md:60, :171 and guardrails-guide.md:23.
  • "block needs --reason, except on prompt-injection" is in both SKILL.md:171 and guardrails-guide.md:18.
  • "No flag turns a rule off: off means delete" is in both SKILL.md:173 and guardrails-guide.md:58.
  • The full delete command and the Rule 8 confirm appear again in the guide's Removing a Rule section (guardrails-guide.md:58).

If one copy changes later (for example the PAT behaviour once the CLI handles it), the others will silently go stale.
Fix:

  • Cut the create bullet down to the command plus "--action: Rule 12; settings and defaults: guide".
  • Drop "When the user did not say which action they want, ask" and the PAT sentence from guardrails-guide.md § Log or Block, and link Rule 12 instead.
  • Keep the "off means delete" fact in only one place, and remove the restated delete command from the guide.

Fix this →

3. SKILL.md:60, Critical Rule 12

OK. It is numbered and explicit, it says to warn about the PAT behaviour even when the user asked for block, and What NOT to Do repeats it as an anti-pattern.

4. references/guardrails-guide.md (new)

OK, apart from the duplication in item 2. The validator-choice table, the settings and defaults table, the entity catalog and the "never guess an entity or model" rule are clear and easy for an agent to follow. The file name ends in -guide.md, the file is linked from SKILL.md's Reference Navigation, and the ../SKILL.md#folder-context and #critical-rules anchors resolve.

5. references/troubleshooting-guide.md

OK. The new ## Guardrails anchor matches the link from SKILL.md, and the #entities and #settings links into the guide resolve. The server-read 403 row comes from cli#5969 rather than the guardrails work, but it is small, in scope for this skill, and the PR body mentions it.

6. tests/tasks/uipath-mcp-servers/guardrail-create, guardrail-update

OK, with one coverage gap (item 7).

  • The task IDs follow skill-<domain>-<capability>.
  • The tags (uipath-mcp-servers, smoke, mode:build, lifecycle:configure) match the sibling user-connection-set task.
  • The prompts are minimal, and the tasks grade side effects from the call log.
  • values() uses (?![\w-]), so --entity never matches --entity-threshold and --mcp never matches --mcp-....
  • entities() handles both repeated --entity flags and comma-separated values, the way the CLI matches them.
  • The after: gating in the manifest matches the shared mock's semantics (mocks/uip:200).
  • The update grader fails on an executed delete or create, which correctly pins the "update in place" rule.
  • One minor point: "match": "agenthub mcp get vendor-mcp" only matches when the slug comes straight after get. The sibling tasks use the same pattern, so this is just a note.

7. Test coverage of the new capability

Medium. The README coverage table credits guardrail-create with Rule 12, but its prompt says "block". The task therefore never tests the behaviour the rule is about: asking when the user did not choose block or log. Activation prompt -058 also says "blocked". Nothing exercises llm-as-judge, guardrail-models list or delete either.
Fix: add a smoke task whose prompt gives no action, for example "add a PII guardrail for emails on vendor-mcp". Grade it with command_not_executed on mcp-guardrails create … --action block, or with a grader that passes if no create ran or the create used --action log. Ideally also add a task that requires guardrail-models list before an llm-as-judge create.

8. tests/tasks/activation/uipath-mcp-servers.jsonl, tests/tasks/uipath-mcp-servers/README.md

Low. In the README's layout tree, the │ ├── task.yaml / fixtures/… subtree used to hang under user-connection-status/ and now hangs under guardrail-update/. This is cosmetic only.

What's Missing

  • No test covers the ask-before-block path of Rule 12, the llm-as-judge judge-model lookup, or delete.
  • assets/uip-catalog-snapshot.json and assets/uip-browser-commands-snapshot.json do not list mcp-guardrails or guardrail-models yet; I found zero matches. The PR body already tracks this as a merge blocker.
  • CODEOWNERS needs no change because no new skill path was added. No skill-flavors/*/uipath-mcp-servers overrides exist, so no flavor review is needed.

Area Ratings

Area Status Notes
Frontmatter OK Under 1024 chars, redirects kept, no status tag
E2E Tests Issue Two good smoke tasks; Rule 12's ask path is untested
Skill Body Issue Guardrail facts copied three ways across Rule 12, the Guardrails section and the guide
References & Assets OK Anchors and links resolve; naming follows conventions
Repo Hygiene OK No secrets or personal paths; fixtures use synthetic values; changes stay within the skill

Issues for Manual Review

  • llm-as-judge threshold scale (guardrails-guide.md:31). The table gives "default 2, values 0, 2, 4 or 6", the same scale as harmful-content. Please confirm this against cli#5954–5956; it may have been copied from the row above.
  • CLI behaviour I could not verify:
    • a PAT caller always gets Unavailable from llm-as-judge;
    • prompt-injection with block needs no --reason;
    • update writes the new rule first and then deletes the old one.
  • "On/off state". update "keeps the rule's on/off state" (SKILL.md:172) while "no flag turns a rule off". Please confirm that the on/off state really is a UI-only toggle that the CLI preserves.
  • Sensitive-content scan not run. I was not permitted to run scripts/check-sensitive-content.py, so the required CI check is the authority. The entity names (USSocialSecurityNumber and so on) are not values, so I expect no block-tier hits.

Conclusion

Approve with minor changes. The content is accurate as far as I can check without the CLI, and the graders are solid. Before merge, I'd remove the copied guardrail facts (item 2) and add a test for Rule 12's ask path (item 7). Merge also still depends on the CLI catalog and Studio Web snapshot refreshes the PR body lists.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @PopescuTudor's task in 48s —— View job


Coder-eval task lint (advisory)

2 task YAMLs changed. Verdicts: 0 Critical, 0 High, 0 Medium, 1 Low, 1 OK. A separate PR-level High is raised under "Evidence of passing run" below.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

  • ❌ High: the PR body says the opposite of a passing-run claim: "The tasks have not run under coder-eval yet: it is not available locally, so this PR's smoke job is their first run." Hand-written call logs and a simulated mock run are useful, but they don't count as a run. Once the smoke job is green, please edit the PR description to add a line like: Ran skill-mcp-servers-guardrail-create and skill-mcp-servers-guardrail-update in the PR smoke job and both passed.

Per-task lint

tests/tasks/uipath-mcp-servers/guardrail-create/task.yaml: verdict OK

The prompt (L20–22) states only the goal and gives the reason string as a ground-truth anchor. It doesn't name a validator, an entity or a flag. The grader (check_guardrail_create.py) reads the mock call log. It requires exactly one executed create with the exact entity set, the reason and the folder, and it fails on any mcp update. A lazy agent can't pass without calling the CLI correctly.

tests/tasks/uipath-mcp-servers/guardrail-update/task.yaml: verdict Low

Issues:

  • [Low] Meaningful coverage: the prompt (L21–22) says "Keep it and its settings", but check_guardrail_update.py only checks that --threshold is kept. An update that rewrites --reason (the stored value is "Prompt injection is not allowed" in fixtures/.../mcp_guardrails_get.json), or changes the enabled state or scope, still passes.

Suggested fixes:

  • In problem() (check_guardrail_update.py ~L47), also reject a --reason that differs from the stored "Prompt injection is not allowed". Apply the same check to any disable/scope flags the update --help fixture exposes. Missing flags should keep passing.

Within-PR duplicates

  • No duplicate clusters detected. The two tasks share a mock scaffold but test different operations (create with entity selection vs. an in-place action switch that keeps the stored settings).

Notes

  • CLI verb reachability not checked: scripts/check-cli-verbs.py couldn't run in this job because of a tool permission. As the PR body says, the 11-17 assets/uip-catalog-snapshot.json doesn't list agenthub mcp-guardrails yet. Both tasks use only command_not_executed patterns plus run_command graders, so no command_executed criterion depends on the catalog. Please rerun /lint-task after the nightly refresh lands.
  • No near-duplicates among siblings: no other task in tests/tasks/uipath-mcp-servers/ covers guardrails.

Conclusion

  • ⚠ 1 task has an issue (max severity Low), and the PR is missing its passing-run claim (High). Advisory only, not blocking merge.

@PopescuTudor PopescuTudor self-assigned this Nov 18, 2026
@PopescuTudor
PopescuTudor force-pushed the feat/mcp-servers-guardrails branch 2 times, most recently from e2bcb86 to 1dc29a6 Compare November 18, 2026 15:36
The skill covers `uip agenthub mcp-guardrails list|get|create|update|delete
--validator <validator>` and `uip agenthub guardrail-models list` (CLI 1.208):
a Guardrails section, Critical Rule 12 (ask before choosing block), a guide
with validators, settings, defaults, entity names, log or block and judge
models, and troubleshooting rows. Two mocked smoke evals, guardrail-create and
guardrail-update, and three activation prompts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant