Repository navigation
feat(uipath-mcp-servers): add MCP server guardrails [PRODEV-1345] - #4930
PopescuTudor wants to merge 1 commit into
Conversation
|
Claude finished @PopescuTudor's task in 1m 39s —— View job SummaryThis PR teaches Change-by-Change Review1.
|
| Area | Status | Notes |
|---|---|---|
| Frontmatter | OK | Under 1024 chars, redirects kept, no status tag |
| E2E Tests | Issue | Two good smoke tasks; Rule 12's ask path is untested |
| Skill Body | Issue | Guardrail facts copied three ways across Rule 12, the Guardrails section and the guide |
| References & Assets | OK | Anchors and links resolve; naming follows conventions |
| Repo Hygiene | OK | No secrets or personal paths; fixtures use synthetic values; changes stay within the skill |
Issues for Manual Review
llm-as-judgethreshold scale (guardrails-guide.md:31). The table gives "default 2, values 0, 2, 4 or 6", the same scale asharmful-content. Please confirm this against cli#5954–5956; it may have been copied from the row above.- CLI behaviour I could not verify:
- a PAT caller always gets
Unavailablefromllm-as-judge; prompt-injectionwithblockneeds no--reason;updatewrites the new rule first and then deletes the old one.
- a PAT caller always gets
- "On/off state".
update"keeps the rule's on/off state" (SKILL.md:172) while "no flag turns a rule off". Please confirm that the on/off state really is a UI-only toggle that the CLI preserves. - Sensitive-content scan not run. I was not permitted to run
scripts/check-sensitive-content.py, so the required CI check is the authority. The entity names (USSocialSecurityNumberand so on) are not values, so I expect no block-tier hits.
Conclusion
Approve with minor changes. The content is accurate as far as I can check without the CLI, and the graders are solid. Before merge, I'd remove the copied guardrail facts (item 2) and add a test for Rule 12's ask path (item 7). Merge also still depends on the CLI catalog and Studio Web snapshot refreshes the PR body lists.
|
Claude finished @PopescuTudor's task in 48s —— View job Coder-eval task lint (advisory)2 task YAMLs changed. Verdicts: 0 Critical, 0 High, 0 Medium, 1 Low, 1 OK. A separate PR-level High is raised under "Evidence of passing run" below. Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge. Evidence of passing run
Per-task lint
|
e2bcb86 to
1dc29a6
Compare
The skill covers `uip agenthub mcp-guardrails list|get|create|update|delete --validator <validator>` and `uip agenthub guardrail-models list` (CLI 1.208): a Guardrails section, Critical Rule 12 (ask before choosing block), a guide with validators, settings, defaults, entity names, log or block and judge models, and troubleshooting rows. Two mocked smoke evals, guardrail-create and guardrail-update, and three activation prompts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1dc29a6 to
1d13d99
Compare
What
The
uipath-mcp-serversskill now covers MCP server guardrails:uip agenthub mcp-guardrails list|get|create|update|delete --validator <validator>anduip agenthub guardrail-models list, from UiPath/cli#5954, #5955, #5956 and #5199 (CLI 1.208). PRODEV-1345.SKILL.md: a Guardrails section, Critical Rule 12 (ask before choosingblock), the description, When to Use and What NOT to Do. Checking a Server now saysrefresh-toolsdeletes the guardrails set on tools, and the server's own rules stay.references/guardrails-guide.md(new): which validator for which request, settings and defaults per validator, entity names, log or block (blockfails closed; a blockingllm-as-judgerule blocks every PAT caller), judge models, changing and removing a rule, rules the CLI does not manage.references/troubleshooting-guide.md: the guardrail messages, and the server-read 403 hint from UiPath/cli#5969.Evals
guardrail-create(smoke, mock): block US Social Security numbers and IBANs on a remote server. Graded from the call log: onemcp-guardrails create --validator pii-detection --action blockwith exactlyUSSocialSecurityNumberandInternationalBankingAccountNumber, the user's--reason, the server's folder; nomcp update.guardrail-update(smoke, mock): add credit card numbers to a PII rule that catches emails, keeping the rest.--entityreplaces the selection, so every executedupdatemust pass both entities and change no action, reason or Email threshold; no delete and create, nomcp update.uipath-mcp-servers-058to-060.Both evals ask for something the CLI's
--helpexamples do not show (those block Email and CreditCardNumber, and switch prompt injection to log), so copying an example does not pass them.The graders were checked against hand-written call logs, passing and failing, and against a simulated run through the shared mock. The tasks have not run under coder-eval yet: it is not available locally, so this PR's smoke job is their first run.
Before merge
assets/uip-catalog-snapshot.json) must list the new commands. The nightly refresh merges itself; the 11-17 snapshot predates them.assets/uip-browser-commands-snapshot.json) must list them too. Its refresh waits on chore: refresh uip browser command surface (cli c0d082ee2) #4629; until then the Studio Web check fails on the guardrail mentions.To ship with the commands, this merges before the 1.208 cut, Sunday 2026-11-29 06:00 UTC.
🤖 Generated with Claude Code