Tests that verify AI agents can correctly use skills from this repository. Tests are defined as coder_eval task YAML files.
-
UiPath private PyPI credentials (optional) — only needed if
coder-evalresolves to packages on the UiPath Azure DevOpsml-packagesfeed. Export these before runningmake installto enable the private feed:export UV_INDEX_UIPATH_USERNAME=<your-ado-username> export UV_INDEX_UIPATH_PASSWORD=<your-ado-pat>
The Makefile composes these into
UV_EXTRA_INDEX_URLforuv pip install. If either variable is empty, install continues against public PyPI only and prints a notice. -
coder-eval — install the version pinned in
.coder-eval-version(creates a local.venv, requires Python 3.13+):cd tests make installDependencies are constrained to the
pip freezeof the pinnedghcr.io/uipath/coder-eval-agentimage, the same constraints CI uses. Needs Docker; without it the install prints a warning and runs unconstrained. -
uip CLI — the UiPath CLI must be available:
npm install -g @uipath/cli
Do not add
@uipath/clitosandbox.node.env_packagesin task YAMLs. The GH smoke runner installs it globally before any task runs. Listing it inenv_packagesis redundant and, when pinned to a version, causes skew against the runner's@latestinstall. -
Environment setup — API keys and other environment variables are required. See the coder_eval README for environment setup (
.env, API keys, etc.).
Platform-specific sandbox driver:
- Linux smoke tests use
driver: dockerfor better isolation. Build the Docker image once before running:cd .coder_eval make docker-image cd ../tests- Windows RPA tests use
driver: tempdir(Docker image not available on Windows runner).
cd tests
# Run everything under the default config (full lifecycle, longest budget)
make all
# Run smoke tests (Linux, fast PR-gate budget)
make smoke
# Run Windows RPA smoke tests (tempdir)
make smoke_rpa
# Run e2e-tagged tests under the default config
make e2e
# Run tests matching a combination of tags (AND semantics — tasks must carry all listed tags) (defaults to experiments/default.yaml):
make tags TAGS="integration connector-feature"
# Optionally override the experiment config
make tags TAGS="integration connector-feature" EXPERIMENT=experiments/smoke.yaml
# Run all tests for a specific skill
make test-uipath-maestro-flow
# Run a single task file
SKILLS_REPO_PATH=$(cd .. && pwd) \
.venv/bin/coder-eval run tasks/uipath-maestro-flow/smoke/init_validate.yaml \
-e experiments/default.yamlThe SKILLS_REPO_PATH environment variable defaults to the parent directory (repo root) when using make.
All make targets run tasks serially by default (-j 1). Override with TASK_PARALLELISM:
# Run smoke tests with 4 tasks in parallel
TASK_PARALLELISM=4 make smoke
# Or export once for the shell session
export TASK_PARALLELISM=4
make allTests are organized into three types, distinguished by tags (not directories). All tests for a skill live together in tests/tasks/<skill-name>/.
| Tag | Purpose | Cadence |
|---|---|---|
smoke |
Skill triggers correctly, CLI produces valid output (1-5 simple scenarios) | Every PR |
integration |
Correct output across diverse scenarios, error paths, anti-patterns | Daily |
e2e |
Full lifecycle: Explore -> Plan -> Build -> Validate -> Deploy -> Run | Daily/weekly (check Dashboard) |
Tags drive make targets, coverage reports, and evalboard drilldown. The tags: list is a flat array of strings; most tag values carry a namespace prefix in key:value form so each dimension is independently queryable (e.g. where tag startswith "connector:" in ADX). Required tags are flat (no prefix) so existing --tags filters keep working.
| Dimension | Form | Purpose | Values |
|---|---|---|---|
| skill | flat, required | Skill under test | uipath-<name> — must match the skill folder (e.g. uipath-maestro-flow) |
| tier | flat, required | Test depth / cost | smoke, integration, e2e |
| mode | mode:X, required |
Coding Agents Scorecard mode | build (creating, designing, editing, deploying), operate (running, triggering, managing live instances/connectors/integrations), diagnose (investigating faults, inspecting traces, debugging) |
| lifecycle | lifecycle:X, required |
Coding Agents Scorecard lifecycle phase | discover (read-only exploration: list/get/inspect existing state), generate (produce a new local artifact: pack, scaffold, render), setup (mutate tenant state: create/edit/delete resources, deploy, configure) |
| shape | shape:X, optional |
Flow composition under test | single-node, multi-node (omit for smoke tests that don't build a flow) |
| node | node:X, repeatable |
Node type(s) under test | decision, switch, subflow, terminate, loop, transform, hitl, ixp (a published IXP model), extract-document (the Extract document node and its inline taxonomy) (omit script/http — ubiquitous) |
| resource | flat, present iff applicable | Marks tasks that exercise any resource-node type (coded-agent, lowcode-agent, api-workflow, rpa). The specific resource is implied by the file path / task_id. |
|
| connector | flat, present iff applicable | Marks tasks that use any IS connector. The specific connector is in the YAML body / file path. | |
| windows | flat, present iff applicable | Marks tasks that require a Windows host (e.g. RPA .xaml/.cs projects that need Studio Helm). Used by smoke-rpa-skills.yml to route the task to a windows-latest runner; Linux/macOS smoke runs skip it. |
|
| path-to-ga | flat, optional | Marks exhaustive, difficult, currently blocked, or historically fragile tasks that represent must-pass scenarios on the path to GA. | path-to-ga |
| autopilot-sw-ga | flat, optional | Marks Studio Web Autopilot use-case tasks tracked for GA (the uipath-agents low-code use-case suite). |
autopilot-sw-ga |
| outcome-graded | flat, optional | Marks tasks whose primary criteria grade the live outcome of an executed run rather than authored file contents. | outcome-graded |
| feature | feature:X, repeatable |
Cross-cutting capability orthogonal to node/resource/connector. Closed vocabulary: http, trigger, registry, transform, eval, approval-gate, write-back, escalation, connections, activities, records, entities, api-workflow, compliance, test-case, hooks, conversational. Do not invent leaf names like feature:ceql-where or directory-name markers like feature:connector-feature — those duplicate the file path. |
- Required on every task:
skill+tier+mode:*+lifecycle:*. These drivemaketargets, coverage, and evalboard dashboards. - One value per singular dimension (
tier,mode,shape). A task doesn't have two tiers. node:andfeature:are repeatable. A flow exercising decision and switch nodes gets bothnode:decisionandnode:switch.connector,resource,windows,path-to-ga,autopilot-sw-ga, andoutcome-gradedare flat boolean markers, not enumerations. Use them once per task; the specific connector/resource is identifiable from the file path,task_id, or YAML body. Addingconnector:slacketc. is no longer the convention.- Use only the vocabularies above. Propose new values in the PR — do not invent tags inline. New values should apply to at least two tasks in practice.
- Don't repeat the skill name as a feature tag. Don't tag a flow task with
rpa(bare) oruipath-rpaas a feature.
tags: [uipath-maestro-flow, e2e, mode:build, shape:multi-node, node:decision, connector, feature:http]make tags TAGS="smoke"→ every skill's entry-gate checks.make tags TAGS="smoke windows"→ Windows-only smoke tasks (the slicesmoke-rpa-skills.ymlruns onwindows-latest).make tags TAGS="integration connector"→ connector coverage across skills.make tags TAGS="e2e mode:build"→ end-to-end build tasks across skills.make tags TAGS="path-to-ga"→ GA-critical exhaustive, blocked, or historically fragile tasks.make tags TAGS="mode:diagnose"→ diagnosis-mode coverage across skills.- Evalboard:
where tag == "connector"→ pass-rate across all connector-using tasks. - Evalboard:
where tag == "shape:multi-node"→ composite-flow reliability.
tests/
├── README.md
├── Makefile
├── experiments/
│ ├── default.yaml # Dev / ad-hoc — tempdir, full lifecycle (no docker image required)
│ ├── nightly.yaml # Nightly cron — docker, full lifecycle, staging tenant
│ ├── smoke.yaml # PR-gate smoke (Linux, docker, faster budget)
│ ├── smoke-windows.yaml # Windows RPA smoke (tempdir)
│ ├── activation.yaml # Opt-in skill-activation benchmark (early-stop)
│ ├── skill-comparison-playbook.md # A/B comparison playbook (research)
│ └── skill-comparison-template.yaml # Template for compare-<a>-vs-<b>.yaml (research)
├── tasks/
│ └── <skill-name>/ # One folder per skill (must match skills/<name>/)
│ ├── _shared/ # Optional — grading helpers/checkers, per-skill pytest; reached only via $REFERENCE_DIR, never staged into the agent's sandbox
│ ├── _setup/ # Optional — pre_run/post_run tooling (seed/cleanup/scaffold scripts); staged into the agent's sandbox via sandbox.template_sources
│ ├── smoke/ # Tier: smoke
│ ├── single_node/ # Tests isolating a single node type (optional)
│ ├── multi_node/ # Composite-flow tests (optional)
│ ├── edit/ # Tests that modify an existing artifact (optional)
│ └── <other>/ # Skill-specific groupings (e.g. hitl/, connector_features/)
└── reports/ # Generated by /test-coverage command
├── <skill-name>.md # Per-skill coverage report
└── SUMMARY.md # Cross-skill roll-up (when analyzing all)
Groupings under a skill are advisory — pick the ones that map to how the skill is exercised. The flow skill uses smoke/, single_node/, multi_node/, edit/, hitl/, connector_features/. Keep dir names short and kebab-case; put only one task YAML per leaf dir (plus its sidecar check scripts).
Experiment files define shared agent defaults per test type. Tasks inherit these defaults and should only override what differs.
Run-time caps live under defaults.run_limits (see coder_eval RunLimits).
turn_timeout bounds one agent turn; task_timeout bounds the turns and
grading under a single watchdog. Raising task_timeout therefore does not give
a turn more room — the orchestrator logs A larger task_timeout cannot extend the agent's single iteration when a task tries. Raise turn_timeout for an
agent that runs out of time mid-build, and task_timeout when the criteria
need room after it: grading only gets what the turn did not spend. A task whose
task_timeout equals its turn_timeout therefore grades only if the agent
finishes early. That is fine where turns reliably come in short (smoke.yaml
runs equal 900s caps on purpose) and a trap for a task that uses its full turn,
because firing that watchdog reports the whole task TIMEOUT, losing even the
criteria that passed.
| Experiment | Driver | Used by | max_turns | task_timeout | turn_timeout |
|---|---|---|---|---|---|
default.yaml |
tempdir | Devs locally, ad-hoc runs | 200 | 1200s | 900s |
nightly.yaml |
docker | Nightly cron (daily.sh) |
200 | 1200s | 900s |
smoke.yaml |
docker | PR-gate smoke (Linux) | 40 | 900s | 900s |
smoke-windows.yaml |
tempdir | PR-gate smoke (Windows RPA only) | 40 | 900s | 900s |
activation.yaml |
tempdir | Skill activation classifier (benchmark) | 3 + early-stop | 360s | 120s |
same-ground-headtohead.yaml |
docker | Campaign-only local comparison arm | 200 | 1200s | 900s |
flow-v2-preview.yaml |
docker | Builder-SDK Maestro skills only (Flow, Case and BPMN all promoted) | 200 | 1200s | 900s |
use-case-tiers.yaml |
docker | Opt-in A/B for the uipath-agents use-case tasks: assisted vs unassisted (lower-tier helper text stripped) |
200 | 1200s | 900s |
same-ground-headtohead.yaml is not a clean-checkout CI experiment. The
campaign runner first builds the pinned skills-image:sg1, prepares isolated
SG_UIPATH_HOME and SG_EMPTY_SKILLS mount sources, and then launches this
single v1 variant; the counterpart arm is launched separately by the comparison
runner. The image build passes the package credential as
--secret id=npm_auth_token,env=NPM_AUTH_TOKEN; the Dockerfile build-arg path
exists only for the external nightly caller during migration. Regular nightly
and smoke jobs continue to use skills-image:latest.
use-case-tiers.yaml runs the uipath-agents use-case tasks
(tasks/uipath-agents/lowcode/use_cases/) in two variants. Those prompts carry
guidance meant only for lower-tier models between <!-- helper:start --> and
<!-- helper:end -->. Every other experiment sends the prompt unchanged, helper
included. The unassisted variant strips the block with a regex
prompt_mutation, so a higher-tier model gets only the context, the user's
request and the closing autonomy line. Helper text is never graded, so both
variants share the same success criteria. A task without the markers runs
identically in both variants. The "Run Coder Eval" workflow always passes
-e experiments/nightly.yaml, so run this experiment locally:
SKILLS_REPO_PATH=$(cd .. && pwd) .venv/bin/coder-eval run \
tasks/uipath-agents/lowcode/use_cases/*/*.yaml -e experiments/use-case-tiers.yamlflow-v2-preview.yaml runs the three builder-SDK skills
uipath-maestro-{flow,case,bpmn} as the ONLY skill catalog, so a run measures the
Flow v2 authoring path rather than a mix of both generations. Flow, Case and BPMN
have all been promoted: their trees live in skills/uipath-maestro-{flow,case,bpmn}
(the default catalog, so nightly.yaml runs them too), and
preview/skills/uipath-maestro-{flow,case,bpmn} are relative symlinks to them so this
arm keeps loading all three. The previous Case and BPMN generations live in
classic/skills/uipath-maestro-{case,bpmn} (shipped only in the Studio Web flavor).
They are in no plugin catalog, but the read-only repo mount leaves them readable
from inside eval containers, so a prompt must not name a v1 Case or BPMN reference
file. preview/ is a Claude Code
plugin root (preview/.claude-plugin/plugin.json + preview/skills/<name>/SKILL.md),
which is the one layout every harness loads: Claude Code requires it, the
Delegate SDK appends /skills to it, Codex and Antigravity accept it. Skills load
as uipath-preview:uipath-maestro-flow (the repo-root catalog is uipath:). Never
point plugins.path at a bare directory of skill folders: Claude Code loads
nothing from it and says so only as a per-task WARNING in task.log (every v2 run
08-20 → 09-03 ran that way). Narrowing plugins.path to preview/ drops the automatic
repo-root bind mount, so the root is remounted explicitly. The symlink needs a
checkout with symlinks enabled (the default on Linux and macOS; on Windows,
git config core.symlinks true before cloning); the arm runs in the Linux
container. Login state mounts at /.uipath,
identical to nightly.yaml. Confirm that mount resolves before a full run, or
every tenant call fails as a capability problem rather than a config one:
docker run --rm --env HOME="$HOME" -v ~/.uipath:/.uipath:rw \
--entrypoint bash skills-codex:latest -c 'uip login status'activation.yaml is a different shape from the tiered configs above — it runs the agent against single-prompt rows to measure whether the right skill fires (precision/recall/F1 per skill). Rows get a small turn budget (max_turns: 3); arming is per-criterion — each skill_triggered criterion carries a stop_early: {on_pass: stop} block that ends a row as soon as its outcome is live-decided. A positive row pass-stops the moment the expected skill engages; a negative row fail-stops on its first engagement. A wrong-skill engagement alone does NOT end a positive row — fail-stop is deferred while the row's positive criterion is still undecided, so a positive row that only misfires runs to the cap, as do rows with no engagement. Decided rows cost ~1 turn and a late-but-correct invocation is no longer truncated. Requires coder_eval >= 0.9.5: 0.9.5 removed stop_when and the run-level run_limits.stop_early: true master arm — both are hard errors now. It's an opt-in benchmark, not a smoke gate. See tasks/activation/README.md.
For A/B comparisons between two skill variants (e.g. main vs a feature branch, or two historical commits), see experiments/skill-comparison-playbook.md and the experiments/skill-comparison-template.yaml. The playbook covers worktree setup, SHA pinning for reproducibility, getting N>1, and interpreting divergent tasks. To automate the whole flow, use the /skill-compare <ref_a> <ref_b> [task_selector] [n_reps] slash command — each ref can be a branch name or a commit SHA, and task_selector accepts a skill name (uipath-maestro-flow), tag list (tags:smoke,init), or path globs (paths:tasks/uipath-maestro-flow/*.yaml).
The experiments set no agent.allowed_tools and no agent.permission_mode, so each
harness uses its own defaults. On Claude Code the agent is offered the CLI's own tool
set and runs bypassPermissions.
On Claude Code, agent.allowed_tools restricts and pre-approves: the agent is
offered only the listed built-in tools, and they run without a prompt. Set it on a
task only when the restriction is part of what the task measures, because a shorter
list changes what the agent does (with only 7 tools it calls Skill first more
often). A task that stages the plugin must keep Skill on its list, or coder-eval plan refuses it. Glob and Grep are not in the CLI's own set; name them on the
list if the task needs them. Use agent.disallowed_tools to withhold a tool from
the full set.
On codex, opencode, pi and antigravity, allowed_tools and permission_mode are
ignored with a warning, and --strict refuses them.
coder-eval 0.12.x differs: there allowed_tools only pre-approves, so a tool left
off the list is still offered, and its own default experiment adds
permission_mode: acceptEdits and the 7-tool list under every experiment.
Task files should not duplicate the full agent: block — the experiment provides the defaults. Only specify fields that differ from the experiment:
# Good — no agent block needed when everything matches the experiment defaults
task_id: skill-flow-init-validate
tags: [uipath-maestro-flow, smoke, mode:build]
sandbox:
driver: docker
python: {}
initial_prompt: |
...
# Good — only override what differs (max_turns: 14 instead of the default 20)
task_id: skill-flow-registry-discovery
tags: [uipath-maestro-flow, smoke, mode:build, feature:registry]
agent:
type: claude-code
max_turns: 14
sandbox:
driver: docker
python: {}
initial_prompt: |
...checker_context.api_route overrides which backend grades llm_judge criteria, decoupled from the agent's own route. Requires coder_eval ≥ 0.11.5, where the simulator resolves its own route independently of this override (Orchestrator.simulator_route). This repo's experiment defaults (default/smoke/smoke-windows/nightly/activation) route llm_judge through litellm → gpt-5.6-luna (the model behind CODEX_BASE_URL/CODEX_API_KEY) instead of the built-in judge (Bedrock/Anthropic):
checker_context:
api_route:
route: litellm
model: azure/gpt-5.6-luna
params:
api_version: "2024-05-01"
num_retries: 5
env_params:
api_base: CODEX_BASE_URL
api_key: CODEX_API_KEYnum_retries is mandatory, not tuning. coder_eval's litellm judge has no retry of its own — judge_litellm.py calls litellm.acompletion bare, while only judge_bedrock.py wraps a RetryConfig — so a transient provider InternalServerError or Timeout raises JudgeInfrastructureError (CE039) and ERRORs the whole task. Worse, checker.py's check_all_async accumulates results in a local list and lets that exception propagate, so every criterion already scored is discarded too: nightly 2026-08-31_04-15-47 lost 20 tasks this way, one of them a 21-criterion e2e whose 18 passing deterministic checks were thrown away by a single 500 on the judge at position 19. params is spliced into the acompletion call verbatim, so retry is configurable here; num_retries is a litellm-level kwarg and is never forwarded to the provider, so drop_params does not strip it. Remove once the litellm judge retries upstream.
route: litellm is llm_judge-only and safe as an experiment default even when simulation.enabled: true: the simulator resolves its own route independently of checker_context.api_route (coder_eval _resolve_routes/simulator_route), so it's unaffected by this override. It is not safe combined with an enabled agent_judge criterion — coder_eval still rejects that combination at setup, since agent_judge shares eval_route with llm_judge. This repo has no agent_judge criteria today; if one is added, override checker_context.api_route back to bedrock/direct on that specific task.
Running these tasks (locally, or a docker-driven experiment) requires:
- The
coder-eval[litellm]extra installed wherever the checker actually executes: on the host fordriver: tempdir(make installincludes it — seetests/Makefile), or baked into the agent image fordriver: docker(coder_eval's owncoder-eval-agentimage bakes--extra litellmin as of 0.11.4; a custom overlay image needs it too if built from an older pin). CODEX_BASE_URL/CODEX_API_KEYset in the environment the checker runs in — exported to the job fortempdir, or listed undersandbox.docker.env_passthrough_extrafordocker(seesmoke.yaml/nightly.yaml).
The two requirements above apply to tasks that actually resolve a checker
route — i.e. those with an llm_judge or agent_judge criterion. A task with
neither needs no credentials at all and runs against your own logged-in
claude CLI:
cd tests && make install # once
SKILLS_REPO_PATH=$(cd .. && pwd) .venv/bin/coder-eval run <task.yaml> \
-e experiments/default.yaml -vWith no judge criteria nothing resolves a checker route, so the agent and the simulated user both run as plain Claude Code subprocesses on your cached CLI auth — no key is injected.
Two traps:
- Do not set
ANTHROPIC_API_KEYorAWS_BEARER_TOKEN_BEDROCKto "help". Either one overrides the cached auth, and a stale or wrong value then fails at call time rather than falling back. - Pinning the same route explicitly does not work. A task-level
checker_context.api_route.route: directgoes through_resolve_backend_route, which does demandANTHROPIC_API_KEY. Absent and inherited succeeds where explicit and identical fails.
tests/scripts/tenant_catalog.py lists the cloud-tenant resources the tasks
depend on. It reads tests/tasks/ on every run, so the list is never
maintained by hand.
python3 tests/scripts/tenant_catalog.py --skill uipath-maestro-case --format mdThe script puts each task in one of three tiers:
| Tier | Meaning |
|---|---|
hard |
A pre_run command or a grader calls the tenant. |
soft |
The grader is local, but the prompt or a command_executed criterion sends the agent to the tenant. |
local |
Nothing reaches the tenant. |
Each dependency gets one disposition, with its reason and the tenant services it needs:
| Disposition | Meaning |
|---|---|
creatable |
Automation Suite can hold it, and a script can create it. |
substitutable |
A third-party SaaS account reached through Integration Service. The eval needs a local stand-in. |
unavailable |
No equivalent exists on Automation Suite. The tasks that depend on it cannot run there. |
The dependency kinds are:
- Integration Service connectors and their pinned connection ids
- solution resources from SDD task details, caseplan bindings and registry ledgers
- folders
- Data Fabric entities
- setup helpers that call the tenant
A dependency that no rule covers is unclassified. --strict exits 1 on a tenant task's
unclassified dependency, and
test_tenant_catalog.py fails on it for the case suite. When you add a tenant
dependency to a task, add its rule in the same change.
Known limits:
- A resource that is named only in prose, or only in a grader's literal assertion, is not caught.
- A runtime state that a task needs, such as a faulted instance, is not caught.
- Resources that must not exist on the tenant are not caught.
tests/tasks/uipath-platform/{orchestrator,resources}/ and
tests/tasks/uipath-solution/ follow the same shape as
orchestrator/job_run_logs_e2e.yaml: the agent receives a process key (and
derived folder) via env var, exercises the operational scenario, and a
check_*.py script verifies tenant state directly. The traces e2e tasks are
the exception — they inline their fixture key instead of reading an env var.
Task sandboxes don't have access to a full host repo checkout, so setup/
cleanup scripts are staged via sandbox.template_sources (agent-visible,
used for pre_run/post_run tooling) and grading scripts are reached via
$REFERENCE_DIR (mirrors reference.directory on disk, used by
run_command/llm_judge criteria, invisible to the agent):
sandbox:
template_sources:
- type: template_dir
path: ../_setup
mount_point: _setup
reference:
directory: .
pre_run:
- command: "E2E_PROCESS_KEY=$E2E_PROCESS_KEY python3 _setup/seed.py"
timeout: 60
post_run:
- command: "python3 _setup/cleanup.py"
timeout: 60
success_criteria:
- type: run_command
command: "python3 $REFERENCE_DIR/check_asset_scoping.py"coder-eval 1.x stages only reference.directory itself, so a grader that
imports a sibling (../_shared) needs directory: .. and
$REFERENCE_DIR/<subdir>/check_*.py commands. When a task changes its
reference.directory, update any grader docstring that states what is staged.
A single helper script (tests/tasks/uipath-platform/_setup/seed.py) writes
seed.json with a fresh uuid8 and — when E2E_PROCESS_KEY is set —
process_key + folder_path (resolved via uip or processes list, matched by Key — the get endpoint doesn't populate FolderPath). Tests
that don't need a process omit the env var assignment; the script just
writes uuid8.
Two pre-existing processes on the tenant, referenced by their keys via CI secrets:
| Secret | Purpose | Used by |
|---|---|---|
E2E_PROCESS_KEY |
Standard coded-agent process, runs to terminal quickly | job/trigger/webhook/resource tests |
E2E_LONG_PROCESS_KEY |
Same shape but sleeps ~30s | job_control_e2e (needs a stop/restart window) |
Both processes live in folders, so folder_path is derived from the
process key — no separate folder secret needed. Tests needing a second
folder create it themselves as part of the scenario.
Keep the stub processes inside the dedicated folder
(Shared/uipath-platform-e2e), not in Shared itself. The seeded
folder is derived from the process, so every resource the e2e tasks
create (assets, queues, buckets, triggers, jobs) lands in the same
folder — pointing the secrets at releases inside the dedicated folder
keeps the shared parent untouched and lets cleanup.py's folder sweep
run without risk to unrelated resources.
There is no post_run. The agent creates and deletes its own ephemeral
resources as part of the test scenario; the check script verifies the
final tenant state. Shared resources (the pre-seeded processes and their
folders) persist on the tenant across runs.
- Create
tests/tasks/<skill-name>/matching the skill folder name underskills/. - Add at minimum 1 smoke test and 1 e2e test (required for every new skill PR).
- Use minimal prompts — the goal is to test whether the skill guides the agent correctly, not to hand-hold it.
- Tag every task using the Tag Taxonomy: required
skill+tier+mode:*, plus optionalshape,node,resource,connector, andfeaturewhere applicable. - Stick to the closed-vocabulary values. Propose new tags in the PR — do not invent them inline.
- When a prompt or grader note cites a skill doc (
references/<file>.md, orskills/<other-skill>/references/<file>.md), the file must exist inskills/orpreview/skills/for that skill (the trees an eval catalog loads; a doc onlyclassic/skills/has does not count in a task YAML). A grader script or fixture beside the task may also cite aclassic/skills/doc whose rule it still enforces. A#sectionmust match a heading of the cited file.python3 scripts/check-task-doc-citations.pychecks every.yaml,.pyand.jsonundertests/tasks/in CI. When a doc is renamed or retired, point the task at the doc that now states the rule, or name the rule instead of the file.
skill-<domain>-<capability>
Examples: skill-flow-init-validate, skill-flow-registry-discovery
This is tasks/uipath-maestro-flow/smoke/init_validate.yaml — a smoke test that verifies the agent can create and validate a Flow project:
task_id: skill-flow-init-validate
description: >
Skill-guided evaluation: agent uses the uipath-maestro-flow skill to create
a new UiPath Flow project inside a solution and validate it. Tests whether
the skill teaches the correct solution-first workflow and CLI usage.
tags: [uipath-maestro-flow, smoke, mode:build]
sandbox:
driver: docker
python: {}
initial_prompt: |
Create a new UiPath Flow project called "WeatherAlert" and make sure it
validates successfully.
Use the `uipath-maestro-flow` skill workflow. A Flow project MUST be created
inside a solution:
1. Create the solution first.
2. Create the Flow project inside that solution.
3. Link the project to the solution.
The correct flow-file path is:
WeatherAlert/WeatherAlert/WeatherAlert.flow
The task is NOT complete until `uip maestro flow validate` has passed for
that exact file path.
Important:
- The `uip` CLI is already available in the environment.
- Do not run `uip maestro flow debug` — just validate locally.
success_criteria:
- type: command_executed
description: "Agent created a solution with uip solution new"
tool_name: "Bash"
command_pattern: '(uip|\$UIP)\s+solution\s+new'
min_count: 1
weight: 1.5
pass_threshold: 1.0
- type: command_executed
description: "Agent initialized a Flow project with uip maestro flow init"
tool_name: "Bash"
command_pattern: '(uip|\$UIP)\s+(maestro\s+)?flow\s+init'
min_count: 1
weight: 1.5
pass_threshold: 1.0
- type: command_executed
description: "Agent validated the .flow file"
tool_name: "Bash"
command_pattern: '(uip|\$UIP)\s+(maestro\s+)?flow\s+validate'
min_count: 1
weight: 1.5
pass_threshold: 1.0
- type: command_executed
description: "Agent used --output json on uip commands"
tool_name: "Bash"
command_pattern: '(uip|\$UIP)\s+.*--output\s+json'
min_count: 1
weight: 1.0
pass_threshold: 1.0
- type: command_executed
description: "Agent linked flow project to solution"
tool_name: "Bash"
command_pattern: '(uip|\$UIP)\s+solution\s+project\s+add'
min_count: 1
weight: 1.0
pass_threshold: 1.0
- type: file_exists
description: "Flow file was created inside the solution"
path: "WeatherAlert/WeatherAlert/WeatherAlert.flow"
weight: 1.5
pass_threshold: 1.0Key patterns to note:
- No
agent:block — inherits everything fromexperiments/default.yaml - No
run_limits:block — inherits turn / timeout caps from the experiment config - Minimal prompt — describes the goal ("create and validate"), not the steps
- Behavior-only criteria —
command_executedandfile_existsverify real operations, not agent self-reports - Weighted scoring — core commands (
weight: 1.5) matter more than supporting checks (weight: 1.0)
Each task defines one or more success criteria. The agent's score is the weighted sum of passing criteria.
Verify the agent ran a specific CLI command (matched by regex). From init_validate.yaml:
- type: command_executed
description: "Agent created a solution with uip solution new"
tool_name: "Bash"
command_pattern: 'uip\s+solution\s+new'
min_count: 1 # minimum times the command must appear
weight: 1.5 # scoring weight
pass_threshold: 1.0 # fraction of min_count required to passScope lookaheads and excludes to ONE command segment. The grader runs one pattern.search() per Bash tool call (re.DOTALL) and also matches a normalized haystack with newlines collapsed to spaces — so (?=[\s\S]*--flag) and exclude_pattern see every command batched into that call (codex chains a && b or stacks lines; a call-wide exclude then vetoes a correct command). Use the segment idiom S = (?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)* ("rest of THIS command", stops at newline, &&, ||, ;, |, or the next uip) and inline negatives instead of exclude_pattern:
# S expanded inline — YAML single quotes, no escaping needed
command_pattern: '(uip|\$UIP)\s+traces\s+feedback\s+list(?=(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--span-id)(?!(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--agent-id)'Verify a file was created in the sandbox. From init_validate.yaml:
- type: file_exists
description: "Flow file was created inside the solution"
path: "WeatherAlert/WeatherAlert/WeatherAlert.flow"
weight: 1.5
pass_threshold: 1.0Verify a file contains (or excludes) expected strings. From uipath-maestro-flow/hitl/smoke_01_hitl_node_placed.yaml:
- type: file_contains
description: "Flow contains the inline HITL node type"
path: "InvoiceApproval/InvoiceApproval/InvoiceApproval.flow"
includes:
- '"uipath.human-in-the-loop.quick-form"'
weight: 3.0
pass_threshold: 1.0excludes: is also supported — useful for asserting a file does not contain a deprecated flag or forbidden value. includes is a required field: an excludes-only criterion fails schema validation, so pair excludes with at least one positive includes entry.
Validate JSON file structure and values using JMESPath assertions. Supported operators: equals, gte, lte, gt, lt, contains.
Execute an arbitrary shell command and check the exit code. Use it for direct verification of state the agent created. From uipath-platform/data-fabric/integration_csv_import.yaml:
- type: run_command
description: "inventory.csv has at least 4 data rows (header + 4)"
command: "awk 'END { exit (NR >= 5 ? 0 : 1) }' inventory.csv"
timeout: 5
expected_exit_code: 0
weight: 2.0
pass_threshold: 1.0Or byte-equality for upload/download round-trips:
- type: run_command
description: "Downloaded file is byte-identical to the original"
command: "cmp -s original.txt downloaded.txt"
timeout: 5
expected_exit_code: 0Verify the agent invoked a Claude Code Skill tool. Useful for "did the agent recognize this scenario calls for skill X?" Both skill_name and expected_skill are required; the expected label is "yes" iff expected_skill == skill_name:
- type: skill_triggered
description: "Agent invoked the uipath-human-in-the-loop skill"
skill_name: "uipath-human-in-the-loop"
expected_skill: "uipath-human-in-the-loop"
weight: 3.0
pass_threshold: 1.0Un-fakeable — the criterion inspects turn_records.commands directly. The negative form (expected_skill: "") is the right primitive for smoke tests where the agent should NOT trigger a particular skill.
Counterpart to command_executed. Verifies the agent did NOT run a prohibited command. Use for refusal / negative-guard tests:
- type: command_not_executed
description: "Agent must not delete an entity"
tool_name: "Bash"
command_pattern: 'uip\s+df\s+entities\s+delete'
weight: 3.0
pass_threshold: 1.0Score is binary: 1.0 when matches ≤ max_count (default 0), else 0.0. Empty turn_records → trivially passes.
weight controls how much a criterion contributes to the overall score. Use higher weights for the core behavior being tested:
| Weight | When to use | Example from existing tests |
|---|---|---|
1.0 |
Supporting checks | --output json flag used, presence of an auxiliary file |
1.5 |
Core behavior | uip solution new executed, .flow file created |
2.0 |
Important artifact content | .flow file contains the expected node type or handle wiring |
3.0 |
Primary artifact validity | uip maestro flow validate passes on the generated flow file |
5.0–6.0 |
End-to-end execution | Check script runs flow debug and verifies output correctness |
pass_threshold is the fraction of the criterion that must pass. For json_check with multiple assertions, 0.75 means 75% of assertions must pass. For most criteria, use 1.0 (all-or-nothing).
After a run, results are written to tests/runs/<experiment-id>/:
runs/
└── <experiment-id>/
├── experiment.md # Overall summary
└── default/
├── variant.md # Variant-level summary
└── <task-id>/
└── task.json # Detailed per-task results
experiment.md— high-level pass/fail summary across all taskstask.json— per-criterion scores, agent transcript, and LLM reviewer output
-
Read the task result:
cat runs/*/default/skill-flow-init-validate/task.json | python -m json.tool
-
Check which criteria failed: Look at the
success_criteriaarray intask.json— each entry has apassedboolean andscore. -
Read the agent transcript: The
transcriptfield intask.jsonshows every agent turn, tool call, and tool result. -
Re-run a single task with verbose output:
SKILLS_REPO_PATH=$(cd .. && pwd) \ .venv/bin/coder-eval run tasks/uipath-maestro-flow/smoke/init_validate.yaml \ -e experiments/default.yaml -v -
Common failure causes:
- Agent used wrong CLI command or flags -> check the skill's SKILL.md for correctness
- Agent didn't activate the skill -> check skill description frontmatter and smoke test
- Agent ran out of turns -> increase
max_turnsor simplify the prompt - Sandbox issue -> check that
uipCLI is available in the test environment
Author tests in this order. Each step has a dedicated slash command.
Generates a coverage report that maps what a skill teaches against what its tests verify.
/test-coverage uipath-maestro-flow # single skill
/test-coverage all # cross-skill roll-upReports are written to tests/reports/<skill-name>.md and include:
- Component, workflow step, critical rule, and anti-pattern coverage (Direct/Indirect/None)
- Weighted overall score
- Priority-ranked coverage gaps with concrete test recommendations
Defined in .claude/commands/test-coverage.md.
Scaffolds a single task YAML from a free-form description of the scenario to cover. The command always infers the target skill from the description — do not pass a skill name.
/generate-task smoke test for folder listing via uip orchestrator
/generate-task e2e flow that uses HITL with an approval gate and write-back
/generate-task cover the new uip flow registry get subcommandOutput lands in tests/tasks/<skill-name>/ as one task YAML (and optional check script). Generated tasks are unverified scaffolds — verify that CLI commands, success criteria, and prompts match the skill's actual behavior.
Defined in .claude/commands/generate-task.md.
Lints the generated YAML against repo conventions (sandbox rules, tag taxonomy, criterion shape, CLI verb reachability) before it lands in a PR. Run this before step 4.
When step 3 surfaces CLI verb reachability findings, run /audit-verbs to see whether the same stale verb appears in other tasks or in skill docs. Writes tests/reports/cli-verb-audit.md and tests/reports/skill-verb-audit.md, both regenerated from the uip catalog at assets/uip-catalog-snapshot.json. Skip this step when /lint-task reports clean.
Defined in .claude/commands/audit-verbs.md.
Run the task end-to-end (see Running Tests) and add a passing-run claim to the PR description. The lint workflow flags missing claims as High severity.
Regression tests for the audit pipeline live under tests/scripts/. Run them when you modify any of scripts/build-uip-catalog.py, scripts/check-cli-verbs.py, or scripts/check-skill-verbs.py:
pytest tests/scripts/- coder_eval repository — framework docs, task definition guide, CLI reference
- CONTRIBUTING.md — skill contribution rules and quality checklist