Repository navigation
Conversation
With `simulation.enabled: true` the simulated user keeps talking after the
agent delivers its diagnosis, so the agent's final turn is usually a one-line
acknowledgement ("Will do."). An `llm_judge` criterion that sets
`include_agent_output: true` without `include_dialog: true` receives only that
final turn, sees no diagnosis, and scores a correct investigation 0.00.
Six troubleshoot tasks failed exactly this way in the 2026-07-28 nightly, each
with a correct root cause in turn 1.
Adds `scripts/check-judge-dialog.py` plus a `Judge Dialog Gate` workflow that
flags the combination on task YAMLs a PR adds or modifies, so tasks predating
the rule are not retroactively blocked.
|
Claude finished @MarinRzv's task in 4m 25s —— View job PR Review: ci: block troubleshoot tasks whose llm_judge grades only the last turn
SummaryAdds a CI gate ( Change-by-Change Review1.
|
| Area | Status | Notes |
|---|---|---|
| Frontmatter | N/A | No skill files changed |
| E2E Tests | N/A | CI tooling only |
| Skill Body | N/A | No skill files changed |
| References & Assets | N/A | No skill files changed |
| Repo Hygiene | OK | No secrets, no cross-skill deps, well-scoped |
Issues for Manual Review
- Confirm the companion PR fix(tests): grade simulated troubleshoot scenarios on the full dialog + CI gate #2293 updates the canonical
llm_judgetemplate intests/tasks/uipath-troubleshoot/CLAUDE.mdto includeinclude_dialog: true. Without that, new tasks generated from the template will immediately fail this gate (which is the intended behavior — but the template should be updated to match).
Conclusion
Clean, focused PR that adds a well-scoped CI guard for a real problem (six nightly failures). One actionable improvement: pass steps.changed.outputs.files through env: instead of direct ${{ }} interpolation in the run: block (line 64) to harden against shell injection via crafted file paths. The Python script is correct and follows established repo patterns.
`${{ }}` expressions are substituted into the run script before the shell
parses it, so a PR-controlled path containing shell metacharacters could be
executed. The file list comes from a `git diff` over PR-controlled paths, so
route it through `env:` - env-var expansion does not trigger command
substitution. Matches the pattern already used in activation-gate.yml.
`$FILES` stays unquoted so word-splitting yields one argument per file.
Added: troubleshoot scenario contract gate (moved here from #2311)Pushed WhyThe troubleshoot suite is ~300 near-identical faithful-replay scenarios grown from one generator template, and nothing verified they stayed in that shape. The drift that motivated this is expensive rather than loud: in the 2026-07-28 nightly, 39 scenarios let the simulated user approve fix application, sending runs into an ungraded edit + validate/build/pack loop worth ~126 minutes of wall clock. Every one of those tasks still passed, so no signal existed. #2299 fixed the 296 files; this adds the gate that keeps them fixed. What
Rules were derived from what is already documented as MUST in the suite CLAUDE.md and currently holds for 100% of scenarios, so the gate is green from the first commit. Constraints that are only mostly-followed ( VerificationNegative tests — injected drift, all correctly failed with an annotated line + fix hint:
Possible consolidation
Pre-existing issues found, deliberately NOT changed
|
Consolidated the judge-dialog rule + fixed three pre-existing bugsThree commits pushed ( 1.
|
| Injected drift | Caught |
|---|---|
include_dialog removed from a scenario |
yes — annotated on the judge criterion line |
include_dialog removed from the generator template only |
yes |
…atching criterion
…string booleans, and a seal-less generator
Problem
With
simulation.enabled: truethe simulated user keeps talking after the agent has delivered its diagnosis ("thanks, I'll get an admin to grant that"). The agent's final turn is then a one-line acknowledgement — "Will do.", "You're welcome." — while the real diagnosis sits two turns back.An
llm_judgecriterion that setsinclude_agent_output: truewithoutinclude_dialog: truereceives only that final turn. The judge sees no diagnosis and scores0.00, so a fully correct investigation fails the task.Six troubleshoot tasks failed exactly this way in the 2026-07-28 nightly, each with a correct root cause in turn 1.
Change
scripts/check-judge-dialog.pyllm_judgecriteria that grade the last turn only. Accepts task YAML paths or a directory..github/workflows/judge-dialog-gate.ymlpull_request(andworkflow_dispatch), scoped to the troubleshoot task YAMLs the PR adds or modifies.A task is reported when all of these hold:
simulation.enabledistrue— multi-turn, so a last-turn-only grade can miss the diagnosis. Single-turn tasks are fine; their last turn is the answer.llm_judgecriterion withinclude_agent_output: true.include_dialog: true.Failures are emitted as
::error file=…,line=…annotations, so they render inline on the PR diff at the offending- type: llm_judgeline.Scoping
The gate checks only the task YAMLs a PR adds or modifies (
--diff-filter=AM), so the ~194 existing tasks that predate this rule are not retroactively blocked.workflow_dispatchscans the whole skill directory instead, for a full sweep on demand.There is no
paths:filter on the workflow, so it can be wired as a required check — it exits 0 when a PR touches no troubleshoot task YAMLs. To cover other skills later, widen the pathspec in the "Collect changed task YAMLs" step.Verification
Run locally (
pip install pyyaml):All three paths confirmed, plus the singular-grammar branch (
1 llm_judge criterion grades).Relationship to #2293
Split out of #2293, which now carries only the task-YAML fixes (
include_dialog: trueon 194 tasks + theCLAUDE.mdjudge template). The two can merge in either order — because the gate is scoped to changed files, merging this first does not retroactively fail anything.