Skip to content

ci: block troubleshoot tasks whose llm_judge grades only the last turn - #2301

Closed
MarinRzv wants to merge 25 commits into
mainfrom
ci/judge-dialog-gate
Closed

MarinRzv wants to merge 25 commits into
mainfrom
ci/judge-dialog-gate

Conversation

@MarinRzv

Copy link
Copy Markdown
Collaborator

Problem

With simulation.enabled: true the simulated user keeps talking after the agent has delivered its diagnosis ("thanks, I'll get an admin to grant that"). The agent's final turn is then a one-line acknowledgement — "Will do.", "You're welcome." — while the real diagnosis sits two turns back.

An llm_judge criterion that sets include_agent_output: true without include_dialog: true receives only that final turn. The judge sees no diagnosis and scores 0.00, so a fully correct investigation fails the task.

Six troubleshoot tasks failed exactly this way in the 2026-07-28 nightly, each with a correct root cause in turn 1.

Change

File Purpose
scripts/check-judge-dialog.py Flags llm_judge criteria that grade the last turn only. Accepts task YAML paths or a directory.
.github/workflows/judge-dialog-gate.yml Runs the check on pull_request (and workflow_dispatch), scoped to the troubleshoot task YAMLs the PR adds or modifies.

A task is reported when all of these hold:

  1. simulation.enabled is true — multi-turn, so a last-turn-only grade can miss the diagnosis. Single-turn tasks are fine; their last turn is the answer.
  2. It has an llm_judge criterion with include_agent_output: true.
  3. That criterion does not set include_dialog: true.

Failures are emitted as ::error file=…,line=… annotations, so they render inline on the PR diff at the offending - type: llm_judge line.

Scoping

The gate checks only the task YAMLs a PR adds or modifies (--diff-filter=AM), so the ~194 existing tasks that predate this rule are not retroactively blocked. workflow_dispatch scans the whole skill directory instead, for a full sweep on demand.

There is no paths: filter on the workflow, so it can be wired as a required check — it exits 0 when a PR touches no troubleshoot task YAMLs. To cover other skills later, widen the pathspec in the "Collect changed task YAMLs" step.

Verification

Run locally (pip install pyyaml):

# current main — every task still grades the last turn only
python3 scripts/check-judge-dialog.py tests/tasks/uipath-troubleshoot
# FAIL - 194 llm_judge criteria grade the last turn only:   (exit 1)

# a task fixed by #2293
python3 scripts/check-judge-dialog.py <fixed-task.yaml>
# OK - every simulated task's llm_judge grades the full dialog.   (exit 0)

# no paths (PR touches no tasks)
python3 scripts/check-judge-dialog.py
# OK - ...   (exit 0)

All three paths confirmed, plus the singular-grammar branch (1 llm_judge criterion grades).

Relationship to #2293

Split out of #2293, which now carries only the task-YAML fixes (include_dialog: true on 194 tasks + the CLAUDE.md judge template). The two can merge in either order — because the gate is scoped to changed files, merging this first does not retroactively fail anything.

With `simulation.enabled: true` the simulated user keeps talking after the
agent delivers its diagnosis, so the agent's final turn is usually a one-line
acknowledgement ("Will do."). An `llm_judge` criterion that sets
`include_agent_output: true` without `include_dialog: true` receives only that
final turn, sees no diagnosis, and scores a correct investigation 0.00.

Six troubleshoot tasks failed exactly this way in the 2026-07-28 nightly, each
with a correct root cause in turn 1.

Adds `scripts/check-judge-dialog.py` plus a `Judge Dialog Gate` workflow that
flags the combination on task YAMLs a PR adds or modifies, so tasks predating
the rule are not retroactively blocked.
@github-actions

github-actions Bot commented Jul 28, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @MarinRzv's task in 4m 25s —— View job


PR Review: ci: block troubleshoot tasks whose llm_judge grades only the last turn

  • Read review guidelines and project conventions
  • Read full PR diff
  • Read changed files in full
  • Check for cross-skill dependencies, secrets, link validity
  • Post review feedback

Summary

Adds a CI gate (judge-dialog-gate.yml + scripts/check-judge-dialog.py) that blocks troubleshoot task YAMLs where an llm_judge criterion grades only the agent's last turn (include_agent_output: true without include_dialog: true) under simulation.enabled: true. Well-scoped: only checks files a PR adds or modifies, so the ~194 existing tasks are not retroactively blocked.

Change-by-Change Review

1. .github/workflows/judge-dialog-gate.yml

Severity: Medium (one injection-hardening improvement)

Clean workflow that follows the established pattern from task-driver-gate.yml. Good use of concurrency, permissions: contents: read, timeout-minutes: 5, and fetch-depth: 0 for the diff. The workflow_dispatch fallback to scan the full directory is a nice touch.

Issue — ${{ }} interpolation in run: block (line 64):

run: python3 scripts/check-judge-dialog.py ${{ steps.changed.outputs.files }}

The steps.changed.outputs.files value originates from git diff --name-only on PR-controlled file paths. Since ${{ }} expressions are string-interpolated into the shell script before the shell parses it, a crafted directory name containing shell metacharacters (e.g. $(cmd), backticks, ;) could be interpreted as code. The risk is low — paths are constrained by the **/task.yaml glob and pull_request runs use a read-only token — but the fix is simple and follows GitHub's own security hardening guide:

- name: Check judge dialog inputs
  env:
    FILES: ${{ steps.changed.outputs.files }}
  run: python3 scripts/check-judge-dialog.py $FILES

Note: $FILES is intentionally unquoted for word-splitting into multiple arguments — this is correct. The key difference is that env-var expansion doesn't trigger command substitution, whereas ${{ }} interpolation does.

Fix this →

Everything else in the workflow is correct:

  • The BASE_REF is already properly passed through env: (line 48-49) — good.
  • --diff-filter=AM correctly limits to added/modified files.
  • The -- 'tests/tasks/uipath-troubleshoot/**/task.yaml' pathspec is well-scoped.

2. scripts/check-judge-dialog.py

Severity: OK

Well-structured script, clear docstring, correct detection logic. Follows the same conventions as check-task-driver.py. Specific observations:

  • Detection logic (lines 89-109): Correctly requires all three conditions — simulation.enabled is True, include_agent_output is True, include_dialog is not True. The is True identity checks are correct for PyYAML-parsed booleans.
  • Line-number correlation (lines 80-86 + 124-126): The regex-based _judge_line_numbers approach to pin GitHub annotations to the right line is clever. The fallback to line=0 when the regex/YAML counts diverge (line 126) is good defensive coding.
  • Error output (lines 139-143): ::error file=…,line=… annotations will render inline on the PR diff — exactly the right UX for this gate.
  • Graceful degradation (lines 118-121): Malformed YAML is silently skipped rather than crashing the gate. The comment explains why ("another gate's problem").
  • Singular/plural grammar (line 133): Nice detail.
  • Empty input (line 129): No arguments → empty file list → OK exit. Correct for the PR-scoped use case.

One minor note: the file is read twice — once with errors="ignore" for line detection (line 84) and once with encoding="utf-8" for YAML parsing (line 117). In practice this is fine for UTF-8 YAML files, but if you wanted to be extra defensive you could read once and reuse. Not worth changing.

What's Missing

Area Ratings

Area Status Notes
Frontmatter N/A No skill files changed
E2E Tests N/A CI tooling only
Skill Body N/A No skill files changed
References & Assets N/A No skill files changed
Repo Hygiene OK No secrets, no cross-skill deps, well-scoped

Issues for Manual Review

Conclusion

Clean, focused PR that adds a well-scoped CI guard for a real problem (six nightly failures). One actionable improvement: pass steps.changed.outputs.files through env: instead of direct ${{ }} interpolation in the run: block (line 64) to harden against shell injection via crafted file paths. The Python script is correct and follows established repo patterns.


`${{ }}` expressions are substituted into the run script before the shell
parses it, so a PR-controlled path containing shell metacharacters could be
executed. The file list comes from a `git diff` over PR-controlled paths, so
route it through `env:` - env-var expansion does not trigger command
substitution. Matches the pattern already used in activation-gate.yml.

`$FILES` stays unquoted so word-splitting yields one argument per file.
@dmorosanu

Copy link
Copy Markdown
Contributor

Added: troubleshoot scenario contract gate (moved here from #2311)

Pushed 33af29dc4 onto this branch, plus a merge of latest main (1bfaebcb0) which this branch needed — it was based on a commit predating #2299, so the new gate could not pass without it. Both gates are green on the merged tree:

$ python3 scripts/check-troubleshoot-tasks.py
OK - 297 scenario(s) + the generator template satisfy the troubleshoot scenario contract.

$ python3 scripts/check-judge-dialog.py
OK - every simulated task's llm_judge grades the full dialog.

Why

The troubleshoot suite is ~300 near-identical faithful-replay scenarios grown from one generator template, and nothing verified they stayed in that shape. The drift that motivated this is expensive rather than loud: in the 2026-07-28 nightly, 39 scenarios let the simulated user approve fix application, sending runs into an ungraded edit + validate/build/pack loop worth ~126 minutes of wall clock. Every one of those tasks still passed, so no signal existed. #2299 fixed the 296 files; this adds the gate that keeps them fixed.

What

  • tests/tasks/uipath-troubleshoot/_shared/scenario-contract.yaml — the machine-readable master contract: required simulation constraints (currently the decline-fix constraint from test(uipath-troubleshoot): simulated user declines fix application in scenario sims #2299), required + forbidden success-criteria types, the canonical llm_judge shape (weight/pass_threshold/include_reference/include_agent_output, no files:), structural keys (reference.file, sandbox.mock_path_dirs, required tags, run_limits), and the exemption list (smoke-manifest-commands, the one non-scenario task).
  • scripts/check-troubleshoot-tasks.py — enforces the contract against every scenario and against the generator template, which it renders as a synthetic scenario. A rule therefore cannot apply to the existing scenarios but not to the generator that produces new ones.
  • .github/workflows/troubleshoot-task-contract.yml — PR gate, no paths: filter so it can be wired as a required check (sub-second full scan, definitive pass/fail). Same shape as task-driver-gate.yml.
  • Suite CLAUDE.md points at the contract and the local command.

Rules were derived from what is already documented as MUST in the suite CLAUDE.md and currently holds for 100% of scenarios, so the gate is green from the first commit. Constraints that are only mostly-followed (Never invent..., Never instruct the agent to stop..., 3-5 scenarios each) were deliberately left out rather than enforced retroactively.

Verification

Negative tests — injected drift, all correctly failed with an annotated line + fix hint:

Injected drift Caught
decline-fix constraint removed from a scenario yes
decline-fix constraint removed from the generator template only yes
file_exists criterion added yes
llm_judge.include_reference: false yes
llm_judge.weight: 2.0 yes
mock_path_dirs: [mocks] yes
uipath-troubleshoot tag removed yes

Possible consolidation

check-judge-dialog.py's rule (llm_judge must grade the full dialog, not only the last turn) is the same kind of suite-wide invariant the contract file now expresses declaratively. Folding it in as a contract rule would leave one gate instead of two — happy to do that here or as a follow-up, your call since this is your PR.

Pre-existing issues found, deliberately NOT changed

  1. The generator template pins sandbox.driver: tempdir, which scripts/check-task-driver.py blocks for task YAMLs. That gate only scans tests/tasks/**.yaml, so the template inside a .py file escapes it — newly generated scenarios would carry a forbidden driver until someone notices.
  2. scripts/check-task-driver.py calls read_text() with no encoding, so it crashes with UnicodeDecodeError on a Windows shell (CI is unaffected; Linux defaults to UTF-8).

@dmorosanu

Copy link
Copy Markdown
Contributor

Consolidated the judge-dialog rule + fixed three pre-existing bugs

Three commits pushed (fd778cc9a, abc9e1f9c, 50c5232a6).

1. ci(troubleshoot): fold the judge-dialog rule into the scenario contract

include_dialog: true is now a required llm_judge flag in _shared/scenario-contract.yaml, enforced by check-troubleshoot-tasks.py. scripts/check-judge-dialog.py and .github/workflows/judge-dialog-gate.yml are removed — 227 net lines deleted, one gate instead of two.

Why this is a clean fold rather than a lossy one:

  • All 297/297 simulated scenarios already set include_dialog: true, so the rule lands green with no migration.
  • The standalone check's guard clauses (simulation.enabled true, include_agent_output true) are unconditionally true inside this contract — it already requires both on every scenario — so the rule collapses to one entry in require_true with no conditional machinery.
  • Enforcement strengthens from changed-files-only to a full 297-scenario scan, which catches drift a path filter misses (a bad merge, a revert, a bulk edit).
  • The rationale from the original docstring (six tasks scoring 0.00 in the 2026-07-28 nightly with correct root causes in turn 1) is preserved verbatim as the rule's why in the contract file, and the suite CLAUDE.md pointer now names the consolidated gate.

This fold also caught a real bug: the generator template never emitted include_dialog, so every newly generated scenario would have shipped with exactly the last-turn-only grading defect this rule exists to prevent. Added to the template.

Also improved the checker while validating the fold: a missing required flag has no line of its own, so annotations were landing on line 0 — they now anchor on the - type: llm_judge line, and each flag carries its own fix hint instead of a shared generic one.

2. fix(troubleshoot): drop the forbidden sandbox.driver pin from the scenario template

The generator template pinned sandbox.driver: tempdir, which scripts/check-task-driver.py blocks for task YAMLs — but that gate only scans tests/tasks/**.yaml, so a template inside a .py file escaped it. Every newly generated scenario would have carried a forbidden driver that opts out of the docker image and runs on the toolless host. Rendered template now yields sandbox: [python, template_sources, mock_path_dirs], no driver.

3. fix(scripts): read task YAMLs as UTF-8 in the driver check

check-task-driver.py called read_text() with no encoding, so it died with UnicodeDecodeError on any non-ASCII task YAML under a Windows shell (CI was unaffected — Linux defaults to UTF-8), making the check unrunnable locally on Windows. Now reads UTF-8 explicitly and tolerates a decode error the same way it already tolerated malformed YAML.

Verification

$ python3 scripts/check-troubleshoot-tasks.py
OK - 297 scenario(s) + the generator template satisfy the troubleshoot scenario contract.

$ python3 scripts/check-task-driver.py tests/tasks     # previously crashed on Windows
OK - no task pins `sandbox.driver: tempdir`.

Negative tests for the folded rule, both correctly failing with an annotated line and the specific rationale:

Injected drift Caught
include_dialog removed from a scenario yes — annotated on the judge criterion line
include_dialog removed from the generator template only yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants