Skip to content

ci(troubleshoot): gate scenario task YAMLs against the suite contract - #2311

Closed
dmorosanu wants to merge 1 commit into
mainfrom
chore/troubleshoot-task-contract-gate
Closed

dmorosanu wants to merge 1 commit into
mainfrom
chore/troubleshoot-task-contract-gate

Conversation

@dmorosanu

Copy link
Copy Markdown
Contributor

Why

The troubleshoot suite is ~300 near-identical faithful-replay scenarios grown from one generator template, and nothing verified they stayed in that shape. The drift that motivated this is expensive rather than loud: in the 2026-07-28 nightly, 39 scenarios let the simulated user approve fix application, sending runs into an ungraded edit + validate/build/pack loop worth ~126 minutes of wall clock. Every one of those tasks still passed, so no signal existed. #2299 fixed the 296 files; this adds the gate that keeps them fixed.

What

  • tests/tasks/uipath-troubleshoot/_shared/scenario-contract.yaml — the machine-readable master contract: required simulation constraints (currently the decline-fix constraint from test(uipath-troubleshoot): simulated user declines fix application in scenario sims #2299), required + forbidden success-criteria types, the canonical llm_judge shape (weight/pass_threshold/include_reference/include_agent_output, no files:), structural keys (reference.file, sandbox.mock_path_dirs, required tags, run_limits), and the exemption list (smoke-manifest-commands, the one non-scenario task).
  • scripts/check-troubleshoot-tasks.py — enforces the contract against every scenario and against the generator template, which it renders as a synthetic scenario. A rule therefore cannot apply to the existing scenarios but not to the generator that produces new ones.
  • .github/workflows/troubleshoot-task-contract.yml — PR gate, no paths: filter so it can be wired as a required check (sub-second full scan, definitive pass/fail). Same shape as task-driver-gate.yml.
  • Suite CLAUDE.md points at the contract and the local command.

Rules were derived from what is already documented as MUST in the suite CLAUDE.md and currently holds for 100% of scenarios, so the gate is green on main from the first commit. Constraints that are only mostly-followed (Never invent..., Never instruct the agent to stop..., 3-5 scenarios each) were deliberately left out rather than enforced retroactively.

Verification

$ python3 scripts/check-troubleshoot-tasks.py
OK - 297 scenario(s) + the generator template satisfy the troubleshoot scenario contract.

Negative tests (all correctly fail with an annotated line + fix hint):

Injected drift Caught
decline-fix constraint removed from a scenario yes
decline-fix constraint removed from the generator template only yes
file_exists criterion added yes
llm_judge.include_reference: false yes
llm_judge.weight: 2.0 yes
mock_path_dirs: [mocks] yes
uipath-troubleshoot tag removed yes

Notes for the reviewer

Two pre-existing issues found while building this, deliberately NOT changed here:

  1. The generator template pins sandbox.driver: tempdir, which scripts/check-task-driver.py blocks for task YAMLs. The existing gate only scans tests/tasks/**.yaml, so the template escapes it — newly generated scenarios would carry a forbidden driver until someone notices.
  2. scripts/check-task-driver.py calls read_text() with no encoding, so it crashes with UnicodeDecodeError on a Windows shell (CI is unaffected because Linux defaults to UTF-8).

@github-actions

github-actions Bot commented Jul 29, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @dmorosanu's task in 4m 23s —— View job


PR Review: ci(troubleshoot): gate scenario task YAMLs against the suite contract

  • Read review guidelines and project conventions
  • Read full diff and changed files
  • Analyze script correctness and edge cases
  • Analyze contract YAML and workflow
  • Post review findings

Summary

Adds a machine-readable contract (scenario-contract.yaml) and enforcement script (check-troubleshoot-tasks.py) to prevent troubleshoot scenario task YAMLs from drifting out of their required shape. Motivated by a real nightly regression where 39 scenarios silently approved fix application, costing ~126 minutes of ungraded wall clock. The workflow, script, and contract are well-structured and follow the existing task-driver-gate.yml pattern.

Change-by-Change Review

1. scripts/check-troubleshoot-tasks.py

Severity: Medium

Well-written enforcement script. Clean separation of concerns, good error messages with fix hints, GitHub Actions annotations. A few observations:

  • include_dialog not enforced (Medium): The contract's require_true list contains include_reference and include_agent_output but NOT include_dialog. The suite CLAUDE.md documents include_dialog: true as mandatory across all scenarios ("What the judge sees — all three flags MUST be true") and explains that six scenarios failed in the 2026-07-28 nightly specifically because of missing include_dialog. All 297 existing scenarios have it. The generator template at generate_scenario.py:142-143 also omits it — so newly generated scenarios will silently lack it. This seems like an oversight rather than a deliberate exclusion, especially given that a separate gate (scripts/check-judge-dialog.py / judge-dialog-gate.yml) referenced in the CLAUDE.md does not actually exist in the repo. Add include_dialog to contract →

  • Latent KeyError in _render_generator_template (Low): At line 190-191, gt = contract.get("generator_template") or {} followed by gt["path"] will raise KeyError if the generator_template section is removed from the contract. The or {} fallback suggests the intent was graceful degradation, but gt["path"] does not use .get(). Either use gt.get("path") with an early return, or drop the or {} fallback to make the crash explicit. Currently academic since the section exists, but inconsistent with the defensive style elsewhere.

  • exec() usage (Low, acknowledged): Line 200 uses exec() to evaluate the extracted template string. The # noqa: S102 acknowledges it. The regex extraction constrains what's executed to the triple-quoted string content, and the source is a repo file trusted in CI — acceptable here, but worth noting that the regex (.*?) with re.S is non-greedy so it captures the first triple-quoted block starting at the symbol, which is correct.

  • Cross-platform _rel improvement (OK): Line 61 adds .replace("\\", "/") which the older check-task-driver.py lacks — a good practice for consistent annotation paths on Windows.

2. tests/tasks/uipath-troubleshoot/_shared/scenario-contract.yaml

Severity: Medium

Clean, well-documented contract. The decline-fix-application constraint, forbidden/required criteria types, llm_judge shape, and structural keys all match what the CLAUDE.md documents and what holds for 100% of existing scenarios.

  • Missing include_dialog (Medium): As noted above — require_true should include include_dialog to match the documented MUST and the 297/297 scenario reality.

  • mock_path_dirs value is ["m"] (string list) in YAML vs - m (OK): The contract specifies mock_path_dirs: ["m"] at line 62 but the checker at line 179 compares with != against the contract value. Since PyYAML parses ["m"] as the Python list ['m'] and YAML - m also becomes ['m'], equality comparison works correctly across both styles.

3. .github/workflows/troubleshoot-task-contract.yml

Severity: OK

Follows the exact shape of task-driver-gate.yml: no paths: filter (suitable as a required check), Python 3.13, pip installs PyYAML, concurrency group with cancel-in-progress. timeout-minutes: 5 is generous for a sub-second scan but consistent with the companion workflow. Nothing to flag.

4. tests/tasks/uipath-troubleshoot/CLAUDE.md

Severity: OK

Six-line addition pointing to the contract and the local command. Well-placed at the top of the "Task YAML requirements" section. The link [_shared/scenario-contract.yaml](./_shared/scenario-contract.yaml) resolves correctly (verified the file exists).

What's Missing

  • include_dialog in contract and generator template: All 297 existing scenarios have include_dialog: true. The CLAUDE.md documents it as mandatory. The contract does not enforce it. The generator template does not produce it. The separate gate referenced in the CLAUDE.md (scripts/check-judge-dialog.py / judge-dialog-gate.yml) does not exist. This means the contract will pass for newly generated scenarios that lack include_dialog, and nothing else catches it.

Area Ratings

Area Status Notes
Frontmatter N/A No skill SKILL.md changed
E2E Tests N/A No task YAMLs added/modified
Script Logic OK Clean, follows existing patterns, good error messages
Contract Issue Missing include_dialog enforcement
Repo Hygiene OK No secrets, scoped changes, workflow follows existing pattern

Issues for Manual Review

  • The PR description notes that the generator template pins sandbox.driver: tempdir, which scripts/check-task-driver.py blocks for task YAMLs but not for the template itself. Confirm this is tracked separately (a future scenario generated from the template would carry a forbidden driver).
  • The CLAUDE.md references scripts/check-judge-dialog.py and judge-dialog-gate.yml as existing CI gates, but neither file exists in the repo. If these are planned, the include_dialog gap in this contract is less urgent; if they were removed or never landed, this contract is the natural place to add the enforcement.

Conclusion

The contract and enforcement script are well-motivated, well-structured, and follow established patterns. The include_dialog gap is the one actionable finding: it's documented as mandatory, present in all 297 scenarios, and was the cause of real nightly failures — adding it to require_true (and to the generator template) would close the loop. Everything else is clean.


@github-actions

github-actions Bot commented Jul 29, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @dmorosanu's task in 36s —— View job


Coder-eval task lint (advisory)

  • Read lint rubric
  • Identify changed task YAMLs
  • Compose result

No coder-eval task YAMLs changed in this PR. The only YAML touched is tests/tasks/uipath-troubleshoot/_shared/scenario-contract.yaml, which lives under _shared/ and is excluded from linting per the rubric.

This PR adds a contract definition and enforcement script — no task files were added or modified, so the lint rubric does not apply.

@dmorosanu

Copy link
Copy Markdown
Contributor Author

dup #2301

@dmorosanu

Copy link
Copy Markdown
Contributor Author

Superseded by #2301 — the commit was moved onto ci/judge-dialog-gate as 33af29dc4 (along with a merge of latest main that branch needed). Full description posted there: #2301 (comment)

@dmorosanu dmorosanu closed this Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant