Repository navigation
Conversation
|
Claude finished @dmorosanu's task in 4m 23s —— View job PR Review: ci(troubleshoot): gate scenario task YAMLs against the suite contract
SummaryAdds a machine-readable contract ( Change-by-Change Review1.
|
| Area | Status | Notes |
|---|---|---|
| Frontmatter | N/A | No skill SKILL.md changed |
| E2E Tests | N/A | No task YAMLs added/modified |
| Script Logic | OK | Clean, follows existing patterns, good error messages |
| Contract | Issue | Missing include_dialog enforcement |
| Repo Hygiene | OK | No secrets, scoped changes, workflow follows existing pattern |
Issues for Manual Review
- The PR description notes that the generator template pins
sandbox.driver: tempdir, whichscripts/check-task-driver.pyblocks for task YAMLs but not for the template itself. Confirm this is tracked separately (a future scenario generated from the template would carry a forbidden driver). - The CLAUDE.md references
scripts/check-judge-dialog.pyandjudge-dialog-gate.ymlas existing CI gates, but neither file exists in the repo. If these are planned, theinclude_dialoggap in this contract is less urgent; if they were removed or never landed, this contract is the natural place to add the enforcement.
Conclusion
The contract and enforcement script are well-motivated, well-structured, and follow established patterns. The include_dialog gap is the one actionable finding: it's documented as mandatory, present in all 297 scenarios, and was the cause of real nightly failures — adding it to require_true (and to the generator template) would close the loop. Everything else is clean.
|
Claude finished @dmorosanu's task in 36s —— View job Coder-eval task lint (advisory)
No coder-eval task YAMLs changed in this PR. The only YAML touched is This PR adds a contract definition and enforcement script — no task files were added or modified, so the lint rubric does not apply. |
|
dup #2301 |
|
Superseded by #2301 — the commit was moved onto |
Why
The troubleshoot suite is ~300 near-identical faithful-replay scenarios grown from one generator template, and nothing verified they stayed in that shape. The drift that motivated this is expensive rather than loud: in the 2026-07-28 nightly, 39 scenarios let the simulated user approve fix application, sending runs into an ungraded edit + validate/build/pack loop worth ~126 minutes of wall clock. Every one of those tasks still passed, so no signal existed. #2299 fixed the 296 files; this adds the gate that keeps them fixed.
What
tests/tasks/uipath-troubleshoot/_shared/scenario-contract.yaml— the machine-readable master contract: required simulation constraints (currently the decline-fix constraint from test(uipath-troubleshoot): simulated user declines fix application in scenario sims #2299), required + forbidden success-criteria types, the canonicalllm_judgeshape (weight/pass_threshold/include_reference/include_agent_output, nofiles:), structural keys (reference.file,sandbox.mock_path_dirs, required tags,run_limits), and the exemption list (smoke-manifest-commands, the one non-scenario task).scripts/check-troubleshoot-tasks.py— enforces the contract against every scenario and against the generator template, which it renders as a synthetic scenario. A rule therefore cannot apply to the existing scenarios but not to the generator that produces new ones..github/workflows/troubleshoot-task-contract.yml— PR gate, nopaths:filter so it can be wired as a required check (sub-second full scan, definitive pass/fail). Same shape astask-driver-gate.yml.CLAUDE.mdpoints at the contract and the local command.Rules were derived from what is already documented as MUST in the suite CLAUDE.md and currently holds for 100% of scenarios, so the gate is green on main from the first commit. Constraints that are only mostly-followed (
Never invent...,Never instruct the agent to stop..., 3-5 scenarios each) were deliberately left out rather than enforced retroactively.Verification
Negative tests (all correctly fail with an annotated line + fix hint):
file_existscriterion addedllm_judge.include_reference: falsellm_judge.weight: 2.0mock_path_dirs: [mocks]uipath-troubleshoottag removedNotes for the reviewer
Two pre-existing issues found while building this, deliberately NOT changed here:
sandbox.driver: tempdir, whichscripts/check-task-driver.pyblocks for task YAMLs. The existing gate only scanstests/tasks/**.yaml, so the template escapes it — newly generated scenarios would carry a forbidden driver until someone notices.scripts/check-task-driver.pycallsread_text()with no encoding, so it crashes withUnicodeDecodeErroron a Windows shell (CI is unaffected because Linux defaults to UTF-8).