Repository navigation
feat(uipath-planner): sdd validate is the case SDD conformance gate; the 35-item walk is a no-sdd fallback (S4, MST-15966) - #3717
Closed
roberts-cliff wants to merge 28 commits into
Conversation
…the 35-item walk is a no-sdd fallback (S4, MST-15966) The CLI now puts every conformance-checklist item behind validate or convert (cli 665a7cdf5: validate 18, warns 2, convert 7, excluded 1, silent 0), so the planner stops walking the checklist and gates the ready flip on `uip maestro case sdd validate` alone. The walk survives only as a fallback when the CLI has no `sdd` command: uip 1.202.1 and 1.203.0-preview.205 both answer `unknown command 'sdd'`, and any CLI that has `sdd validate` carries the 16 gate checks (cli#4766 is in the cli#4743 rollup). Measured on the planner case lane, 2 runs per arm, same archive (cli 665a7cdf5), walk vs no walk: 10 shared tasks 19/20 vs 19/19 (+1 judge no-verdict), mean score 0.976 vs 0.986, list-price cost $1.98 vs $1.44 mean ($1.63 vs $1.39 median), 13.7 vs 8.6 min, 69k vs 49k output tokens; cheaper on 8 of 10 tasks. The measured arm left the template footer pointing at the checklist and 4 of 24 runs still read it; this change also moves the footer. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…finalization skips the layers guide before the first Read Checklist item 6 asked execute-connector-activity for **Resolved Resource:**, which the template's Connector Task Detail does not have; the gate built from it (cli#4766) fired SDD_TASK_BLOCK_INCOMPLETE 86 times across 48 S4 runs and drove 15-28 extra repair ops on the two dearest tasks. cli 54d04b951 drops the requirement; the checklist now matches the template. The finalization read budget sat ten lines below "read all three, in parallel", so 8 of 10 finalize runs read the layers guide anyway. SKILL.md is loaded before any Read, so the exception now lives there. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nd sees The coverage record now seeds every checklist item (cli 54d04b951): validate 20, validate-warns 2, convert 7, excluded 1, silent 3 (17-19: data closure), needs-draft 4 (30-33: draft parity, which no CLI command can read). Items 28 and 35 are caught, by neighbouring rules (SDD_RULE_NOT_FOR_SLOT, SDD_WAIT_FOR_USER_UNPAIRED). The planner checks those seven inline on the written file — 17-19 always, 30-33 when finalizing a draft — each stated in full in the lane, no subagent, no checklist read. Everything else stays sdd validate's; the full walk remains only the no-sdd fallback. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… Value, as the template does The template's Timer Task Detail has no **Duration:**; the gate built from the checklist fired SDD_TASK_BLOCK_INCOMPLETE "lacks **Duration:**" 4 times in two S4 runs on cli 36b85f8e4. Same class as the Resolved Resource fix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… 13 moves to the gate
SDD_SLA_REFERENCE_UNRESOLVED (cli#4766) catches both item 13 seeds: an
sla-status-change naming an undeclared SLA title, and the unclosed form from a
failed S4 run (`sla-status-change("Assess (`assess`)`). Tally: validate 22,
validate-warns 2, convert 6, excluded 1, silent 3 (17-19), needs-draft 4
(30-33). cli c2b3c72db (the #4743 rollup tip, adds #4774) changes no gate check
or convert output, so the verdicts hold there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…th arms runnability_disclosed builds the all-placeholder sla_from_sdd case and grades that the completion report says the case cannot run and names where it stops. runnability_silent builds the runnable expense case (all resources deployed) and grades that the report says nothing about runnability. Each arm's precondition is deterministic (check_runnability_shape.py --expect blocked|runnable, falsified on a 29-placeholder user case and a resolved fixture); only the reply is judged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… line, not a numbered report field runnability_silent failed 2/2: on a fully runnable case both completion reports printed a Runnability section with zero counts, although field 6 said to omit it at zero. A numbered field reads as a slot to fill. The disclosure now sits after the field list as two lines written only when a count is non-zero, with an explicit 'write nothing' for the clean case. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ity items cli 6f3b4ba96 (the #4743 rollup; items 17-19 in cli#4766 at 7619dd18b) puts data closure behind sdd validate: SDD_VARIABLE_NEVER_PRODUCED, SDD_OUT_VARIABLE_UNSET, SDD_BUTTON_TARGET_UNKNOWN. Coverage over all 35 items: validate 25, validate-warns 2, convert 6, excluded 1, needs-draft 4, silent 0. The planner now checks only items 30-33, and only when finalizing a draft; checklist item 17 also names a Buttons assignment as a producer, as the gate does. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…; never stamp a marker into one resolve wrote #3671's "never change a field resolve wrote" did not move the refusal rate (3/6 runs before, 3/6 after, hitl smoke, cli f7c68a2e9): planning §3.4 still told the agent to record <UNRESOLVED: ...> in the entry's taskTypeId slot, and Step 9.1 / placeholder-tasks / Rule 9 keyed the placeholder on that marker. So the agent opened resolve's entry to stamp it, re-ran the lookup under the task name ("37 entries checked"), rewrote searchQuery.name and cacheFile, and convert --resolved refused. Resolve already writes selected:null and a rationale for an empty lookup; that is now the placeholder signal, and the marker belongs only to entries the agent writes itself. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…verbatim, not by reference runnability_disclosed passed 2 of 4: the misses listed every placeholder but never said a run would stop or where. The report paragraph now carries the two lines as a fill-in template instead of pointing at the Phase 6 section. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…le, receipt or not; normalize only what the CLI cannot read #3690 sent every SDD without the planner receipt to planner normalization. Most case-eval SDDs (44 of 49) are real designs written outside the planner, and the route fired inconsistently on them (init-validate normalized, case_exit_full and timer_connector_from_sdd built directly). Rule 2 now decides three ways: receipt present → build; no receipt but sdd parse reads it whole → build it directly, exactly the same; no receipt and the parse fails → normalize through the planner, keeping sdd.original.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…interrupts; parallel oversight is start-task or notify-only The docs taught a non-interrupting secondary SLA "oversight lane" (Interrupting: No). The case compiler sets cancelStagesAndTasks for any secondary-stage entry, whatever isInterrupting says, and cli#4804 now refuses the shape. Falsified 2026-10-05 rather than taken on report: an isInterrupting:false secondary case-entered entry packs to "cancelStagesAndTasks":true, and debug instance 6ecbe140 ran only the secondary lane while the primary stage, also on case-entered, never started. Rule 22, the secondary-lane anti-pattern, sla-response-shapes §1/§4/§5, the stages and stage-entry plugins, the planner layers guide (carve-out, response table, Interrupting cell) and the template's secondary rule now say Yes on every secondary entry, and route work-keeps-running requirements to start-task or notify-only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… an at-risk SLA rule All 14 sla-status-change rows in the eval fixtures were 2-argument breach rows, so nothing exercised the template's 3-argument at-risk form. This variant of sla_from_sdd moves Intake's start-task to at-risk and grades the built rule: Intake's slaId plus the escalationId of the 'Intake at risk' at-risk escalation. Falsified on cli 6f3b4ba96, which read argument 3 as the SLA title: convert left the rule with no slaId at all. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… head (cli 90b5247e6) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d-stage-exited carries an IF A secondary stage entered on selected-stage-exited(X) with no IF takes over every time X completes normally. Seen in a planner run (loan-origination, Withdrawn lane on any exit of Intake and Loan Setup) and in 10 corpus documents. cli#4766 now refuses it as SDD_SECONDARY_ENTERS_ON_COMPLETION (item 36). The checklist (the planner's no-sdd fallback walk), the template count and the coverage seed follow; coverage on cli fe4c3e3e2 reads it validate. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ceipt-carrying SDD; without the receipt it normalizes Rule 2's three-way decision sends an SDD with no planner receipt that fails sdd parse to planner normalization (c), but its refusal clause said any parse/format/resolve/convert refusal stops the build. The suite showed which one wins: all four no-receipt SDDs refused on a Case Variables Type `number` (io-binding-matrix, three hitl-quality tasks) stopped and asked the user, scoring 0.00 at $0.18-0.46. The stop now names case (a) only, and (c) names a single-cell refusal as its own. E1a's fixture carries the receipt, so its hand-back is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…lookup fields are untouchable registry-handoff-ambiguous passed 14/14 nightlies on main and failed both stack runs (0.14, 0.57). #3671's "never change a field resolve wrote" listed `selected`, so after the user picked a candidate the agent recorded a gateDecision but left selected null and bound nothing. convert --resolved compares stage, task and searchQuery, not selected: the rule now protects the lookup fields and names the one write a gate answer requires (selected to the chosen match, with gateDecision pick:<name>). An ambiguous entry goes to the gate before it can become a placeholder. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ompleted
The Lifecycle gates table allowed only required-tasks-completed and
wait-for-connector on a completing stage row, matching the CLI gate's item 9.
Both were wrong (cli-62, from @uipath/case-schema: the schema, compiler and
scheduler all accept selected-tasks-completed as a TasksCompleted completion
condition; cli fixed in 990b37cda). Planner normalization rewrote a real SDD's
selected-tasks-completed("Approve Contract") completion row to
required-tasks-completed, changing which tasks complete the stage, and
hitl quality_04 failed on it. Checklist item 9 reads this table, so the
fallback walk follows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…gainst runtime truth case_breach_oversight graded isInterrupting false, which the runtime falsified (debug instance 72162d9b: the breach exited the active stage and cancelled its task). It now requires isInterrupting true on the still-secondary, non-required lane and judges whether the reply tells the user the work will pause. quality_01 accepts one trivial required task in Closed: an empty stage with a required-tasks-completed exit never completes (instance cc7223dc). io-binding accepts double or integer where the SDD wrote number, the form planner normalization produces. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…et layout The SDD declares no task set, so convert puts each timer, with its own current-stage-entered rule, in its own set. Debug instance 2df59883 shows two timers in separate sets start and finish together, so the shared-set assertion graded layout, not behaviour. The entry-rule assertion stays. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The oversight eval accepted only a lane, but the skill's parallel-oversight shape is a start-task. Debug instance 583c4102 shows a root-SLA breach during Intake starts such a task even in not-yet-entered Decision, cancels nothing, and the case completes, so it truly runs alongside. The checker now passes either a non-required breach task with no new stage, or a secondary lane with isInterrupting true; the judge grades the reply's claim against the shape built. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ng input The io_binding fixture writes the non-template type number on purpose: its eval grades the no-receipt, parse-fails route to planner normalization (run 37494190705: the planner rewrote exactly those three cells, build 1.00). None of the parse-clean test's three verdicts fit, and an exclusion would stop checking the file. A fourth verdict, routing-input, pins the exact refusal instead, so a new defect in the file or a CLI that starts accepting number both fail. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…st their seed read-as-seed required sdd validate to exit zero, but the seeds are hand-written pre-template SDDs that strict validate rejects on their own (32 issues on e2e_expense_runnable: no Planner Handoff, task envelopes or activation modes). The fixture now must give the seed's exit code and issues plus exactly its recorded warning, and the seed's Model. table-row-missing-leading-pipe passes on 1.205: the row and everything below it are read, with the warning at 312. Records the 1.205 measurement beside the old reader's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…defect The norm() merge it was built for is fixed; what remained was a non-identifier variable name that convert wrote as an unreachable id, retyped to string, and both validators passed. cli #4804 now refuses it at line 70 with SDD_VARIABLE_NAME_NOT_IDENTIFIER, so reject-naming-line passes as recorded. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…names cells A no-receipt SDD one cell from parsing (a Case Variables Type of number) took the full re-render: about 150 s on Opus, 20 s of it re-typing the original and 75 s regenerating a document that needed one word changed. Nine of the ten normalizing evals were cell-level. The lane now copies the original with one cp, edits only the named cells when parse read the stages, and hands back once it parses whole (case Rule 2 (b)); the template render runs only for a refusal that is not cell-level. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…from sdd format The test hand-built a sidecar with no schemaVersion; the CLI's model schema moved to 2.0.0 and refuses an unversioned file as 1.0.0. The envelope now comes from a real sdd format run, so the test follows the CLI's schema. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On Opus the design lane's gate got 40% faster once sdd validate replaced the checklist walk, but the writing phase gave it back: both finalize tasks ran validate twice on half-written files and repaired errors the next append would have cleared, and 4 of 6 design tasks ran --help or --version to rediscover commands the lane already states. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Closing the MST-15966 SDD-quest stack; it isn't landing as is. The head is kept locally. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is S4, stacked on #3712. #3691 merges the whole stack.
Change. In the planner's Case Design Lane,
uip maestro case sdd validatenow gates the ready flip on its own. The 35-item conformance-checklist walk, and the subagent that ran it, are gone from every mode. The template's § Validation footer, planner SKILL.md andsdd-generation-guide.mdnow say the same.Fallback. The walk survives only when the CLI has no
sddcommand. I checked the released CLIs: uip 1.202.1 (latest) and 1.203.0-preview.205 both answerunknown command 'sdd'. Any CLI that does havesdd validatecomes from the cli#4743 rollup, which carries the 16 gate checks (cli#4766). That makes the unknown-command answer an exact signal for "no real gate". So this PR is safe to merge before the CLI ships, because today's users take the fallback. It still belongs with cli#4743, because the saving only appears on a CLI that hassdd validate.Why now. With cli 665a7cdf5, the checklist coverage record (
checklist_coverage.json, committed in #3712's branch at d29b9dd) reads validate 18, validate-warns 2, convert 7, excluded 1 (Design Rationale, by decision), silent 0. Every checklist item is now behind validate or convert. S4 used to be blocked at "validate catches 2 of 26".Evidence: planner case-lane A/B, run in CI. 12 tasks, 2 runs per arm, claude-sonnet-5, all on archive 37035709980 (cli 665a7cdf5). Cost is the agent SDK's list-price estimate on Bedrock, not billed money.
procurement, which passed in both no-walk runs.tmp/planner-ab-nowalk) spawned no checklist subagent in 24 runs, against 11 of 24 with the walk. 4 of its 24 runs still read the checklist, because the template footer still pointed at it. This PR also moves the footer.Checks:
skills:validatepasses, and the planner and case checker tests pass (375).🤖 Generated with Claude Code