Skip to content

feat(uipath-planner): sdd validate is the case SDD conformance gate; the 35-item walk is a no-sdd fallback (S4, MST-15966) - #3717

Closed
roberts-cliff wants to merge 28 commits into
feat/case-unfinished-connector-rulefrom
feat/planner-s4-validate-is-the-gate
Closed

roberts-cliff wants to merge 28 commits into
feat/case-unfinished-connector-rulefrom
feat/planner-s4-validate-is-the-gate

Conversation

@roberts-cliff

@roberts-cliff roberts-cliff commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

This is S4, stacked on #3712. #3691 merges the whole stack.

Change. In the planner's Case Design Lane, uip maestro case sdd validate now gates the ready flip on its own. The 35-item conformance-checklist walk, and the subagent that ran it, are gone from every mode. The template's § Validation footer, planner SKILL.md and sdd-generation-guide.md now say the same.

Fallback. The walk survives only when the CLI has no sdd command. I checked the released CLIs: uip 1.202.1 (latest) and 1.203.0-preview.205 both answer unknown command 'sdd'. Any CLI that does have sdd validate comes from the cli#4743 rollup, which carries the 16 gate checks (cli#4766). That makes the unknown-command answer an exact signal for "no real gate". So this PR is safe to merge before the CLI ships, because today's users take the fallback. It still belongs with cli#4743, because the saving only appears on a CLI that has sdd validate.

Status after correction (126b125). An earlier version of this PR claimed every checklist item was behind validate or convert. That was wrong: the coverage record measured only 26 of the 35 items. It now seeds all of them (cli 54d04b951): validate 20, validate-warns 2, convert 7, excluded 1, silent 3 (17–19, data closure) and needs-draft 4 (30–33, draft parity, which no CLI command reads). The planner now checks those seven CLI-blind items inline on the written file: 17–19 always, 30–33 when finalizing a draft. Each is stated in full in the lane, with no subagent and no checklist read. Everything else stays with sdd validate. The first S4 confirmation run had failed item 32's territory (the Credit Analyst >$5M gate in finalize-draft-loan). Five runs of this version are in CI to measure whether that recurs (37054889396, 37054902644, 37054916203, 37054929031, 37054941191).

Why now. With cli 665a7cdf5, the checklist coverage record (checklist_coverage.json, committed in #3712's branch at d29b9dd) reads validate 18, validate-warns 2, convert 7, excluded 1 (Design Rationale, by decision), silent 0. Every checklist item is now behind validate or convert. S4 used to be blocked at "validate catches 2 of 26".

Evidence: planner case-lane A/B, run in CI. 12 tasks, 2 runs per arm, claude-sonnet-5, all on archive 37035709980 (cli 665a7cdf5). Cost is the agent SDK's list-price estimate on Bedrock, not billed money.

10 shared tasks Walk (stack tip) No walk (this change)
Pass 19/20 19/19 (+1 run where the judge returned no verdict)
Mean score 0.976 0.986
Cost, mean / median $1.98 / $1.63 $1.44 / $1.39
Duration 13.7 min 8.6 min
Output tokens 69k 49k
  • No-walk is cheaper on 8 of the 10 shared tasks.
  • The two stack-added tasks are 4/4 in both arms.
  • The one walk-arm failure is a judge's design call on procurement, which passed in both no-walk runs.
  • The measured arm (tmp/planner-ab-nowalk) spawned no checklist subagent in 24 runs, against 11 of 24 with the walk. 4 of its 24 runs still read the checklist, because the template footer still pointed at it. This PR also moves the footer.

Checks: skills:validate passes, and the planner and case checker tests pass (375).

🤖 Generated with Claude Code

…the 35-item walk is a no-sdd fallback (S4, MST-15966)

The CLI now puts every conformance-checklist item behind validate or convert
(cli 665a7cdf5: validate 18, warns 2, convert 7, excluded 1, silent 0), so the
planner stops walking the checklist and gates the ready flip on
`uip maestro case sdd validate` alone. The walk survives only as a fallback
when the CLI has no `sdd` command: uip 1.202.1 and 1.203.0-preview.205 both
answer `unknown command 'sdd'`, and any CLI that has `sdd validate` carries
the 16 gate checks (cli#4766 is in the cli#4743 rollup).

Measured on the planner case lane, 2 runs per arm, same archive (cli
665a7cdf5), walk vs no walk: 10 shared tasks 19/20 vs 19/19 (+1 judge
no-verdict), mean score 0.976 vs 0.986, list-price cost $1.98 vs $1.44 mean
($1.63 vs $1.39 median), 13.7 vs 8.6 min, 69k vs 49k output tokens; cheaper
on 8 of 10 tasks. The measured arm left the template footer pointing at the
checklist and 4 of 24 runs still read it; this change also moves the footer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
roberts-cliff and others added 27 commits October 2, 2026 12:24
…finalization skips the layers guide before the first Read

Checklist item 6 asked execute-connector-activity for **Resolved Resource:**,
which the template's Connector Task Detail does not have; the gate built from
it (cli#4766) fired SDD_TASK_BLOCK_INCOMPLETE 86 times across 48 S4 runs and
drove 15-28 extra repair ops on the two dearest tasks. cli 54d04b951 drops the
requirement; the checklist now matches the template.

The finalization read budget sat ten lines below "read all three, in
parallel", so 8 of 10 finalize runs read the layers guide anyway. SKILL.md is
loaded before any Read, so the exception now lives there.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nd sees

The coverage record now seeds every checklist item (cli 54d04b951): validate
20, validate-warns 2, convert 7, excluded 1, silent 3 (17-19: data closure),
needs-draft 4 (30-33: draft parity, which no CLI command can read). Items 28
and 35 are caught, by neighbouring rules (SDD_RULE_NOT_FOR_SLOT,
SDD_WAIT_FOR_USER_UNPAIRED).

The planner checks those seven inline on the written file — 17-19 always,
30-33 when finalizing a draft — each stated in full in the lane, no
subagent, no checklist read. Everything else stays sdd validate's; the full
walk remains only the no-sdd fallback.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… Value, as the template does

The template's Timer Task Detail has no **Duration:**; the gate built from the
checklist fired SDD_TASK_BLOCK_INCOMPLETE "lacks **Duration:**" 4 times in two
S4 runs on cli 36b85f8e4. Same class as the Resolved Resource fix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… 13 moves to the gate

SDD_SLA_REFERENCE_UNRESOLVED (cli#4766) catches both item 13 seeds: an
sla-status-change naming an undeclared SLA title, and the unclosed form from a
failed S4 run (`sla-status-change("Assess (`assess`)`). Tally: validate 22,
validate-warns 2, convert 6, excluded 1, silent 3 (17-19), needs-draft 4
(30-33). cli c2b3c72db (the #4743 rollup tip, adds #4774) changes no gate check
or convert output, so the verdicts hold there.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…th arms

runnability_disclosed builds the all-placeholder sla_from_sdd case and grades
that the completion report says the case cannot run and names where it stops.
runnability_silent builds the runnable expense case (all resources deployed)
and grades that the report says nothing about runnability. Each arm's
precondition is deterministic (check_runnability_shape.py --expect
blocked|runnable, falsified on a 29-placeholder user case and a resolved
fixture); only the reply is judged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… line, not a numbered report field

runnability_silent failed 2/2: on a fully runnable case both completion
reports printed a Runnability section with zero counts, although field 6 said
to omit it at zero. A numbered field reads as a slot to fill. The disclosure now
sits after the field list as two lines written only when a count is non-zero,
with an explicit 'write nothing' for the clean case.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ity items

cli 6f3b4ba96 (the #4743 rollup; items 17-19 in cli#4766 at 7619dd18b) puts
data closure behind sdd validate: SDD_VARIABLE_NEVER_PRODUCED,
SDD_OUT_VARIABLE_UNSET, SDD_BUTTON_TARGET_UNKNOWN. Coverage over all 35
items: validate 25, validate-warns 2, convert 6, excluded 1, needs-draft 4,
silent 0. The planner now checks only items 30-33, and only when finalizing a
draft; checklist item 17 also names a Buttons assignment as a producer, as
the gate does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…; never stamp a marker into one resolve wrote

#3671's "never change a field resolve wrote" did not move the refusal rate
(3/6 runs before, 3/6 after, hitl smoke, cli f7c68a2e9): planning §3.4 still
told the agent to record <UNRESOLVED: ...> in the entry's taskTypeId slot, and
Step 9.1 / placeholder-tasks / Rule 9 keyed the placeholder on that marker. So
the agent opened resolve's entry to stamp it, re-ran the lookup under the task
name ("37 entries checked"), rewrote searchQuery.name and cacheFile, and
convert --resolved refused. Resolve already writes selected:null and a
rationale for an empty lookup; that is now the placeholder signal, and the
marker belongs only to entries the agent writes itself.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…verbatim, not by reference

runnability_disclosed passed 2 of 4: the misses listed every placeholder but
never said a run would stop or where. The report paragraph now carries the two
lines as a fill-in template instead of pointing at the Phase 6 section.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…le, receipt or not; normalize only what the CLI cannot read

#3690 sent every SDD without the planner receipt to planner normalization.
Most case-eval SDDs (44 of 49) are real designs written outside the planner,
and the route fired inconsistently on them (init-validate normalized,
case_exit_full and timer_connector_from_sdd built directly). Rule 2 now
decides three ways: receipt present → build; no receipt but sdd parse reads
it whole → build it directly, exactly the same; no receipt and the parse
fails → normalize through the planner, keeping sdd.original.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…interrupts; parallel oversight is start-task or notify-only

The docs taught a non-interrupting secondary SLA "oversight lane"
(Interrupting: No). The case compiler sets cancelStagesAndTasks for any
secondary-stage entry, whatever isInterrupting says, and cli#4804 now refuses
the shape. Falsified 2026-10-05 rather than taken on report: an
isInterrupting:false secondary case-entered entry packs to
"cancelStagesAndTasks":true, and debug instance 6ecbe140 ran only the
secondary lane while the primary stage, also on case-entered, never started.

Rule 22, the secondary-lane anti-pattern, sla-response-shapes §1/§4/§5,
the stages and stage-entry plugins, the planner layers guide (carve-out,
response table, Interrupting cell) and the template's secondary rule now say
Yes on every secondary entry, and route work-keeps-running requirements to
start-task or notify-only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… an at-risk SLA rule

All 14 sla-status-change rows in the eval fixtures were 2-argument breach rows,
so nothing exercised the template's 3-argument at-risk form. This variant of
sla_from_sdd moves Intake's start-task to at-risk and grades the built rule:
Intake's slaId plus the escalationId of the 'Intake at risk' at-risk
escalation. Falsified on cli 6f3b4ba96, which read argument 3 as the SLA
title: convert left the rule with no slaId at all.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… head (cli 90b5247e6)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d-stage-exited carries an IF

A secondary stage entered on selected-stage-exited(X) with no IF takes over
every time X completes normally. Seen in a planner run (loan-origination,
Withdrawn lane on any exit of Intake and Loan Setup) and in 10 corpus
documents. cli#4766 now refuses it as SDD_SECONDARY_ENTERS_ON_COMPLETION
(item 36). The checklist (the planner's no-sdd fallback walk), the template
count and the coverage seed follow; coverage on cli fe4c3e3e2 reads it validate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ceipt-carrying SDD; without the receipt it normalizes

Rule 2's three-way decision sends an SDD with no planner receipt that fails
sdd parse to planner normalization (c), but its refusal clause said any
parse/format/resolve/convert refusal stops the build. The suite showed which
one wins: all four no-receipt SDDs refused on a Case Variables Type `number`
(io-binding-matrix, three hitl-quality tasks) stopped and asked the user,
scoring 0.00 at $0.18-0.46. The stop now names case (a) only, and (c) names a
single-cell refusal as its own. E1a's fixture carries the receipt, so its
hand-back is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…lookup fields are untouchable

registry-handoff-ambiguous passed 14/14 nightlies on main and failed both
stack runs (0.14, 0.57). #3671's "never change a field resolve wrote" listed
`selected`, so after the user picked a candidate the agent recorded a
gateDecision but left selected null and bound nothing. convert --resolved
compares stage, task and searchQuery, not selected: the rule now protects the
lookup fields and names the one write a gate answer requires (selected to the
chosen match, with gateDecision pick:<name>). An ambiguous entry goes to the
gate before it can become a placeholder.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ompleted

The Lifecycle gates table allowed only required-tasks-completed and
wait-for-connector on a completing stage row, matching the CLI gate's item 9.
Both were wrong (cli-62, from @uipath/case-schema: the schema, compiler and
scheduler all accept selected-tasks-completed as a TasksCompleted completion
condition; cli fixed in 990b37cda). Planner normalization rewrote a real SDD's
selected-tasks-completed("Approve Contract") completion row to
required-tasks-completed, changing which tasks complete the stage, and
hitl quality_04 failed on it. Checklist item 9 reads this table, so the
fallback walk follows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…gainst runtime truth

case_breach_oversight graded isInterrupting false, which the runtime falsified
(debug instance 72162d9b: the breach exited the active stage and cancelled its
task). It now requires isInterrupting true on the still-secondary, non-required
lane and judges whether the reply tells the user the work will pause.

quality_01 accepts one trivial required task in Closed: an empty stage with a
required-tasks-completed exit never completes (instance cc7223dc).

io-binding accepts double or integer where the SDD wrote number, the form
planner normalization produces.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…et layout

The SDD declares no task set, so convert puts each timer, with its own
current-stage-entered rule, in its own set. Debug instance 2df59883 shows two
timers in separate sets start and finish together, so the shared-set
assertion graded layout, not behaviour. The entry-rule assertion stays.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The oversight eval accepted only a lane, but the skill's parallel-oversight
shape is a start-task. Debug instance 583c4102 shows a root-SLA breach during
Intake starts such a task even in not-yet-entered Decision, cancels nothing, and
the case completes, so it truly runs alongside. The checker now passes either a
non-required breach task with no new stage, or a secondary lane with
isInterrupting true; the judge grades the reply's claim against the shape built.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ng input

The io_binding fixture writes the non-template type number on purpose: its eval
grades the no-receipt, parse-fails route to planner normalization (run
37494190705: the planner rewrote exactly those three cells, build 1.00). None of
the parse-clean test's three verdicts fit, and an exclusion would stop checking
the file. A fourth verdict, routing-input, pins the exact refusal instead, so a
new defect in the file or a CLI that starts accepting number both fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…st their seed

read-as-seed required sdd validate to exit zero, but the seeds are hand-written
pre-template SDDs that strict validate rejects on their own (32 issues on
e2e_expense_runnable: no Planner Handoff, task envelopes or activation modes).
The fixture now must give the seed's exit code and issues plus exactly its
recorded warning, and the seed's Model. table-row-missing-leading-pipe passes
on 1.205: the row and everything below it are read, with the warning at 312.
Records the 1.205 measurement beside the old reader's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…defect

The norm() merge it was built for is fixed; what remained was a non-identifier
variable name that convert wrote as an unreachable id, retyped to string, and
both validators passed. cli #4804 now refuses it at line 70 with
SDD_VARIABLE_NAME_NOT_IDENTIFIER, so reject-naming-line passes as recorded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…names cells

A no-receipt SDD one cell from parsing (a Case Variables Type of number) took
the full re-render: about 150 s on Opus, 20 s of it re-typing the original and
75 s regenerating a document that needed one word changed. Nine of the ten
normalizing evals were cell-level. The lane now copies the original with one
cp, edits only the named cells when parse read the stages, and hands back once
it parses whole (case Rule 2 (b)); the template render runs only for a refusal
that is not cell-level.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…from sdd format

The test hand-built a sidecar with no schemaVersion; the CLI's model schema
moved to 2.0.0 and refuses an unversioned file as 1.0.0. The envelope now
comes from a real sdd format run, so the test follows the CLI's schema.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On Opus the design lane's gate got 40% faster once sdd validate replaced the
checklist walk, but the writing phase gave it back: both finalize tasks ran
validate twice on half-written files and repaired errors the next append would
have cleared, and 4 of 6 design tasks ran --help or --version to rediscover
commands the lane already states.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@roberts-cliff

Copy link
Copy Markdown
Contributor Author

Closing the MST-15966 SDD-quest stack; it isn't landing as is. The head is kept locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant