fix: azure-app-onboard & prereq - Harden app-onboard family: naming, schemas, cleanup

[vaibbavis]

Description

Harden app-onboard family: naming, schemas, cleanup

Consistency/hardening pass across the app-onboard pipeline — no new features. It tightens the contracts between phases so downstream steps read stable data, and removes dead reference files. Grouped by phase for review.

Cross-cutting — session schema (session-schemas.ts, both prereq + orchestrator copies)

• verdicts on each component is now required (was optional) and documented as separate from the top-level overallHealth enum — prepare/readiness scoring depend on it.
• New fields: tenantId + userDisplayName (deploy identity/tags), lastScanCommit (prereq staleness guard).
• Trimmed AppOnboardPhase to the 5 real phases; expanded routeToSkill/routeReason docs.

Prereq

• deployability-check.md: apps using SQLite/Postgres/Redis/Kafka that map to an Azure service now classify readyWithCaveats, not blocked — they're a mapping concern, not a blocker. (behavior change — worth a close look.)
• prereq-schemas.ts: new CloudSdkFinding type + cloudSdkFindings[] so the cloud-SDK gate can surface AWS/GCP deps and route to azure-cloud-migrate.
• prereq-artifacts.md: per-component verdicts must persist; clarified it's a plural object, not singular.
• completeness-check.md: fixPhase spelling postdeploy → post-deploy; SKILL.md mandatory-read phrasing hardened.

Prepare

• Minor schema alignment (prepare-schemas.ts); intent now written after Step 4 (was Step 2).

Scaffold

• Naming — plan is source of truth (bicep-patterns.md, subagent-validate.md): scaffold uses prepare-plan.json.naming.resources[] verbatim instead of re-deriving with uniqueString()/take()/substring(). Why: re-deriving produced names that didn't match the plan and failed the conformance gate.
• self-review-checklist.md: no az deployment sub what-if at scaffold (needs real secrets) — deploy runs it.
• Validation checks standardized to { passed: true }; ConformanceResult.source always "script"; manifest written to the exact session path.
• env-var-secrets.md: use bare administratorLogin (Flexible Server rejects the legacy {admin}@{server} form).
• Deleted: cicd-pipelines.md, error-handling.md, self-review-procedure.md, waf-checklist.md.

Deploy

• SKILL.md: portal-link generation inlined into Step 6 (added a Terraform activity-log variant; re-emits a fresh link on redeploy). Failures recorded into resourceResults[].error + healingAttempts[].
• preflight-checks.md: rename-on-collision writes the accepted name back into the plan before redeploy (keeps scaffold/deploy in sync); dropped --what-if-result-format FullResourcePayloads.
• deploy-schemas.ts: completedUtc optional (set at finalize); ResourceDeployStatus drops unused skipped.
• Deleted: portal-links.md (moved into SKILL.md).

Orchestrator

• approval-gates.md: after deploy approval, read deploy-checklist.md (compaction-safe anchor) first, then deploy/SKILL.md; existing azure.yaml → rename infra/ to infra.bak/.
• SKILL.md: MCP-unavailable disclaimer now targets the specific prepare-plan.json.assumptions (scaffold gate) and costEstimate.disclaimer (deploy gate) fields.

Evals & tooling

• onboard.eval.yaml / prepare.eval.yaml: early-stop matchers now fire on the prepare-plan.json / scaffold-manifest.json write-result (not spoken text); added a cloudSdkFindings grader.
• run-onboard-vally.prompt.md tweaks + logged the 07-22/07-23 runs (README index).

---

Validation — eval run 2026-07-27 (vally v0.7, claude-sonnet-4.6, real Azure for deploy)

Combined: 51/57 tests · 448/464 graders. All 8 suites clear the 80% weighted threshold.

Phase	Tests	Weighted
Prereq	19/22	97.9% ✅
Prepare	10/10	100% ✅
Onboard	14/14	100% ✅
Scaffold	5/6	97.9% ✅
Deploy (fasttrack · wetty · deploy-depth · seeded-deploy)	3/5	96.4–100% ✅ / seeded 80.8%

3 of 5 real deploys reached status:"succeeded" + healthStatus:"healthy" with live endpoints. 0 harness errors.

Why seeded-deploy is 80.8% (vs 96–100% elsewhere): it's 2 stimuli — App Service Pipeline 100% (25/25) and Container Apps Pipeline 61.5% (16/26). The Container Apps half early-stopped at the approval gate and never ran az acr build/az deployment, so it lost the 10 downstream graders (succeeded status, live endpoint, healthy, deployment-summary.md, cleanup). Averaging a perfect run with one early-stop pulls the suite to 80.8% — it's one flaky stop, not a systemic seeded-deploy problem (no RG was even created).

All 6 failures are known recurring flaky patterns, not regressions from this PR: 1 approval-gate early-stop (above) · 2 missing scaffold-manifest.json (async-flush; one deploy still succeeded) · 3 prereq non-determinism (overallHealth classification, missing readiness-report.md + axis verdicts, routeToSkill not persisted).

The new schema fields are exercised and passing where the model persists them (Cloud SDK Continue 7/7, Go Gin routing 5/5) — the 3 prereq misses are occasional non-persistence, not schema errors.

---

Checklist

• Tests pass locally (cd tests && npm test)
• Title has one of the prefixes: fix:, feat:, feature:, chore:, misc:, test:, eval:
• If modifying skill descriptions: verified routing correctness with integration tests (In tests/, npm run test:integration -- <skill> or npm run test:vally -- --skill <skill>)

Related Issues

[Copilot]

Pull request overview

This PR hardens and aligns the azure-app-onboard skill family’s phase contracts (session schemas, artifact expectations, naming, validation/manifests) and updates deploy guidance + Vally eval termination behavior to be more deterministic.

Changes:

• Tightens session/artifact schemas and documentation (e.g., required per-component verdicts, added tenant/user identity fields, phase enum trimming).
• Makes scaffold/deploy more contract-driven (plan-as-source-of-truth naming, standardized validation result shape, deploy skeleton/portal link + redeploy behavior).
• Updates Vally eval suites to early-terminate on artifact writes rather than ambiguous assistant text.

Show a summary per file

File	Description
plugins/azure-skills/skills/azure-app-onboard/SKILL.md	Orchestrator step table hardening and updated MCP-unavailable disclaimer targets.
plugins/azure-skills/skills/azure-app-onboard/scaffold/SKILL.md	Scaffold phase guidance updates (backup behavior, troubleshooting additions).
plugins/azure-skills/skills/azure-app-onboard/scaffold/references/validation-and-manifest.md	Manifest/validation check schema normalization and explicit manifest write path guidance.
plugins/azure-skills/skills/azure-app-onboard/scaffold/references/subagent-validate.md	Conformance validation guidance updated to enforce plan-verbatim naming.
plugins/azure-skills/skills/azure-app-onboard/scaffold/references/subagent-iac-gen.md	IaC generation guidance tightened (deployerObjectId resolution expectations).
plugins/azure-skills/skills/azure-app-onboard/scaffold/references/self-review-checklist.md	Removes scaffold-stage what-if; clarifies validation responsibilities.
plugins/azure-skills/skills/azure-app-onboard/scaffold/references/scaffold-schemas.ts	Conformance result schema tightened (source always "script"), doc wording updates.
plugins/azure-skills/skills/azure-app-onboard/scaffold/references/env-var-secrets.md	Flexible Server admin username mapping clarified.
plugins/azure-skills/skills/azure-app-onboard/scaffold/references/bicep-patterns.md	Naming guidance changed to “plan is source of truth,” with fallback guidance.
plugins/azure-skills/skills/azure-app-onboard/scaffold/references/bicep-patterns-security.md	Clarifies deployer RBAC source (deployerObjectId).
plugins/azure-skills/skills/azure-app-onboard/references/session-schemas.ts	Session schema hardened (required component verdicts, new identity/staleness fields, phase enum trim).
plugins/azure-skills/skills/azure-app-onboard/references/session-protocol.md	Session artifact location table updated (and backup-path guidance removed).
plugins/azure-skills/skills/azure-app-onboard/references/pipeline-rules.md	Adjusts pipeline rule timing (intent write step).
plugins/azure-skills/skills/azure-app-onboard/references/pipeline-rules-runtime.md	Updates runtime guidance for secrets with shell-special chars.
plugins/azure-skills/skills/azure-app-onboard/references/handoff-protocol.md	Step number alignment with orchestrator.
plugins/azure-skills/skills/azure-app-onboard/references/azd-template-routing.md	“Start fresh” backup behavior clarified; avoids replaced-files path behavior.
plugins/azure-skills/skills/azure-app-onboard/references/approval-gates.md	Deploy approval ordering clarified and backup behavior aligned with azd-template routing.
plugins/azure-skills/skills/azure-app-onboard/prepare/SKILL.md	Prepare phase step ordering revised; adds reference pointer for MCP tool parameters.
plugins/azure-skills/skills/azure-app-onboard/prepare/references/prepare-schemas.ts	Schema docs aligned with new prepare step numbering; naming suffix field added.
plugins/azure-skills/skills/azure-app-onboard/deploy/SKILL.md	Deploy instructions hardened (portal link regen on redeploy, status messaging guardrails, error capture guidance).
plugins/azure-skills/skills/azure-app-onboard/deploy/references/subagent-preflight.md	Updates deploy-result skeleton shape (duration nesting).
plugins/azure-skills/skills/azure-app-onboard/deploy/references/preflight-checks.md	Preflight name-collision sync-back and what-if formatting changes.
plugins/azure-skills/skills/azure-app-onboard/deploy/references/error-classification.md	Healing guidance aligned to plan-verbatim naming (plan name updates).
plugins/azure-skills/skills/azure-app-onboard/deploy/references/deploy-schemas.ts	Deploy schema adjustments (completedUtc optional, remove unused status).
plugins/azure-skills/skills/azure-app-onboard/deploy/references/deploy-safety.md	Tightens “sanctioned secret file” and variable-passing guidance.
plugins/azure-skills/skills/azure-app-onboard/deploy/references/deploy-checklist-template.md	Adds explicit “warnings are not a stop signal” rules and status gating.
plugins/azure-skills/skills/azure-app-onboard/deploy/references/blocked-patterns.md	Refines secret-handling blocks (literal vs variable; sanctioned cache file).
plugins/azure-skills/skills/azure-app-onboard-prereq/SKILL.md	Strengthens mandatory-read language and cloud SDK gate phrasing.
plugins/azure-skills/skills/azure-app-onboard-prereq/references/subscription-resolution.md	Adds login attempt cap + clearer halt behavior.
plugins/azure-skills/skills/azure-app-onboard-prereq/references/session-schemas.ts	Prereq copy of shared session schema updated to match orchestrator copy.
plugins/azure-skills/skills/azure-app-onboard-prereq/references/session-protocol.md	Azure login gate updated (timeouts, single-write merge incl. displayName, .gitignore enforcement).
plugins/azure-skills/skills/azure-app-onboard-prereq/references/remediation-protocol.md	Clarifies “no lockfile regeneration” as part of static-only remediation.
plugins/azure-skills/skills/azure-app-onboard-prereq/references/readiness-gate.md	Tightens semantics of overallHealth vs axis verdicts; updates user-choice phrasing.
plugins/azure-skills/skills/azure-app-onboard-prereq/references/prereq-schemas.ts	Adds Cloud SDK findings schema for migration gating.
plugins/azure-skills/skills/azure-app-onboard-prereq/references/prereq-artifacts.md	Clarifies verdicts persistence contract and enum separation.
plugins/azure-skills/skills/azure-app-onboard-prereq/references/deployability-check.md	Reclassifies external managed-service deps as WARN (mapping concern), not blockers.
plugins/azure-skills/skills/azure-app-onboard-prereq/references/completeness-check.md	Fixes fixPhase spelling (post-deploy).
evals/azure-app-onboard/prepare.eval.yaml	Early-terminate rules updated to stop on artifact writes rather than assistant text.
evals/azure-app-onboard/onboard.eval.yaml	Early-terminate rules refined and adds artifact-based remediation proof checks.
evals/azure-app-onboard-prereq/eval.yaml	Terminator regex adjusted and readiness-report existence checks repositioned.

Review details

• Files reviewed: 40/40 changed files
• Comments generated: 3
• Review effort level: Low

[JasonYeMSFT (JasonYeMSFT)]

vaibbavis The changes look good to me but I am not sure why there are merge conflicts. Please resolve them and let me know when it's done.