Fleet controller - Compensate VMSS capacity for phantom drift one settled mutation at a time

[potatoqualitee]

Problem

On Flexible orchestration, deleting spent ephemeral VMs one at a time leaves sku.capacity (nominal) above the number of instances that exist (actual) — observed in production as sku=9 with 6 VMs. A capacity PATCH creates newValue − nominal instances, not newValue − actual, so scaling "to target" under drift silently under-creates: a ten-runner lane sat at six VMs. Two earlier designs failed: normalize-then-scale cancels in-flight creations, and one-plan-step-per-pass lets churn-minted drift starve creation entirely (the fleet drained 8→5→3 against a target of 20).

⛔→✅ Merge gate: RESOLVED — scale-in is destructive, branches reworked

The experiment never had to be run; production answered on 2026-08-08:

• Down-PATCHes delete live instances. During the drain, six consecutive normalization passes each deleted exactly nominal − newValue live VMs (App Insights decision traces matched against per-VM lifecycle, verified pass by pass).
• It killed this PR's own CI run. Run 31273376357 (19:01 UTC): all ten SQL lanes died with "self-hosted runner lost communication" and zero failed steps — healthy jobs whose VMs were deleted underneath them, in simultaneous batches (3 at 19:18:40, 3 at 19:20:40 — the multi-VM down-PATCH signature). The deployed controller's normalization branch woodchipped its own busy runners mid-job.
• The gap is conserved. Every capacity PATCH (up or down) preserves the nominal-over-actual gap; per-VM deletes grow it; it clears only at the zero-crossing, where a down-PATCH has no members left to take. Normalization can therefore never catch the drift it chases — it just deletes runners forever.

The original normalization/reclaim branches were destructive and are gone. What replaced them:

What this does (reworked)

At most one capacity mutation per settled pass, chosen by Get-FleetCapacityStep:

• Mutate only from an ARM terminal state (Succeeded/Failed/Canceled); unknown/transitional states and negative telemetry skip the pass.
• Shortfall → compensated scale-out: PATCH to min(35, nominal + (target − actual)), which creates exactly target − actual instances regardless of drift, in one fire-and-forget mutation. If the fleet is empty and the step would clip at the 35 ceiling (creating fewer than target, then pinning there), the reclaim to zero wins the pass instead and the next pass creates the full target from a clean nominal.
• The only down-step is the reclaim to zero, on a settled pass with no members — it deletes nothing and repays the entire drift at once. A drifted nominal above an occupied fleet stays untouched, costing at most gap-many slots of ceiling headroom until the fleet next empties (which it does nightly under the 20-minute warm-down).
• The fleet inventory is listed after the settled-state read, so a down-PATCH can never act on a list that predates a just-completed scale-out.
• Fail closed on garbled telemetry: a capacity read without a usable integral sku.capacity throws TransientFleetException instead of coercing to a false nominal-0 (which would turn compensation into a down-PATCH from Azure's real figure) or crashing at the [int] cast. List payloads must carry their arrays (value, runners) as actual arrays — any other shape throws rather than flattening into a false empty inventory.
• The zero reclaim needs two witnesses: GitHub must show zero online runners (a runner cannot be online without a live VM behind it) and a confirming second inventory read must come back empty; one recanting witness skips the PATCH for the pass. A null inventory read fails the pass outright.
• Nominal/actual arrive from ARM unvalidated (the janitor runbook treats capacity above the ceiling as a real state); only the controller-chosen target keeps its range gate. A runaway nominal no longer crash-loops — it rides along and unwinds through the zero-crossing.

Adversarial review status

#	Finding	Status
1	Scale-in count semantics on a drifted Flex VMSS unverified	ANSWERED — destructive (production telemetry above); branches reworked here
2	Stale-low VM list could drive a down-PATCH below real membership	Fixed (read-order + GitHub corroboration)
3	ValidateRange(0,35) on ARM-derived inputs → uncaught crash-loop	Fixed here
4	Nominal ratchets to the 35 ceiling under sustained churn	Accepted: unwinds at the nightly zero-crossing; worst case is gap-many slots of headroom
5	Persistent allocation failure ratchets silently with no distinct alarm	Follow-up
6	Legacy CLI (reconcile-runner-fleet.ps1) is a second, unserialized capacity writer if run manually	Follow-up / ops note
7	Unit-test gaps	Fixed (126 tests green; destructive-move tests flipped to the proven semantics)

Deploy status

Already deployed (operator ran deploy-controller.ps1 from this branch, 2026-08-08). Verified in production since: the fleet sits drift-free at capacity 4 with exactly 4 VMs, and a full 10-lane CI run completed with every runner surviving — the sawtooth is gone. Merging (squash, per repo convention) aligns development with what is already running.

🤖 Generated with Claude Code

[potatoqualitee]

damn, bout time