Agent lessons — sharpening the axe

Lessons from wootc's boot chain and E2E harness. Six defects in the harness hid the causes of real failures in one session. About half the failures came from our own changes.

Read this before you change the E2E harness, deployer, or runners. Most entries cost at least one run of 60–90 minutes.

1. The dominant bug class

A proxy can produce a status without the required observation. Every serious defect in this session followed that pattern. Ask what a check would report if the event it asserts never occurred.

These checks all shipped with defects:

Check	Passed when	Consequence
`qga_read marker >/dev/null`	a stale marker from a previous run existed	harness monitored a deployer that was never staged
Phase-2 boot checks	advisory only	`ALL TESTS PASSED` for a boot that never happened
`compose_up_windows` (return ignored)	always	15-minute wait, then "QEMU did not start" — wrong cause entirely
`snapshot_serial` failure swallowed	`podman cp` failed silently	harness read a frozen serial file for 2 hours
`[ -f "$PTY" ]`	a previous run's file existed	a run analysed another run's serial
counter `ELAPSED`	never	timeouts 1.5× nominal; progress lines off by 30 minutes

Rule: a check must fail when the required event is absent. Break the code and confirm that the test fails.

2. Liveness: what lies and what doesn't

These signals were unreliable in this session:

pgrep -f "run-e2e.sh" over ssh matches your own ssh command. Its command line contains the pattern, so it can report a run that does not exist. Filter your process with pgrep -f ... | grep -v $$, or inspect the process tree.
systemctl --user is-active cannot report a run that you launched with nohup outside that unit. Check how you launched the run.
Log tail can mix runs when StandardOutput=file: truncates a file but StandardError=append: preserves old errors. Use append for both streams. Locate the current run by its Run ID line.

Use these observations to diagnose the current run:

log mtime (stat -c %Y) — did the log change?
guest CPU (podman exec <container> ps -eo pcpu,args) — does QEMU consume CPU in the current run?
process tree (systemctl --user status <unit>) — which children does this unit own? A sleep 3 child can explain a pause.

These observations do not prove the identity of a guest, its installation, or a successful boot. Each claim needs a successful command and current facts.

3. Serial silence is not death

bootc install can produce no serial output for 10+ minutes as it extracts layers. The time since the last output cannot prove a failure.

In the recorded runs, useful work continued without output at high CPU (130–170%). An idle CPU with no output was a useful clue. A deploy that appeared stuck for 13 minutes succeeded. A similar run did not. Check CPU as a diagnostic (#40), then inspect the guest for the cause.

A sample of CPU use cannot identify a prompt in Setup or prove a product verdict. If guest identity remains unknown, record an infrastructure failure.

4. Timeouts must be wall-clock, and bounded at the blocking call

Two defects affected the timeouts.

Counters drift. ELAPSED=$((ELAPSED+5)) beside sleep 5 does not measure elapsed time. QGA probes and snapshots consume time that the counter omits. The measured rate was 0.68× wall clock: a wait of 45 minutes took about 66 minutes. Use deadline_in and past_deadline.

The defect recurred on 2026-07-22 in wootc-attach-loop.sh. A nominal 60-second wait for host NTFS added 3 seconds per iteration. udevadm settle returned at once on an empty queue. The budget expired in about 2 wall-seconds, before virtio-scsi discovered the disk. Phase 2 reached the emergency shell.

The timestamps in kernel messages showed the error: the hook entered at 1.07 seconds and exited at 3.37 seconds. In an initramfs, use a /proc/uptime deadline and an unconditional sleep per iteration. Never let a probe's status control that sleep. The raw-loopback.bats check is "wall-clock with an unconditional sleep".

A deadline cannot stop a call that never returns. qga_call had no timeout. A stuck podman exec stopped the loop before its next deadline check. Set a timeout on every external call that can wait.

An accurate clock can reveal another defect: a budget that was never enough for the elapsed time. Recheck the budget after you correct the clock.

5. Removing something can expose what it was hiding

The snapshot before the deployer spent 10–20 minutes on an fsfreeze and a 28 GiB copy. That delay gave time for Windows OEM setup to stage BootNext. When we removed the snapshot (see §7), a barrier could pass on a stale marker before setup finished.

Both changes were necessary. When you remove a slow step, check which races its delay concealed. Check a current observation at the barrier; another sleep cannot prove the event occurred.

6. Testing traps in this repo

/tmp is noexec on the dev box. PATH stubs in BATS_TEST_TMPDIR cannot execute there. A test can then report that no command ran because the stub itself failed. setup() must create and execute a stub; use $HOME/.cache if that fails. See go-native.bats and pick-blank-disk.bats.
Case-sensitive guards cover only their pattern. A check for $ELAPSED missed three loops that used lowercase $elapsed.
A test can match its own documentation. A check for need at least matched a comment about the guard, not the code. It then reported a defect that did not exist.
Test the counterexample. Break the code, confirm failure, then restore it. Do this for every test that protects a safety requirement.

7. Runner operations

Never podman system prune -af on a host with a live run. It killed three runs at once and deleted the local image wootc-e2e-windows-ssh:latest. Compose then tried to pull from a registry named localhost.
loginctl enable-linger <user> is required. Without it, systemd killed runs when the ssh session closed.
Launch with systemd-run --user. Pass XDG_RUNTIME_DIR and HOME explicitly. Otherwise rootless podman can use storage paths for root and fail with permission denied on /run/containers/storage.
Check for a live run before any cleanup. Keep its storage, snapshots, and evidence. Remove only disposable files that you own.
Prefer runners hosted by GitHub (e2e-hosted.yml, ubuntu-latest with /dev/kvm). The laptops each failed for a different reason; see the table below.

8. Domain knowledge worth keeping

dracut/printk logging. Neither info() nor warn() is reliable in an initramfs:

info() writes <30> (KERN_INFO) and only echoes to stderr when DRACUT_QUIET != yes — which check_quiet() defaults to yes.
warn() writes <28> (level 4), but quiet sets console_loglevel=4 and printk prints only levels strictly below it — so warn is dropped too.

Use <27> (KERN_ERR, level 3) to kmsg. Do not also write to /dev/console in high-volume paths — deploy.sh emitting every line three times saturated a 115200-baud serial and stalled every deploy. Low-volume boot hooks may.

Windows VSS freeze limits. guest-fsfreeze-freeze on Windows goes through VSS, which enforces ~10s for writers and ~60s overall. A freeze held across a 20-minute copy is not honoured — the guest auto-thaws mid-copy, so the "crash consistent" snapshot is not, and the volume can be left dirty.

NTFS dirty bit. ntfs3 refuses a dirty volume even read-only. A volume formatted by Windows and rebooted immediately is dirty. Mount fallbacks: ntfs3 → ntfs3 -o ro,force → ntfs-3g. This was the BitLocker bug (#36).

Cross-image binaries need their whole closure. The target bootc image ships no qemu-nbd, so the deployer stages its own — but the deployer is Fedora-based and the initramfs is assembled from the target's libraries. Measured skew: libfuse3.so.4 vs .so.3 (a soname major bump). The binary lands and dies at runtime, failing exactly like a missing one.

Ship the full closure: binary + every NEEDED lib + the loader, invoked as ld.so --library-path <dir> <binary>.
Never symlink .so.4 onto .so.3 — a soname major bump is an ABI break, and this driver writes the root filesystem.
Never match the deployer base to the target image — wootc supports arbitrary bootc images, so target library versions are unknowable.
ldd reports only the first missing library. Test by actually running the binary in the target image.

Container probes report the host kernel. /proc/filesystems inside podman run shows the host's kernel, not the image's. It says nothing about whether the image's kernel supports ntfs3. This produced one wrong root cause already.

9. Diagnosis discipline

Wrong root causes reached confidently in one session: "EL10 lacks NTFS support", "multi-disk layout breaks Phase 2", "the deploy wait has no timeout", "netavark errors are killing the deploy", "the QGA MSI was re-downloaded with hardened defaults". Each felt convincing.

What actually worked:

Reproduce outside the VM when possible. The qemu-nbd library mismatch was diagnosed and its fix verified in a container in minutes, versus a 90-minute run. Always ask: can this be tested without booting?
Check the discriminator before asserting. Guest CPU for silence. Marker contents for barriers. ldd inside the target image, not the host.
n=1 is not evidence. The multi-disk hypothesis came from one run per arm.
Compare against a known-good run before blaming something new. The netavark errors looked damning until they turned up in the morning's working runs too.
State what is established vs claimed. docs/phase2-debug-plan.md keeps these separate deliberately.

10. Process

One deployer/initramfs change per run. Every change costs 60–90 minutes; batching makes attribution guesswork. Three times a "fix" was applied to something that was not broken while introducing something that was.
A fix for an observability gap is not free. The /dev/console change was meant only to make failures visible and it stalled every deploy.
Put the reasoning in the test, not just the commit. Tests here carry the failure they prevent, so the next person cannot "simplify" it away.
Retention matters. Each run writes ~3 GiB of artifacts and needs ~45 GiB resident. Without pruning, a successful run breaks the next one.

11. Current known-bad hosts

Host	Issue
runner-a	podman resolves root storage under `systemd-run --user` (#41); runs die without a `[FAIL]` line
runner-b	container QEMU lost KVM after `podman system migrate` (#42); 238 GiB disk is undersized for a ~45 GiB/run workload
runner-c	healthiest of the three; 952 GiB

Prefer e2e-hosted.yml on ubuntu-latest over all of them.

Part 2 — the Phase-2 hunt (2026-07-19)

The first half of this document was written mid-session. What follows is what the rest of the day taught, including six regressions I introduced myself. Read §12 first; it is the one that would have saved the most time.

12. Get inside a live box. Do not wait for the run to finish.

The single highest-value technique of the entire session, and it came from the user telling me to stop waiting.

Four consecutive 90-minute runs failed without producing an attributable cause. Then one command answered it:

podman exec <container> python3 /tmp/qga.py powershell '$env:OS'
# -> Windows_NT   ... while the harness was 61 minutes into "Deploying..."

The guest was not running the deployer at all. From there, two minutes of live inspection found what four runs had not:

# which OS is actually running?
qga.py exec /bin/sh -c "uname -sr"
# what is the process actually doing?
qga.py exec /bin/sh -c "ps -eo pid,ppid,stat,wchan:20,args"
# what did it last say? (the deployer's own journal, live)
qga.py exec /bin/sh -c "journalctl --no-pager | grep -a wootc | tail -20"

wchan is the key column: do_wait means blocked on a child, hrtimer_nanosleep means a sleep, anon_pipe_read means blocked on a pipe.

Post-hoc artifacts repeatedly failed where live inspection worked: the deployer log could not be read because the guest was in an emergency shell, and the journal artifact came back at 111 bytes. If a VM is hung, go in NOW — the evidence disappears when the run cleans up.

The deployer initramfs ships qemu-ga, so this works during Phase 1 too, not just once Windows is back.

13. Phase 2 was never failing. It was never reachable.

Worth internalising as a class of mistake, not just a fact.

For most of the session I debugged Phase 2 as though it were broken: the attach hook, the qemu-nbd closure, the NTFS driver, hypotheses A1–A6. All of it was reasoning about components that were never installed, because the deploy died before staging them.

The proof took one manual boot: force the BCD one-shot by hand and look at the GRUB menu.

*Install wootc (automatic)
 Install wootc (debug)

No entry for the installed system — while root.vhdx held a complete 6.6 GB ostree deployment. The OS was installed and unreachable.

Lesson: before debugging why a stage fails, verify the stage can be entered at all. One QGA call would have shown this on day one.

14. A watchdog you have to signal is a bug factory

Three designs, two of which I made worse:

( sleep 2700; force_reboot ) & — never cancelled. dracut-initqueue blocked in wait() for the full 45 minutes after the deployer returned, so Phase-2 setup never ran and the exit status was never printed.
Added kill "$pid"; wait "$pid". When the kill misses, that wait blocks forever — a permanent hang replacing a 45-minute one.
Wrapped the sleep in setsid so it could not outlive its subshell. That put it in its own session, beyond the reach of both the pid kill and the process-group kill. Strictly worse. Confirmed live:
```
453  1    S   do_wait            /usr/bin/sh /usr/bin/dracut-initqueue
455  453  Ss  hrtimer_nanosleep  sleep 2700     <- Ss = session leader
```

The working design signals nothing: the watchdog polls a flag file, cancelling is : > /run/wootc-deploy-done, and the loop exits within one tick by itself.

Generalisation: in a shell, prefer a background task that observes a condition and exits over one you must find and kill. wait on a pid you do not control is an unbounded block; kill is unreliable the moment process groups or sessions are involved.

15. Things that match themselves

Three variants bit in one day. All produce confident, wrong answers.

pgrep -f "run-e2e.sh" over ssh matches the ssh command running it. Reported dead runs as alive.
Polling journalctl | grep verify: logs your own command, which then matches the grep. The output was entirely my own polling.
A test grepping for a string that appears in its own comment. Happened three times: a preflight guard matched the comment quoting the error, and a setsid removal test matched the comment explaining the removal.

Rule: when grepping for a pattern, exclude the searcher. grep -v qemu-ga, grep -v $$, grep -nE '^[^#]*pattern'.

16. My regression rate, and its cause

Six regressions introduced while fixing things, in one session:

#	Change	Damage
1	`podman system prune -af` on live hosts	killed 3 runs; deleted the built ssh image
2	triple `/dev/console` logging	saturated serial, stalled every deploy
3	folding kernel reboot into DEPLOYER_REBOOT_SEEN	false `[PASS]` on a dead deploy
4	`kill` + `wait` in cancel_watchdog	permanent hang
5	`setsid` on the watchdog sleep	made it uncancellable
6	RunId barrier without refreshing C:\OEM	mutual deadlock, guest timed out

The cause is single and structural: I changed code faster than a 20–90 minute feedback loop could validate it. Every one of these looked correct when written and was wrong in an interaction I could not test locally.

Mitigations that actually work, in order of value:

Reproduce in a container first. The qemu-nbd closure was diagnosed AND its fix verified in a container in minutes.
One deployer change per run. Repeatedly violated under pressure to show progress; every violation cost more than it saved.
Round-robin the runners (§17) so a fix is validated sooner.
Prefer designs that cannot fail the same way — the flag-file watchdog over any amount of careful signalling.

17. Round-robin the fleet

With three runners staggered by ~15 minutes, a fix is validated against whichever reaches the interesting stage first, instead of waiting a whole cycle. It also supplies what #34 needs (a pass RATE at one commit) and covers "multiple cases".

Launch notes per host are in AGENTS.md. runner-a needs nohup rather than systemd-run (#41); all hosts need loginctl enable-linger.

18. Reuse the helper that already exists

I wrote an ad-hoc loop to push files into the guest, using $OEM_DIR — a HOST path — while qga.py runs inside the container and sees that mount at /oem. Every write failed. qga_sync_oem() already existed forty lines away, did it correctly, and was merely gated on --skip-install.

Before writing a helper, grep for one. Before passing a path to something that runs in a container, ask whose filesystem that path is on.

19. Guest/host state must be refreshed, not assumed

C:\OEM is populated from the ISO at Windows install time. Any guest whose Windows was installed by an earlier run carries THAT run's scripts. Introducing a protocol change (the RunId barrier) without refreshing the guest deadlocked both sides: the guest stamped an old constant, the host never matched, and each waited on the other until the guest's 10-minute deadline expired.

If you change a host/guest protocol, push the guest half of it in the same change.

20. A retried command is a second command

qga_call retries three times to survive flaky agents. Sent through a boot transition, the retry is poison: a systemctl reboot the dying guest never consumed sat in the virtio-serial channel until the NEXT agent opened it — the freshly booted native install, which obeyed and rebooted straight back to Windows (BootNext one-shot, already consumed). Earlier runs "survived" only because guest-exec was still blacklisted natively; fixing the blacklist armed the trap.

Side-effecting commands must never ride a retried channel across a boot boundary. Use the QEMU monitor (system_reset) — it queues nothing anywhere. And remember any single reboot of the native system lands in Windows: BootNext fires once, BootOrder still lists Windows first.

21. A service that can succeed while doing nothing is unobservable

wootc-mount-user-dirs exited 0 with zero binds and an empty journal — each profile was skipped by a silent continue. The condition it skipped on ([[ -d $home ]]) was itself a deployment bug: fisherman's useradd --create-home followed the deployment's /home -> var/home symlink into the deployment's OWN var, which the stateroot var mount masks at runtime. Two rules: every skip on an "impossible" condition logs an ERROR naming the bug, and every unit that can no-op logs a final summary line ("N binds across M users") so the journal alone convicts or acquits it.

22. Forensics beat theorizing: the disks were all still there

The failed run's whole story was recoverable offline: qemu-nbd the graduated qcow2 → native journal showed the exact reboot command and its source pid; loop-mount root.disk inside the NTFS qcow2 → the orphaned home in the deployment's masked var. Two dead ends to skip next time: a qemu-nbd --connect dies with its ssh session (do all reads in ONE session, or it reads as a corrupt/empty GPT with fresh random GUIDs per call), and the serial pty spans every boot of the run — sed from the FIRST "GRUB version" banner lands in Phase-1, not the boot you care about (anchor on BdsDxe: starting BootNNNN line numbers instead).

23. `--root` is not chroot

useradd --root <dir> chroots — after initializing the HOST's PAM/SELinux stack. From the deployer initramfs it worked for months; from booted Phase 2 it failed with "failure while writing changes to /etc/passwd" against an etc that append/touch proved writable. chroot <dir> useradd succeeded on the same files, same second. Offline user tooling must run the TARGET's binaries via plain chroot; the flag that looks equivalent is environment-sensitive in exactly the way an initramfs test bed cannot reveal.

The debugging move that cracked it in minutes: while the failed Phase 2 was still running, reproduce the exact failing command by hand over QGA, then bisect the variants (--root vs chroot vs --prefix) live. One boot's evidence beats three relaunch cycles.

24. A verdict is only trustworthy if the run actually ended

run-matrix.sh polled for any [FAIL] line and concluded on the first one. But several run-e2e checks are invoked as ... || true, so a non-fatal failure line looks identical in the log to a fatal one. A cosmetic seed_user_data failure therefore ended the poll ~35 minutes early, recorded itself as the case's verdict, and sent me hunting a "composefs" bug that was neither composefs nor the reason the case died — while the run was still deploying.

The rule: conclude on a terminal state, never on the presence of a scary line. run-e2e.sh stamps stage=exited from an EXIT trap, so an ending is always observable; the last [FAIL] then supplies the reason — it does not supply the fact of failure. And delete the previous case's state file with its log, or a stale stamp declares the new case finished the moment it starts.

This is lesson §21's sibling: there, a service could succeed while doing nothing; here, an observer could report a failure that had not happened yet.

25. The guest chooses the drive letter, not you

On the BitLocker axis setup-wootc.ps1 carves an unencrypted volume and puts the whole tree on it — E:\wootc\{disks,install,logs}. The harness hardcoded C:\wootc\... in four places. So el10-gnome-win11pro-bitlocker deployed successfully, passed both FDE assertions (C: still encrypted, Linux on unencrypted E:), and then died 39 minutes in on "Could not read wootc BCD GUID from Windows" — reading a file from a drive it was never written to. The VDL extension had already silently skipped for the same reason, reporting only size='0' at INFO level.

Two habits fall out of this:

When the guest-side script computes a location ($storageRoot), the host-side must ask for it, not re-derive it. guest_wootc_root() queries the guest and caches only a positive answer.
info "Could not read X (size='0'), continuing" is a bug report wearing a progress message. A read that returns nothing where something must exist is never a continue-quietly condition.

26. `podman build` blames your Containerfile for the runtime's crash

A GUI E2E died at 6 minutes with what read as a broken package list:

Error: building at STEP "RUN dnf install -y  systemd systemd-udev ...": while running runtime: exit status 1
[FAIL] Deployer build failed

Nothing was wrong with the packages. Two lines up, unhighlighted, was the real event: error running container: from /usr/bin/crun creating container / did not get container create message from subprocess: EOF. The container was never created, so dnf never ran — but the error names the RUN step, so the instinct is to go read the package list.

The cause was a hosted runner-image bump: podman 4.9.3 → 5.8.4 with /usr/bin/crun left at 1.14.1. Podman 5 emits an OCI spec version crun 1.14 rejects (crun: unknown version specified). Note the good error text exists — it appears verbatim in podman run, and it was podman build that buried it.

Three habits:

Check the blast radius before diagnosing. The same push turned the container test suite red, and that log printed the unswallowed crun: unknown version specified. A failure appearing in two unrelated workflows is environmental; one that appears in only one is yours. This is what turned a wrong guess (AppArmor userns restrictions — the other cause of EOF at create) into the actual answer in one step.
crun --version does not tell you what podman runs. Hosted runners carry more than one crun; $PATH prefers /usr/local/bin while podman prefers /usr/bin. A bare crun --version reported 1.28 before and after installing 1.28, which nearly credited the fix to an unrelated image patch. Ask the engine: podman info --format '{{.Host.OCIRuntime.Path}} {{.Host.OCIRuntime.Version}}'. For the same reason .github/actions/podman-runtime installs over /usr/bin/crun — a /usr/local/bin copy is on $PATH but invisible to podman.
The hosted pool is mixed. Jobs in the same run landed on podman 4.9.3 and 5.8.4, so an environment repair must be unconditional and then verified by a real podman run, not gated on a version comparison. Version strings are a proxy; a container that starts is the fact.

27. Rebooting is not releasing. Phase 2 owed Windows a clean unmount.

Both fedora-gnome cells of run 30704513401 failed identically — so, not a flake — with [FAIL] QGA did not become available for Windows return after Phase 2 Linux within 10 minutes. Phase 2 itself was flawless: it booted, passed every passthrough and user-data check, and rebooted on cue. The damage was done on the way out, and it was visible in five lines of serial nobody had been reading:

(sd-remount)[...]: Failed to remount '/run/initramfs/wootc-host' read-only: Device or resource busy
systemd-shutdown[1]: Not all file systems unmounted, 1 left.
systemd-shutdown[1]: Not all loop devices detached, 1 left.
systemd-shutdown[1]: Cannot finalize remaining file systems, loop devices, continuing.
reboot: machine restart

Windows then showed a desktop for ~26 s, rebooted itself, looped the boot manager three or four times and died in Startup Repair. The QGA timeout was a symptom four reboots downstream of the actual bug.

A boot stack that lives on the thing it must release cannot release it. Phase-2 / is a loop device backed by root.disk, a file on the rw NTFS mount. Nothing running inside that stack can unmount the volume. The deployer already knew this — deploy.sh has a whole block ending "a still-mounted rw NTFS would be flagged dirty" — and Phase 2 simply never got the same treatment. When one side of a symmetric operation has a hard-won teardown, go look for the other side.
A missing file can disable an entire subsystem in total silence. dracut-shutdown.service populates /run/initramfs from /boot/initramfs-$(uname -r).img. On ostree/composefs that path does not exist, dracut-initramfs-restore no-ops, and systemd — which only pivots when /run/initramfs/shutdown is executable — quietly skips the whole shutdown-initramfs mechanism. No error, no warning, and the absence of "Returning to initrd..." in the log is the only tell.
Corruption sorts by driver, and the passing cells were the clue. Every cell mounting the host volume with kernel ntfs3 failed; every cell that fell back to the ntfs-3g FUSE driver passed. That correlation is what turned "flaky Fedora cell" into "structural teardown bug" — a green cell is evidence too, and diffing it against the red one costs minutes.
Make the dangerous new path revert to the old one. The fix arms a shutdown pivot that did not previously happen, on every cell's boot path, and cannot be tested outside a 60–90 minute VM run. So every failure path in the staging hook ends by deleting /run/initramfs/shutdown — without it systemd does not pivot, which is exactly today's behaviour. A partial or broken staging can only reproduce the bug it is fixing; it cannot invent a new one.

28. The initramfs is not your laptop, and `[FAIL]` is not a log level

bluefin-dakota-win11pro (run 30707067821) died 11 minutes into the deploy on two lines:

[wootc] ABORT: line 1281: awk '{for(i=1;i<=NF;i++) if ($i ~ /^\//) print $i}' (exit 127)
[FAIL] qga: ldd on the deployer's qemu-ga surfaced no dynamic loader

A dracut initramfs contains exactly what module-setup.sh names. Every closure builder in deploy.sh resolved libraries with ldd "$bin" | awk …. ldd is a glibc-common shell script; no library dependency drags it in, nothing listed it, so it was never in the image. Reach for the thing whose presence is structurally guaranteed instead: the dynamic loader has to be there or the deployer itself could not run, and $ldso --list is what ldd execs anyway. dso_closure() now does that in pure bash, so no missing text tool can break a closure either.
Exit 127 names the wrong command. Under set -o pipefail the ERR trap reported the awk stage of a pipeline whose first stage was the missing one. When a trap blames a command that obviously exists, suspect the rest of the pipeline before you suspect the trap.
[FAIL] on the deployer serial is an API, not a severity. run-e2e.sh greps the serial for fatal|panic|[FAIL] and ends the deploy on the first hit. The qga stager printed [FAIL] for a condition its own caller absorbs with [WARN] no fallback qemu-ga staged — so a survivable best-effort miss killed a cell that was otherwise fine. A function may only shout [FAIL] if every one of its callers treats the failure as fatal.
The same rule bites the harness. run-e2e.sh also printed [FAIL] run-e2e.sh aborted: awk … on every hosted cell, because -E propagates the ERR trap into the command substitution probing for a tailscale0 that hosted runners do not have. Nothing aborted. The matrix takes the last [FAIL] as the verdict, so an optional probe was one silent timeout away from becoming a run's official cause of death.

29. The harness destroyed the run it had already passed

fedora-gnome-win11pro-btrfs (run 30710282779) failed on:

[FAIL] Windows QGA did not become available within 10 minutes

It had already succeeded. Every Linux assertion passed, §27's shutdown pivot worked (wootc: shutdown: unmounted the Windows host NTFS at /oldsys/run/initramfs/wootc-host), and the serial shows the return happening two seconds after the reboot request:

[   75.9] reboot: machine restart
BdsDxe: starting Boot0003 "Windows Boot Manager" ...

Then the harness power-cut it. The two boot-manager loads after that one are its own system_reset and the loop that followed; the final screenshot is a recovery screen.

A ping is not an identity. The reboot step watched for qga_probe to fail and reset the VM if it never did. qga_probe is guest-ping, which answers for whichever agent is up — so "Phase 2 has not rebooted" and "Windows is already back" produced the identical reading, and the fallback for the first is a hard power cut to the second. §1 again, in its purest form: the observable is which OS answers, and the code only ever sampled whether one did. qga_windows_probe had existed for months, six lines above; qga_wait_down already used it. This loop just didn't.
A destructive fallback needs positive evidence, not the absence of evidence. p2_reboot_observe() now returns one of down / windows / linux / unknown, and only linux — a guest that answered uname -s — may be reset. unknown (a Windows agent pinging before PowerShell is up) deliberately does nothing: the cost of waiting is one honest timeout, the cost of guessing wrong is the whole run.
Blind time is where the state changes underneath you. The request went out through an unbounded qga_call exec, which polls guest-exec-status until exit and then retries three times at 60s. Against a guest that is busy dying that is ~110 seconds of not looking — long enough for Phase 2 to leave and Windows to come back, so the observation opened its eyes on the wrong OS. §4 says bound the blocking call; the corollary is that anything you do not watch is free to become its own opposite while you wait.
This is why §27 read as a partial fix. That session saw Windows "show a desktop for ~26 s, reboot, and die in Startup Repair" and attributed the reboot to the dirty volume. The unclean unmount was real and worth fixing, but the reboot at ~26 s was this bug: the harness resetting a Windows that had come back fine. When a fix lands and the symptom shifts but survives, check whether you are now looking at a second cause rather than a an incomplete first one.

30. The product refused, and it was right to

el10-gnome-win10pro, the first GUI-driven Win10 cell (run 30717631172), failed with the app's own words:

[FAIL] the GUI REFUSES this configuration — Install is disabled, so no install can be driven
[FAIL]   the app says: Windows has an update waiting to finish (servicing,windows-update).
         Restart the PC, let it complete, then run wootc again

Nothing was broken. pendingReboot() read the two registry keys it is meant to read, both were set, and the preflight declined a machine that is mid-servicing — exactly the gate #63 exists for.

A red cell is not automatically a bug in the thing under test. The harness had handed the product a machine in a state no user is supposed to install from: Win10 media stage servicing work that outlives OOBE, and the harness's own WebView2 bootstrapper (installed seconds earlier, because wails cannot render without it) adds an update of its own. The failing component was the setup, and the fix is to provision the guest into a state a real user would be in — not to loosen a gate that protects boot configuration during a migration.
When the product tells the user what to do, the harness should do it. gui_settle_pending_servicing() reads the same two keys the app reads, restarts the guest when either is set, and re-reads them. If a restart does not clear them, it says so and lets the app's refusal stand as the verdict: the harness must never assert a machine is ready, only observe that it is.
Coming back is not being logged on. QGA answers from session 0 long before autologon finishes, and the GUI is launched via schtasks /IT, which needs a real interactive session or fails with "the system cannot find the file specified". §1 again: ping is a proxy, the logged-on user is the observable, so the restart waits for Win32_ComputerSystem.UserName.

31. The first sample of a just-started process is not its argv

el10-gnome-win11pro-bitlocker (run 31076749824) died three seconds after the container came up:

[INFO] Container wootc-e2e-windows started
[FAIL] QEMU is not using KVM acceleration

The diagnostics dump 300 ms later printed the very same PID with accel=kvm -enable-kvm on it, at 0 s of CPU time. Nothing was misconfigured: the harness read ps -ef while QEMU was still inside execve, where /proc/<pid>/cmdline is only partially written, and judged a command line that had been truncated mid-argument.

A racy read is a proxy (§1). The observable is the settled argv, not the first one that happens to be readable. -enable-kvm, -tpmdev emulator, property=secure,value=on and /storage2/data2.qcow2 all sit late on a ~2 KB command line, so a short read fails every one of those assertions and reports the most alarming of them as the verdict.
Poll until the assertion is satisfiable, then fail on a deadline. The check re-samples for up to 60 s and prints the settled command line with the failure, so a genuine "dockur disabled KVM" still fails, and fails legibly.
The generic form: when polling for a process, the appearance of the process and the readability of its state are two different events. Waiting for the first tells you nothing about the second.

32. `schtasks /Run` does not consume the trigger you created

bluefin-dakota-win11pro and fedora-gnome-win11pro (run 31081727936) both died at 65% with the product's own foreign-ESP refusal:

    esp-sweep: before: clean
    esp-sweep: after: clean
...
Setting up ESP: this PC's EFI boot partition already contains
EFI\fedora\shimx64.efi, which wootc did not put there

One cell named shimx64.efi, the other grubx64.efi — two different members of the four files setupSignedChain copies, which is what reading a set mid-write looks like, not what leftover state looks like. (reset_oem_attempt reads the same run differently — that before: clean was itself an unreliable print, because a drive letter assigned mid-process is not visible to that process's FileSystem provider. Both can be true; the argument below rests on neither.)

gui_install_arm launched the app with

schtasks /Create /TN wootc-gui-e2e /SC ONCE /ST <now+1m> ... ; schtasks /Run ...

/Run starts the task immediately and leaves the ONCE trigger armed, so Task Scheduler started launch-gui.cmd again inside the minute. Nothing makes wootc.exe single-instance, both apps polled the same e2e-drive.json, and both clicked Install. The install mutex in StartInstall is per-process, so two pipelines raced onto one ESP — and because recordESPOwnership runs only after all four copies land, whichever pipeline reached the guard during the other's copy loop saw an unattributable vendor binary and refused, correctly.

A manual start and a schedule are two launches. /Run is not "run instead of the trigger"; it is "run as well as".
The bug was in the launch, not in the logs. Two schtasks calls are two launches whatever the run printed, so this needed no verdict to be trusted — and every GUI cell to date had been running two installers.
Which file it blamed was the tell. A random member of a set that one code path writes in Go map order is being read mid-write.
The harness now drops the task once the app is up and then counts wootc.exe before writing the directive — no instance can install until the directive exists, so an extra killed at that point provably never installed.
The drop was a silent no-op for a month (2026-10). It was written cmd.exe /d /c "schtasks.exe /Delete /TN \"wootc-gui-e2e\" /F >NUL 2>&1". PowerShell has no backslash escape, so \" ended the string; cmd.exe got two arguments, schtasks tried to delete a task named ", and >NUL hid the error. The count still caught the extra while launch confirmation was slower than the trigger's minute. When the ready marker (#364) made confirmation fast, the count ran first, the trigger fired after it, and the second app's pipeline died on the first one's file: Making room for Linux: root.disk already exists on every image (runs 36450824541, 38735109866). The ESP at failure time held a complete deployer chain from the other pipeline — the failing one had stopped before it ever wrote there. The delete is now verified from a successful task listing, and the directive is refused without both "task absent" and "one wootc.exe" (tests/unit/test_gui_launch_task.py, which also parses the line with pwsh when present). A fix that hides its own failure behind >NUL is a proxy for itself (§1).

33. A deaf channel has no opinion about the product

el10-gnome-win11pro (run 32556250889) reported "install stalled at Finding your files" after waiting out its full 30-minute drive deadline. Printed underneath that verdict, in the same block, was the fact that invalidated it:

[FAIL] GUI-driven install did not reach the done screen in 30m
[FAIL]   QGA does NOT answer ping — the verdict above may be a lost channel

The QGA virtio-serial channel had gone deaf. A stalled installer and a dead channel produce the same observable — the drive-state file stops changing — and opposite causes, so the run had no standing to say anything about the install. It said it anyway, thirty minutes late, and a human had to read the log to find out the red was not real.

Ask the discriminator first, not last. guest-ping decides which verdict is even available, so it runs before anything is written down. Appending the channel state under a product verdict is the wrong order: the caveat does not retract the claim, it just sits below it.
Waiting longer cannot make a deaf socket talk. The remaining ~28 minutes bought nothing except the appearance of diligence. Once ping is dead, one bounded reconnect cycle — reap the clients that may still hold the single-client socket, drain what the killed one left queued (§20), reopen with a delimited sync — and then a verdict either way.
Say what you do not know. The classification is qga-channel-lost and it states that the run has no verdict on the product. The install may have finished, stalled or failed; the harness cannot tell, so it does not guess.
Name the class in the ledger. note_flake writes the machine-readable verdict the retry gate reads, and the ledger line names the same class, so the re-dispatch decision needs no human in it.
Exit 1 from a socket that will not open was the same mistake in miniature. qga.py connected outside its own error handler, so ENOENT on a dead channel exited 1 — indistinguishable from a guest command that ran and returned 1, which qga_call_retry must never replay (§#40). A transport failure has to say so: it exits 42.

Guarded by tests/unit/qga-channel-lost.bats and a synthetic channel kill against a real socket in tests/unit/test_qga_reconnect.py.

34. A cached Windows password ages even when its VM is powered off

The GUI nightly failed in run 36240171646 (2026-09-26, #399). QGA answered, schtasks /Run exited 0, but no interactive user existed. The final screenshot was black. The video showed this Windows prompt: Your password has expired and must be changed. The restored fixture had aged past its account's password lifetime.

Use the video if the final screenshot hides the original screen.
Set PasswordNeverExpires on the local account that the fixture uses for autologon. Apply this policy when you create and restore snapshots. Keep the credentials intact. Do not change the policy for other accounts.
After the repair of an expired fixture, restart once so Windows retries autologon. Then verify that an interactive user exists. The policy write alone does not prove this.
An interactive-session deadline must block GUI launch. The old loop timed out, substituted a username, and scheduled a task that could never run. autologon-no-session is an environment failure. It is not a product verdict or an automatic retry trigger. A second identical snapshot has the same bug.

35. Guest free space does not prove KubeVirt node capacity

On 2026-09-26, storage pressure on the shared node stopped a helper trial inside Windows after 4,710.74 seconds (78.5 minutes). The helper had started to install OS layers. It had not returned a success receipt. Another test pod had already left the node. Free space had fallen below the kubelet threshold of 23,502,050,623 bytes. The experiment's former 8 GiB node guard was too low.

Check the node filesystem availableBytes, or df -B1 on the volume that stores VM disks, before and during a trial. Windows C: free space and a sparse disk's logical size do not measure this shared capacity.
This rig now reserves at least 26 GiB on the node, above the observed eviction threshold with headroom. Recheck the node's threshold before a future run. Other pods can consume the same filesystem.
Keep this infrastructure reserve separate from the application's 8 GiB consumer reserve. A kubelet eviction policy is not a laptop requirement.
If the node crosses the rig reserve, stop only the identified test writer. Do not wait below the reserve for another cleanup job. Preserve the outer Windows fixture and record a storage-pressure stop, not a helper failure.
After the writer exits, archive the evidence, then delete its disposable scratch disk. A file deletion in Windows does not prove that the node has recovered physical space. This trial needed ReTrim inside Windows. A new node measurement showed that available bytes rose from 24,140,267,520 to 34,515,693,568. Confirm free space on the node before you start another trial.

The recorded result retains the empty helper receipt and the harness termination reason. No claim of an installed desktop or a complete installation follows from this run.

FilesExpand file tree

agent-lessons.md

Latest commit

History