Skip to content

AGENTS.md

Latest commit

 

History

History
278 lines (222 loc) · 12.8 KB

AGENTS.md

File metadata and controls

278 lines (222 loc) · 12.8 KB
Raw

wootc — Agent Guidance

Start here

  • docs/status.md#buildtest-matrix — the one current status source (the roadmap itself is ROADMAP.md). Green/red per image family × phase and per axis, from the E2E rig and the hosted matrix; a cell is only green once the hosted E2E passed it end-to-end (see docs/RELEASING.md). Open work is tracked on the org TunaOS GA Roadmap project (Tier 1 = silent-failure class, Tier 2 = user-facing correctness, Tier 3 = harness + process).
  • docs/milestones.md — the verification ladder the matrix rolls up.
  • docs/agent-lessons.md — traps that have each cost a 60–90 minute VM run. Read before touching the E2E harness, the deployer, or the runners.
  • docs/docs-truth-pass.md — the dated record of each docs-vs-build pass (#233): what was checked, what was wrong, and what is still owed to an RC walk. Every ✘ it found is pinned by tests/unit/docs-truth.bats, so change a path, a default image, or a channel gate and that suite tells you which doc you just falsified.
  • docs/phase2-debug-plan.md — historical (2026-07-18): its Phase-2 hypotheses were resolved; the debugging story is in docs/phase2-attach-postmortem.md.

The single most useful heuristic in this codebase: status derived from a proxy rather than an observable is the dominant bug class here. When adding a check, ask what it would print if the thing it asserts never happened — then break the code and confirm the test goes red.

Project layers

The codebase has four distinct layers. Delegate to the matching agent when working in a specific layer.

Layer Agent Key files
Windows OEM wootc.wootc-windows-oem autounattend.xml, setup-wootc.ps1, install.bat
QGA control plane wootc.wootc-qga-control qga.py, QGA socket wiring in compose.yml
Deployer initramfs wootc.wootc-deployer module-setup.sh, deploy.sh, deploy-hook.sh
E2E test runner wootc.wootc-e2e-runner run-e2e.sh, Kanpur infrastructure

All four agents have the wootc-e2e skill loaded, which covers shared knowledge: PowerShell safety rules, QGA primitives, Kanpur quirks, and the debug cycle.

Project status

QGA control plane — Live (commit 377a2ff)

The E2E control plane has been migrated from WinRM to QEMU Guest Agent. The QGA client (tests/e2e/qga.py, 131-line stdlib Python) talks JSON-lines over a virtio-serial Unix socket at /run/shm/qga.sock.

QGA provides:

  • guest-ping for generic liveness (no credentials needed) — but it answers for whichever agent is up, Windows or Linux, so it cannot tell them apart on its own
  • guest-exec for running PowerShell as SYSTEM (Windows) or /bin/sh (deployer / Phase-2 Linux)
  • guest-file-read for reading OEM logs and the deployer journal
  • Reboot detection via guest-ping down/up cycle

Liveness ≠ identity. Before every OS transition the runner asserts identity positively: $env:OS must match Windows_NT (qga_windows_probe) or uname -s must say Linux (qga_linux_probe) in tests/e2e/run-e2e.sh. Relying on guest-ping alone has cost runs by answering for the wrong guest.

See docs/e2e-architecture.md for the current control-plane topology; HANDOFF.md is the historical (2026-07-15) design rationale.

E2E Testing — autounattend.xml v3 Fixed (commit 0779d8d)

The critical fix: autounattend.xml was missing a DiskConfiguration block. Without it, Windows Setup waits forever at "Where do you want to install Windows?" — disk never grows past 1.2MB. v3 adds explicit UEFI GPT partitioning (EFI 100MB + MSR 16MB + Primary), removes EnableLUA=false (breaks Windows 11 boot), and merges WinRM setup into consolidated FirstLogonCommands.

BCD Chainload — Proven Working

bcdedit /copy {bootmgr} (not /create /application firmware) is the correct approach. /application firmware is not a valid bcdedit type on Windows 11. Fixed in setup-wootc.ps1 and app/installer_windows.go.

wubildr.efi — Built and Tested

Custom GRUB core image (1.3MB) with embedded bootstrap config, ntfs + loopback modules. Built via grub2-mkimage inside the deployer container. Stock Fedora grubx64.efi drops to rescue shell — wubildr.efi fixes this.

Secure Boot chainload — shim + signed grub ✅ (commit 8a58274)

wubildr.efi is unsigned, so Secure Boot rejects it with Access Denied on the serial console. The fix is a Microsoft-signed intermediate bootloader:

UEFI → shimx64.efi → grubx64.efi → grub.cfg → deployer

Component Signer Source
shimx64.efi Microsoft Fedora shim-x64 package
grubx64.efi Fedora (in-shim MOK) Fedora grub2-efi-x64 package
grub.cfg N/A (on ESP) same logic as wubildr.cfg

Fedora's signed grubx64.efi does NOT embed ntfs+loopback modules, but the deployer and Phase 2 kernels + initramfs live on the FAT32 ESP — GRUB can natively read FAT32 and never needs NTFS modules. The kernel command line passes loop=/wootc/disks/root.disk wootc.host_uuid=… so the initramfs (not GRUB) mounts the NTFS volume and attaches the loop device. No .mod files are needed on the ESP.

E2E boot chain progress

Each row is a separate reboot from one step to the next:

Step Status Evidence
OEM setup complete ✅ root.disk created, BCD configured, Fast Startup disabled
Boot via wubildr.efi ❌ Access Denied — unsigned binary, Secure Boot blocks it
Boot via shimx64.efi ✅ BdsDxe: starting Boot0005...shimx64.efi — no Access Denied
GRUB loads grub.cfg ✅ GRUB version 2.12 visible on serial console
Deployer boots ✅ Kernel + initramfs loaded from FAT32 ESP (no NTFS needed)
fisherman runs ✅ deploy.sh completes, bootc installs the target OS
Phase 2 boots (shim→grub→kernel) ✅ ESP grub.cfg loads phase2-vmlinuz from FAT32
Linux installed + Windows returns ✅ Full cycle green on el10-gnome-win11pro-bitlocker (E2E runner, 2026-07-28)

Serial logging

The compose.yml overrides Dockur's default SERIAL=mon:stdio with SERIAL=file:/storage/deployer-serial.log so the deployer's console output is written to a persistent file on the mounted /storage volume. This file survives container teardown and single-handedly made the Secure Boot rejection visible.

Snapshot before Phase 2

Always copy storage/data.qcow2 before the first deployer boot. If the deployer fails or corrupts the disk, restore from the snapshot and retry without reinstalling Windows:

cp storage/data.qcow2 storage/data.qcow2.snap
# ... attempt deployer boot, fails ...
cp storage/data.qcow2.snap storage/data.qcow2
# restart VM --skip-install

PowerShell safety rules — Established (commit 09060c4)

Three rules that have burned multiple E2E cycles. Always validate before committing changes to setup-wootc.ps1 or any Windows script.

R1: No trailing backslash in double-quoted strings. PowerShell sees \" as an escaped quote and the string never terminates. The runner has a pre-flight check: grep -n '\\\\"$' setup-wootc.ps1.

R2: Use variable expansion — no -f or + concatenation inside parenthesized expressions. Both -f format strings and + concatenation inside (...) trigger parser confusion with closing parentheses. Use plain variable expansion: Write-Host " root.disk: $diskPath ($DiskSizeGB GB)"

R3: Single-quote here-strings for GRUB config. GRUB config contains $prefix, $root, { etc. Use @'...'@ not @"..."@.

R4: CRLF line endings + UTF-8 BOM for Windows PowerShell 5.1. PowerShell 5.1's Get-Content -Raw and internal script parser corrupt UTF-8 files with LF-only line endings. The E2E runner automatically converts setup-wootc.ps1 before staging it in the OEM payload. Always use printf '\xEF\xBB\xBF' > file.ps1; sed 's/$/\r/' file.ps1 >> file.ps1.crlf when writing PowerShell scripts intended for Windows 10/11 VMs.

Kanpur quirks

The E2E host (kanpur) is Bluefin (Fedora Silverblue, immutable). These quirks are documented in the wootc-e2e skill and the e2e-runner agent:

  • podman-compose is at ~/.local/bin/podman-compose, not on default PATH
  • Podman creates root-owned files in tests/e2e/ — chown -R before re-running
  • Stale rootlessport processes hold port 3389 across runs — kill them
  • Container name is always wootc-e2e-windows

Phase 2 target — green end-to-end

Green since 2026-07-23 (rung 2 of docs/milestones.md): the full cycle passes repeatably via run-e2e.sh on the E2E runner, including the BitLocker cell (el10-gnome-win11pro-bitlocker, full cycle green 2026-07-28), and the same chain is green GUI-driven on bluefin:lts (docs/status.md matrix, pages/e2e/latest). The acceptance criteria below are the definition of that target:

  1. Fresh Windows 11 install under KVM + TPM 2.0 + Secure Boot.
  2. Windows creates root.disk, copies the deployer, installs wubildr.efi, and configures the one-shot BCD entry.
  3. The deployer boots and completes Linux installation.
  4. Windows returns after that one-shot deployer boot.
  5. The test explicitly schedules the installed Phase 2 Linux root, observes a successful Linux boot, then observes a successful Windows return.

Key artifacts

  • CONTEXT.md — Domain glossary (Phases 1-3, root.disk, User Data Bridge)
  • HANDOFF.md — QGA migration design rationale (historical, 2026-07-15; superseded by docs/e2e-architecture.md)
  • docs/adr/0001-phase1-first-architecture.md — Phase 1 VM-first architecture
  • tests/e2e/wootc-files/wubildr.efi — Custom GRUB image (1.3MB)
  • tests/e2e/qga.py — QGA JSON-lines client (stdlib only)
  • AGENTS.md — This file

Running E2E

Read docs/agent-lessons.md first. It records the traps that have each cost at least one 60–90 minute run — liveness signals that lie, timeouts that are not wall-clock, vacuous tests on a noexec /tmp, and the runner operations that have killed live runs.

Prefer a GitHub hosted runner over the laptops:

gh workflow run e2e-nightly.yml --ref <branch> \
  -f image=ghcr.io/tuna-os/yellowfin:gnome -f win_version=11 -f bitlocker=off

ubuntu-latest exposes /dev/kvm, and e2e-hosted.yml handles disk reclaim and storage placement. The three laptop runners have each failed in a different host-specific way (see the table at the end of docs/agent-lessons.md).

On a laptop runner, launch through the just recipes — they encode all of the rules below (systemd user unit, live-run guard, no storage wipe) so you don't have to remember them:

loginctl enable-linger $USER          # once per host
just remote-e2e          # fresh Windows install + deploy (~60-90 min)
just remote-e2e-quick    # restore pristine Windows, re-arm, deploy (~20-40 min)
just remote-e2e-phase3   # quick + graduate to a blank native disk (rung 3)
# host defaults to the primary E2E runner; override with WOOTC_E2E_HOST=<host>
# logs: /tmp/wootc-e2e-<short-sha>.log on the host (just remote-logs tails it)

If launching by hand instead, it must be a systemd user unit — not nohup. Lingering must be enabled or systemd kills the run when your ssh session closes:

ssh <host> 'cd ~/wootc
  systemd-run --user --unit=wootc-e2e --collect \
    --setenv=XDG_RUNTIME_DIR=/run/user/$(id -u) \
    --setenv=HOME=$HOME \
    -p StandardOutput=append:/tmp/wootc-e2e-run.log \
    -p StandardError=append:/tmp/wootc-e2e-run.log \
    -p WorkingDirectory=$HOME/wootc \
    ./tests/e2e/run-e2e.sh ghcr.io/tuna-os/yellowfin:gnome'

XDG_RUNTIME_DIR and HOME are required, or rootless podman resolves root storage paths and fails with permission denied on /run/containers/storage.

Checking on a run — do not trust pgrep (it matches your own ssh command) or systemctl is-active (meaningless if launched via nohup). Use:

ssh <host> '
  # is it writing?
  echo "age=$(( $(date +%s) - $(stat -c %Y /tmp/wootc-e2e-run.log) ))s"
  # is the guest working? silence + high CPU = fine, silence + idle = wedged
  podman exec wootc-e2e-windows sh -c "ps -eo pcpu,args | grep [q]emu-system"
  # read only the CURRENT run (logs are appended across runs)
  L=$(grep -an "Run ID" /tmp/wootc-e2e-run.log | tail -1 | cut -d: -f1)
  tail -n +$L /tmp/wootc-e2e-run.log | tail -5'

Decisive markers to grep for, in order of the run: closure staged and verified → deployer active → deployer rebooted → attach-loop hook entered → either an EXIT: <reason> line or post-attach by-uuid.

Never podman system prune -af on a host with a live run. It has killed three runs at once and deleted the locally-built wootc-e2e-windows-ssh:latest, after which compose tries to pull from a registry literally named localhost. Rebuild with bash tests/e2e/build-ssh-image.sh.

For delegated work, use the wootc.wootc-e2e-runner agent.