TESTING.md

Testing Guide

Local Checks Before Image or VM Tests

Run these commands from the repository root. Choose the checks for the files you change:

Check	Command	Purpose
Format and lint	just fix && just check	Required before each commit; see the contributor workflow. Run just ste for Markdown changes; see prose checks.
Unit suites	just test-bats, just test-pytest, or just test	Run shell tests, Python tests, or the full test recipe.
Image and VM checks	just test-cell <variant> <flavor> or just test-e2e <iso>	Check a published container image or boot an ISO. These need Podman or QEMU; see the ISO instructions below.

The source recipes define the checks and their setup steps. just setup installs check and test dependencies through Homebrew and pip. Setup can download dependencies. Some tests also need network access or containers; inspect the selected tests before you run them.

For a focused check, the test-bats recipe runs each Bats file with --formatter tap. Run only the functional dispatcher's unit tests with:

bats --formatter tap tests/bats/test_functional_run.bats

This direct command needs Bash, Bats, Python 3, and standard shell utilities on PATH. The test file creates temporary files and stubs systemctl, bootc, and flatpak. It tests the dispatcher without an image build, VM, or network access. These tests check the dispatcher logic; use the image and VM checks to verify a built system.

The CI BATS runner runs two files at a time and each file's cases in series. It uses a private /tmp/tb.XXXXXX directory and numeric worker IDs for temporary files. In CI #4179, a long RUNNER_TEMP plus the test filename and the suffix from BATS caused AF_UNIX path too long in the poweroff test. Keep paths for sockets short; do not base them on RUNNER_TEMP or test filenames.

The runner removes only its own scratch directory. It prints each file's complete log in stable order. If worker storage is not available, it runs all files in series. The runner behavior tests include a real nested BATS run that binds a socket with a long RUNNER_TEMP.

PR Tuna image coverage

tuna-pr-images.yml calls tuna-pr-build-inputs.sh --variants for its matrix. The default script interface remains boolean: exit 0 for image inputs and 1 for unrelated paths. Docs-only PRs select no images. A rename includes the old path and the new path.

The selector unions the consumers of known local inputs:

Input	Tuna variants
Arch recipe or tuna-arch.yaml	marlin
Debian recipe or tuna-debian.yaml	flounder, flounder-sid
Ubuntu or openSUSE recipe	grouper or sailfin, respectively
Variant override or desktop-contract directory	Its admitted variant; bonito and flounder overrides include their rolling siblings
Isolated existing variant edits in tunaos.toml	The changed admitted variants, compared with the PR base SHA

resolve-flavor.sh, install-desktop.sh, and the variant override installer show those consumers. Shared recipes (including EL10, final and overlay), build logic, shared manifests, pins, shared config, and unknown relevant paths keep all eleven admitted :tuna cells. Missing base history or unreadable config also keeps the full matrix. Shared changes never select only representative cells.

The four family canaries stay independent of this subset: albacore:gnome, bonito:kde, marlin:cosmic, and flounder:xfce. All four run on the existing Containerfile*, build_scripts/, or tunaos.toml inputs. They keep the same desktop contract and boot checks. PR jobs do not log in or push. They get no publication secrets. Regression coverage lives in tests/test_pr_canary_cells.py and tests/test_tuna_admission.py.

TunaOS uses a QEMU-based end-to-end test harness to verify ISO images boot correctly and reach the live desktop environment.

Manual Mesa comparison

The Mesa #3969 lab can check code safety. Graphics need separate proof. The existing Test workflow discovers the new pytest cases, including tests/manual/test_mesa_3969_source_smoke.py. The smoke can check CLI refusals, archive integrity and cleanup with synthetic data. It blocks build tools and does not verify signatures, load an image or start a guest. Run these checks locally; CI runs them too.

mesa-3969-compare.yml is a separate, manual-only experiment. Do not dispatch it for a source check. A later run needs owner approval of the exact source head, the matching MESA_3969_APPROVED_HEAD repository variable, reviewed_head, and confirmation RUN_MESA_3969_ONCE. This workflow does not create variables or secrets. Admission rejects forks, automatic events, head mismatches, repeated runs and a head used by a prior lab run. Configure approval before the first dispatch; a failed run is not a license to retry the same head.

The lab builds both complete Mesa sets from one frozen buildroot and one baseline. It uses the unchanged native Containerfile once, not the normal wrapper's retry/global-cleanup loop. Only the run-bound lab container may replace RPMs. Never change a host package database.

The pinned probe runs before GDM. The existing real login, lock, unlock and VM cleanup stay in use. Reports must link observed output from the software renderer to the observed compositor and session. Exit 3 or 4 means an unexercised device path, not successful EGL initialization.

An installed package is not proof of loaded code. Each arm records the Mesa files mapped by the probe (traced until it exits) and by the greeter's gnome-shell. It then compares their SHA-256 with the files in that arm's transferred RPMs. comparison.json byte_provenance gives the same files' digests in the committed image, Corral's derived image, the guest's ostree deployment and the root it booted. The first stage that differs is where the patched bytes were lost.

Limits are 8 CPU, 80 GB and 180 minutes for preparation, then 60 minutes for sequential KVM control/candidate guests. The lab accepts archives up to 24 GiB and extracts up to 32 GiB. The evidence limit is 32 MiB, with one day of retention. Never upload private keys, raw SSH state, VM disks or ISOs. No R2 or production publication is part of this lab.

Runtime state uses the private, run-marked /tmp/m3969-<run-id> root. Corral homes and VM names stay short enough for Linux socket paths. Admission checks the paths of sockets before packages build. Local smoke checks with QEMU and swtpm cover sockets and cleanup, not a booted desktop.

Source CI does not qualify Mesa, GDM, encrypted installation or either configuration/ISO epic. A real run and review of its retained evidence remain separate acceptance.

Publish Gating

Nothing user-facing is published without a boot check. The pipeline follows Bluefin's testing→stable promotion model:

Artifact	Gate	Where
GHCR image :<flavor>	Manifest is pushed as :<flavor>-candidate; a boot gate (verify_boot job) installs it to a qcow2 with bootc, boots it in QEMU (scripts/iso-e2e.sh --disk), and only then the promote job writes :<flavor>, :<flavor>-YYYYMMDD, and per-arch tags. base* flavors skip the boot gate (no desktop) and promote after manifest.	reusable-build-image.yml
ISOs on R2 / GitHub Releases	ISO is built, boot-verified in QEMU (scripts/iso-e2e.sh, readiness marker or screenshot-sanity fallback), and only uploaded if the gate passes.	reusable-build-artifacts.yml, publish-iso-groups.yml
PRs	Build + QEMU boot verification of the locally built image (amd64).	reusable-build-image.yml

The screenshot-sanity fallback exists because the EL10 bootc kernels ship CONFIG_SERIAL_8250=m, so readiness markers often never reach the serial console; a rendered (non-blank) framebuffer captured via -vga virtio counts as booted, a black/absent one fails the gate.

Run the same gates locally:

just verify-disk yellowfin.qcow2      # boot-gate a disk image
./scripts/iso-e2e.sh some.iso         # boot-gate an ISO

Automated screenshots: weekly-desktop-screenshots.yml commits real desktop captures for every variant × DE to docs/images/desktops/, and installer-screenshots.yml drives the GUI installer and commits the flow captures to docs/images/installer/ (then boot-verifies the disk the walkthrough installed).

Each base-DE pair has a LUKS E2E WebM.

See the latest LUKS install timelapses. HWE and NVIDIA overlays stay in the per-cell artifacts. The source run links them.

Code Coverage

The Unit Tests job writes coverage-python.xml and sends it to Codecov with the python flag. It also keeps the XML as a workflow artifact for 30 days. Codecov applies the project and patch targets in codecov.yml to pull requests and pushes to main. GitHub OIDC authenticates each upload. The workflow does not use a long-lived token.

The report measures Python only. BATS checks how the shell behaves. The QEMU suites check built images and the installer. These suites do not emit data about line coverage. Thus, the Codecov percentage does not include shell or end-to-end coverage.

ISO End-to-End Tests

The scripts/iso-e2e.sh script boots a TunaOS live ISO in QEMU with OVMF (UEFI), waits for the live environment to be ready, captures screenshots, and collects serial logs.

Framebuffer capture and paint verification live in scripts/lib/e2e-visual.sh. The top-level harness supplies its output directory, monitor socket, and image identity, then calls the module's screenshot, screenshot_compare, screenshot_sane, and wait_for_paint interface. Keep visual-policy changes in that module; tests/bats/test_e2e_visual.bats exercises it without QEMU.

Running Locally

set -euo pipefail

# Resolve the current pre-built ISO. During a rollback, the stable object stays
# frozen and this resolver points at a checksum-addressed snapshot. Only a
# confirmed missing resolver selects the legacy stable object.
base=https://download.tunaos.org/live-isos
stable=yellowfin-gnome-latest.iso
resolution_file="$(mktemp)"
trap 'rm -f "$resolution_file"' EXIT
status="$(curl --silent --show-error --location --output "$resolution_file" \
  --write-out '%{http_code}' "$base/${stable%.iso}.json")"
case "$status" in
  200)
    resolved="$(jq -er '
      select(.schema == "TunaOSIsoResolution/v1")
      | select(.iso | test("^[A-Za-z0-9][A-Za-z0-9._+~-]*[.]iso$"))
      | select(.digest | test("^sha256:[0-9a-f]{64}$"))
      | [.iso, .digest] | @tsv
    ' "$resolution_file")"
    IFS=$'\t' read -r iso expected <<<"$resolved"
    ;;
  404) iso="$stable"; expected="" ;;
  *) echo "resolver request failed with HTTP $status" >&2; exit 1 ;;
esac
curl --fail --location --remote-name "$base/$iso"
if [[ -n "$expected" ]]; then
  actual="sha256:$(sha256sum "$iso" | awk '{print $1}')"
  [[ "$actual" == "$expected" ]] || {
    echo "ISO digest mismatch: expected $expected, got $actual" >&2
    exit 1
  }
fi

# Run the e2e test
./scripts/iso-e2e.sh yellowfin gnome "./$iso"

Requirements:

• qemu-system-x86_64 with KVM support
• qemu-utils for qemu-img
• socat for QEMU monitor communication
• At least 4GB RAM available for the VM

What Gets Tested

The harness validates:

1. Boot success — ISO reaches the live environment within 90 seconds
2. Desktop readiness — gdm.service or equivalent display manager is active
3. No critical failures — no Failed to start in the systemd journal
4. Screenshot capture — visual confirmation of the desktop
5. Serial logs — full boot output for debugging

CI Integration

The .github/workflows/iso-e2e.yml workflow runs automatically:

• On PRs that modify build inputs (Containerfile*, build_scripts/**, live-iso/**, system_files*/**)
• Weekly on a schedule for regression detection
• Posts PR comments with screenshots and serial log summaries

Test Artifacts

Test outputs are uploaded as GitHub Actions artifacts:

• screenshot.png — Desktop screenshot via QEMU monitor
• serial.log — Full boot serial console output

Test Files

File	Purpose
tests/anaconda-ks.cfg	Kickstart configuration for automated Anaconda install testing
tests/lima-template.yaml	Lima VM template for macOS testing
tests/live-iso-verify.yaml	Live ISO verification manifest
scripts/iso-e2e.sh	Main QEMU-based end-to-end test runner
scripts/lib/e2e-visual.sh	Screenshot capture, comparison, blank-frame detection, and bounded paint polling
.github/workflows/iso-e2e.yml	CI workflow for automated ISO testing
tests/functional/run.sh	Tier-1 functional checks for a booted image (per-desktop SSH assertions, #576)
tests/bats/test_functional_run.bats	Unit tests for the functional-check dispatcher (stubbed systemctl/bootc/flatpak)

Functional Checks (per-image, per-desktop)

The boot gate proves an image boots; tests/functional/run.sh proves features a user needs are present and working on the running system (#576). It runs over SSH against a booted image — corral VM, gate VM, or local install — and emits TAP-style ok/not ok lines; the exit code is the failure count.

Checks: system running + graphical.target; no failed units outside the VM-noise allowlist (libstoragemgmt, mcelog); the desktop's display manager active; the desktop's session binary and a session entry present; bootc status healthy; Flathub configured; and (when a variant is given) image-info.json

• installer recipe.json branding.

# Against a corral VM running a booted image. `corral ssh` takes at most one
# positional arg (the VM name) — its -c flag is a SINGLE command string, so
# the desktop/variant args go inside that string, not after the redirection.
corral ssh <vm> -u root -c 'bash -s gnome yellowfin' < tests/functional/run.sh

# Or copy it in and run inside the guest
scp tests/functional/run.sh root@<guest>:/tmp/ && ssh root@<guest> bash /tmp/run.sh kde yellowfin

# Composefs variant (e.g. grouper): opt in to the composefs assertion
FUNCTIONAL_EXPECT_COMPOSEFS=1 tests/functional/run.sh gnome grouper

just boot-gate <variant> [flavor] / scripts/boot-gate.sh and CI (reusable-build-image.yml verify_boot step) run this dispatcher automatically after their own graphical.target/display-manager checks (advisory in verify_boot — see the comment in boot-gate.sh for the overlay-suffix-to-desktop-name mapping and why *-nvidia/*-hwe/etc. flavors still resolve to a bare desktop name run.sh understands).

Run the dispatcher's unit tests with the rest of the suite:

bats tests/bats/test_functional_run.bats

Writing New Tests

For desktop environment changes:

1. Build the ISO: sudo just iso yellowfin gnome local
2. Run the e2e harness: ./scripts/iso-e2e.sh yellowfin gnome .build/iso/yellowfin-gnome.iso
3. Verify the screenshot shows the expected desktop
4. Check serial logs for any unexpected failures

Troubleshooting

Symptom	Likely Cause
ISO doesn't boot	Missing KVM support; try adding --no-kvm to the QEMU command
Timeout waiting for desktop	Desktop environment failed to start; check serial logs
qemu-img not found	Install qemu-utils package
Screenshot is blank	Not necessarily a failure. Under plain virtio-vga (no render node) the guest paints with Mesa's llvmpipe software rasteriser, and first paint can trail the serial markers by a minute or more on a 2-4 vCPU runner (tunaOS#581). The gates key off the serial markers (TUNAOS_DESKTOP_CONTRACT_OK / readiness marker); iso-e2e.sh waits for the framebuffer to paint before the evidence screenshot (wait_for_paint, bounded by TBOX_E2E_PAINT_TIMEOUT, default 120s) instead of a fixed sleep. A still-blank capture after that cap means the image genuinely never painted — check serial.log for whether the display manager started