Port tests to barrage

.github/workflows/ci.yml

@@ -30,10 +30,10 @@ jobs:
           mkosi -f box -- true
 
       - name: Run unit tests
-        run: mkosi box -- python3 -m pytest -sv
+        run: mkosi box -- python3 -m barrage -v
 
       - name: Run installation tests
-        run: mkosi box -- python3 -m pytest -sv -m install
+        run: mkosi box -- python3 -m barrage -v --pattern 'install_*.py'
 
       - name: Test execution from current working directory (sudo call)
         run: sudo python3 -m mkosi -h
@@ -148,14 +148,14 @@ jobs:
       - name: Run integration tests
         run: |
           # Without KVM the tests are way too slow and time out
+          # barrage currently has no dynamic runtime throttling to resources
+          # each test runs a ~ 1.5 GiB VM, and GitHub runners have 4 GiB
           if [[ -e /dev/kvm ]]; then
             sudo mkosi box -- \
               timeout -k 30 1h \
-              python3 -m pytest \
-              --tb=no \
-              --capture=no \
+              python3 -m barrage \
               --verbose \
-              -m integration \
-              --distribution ${{ matrix.distro }} \
+              --max-concurrency=2 \
+              --pattern 'integration_*.py' \
               tests/
           fi

.gitignore

@@ -5,7 +5,6 @@
 .mypy_cache/
 .project
 .pydevproject
-.pytest_cache/
 /.mkosi-*
 /SHA256SUMS
 /SHA256SUMS.gpg

AGENTS.md

@@ -13,13 +13,12 @@ Always consult these files as needed:
 ## Build and Test Commands
 
 - Running tests: See the "Hacking on mkosi" section in `README.md` for complete instructions.
-- `bin/mkosi box -- pytest` to run all unit tests including linters and type checkers
-- append usual pytest options like `-k test_mypy` to run a specific check
+- `bin/mkosi box -- python3 -m barrage` to run all unit tests including linters and type checkers
+- select a subset by name, e.g. `bin/mkosi box -- python3 -m barrage test_mypy` to run a specific check
 - `bin/mkosi box -- ruff format mkosi tests kernel-install/*.install` to format code
 - `bin/mkosi box -- ruff check --fix mkosi tests kernel-install/*.install` to fix ruff issues
-- `python3 -m pytest -m integration ...` to run integration tests. No need to run these by default.
-- `bin/mkosi box -- pytest -m install` to run installation tests (venv/pip/zipapp). Skipped by default as they install from the network. No need to run these by default.
-
+- `bin/mkosi box -- python3 -m barrage --pattern 'integration_*.py' tests/` to run integration tests (after `tools/integration-test-setup.sh`). No need to run these by default.
+- `bin/mkosi box -- python3 -m barrage --pattern 'install_*.py'` to run installation tests (venv/pip/zipapp). Skipped by default as they install from the network. No need to run these by default.
 - Never invent your own build commands or try to optimize the build process.
 - Never use `head`, `tail`, or pipe (`|`) the output of build or test commands. Always let the full output
 display. This is critical for diagnosing build and test failures.

README.md

@@ -125,7 +125,7 @@ modifications.
 
 To hack on mkosi itself, you can run the full test suite locally, just
 like CI does. The tests include linting, type checking, and unit tests,
-all runnable via [pytest](https://github.com/pytest-dev/pytest).
+all runnable via [barrage](https://github.com/amutable-systems/barrage).
 
 All linters such as `ruff` or `mypy` are run inside `mkosi box`
 (i.e. from inside `mkosi.tools/`) for a consistent environment. Build that with:
@@ -134,35 +134,36 @@ All linters such as `ruff` or `mypy` are run inside `mkosi box`
 bin/mkosi -f box -- true
 ```
 
-Then run the full test suite inside the tools tree:
+Then run the full unit test suite inside the tools tree:
 
 ```bash
-bin/mkosi box -- pytest
+bin/mkosi box -- python3 -m barrage
 ```
 
-You can use [pytest options](https://docs.pytest.org/en/stable/how-to/usage.html) to only run a subset, for
-example only run the linters:
+You can select a subset, for example only run the linters or only a single test:
 
 ```sh
-bin/mkosi box -- pytest -k test_linters
+bin/mkosi box -- python3 -m barrage tests/test_linters.py
+
+bin/mkosi box -- python3 -m barrage test_conversion
 ```
 
-Installation tests (venv, editable and zipapp installs) are marked with the
-`install` marker and are skipped by default, as they create virtual
-environments and install packages from the network. Run them explicitly with:
+Installation tests (venv, editable and zipapp installs) live in `install_*.py`
+files and are skipped by default, as they create virtual environments and
+install packages from the network. Run them explicitly with:
 
 ```sh
-bin/mkosi box -- pytest -m install
+bin/mkosi box -- python3 -m barrage --pattern 'install_*.py'
 ```
 
 When a tool that mkosi runs inside its sandbox fails, see
 [Debugging failing sandboxed commands](docs/debugging.md) for how to replay the command by hand.
 
 # Integration tests
 
-Integration tests build and boot full images. They are marked with the
-`integration` marker, and are skipped by default. They need a tools tree and an
-image to be built first. `tools/integration-test-setup.sh` writes a local
+Integration tests build and boot full images. They live in `integration_*.py`
+files and are not run as part of the normal test suite. They need a tools tree
+and an image to be built first; `tools/integration-test-setup.sh` writes a local
 configuration for the given image and tools tree distribution and builds both:
 
 ```sh
@@ -175,13 +176,37 @@ single integration test:
 ```sh
 tools/integration-test-setup.sh arch fedora
 
-bin/mkosi box -- pytest -m integration --distribution arch --capture=no --verbose \
-    'tests/test_boot.py::test_bootloader[systemd-boot]'
+bin/mkosi box -- python3 -m barrage -v \
+    'tests/integration_boot.py::test_bootloader_systemd_boot'
+```
+
+To run all integration tests, select them by file name pattern:
+
+```sh
+bin/mkosi box -- python3 -m barrage --pattern 'integration_*.py' tests/
 ```
 
 The integration tests require KVM and are skipped (or very slow) without
 `/dev/kvm`.
 
+To debug a failing build, set `TEST_DEBUG_SHELL=1` to pass `--debug-shell` to
+mkosi, which drops into an interactive shell when a build step fails. This only
+makes sense when running a single test interactively (`barrage -i`):
+
+```sh
+TEST_DEBUG_SHELL=1 bin/mkosi box -- python3 -m barrage -i \
+    'tests/integration_boot.py::test_bootloader_systemd_boot'
+```
+
+barrage runs tests concurrently with no limit by default. Pass
+`--max-concurrency N` to restrict the parallelism.
+
+When running unprivileged (i.e. not as root), building several images in the
+same session tends to exhaust systemd-nsresourced's pool of dynamic UID ranges,
+which makes builds fail with `io.systemd.NamespaceResource.NoDynamicRange`. Run
+the tests one at a time (or with `--max-concurrency 1`) to avoid this; it does
+not happen when the tests run as root, as they do in CI.
+
 # References
 
 * [Primary mkosi git repository on GitHub](https://github.com/systemd/mkosi/)

REUSE.toml

@@ -39,6 +39,7 @@ path = [
     "**.zsh",
     "mkosi.credentials/*",
     "mkosi/resources/pandoc/*.lua",
+    "mkosi.tools.conf/test-requirements.txt",
 ]
 precedence = "aggregate"
 SPDX-FileCopyrightText = "Mkosi Contributors"

mkosi.tools.conf/mkosi.conf.d/arch.conf

@@ -10,7 +10,7 @@ Packages=
         mkinitcpio
         mypy
         pyright
-        python-pytest
+        python-pip
         reuse
         ruff
         sequoia-sop

mkosi.tools.conf/mkosi.conf.d/azure-centos-fedora.conf

@@ -10,6 +10,6 @@ Packages=
         codespell
         cryptsetup
         python3-mypy
+        python3-pip
         npm
-        python3-pytest
         shellcheck

mkosi.tools.conf/mkosi.conf.d/debian-kali-ubuntu.conf

@@ -12,7 +12,7 @@ Packages=
         fdisk
         mypy
         npm
-        python3-pytest
+        python3-pip
         reuse
         sqop
         shellcheck

mkosi.tools.conf/mkosi.conf.d/opensuse.conf

@@ -10,7 +10,7 @@ Packages=
         grub2 # TODO: Move to default tools tree when https://bugzilla.opensuse.org/show_bug.cgi?id=1227464 is resolved.
         mypy
         npm
-        python3-pytest
+        python3-pip
         reuse
         ruff
         ShellCheck

mkosi.tools.conf/mkosi.conf.d/postmarketos.conf

@@ -6,5 +6,6 @@ Distribution=postmarketos
 [Content]
 Packages=
         py3-codespell
+        py3-pip
         py3-pytest
         ruff

mkosi.tools.conf/mkosi.prepare.chroot

@@ -8,3 +8,5 @@ if ! command -v pyright &>/dev/null; then
         fi
     done
 fi
+
+python3 -m pip install --break-system-packages -r "$SRCDIR/mkosi.tools.conf/test-requirements.txt"

mkosi.tools.conf/test-requirements.txt

@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+# Test runner installed into the tools tree.
+# FIXME: move to version pin once published on pypi, to get dependabot coverage
+barrage @ git+https://github.com/amutable-systems/barrage.git@8361c35b7ad7c00e6a446f16d5a269f07a35393b

mkosi/config.py

@@ -5445,7 +5445,12 @@ def want_default_initrd(config: Config) -> bool:
     return Path("default") in config.initrds
 
 
-def finalize_historydir(args: Args) -> Path:
+def finalize_historydir(args: Args, output_dir: Optional[Path] = None) -> Path:
+    # When an output directory is given, the build history is also stored there so that builds into
+    # distinct output directories don't read each other's history. Otherwise it lives in the config dir.
+    if output_dir is not None:
+        return output_dir / ".mkosi-private/history"
+
     configdir = finalize_configdir(args.directory)
     return (configdir or Path.cwd()) / ".mkosi-private/history"
 
@@ -5502,7 +5507,13 @@ def parse_config(
         return args, None, ()
 
     configdir = finalize_configdir(args.directory)
-    historydir = finalize_historydir(args)
+    config_historydir = finalize_historydir(args)
+    # Prefer the history stored in the output directory (when --output-directory is given on the CLI) so a
+    # consumer reads back exactly the build that wrote into that directory. Fall back to the config dir for
+    # consumers that don't pass --output-directory and for history written by older mkosi versions.
+    historydir = finalize_historydir(args, context.cli.get("output_dir"))
+    if not (historydir / "latest.json").exists():
+        historydir = config_historydir
 
     if have_history(args, historydir):
         history = Config.from_partial_json((historydir / "latest.json").read_text())
@@ -5555,8 +5566,14 @@ def parse_config(
     maincontext = copy.deepcopy(context)
 
     if config["history"] and want_new_history(args):
-        historydir.mkdir(parents=True, exist_ok=True)
-        (historydir / "latest.json").write_text(dump_json(Config.to_partial_dict(cli)))
+        # Store the history in the config dir (so consumers that don't pass --output-directory find it) and,
+        # when an output directory is configured, in the output directory too (so builds into distinct
+        # output directories stay isolated). These coincide when no output directory is set. This keys on
+        # the finalized output dir (config or CLI), unlike the read above which can only use the CLI value
+        # (the configuration isn't parsed yet there), so don't collapse the two into one variable.
+        for hd in {config_historydir, finalize_historydir(args, config.get("output_dir"))}:
+            hd.mkdir(parents=True, exist_ok=True)
+            (hd / "latest.json").write_text(dump_json(Config.to_partial_dict(cli)))
 
     tools = None
     if config.get("tools_tree") in (Path("default"), Path("yes")):

pyproject.toml

@@ -84,6 +84,18 @@ warn_unreachable = true
 strict_equality = true
 scripts_are_modules = true
 
+# barrage (the async test runner used by tests/) requires Python 3.12+ syntax, which mypy cannot parse
+# when checking under our supported floor (3.9/3.10). Skip following it and don't flag the test classes
+# that subclass its (now Any-typed) AsyncTestCase. pyright still type-checks tests/ fully.
+[[tool.mypy.overrides]]
+module = "barrage.*"
+ignore_missing_imports = true
+follow_imports = "skip"
+
+[[tool.mypy.overrides]]
+module = "tests.*"
+disallow_subclassing_any = false
+
 [tool.ruff]
 target-version = "py39"
 line-length = 109
@@ -98,11 +110,3 @@ lint.select = [
 
 [tool.ruff.lint.flake8-bugbear]
 extend-immutable-calls = ["mkosi.run.nosandbox"]
-
-[tool.pytest.ini_options]
-markers = [
-    "integration: mark a test as an integration test.",
-    "install: mark a test as an installation test (venv/pip/zipapp).",
-]
-addopts = "-m \"not integration and not install\""
-testpaths = ["tests"]