Skip to content

Release smoke test: Playwright verifies login page from installed wheel (#1611)#1705

Merged
mcdonc merged 4 commits into
mainfrom
issue-1611-release-smoke-test
Jul 21, 2026
Merged

Release smoke test: Playwright verifies login page from installed wheel (#1611)#1705
mcdonc merged 4 commits into
mainfrom
issue-1611-release-smoke-test

Conversation

@mcdonc

@mcdonc mcdonc commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Closes #1611.

Summary

Add a dist-smoke-test job to release.yml that gates build-and-release (image push + GH release). Downloads the wheel built by build-wheel, installs it into an isolated venv (not editable — catches packaging bugs an editable install would hide), starts the real klangkd entrypoint (uvicorn on UDS + nginx child), polls /health, and runs one minimal Playwright spec against it. If this fails, the release stops before images are pushed or the GitHub release is created.

What this catches before users see a broken release

Failure mode Symptom
Frontend not included in wheel 404 / blank page
_DEFAULT_FRONTEND_DIR resolves wrong post-install Same
Flutter build broken / incomplete flutter-view never attaches
main.dart.js missing or corrupt Engine doesn't boot
Auth config endpoint broken Login form never renders (title never becomes "Login")
klangkd entrypoint registration broken Server never starts (caught by /health poll)
nginx not found or config render broken nginx refuses to start
UDS proxy_pass misconfigured nginx 502s
location / missing in full template Static files not served
UDS path too long (sun_path overflow) uvicorn bind fails

Pieces

  • scripts/dist-smoke-test.sh — the runner. Installs the wheel into an isolated venv, starts klangkd --config=none with password-mode env vars (so the login form is the rendered route), polls /health, runs npx playwright test --project=dist-smoke, traps klangkd teardown on exit. shellcheck clean.
  • src/frontend/e2e-tests/e2e/dist-smoke.spec.ts — the one-test spec. Navigates /, waits for "Loading, please wait" to clear (Flutter boot), waits for <flutter-view> (engine hydrated), asserts page title is /Login/i. Mirrors the e2e suite's canonical login-page check.
  • src/frontend/e2e-tests/playwright.config.ts — new dist-smoke project matching only dist-smoke.spec.ts on chromium. Other projects untouched.
  • .github/workflows/release.yml — new dist-smoke-test job (needs: build-wheel) that downloads the wheel artifact and runs scripts/dist-smoke-test.sh inside devenv shell (devenv provides nginx on PATH + the nix-bundled Playwright Chromium via PLAYWRIGHT_BROWSERS_PATH). build-and-release now needs: dist-smoke-test.

Dependency chain:

build-wheel ─→ dist-smoke-test ─→ build-and-release
   │
   └─→ PyPI publish (stays parallel with the smoke test)

PyPI publish stays parallel: pip/uv users get the wheel immediately, and if the smoke test reveals the wheel is broken we yank. The expensive image build + GH release only happens once the smoke test passes.

Reuses existing harness

The smoke job reuses the existing global-setup / global-teardown KLANGK_TEST_URL short-circuit — they were already designed for "external server" mode (skip own server startup, just poll /health on the supplied URL; skip teardown). Setting KLANGK_TEST_URL=http://127.0.0.1:18997 makes the existing harness work for the smoke run with no harness changes.

Verification

  • release.yml YAML parses; devenv-setup action correctly skips the Build step for build-tasks: '' (the action's if: inputs.build-tasks != '' guard handles it).
  • npx playwright test --project=dist-smoke --list matches exactly the one new test — no other projects disturbed (206 total tests across all projects, +1 vs. baseline).
  • shellcheck scripts/dist-smoke-test.sh — clean.

A live end-to-end run happens the first time this fires in CI on a v* tag push. The unit-level checks above cover everything that can be validated without a full Flutter web build.

mcdonc added 2 commits July 21, 2026 11:26
@mcdonc
feat(1611): release smoke test — Playwright verifies login page from …
7f59f98 
…installed wheel (#1611)

Add a dist-smoke-test job to release.yml that gates build-and-release
(image push + GH release creation). Downloads the wheel built by build-wheel,
installs it into an isolated venv (not editable — catches packaging bugs an
editable install would hide), starts the real klangkd entrypoint (uvicorn on
UDS + nginx child), polls /health, and runs one minimal Playwright spec
against it. If this fails, the release stops before images are pushed.

What this catches before users see a broken release:

  - Frontend not included in wheel → 404 / blank page
  - _DEFAULT_FRONTEND_DIR resolves wrong post-install → same
  - Flutter build broken / incomplete → flutter-view never attaches
  - main.dart.js missing or corrupt → engine doesn't boot
  - klangkd entrypoint registration broken → server never starts
  - nginx not found or config render broken → nginx refuses to start
  - UDS proxy_pass misconfigured → nginx 502s
  - location / missing in full template → static files not served

Pieces:

- scripts/dist-smoke-test.sh — the runner. Installs the wheel into an
  isolated venv, starts klangkd --config=none with password-mode env vars
  (so the login form is the rendered route), polls /health, runs
  `npx playwright test --project=dist-smoke`, traps klangkd teardown on
  exit. Shellcheck clean.

- src/frontend/e2e-tests/e2e/dist-smoke.spec.ts — the one-test spec.
  Navigates /, waits for the "Loading, please wait" text to clear
  (Flutter boot), waits for <flutter-view> (engine hydrated), asserts
  page title is /Login/i. Mirrors the existing e2e helpers' canonical
  login-page check.

- src/frontend/e2e-tests/playwright.config.ts — new `dist-smoke` project
  matching only dist-smoke.spec.ts on chromium. Other projects untouched.

- .github/workflows/release.yml — new dist-smoke-test job (needs: build-wheel)
  that downloads the wheel artifact and runs scripts/dist-smoke-test.sh
  inside devenv shell (devenv provides nginx on PATH + the nix-bundled
  Playwright Chromium via PLAYWRIGHT_BROWSERS_PATH). build-and-release
  now needs: dist-smoke-test. Dependency chain:
    build-wheel → dist-smoke-test → build-and-release
  PyPI publish (inside build-wheel) stays parallel with the smoke test
  (pip/uv users get the wheel immediately; if the smoke fails we yank).

The smoke job reuses the existing global-setup/global-teardown
KLANGK_TEST_URL short-circuit (they were already designed for "external
server" mode) — no harness changes needed.

Verified locally: YAML parses; devenv-setup action correctly skips the
Build step for build-tasks: ''; `npx playwright test --project=dist-smoke
--list` matches exactly the one new test (no other projects disturbed);
shellcheck clean on the runner script.
@mcdonc
test(1611): apply #1705 review — wheel-bundled frontend, serial relea…
dbce6a8 
…se topology (#1705 review)

Five fixes from the adversarial review of #1705:

B1 (blocker) — KLANGK_FRONTEND_DIR was leaking from devenv shell into the
klangkd child, defeating the gate: in CI (no Flutter build) it would make
the smoke fail on first run; locally (with a dev Flutter build) it would
make the smoke pass while testing the dev tree, not the wheel. Use
`env -u KLANGK_FRONTEND_DIR` to strip the inherited var so settings.py
falls back to _DEFAULT_FRONTEND_DIR = <site-packages>/klangk/frontend (the
wheel-bundled copy). Empty string does NOT work — pydantic-settings accepts
it literally and Path("").exists() is False, so the UI mount is skipped
with a warning; unsetting is what triggers the fall-back.

I1 — Topology changed: make PyPI publish wait on the smoke test. Pre-PR
build-wheel published in parallel with the image build; that put a broken
wheel on PyPI before the smoke caught it and required a yank. Restructured
to build-wheel (no publish) → dist-smoke-test → publish-and-release, where
publish-and-release publishes PyPI + pushes images + cuts the GH release,
all gated by the smoke. No yank path exists, by design — broken wheels
never reach the index.

I2 — Acknowledged in publish-and-release's job comment: the host-image
build (slow step, ~25-30 min) runs in publish-and-release, not parallel
with wheel+smoke. Splitting it into a separate job would require
docker save/load of the multi-GB host image (which embeds the workspace
tarball) between runners — roughly the time the parallelism would save.
Tag-push-only frequency makes the serial SLO acceptable.

I3 — Redirect klangkd's stdio to $STATE_DIR/server.log (was inherited
stdio → only landed in the GH Actions step log, not the failure artifact).
Mirrors global-setup.ts's openSync(logFd) pattern. Poll-loop failure
branches now `tail -n 50` the log before exiting so the actual startup
error is in the step output, not just the artifact.

I4 — Replaced `klangkd --help` sanity check with
`python -c 'from klangk.main import build_app'`. Typer handles --help and
exits before main(), and build_app is imported lazily inside main(), so
--help proved only the entry-point registration. The import check surfaces
a ModuleNotFoundError with a real traceback if a transitive dep is missing
from pyproject.toml.

Verified: YAML valid; topology build-wheel → dist-smoke-test →
publish-and-release; dist-smoke project still matches exactly one test;
shellcheck clean.
@mcdonc
mcdonc force-pushed the issue-1611-release-smoke-test branch from 17e79e7 to dbce6a8 Compare July 21, 2026 15:27
mcdonc added 2 commits July 21, 2026 11:33
@mcdonc
feat(1611): run dist-smoke against stock Ubuntu, not devenv (#1611)
cfd54c2 
The whole point of the dist-smoke test is to prove a bare `pip install
klangk && klangkd` works in the environment the #1607 / #1645 first-run
audience actually uses — a stock OS with caddy + Python, no devenv/nix.
The previous version ran the smoke inside `devenv shell`, which:
  - bundled caddy, Python, AND Playwright Chromium from nix (defeating
    the "stock Ubuntu" claim);
  - leaked KLANGK_FRONTEND_DIR from devenv.nix, which the runner had to
    strip with `env -u` to avoid serving the dev tree's frontend instead
    of the wheel's.

Restructured:

- `.github/workflows/release.yml` — dist-smoke-test drops its
  `.github/actions/devenv-setup` step. apt-get installs caddy + the
  Playwright runtime libs + python3-venv directly from the Ubuntu archive;
  npm/node come from the GitHub Actions runner image; nothing from nix.
  publish-and-release still uses devenv (for build-host-image + podman);
  build-wheel still uses devenv (for flutterbuildweb via the build hook).
  Only the smoke step is devenv-free.

- `scripts/dist-smoke-test.sh` — no longer assumes it runs inside
  `devenv shell`. Adds `npx playwright install --with-deps chromium` so
  Playwright fetches its own pinned Chromium (was relying on
  PLAYWRIGHT_BROWSERS_PATH from devenv.nix). Documents the stock-Ubuntu
  PATH requirements (caddy, curl, python3, python3-venv, npm, node) in
  the script header.

Keeps the prior review fixes (#1705 review):
  - env -u KLANGK_FRONTEND_DIR (B1) — still needed even outside devenv in
    case a dev runs the script locally from a devenv shell.
  - python -c 'from klangk.main import build_app' sanity check (I4).
  - $STATE_DIR/server.log stdio redirect + log-tail on poll failure (I3).
  - Serial topology build-wheel → dist-smoke-test → publish-and-release
    (I1 + I2): PyPI publish, image push, and GH release all gate on the
    smoke passing — no yank path exists by design.

Verified: shellcheck clean; YAML valid; dist-smoke-test job has zero
devenv/nix references in its steps; dist-smoke project still matches
exactly one test.
@mcdonc
feat(1611): workflow_dispatch runs dist-smoke without publishing (#1611)
0561e85 
Add workflow_dispatch to release.yml so a maintainer can run the dist-smoke
gate against the current branch without cutting a release. The chain
build-wheel → dist-smoke-test runs on any dispatch; publish-and-release is
gated on `github.event_name == 'push'` so a manual run stops at the smoke
gate and never touches PyPI / GHCR / GitHub Release.

Use case: validate a release candidate before tagging. Run
`gh workflow run release.yml` from the release branch, watch the smoke
pass, then tag.

Also documents the fully-local option in scripts/dist-smoke-test.sh's
header — build the wheel with scripts/build_wheel.sh + run the script
directly. No CI round-trip needed.
@mcdonc
mcdonc merged commit 551e4a1 into main Jul 21, 2026
8 checks passed
@mcdonc
mcdonc deleted the issue-1611-release-smoke-test branch July 21, 2026 15:43
mcdonc added a commit that referenced this pull request Jul 21, 2026
@mcdonc
fix(1611): delegate Playwright browser-dep install to npx (#1705 foll…
043bcee 
…ow-up)

The dist-smoke-test job's hand-maintained apt list broke on the Ubuntu 24.04
t64 transition: libasound2 became a virtual package (provided by
libasound2t64), libatk1.0-0 → libatk1.0-0t64, libcups2 → libcups2t64, etc.
apt refused: "Package 'libasound2' has no installation candidate" → the
smoke job failed before the wheel was even downloaded
(https://github.com/mcdonc/klangk/actions/runs/29845509929/job/88686361663).

Install only caddy + curl + python3 + python3-venv in the workflow; let
`npx playwright install --with-deps chromium` (which the script already
runs) provide the browser runtime libs. Playwright's install-deps list
tracks per-OS package renames automatically, so the t64 transition (and
any future rename) is handled upstream rather than maintained here.
@mcdonc mcdonc mentioned this pull request Jul 21, 2026
Merged
mcdonc added a commit that referenced this pull request Jul 21, 2026
@mcdonc
fix(1611): delegate Playwright browser-dep install to npx (#1705 foll…
6da6084 
…ow-up) (#1707)

The dist-smoke-test job's hand-maintained apt list broke on the Ubuntu 24.04
t64 transition: libasound2 became a virtual package (provided by
libasound2t64), libatk1.0-0 → libatk1.0-0t64, libcups2 → libcups2t64, etc.
apt refused: "Package 'libasound2' has no installation candidate" → the
smoke job failed before the wheel was even downloaded
(https://github.com/mcdonc/klangk/actions/runs/29845509929/job/88686361663).

Install only caddy + curl + python3 + python3-venv in the workflow; let
`npx playwright install --with-deps chromium` (which the script already
runs) provide the browser runtime libs. Playwright's install-deps list
tracks per-OS package renames automatically, so the t64 transition (and
any future rename) is handled upstream rather than maintained here.
mcdonc added a commit that referenced this pull request Jul 21, 2026
@mcdonc
fix(1611): mask caddy.service so klangkd's child can bind admin (#1705 …
ef0b2ef 
…follow-up) (#1708)

The dist-smoke-test job's second workflow_dispatch run failed: klangkd's
caddy child kept crashing on startup with

  Error: loading initial config: loading new config: starting caddy
  administration endpoint: listen tcp 127.0.0.1:2019: bind: address
  already in use

(https://github.com/mcdonc/klangk/actions/runs/29846484392/job/88689613725)

Root cause: the Debian/Ubuntu caddy package ships a systemd unit that
autostarts on install. apt-get install caddy created
/etc/systemd/system/multi-user.target.wants/caddy.service and started
caddy binding 127.0.0.1:2019 (the caddy default admin port). klangkd
correctly bootstraps its own child with CADDY_ADMIN=unix//<sock>, but
caddy's INITIAL config-load still tries to bind 2019 before applying the
new admin address — so the system service sitting on 2019 makes every
klangkd-managed caddy fail to start.

Fix: stop, disable, and mask caddy.service after the apt install. The
apt-installed caddy is now just a binary on PATH, which is what klangkd's
contract expects (it owns its own child process; no coexisting system
service). This mirrors how the production host container does it — caddy
runs as a klangkd-supervised child, not as an independent service.
@mcdonc mcdonc mentioned this pull request Jul 21, 2026
Merged
mcdonc added a commit that referenced this pull request Jul 21, 2026
@mcdonc
feat(1611): split dist-smoke into its own non-blocking workflow (#1611)…
a4faff4 
… (#1710)

The dist-smoke-test gate is currently red on #1709 (klangkd's caddy child
can't coexist with another caddy / binds 2019 on bootstrap instead of the
UDS) — a klangkd bug, not a wheel-packaging bug. Holding the release
pipeline hostage to the smoke being green means no release can happen
until #1709 is fixed, which is the wrong coupling.

Restructure:

- New .github/workflows/dist-smoke.yml — standalone workflow, workflow_dispatch
  only, two jobs (build-wheel → dist-smoke-test). No publish. This is now a
  release-engineering tool the release manager runs against a release-
  candidate branch before tagging (`gh workflow run dist-smoke.yml --ref <branch>`).

- Revert .github/workflows/release.yml to its pre-#1705 shape:
  build-and-release + build-wheel (env: pypi, publishes) run in parallel,
  no smoke in the chain. The "never yank" invariant now holds by workflow:
  the release manager runs dist-smoke.yml against the RC branch first, and
  only tags once it's green (or after manually assessing any failures).

- scripts/dist-smoke-test.sh header: updated the CI-without-tagging pointer
  to reference dist-smoke.yml instead of release.yml's workflow_dispatch.

Also notes in dist-smoke.yml that the smoke is currently expected to fail
until #1709 lands, so a maintainer running it isn't surprised.

Follow-up to #1705, #1707, #1708.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Release smoke test: Playwright verifies login page renders from installed wheel via nginx

1 participant

@mcdonc