Fix the self-hosted APT repo publish workflow - #15164
Open
rtibbles wants to merge 4 commits into
Open
Conversation
KOLIBRI_PUBLIC_RELEASE_GCS_BUCKET holds a bare bucket name, so the "<bucket>/apt" the workflow passed never matched publish.sh's gs:// branch. Every publish rsynced into a directory of that name on the runner and exited 0 — a green job that shipped nothing. publish.sh now derives gs://<bucket>/downloads/kolibri/apt itself. The local-directory branch it fell through to only ever ran under test, so the tests proved nothing about the path CI takes. It is gone, and the tests now stub `gcloud` at the CLI boundary and drive the real one. The stub rejects flags gcloud does not have, which is what the up-leg was passing: `-d` does not exist and `-c` is --continue-on-error, not a checksum compare, so it would have aborted before writing anything. Publishes are also idempotent now — a version already in the repo is kept rather than re-included, since reprepro hard-errors on same-version bytes that differ and CI rebuilds the keyring .deb every run. Cache-Control is set per prefix so a cached index is never served against a newer pool. build-essential rounds it off: it is not on the runner image, and dpkg-checkbuilddeps demands it even for the arch-all keyring package.
The imported GPG_SIGNING_KEY is Learning Equality's general signing key. The repo has to be signed by the Debian repository key instead: that is what kolibri-archive-keyring ships, what the Pi trusts, and what signs the Pages repos this one replaces. Signed by anything else, every client pinning the shipped keyring with Signed-By rejects the repo, and nothing in the pipeline notices — reprepro signs happily with whatever key it is handed. Take DEBIAN_REPO_SIGNING_KEY, the secret the kolibri-installer-debian Pages publish already uses, with that workflow's keyring layout: an isolated GNUPGHOME, loopback pinentry and an ownertrust import. It carries no passphrase, so the preset-passphrase dance goes with it. The key id comes from the committed public key rather than a secret or a variable, so the id cannot drift from the key we ship, and the job aborts up front if the secret does not hold it.
Both Pages repos being replaced advertise amd64, i386, arm64 and armhf, and those clients are real: the published wheel bundles C extensions for armv6l, armv7l and i686, and the .deb is Architecture: all. reprepro only lists an arch-all package under the architectures a suite declares, so migrating a 32-bit host to a suite declaring amd64 and arm64 leaves it with a repo it can read but no package it can install.
github-actions
Bot
added
DEV: dev-ops
Contributor
Build Artifacts
Smoke test screenshot |
roles/storage.objectCreator, which the release uploads run on, cannot overwrite: a GCS overwrite is a delete plus a create. Publishing into an empty prefix therefore succeeds, and every publish after it fails on the indices with the pool already ahead of the metadata. Convert the README to Markdown while here, matching the rest of platforms/, and drop two stale claims: the signing key needs no passphrase preset, and the host it says ops has yet to provision is serving.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a range of issues with the apt repo publishing workflow:
References
Fixes #15149
Follow-up to: #15054
Ops prerequisites: #13720
Reviewer guidance
Manually tested by dispatching this branch against the live bucket, twice:
kolibriinto the empty prefix: https://github.com/learningequality/kolibri/actions/runs/31230971790. A freshubuntu:24.04then bootstrapped from the repo root deb, verifiedInReleaseagainst the shipped keyring, and installedkolibri 0.19.5-0ubuntu1(kolibri, version 0.19.5).kolibri-serverinto the now-populated prefix, exercising the read-modify-write: https://github.com/learningequality/kolibri/actions/runs/31231573658. Same container flow installedkolibri-server 0.5.1-0ubuntu1, and the suite still lists all three packages — checked inbinary-armhfas well asbinary-amd64.objectCreator, so thepoolcreates succeeded and everydistsoverwrite was denied. The published index stayed valid and signed throughout, and once the grant was added the retry healed the orphan pool object with no manual cleanup — the pool-first ordering behaving as intended.Left for a reviewer to weigh:
platforms/apt-repo/publish.sh:63—include_debkeeps the published build when that version already exists, so a rebuilt artifact would leave the repo serving bytes that differ from the release asset; check that skipping is preferable to failing the job.AI usage
Used Claude Code to diagnose the failing publish job, write the fixes and build the gcloud test double. Verified with the apt-repo shell test suite in a Debian container, prek, and gcloud's own argument parsing for the flag changes.