From f06ecd31e20b84ea7408474a27534e64191fbaea Mon Sep 17 00:00:00 2001 From: Richard Tibbles Date: Fri, 7 Aug 2026 17:03:08 -0700 Subject: [PATCH 1/4] Publish the APT repo to the bucket instead of a runner directory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit KOLIBRI_PUBLIC_RELEASE_GCS_BUCKET holds a bare bucket name, so the "/apt" the workflow passed never matched publish.sh's gs:// branch. Every publish rsynced into a directory of that name on the runner and exited 0 — a green job that shipped nothing. publish.sh now derives gs:///downloads/kolibri/apt itself. The local-directory branch it fell through to only ever ran under test, so the tests proved nothing about the path CI takes. It is gone, and the tests now stub `gcloud` at the CLI boundary and drive the real one. The stub rejects flags gcloud does not have, which is what the up-leg was passing: `-d` does not exist and `-c` is --continue-on-error, not a checksum compare, so it would have aborted before writing anything. Publishes are also idempotent now — a version already in the repo is kept rather than re-included, since reprepro hard-errors on same-version bytes that differ and CI rebuilds the keyring .deb every run. Cache-Control is set per prefix so a cached index is never served against a newer pool. build-essential rounds it off: it is not on the runner image, and dpkg-checkbuilddeps demands it even for the arch-all keyring package. --- .../workflows/platform-apt-repo-publish.yml | 4 +- platforms/apt-repo/README.rst | 17 ++- platforms/apt-repo/publish.sh | 112 ++++++++++-------- platforms/apt-repo/tests/lib.sh | 92 ++++++++++++++ .../tests/test_publish_empty_prefix_guard.sh | 51 ++------ .../apt-repo/tests/test_publish_roundtrip.sh | 13 +- 6 files changed, 186 insertions(+), 103 deletions(-) diff --git a/.github/workflows/platform-apt-repo-publish.yml b/.github/workflows/platform-apt-repo-publish.yml index a55ccc1c726..d21616f799f 100644 --- a/.github/workflows/platform-apt-repo-publish.yml +++ b/.github/workflows/platform-apt-repo-publish.yml @@ -49,7 +49,7 @@ jobs: - name: Install reprepro and Debian packaging tools run: | sudo apt-get update - sudo apt-get install -y reprepro dpkg-dev debhelper + sudo apt-get install -y reprepro dpkg-dev debhelper build-essential - name: Import the GPG signing key and preset its passphrase id: gpg env: @@ -86,7 +86,7 @@ jobs: run: dpkg-buildpackage -b -us -uc - name: Publish the .deb into the reprepro repo env: - KOLIBRI_APT_BUCKET: "${{ secrets.KOLIBRI_PUBLIC_RELEASE_GCS_BUCKET }}/apt" + KOLIBRI_APT_BUCKET: ${{ secrets.KOLIBRI_PUBLIC_RELEASE_GCS_BUCKET }} REPREPRO_SIGN_KEY: ${{ steps.gpg.outputs.key-id }} run: | KOLIBRI_KEYRING_DEB=$(ls "$GITHUB_WORKSPACE"/platforms/apt-repo/kolibri-archive-keyring_*.deb) diff --git a/platforms/apt-repo/README.rst b/platforms/apt-repo/README.rst index 1414a3e1174..3c3e87d8db7 100644 --- a/platforms/apt-repo/README.rst +++ b/platforms/apt-repo/README.rst @@ -11,14 +11,21 @@ Publishing model ---------------- The repo is a `reprepro `_ tree that -lives on the release GCS bucket under its ``apt/`` path. Each release does a -**read-modify-write** so prior packages and versions persist (a ``kolibri``-only -release leaves the existing ``kolibri-server`` package live): +lives on the release GCS bucket under ``downloads/kolibri/apt``, alongside the +release downloads. Each release does a **read-modify-write** so prior packages +and versions persist (a ``kolibri``-only release leaves the existing +``kolibri-server`` package live): 1. ``gcloud storage rsync`` the full repo tree **down** from the bucket. -2. ``reprepro includedeb stable`` the new ``.deb``\(s). +2. ``reprepro includedeb stable`` the new ``.deb``\(s), skipping any version + already published — a published version is immutable, and reprepro rejects + same-version bytes that differ. 3. Export the served ``pubkey.asc`` from the signing key. -4. ``gcloud storage rsync -c -d`` the tree back **up**. +4. Sync back **up** a prefix at a time, ``pool`` first so no index is published + naming a file that is not there yet. ``--checksums-only`` because reprepro + rewrites ``db/*.db`` in place without changing size or mtime, and + ``Cache-Control`` per prefix: pool files never change once published, while a + cached index served against a newer pool is a client-side hash mismatch. ``publish.sh`` implements this; its header documents the env contract. ``conf/distributions.in`` is the suite config template, with the ``SignWith`` diff --git a/platforms/apt-repo/publish.sh b/platforms/apt-repo/publish.sh index 6168dc4de34..3ca342c252f 100755 --- a/platforms/apt-repo/publish.sh +++ b/platforms/apt-repo/publish.sh @@ -5,7 +5,9 @@ # Usage: publish.sh DEB_PATH [DEB_PATH...] # # Env: -# KOLIBRI_APT_BUCKET repo root — "gs://.../apt" in CI, a local dir in tests (required) +# KOLIBRI_APT_BUCKET GCS bucket name; the repo lives under its +# downloads/kolibri/apt prefix, alongside the release +# downloads (required) # REPREPRO_SIGN_KEY key id/fingerprint for SignWith + pubkey.asc export (required) # KOLIBRI_KEYRING_DEB optional path to the kolibri-archive-keyring .deb; when set it # is includedeb'd AND copied to the bucket root as the version-less @@ -16,9 +18,11 @@ if [ "$#" -lt 1 ]; then echo "usage: publish.sh DEB_PATH [DEB_PATH...]" >&2 exit 2 fi -: "${KOLIBRI_APT_BUCKET:?KOLIBRI_APT_BUCKET must be set (repo root: gs://.../apt or a local dir)}" +: "${KOLIBRI_APT_BUCKET:?KOLIBRI_APT_BUCKET must be set (GCS bucket name)}" : "${REPREPRO_SIGN_KEY:?REPREPRO_SIGN_KEY must be set (signing key id/fingerprint)}" +REPO_ROOT="gs://$KOLIBRI_APT_BUCKET/downloads/kolibri/apt" + HERE=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) WORKDIR=$(mktemp -d) @@ -26,9 +30,9 @@ trap 'rm -rf "$WORKDIR"' EXIT mkdir -p "$WORKDIR/repo/conf" # --- sync down (full tree) -------------------------------------------------- -# The whole tree must be fetched so the `-d` up-leg does not delete state we -# never saw. A masked download failure followed by `-c -d` would wipe the live -# repo, so only a genuinely empty prefix may skip the down-leg. +# The whole tree must be fetched so the deleting up-leg does not delete state we +# never saw. A masked download failure followed by the deleting up-leg would +# wipe the live repo, so only a genuinely empty prefix may skip the down-leg. # # `gcloud storage ls` exits non-zero both for a genuinely empty prefix and for # a transient network/throttle/auth failure — exit status alone cannot tell @@ -36,31 +40,38 @@ mkdir -p "$WORKDIR/repo/conf" # (sync down); the unambiguous "matched no objects" error means the prefix is # empty (skip); any other failure is fatal and must abort *before* the # destructive up-leg rather than fall through to "starting fresh". -case "$KOLIBRI_APT_BUCKET" in - gs://*) - if ls_err=$(gcloud storage ls "$KOLIBRI_APT_BUCKET/**" 2>&1 >/dev/null); then - gcloud storage rsync -r "$KOLIBRI_APT_BUCKET" "$WORKDIR/repo" - elif printf '%s' "$ls_err" | grep -qiF 'matched no objects'; then - echo "empty prefix — first publish, starting fresh" - else - echo "aborting: cannot list $KOLIBRI_APT_BUCKET to confirm it is empty" >&2 - printf '%s\n' "$ls_err" >&2 - exit 1 - fi - ;; - *) - mkdir -p "$KOLIBRI_APT_BUCKET" - cp -a "$KOLIBRI_APT_BUCKET/." "$WORKDIR/repo/" 2>/dev/null || true - ;; -esac +if ls_err=$(gcloud storage ls "$REPO_ROOT/**" 2>&1 >/dev/null); then + gcloud storage rsync -r "$REPO_ROOT" "$WORKDIR/repo" +elif printf '%s' "$ls_err" | grep -qiF 'matched no objects'; then + echo "empty prefix — first publish, starting fresh" +else + echo "aborting: cannot list $REPO_ROOT to confirm it is empty" >&2 + printf '%s\n' "$ls_err" >&2 + exit 1 +fi # --- render conf (committed template always wins) --------------------------- sed "s/__REPREPRO_SIGN_KEY__/$REPREPRO_SIGN_KEY/" \ "$HERE/conf/distributions.in" > "$WORKDIR/repo/conf/distributions" # --- add the release .deb(s) ------------------------------------------------ +# A published version is immutable, and reprepro refuses a same-version .deb whose +# bytes differ with a hard error. That would fail any re-publish of a rebuilt +# artifact — CI rebuilds the keyring .deb every run, and a re-run of a release job +# rebuilds its .deb too. Keep what is already published and say so; ship changes by +# bumping the version. +include_deb() { + pkg=$(dpkg-deb -f "$1" Package) + ver=$(dpkg-deb -f "$1" Version) + if reprepro -b "$WORKDIR/repo" list stable "$pkg" | grep -qF "$pkg $ver"; then + echo "$pkg $ver already published — keeping the published build" + else + reprepro -b "$WORKDIR/repo" includedeb stable "$1" + fi +} + for deb in "$@"; do - reprepro -b "$WORKDIR/repo" includedeb stable "$deb" + include_deb "$deb" done # --- export the served public key ------------------------------------------- @@ -68,33 +79,34 @@ gpg --armor --export "$REPREPRO_SIGN_KEY" > "$WORKDIR/repo/pubkey.asc" # --- bootstrap deb at the repo root (optional) ------------------------------ if [ -n "${KOLIBRI_KEYRING_DEB:-}" ]; then - # The keyring version is fixed across releases, but every CI run rebuilds the - # .deb, so its bytes differ each time. A blind re-includedeb would hit - # reprepro's same-version-different-checksum refusal (a hard error) and fail - # every release after the first. Only add it when this version isn't already - # published; bump the keyring version to ship a change. - keyring_pkg=$(dpkg-deb -f "$KOLIBRI_KEYRING_DEB" Package) - keyring_ver=$(dpkg-deb -f "$KOLIBRI_KEYRING_DEB" Version) - if reprepro -b "$WORKDIR/repo" list stable "$keyring_pkg" \ - | grep -qF "$keyring_pkg $keyring_ver"; then - echo "$keyring_pkg $keyring_ver already published — skipping keyring includedeb" - else - reprepro -b "$WORKDIR/repo" includedeb stable "$KOLIBRI_KEYRING_DEB" - fi + include_deb "$KOLIBRI_KEYRING_DEB" cp "$KOLIBRI_KEYRING_DEB" "$WORKDIR/repo/kolibri-archive-keyring.deb" fi -# --- sync up (-c -d) -------------------------------------------------------- -case "$KOLIBRI_APT_BUCKET" in - gs://*) - gcloud storage rsync -r -c -d "$WORKDIR/repo" "$KOLIBRI_APT_BUCKET" - ;; - *) - # --checksum (mirroring the gs:// leg's -c) is required, not cosmetic: - # reprepro rewrites db/*.db in place, often without changing size or mtime - # (fixed-size BDB pages, sub-second rewrite). rsync's default size+mtime - # quick-check then skips the modified db, leaving the published db stale - # against the freshly-exported pool/dists — a silently inconsistent repo. - rsync -a --delete --checksum "$WORKDIR/repo/" "$KOLIBRI_APT_BUCKET/" - ;; -esac +# --- sync up (checksum compare, delete extras) ------------------------------ +# --checksums-only is required, not cosmetic: reprepro rewrites db/*.db in place, +# often without changing size or mtime (fixed-size BDB pages, sub-second rewrite). +# A default mtime+size comparison then skips the modified db, leaving the +# published db stale against the freshly-exported pool/dists — a silently +# inconsistent repo. +# +# Cache-Control is set per prefix, not once for the tree: a cached index served +# against a newer pool is a Hash Sum mismatch for the client, while pool files +# never change once published. +sync_up() { + gcloud storage rsync -r --checksums-only --delete-unmatched-destination-objects \ + --cache-control="$2" "$WORKDIR/repo/$1" "$REPO_ROOT/$1" +} + +# pool goes first, so no index is ever published naming a file that is not there. +sync_up pool 'public, max-age=2592000' +sync_up dists 'no-cache' +# reprepro's own state: read back by the next publish, never by a client. +sync_up db 'no-cache' +sync_up conf 'no-cache' + +# Root files are rewritten in place rather than versioned, so they stay uncached. +for root_file in "$WORKDIR"/repo/*; do + [ -f "$root_file" ] || continue + gcloud storage cp --cache-control='no-cache' "$root_file" "$REPO_ROOT/${root_file##*/}" +done diff --git a/platforms/apt-repo/tests/lib.sh b/platforms/apt-repo/tests/lib.sh index 01fc98ec201..e1114742107 100644 --- a/platforms/apt-repo/tests/lib.sh +++ b/platforms/apt-repo/tests/lib.sh @@ -63,6 +63,98 @@ setup_workdir() { trap 'gpgconf --kill gpg-agent >/dev/null 2>&1 || true; rm -rf "$WORK"' EXIT } +# fake_gcloud — put a stub `gcloud` on PATH that serves the `gcloud storage` +# subset publish.sh uses out of a local tree, so the tests drive publish.sh's +# real (and only) gs:// path without touching GCS. Buckets live under +# $FAKE_GCS_ROOT//…; every rsync invocation is appended to +# $FAKE_GCS_RSYNC_LOG. Unknown flags are rejected, as the real CLI does, so a +# flag that only exists in gsutil fails the test rather than passing silently. +# Export FAKE_LS_MODE=transient to make `storage ls` fail the way a throttled or +# unauthenticated call does. +fake_gcloud() { + FAKE_GCS_ROOT="$WORK/gcs" + FAKE_GCS_RSYNC_LOG="$WORK/gcloud_rsync.log" + mkdir -p "$FAKE_GCS_ROOT" "$WORK/bin" + : > "$FAKE_GCS_RSYNC_LOG" + export FAKE_GCS_ROOT FAKE_GCS_RSYNC_LOG + cat > "$WORK/bin/gcloud" <<'EOF' +#!/bin/sh +set -eu +no_objects() { + echo "ERROR: (gcloud.storage.ls) One or more URLs matched no objects." >&2 + exit 1 +} +# gs://bucket/path -> $FAKE_GCS_ROOT/bucket/path; local paths pass through. +localise() { + case "$1" in + gs://*) echo "$FAKE_GCS_ROOT/${1#gs://}" ;; + *) echo "$1" ;; + esac +} + +[ "${1:-}" = storage ] || { echo "fake gcloud: unhandled args: $*" >&2; exit 100; } +cmd=$2 +shift 2 + +case "$cmd" in + ls) + [ "${FAKE_LS_MODE:-}" != transient ] || { + echo "ERROR: (gcloud.storage.ls) HTTPError 503: The service is currently unavailable." >&2 + exit 1 + } + dir=$(localise "${1%"/**"}") + [ -d "$dir" ] || no_objects + found=$(find "$dir" -type f) + [ -n "$found" ] || no_objects + echo "$found" + ;; + rsync) + echo "rsync $*" >> "$FAKE_GCS_RSYNC_LOG" + delete=0 + src="" + dst="" + for arg in "$@"; do + case "$arg" in + -r|--recursive|--checksums-only|--cache-control=*) ;; + --delete-unmatched-destination-objects) delete=1 ;; + -*) echo "ERROR: (gcloud.storage.rsync) unrecognized arguments: $arg" >&2; exit 2 ;; + *) if [ -z "$src" ]; then src=$arg; else dst=$arg; fi ;; + esac + done + src=$(localise "$src") + dst=$(localise "$dst") + mkdir -p "$src" "$dst" + if [ "$delete" -eq 1 ]; then + rsync -a --delete "$src/" "$dst/" + else + rsync -a "$src/" "$dst/" + fi + ;; + cp) + src="" + dst="" + for arg in "$@"; do + case "$arg" in + --cache-control=*) ;; + -*) echo "ERROR: (gcloud.storage.cp) unrecognized arguments: $arg" >&2; exit 2 ;; + *) if [ -z "$src" ]; then src=$arg; else dst=$arg; fi ;; + esac + done + dst=$(localise "$dst") + mkdir -p "${dst%/*}" + cp "$(localise "$src")" "$dst" + ;; + *) + echo "fake gcloud: unhandled storage command: $cmd" >&2 + exit 100 + ;; +esac +EOF + chmod +x "$WORK/bin/gcloud" + PATH="$WORK/bin:$PATH" + export PATH +} + # make_ephemeral_signing_key — generate an unprotected ed25519 key under # $WORK/gnupg; set GNUPGHOME + FPR (its fingerprint). %no-protection stops # reprepro's non-interactive Release signing hanging on a pinentry prompt. diff --git a/platforms/apt-repo/tests/test_publish_empty_prefix_guard.sh b/platforms/apt-repo/tests/test_publish_empty_prefix_guard.sh index c33b976614e..1ceb3d521b4 100755 --- a/platforms/apt-repo/tests/test_publish_empty_prefix_guard.sh +++ b/platforms/apt-repo/tests/test_publish_empty_prefix_guard.sh @@ -1,9 +1,9 @@ #!/bin/sh # Guards the gs:// sync-down guard in publish.sh: a genuinely empty prefix may # skip the down-leg, but a transient `gcloud storage ls` failure must NOT be -# mistaken for "empty" — otherwise the destructive `-c -d` up-leg wipes the -# live repo. Both cases are driven through a fake `gcloud` on PATH so no real -# GCS access is needed. +# mistaken for "empty" — otherwise the destructive up-leg wipes the live repo. +# Both cases are driven through the fake `gcloud` from lib.sh, so no real GCS +# access is needed. set -eu HERE=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) @@ -14,43 +14,14 @@ require_tools reprepro gpg dpkg-deb rsync PUBLISH="$HERE/../publish.sh" setup_workdir - -# --- fake gcloud on PATH ---------------------------------------------------- -# `storage ls` -> behaviour picked by $FAKE_LS_MODE (empty|transient|objects) -# `storage rsync` -> record the invocation so we can assert whether the up-leg -# (the destructive `-c -d` leg) ever ran. -BIN="$WORK/bin" -mkdir -p "$BIN" -RSYNC_LOG="$WORK/gcloud_rsync.log" -cat > "$BIN/gcloud" <&2; exit 1 ;; - transient) echo "ERROR: (gcloud.storage.ls) HTTPError 503: The service is currently unavailable." >&2; exit 1 ;; - *) echo "fake gcloud: FAKE_LS_MODE unset" >&2; exit 99 ;; - esac -fi -if [ "\$1" = "storage" ] && [ "\$2" = "rsync" ]; then - echo "rsync \$*" >> "$RSYNC_LOG" - exit 0 -fi -echo "fake gcloud: unhandled args: \$*" >&2 -exit 100 -EOF -chmod +x "$BIN/gcloud" -PATH="$BIN:$PATH" -export PATH - +fake_gcloud make_ephemeral_signing_key export REPREPRO_SIGN_KEY="$FPR" build_min_deb aaa-test -export KOLIBRI_APT_BUCKET="gs://fake-bucket/apt" +export KOLIBRI_APT_BUCKET="fake-bucket" # --- transient failure: must abort before the destructive up-leg ------------ -: > "$RSYNC_LOG" set +e FAKE_LS_MODE=transient bash "$PUBLISH" "$WORK/aaa-test_1.0_all.deb" >"$WORK/transient.out" 2>&1 rc=$? @@ -59,16 +30,16 @@ set -e assert_contains "$WORK/transient.out" 'aborting: cannot list' \ "transient failure did not abort with the guard message" # The up-leg is the only rsync that would delete state; it must never run. -[ ! -s "$RSYNC_LOG" ] \ - || fail "transient ls failure still reached a gcloud rsync leg: $(cat "$RSYNC_LOG")" +[ ! -s "$FAKE_GCS_RSYNC_LOG" ] \ + || fail "transient ls failure still reached a gcloud rsync leg: $(cat "$FAKE_GCS_RSYNC_LOG")" # --- genuinely empty prefix: skip down-leg, still publish + run up-leg ------- -: > "$RSYNC_LOG" -FAKE_LS_MODE=empty bash "$PUBLISH" "$WORK/aaa-test_1.0_all.deb" >"$WORK/empty.out" 2>&1 \ +# No FAKE_LS_MODE: the bucket really is empty, so `storage ls` reports it. +bash "$PUBLISH" "$WORK/aaa-test_1.0_all.deb" >"$WORK/empty.out" 2>&1 \ || { cat "$WORK/empty.out"; fail "empty prefix should publish cleanly, not abort"; } assert_contains "$WORK/empty.out" 'empty prefix' \ "empty prefix was not detected as a first publish" -assert_contains "$RSYNC_LOG" '^rsync .* -d ' \ - "empty-prefix publish never reached the -d up-leg" +assert_contains "$FAKE_GCS_RSYNC_LOG" '^rsync .* --delete-unmatched-destination-objects ' \ + "empty-prefix publish never reached the deleting up-leg" echo "PASS: transient ls failure aborts before the up-leg; empty prefix publishes fresh" diff --git a/platforms/apt-repo/tests/test_publish_roundtrip.sh b/platforms/apt-repo/tests/test_publish_roundtrip.sh index c62491f0ec2..9c3406e8c95 100755 --- a/platforms/apt-repo/tests/test_publish_roundtrip.sh +++ b/platforms/apt-repo/tests/test_publish_roundtrip.sh @@ -1,7 +1,8 @@ #!/bin/sh # AC#1/#2/#3 harness for publish.sh — proves the read-modify-write publish -# preserves prior packages, signs the Release, and serves pubkey.asc, using -# only local tooling (no GCS). +# preserves prior packages, signs the Release, and serves pubkey.asc. The bucket +# is served by the fake `gcloud` from lib.sh, so this drives publish.sh's real +# gs:// path with no GCS access. set -eu HERE=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) @@ -12,6 +13,7 @@ require_tools reprepro gpg dpkg-deb rsync PUBLISH="$HERE/../publish.sh" setup_workdir +fake_gcloud make_ephemeral_signing_key export REPREPRO_SIGN_KEY="$FPR" @@ -19,10 +21,9 @@ export REPREPRO_SIGN_KEY="$FPR" build_min_deb aaa-test build_min_deb bbb-test -# --- fake "bucket" = empty local dir ---------------------------------------- -BUCKET="$WORK/bucket" -mkdir -p "$BUCKET" -export KOLIBRI_APT_BUCKET="$BUCKET" +# --- the repo root publish.sh derives from the bucket name ------------------ +export KOLIBRI_APT_BUCKET="fake-bucket" +BUCKET="$FAKE_GCS_ROOT/fake-bucket/downloads/kolibri/apt" # --- two sequential publishes ----------------------------------------------- bash "$PUBLISH" "$WORK/aaa-test_1.0_all.deb" From 7f110e05c192fe3a83e42fffe1e21b2e5cc9c766 Mon Sep 17 00:00:00 2001 From: Richard Tibbles Date: Fri, 7 Aug 2026 17:04:40 -0700 Subject: [PATCH 2/4] Sign the APT repo with the Debian repo signing key MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The imported GPG_SIGNING_KEY is Learning Equality's general signing key. The repo has to be signed by the Debian repository key instead: that is what kolibri-archive-keyring ships, what the Pi trusts, and what signs the Pages repos this one replaces. Signed by anything else, every client pinning the shipped keyring with Signed-By rejects the repo, and nothing in the pipeline notices — reprepro signs happily with whatever key it is handed. Take DEBIAN_REPO_SIGNING_KEY, the secret the kolibri-installer-debian Pages publish already uses, with that workflow's keyring layout: an isolated GNUPGHOME, loopback pinentry and an ownertrust import. It carries no passphrase, so the preset-passphrase dance goes with it. The key id comes from the committed public key rather than a secret or a variable, so the id cannot drift from the key we ship, and the job aborts up front if the secret does not hold it. --- .../workflows/platform-apt-repo-publish.yml | 45 +++++++++---------- .../platform-debian-server-release.yml | 5 ++- .github/workflows/release_kolibri.yml | 1 + platforms/apt-repo/README.rst | 13 +++--- 4 files changed, 32 insertions(+), 32 deletions(-) diff --git a/.github/workflows/platform-apt-repo-publish.yml b/.github/workflows/platform-apt-repo-publish.yml index d21616f799f..c9d8d7125c4 100644 --- a/.github/workflows/platform-apt-repo-publish.yml +++ b/.github/workflows/platform-apt-repo-publish.yml @@ -12,9 +12,7 @@ on: required: true type: string secrets: - GPG_SIGNING_KEY: - required: true - GPG_PASSPHRASE: + DEBIAN_REPO_SIGNING_KEY: required: true GH_UPLOADER_GCP_SA_CREDENTIALS: required: true @@ -50,29 +48,28 @@ jobs: run: | sudo apt-get update sudo apt-get install -y reprepro dpkg-dev debhelper build-essential - - name: Import the GPG signing key and preset its passphrase + - name: Import the Debian repo signing key id: gpg - env: - GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }} run: | - echo -n "${{ secrets.GPG_SIGNING_KEY }}" | base64 --decode | gpg --import --no-tty --batch --yes - # Derive the signing key id from the imported key (no separate secret), - # mirroring the release workflow. - GPG_KEY_ID=$(gpg --list-secret-keys --keyid-format long --with-colons | grep ^sec | head -1 | cut -d: -f5) + # Same key, keyring layout and loopback pinentry as the release_ppa + # workflow in learningequality/kolibri-installer-debian, which publishes + # the Pages repos this one replaces. + GNUPGHOME=$(mktemp -d) + export GNUPGHOME + echo "GNUPGHOME=$GNUPGHOME" >> "$GITHUB_ENV" + echo "pinentry-mode loopback" > "$GNUPGHOME/gpg.conf" + echo "allow-loopback-pinentry" > "$GNUPGHOME/gpg-agent.conf" + echo "${{ secrets.DEBIAN_REPO_SIGNING_KEY }}" | gpg --batch --import + # The repo must be signed by the key kolibri-archive-keyring ships, or no + # client trusting that keyring can verify it. Take the key id from the + # committed key and require the imported secret key to match. + GPG_KEY_ID=$(gpg --show-keys --with-colons platforms/apt-repo/keyring/kolibri-archive-keyring.asc | awk -F: '/^fpr:/ {print $10; exit}') + gpg --list-secret-keys "$GPG_KEY_ID" >/dev/null 2>&1 || { + echo "::error::DEBIAN_REPO_SIGNING_KEY does not hold $GPG_KEY_ID, the key shipped in kolibri-archive-keyring.asc" + exit 1 + } + echo "$GPG_KEY_ID:6:" | gpg --batch --import-ownertrust echo "key-id=$GPG_KEY_ID" >> "$GITHUB_OUTPUT" - # reprepro signs Release via gpgme/gpg-agent (no CLI passphrase flag), - # so preset the passphrase or signing hangs on the pinentry prompt. - echo "allow-preset-passphrase" >> ~/.gnupg/gpg-agent.conf - gpg-connect-agent reloadagent /bye - # gpg-preset-passphrase ships in a version-specific libexec dir, not on PATH. - for candidate in /usr/lib/gnupg*/gpg-preset-passphrase; do - [ -x "$candidate" ] && PRESET="$candidate" && break - done - KEYGRIP=$(gpg --with-keygrip --list-secret-keys | grep Keygrip | head -1 | awk '{print $3}') - # gpg-preset-passphrase reads the passphrase from stdin only, so pipe - # it in: printf is a shell builtin, so the secret never reaches argv - # or the log, and no copy is written to disk. - printf '%s' "$GPG_PASSPHRASE" | "$PRESET" --preset "$KEYGRIP" - uses: "google-github-actions/auth@v3" with: credentials_json: "${{ secrets.GH_UPLOADER_GCP_SA_CREDENTIALS }}" @@ -96,4 +93,4 @@ jobs: if: always() run: | gpgconf --kill gpg-agent || true - rm -rf ~/.gnupg + rm -rf "$GNUPGHOME" diff --git a/.github/workflows/platform-debian-server-release.yml b/.github/workflows/platform-debian-server-release.yml index 9b59ee7f65d..1b34aa8455d 100644 --- a/.github/workflows/platform-debian-server-release.yml +++ b/.github/workflows/platform-debian-server-release.yml @@ -14,6 +14,8 @@ on: required: true GPG_PASSPHRASE: required: true + DEBIAN_REPO_SIGNING_KEY: + required: true GH_UPLOADER_GCP_SA_CREDENTIALS: required: true KOLIBRI_PUBLIC_RELEASE_GCS_BUCKET: @@ -266,7 +268,6 @@ jobs: with: deb-artifact-name: ${{ inputs.deb-artifact-name }} secrets: - GPG_SIGNING_KEY: ${{ secrets.GPG_SIGNING_KEY }} - GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }} + DEBIAN_REPO_SIGNING_KEY: ${{ secrets.DEBIAN_REPO_SIGNING_KEY }} GH_UPLOADER_GCP_SA_CREDENTIALS: ${{ secrets.GH_UPLOADER_GCP_SA_CREDENTIALS }} KOLIBRI_PUBLIC_RELEASE_GCS_BUCKET: ${{ secrets.KOLIBRI_PUBLIC_RELEASE_GCS_BUCKET }} diff --git a/.github/workflows/release_kolibri.yml b/.github/workflows/release_kolibri.yml index 620c2c031d5..996fc2b036c 100644 --- a/.github/workflows/release_kolibri.yml +++ b/.github/workflows/release_kolibri.yml @@ -249,6 +249,7 @@ jobs: LP_CREDENTIALS: ${{ secrets.LP_CREDENTIALS }} GPG_SIGNING_KEY: ${{ secrets.GPG_SIGNING_KEY }} GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }} + DEBIAN_REPO_SIGNING_KEY: ${{ secrets.DEBIAN_REPO_SIGNING_KEY }} GH_UPLOADER_GCP_SA_CREDENTIALS: ${{ secrets.GH_UPLOADER_GCP_SA_CREDENTIALS }} KOLIBRI_PUBLIC_RELEASE_GCS_BUCKET: ${{ secrets.KOLIBRI_PUBLIC_RELEASE_GCS_BUCKET }} container_image_publish: diff --git a/platforms/apt-repo/README.rst b/platforms/apt-repo/README.rst index 3c3e87d8db7..424a0a623d9 100644 --- a/platforms/apt-repo/README.rst +++ b/platforms/apt-repo/README.rst @@ -106,9 +106,10 @@ CI secrets, actually running ``seed_old_pages.sh`` for the cutover release, the readthedocs user-manual update, and archiving the old Pages repos are **admin/ops tasks tracked on #13720**, not code in this directory. -**Key-match assumption:** the private half of ``GPG_SIGNING_KEY`` must match the -committed client trust key ``platforms/raspberry-pi/files/learningequality.asc`` -(the byte-identical source of ``keyring/kolibri-archive-keyring.asc``). If the -self-hosted repo historically used a different key than the PPA, ops must -reconcile them before the cutover (reprepro fails loudly at publish rather than -shipping a silently-broken repo). +**Key match:** the private half of ``DEBIAN_REPO_SIGNING_KEY`` — the same secret +the ``kolibri-installer-debian`` Pages publish uses — must be the committed +client trust key ``platforms/raspberry-pi/files/learningequality.asc`` (the +byte-identical source of ``keyring/kolibri-archive-keyring.asc``), which is also +the key that signs the old Pages repos. The publish workflow derives the signing +key id from that committed key and aborts if the imported secret does not hold +it, rather than shipping a repo no client can verify. From 805d3ffd6321ffb948b748e1f7ead23c88082783 Mon Sep 17 00:00:00 2001 From: Richard Tibbles Date: Fri, 7 Aug 2026 17:05:14 -0700 Subject: [PATCH 3/4] Serve i386 and armhf from the APT repo Both Pages repos being replaced advertise amd64, i386, arm64 and armhf, and those clients are real: the published wheel bundles C extensions for armv6l, armv7l and i686, and the .deb is Architecture: all. reprepro only lists an arch-all package under the architectures a suite declares, so migrating a 32-bit host to a suite declaring amd64 and arm64 leaves it with a repo it can read but no package it can install. --- platforms/apt-repo/conf/distributions.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/platforms/apt-repo/conf/distributions.in b/platforms/apt-repo/conf/distributions.in index cb44ad69faf..2389a665aa8 100644 --- a/platforms/apt-repo/conf/distributions.in +++ b/platforms/apt-repo/conf/distributions.in @@ -2,7 +2,7 @@ Origin: Learning Equality Label: Kolibri Codename: stable Suite: stable -Architectures: amd64 arm64 +Architectures: amd64 i386 arm64 armhf Components: main Description: Kolibri self-hosted APT repository SignWith: __REPREPRO_SIGN_KEY__ From 630a78d629c9f696d87474608af6f67f27a69998 Mon Sep 17 00:00:00 2001 From: Richard Tibbles Date: Fri, 7 Aug 2026 18:27:35 -0700 Subject: [PATCH 4/4] Document the bucket permissions the publish needs roles/storage.objectCreator, which the release uploads run on, cannot overwrite: a GCS overwrite is a delete plus a create. Publishing into an empty prefix therefore succeeds, and every publish after it fails on the indices with the pool already ahead of the metadata. Convert the README to Markdown while here, matching the rest of platforms/, and drop two stale claims: the signing key needs no passphrase preset, and the host it says ops has yet to provision is serving. --- platforms/apt-repo/README.md | 58 +++++++++++++++++ platforms/apt-repo/README.rst | 115 ---------------------------------- platforms/apt-repo/publish.sh | 2 +- 3 files changed, 59 insertions(+), 116 deletions(-) create mode 100644 platforms/apt-repo/README.md delete mode 100644 platforms/apt-repo/README.rst diff --git a/platforms/apt-repo/README.md b/platforms/apt-repo/README.md new file mode 100644 index 00000000000..616b4709153 --- /dev/null +++ b/platforms/apt-repo/README.md @@ -0,0 +1,58 @@ +# Kolibri self-hosted APT repository + +Publishing infrastructure for `https://apt.learningequality.org/` (suite `stable`, component `main`), replacing the `learningequality.github.io/kolibri-server/` and `…/kolibri-installer-debian/` Pages repos. + +## Publishing model + +A [reprepro](https://salsa.debian.org/debian/reprepro) tree on the release GCS bucket under `downloads/kolibri/apt`. Each release read-modify-writes it, so prior packages and versions persist: + +1. `gcloud storage rsync` the tree **down**. +2. `reprepro includedeb stable` the new `.deb`(s), skipping any version already published — reprepro rejects same-version bytes that differ. +3. Export `pubkey.asc` from the signing key. +4. Sync **up** one prefix at a time: + - `pool` first, so no index is published naming a file that is not there yet. + - `--checksums-only`, because reprepro rewrites `db/*.db` in place without changing size or mtime. + - `Cache-Control` per prefix: pool objects never change once published, while a cached index served against a newer pool is a client-side hash mismatch. + +`publish.sh` implements this; its header documents the env contract. `conf/distributions.in` is the suite template, with `SignWith` rendered in at runtime. + +`.github/workflows/platform-apt-repo-publish.yml` runs it in CI, serialized by a static `concurrency` group so two releases cannot clobber the shared state mid-write. + +The workflow never builds the `.deb` it publishes: `release_kolibri.yml` builds `kolibri-server` and passes the artifact name down through `platform-debian-server-release.yml`. Dispatch it by hand with `deb-url` to publish an already-released `.deb`. + +## New-user install + +`kolibri-archive-keyring` (`keyring/`) ships the apt source file (`/etc/apt/sources.list.d/kolibri.sources`) and the signing key (`/usr/share/keyrings/kolibri-archive-keyring.asc`). The workflow serves it at the repo root, so a fresh host can bootstrap before it has apt configured: + +```sh +curl -fsSLO https://apt.learningequality.org/kolibri-archive-keyring.deb +sudo dpkg -i kolibri-archive-keyring.deb +sudo apt update && sudo apt install kolibri +``` + +## Self-migration of the installed base + +`migrate-apt-source.sh` defines `migrate_kolibri_apt_source()`, which rewrites any existing `github.io` Kolibri source under `/etc/apt/sources.list.d/` to `apt.learningequality.org`. It is idempotent, and a no-op when no such source is present (e.g. a Launchpad-PPA install). `kolibri-server`'s `postinst` calls it on `configure`, so existing installs migrate on their next `apt upgrade` with no user action. + +## seed_old_pages.sh — run once + +> **Warning:** a one-shot script for the cutover release only — the first release carrying the `postinst` migration snippet. It is not part of the ongoing release process. + +It pushes the cutover `.deb` into the old, soon-to-be-archived Pages repo(s), so stragglers still resolving the old URL receive it on their next upgrade and self-migrate. Each old repo is read-modify-written against its own committed `conf/`, and its `Release` re-signed with the key that repo's `SignWith` names, which must already be in the gpg keyring. + +```sh +./seed_old_pages.sh --deb path/to/kolibri-server__all.deb --repo kolibri-server +./seed_old_pages.sh --deb path/to/kolibri__all.deb --repo kolibri-installer-debian +``` + +## Verification + +Each script in `tests/` covers one acceptance criterion and names it in its header. CI runs them with `APT_REPO_TESTS_STRICT=1`, so a missing tool fails rather than skips; standalone on a dev box, each skips cleanly when its tooling is absent. + +`e2e_cutover.sh` is the full containerized cutover: an old-source client is auto-rewritten and fetches its next update from the new host. + +## Ops prerequisites (#13720) + +- The uploader service account needs `storage.objects.delete` and `storage.objects.update` under `downloads/kolibri/apt` — `roles/storage.objectCreator`, which the release uploads run on, is not enough. A GCS overwrite is a delete plus a create, and every publish after the first rewrites `dists/`, `db/` and the root files. Reads come from the bucket's public `allUsers` grant. +- `DEBIAN_REPO_SIGNING_KEY` must hold the private half of `platforms/raspberry-pi/files/learningequality.asc`, the key `keyring/kolibri-archive-keyring.asc` ships and the old Pages repos sign with. The workflow takes the key id from that committed key and aborts if the secret does not hold it. +- Standing up the subdomain (DNS + Cloudflare in front of the bucket), running `seed_old_pages.sh` for the cutover release, the readthedocs user-manual update, and archiving the old Pages repos. diff --git a/platforms/apt-repo/README.rst b/platforms/apt-repo/README.rst deleted file mode 100644 index 424a0a623d9..00000000000 --- a/platforms/apt-repo/README.rst +++ /dev/null @@ -1,115 +0,0 @@ -Kolibri self-hosted APT repository -================================== - -Publishing infrastructure for the Kolibri Debian/Ubuntu APT repo served at -``https://apt.learningequality.org/`` — one unified, backend-independent repo -(suite ``stable``, component ``main``) that replaces the per-installer -``learningequality.github.io/kolibri-server/`` and -``…/kolibri-installer-debian/`` GitHub Pages repos. - -Publishing model ----------------- - -The repo is a `reprepro `_ tree that -lives on the release GCS bucket under ``downloads/kolibri/apt``, alongside the -release downloads. Each release does a **read-modify-write** so prior packages -and versions persist (a ``kolibri``-only release leaves the existing -``kolibri-server`` package live): - -1. ``gcloud storage rsync`` the full repo tree **down** from the bucket. -2. ``reprepro includedeb stable`` the new ``.deb``\(s), skipping any version - already published — a published version is immutable, and reprepro rejects - same-version bytes that differ. -3. Export the served ``pubkey.asc`` from the signing key. -4. Sync back **up** a prefix at a time, ``pool`` first so no index is published - naming a file that is not there yet. ``--checksums-only`` because reprepro - rewrites ``db/*.db`` in place without changing size or mtime, and - ``Cache-Control`` per prefix: pool files never change once published, while a - cached index served against a newer pool is a client-side hash mismatch. - -``publish.sh`` implements this; its header documents the env contract. -``conf/distributions.in`` is the suite config template, with the ``SignWith`` -key rendered in at runtime. - -The reusable workflow ``.github/workflows/platform-apt-repo-publish.yml`` runs -this in CI, serialized by a static ``concurrency`` group so two releases can -never clobber the shared repo state mid-write. - -The workflow never builds the ``.deb`` it publishes: ``release_kolibri.yml`` -builds ``kolibri-server`` once and passes that artifact name down through -``platform-debian-server-release.yml``. It can also be dispatched by hand with -``deb-url`` — a URL to an already-released ``.deb`` — to test the workflow or to -populate the repo from an existing release. - -New-user install ------------------ - -The ``kolibri-archive-keyring`` package (``keyring/``) ships the apt source file -(``/etc/apt/sources.list.d/kolibri.sources``) and the signing key -(``/usr/share/keyrings/kolibri-archive-keyring.asc``), replacing the manual -``curl``/``echo`` recipe. The publish workflow serves it at the repo root so a -fresh host can bootstrap before it has apt configured:: - - curl -fsSLO https://apt.learningequality.org/kolibri-archive-keyring.deb - sudo dpkg -i kolibri-archive-keyring.deb - sudo apt update && sudo apt install kolibri - -Self-migration of the existing installed base ---------------------------------------------- - -``migrate-apt-source.sh`` defines ``migrate_kolibri_apt_source()``, the cutover -snippet that rewrites any existing ``github.io`` Kolibri source under -``/etc/apt/sources.list.d/`` to ``apt.learningequality.org``. It is idempotent -and a no-op when no such source is present (e.g. a Launchpad-PPA install). -``kolibri-server``'s ``postinst`` sources and calls it on ``configure``; the -``kolibri`` package reuses the same snippet in its own migration (out of scope -here). Because the source rides with an auto-upgrading package, existing -installs migrate on their next ``apt upgrade`` with no user action. - -seed_old_pages.sh — RUN ONCE ----------------------------- - -.. warning:: - - ``seed_old_pages.sh`` is a **one-shot** script for the **cutover release - only** — run it exactly once, for the first release carrying the ``postinst`` - migration snippet. It is **not** part of the ongoing release process. - -It pushes the cutover ``.deb`` into the old (soon-to-be-archived) Pages repo(s) -so stragglers still resolving the old URL receive it on their next upgrade and -self-migrate. It read-modify-writes each old repo's own committed ``conf/`` (so -prior packages persist) and re-signs ``Release`` with the key named by that -repo's own ``SignWith`` — the matching secret key must be imported into the gpg -keyring first (with its passphrase preset, as the publish workflow does):: - - ./seed_old_pages.sh --deb path/to/kolibri-server__all.deb --repo kolibri-server - ./seed_old_pages.sh --deb path/to/kolibri__all.deb --repo kolibri-installer-debian - -Verification ------------- - -Each script in ``tests/`` covers one acceptance criterion and states which in -its header. They run in CI (``.github/workflows/platform-apt-repo-test.yml``, -with ``APT_REPO_TESTS_STRICT=1`` so a missing tool fails rather than skips) and -standalone on a dev box, where each skips cleanly when its tooling is absent. - -``e2e_cutover.sh`` is the full containerized cutover: an old-source client is -auto-rewritten and fetches its next update from the new host. To reproduce the -real Debian 13 check, run it against the live ``apt.learningequality.org`` once -ops has provisioned the host. - -Ops prerequisites — out of scope here (tracked on #13720) ---------------------------------------------------------- - -Standing up the subdomain (DNS + Cloudflare in front of the GCS bucket), the -CI secrets, actually running ``seed_old_pages.sh`` for the cutover release, the -readthedocs user-manual update, and archiving the old Pages repos are **admin/ops -tasks tracked on #13720**, not code in this directory. - -**Key match:** the private half of ``DEBIAN_REPO_SIGNING_KEY`` — the same secret -the ``kolibri-installer-debian`` Pages publish uses — must be the committed -client trust key ``platforms/raspberry-pi/files/learningequality.asc`` (the -byte-identical source of ``keyring/kolibri-archive-keyring.asc``), which is also -the key that signs the old Pages repos. The publish workflow derives the signing -key id from that committed key and aborts if the imported secret does not hold -it, rather than shipping a repo no client can verify. diff --git a/platforms/apt-repo/publish.sh b/platforms/apt-repo/publish.sh index 3ca342c252f..c4883f08722 100755 --- a/platforms/apt-repo/publish.sh +++ b/platforms/apt-repo/publish.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Read-modify-write publish of the Kolibri self-hosted APT repo. -# See README.rst for the publishing model. +# See README.md for the publishing model. # # Usage: publish.sh DEB_PATH [DEB_PATH...] #