Repository navigation

🐛 fix(deploy): reflect intercepted 403/404 from the ingress error backend instead of answering 503 #2964

Workflow file for this run

.github/workflows/podman-rootful-lane.yml at 391d212

	name: Podman Rootful Lane

	# The rootful Podman lane on hosted amd64 (#4335), lane 2 of the map in
	# src/docs/podman-ci-runner-map.md (#4211).
	#
	# #4200 established the rootful baseline and shipped
	# src/deploy/probe_podman_rootful_netadmin.sh, but nothing ran it per PR.
	# Rootful is the reference the rootless result is measured against, so a silent
	# regression here invalidates that comparison. This workflow is the wrapper
	# that makes the existing probe run on every pull request; the probe itself is
	# not modified by this lane.
	#
	# The three cases it runs, all as real root:
	#
	# default rootful, no added capability -> fail closed, exit 77
	# netadmin rootful + --cap-add NET_ADMIN -> egress redirect installed
	# advisory HIVE_PROXY_ADVISORY_OK=true -> starts with the explicit warning
	#
	# The probe exits non-zero when a case stops matching the contract recorded in
	# src/docs/podman-rootful-egress-baseline.md, and this job does not swallow
	# that status.
	#
	# WHAT A GREEN TICK HERE DOES AND DOES NOT MEAN. The probe pulls a published
	# image, so this lane asserts that the published image's egress gate still
	# behaves under rootful Podman and that the runner still provides a working
	# rootful stack. It does NOT build the PR's own image, so a change to
	# src/deploy/entrypoint.sh is not covered until that change is published.
	# Probing a PR-built image is the map's lane 3 and is outside #4335's boundary,
	# which is "wrap the existing probe in a workflow".
	#
	# LANE 3 NOW DOES THAT (#5370). podman-arm64-lane.yml builds the PR's own image
	# and probes it on pull_request, so a change to src/deploy/entrypoint.sh IS
	# covered pre-merge — on arm64, for startup. This lane still probes the
	# published image, so the gap that remains here is narrower than the paragraph
	# above implies: the PR's own EGRESS-GATE behaviour under rootful Podman stays
	# unverified until publication. Closing that means building the image in this
	# lane too; #5370 judged the startup half the one worth paying for first,
	# because that is the half that had already shipped a regression (#5360).
	#
	# No path filters, on purpose. #4339 is what a skipped guard looks like: a
	# workflow scoped so narrowly that it reports green by never running.
	#
	# SAFETY BOUNDARIES, from the map's "Permissions and safety boundaries". This
	# lane runs containers as real root on the runner VM, so they are tighter here
	# than anywhere else:
	#
	# * contents: read and nothing more.
	# * No secrets, and therefore no pull_request_target. Fork PRs are the
	# coverage that matters most and they get no secrets anyway;
	# pull_request_target would run untrusted code with repository credentials
	# as root, which is the one combination the map forbids outright.
	# * No engine socket is mounted anywhere.
	# * Storage isolation is the probe's, and this lane leaves it alone: --store
	# is deliberately NOT passed, so the probe builds its own throwaway
	# graphroot/runroot under /var/tmp and removes it itself. Pointing this job
	# at a shared store — including the runner's own /var/lib/containers — would
	# defeat the isolation #4200 built in. The post-run step below asserts that
	# the host's rootful store really was left untouched.
	#
	# Rootful on an ephemeral hosted runner is acceptable precisely because the VM
	# is single-use. Do not copy this lane onto a persistent self-hosted runner
	# without re-reading that section of the map first.

	on:
	push:
	paths:
	- '.github/workflows/podman-rootful-lane.yml'
	- 'src/deploy/**'
	- 'justfile'
	branches:
	- v4
	- v5
	- v6
	pull_request:
	paths:
	- '.github/workflows/podman-rootful-lane.yml'
	- 'src/deploy/**'
	- 'justfile'
	branches:
	- v4
	- v5
	- v6
	workflow_dispatch:
	inputs:
	somark:
	description: 'Also isolate the SO_MARK exemption from the owner-UID one (needs outbound egress)'
	type: boolean
	default: false

	permissions:
	contents: read

	concurrency:
	group: podman-rootful-lane-${{ github.ref }}
	cancel-in-progress: true

	jobs:
	rootful-probe:
	name: rootful startup, exit-77, and egress gate
	runs-on: ubuntu-latest
	timeout-minutes: 25

	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

	# Assert the environment, do not assume it. Podman is preinstalled on
	# this runner image and sudo is passwordless, so there is no setup step —
	# which is exactly why both have to be checked: a runner-image change is
	# silent, and would otherwise turn a real regression into a test that
	# quietly stopped covering anything.
	#
	# The floor is asserted; the exact version is reported. Pinning the exact
	# version would make every benign runner bump a red build, so a
	# difference from the version #4211 recorded is surfaced as a notice
	# instead — visible, not fatal.
	- name: Assert the rootful Podman stack this lane depends on
	id: stack
	env:
	RECORDED_PODMAN: '5.8.4'
	run: \|
	# shellcheck disable=SC2016 # the printf format strings below carry
	# literal markdown backticks for the step summary, not expansions.
	set -euo pipefail

	if ! command -v podman >/dev/null 2>&1; then
	echo "::error::podman is not installed on this runner; #4211 recorded it preinstalled at /usr/local/bin/podman"
	exit 1
	fi

	if ! command -v jq >/dev/null 2>&1; then
	echo "::error::jq is not installed on this runner; this step reads 'podman info --format json' with it"
	exit 1
	fi

	# -n, never a prompt: without passwordless sudo this lane cannot run
	# rootful at all, and a hang is a worse failure than a red X.
	if ! sudo -n true 2>/dev/null; then
	echo "::error::passwordless sudo is not available; this lane cannot exercise rootful Podman. Record the missing facility in src/docs/podman-ci-runner-map.md."
	exit 1
	fi

	# ONE JSON read, not per-field Go templates (#5678). A Go template
	# hard-fails the whole `podman info` call (exit 125) the moment a
	# field leaves Podman's info schema — a runner-image Podman bump
	# without .Host.RootlessNetworkCmd did exactly that to the rootless
	# and arm64 lanes, on a REPORT-ONLY field. jq with `// "unknown"`
	# degrades per field instead; the load-bearing assertions below
	# still fail loudly on a bad value.
	info="$(sudo -n podman info --format json)"

	version="$(jq -r '.version.Version // "unknown"' <<<"$info")"
	# Boolean, so NOT `// "unknown"`: jq's `//` treats false as falsy
	# and would turn a correct rootless=false — the one value this
	# ROOTFUL lane requires — into "unknown". Type-check instead, so
	# only a genuinely absent field degrades.
	rootless="$(jq -r '.host.security.rootless \| if type == "boolean" then tostring else "unknown" end' <<<"$info")"
	cgroups="$(jq -r '.host.cgroupVersion // "unknown"' <<<"$info")"
	backend="$(jq -r '.host.networkBackend // "unknown"' <<<"$info")"
	runtime="$(jq -r '.host.ociRuntime.name // "unknown"' <<<"$info")"

	# The host store path is handed to the post-run isolation check, so
	# "unknown" is not an acceptable degraded value here: without the
	# real path that check would compare against garbage and prove
	# nothing. Missing means this step fails, loudly.
	hoststore="$(jq -r '.store.graphRoot // empty' <<<"$info")"
	if [ -z "$hoststore" ]; then
	echo "::error::podman info did not report store.graphRoot; the post-run isolation check compares against this path and cannot run without it"
	exit 1
	fi

	printf 'podman %s (%s)\n' "$version" "$(command -v podman)"
	printf 'rootless %s\n' "$rootless"
	printf 'cgroups %s\n' "$cgroups"
	printf 'backend %s\n' "$backend"
	printf 'runtime %s\n' "$runtime"
	printf 'host store %s (the probe must NOT use this)\n' "$hoststore"

	# If the FLAG left the schema, probe the PROPERTY before asserting:
	# `sudo -n podman` running as euid 0 against the default rootful
	# store is rootful by definition. This only rescues the
	# field-missing case — an explicit rootless=true above stays a hard
	# failure, because that is a real mode swap, not schema drift.
	if [ "$rootless" = "unknown" ] && [ "$(sudo -n id -u)" = "0" ]; then
	case "$hoststore" in
	/var/lib/containers*)
	echo "::notice::podman info no longer reports host.security.rootless; derived ROOTFUL from euid 0 and the default rootful store ${hoststore}"
	rootless="false"
	;;
	esac
	fi

	# THE assertion of this lane. Rootless is #4334's lane; if `sudo
	# podman` ever came back rootless — a sudo/XDG_RUNTIME_DIR change is
	# all it would take — every case below would still pass and would
	# mean something else entirely. The probe refuses to run in that
	# state too, but it must not be the only thing checking: a lane that
	# can silently swap root modes is not a baseline.
	if [ "$rootless" != "false" ]; then
	echo "::error::sudo podman reports rootless=${rootless}; this lane must run ROOTFUL (rootless is #4334)"
	exit 1
	fi

	# A floor, not a pin. 4.x is the oldest series with the --cap-add
	# NET_ADMIN behaviour the whole matrix turns on. An unreadable
	# version fails the floor check explicitly rather than as a bash
	# integer-comparison error.
	major="${version%%.*}"
	case "$major" in
	''\|[!0-9])
	echo "::error::could not read a numeric Podman version from 'podman info --format json' (got '${version}'); the 4.x floor cannot be verified"
	exit 1
	;;
	esac
	if [ "$major" -lt 4 ]; then
	echo "::error::podman ${version} is below the 4.x floor this lane needs for --cap-add NET_ADMIN"
	exit 1
	fi

	if [ "$version" != "$RECORDED_PODMAN" ]; then
	echo "::notice::runner Podman is ${version}; src/docs/podman-ci-runner-map.md recorded ${RECORDED_PODMAN}. Re-check the map if this lane starts behaving differently."
	fi

	# Handed to the post-run isolation check so it compares against the
	# store this job actually saw, not a hardcoded path. The image count
	# is the baseline that check compares against: the probe pulls a
	# multi-gigabyte image, so a store it was wrongly pointed at grows by
	# at least one image and stays grown — a far more durable trace than
	# its containers, which are all --rm.
	echo "host_store=${hoststore}" >>"$GITHUB_OUTPUT"
	echo "images_before=$(sudo -n podman images -q \| wc -l)" >>"$GITHUB_OUTPUT"

	{
	printf '### Rootful Podman lane environment\n\n'
	printf '\| \| \|\n\| --- \| --- \|\n'
	printf '\| Podman \| `%s` (map recorded `%s`) \|\n' "$version" "$RECORDED_PODMAN"
	printf '\| Root mode \| rootless=`%s` \|\n' "$rootless"
	printf '\| cgroups \| `%s` \|\n' "$cgroups"
	printf '\| Network backend \| `%s` \|\n' "$backend"
	printf '\| OCI runtime \| `%s` \|\n' "$runtime"
	printf '\| Host rootful store \| `%s` (probe uses a throwaway store) \|\n' "$hoststore"
	printf '\| Runner \| `%s`, image `%s` \|\n' "$RUNNER_OS" "${ImageVersion:-unknown}"
	} >>"$GITHUB_STEP_SUMMARY"

	# The probe's exit status is the result. 0 means every case matched the
	# documented contract; 1 means one did not — a gate that failed to
	# install lands here; 78 means a prerequisite was missing, which is a
	# broken lane rather than a passing one and must be just as loud.
	#
	# There is deliberately no continue-on-error and no `\|\| true`: the whole
	# point of the lane is that a failed gate fails the job.
	#
	# No --store argument. See the storage note in the header.
	- name: Rootful startup, exit-77, and egress-gate matrix (#4200)
	env:
	SOMARK: ${{ inputs.somark }}
	run: \|
	set -uo pipefail

	args=""
	if [ "${SOMARK:-false}" = "true" ]; then
	args="--somark"
	fi

	# The runner invokes this with `bash -e`, so errexit has to come off
	# deliberately: the probe's exit status IS the result, and it has to
	# survive to the diagnosis below instead of aborting the step early.
	set +e
	# shellcheck disable=SC2086 # args is a deliberate single optional flag
	sudo -n bash src/deploy/probe_podman_rootful_netadmin.sh $args
	status=$?
	set -e

	case "$status" in
	0)
	echo "::notice::rootful matrix matched the documented baseline"
	printf '\n✅ The three-case rootful matrix matched the baseline in `src/docs/podman-rootful-egress-baseline.md`.\n' >>"$GITHUB_STEP_SUMMARY"
	;;
	78)
	echo "::error::the probe could not run — a prerequisite is missing (EX_CONFIG). A lane that cannot run is not a lane that passed."
	printf '\n❌ Prerequisite missing (exit 78). See the log for which one.\n' >>"$GITHUB_STEP_SUMMARY"
	;;
	*)
	echo "::error::the rootful matrix no longer matches the documented baseline (exit ${status})"
	printf '\n❌ The rootful matrix did not match the baseline (exit %s).\n' "$status" >>"$GITHUB_STEP_SUMMARY"
	;;
	esac

	exit "$status"

	# The probe's storage isolation is an acceptance criterion, so it is
	# checked rather than trusted. If a future edit ever pointed this lane at
	# a shared store, the probe's containers and the pulled image would land
	# in the host's rootful store instead of the throwaway one — and on a
	# persistent runner that is the collision the map warns about. `always()`
	# because a failed matrix is exactly when leftovers matter.
	- name: Assert the probe left the host's rootful store alone
	if: always()
	env:
	HOST_STORE: ${{ steps.stack.outputs.host_store }}
	IMAGES_BEFORE: ${{ steps.stack.outputs.images_before }}
	run: \|
	# shellcheck disable=SC2016 # {{.Names}} is a Podman Go template
	set -uo pipefail

	# Nothing to compare against if the assertion step never got that
	# far; that failure is already the job's failure.
	if [ -z "${IMAGES_BEFORE:-}" ]; then
	echo "::notice::stack assertion did not complete, so there is no store baseline to compare against"
	exit 0
	fi

	after="$(sudo -n podman images -q \| wc -l)"
	leaked="$(sudo -n podman ps -a --format '{{.Names}}' 2>/dev/null \| grep '^hive-rootful-probe-' \|\| true)"

	printf 'host store %s: %s image(s) before, %s after\n' "$HOST_STORE" "$IMAGES_BEFORE" "$after"

	if [ "$after" -ne "$IMAGES_BEFORE" ] \|\| [ -n "$leaked" ]; then
	echo "::error::the probe wrote into the host's rootful store (${HOST_STORE}); its --root/--runroot isolation was bypassed"
	sudo -n podman images
	[ -n "$leaked" ] && printf 'leftover containers:\n%s\n' "$leaked"
	exit 1
	fi

	echo "the host's rootful store is unchanged: the probe's throwaway store held"

Navigation Menu

Repository navigation

🐛 fix(deploy): reflect intercepted 403/404 from the ingress error backend instead of answering 503 #2964

Workflow file

🐛 fix(deploy): reflect intercepted 403/404 from the ingress error backend instead of answering 503 #2964

Uh oh!

Workflow file for this run