Repository navigation

🐛 fix(crux): /v1/health reports tmux and returns 503 when it is missing #24799

Workflow file for this run

.github/workflows/v2-tests.yml at f727b08

	name: v2 Tests

	on:
	workflow_call:
	inputs:
	integration_candidate:
	description: Run an installed immutable Integration-aware candidate with isolated read-only technical jobs.
	type: boolean
	default: false
	merge_group:
	types: [checks_requested]
	schedule:
	# Hourly, at :30 — deliberately offset from coverage-hourly.yml's :00 run
	# so the two scheduled nets alternate every half hour instead of firing
	# together. This schedule is the post-merge safety net for the default
	# branch (there is no push trigger; pull_request below is the real gate),
	# and it is what keeps create-issue-on-failure armed. It ran every 15
	# minutes until #4650: ~96 runs/day at ~3m15s each — ~5 runner-hours/day —
	# almost all on an unchanged, already-gated SHA, while coverage-hourly
	# re-ran the identical suite with the same floors every hour anyway.
	- cron: '30 * * * *'
	workflow_dispatch:
	# Gate PRs, not just the cron. `go test ./pkg/...` runs nowhere else — v2-ci.yml
	# builds and vets but never runs the unit tests — so before this, a Go
	# regression test could only fail AFTER merge, as a cron-filed ci-failure
	# issue. That is the wrong end of the loop for security regressions: the test
	# proving a fix works has to be able to block the PR that breaks it again.
	#
	# create-issue-on-failure below stays gated on `github.event_name == 'schedule'`,
	# so a red PR fails the check without also filing an issue.
	# Push to the release branches is what produces the per-commit evidence the
	# stable promotion gate requires. Without it v2-tests only ever lands on
	# whatever SHA happens to be HEAD at :30, so no merged commit carries a
	# v2-tests success and the gate can never promote - it held on every
	# candidate, including under an emergency exception, which by design refuses
	# to bypass missing green evidence. The paths filter mirrors pull_request, so
	# this adds a run only where the suite is relevant.
	push:
	branches: [v4, v5, v6]
	paths:
	- 'src/**'
	- 'bin/**'
	- 'config/**'
	- 'Justfile'
	- 'dashboard/openapi.json'
	# release PRs touch only these; the required contexts must report — #10795
	- 'CHANGELOG.md'
	- 'changelog.d/**'
	- '.github/workflows/**'
	- '.github/scripts/**'
	pull_request:
	branches: [v4, v5, v6]
	# Required contexts from this workflow must appear on every PR head. Keep
	# the trigger unfiltered and let required-context-scope below decide whether
	# to run the expensive shards or report success for out-of-scope diffs.


	permissions:
	contents: read

	env:
	# The race-enabled shards require cgo even on lean self-hosted runner images.
	CGO_ENABLED: '1'
	# Self-hosted runners sit inside OpenShift; keep these hermetic unit tests from auto-discovering that cluster.
	KUBERNETES_SERVICE_HOST: ''
	KUBERNETES_SERVICE_PORT: ''
	# A slow cache-segment download must never cost more than a recompile.
	# Some cluster nodes crawl when pulling from the Actions cache backend —
	# observed 16-minute "Restore warm Go build cache" steps against a ~2-minute
	# cold compile. Abort a segment after 1 minute and let the job build instead.
	ACTIONS_CACHE_SEGMENT_DOWNLOAD_TIMEOUT_MINS: '1'
	# Build-cache isolation for the self-hosted pool (see
	# .github/scripts/go-cache-guard.sh). `job` gives each job its own GOCACHE
	# under RUNNER_TEMP instead of the cephfs volume every runner pod shares;
	# GOMODCACHE stays shared. Set the repo variable HIVE_GO_BUILD_CACHE=shared
	# to go back to the shared build cache without a code change.
	GO_CACHE_GUARD_BUILD_CACHE: ${{ vars.HIVE_GO_BUILD_CACHE \|\| 'job' }}


	# Cancel superseded PR runs only (#4623). A new push to a PR makes the
	# in-flight run's verdict irrelevant, so cancelling it is pure runner-minute
	# savings. Scheduled and workflow_dispatch runs must NEVER be cancelable:
	# the scheduled run feeds create-issue-on-failure (the merge-race/flake
	# auto-filer) — cancelling one blinds it for that window. Keying the non-PR
	# group on run_id gives every such run its own group, so they can neither
	# cancel each other nor queue behind one another.
	concurrency:
	group: ${{ inputs.integration_candidate && 'integration-tests' \|\| github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.number \|\| github.run_id }}
	cancel-in-progress: ${{ !inputs.integration_candidate && github.event_name == 'pull_request' }}

	# ── Sharding layout (#4118) ──────────────────────────────────────────────────
	# The suite is split so PR wall-clock ≈ the slowest single piece, not the sum.
	# Measured on CI (v4, ubuntu-latest, -short -race): pkg/hub ≈ 117s and
	# crux/agent ≈ 429s of TEST time each — every other package is ≤ 51s and they
	# all run concurrently inside one `go test` invocation. Round-robin over the
	# 840 sorted agent test names measures a worst slice of ~127s at N=5. So:
	#
	# test (hub i/H) ./pkg/hub H jobs, each running a
	# deterministic 1/H slice of the
	# package's test FUNCTIONS via -run
	# (one ~2,700-function package at
	# ~125s serial: it was the PR gate's
	# critical path until sliced the
	# same way as agent) — starts
	# immediately, self-partitions
	# test (agent i/N) ./crux/agent N jobs, each running a
	# deterministic 1/N slice of the
	# package's test FUNCTIONS via -run
	# (the package is a serial 429s
	# monolith; package-level sharding
	# cannot help it) — starts
	# immediately, self-partitions
	# test (dashboard i/D) ./pkg/dashboard D jobs; same function slicing
	# test (scheduler i/S) ./pkg/scheduler S jobs; same function slicing
	# (one test-sliced job with an
	# include matrix; both packages
	# had grown into unsplittable
	# ~200s+ rest-bucket floors —
	# see the job comment)
	# list-packages go list minus hub/agent/dashboard/scheduler, greedily
	# bucketed by measured weight into R balanced groups
	# (job outputs → matrix)
	# test (rest i/R) one bucket each
	#
	# Shard counts (H=4, N=5, R=5) were re-tuned on 2026-09-25 once the self-hosted
	# runner fleet dropped per-shard setup overhead to ~25s: at H=3/R=4 the gate
	# was bounded by hub 2/3 (185s) and rest 2/4 (180s) against a 4.0-minute run,
	# so one more slice of each shaves ~1 minute off the critical path at the cost
	# of two runner pods. Re-tuned again on 2026-10-26 (D=4, S=3, R=4): the gate
	# had drifted to `test (rest 1/5)` at 325s because pkg/dashboard alone ran
	# 224s inside it (the weight table said 123s) and pkg/scheduler 194s (table:
	# 74s). Slicing both out of the rest partition caps every rest bucket near its
	# next-heaviest package (pkg/hub/spoke, ~85s) and makes the dashboard shuffle
	# lane (~190s) the expected critical path. Re-tune when the slowest shard
	# drifts past ~150s; the per-job step summary ranks each shard's slowest tests.
	#
	# Every shard uses the exact same flags: -short -race -count=1 (untouched),
	# plus -coverprofile for the parse-only coverage gate below, plus:
	# - -v, with the `=== RUN/PAUSE/CONT` lines stripped before tee, so the log
	# carries a per-test-function duration and .github/scripts/slowest-tests.sh
	# can rank the slice's slowest tests into the step summary. That table is
	# how a slow shard gets diagnosed without a local reproduction.
	# - -shuffle=on on every NON-pull_request run (the hourly schedule and manual
	# dispatches). Order-dependent tests (#5102's golden splash race, the
	# package-var seams in #4795) pass in a fixed order and only fail when the
	# order moves; shuffling the post-merge net finds them and files an issue
	# with the seed in the log, while the PR gate stays deterministic so a
	# contributor is never failed by an ordering their change did not cause.
	# - pkg/dashboard additionally gets a dedicated shuffled lane on EVERY event,
	# PRs included (#5570): it is the package with a record of package-level
	# cache leaks (#5553's cachedHealth), and a leak merged today otherwise
	# fails weeks later on whichever scheduled run draws a hostile order. The
	# lane bounds parallelism and surfaces the failing seed — see
	# test-dashboard-shuffle below for why both bounds are load-bearing.
	#
	# Completeness invariants (nothing can be silently untested):
	# - rest buckets are computed by EXCLUDING exactly pkg/hub, crux/agent,
	# pkg/dashboard and pkg/scheduler from the full `go list` and PARTITIONING
	# the remainder — a new package always lands in a bucket (a future
	# subpackage of any sliced package included, since the slices run the bare
	# package path), a removed one just disappears from its bucket.
	# - hub, dashboard, scheduler and agent slices partition the sorted
	# `go test -list` output round-robin — a new test function always lands in
	# a slice. Anchored full-name regexes (^(A\|B\|...)$) make slices pairwise
	# disjoint and jointly exhaustive.
	# - the aggregate `test` job (the required-check name) fails closed: it
	# requires list-packages AND every shard to succeed.
	#
	# Affected-package selection (#12454), pull requests only: list-packages diffs
	# the PR merge commit against its base parent and runs
	# .github/scripts/affected-go-packages.sh, which selects the changed packages
	# plus their reverse dependencies and falls back to the full suite for
	# anything it cannot reason about (workflow/script/harness/module/asset
	# changes, analysis failures). Every shard job still runs and reports; a lane
	# with nothing selected succeeds without testing, so `test` and `coverage`
	# never wait on a missing context. Push and scheduled runs are always full.
	#
	# Crux boundary filter (#13536), pull requests only, layered on the selection
	# above and consulted ONLY when it is `partial` (an `all` verdict from the
	# dependency analysis is never narrowed). .github/scripts/crux-scope.sh
	# classifies the diff against the Crux package set — the nested Go module at
	# src/crux/ (module github.com/hivecommons/hive/crux: crux/agent/... and its
	# ring, addressed from src/ as ./crux/... through src/go.work) — as:
	# crux-only every change is inside the set: the agent slices run, the rest
	# buckets keep only the selected Crux packages plus the
	# always-on tree-scanning packages (internal/..., pkg/config,
	# pkg/rotation), and the hub, dashboard and scheduler slices and
	# the dashboard shuffle lane report success without testing.
	# The inbound consumers of crux/agent (dashboard, hub,
	# cmd/hive, ...) are deliberately NOT
	# run on such a PR; v2-ci.yml still builds and vets the whole
	# tree, and the push/scheduled full runs are the behavioural net.
	# none no change touches the set: the agent slices report success
	# without testing and crux/agent is dropped from the selection
	# EVEN IF the dependency analysis selected it (a PR that only
	# edits a root-module package, e.g. pkg/github, does not run
	# crux/agent tests on the PR). This is the mirror of
	# the crux-only trade: the boundary is tested from inside the
	# set, and the push/scheduled full runs are the net. Everything
	# else follows the selection.
	# mixed both sides changed: the selection is used unchanged.
	# full a fail-safe path changed — src/go.mod, src/go.sum,
	# src/go.work, src/crux/go.mod, src/crux/go.sum,
	# src/internal/, src/pkg/config/, .github/workflows/**,
	# .github/scripts/, src/scripts/ — or the diff could not be
	# read: everything runs.
	# The verdict and the resulting per-lane run flags are computed once, in
	# list-packages, and every lane and the aggregate `test` job read the same
	# outputs: a lane may report success without testing ONLY when its run flag
	# is false, the aggregate prints that allowlist, and it still fails closed on
	# any shard result other than success (a GitHub-level `skipped` is never
	# accepted). HIVE_AFFECTED_TEST_SELECTION=off disables this filter together
	# with the selection.
	jobs:
	required-context-scope:
	name: required context scope
	runs-on: ${{ !inputs.integration_candidate && (github.event_name != 'pull_request' \|\| github.event.pull_request.head.repo.full_name == github.repository) && fromJSON(vars.HIVE_RUNNER_LABELS \|\| '["ubuntu-latest"]') \|\| 'ubuntu-latest' }}
	timeout-minutes: 10
	outputs:
	run-tests: ${{ steps.scope.outputs.run-tests }}
	steps:
	- name: Decide whether the test and coverage gates need the real suite
	id: scope
	env:
	GITHUB_TOKEN: ${{ github.token }}
	PR_NUMBER: ${{ github.event.pull_request.number }}
	run: \|
	set -euo pipefail
	if [ "${{ inputs.integration_candidate }}" = true ] &&
	{ [ "$GITHUB_EVENT_NAME" != workflow_dispatch ] \|\| [[ "$GITHUB_REF" != refs/tags/hive-integration/* ]]; }; then
	echo "::error::Integration candidates require an explicitly dispatched immutable candidate tag"
	exit 1
	fi
	if [ "${{ github.event_name }}" != "pull_request" ]; then
	echo "run-tests=true" >> "$GITHUB_OUTPUT"
	exit 0
	fi
	: > changed-files.txt
	page=1
	api_root="${GITHUB_API_URL:-https://api.github.com}"
	while :; do
	curl -fsSL -H "Authorization: Bearer ${GITHUB_TOKEN}" -H "Accept: application/vnd.github+json" "${api_root}/repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}/files?per_page=100&page=${page}" -o changed-files-page.json
	python3 - <<'PYFILES' >> changed-files.txt
	import json
	with open('changed-files-page.json', encoding='utf-8') as f:
	for item in json.load(f):
	print(item['filename'])
	PYFILES
	count=$(python3 - <<'PYCOUNT'
	import json
	with open('changed-files-page.json', encoding='utf-8') as f:
	print(len(json.load(f)))
	PYCOUNT
	)
	[ "$count" -lt 100 ] && break
	page=$((page + 1))
	done
	rm -f changed-files-page.json
	run=false
	while IFS= read -r path; do
	case "$path" in
	src/\|bin/\|config/\|Justfile\|dashboard/openapi.json\|CHANGELOG.md\|changelog.d/\|.github/workflows/\|.github/scripts/)
	run=true
	;;
	esac
	done < changed-files.txt
	echo "run-tests=${run}" >> "$GITHUB_OUTPUT"
	if [ "$run" = true ]; then
	echo "v2 test/coverage gates will run real shards."
	else
	echo "No v2 test paths changed; required test/coverage contexts will report success without shards."
	fi

	# pkg/hub is one package of ~2,700 test functions that takes ~125s serially
	# under -race — after #4118 sliced crux/agent, this single job was the PR
	# gate's critical path (~3.3 min wall clock against ~2.4 min for the widest
	# rest bucket). It is sliced exactly like test-agent below: each job derives
	# the same sorted `go test -list` output and runs rows where NR mod H ==
	# idx-1. The coverage job merges the H partial profiles before scoring the
	# hub floor. Adding a hub test no longer moves the gate's wall clock by its
	# full duration — only by 1/H of it.
	test-hub:
	needs: [required-context-scope, list-packages]
	if: ${{ (needs.required-context-scope.result != 'success' \|\| needs.required-context-scope.outputs.run-tests == 'true') && !(github.event_name == 'pull_request' && startsWith(github.head_ref, 'release-gate/')) }}
	name: test (hub ${{ matrix.idx }}/4)
	strategy:
	fail-fast: false
	matrix:
	idx: [1, 2, 3, 4]
	runs-on: ${{ !inputs.integration_candidate && (github.event_name != 'pull_request' \|\| github.event.pull_request.head.repo.full_name == github.repository) && fromJSON(vars.HIVE_RUNNER_LABELS \|\| '["ubuntu-latest"]') \|\| 'ubuntu-latest' }}
	timeout-minutes: 20
	defaults:
	run:
	working-directory: src
	steps:
	# Affected-package selection (#12454): when this lane has nothing to
	# test on a PR it still runs and reports success, so no context the
	# aggregates or branch protection wait on can go missing. Anything but
	# an explicit "nothing selected" runs the lane (fail toward testing).
	- name: Affected-package selection
	id: select
	working-directory: .
	env:
	SELECTION: ${{ needs.list-packages.outputs.selection }}
	RUN_LANE: ${{ needs.list-packages.outputs.run-hub }}
	run: \|
	if [ "$SELECTION" = partial ] && [ "$RUN_LANE" = false ]; then
	echo "Affected-package selection: pkg/hub is not affected by this diff; reporting success without running tests. Push and scheduled runs test it in full."
	echo "run=false" >> "$GITHUB_OUTPUT"
	else
	echo "run=true" >> "$GITHUB_OUTPUT"
	fi

	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	if: steps.select.outputs.run == 'true'
	with:
	persist-credentials: ${{ !inputs.integration_candidate }}

	# Populate the shared node-local Go tool cache under the runner pods'
	# node-wide lock before setup-go, so parallel jobs on a cold cache (a
	# src/go.mod bump) stop clobbering each other's rm -rf + copy (#11175,
	# #10234). Never fails the job.
	- name: Warm shared Go tool cache (serialize the cold-cache population)
	if: steps.select.outputs.run == 'true'
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-toolcache-guard.sh"
	- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
	if: steps.select.outputs.run == 'true'
	with:
	cache: false
	go-version-file: src/go.mod
	- name: Guard shared Go cache (fall back when the runner volume is full)
	if: steps.select.outputs.run == 'true'
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-cache-guard.sh"
	# Fill GOMODCACHE up front with bounded retries so a transient
	# proxy.golang.org stream reset (INTERNAL_ERROR, #14976) cannot fail the
	# job's first module-fetching step; see .github/actions/go-mod-download-retry.
	- name: Download Go modules (retry transient proxy resets)
	if: steps.select.outputs.run == 'true'
	uses: ./.github/actions/go-mod-download-retry

	- name: Install gotestsum
	if: steps.select.outputs.run == 'true'
	run: \|
	echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
	go install gotest.tools/gotestsum@v1.13.0
	"$(go env GOPATH)/bin/gotestsum" --version

	# Warm compile cache (#7480 follow-up): the warm-build-cache job on every
	# v4 push compiles the whole tree (race + norace) and saves GOCACHE keyed
	# by SHA; PR jobs restore the nearest ancestor via the prefix restore-key,
	# turning the ~2-minute per-job -race recompile into an incremental one.
	# Restore-only in PR jobs: balance/miss costs seconds, never correctness.
	- name: Restore warm Go build cache
	if: ${{ steps.select.outputs.run == 'true' && (!contains(vars.HIVE_RUNNER_LABELS, 'hive-runners')) }}
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: ~/.cache/go-build
	key: hive-go-build-${{ runner.os }}-${{ runner.arch }}-${{ github.sha }}
	restore-keys: \|
	hive-go-build-${{ runner.os }}-${{ runner.arch }}-

	- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
	if: steps.select.outputs.run == 'true'
	with:
	node-version: '22'
	- name: Guard shared node tool cache (repair a half-written entry)
	# setup-node reports "Found in cache" for a node/<ver>/x64 entry whose
	# .complete marker exists but bin/npm does not (#10505, #10518). The
	# canary repairs this too, but its cron is starved to a ~5h cadence, so
	# heal the node from the hot path: drop the marker (never the directory,
	# a sibling job may be mid-run) so the next job re-extracts.
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: \|
	set -u
	command -v npm >/dev/null 2>&1 && exit 0
	for marker in "${RUNNER_TOOL_CACHE:-/nonexistent}"/node/*/x64.complete; do
	[ -e "$marker" ] \|\| continue
	dir="${marker%.complete}"
	[ -x "$dir/bin/npm" ] && continue
	echo "::warning::node tool cache $dir has a .complete marker but no executable bin/npm; removing the marker so the next job on $RUNNER_NAME rebuilds it"
	rm -f "$marker"
	done
	echo "::error::npm is not on PATH after actions/setup-node on $RUNNER_NAME (half-written shared tool cache, see #10518); marker removed, re-run this job"
	exit 1

	- name: Restore cgo toolchain package cache
	if: steps.select.outputs.run == 'true'
	id: restore-cgo-apt-cache
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2
	restore-keys: \|
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-
	hive-cgo-race-apt-

	# Bounded + annotated by .github/scripts/ci-install-tool.sh (#6648/#6870/#6935):
	# skip apt when the tool already exists, install restored .debs offline
	# when possible, retry short egress flakes with backoff, and fail loudly
	# with a ::error:: if neither cache nor mirrors can provide the tool.
	- name: Prepare cgo toolchain for race tests
	if: steps.select.outputs.run == 'true'
	id: prepare-cgo-toolchain
	env:
	HIVE_CI_APT_CACHE_DIR: ${{ github.workspace }}/.ci/apt-cache/cgo-race
	HIVE_CI_MISSING_TOOLCHAIN_SOFT_FAIL: ${{ !inputs.integration_candidate && (github.event_name == 'schedule' \|\| github.event_name == 'workflow_dispatch') && '1' \|\| '0' }}
	run: \|
	bash "$GITHUB_WORKSPACE/.github/scripts/ci-install-tool.sh" \
	--require gcc --for '-race tests' \
	--apt 'gcc libc6-dev' --apk 'gcc musl-dev' --verify 'gcc --version'

	- name: Save cgo toolchain package cache
	if: ${{ steps.select.outputs.run == 'true' && (always() && steps.restore-cgo-apt-cache.outputs.cache-hit != 'true' && steps.prepare-cgo-toolchain.outputs.apt-cache-populated == 'true') }}
	continue-on-error: true
	uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2

	- name: Test slice of pkg/hub
	if: ${{ steps.select.outputs.run == 'true' && (steps.prepare-cgo-toolchain.outputs.toolchain-ready == 'true') }}
	env:
	IDX: ${{ matrix.idx }}
	TOTAL: ${{ strategy.job-total }}
	# The converted skips in pkg/hub gate on repo files being present in
	# the checkout (deploy/k8s/error-pages.yaml) — a precondition
	# actions/checkout above GUARANTEES here, so a skip at a converted
	# site means the test broke, not that the environment is unsuitable
	# (#5388). Only testutil.SkipUnlessRequired sites become fatal;
	# plain t.Skip capability gates are unaffected.
	HIVE_TEST_REQUIRE_BEHAVIOURAL: '1'
	run: \|
	set -o pipefail
	ROOT="$PWD"
	export LC_ALL=C # Regex length guards below must count bytes.
	export TEST_PACKAGE=github.com/hivecommons/hive/pkg/hub
	export TEST_DIR="$ROOT/pkg/hub"
	export TEST_BINARY="$ROOT/hub.test"
	go test -c -race -cover -o "$TEST_BINARY" ./pkg/hub
	(cd "$TEST_DIR" && "$TEST_BINARY" -test.short -test.list '.*') \| grep -E '^(Test\|Example\|Fuzz)' \| LC_ALL=C sort > all-tests.txt
	if [ ! -s all-tests.txt ]; then
	echo "::error::go test -list produced no test functions for pkg/hub — refusing to run an empty slice (fail closed)"
	exit 1
	fi
	awk -v n="$TOTAL" -v i="$IDX" 'NR % n == i - 1' all-tests.txt > slice.txt
	echo "slice $IDX/$TOTAL: $(wc -l < slice.txt) of $(wc -l < all-tests.txt) test functions"
	if [ ! -s slice.txt ]; then
	echo "::error::empty hub slice — failing closed"
	exit 1
	fi
	export CHUNK_DIR TEST_SHUFFLE=off
	CHUNK_DIR=$(mktemp -d "$ROOT/test-chunks.XXXXXX")
	split -l 1000 slice.txt "$CHUNK_DIR/chunk-"
	if [ "$GITHUB_EVENT_NAME" != "pull_request" ]; then TEST_SHUFFLE=on; fi
	# raw-command preserves one JUnit/flake report across all chunks.
	# gotestsum appends -test.run and package arguments on reruns.
	cat > "$CHUNK_DIR/run.sh" <<'RUNNER'
	#!/usr/bin/env bash
	set -uo pipefail
	cd "$TEST_DIR" \|\| exit 1
	run_chunk() {
	local RUN="$1" PROFILE="$2"
	# MAX_ARG_STRLEN is 131072 bytes; never approach it, even if a
	# future test has an exceptionally long name. Guard before exec.
	if [ "${#RUN}" -gt 120000 ]; then
	echo "::error::chunk regex exceeds 120000 bytes; reduce chunk size" >&2
	return 1
	fi
	go tool test2json -t -p "$TEST_PACKAGE" "$TEST_BINARY" \
	-test.short -test.count=1 -test.v=test2json -test.timeout=10m \
	-test.shuffle="$TEST_SHUFFLE" "$RUN" -test.coverprofile="$PROFILE"
	}
	if [ "$#" -gt 0 ]; then
	# Reruns must not overwrite the first-pass coverage.
	run_chunk "$1" "$CHUNK_DIR/rerun.out"
	exit $?
	fi
	status=0
	for chunk in "$CHUNK_DIR"/chunk-*; do
	RUN="-test.run=^($(paste -sd'\|' "$chunk"))$"
	run_chunk "$RUN" "$chunk.out" \|\| status=1
	done
	exit "$status"
	RUNNER
	status=0
	if gotestsum --format=standard-verbose --rerun-fails=2 --rerun-fails-max-failures=5 \
	--rerun-fails-report=flaky-tests-hub-"$IDX".txt \
	--junitfile junit-hub-"$IDX".xml --raw-command -- bash "$CHUNK_DIR/run.sh" \
	2>&1 \| grep --line-buffered -vE '^=== (RUN\|PAUSE\|CONT) ' \| tee test-hub-"$IDX".log; then
	:
	else
	status=1
	fi
	# The test binary runs under test2json, not the `go test` driver, so
	# nothing prints the `ok <pkg>` / `FAIL <pkg>` summary line the
	# coverage scorer classifies packages by (#14135). Append it.
	if [ "$status" -eq 0 ]; then
	printf 'ok \t%s\t(slice %s/%s)\n' "$TEST_PACKAGE" "$IDX" "$TOTAL" >> test-hub-"$IDX".log
	else
	printf 'FAIL\t%s\t(slice %s/%s)\n' "$TEST_PACKAGE" "$IDX" "$TOTAL" >> test-hub-"$IDX".log
	fi
	# Sum first-pass counters for identical blocks; never count reruns.
	# Do this even after test failure so the coverage gate has evidence.
	awk 'FNR == 1 { next } { key=$1 " " $2; hits[key]+=$3 }
	END { print "mode: atomic"; for (key in hits) print key, hits[key] }' \
	"$CHUNK_DIR"/chunk-*.out > coverage-hub-"$IDX".out
	exit "$status"

	- name: Surface flaky reruns
	id: flaky-reruns
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: bash "$GITHUB_WORKSPACE/.github/scripts/ci-gotestsum-flake-summary.sh" "flaky-tests-hub-${{ matrix.idx }}.txt" "hub ${{ matrix.idx }}/4"

	- name: Upload flaky rerun report
	if: ${{ steps.select.outputs.run == 'true' && (always() && steps.flaky-reruns.outputs.has-flakes == 'true') }}
	uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	with:
	name: flaky-tests-hub-${{ matrix.idx }}
	path: src/flaky-tests-hub-${{ matrix.idx }}.txt
	if-no-files-found: error
	retention-days: 14

	- name: Slowest tests in this slice
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: bash "$GITHUB_WORKSPACE/.github/scripts/slowest-tests.sh" "hub ${{ matrix.idx }}/4" test-hub-${{ matrix.idx }}.log

	# Uploaded even on failure so the coverage job (and humans) can inspect
	# the raw `go test` output. `warn` not `error`: on a compile failure the
	# coverprofile is never written, and this upload must not mask the real
	# test failure with an artifact error.
	- name: Upload shard coverage artifacts
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	with:
	name: test-shard-hub-${{ matrix.idx }}
	path: \|
	src/coverage-hub-${{ matrix.idx }}.out
	src/test-hub-${{ matrix.idx }}.log
	if-no-files-found: warn
	retention-days: 1

	# Function-sliced lanes for the two root-module packages whose serial
	# -short -race wall clock is the rest lane's floor: pkg/dashboard (~220s,
	# ~7,300 test functions) and pkg/scheduler (~200s, mostly serial; one
	# 75s test). Measured 2026-10-26 (v5 push runs 40383615331, 40382244011):
	# `test (rest 1/5)` took 325s with pkg/dashboard alone running 224s of it,
	# `test (rest 3/5)` 253s with pkg/scheduler at 194s, every other rest
	# bucket ≤ 154s. A rest bucket's wall clock is its heaviest package (the
	# packages run concurrently under one go test), so package-level
	# rebalancing cannot move either floor. Both are sliced exactly like
	# pkg/hub above and EXCLUDED from the rest partition (their subpackages —
	# pkg/dashboard/webstatic, pkg/dashboard/collect, pkg/dashboard/theme —
	# stay in the rest buckets). The context name is `test (<lane> i/N)`; the
	# coverage job merges the N partial profiles exactly as for hub and
	# agent. Adding a lane = one include row per slice below plus a
	# run-<lane> flag in list-packages and the aggregate `test` allowlist.
	test-sliced:
	needs: [required-context-scope, list-packages]
	if: ${{ (needs.required-context-scope.result != 'success' \|\| needs.required-context-scope.outputs.run-tests == 'true') && !(github.event_name == 'pull_request' && startsWith(github.head_ref, 'release-gate/')) }}
	name: test (${{ matrix.lane }} ${{ matrix.idx }}/${{ matrix.total }})
	strategy:
	fail-fast: false
	matrix:
	include:
	- { lane: dashboard, pkg: pkg/dashboard, idx: 1, total: 4 }
	- { lane: dashboard, pkg: pkg/dashboard, idx: 2, total: 4 }
	- { lane: dashboard, pkg: pkg/dashboard, idx: 3, total: 4 }
	- { lane: dashboard, pkg: pkg/dashboard, idx: 4, total: 4 }
	- { lane: scheduler, pkg: pkg/scheduler, idx: 1, total: 3 }
	- { lane: scheduler, pkg: pkg/scheduler, idx: 2, total: 3 }
	- { lane: scheduler, pkg: pkg/scheduler, idx: 3, total: 3 }
	runs-on: ${{ !inputs.integration_candidate && (github.event_name != 'pull_request' \|\| github.event.pull_request.head.repo.full_name == github.repository) && fromJSON(vars.HIVE_RUNNER_LABELS \|\| '["ubuntu-latest"]') \|\| 'ubuntu-latest' }}
	timeout-minutes: 20
	defaults:
	run:
	working-directory: src
	steps:
	# Affected-package selection (#12454): when this lane has nothing to
	# test on a PR it still runs and reports success, so no context the
	# aggregates or branch protection wait on can go missing. Anything but
	# an explicit "nothing selected" runs the lane (fail toward testing).
	- name: Affected-package selection
	id: select
	working-directory: .
	env:
	SELECTION: ${{ needs.list-packages.outputs.selection }}
	LANE: ${{ matrix.lane }}
	PKG: ${{ matrix.pkg }}
	RUN_DASHBOARD: ${{ needs.list-packages.outputs.run-dashboard }}
	RUN_SCHEDULER: ${{ needs.list-packages.outputs.run-scheduler }}
	run: \|
	case "$LANE" in
	dashboard) RUN_LANE=$RUN_DASHBOARD ;;
	scheduler) RUN_LANE=$RUN_SCHEDULER ;;
	*) RUN_LANE=true ;;
	esac
	if [ "$SELECTION" = partial ] && [ "$RUN_LANE" = false ]; then
	echo "Affected-package selection: $PKG is not affected by this diff; reporting success without running tests. Push and scheduled runs test it in full."
	echo "run=false" >> "$GITHUB_OUTPUT"
	else
	echo "run=true" >> "$GITHUB_OUTPUT"
	fi

	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	if: steps.select.outputs.run == 'true'
	with:
	persist-credentials: ${{ !inputs.integration_candidate }}

	# Populate the shared node-local Go tool cache under the runner pods'
	# node-wide lock before setup-go, so parallel jobs on a cold cache (a
	# src/go.mod bump) stop clobbering each other's rm -rf + copy (#11175,
	# #10234). Never fails the job.
	- name: Warm shared Go tool cache (serialize the cold-cache population)
	if: steps.select.outputs.run == 'true'
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-toolcache-guard.sh"
	- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
	if: steps.select.outputs.run == 'true'
	with:
	cache: false
	go-version-file: src/go.mod
	- name: Guard shared Go cache (fall back when the runner volume is full)
	if: steps.select.outputs.run == 'true'
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-cache-guard.sh"
	# Fill GOMODCACHE up front with bounded retries so a transient
	# proxy.golang.org stream reset (INTERNAL_ERROR, #14976) cannot fail the
	# job's first module-fetching step; see .github/actions/go-mod-download-retry.
	- name: Download Go modules (retry transient proxy resets)
	if: steps.select.outputs.run == 'true'
	uses: ./.github/actions/go-mod-download-retry

	- name: Install gotestsum
	if: steps.select.outputs.run == 'true'
	run: \|
	echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
	go install gotest.tools/gotestsum@v1.13.0
	"$(go env GOPATH)/bin/gotestsum" --version

	# Warm compile cache (#7480 follow-up): the warm-build-cache job on every
	# v4 push compiles the whole tree (race + norace) and saves GOCACHE keyed
	# by SHA; PR jobs restore the nearest ancestor via the prefix restore-key,
	# turning the ~2-minute per-job -race recompile into an incremental one.
	# Restore-only in PR jobs: balance/miss costs seconds, never correctness.
	- name: Restore warm Go build cache
	if: ${{ steps.select.outputs.run == 'true' && (!contains(vars.HIVE_RUNNER_LABELS, 'hive-runners')) }}
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: ~/.cache/go-build
	key: hive-go-build-${{ runner.os }}-${{ runner.arch }}-${{ github.sha }}
	restore-keys: \|
	hive-go-build-${{ runner.os }}-${{ runner.arch }}-

	- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
	if: steps.select.outputs.run == 'true'
	with:
	node-version: '22'
	- name: Guard shared node tool cache (repair a half-written entry)
	# setup-node reports "Found in cache" for a node/<ver>/x64 entry whose
	# .complete marker exists but bin/npm does not (#10505, #10518). The
	# canary repairs this too, but its cron is starved to a ~5h cadence, so
	# heal the node from the hot path: drop the marker (never the directory,
	# a sibling job may be mid-run) so the next job re-extracts.
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: \|
	set -u
	command -v npm >/dev/null 2>&1 && exit 0
	for marker in "${RUNNER_TOOL_CACHE:-/nonexistent}"/node/*/x64.complete; do
	[ -e "$marker" ] \|\| continue
	dir="${marker%.complete}"
	[ -x "$dir/bin/npm" ] && continue
	echo "::warning::node tool cache $dir has a .complete marker but no executable bin/npm; removing the marker so the next job on $RUNNER_NAME rebuilds it"
	rm -f "$marker"
	done
	echo "::error::npm is not on PATH after actions/setup-node on $RUNNER_NAME (half-written shared tool cache, see #10518); marker removed, re-run this job"
	exit 1

	- name: Restore cgo toolchain package cache
	if: steps.select.outputs.run == 'true'
	id: restore-cgo-apt-cache
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2
	restore-keys: \|
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-
	hive-cgo-race-apt-

	# Bounded + annotated by .github/scripts/ci-install-tool.sh (#6648/#6870/#6935):
	# skip apt when the tool already exists, install restored .debs offline
	# when possible, retry short egress flakes with backoff, and fail loudly
	# with a ::error:: if neither cache nor mirrors can provide the tool.
	- name: Prepare cgo toolchain for race tests
	if: steps.select.outputs.run == 'true'
	id: prepare-cgo-toolchain
	env:
	HIVE_CI_APT_CACHE_DIR: ${{ github.workspace }}/.ci/apt-cache/cgo-race
	HIVE_CI_MISSING_TOOLCHAIN_SOFT_FAIL: ${{ !inputs.integration_candidate && (github.event_name == 'schedule' \|\| github.event_name == 'workflow_dispatch') && '1' \|\| '0' }}
	run: \|
	bash "$GITHUB_WORKSPACE/.github/scripts/ci-install-tool.sh" \
	--require gcc --for '-race tests' \
	--apt 'gcc libc6-dev' --apk 'gcc musl-dev' --verify 'gcc --version'

	- name: Save cgo toolchain package cache
	if: ${{ steps.select.outputs.run == 'true' && (always() && steps.restore-cgo-apt-cache.outputs.cache-hit != 'true' && steps.prepare-cgo-toolchain.outputs.apt-cache-populated == 'true') }}
	continue-on-error: true
	uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2

	- name: Test slice of ${{ matrix.pkg }}
	if: ${{ steps.select.outputs.run == 'true' && (steps.prepare-cgo-toolchain.outputs.toolchain-ready == 'true') }}
	env:
	IDX: ${{ matrix.idx }}
	# Per-lane slice count: strategy.job-total would be the size of the
	# whole include matrix, not of this package's slices.
	TOTAL: ${{ matrix.total }}
	LANE: ${{ matrix.lane }}
	PKG: ${{ matrix.pkg }}
	# The converted skips in these packages gate on repo files being
	# present in the checkout — a precondition
	# actions/checkout above GUARANTEES here, so a skip at a converted
	# site means the test broke, not that the environment is unsuitable
	# (#5388). Only testutil.SkipUnlessRequired sites become fatal;
	# plain t.Skip capability gates are unaffected.
	HIVE_TEST_REQUIRE_BEHAVIOURAL: '1'
	run: \|
	set -o pipefail
	ROOT="$PWD"
	export LC_ALL=C # Regex length guards below must count bytes.
	export TEST_PACKAGE="github.com/hivecommons/hive/$PKG"
	export TEST_DIR="$ROOT/$PKG"
	export TEST_BINARY="$ROOT/$LANE.test"
	go test -c -race -cover -o "$TEST_BINARY" "./$PKG"
	(cd "$TEST_DIR" && "$TEST_BINARY" -test.short -test.list '.*') \| grep -E '^(Test\|Example\|Fuzz)' \| LC_ALL=C sort > all-tests.txt
	if [ ! -s all-tests.txt ]; then
	echo "::error::go test -list produced no test functions for $PKG — refusing to run an empty slice (fail closed)"
	exit 1
	fi
	awk -v n="$TOTAL" -v i="$IDX" 'NR % n == i - 1' all-tests.txt > slice.txt
	echo "slice $IDX/$TOTAL: $(wc -l < slice.txt) of $(wc -l < all-tests.txt) test functions"
	if [ ! -s slice.txt ]; then
	echo "::error::empty $LANE slice — failing closed"
	exit 1
	fi
	export CHUNK_DIR TEST_SHUFFLE=off
	CHUNK_DIR=$(mktemp -d "$ROOT/test-chunks.XXXXXX")
	split -l 1000 slice.txt "$CHUNK_DIR/chunk-"
	if [ "$GITHUB_EVENT_NAME" != "pull_request" ]; then TEST_SHUFFLE=on; fi
	# raw-command preserves one JUnit/flake report across all chunks.
	# gotestsum appends -test.run and package arguments on reruns.
	cat > "$CHUNK_DIR/run.sh" <<'RUNNER'
	#!/usr/bin/env bash
	set -uo pipefail
	cd "$TEST_DIR" \|\| exit 1
	run_chunk() {
	local RUN="$1" PROFILE="$2"
	# MAX_ARG_STRLEN is 131072 bytes; never approach it, even if a
	# future test has an exceptionally long name. Guard before exec.
	if [ "${#RUN}" -gt 120000 ]; then
	echo "::error::chunk regex exceeds 120000 bytes; reduce chunk size" >&2
	return 1
	fi
	go tool test2json -t -p "$TEST_PACKAGE" "$TEST_BINARY" \
	-test.short -test.count=1 -test.v=test2json -test.timeout=10m \
	-test.shuffle="$TEST_SHUFFLE" "$RUN" -test.coverprofile="$PROFILE"
	}
	if [ "$#" -gt 0 ]; then
	# Reruns must not overwrite the first-pass coverage.
	run_chunk "$1" "$CHUNK_DIR/rerun.out"
	exit $?
	fi
	status=0
	for chunk in "$CHUNK_DIR"/chunk-*; do
	RUN="-test.run=^($(paste -sd'\|' "$chunk"))$"
	run_chunk "$RUN" "$chunk.out" \|\| status=1
	done
	exit "$status"
	RUNNER
	status=0
	if gotestsum --format=standard-verbose --rerun-fails=2 --rerun-fails-max-failures=5 \
	--rerun-fails-report=flaky-tests-"$LANE"-"$IDX".txt \
	--junitfile junit-"$LANE"-"$IDX".xml --raw-command -- bash "$CHUNK_DIR/run.sh" \
	2>&1 \| grep --line-buffered -vE '^=== (RUN\|PAUSE\|CONT) ' \| tee test-"$LANE"-"$IDX".log; then
	:
	else
	status=1
	fi
	# The test binary runs under test2json, not the `go test` driver, so
	# nothing prints the `ok <pkg>` / `FAIL <pkg>` summary line the
	# coverage scorer classifies packages by (#14135). Append it.
	if [ "$status" -eq 0 ]; then
	printf 'ok \t%s\t(slice %s/%s)\n' "$TEST_PACKAGE" "$IDX" "$TOTAL" >> test-"$LANE"-"$IDX".log
	else
	printf 'FAIL\t%s\t(slice %s/%s)\n' "$TEST_PACKAGE" "$IDX" "$TOTAL" >> test-"$LANE"-"$IDX".log
	fi
	# Sum first-pass counters for identical blocks; never count reruns.
	# Do this even after test failure so the coverage gate has evidence.
	awk 'FNR == 1 { next } { key=$1 " " $2; hits[key]+=$3 }
	END { print "mode: atomic"; for (key in hits) print key, hits[key] }' \
	"$CHUNK_DIR"/chunk-*.out > coverage-"$LANE"-"$IDX".out
	exit "$status"

	- name: Surface flaky reruns
	id: flaky-reruns
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: bash "$GITHUB_WORKSPACE/.github/scripts/ci-gotestsum-flake-summary.sh" "flaky-tests-${{ matrix.lane }}-${{ matrix.idx }}.txt" "${{ matrix.lane }} ${{ matrix.idx }}/${{ matrix.total }}"

	- name: Upload flaky rerun report
	if: ${{ steps.select.outputs.run == 'true' && (always() && steps.flaky-reruns.outputs.has-flakes == 'true') }}
	uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	with:
	name: flaky-tests-${{ matrix.lane }}-${{ matrix.idx }}
	path: src/flaky-tests-${{ matrix.lane }}-${{ matrix.idx }}.txt
	if-no-files-found: error
	retention-days: 14

	- name: Slowest tests in this slice
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: bash "$GITHUB_WORKSPACE/.github/scripts/slowest-tests.sh" "${{ matrix.lane }} ${{ matrix.idx }}/${{ matrix.total }}" test-${{ matrix.lane }}-${{ matrix.idx }}.log

	# Uploaded even on failure so the coverage job (and humans) can inspect
	# the raw `go test` output. `warn` not `error`: on a compile failure the
	# coverprofile is never written, and this upload must not mask the real
	# test failure with an artifact error.
	- name: Upload shard coverage artifacts
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	with:
	name: test-shard-${{ matrix.lane }}-${{ matrix.idx }}
	path: \|
	src/coverage-${{ matrix.lane }}-${{ matrix.idx }}.out
	src/test-${{ matrix.lane }}-${{ matrix.idx }}.log
	if-no-files-found: warn
	retention-days: 1

	# crux/agent (the Crux module's manager package, pkg/agent before the
	# module cut) is a single package whose tests take ~429s serially on CI — it
	# was the entire "rest" shard's wall clock. Package-granular sharding cannot
	# split a package, so each of these jobs runs a deterministic 1/N slice of
	# the package's test functions. The slice is computed INSIDE the job (no
	# dependency on list-packages, so these start at t=0): every job derives the
	# same sorted `go test -list` output from the same commit and takes rows
	# where NR mod N == idx-1 — pairwise disjoint, jointly exhaustive, and a new
	# test function always lands in exactly one slice. N is ${{ strategy.job-total }},
	# so adding an index to the matrix rebalances automatically.
	#
	# Each slice produces a PARTIAL coverage profile for crux/agent; the coverage
	# job below merges them by summing per-block counts before scoring, so the
	# agent floor is applied to the union, never to a partial run.
	test-agent:
	needs: [required-context-scope, list-packages]
	if: ${{ (needs.required-context-scope.result != 'success' \|\| needs.required-context-scope.outputs.run-tests == 'true') && !(github.event_name == 'pull_request' && startsWith(github.head_ref, 'release-gate/')) }}
	name: test (agent ${{ matrix.idx }}/5)
	strategy:
	fail-fast: false
	matrix:
	idx: [1, 2, 3, 4, 5]
	runs-on: ${{ !inputs.integration_candidate && (github.event_name != 'pull_request' \|\| github.event.pull_request.head.repo.full_name == github.repository) && fromJSON(vars.HIVE_RUNNER_LABELS \|\| '["ubuntu-latest"]') \|\| 'ubuntu-latest' }}
	timeout-minutes: 25
	defaults:
	run:
	working-directory: src
	steps:
	# Affected-package selection (#12454): when this lane has nothing to
	# test on a PR it still runs and reports success, so no context the
	# aggregates or branch protection wait on can go missing. Anything but
	# an explicit "nothing selected" runs the lane (fail toward testing).
	- name: Affected-package selection
	id: select
	working-directory: .
	env:
	SELECTION: ${{ needs.list-packages.outputs.selection }}
	RUN_LANE: ${{ needs.list-packages.outputs.run-agent }}
	run: \|
	if [ "$SELECTION" = partial ] && [ "$RUN_LANE" = false ]; then
	echo "Affected-package selection: crux/agent is not affected by this diff; reporting success without running tests. Push and scheduled runs test it in full."
	echo "run=false" >> "$GITHUB_OUTPUT"
	else
	echo "run=true" >> "$GITHUB_OUTPUT"
	fi

	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	if: steps.select.outputs.run == 'true'
	with:
	persist-credentials: ${{ !inputs.integration_candidate }}

	# Populate the shared node-local Go tool cache under the runner pods'
	# node-wide lock before setup-go, so parallel jobs on a cold cache (a
	# src/go.mod bump) stop clobbering each other's rm -rf + copy (#11175,
	# #10234). Never fails the job.
	- name: Warm shared Go tool cache (serialize the cold-cache population)
	if: steps.select.outputs.run == 'true'
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-toolcache-guard.sh"
	- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
	if: steps.select.outputs.run == 'true'
	with:
	cache: false
	go-version-file: src/go.mod
	- name: Guard shared Go cache (fall back when the runner volume is full)
	if: steps.select.outputs.run == 'true'
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-cache-guard.sh"
	# Fill GOMODCACHE up front with bounded retries so a transient
	# proxy.golang.org stream reset (INTERNAL_ERROR, #14976) cannot fail the
	# job's first module-fetching step; see .github/actions/go-mod-download-retry.
	- name: Download Go modules (retry transient proxy resets)
	if: steps.select.outputs.run == 'true'
	uses: ./.github/actions/go-mod-download-retry

	- name: Install gotestsum
	if: steps.select.outputs.run == 'true'
	run: \|
	echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
	go install gotest.tools/gotestsum@v1.13.0
	"$(go env GOPATH)/bin/gotestsum" --version

	# Warm compile cache (#7480 follow-up): the warm-build-cache job on every
	# v4 push compiles the whole tree (race + norace) and saves GOCACHE keyed
	# by SHA; PR jobs restore the nearest ancestor via the prefix restore-key,
	# turning the ~2-minute per-job -race recompile into an incremental one.
	# Restore-only in PR jobs: balance/miss costs seconds, never correctness.
	- name: Restore warm Go build cache
	if: ${{ steps.select.outputs.run == 'true' && (!contains(vars.HIVE_RUNNER_LABELS, 'hive-runners')) }}
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: ~/.cache/go-build
	key: hive-go-build-${{ runner.os }}-${{ runner.arch }}-${{ github.sha }}
	restore-keys: \|
	hive-go-build-${{ runner.os }}-${{ runner.arch }}-

	- name: Restore cgo toolchain package cache
	if: steps.select.outputs.run == 'true'
	id: restore-cgo-apt-cache
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2
	restore-keys: \|
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-
	hive-cgo-race-apt-

	# Bounded + annotated by .github/scripts/ci-install-tool.sh (#6648/#6870/#6935):
	# skip apt when the tool already exists, install restored .debs offline
	# when possible, retry short egress flakes with backoff, and fail loudly
	# with a ::error:: if neither cache nor mirrors can provide the tool.
	- name: Prepare cgo toolchain for race tests
	if: steps.select.outputs.run == 'true'
	id: prepare-cgo-toolchain
	env:
	HIVE_CI_APT_CACHE_DIR: ${{ github.workspace }}/.ci/apt-cache/cgo-race
	HIVE_CI_MISSING_TOOLCHAIN_SOFT_FAIL: ${{ !inputs.integration_candidate && (github.event_name == 'schedule' \|\| github.event_name == 'workflow_dispatch') && '1' \|\| '0' }}
	run: \|
	bash "$GITHUB_WORKSPACE/.github/scripts/ci-install-tool.sh" \
	--require gcc --for '-race tests' \
	--apt 'gcc libc6-dev' --apk 'gcc musl-dev' --verify 'gcc --version'

	- name: Save cgo toolchain package cache
	if: ${{ steps.select.outputs.run == 'true' && (always() && steps.restore-cgo-apt-cache.outputs.cache-hit != 'true' && steps.prepare-cgo-toolchain.outputs.apt-cache-populated == 'true') }}
	continue-on-error: true
	uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2

	# tmux is what makes HIVE_TEST_REQUIRE_BEHAVIOURAL legitimate below.
	# ubuntu-latest happens to ship it, but "happens to" is not a guarantee —
	# a runner image change would silently turn ~50 crux/agent tests back into
	# skips, which is the exact failure #5388 exists to prevent. Install it
	# explicitly and fail the job if it is absent, so the flag below is
	# asserting something this lane actually enforces.
	# Same bounded/annotated install path as the cgo steps (#6648).
	- name: Install tmux (precondition for the behavioural gate)
	if: ${{ steps.select.outputs.run == 'true' && (steps.prepare-cgo-toolchain.outputs.toolchain-ready == 'true') }}
	run: \|
	bash "$GITHUB_WORKSPACE/.github/scripts/ci-install-tool.sh" \
	--require tmux --for 'the behavioural gate' \
	--apt tmux --apk tmux --verify 'tmux -V'

	- name: Test slice of crux/agent
	if: ${{ steps.select.outputs.run == 'true' && (steps.prepare-cgo-toolchain.outputs.toolchain-ready == 'true') }}
	env:
	IDX: ${{ matrix.idx }}
	TOTAL: ${{ strategy.job-total }}
	# This lane GUARANTEES the preconditions the crux/agent skips gate on:
	# tmux is installed and verified above, and the rest are t.TempDir()
	# writes and TestMain-built fixtures. So a skip at a converted site
	# here means the test broke, not that the environment is unsuitable —
	# make it fatal (#5388). Only sites judged guaranteed were converted;
	# genuine capability gates (tmuxAvailable() itself) stay permissive,
	# matching the shell half's treatment of docker/tmux gates.
	HIVE_TEST_REQUIRE_BEHAVIOURAL: '1'
	run: \|
	set -o pipefail
	ROOT="$PWD"
	export LC_ALL=C # Regex length guards below must count bytes.
	export TEST_PACKAGE=github.com/hivecommons/hive/crux/agent
	export TEST_DIR="$ROOT/crux/agent"
	export TEST_BINARY="$ROOT/agent.test"
	go test -c -race -cover -o "$TEST_BINARY" ./crux/agent
	# Include Test/Example/Fuzz, but not opt-in benchmarks.
	(cd "$TEST_DIR" && "$TEST_BINARY" -test.short -test.list '.*') \| grep -E '^(Test\|Example\|Fuzz)' \| LC_ALL=C sort > all-tests.txt
	if [ ! -s all-tests.txt ]; then
	echo "::error::go test -list produced no test functions for crux/agent — refusing to run an empty slice (fail closed)"
	exit 1
	fi
	awk -v n="$TOTAL" -v i="$IDX" 'NR % n == i - 1' all-tests.txt > slice.txt
	echo "slice $IDX/$TOTAL: $(wc -l < slice.txt) of $(wc -l < all-tests.txt) test functions"
	if [ ! -s slice.txt ]; then
	echo "::error::empty agent slice — failing closed"
	exit 1
	fi
	export CHUNK_DIR TEST_SHUFFLE=off
	CHUNK_DIR=$(mktemp -d "$ROOT/test-chunks.XXXXXX")
	split -l 1000 slice.txt "$CHUNK_DIR/chunk-"
	if [ "$GITHUB_EVENT_NAME" != "pull_request" ]; then TEST_SHUFFLE=on; fi
	# raw-command preserves one JUnit/flake report across all chunks.
	# gotestsum appends -test.run and package arguments on reruns.
	cat > "$CHUNK_DIR/run.sh" <<'RUNNER'
	#!/usr/bin/env bash
	set -uo pipefail
	cd "$TEST_DIR" \|\| exit 1
	run_chunk() {
	local RUN="$1" PROFILE="$2"
	# MAX_ARG_STRLEN is 131072 bytes; never approach it, even if a
	# future test has an exceptionally long name. Guard before exec.
	if [ "${#RUN}" -gt 120000 ]; then
	echo "::error::chunk regex exceeds 120000 bytes; reduce chunk size" >&2
	return 1
	fi
	go tool test2json -t -p "$TEST_PACKAGE" "$TEST_BINARY" \
	-test.short -test.count=1 -test.v=test2json -test.timeout=10m \
	-test.shuffle="$TEST_SHUFFLE" "$RUN" -test.coverprofile="$PROFILE"
	}
	if [ "$#" -gt 0 ]; then
	# Reruns must not overwrite the first-pass coverage.
	run_chunk "$1" "$CHUNK_DIR/rerun.out"
	exit $?
	fi
	status=0
	for chunk in "$CHUNK_DIR"/chunk-*; do
	RUN="-test.run=^($(paste -sd'\|' "$chunk"))$"
	run_chunk "$RUN" "$chunk.out" \|\| status=1
	done
	exit "$status"
	RUNNER
	status=0
	if gotestsum --format=standard-verbose --rerun-fails=2 --rerun-fails-max-failures=5 \
	--rerun-fails-report=flaky-tests-agent-"$IDX".txt \
	--junitfile junit-agent-"$IDX".xml --raw-command -- bash "$CHUNK_DIR/run.sh" \
	2>&1 \| grep --line-buffered -vE '^=== (RUN\|PAUSE\|CONT) ' \| tee test-agent-"$IDX".log; then
	:
	else
	status=1
	fi
	# The test binary runs under test2json, not the `go test` driver, so
	# nothing prints the `ok <pkg>` / `FAIL <pkg>` summary line the
	# coverage scorer classifies packages by (#14135). Append it.
	if [ "$status" -eq 0 ]; then
	printf 'ok \t%s\t(slice %s/%s)\n' "$TEST_PACKAGE" "$IDX" "$TOTAL" >> test-agent-"$IDX".log
	else
	printf 'FAIL\t%s\t(slice %s/%s)\n' "$TEST_PACKAGE" "$IDX" "$TOTAL" >> test-agent-"$IDX".log
	fi
	# Sum first-pass counters for identical blocks; never count reruns.
	# Do this even after test failure so the coverage gate has evidence.
	awk 'FNR == 1 { next } { key=$1 " " $2; hits[key]+=$3 }
	END { print "mode: atomic"; for (key in hits) print key, hits[key] }' \
	"$CHUNK_DIR"/chunk-*.out > coverage-agent-"$IDX".out
	exit "$status"

	- name: Surface flaky reruns
	id: flaky-reruns
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: bash "$GITHUB_WORKSPACE/.github/scripts/ci-gotestsum-flake-summary.sh" "flaky-tests-agent-${{ matrix.idx }}.txt" "agent ${{ matrix.idx }}/5"

	- name: Upload flaky rerun report
	if: ${{ steps.select.outputs.run == 'true' && (always() && steps.flaky-reruns.outputs.has-flakes == 'true') }}
	uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	with:
	name: flaky-tests-agent-${{ matrix.idx }}
	path: src/flaky-tests-agent-${{ matrix.idx }}.txt
	if-no-files-found: error
	retention-days: 14

	- name: Slowest tests in this slice
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: bash "$GITHUB_WORKSPACE/.github/scripts/slowest-tests.sh" "agent ${{ matrix.idx }}/5" test-agent-${{ matrix.idx }}.log

	- name: Upload shard coverage artifacts
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	with:
	name: test-shard-agent-${{ matrix.idx }}
	path: \|
	src/coverage-agent-${{ matrix.idx }}.out
	src/test-agent-${{ matrix.idx }}.log
	if-no-files-found: warn
	retention-days: 1

	# Partition everything EXCEPT pkg/hub, crux/agent, pkg/dashboard and
	# pkg/scheduler (which have dedicated sliced jobs above) into R balanced
	# buckets, published as a matrix for test-rest. Greedy
	# longest-processing-time assignment over static weight hints (measured
	# seconds on CI under -short -race; unlisted packages default to 1s — they
	# are all ≤ 5s). The hints only affect BALANCE, never MEMBERSHIP: a stale
	# weight makes a bucket slower, not a package untested.
	list-packages:
	# Discovery uses only the checkout/event, not the scope verdict. Start it
	# alongside scope instead of paying another runner-queue round trip.
	# Shards and both required aggregates still depend on successful scope.
	if: ${{ !(github.event_name == 'pull_request' && startsWith(github.head_ref, 'release-gate/')) }}
	runs-on: ${{ !inputs.integration_candidate && (github.event_name != 'pull_request' \|\| github.event.pull_request.head.repo.full_name == github.repository) && fromJSON(vars.HIVE_RUNNER_LABELS \|\| '["ubuntu-latest"]') \|\| 'ubuntu-latest' }}
	timeout-minutes: 15
	outputs:
	matrix: ${{ steps.partition.outputs.matrix }}
	# Affected-package selection (#12454). `full` on every non-PR event.
	selection: ${{ steps.select.outputs.selection }}
	selected: ${{ steps.select.outputs.selected }}
	run-hub: ${{ steps.select.outputs.run-hub }}
	run-agent: ${{ steps.select.outputs.run-agent }}
	run-dashboard: ${{ steps.select.outputs.run-dashboard }}
	run-scheduler: ${{ steps.select.outputs.run-scheduler }}
	# Crux boundary filter (#13536): crux-only \| none \| mixed \| full.
	crux-scope: ${{ steps.select.outputs.crux-scope }}
	defaults:
	run:
	working-directory: src
	steps:
	# Depth 2 so the PR merge commit's two parents (base tip, PR head) are
	# present for the affected-package diff below.
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	persist-credentials: ${{ !inputs.integration_candidate }}
	fetch-depth: 2

	# Populate the shared node-local Go tool cache under the runner pods'
	# node-wide lock before setup-go, so parallel jobs on a cold cache (a
	# src/go.mod bump) stop clobbering each other's rm -rf + copy (#11175,
	# #10234). Never fails the job.
	- name: Warm shared Go tool cache (serialize the cold-cache population)
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-toolcache-guard.sh"
	- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
	with:
	cache: false
	go-version-file: src/go.mod
	- name: Guard shared Go cache (fall back when the runner volume is full)
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-cache-guard.sh"
	# Fill GOMODCACHE up front with bounded retries so a transient
	# proxy.golang.org stream reset (INTERNAL_ERROR, #14976) cannot fail the
	# job's first module-fetching step; see .github/actions/go-mod-download-retry.
	- name: Download Go modules (retry transient proxy resets)
	uses: ./.github/actions/go-mod-download-retry

	# Warm compile cache (#7480 follow-up): the warm-build-cache job on every
	# v4 push compiles the whole tree (race + norace) and saves GOCACHE keyed
	# by SHA; PR jobs restore the nearest ancestor via the prefix restore-key,
	# turning the ~2-minute per-job -race recompile into an incremental one.
	# Restore-only in PR jobs: balance/miss costs seconds, never correctness.
	- name: Restore warm Go build cache
	if: ${{ !contains(vars.HIVE_RUNNER_LABELS, 'hive-runners') }}
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: ~/.cache/go-build
	key: hive-go-build-${{ runner.os }}-${{ runner.arch }}-${{ github.sha }}
	restore-keys: \|
	hive-go-build-${{ runner.os }}-${{ runner.arch }}-

	- name: Restore cgo toolchain package cache
	id: restore-cgo-apt-cache
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2
	restore-keys: \|
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-
	hive-cgo-race-apt-

	# Bounded + annotated by .github/scripts/ci-install-tool.sh (#6648/#6870/#6935):
	# skip apt when the tool already exists, install restored .debs offline
	# when possible, retry short egress flakes with backoff, and fail loudly
	# with a ::error:: if neither cache nor mirrors can provide the tool.
	- name: Prepare cgo toolchain for race tests
	id: prepare-cgo-toolchain
	env:
	HIVE_CI_APT_CACHE_DIR: ${{ github.workspace }}/.ci/apt-cache/cgo-race
	HIVE_CI_MISSING_TOOLCHAIN_SOFT_FAIL: ${{ !inputs.integration_candidate && (github.event_name == 'schedule' \|\| github.event_name == 'workflow_dispatch') && '1' \|\| '0' }}
	run: \|
	bash "$GITHUB_WORKSPACE/.github/scripts/ci-install-tool.sh" \
	--require gcc --for '-race tests' \
	--apt 'gcc libc6-dev' --apk 'gcc musl-dev' --verify 'gcc --version'

	- name: Save cgo toolchain package cache
	if: always() && steps.restore-cgo-apt-cache.outputs.cache-hit != 'true' && steps.prepare-cgo-toolchain.outputs.apt-cache-populated == 'true'
	continue-on-error: true
	uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2

	# Affected-package selection (#12454). Pull requests only: push to the
	# release branches, the hourly schedule and manual dispatches always run
	# the full suite and are the safety net for anything selection misses.
	# The PR checkout is the merge commit, so HEAD^1..HEAD is exactly the
	# change this PR makes to the current base; anything unexpected (not a
	# merge commit, second parent is not the PR head, analysis failure) and
	# every fallback in .github/scripts/affected-go-packages.sh mean `all`.
	# Set the repo variable HIVE_AFFECTED_TEST_SELECTION=off to force the
	# full suite on PRs without a code change.
	- name: Select affected packages
	id: select
	env:
	PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
	SELECTION_SWITCH: ${{ vars.HIVE_AFFECTED_TEST_SELECTION }}
	run: \|
	set -uo pipefail
	sel=../affected-pkgs.txt
	if [ "$GITHUB_EVENT_NAME" != pull_request ]; then
	echo "$GITHUB_EVENT_NAME run: full suite (only pull requests use affected-package selection)."
	echo all > "$sel"
	elif [ "${SELECTION_SWITCH:-}" = off ]; then
	echo "HIVE_AFFECTED_TEST_SELECTION=off: full suite."
	echo all > "$sel"
	elif [ -z "${PR_HEAD_SHA:-}" ] \|\| [ "$(git rev-parse -q --verify 'HEAD^2' \|\| true)" != "$PR_HEAD_SHA" ]; then
	echo "::notice::checkout is not the PR merge commit of $PR_HEAD_SHA; running the full suite"
	echo all > "$sel"
	else
	bash "$GITHUB_WORKSPACE/.github/scripts/affected-go-packages.sh" 'HEAD^1' HEAD "$sel" \|\| echo all > "$sel"
	fi
	[ -f "$sel" ] \|\| echo all > "$sel"
	# src/go.work puts `go list -m` in workspace mode, where it prints
	# every workspace module; the selection file is keyed by the root
	# module's import paths, so ask for that module alone.
	mod=$(GOWORK=off go list -m)
	crux_scope=full
	if [ "$(head -n1 "$sel")" = all ]; then
	selection=full
	run_hub=true run_agent=true run_dashboard=true run_scheduler=true
	else
	selection=partial
	has() { grep -Fxq "$mod/$1" "$sel" && echo true \|\| echo false; }
	run_hub=$(has pkg/hub) run_agent=$(has crux/agent) run_dashboard=$(has pkg/dashboard) run_scheduler=$(has pkg/scheduler)
	# Crux boundary filter (#13536) — see the header comment. Only a
	# partial selection is ever narrowed; the script answers `full`
	# for every fail-safe path and for anything it cannot read, and
	# `full` leaves the selection exactly as the dependency analysis
	# computed it.
	crux_scope=$(bash "$GITHUB_WORKSPACE/.github/scripts/crux-scope.sh" 'HEAD^1' HEAD) \|\| crux_scope=full
	case "$crux_scope" in
	crux-only)
	run_hub=false run_dashboard=false run_scheduler=false run_agent=true
	# Rest buckets: the selected Crux packages plus the always-on
	# tree-scanning packages (mirrors ALWAYS_PATTERNS in
	# affected-go-packages.sh); a failure here falls back to the
	# unnarrowed selection.
	if go list ./crux/... \
	./internal/... ./pkg/config ./pkg/rotation > ../crux-pkgs.txt 2>/dev/null \
	&& [ -s ../crux-pkgs.txt ]; then
	grep -Fx -f ../crux-pkgs.txt "$sel" > "$sel.crux" \|\| true
	echo "crux-only diff: narrowing $(grep -c . "$sel" \|\| true) selected packages to $(grep -c . "$sel.crux" \|\| true) Crux/always-on packages; hub, dashboard and scheduler slices and the dashboard shuffle lane will not run"
	mv "$sel.crux" "$sel"
	else
	echo "::warning::crux-only diff but go list of the Crux set failed; keeping the unnarrowed selection"
	crux_scope=mixed
	fi
	;;
	none)
	# Drop crux/agent from the selection too: the coverage job
	# requires every selected package to have produced a result,
	# and the agent slices are the only lane that tests it.
	run_agent=false
	grep -Fxv "$mod/crux/agent" "$sel" > "$sel.nocrux" \|\| true
	mv "$sel.nocrux" "$sel"
	echo "no Crux package changed: agent slices will not run"
	;;
	mixed\|full) ;;
	*)
	echo "::warning::unexpected crux-scope verdict '$crux_scope'; treating as mixed"
	crux_scope=mixed
	;;
	esac
	fi
	{
	echo "selection=$selection"
	echo "selected=$(paste -sd' ' "$sel")"
	echo "run-hub=$run_hub"
	echo "run-agent=$run_agent"
	echo "run-dashboard=$run_dashboard"
	echo "run-scheduler=$run_scheduler"
	echo "crux-scope=$crux_scope"
	} >> "$GITHUB_OUTPUT"
	{
	echo "## Go test selection"
	echo ""
	if [ "$selection" = full ]; then
	echo "Full suite."
	else
	echo "Affected packages only ($(grep -c . "$sel" \|\| true)); crux-scope=$crux_scope; hub=$run_hub agent=$run_agent dashboard=$run_dashboard scheduler=$run_scheduler."
	echo ""
	sed 's/^/- /' "$sel"
	fi
	} >> "$GITHUB_STEP_SUMMARY"
	echo "selection=$selection crux-scope=$crux_scope hub=$run_hub agent=$run_agent dashboard=$run_dashboard scheduler=$run_scheduler"

	- name: Partition packages into balanced buckets
	id: partition
	run: \|
	R=4
	# Drop only the monolithic packages that have function-sliced jobs
	# of their own: pkg/hub (test-hub), pkg/dashboard and pkg/scheduler
	# (test-sliced). All their subpackages belong in the rest buckets
	# so coverage floors and the no-test-files gate apply to them too.
	# pkg/hubbackup, pkg/dashboard/webstatic et al. stay here.
	# ./internal/... carries the shared test helpers (internal/testutil)
	# and their contract tests, such as the time.Sleep ratchet in
	# sleep_ratchet_test.go. ./crux/... is the nested Crux module
	# (#13536), reachable from src/ through src/go.work; its manager
	# package crux/agent has the dedicated agent slices.
	go list ./pkg/... ./cmd/... ./internal/... ./crux/... \
	\| awk '$0 == "github.com/hivecommons/hive/pkg/hub" { next }
	$0 == "github.com/hivecommons/hive/pkg/dashboard" { next }
	$0 == "github.com/hivecommons/hive/pkg/scheduler" { next }
	{ print }' \
	\| grep -Ev '^github\.com/hivecommons/hive/crux/agent$' \
	\| LC_ALL=C sort > rest-pkgs.txt
	if [ ! -s rest-pkgs.txt ]; then
	echo "::error::go list produced no rest packages — failing closed"
	exit 1
	fi
	# Affected-package selection narrows membership only after the
	# fail-closed check above ran on the full list. The bucket count
	# stays R, so every `test (rest i/R)` context still reports; a
	# bucket left empty succeeds without running (see test-rest).
	if [ "$(head -n1 ../affected-pkgs.txt)" != all ]; then
	grep -Fx -f ../affected-pkgs.txt rest-pkgs.txt > rest-selected.txt \|\| true
	echo "affected-package selection: $(wc -l < rest-selected.txt) of $(wc -l < rest-pkgs.txt) rest packages"
	mv rest-selected.txt rest-pkgs.txt
	fi

	# Weights = measured -short -race package wall seconds on the
	# hive-runners fleet, mean of two v5 push runs (40383615331 and
	# 40382244011, 2026-10-26, rounded). Re-measure with: for each
	# `test (rest i/R)` job log, grep the
	# `ok github.com/hivecommons/hive/<pkg> <N>s` lines. Anything
	# under ~5s is noise and takes the 1s default. Packages inside a
	# bucket run concurrently under one go test, so a bucket's wall
	# clock is its heaviest package plus compile time — the table
	# mainly has to keep the big ones apart, and the sum-based greedy
	# below does that while spreading the compile load. pkg/hub/spoke
	# is the floor now that pkg/dashboard and pkg/scheduler are sliced
	# (test-sliced above): the widest bucket can never be faster than
	# it. pkg/dashboard and pkg/scheduler are listed so a stale
	# exclusion above still yields a sane partition. pkg/github is
	# ~1s with ~1% coverage on push runs but 70s on pull requests (PR
	# 14407's run 40387365403), where it is weighed — measure it from a
	# PR run, not a push run.
	weight() {
	case "$1" in
	*/pkg/dashboard) echo 219 ;;
	*/pkg/scheduler) echo 199 ;;
	*/pkg/hub/spoke) echo 85 ;;
	*/pkg/config) echo 82 ;;
	*/pkg/github) echo 70 ;;
	*/cmd/hive) echo 69 ;;
	*/crux/proxy) echo 56 ;;
	*/pkg/dashboard/webstatic) echo 35 ;;
	*/pkg/hubbackup) echo 30 ;;
	*/pkg/advisory) echo 29 ;;
	*/internal/testutil) echo 22 ;;
	*/pkg/ioscan) echo 16 ;;
	*/pkg/hivectl) echo 15 ;;
	*/pkg/tui) echo 14 ;;
	*/pkg/discord) echo 12 ;;
	*/pkg/fleetreport) echo 11 ;;
	*/pkg/hivectl/commands) echo 10 ;;
	*/pkg/stageengine) echo 10 ;;
	*/pkg/apphealth) echo 9 ;;
	*/pkg/rotation) echo 9 ;;
	*/pkg/beads) echo 8 ;;
	*/pkg/github/automerge) echo 8 ;;
	*/crux/api/callback) echo 6 ;;
	*/crux/inference) echo 6 ;;
	*/cmd/hive-backup) echo 5 ;;
	*/pkg/mint) echo 5 ;;
	*/pkg/policies) echo 5 ;;
	*) echo 1 ;;
	esac
	}

	# Deterministic greedy LPT: stable-sort by weight desc (name asc on
	# ties), assign each package to the currently lightest bucket
	# (lowest index on ties).
	while read -r p; do
	echo "$(weight "$p") $p"
	done < rest-pkgs.txt \| sort -k1,1nr -k2,2 > weighted.txt

	declare -a LOAD BUCKET
	for ((i=0; i<R; i++)); do LOAD[i]=0; BUCKET[i]=""; done
	while read -r w p; do
	best=0
	for ((i=1; i<R; i++)); do
	if [ "${LOAD[i]}" -lt "${LOAD[best]}" ]; then best=$i; fi
	done
	BUCKET[best]="${BUCKET[best]} $p"
	LOAD[best]=$(( LOAD[best] + w ))
	done < weighted.txt

	INCLUDE=""
	for ((i=0; i<R; i++)); do
	pkgs="${BUCKET[i]# }"
	echo "bucket $((i+1))/$R (weight ${LOAD[i]}): $pkgs"
	[ -n "$INCLUDE" ] && INCLUDE="$INCLUDE,"
	INCLUDE="$INCLUDE{\"idx\":$((i+1)),\"total\":$R,\"pkgs\":\"$pkgs\"}"
	done
	echo "matrix={\"include\":[$INCLUDE]}" >> "$GITHUB_OUTPUT"

	test-rest:
	if: ${{ (needs.required-context-scope.result != 'success' \|\| needs.required-context-scope.outputs.run-tests == 'true') && !(github.event_name == 'pull_request' && startsWith(github.head_ref, 'release-gate/')) }}
	name: test (rest ${{ matrix.idx }}/${{ matrix.total }})
	needs: [required-context-scope, list-packages]
	strategy:
	fail-fast: false
	matrix: ${{ fromJSON(needs.list-packages.outputs.matrix) }}
	runs-on: ${{ !inputs.integration_candidate && (github.event_name != 'pull_request' \|\| github.event.pull_request.head.repo.full_name == github.repository) && fromJSON(vars.HIVE_RUNNER_LABELS \|\| '["ubuntu-latest"]') \|\| 'ubuntu-latest' }}
	timeout-minutes: 25
	defaults:
	run:
	working-directory: src
	steps:
	# Affected-package selection (#12454): when this lane has nothing to
	# test on a PR it still runs and reports success, so no context the
	# aggregates or branch protection wait on can go missing. Anything but
	# an explicit "nothing selected" runs the lane (fail toward testing).
	- name: Affected-package selection
	id: select
	working-directory: .
	env:
	SELECTION: ${{ needs.list-packages.outputs.selection }}
	PKGS: ${{ matrix.pkgs }}
	run: \|
	if [ "$SELECTION" = partial ] && [ -z "$PKGS" ]; then
	echo "Affected-package selection: no affected package landed in this bucket; reporting success without running tests. Push and scheduled runs test it in full."
	echo "run=false" >> "$GITHUB_OUTPUT"
	else
	echo "run=true" >> "$GITHUB_OUTPUT"
	fi

	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	if: steps.select.outputs.run == 'true'
	with:
	persist-credentials: ${{ !inputs.integration_candidate }}

	# Populate the shared node-local Go tool cache under the runner pods'
	# node-wide lock before setup-go, so parallel jobs on a cold cache (a
	# src/go.mod bump) stop clobbering each other's rm -rf + copy (#11175,
	# #10234). Never fails the job.
	- name: Warm shared Go tool cache (serialize the cold-cache population)
	if: steps.select.outputs.run == 'true'
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-toolcache-guard.sh"
	- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
	if: steps.select.outputs.run == 'true'
	with:
	cache: false
	go-version-file: src/go.mod
	# Dashboard tests execute JavaScript even on runners without preinstalled Node.
	- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
	if: steps.select.outputs.run == 'true'
	with:
	node-version: '22'
	- name: Guard shared node tool cache (repair a half-written entry)
	# setup-node reports "Found in cache" for a node/<ver>/x64 entry whose
	# .complete marker exists but bin/npm does not (#10505, #10518). The
	# canary repairs this too, but its cron is starved to a ~5h cadence, so
	# heal the node from the hot path: drop the marker (never the directory,
	# a sibling job may be mid-run) so the next job re-extracts.
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: \|
	set -u
	command -v npm >/dev/null 2>&1 && exit 0
	for marker in "${RUNNER_TOOL_CACHE:-/nonexistent}"/node/*/x64.complete; do
	[ -e "$marker" ] \|\| continue
	dir="${marker%.complete}"
	[ -x "$dir/bin/npm" ] && continue
	echo "::warning::node tool cache $dir has a .complete marker but no executable bin/npm; removing the marker so the next job on $RUNNER_NAME rebuilds it"
	rm -f "$marker"
	done
	echo "::error::npm is not on PATH after actions/setup-node on $RUNNER_NAME (half-written shared tool cache, see #10518); marker removed, re-run this job"
	exit 1
	- name: Guard shared Go cache (fall back when the runner volume is full)
	if: steps.select.outputs.run == 'true'
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-cache-guard.sh"
	# Fill GOMODCACHE up front with bounded retries so a transient
	# proxy.golang.org stream reset (INTERNAL_ERROR, #14976) cannot fail the
	# job's first module-fetching step; see .github/actions/go-mod-download-retry.
	- name: Download Go modules (retry transient proxy resets)
	if: steps.select.outputs.run == 'true'
	uses: ./.github/actions/go-mod-download-retry

	- name: Install gotestsum
	if: steps.select.outputs.run == 'true'
	run: \|
	echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
	go install gotest.tools/gotestsum@v1.13.0
	"$(go env GOPATH)/bin/gotestsum" --version

	# Warm compile cache (#7480 follow-up): the warm-build-cache job on every
	# v4 push compiles the whole tree (race + norace) and saves GOCACHE keyed
	# by SHA; PR jobs restore the nearest ancestor via the prefix restore-key,
	# turning the ~2-minute per-job -race recompile into an incremental one.
	# Restore-only in PR jobs: balance/miss costs seconds, never correctness.
	- name: Restore warm Go build cache
	if: ${{ steps.select.outputs.run == 'true' && (!contains(vars.HIVE_RUNNER_LABELS, 'hive-runners')) }}
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: ~/.cache/go-build
	key: hive-go-build-${{ runner.os }}-${{ runner.arch }}-${{ github.sha }}
	restore-keys: \|
	hive-go-build-${{ runner.os }}-${{ runner.arch }}-

	- name: Restore cgo toolchain package cache
	if: steps.select.outputs.run == 'true'
	id: restore-cgo-apt-cache
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2
	restore-keys: \|
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-
	hive-cgo-race-apt-

	# Bounded + annotated by .github/scripts/ci-install-tool.sh (#6648/#6870/#6935):
	# skip apt when the tool already exists, install restored .debs offline
	# when possible, retry short egress flakes with backoff, and fail loudly
	# with a ::error:: if neither cache nor mirrors can provide the tool.
	- name: Prepare cgo toolchain for race tests
	if: steps.select.outputs.run == 'true'
	id: prepare-cgo-toolchain
	env:
	HIVE_CI_APT_CACHE_DIR: ${{ github.workspace }}/.ci/apt-cache/cgo-race
	HIVE_CI_MISSING_TOOLCHAIN_SOFT_FAIL: ${{ !inputs.integration_candidate && (github.event_name == 'schedule' \|\| github.event_name == 'workflow_dispatch') && '1' \|\| '0' }}
	run: \|
	bash "$GITHUB_WORKSPACE/.github/scripts/ci-install-tool.sh" \
	--require gcc --for '-race tests' \
	--apt 'gcc libc6-dev' --apk 'gcc musl-dev' --verify 'gcc --version'

	- name: Save cgo toolchain package cache
	if: ${{ steps.select.outputs.run == 'true' && (always() && steps.restore-cgo-apt-cache.outputs.cache-hit != 'true' && steps.prepare-cgo-toolchain.outputs.apt-cache-populated == 'true') }}
	continue-on-error: true
	uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2

	# ./cmd/... is included in the partition deliberately. It used to be
	# ./pkg/... only, so no workflow anywhere ran the cmd tests — four cmd/bd
	# tests were consequently merged already-failing and stayed red on the
	# default branch, undetected. All of ./cmd/... passes under -short -race.
	- name: Test
	if: ${{ steps.select.outputs.run == 'true' && (steps.prepare-cgo-toolchain.outputs.toolchain-ready == 'true') }}
	env:
	PKGS: ${{ matrix.pkgs }}
	# The converted skips in the rest packages gate on repo files being
	# present in the checkout (deploy/entrypoint.sh, Dockerfile,
	# deploy/nginx.conf, config/backends.conf, bin/gh-app-token.sh,
	# bin/hive-open-pr.sh) — a precondition actions/checkout above
	# GUARANTEES here, so a skip at a converted site means the test
	# broke, not that the environment is unsuitable (#5388). Only
	# testutil.SkipUnlessRequired sites become fatal; plain t.Skip
	# capability gates (root, symlinks, git, /data writability, tool
	# availability) are unaffected and stay permissive.
	HIVE_TEST_REQUIRE_BEHAVIOURAL: '1'
	run: \|
	set -o pipefail
	if [ -z "$PKGS" ]; then
	echo "::error::empty package bucket — failing closed"
	exit 1
	fi
	# -timeout=20m: kept from when pkg/dashboard lived in these buckets
	# and nearly exhausted go test's default 10m on loaded PR runners
	# (#12113); a loaded runner can still stretch pkg/hub/spoke.
	SHUFFLE=""
	if [ "$GITHUB_EVENT_NAME" != "pull_request" ]; then SHUFFLE="-shuffle=on"; fi
	# shellcheck disable=SC2086 # PKGS and SHUFFLE are deliberate word-split lists
	bash "$GITHUB_WORKSPACE/.github/scripts/ci-gotestsum-preserve-coverprofile.sh" \
	gotestsum --format=standard-verbose --rerun-fails=2 --rerun-fails-max-failures=5 \
	--rerun-fails-report=flaky-tests-rest-${{ matrix.idx }}.txt \
	--packages="$PKGS" --junitfile junit-rest-${{ matrix.idx }}.xml \
	-- -short -race -count=1 -timeout=20m -v $SHUFFLE -coverprofile=coverage-rest-${{ matrix.idx }}.out \
	2>&1 \| grep --line-buffered -vE '^=== (RUN\|PAUSE\|CONT) ' \| tee test-rest-${{ matrix.idx }}.log

	- name: Surface flaky reruns
	id: flaky-reruns
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: bash "$GITHUB_WORKSPACE/.github/scripts/ci-gotestsum-flake-summary.sh" "flaky-tests-rest-${{ matrix.idx }}.txt" "rest ${{ matrix.idx }}/${{ matrix.total }}"

	- name: Upload flaky rerun report
	if: ${{ steps.select.outputs.run == 'true' && (always() && steps.flaky-reruns.outputs.has-flakes == 'true') }}
	uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	with:
	name: flaky-tests-rest-${{ matrix.idx }}
	path: src/flaky-tests-rest-${{ matrix.idx }}.txt
	if-no-files-found: error
	retention-days: 14

	- name: Slowest tests in this bucket
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: bash "$GITHUB_WORKSPACE/.github/scripts/slowest-tests.sh" "rest ${{ matrix.idx }}/${{ matrix.total }}" test-rest-${{ matrix.idx }}.log

	- name: Upload shard coverage artifacts
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	with:
	name: test-shard-rest-${{ matrix.idx }}
	path: \|
	src/coverage-rest-${{ matrix.idx }}.out
	src/test-rest-${{ matrix.idx }}.log
	if-no-files-found: warn
	retention-days: 1

	# Dedicated shuffled-order lane for pkg/dashboard (#5570). The rest bucket
	# above shuffles only on NON-pull_request runs, so an order-dependent test
	# merged today fails weeks later on an unlucky scheduled run — #5553 was
	# exactly that shape (a cachedHealth cache leak that passed five runs before
	# and after the one hostile ordering). This lane also runs at PR time, with
	# the two bounds the #5570 investigation showed the lane needs to stay
	# signal rather than noise:
	# - parallelism is constrained: the package stands up hundreds of httptest
	# servers, and unconstrained parallel fan-out exhausts ephemeral ports on
	# a loaded machine — that noise (`connect: can't assign requested
	# address`) is what inflated #5570's original "83 failures" figure. A
	# flaky detector gets ignored, which is worse than no detector.
	# - the seed is surfaced in the failure annotation: -shuffle=on draws a
	# fresh seed each run, and without the seed a real order-dependence
	# finding is unreproducible.
	# No -coverprofile here: pkg/dashboard's coverage floor is scored from the
	# merged test-sliced `test (dashboard i/D)` profiles; this lane exists only
	# to vary the execution order.
	test-dashboard-shuffle:
	needs: [required-context-scope, list-packages]
	if: ${{ (needs.required-context-scope.result != 'success' \|\| needs.required-context-scope.outputs.run-tests == 'true') && !(github.event_name == 'pull_request' && startsWith(github.head_ref, 'release-gate/')) }}
	name: test (dashboard shuffle ${{ matrix.idx }}/3)
	strategy:
	fail-fast: false
	matrix:
	idx: [1, 2, 3]
	runs-on: ${{ !inputs.integration_candidate && (github.event_name != 'pull_request' \|\| github.event.pull_request.head.repo.full_name == github.repository) && fromJSON(vars.HIVE_RUNNER_LABELS \|\| '["ubuntu-latest"]') \|\| 'ubuntu-latest' }}
	timeout-minutes: 20
	defaults:
	run:
	working-directory: src
	steps:
	# Affected-package selection (#12454): when this lane has nothing to
	# test on a PR it still runs and reports success, so no context the
	# aggregates or branch protection wait on can go missing. Anything but
	# an explicit "nothing selected" runs the lane (fail toward testing).
	- name: Affected-package selection
	id: select
	working-directory: .
	env:
	SELECTION: ${{ needs.list-packages.outputs.selection }}
	RUN_LANE: ${{ needs.list-packages.outputs.run-dashboard }}
	run: \|
	if [ "$SELECTION" = partial ] && [ "$RUN_LANE" = false ]; then
	echo "Affected-package selection: pkg/dashboard is not affected by this diff; reporting success without running tests. Push and scheduled runs test it in full."
	echo "run=false" >> "$GITHUB_OUTPUT"
	else
	echo "run=true" >> "$GITHUB_OUTPUT"
	fi

	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	if: steps.select.outputs.run == 'true'
	with:
	persist-credentials: ${{ !inputs.integration_candidate }}

	# Populate the shared node-local Go tool cache under the runner pods'
	# node-wide lock before setup-go, so parallel jobs on a cold cache (a
	# src/go.mod bump) stop clobbering each other's rm -rf + copy (#11175,
	# #10234). Never fails the job.
	- name: Warm shared Go tool cache (serialize the cold-cache population)
	if: steps.select.outputs.run == 'true'
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-toolcache-guard.sh"
	- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
	if: steps.select.outputs.run == 'true'
	with:
	cache: false
	go-version-file: src/go.mod
	# Dashboard tests execute JavaScript even on runners without preinstalled Node.
	- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
	if: steps.select.outputs.run == 'true'
	with:
	node-version: '22'
	- name: Guard shared node tool cache (repair a half-written entry)
	# setup-node reports "Found in cache" for a node/<ver>/x64 entry whose
	# .complete marker exists but bin/npm does not (#10505, #10518). The
	# canary repairs this too, but its cron is starved to a ~5h cadence, so
	# heal the node from the hot path: drop the marker (never the directory,
	# a sibling job may be mid-run) so the next job re-extracts.
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: \|
	set -u
	command -v npm >/dev/null 2>&1 && exit 0
	for marker in "${RUNNER_TOOL_CACHE:-/nonexistent}"/node/*/x64.complete; do
	[ -e "$marker" ] \|\| continue
	dir="${marker%.complete}"
	[ -x "$dir/bin/npm" ] && continue
	echo "::warning::node tool cache $dir has a .complete marker but no executable bin/npm; removing the marker so the next job on $RUNNER_NAME rebuilds it"
	rm -f "$marker"
	done
	echo "::error::npm is not on PATH after actions/setup-node on $RUNNER_NAME (half-written shared tool cache, see #10518); marker removed, re-run this job"
	exit 1
	- name: Guard shared Go cache (fall back when the runner volume is full)
	if: steps.select.outputs.run == 'true'
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-cache-guard.sh"
	# Fill GOMODCACHE up front with bounded retries so a transient
	# proxy.golang.org stream reset (INTERNAL_ERROR, #14976) cannot fail the
	# job's first module-fetching step; see .github/actions/go-mod-download-retry.
	- name: Download Go modules (retry transient proxy resets)
	if: steps.select.outputs.run == 'true'
	uses: ./.github/actions/go-mod-download-retry

	- name: Install gotestsum
	if: steps.select.outputs.run == 'true'
	run: \|
	echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
	go install gotest.tools/gotestsum@v1.13.0
	"$(go env GOPATH)/bin/gotestsum" --version

	# Warm compile cache (#7480 follow-up): the warm-build-cache job on every
	# v4 push compiles the whole tree (race + norace) and saves GOCACHE keyed
	# by SHA; PR jobs restore the nearest ancestor via the prefix restore-key,
	# turning the ~2-minute per-job -race recompile into an incremental one.
	# Restore-only in PR jobs: balance/miss costs seconds, never correctness.
	- name: Restore warm Go build cache
	if: ${{ steps.select.outputs.run == 'true' && (!contains(vars.HIVE_RUNNER_LABELS, 'hive-runners')) }}
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: ~/.cache/go-build
	key: hive-go-build-${{ runner.os }}-${{ runner.arch }}-${{ github.sha }}
	restore-keys: \|
	hive-go-build-${{ runner.os }}-${{ runner.arch }}-

	- name: Restore cgo toolchain package cache
	if: steps.select.outputs.run == 'true'
	id: restore-cgo-apt-cache
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2
	restore-keys: \|
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-
	hive-cgo-race-apt-

	# Bounded + annotated by .github/scripts/ci-install-tool.sh (#6648/#6870/#6935):
	# skip apt when the tool already exists, install restored .debs offline
	# when possible, retry short egress flakes with backoff, and fail loudly
	# with a ::error:: if neither cache nor mirrors can provide the tool.
	- name: Prepare cgo toolchain for race tests
	if: steps.select.outputs.run == 'true'
	id: prepare-cgo-toolchain
	env:
	HIVE_CI_APT_CACHE_DIR: ${{ github.workspace }}/.ci/apt-cache/cgo-race
	HIVE_CI_MISSING_TOOLCHAIN_SOFT_FAIL: ${{ !inputs.integration_candidate && (github.event_name == 'schedule' \|\| github.event_name == 'workflow_dispatch') && '1' \|\| '0' }}
	run: \|
	bash "$GITHUB_WORKSPACE/.github/scripts/ci-install-tool.sh" \
	--require gcc --for '-race tests' \
	--apt 'gcc libc6-dev' --apk 'gcc musl-dev' --verify 'gcc --version'

	- name: Save cgo toolchain package cache
	if: ${{ steps.select.outputs.run == 'true' && (always() && steps.restore-cgo-apt-cache.outputs.cache-hit != 'true' && steps.prepare-cgo-toolchain.outputs.apt-cache-populated == 'true') }}
	continue-on-error: true
	uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2

	- name: Test pkg/dashboard in shuffled order
	if: ${{ steps.select.outputs.run == 'true' && (steps.prepare-cgo-toolchain.outputs.toolchain-ready == 'true') }}
	env:
	IDX: ${{ matrix.idx }}
	TOTAL: ${{ strategy.job-total }}
	run: \|
	set -o pipefail
	# Sliced like test-hub (NR mod N — pairwise disjoint, jointly
	# exhaustive) so no single job carries the whole ~6-minute lane;
	# -shuffle still varies order within each slice every run.
	ROOT="$PWD"
	export TEST_PACKAGE=github.com/hivecommons/hive/pkg/dashboard
	export TEST_DIR="$ROOT/pkg/dashboard"
	export TEST_BINARY="$ROOT/dashboard.test"
	go test -c -race -o "$TEST_BINARY" ./pkg/dashboard
	(cd "$TEST_DIR" && "$TEST_BINARY" -test.short -test.list '.*') \| grep -E '^(Test\|Example\|Fuzz)' \| LC_ALL=C sort > all-tests.txt
	if [ ! -s all-tests.txt ]; then
	echo "::error::go test -list produced no test functions for pkg/dashboard — refusing to run an empty slice (fail closed)"
	exit 1
	fi
	awk -v n="$TOTAL" -v i="$IDX" 'NR % n == i - 1' all-tests.txt > slice.txt
	echo "slice $IDX/$TOTAL: $(wc -l < slice.txt) of $(wc -l < all-tests.txt) test functions"
	if [ ! -s slice.txt ]; then
	echo "::error::empty dashboard slice — failing closed"
	exit 1
	fi
	export CHUNK_DIR
	CHUNK_DIR=$(mktemp -d "$ROOT/test-chunks.XXXXXX")
	split -l 1000 slice.txt "$CHUNK_DIR/chunk-"
	# raw-command preserves one JUnit/flake report across all chunks.
	# gotestsum appends -test.run and package arguments on reruns.
	cat > "$CHUNK_DIR/run.sh" <<'RUNNER'
	#!/usr/bin/env bash
	set -uo pipefail
	export LC_ALL=C # ${#RUN} must count bytes, not Unicode characters.
	cd "$TEST_DIR" \|\| exit 1
	run_chunk() {
	local RUN="$1"
	# MAX_ARG_STRLEN is 131072 bytes; fail clearly before execve.
	if [ "${#RUN}" -gt 120000 ]; then
	echo "::error::chunk regex exceeds 120000 bytes; reduce chunk size" >&2
	return 1
	fi
	# Bound t.Parallel to avoid exhausting httptest ephemeral ports.
	go tool test2json -t -p "$TEST_PACKAGE" "$TEST_BINARY" \
	-test.short -test.count=1 -test.v=test2json -test.timeout=10m \
	-test.shuffle=on -test.parallel=4 "$RUN"
	}
	if [ "$#" -gt 0 ]; then
	run_chunk "$1"
	exit $?
	fi
	status=0
	for chunk in "$CHUNK_DIR"/chunk-*; do
	RUN="-test.run=^($(paste -sd'\|' "$chunk"))$"
	run_chunk "$RUN" \|\| status=1
	done
	exit "$status"
	RUNNER
	if gotestsum --format=standard-verbose --rerun-fails=2 --rerun-fails-max-failures=5 \
	--rerun-fails-report=flaky-tests-dashboard-shuffle-"$IDX".txt \
	--junitfile junit-dashboard-shuffle-"$IDX".xml --raw-command -- bash "$CHUNK_DIR/run.sh" \
	2>&1 \| grep --line-buffered -vE '^=== (RUN\|PAUSE\|CONT) ' \| tee test-dashboard-shuffle-"$IDX".log; then
	exit 0
	fi
	# Each chunk prints its own -test.shuffle seed immediately before
	# its tests; do not report the first chunk's seed for a later failure.
	echo "::error::pkg/dashboard failed (slice $IDX/$TOTAL); see the failing chunk's -test.shuffle seed in test-dashboard-shuffle-$IDX.log. Reproduce the failing test with: go test ./pkg/dashboard -short -race -count=1 -p 1 -parallel 4 -run '^TestName$' -shuffle=<seed>"
	exit 1

	- name: Surface flaky reruns
	id: flaky-reruns
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: bash "$GITHUB_WORKSPACE/.github/scripts/ci-gotestsum-flake-summary.sh" "flaky-tests-dashboard-shuffle-${{ matrix.idx }}.txt" "dashboard shuffle ${{ matrix.idx }}/3"

	- name: Upload flaky rerun report
	if: ${{ steps.select.outputs.run == 'true' && (always() && steps.flaky-reruns.outputs.has-flakes == 'true') }}
	uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	with:
	name: flaky-tests-dashboard-shuffle-${{ matrix.idx }}
	path: src/flaky-tests-dashboard-shuffle-${{ matrix.idx }}.txt
	if-no-files-found: error
	retention-days: 14

	- name: Slowest tests in this lane
	if: ${{ steps.select.outputs.run == 'true' && (always()) }}
	run: bash "$GITHUB_WORKSPACE/.github/scripts/slowest-tests.sh" "dashboard shuffle ${{ matrix.idx }}/3" test-dashboard-shuffle-${{ matrix.idx }}.log

	- name: Upload shuffle log
	if: ${{ steps.select.outputs.run == 'true' && (failure()) }}
	uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	with:
	name: test-dashboard-shuffle-log-${{ matrix.idx }}
	path: src/test-dashboard-shuffle-${{ matrix.idx }}.log
	if-no-files-found: warn
	retention-days: 1

	# Stable aggregate context named "test", preserved from the pre-shard job so
	# anything keyed on the check name (branch protection, tide, humans' muscle
	# memory) keeps working. Fails closed for genuine failures, skipped shards,
	# and unknown states. A cancelled-only result is reported distinctly as an
	# infrastructure cancellation instead of being counted as a test failure: no
	# tests ran in that shard, so it is not evidence of a code defect, and the
	# summary says it is not complete green test evidence.
	test:
	# release-gate/* is created only by tagged-release.yml and carries only CHANGELOG.md/changelog.d changes; report the required context immediately (#10863).
	if: ${{ always() && !cancelled() && !(github.event_name == 'pull_request' && startsWith(github.head_ref, 'release-gate/')) }}
	# Do not run this aggregate in concurrency-cancelled twin runs (#6293);
	# the surviving run still fail-closes on any non-success shard result.
	# Required on v5: keep on the self-hosted pool so a GitHub-hosted runner
	# outage does not make otherwise-mergeable PRs impossible to merge.
	runs-on: ${{ !inputs.integration_candidate && (github.event_name != 'pull_request' \|\| github.event.pull_request.head.repo.full_name == github.repository) && fromJSON(vars.HIVE_RUNNER_LABELS \|\| '["ubuntu-latest"]') \|\| 'ubuntu-latest' }}
	timeout-minutes: 10
	needs: [required-context-scope, list-packages, test-hub, test-sliced, test-agent, test-rest, test-dashboard-shuffle]
	steps:
	- name: Check shard results
	env:
	LIST_PACKAGES_RESULT: ${{ needs.list-packages.result }}
	TEST_HUB_RESULT: ${{ needs.test-hub.result }}
	TEST_SLICED_RESULT: ${{ needs.test-sliced.result }}
	TEST_AGENT_RESULT: ${{ needs.test-agent.result }}
	TEST_REST_RESULT: ${{ needs.test-rest.result }}
	TEST_DASHBOARD_SHUFFLE_RESULT: ${{ needs.test-dashboard-shuffle.result }}
	RUN_TESTS: ${{ needs.required-context-scope.outputs.run-tests }}
	SCOPE_RESULT: ${{ needs.required-context-scope.result }}
	SELECTION: ${{ needs.list-packages.outputs.selection }}
	CRUX_SCOPE: ${{ needs.list-packages.outputs.crux-scope }}
	RUN_HUB: ${{ needs.list-packages.outputs.run-hub }}
	RUN_AGENT: ${{ needs.list-packages.outputs.run-agent }}
	RUN_DASHBOARD: ${{ needs.list-packages.outputs.run-dashboard }}
	RUN_SCHEDULER: ${{ needs.list-packages.outputs.run-scheduler }}
	run: \|
	set -euo pipefail

	if [ "${SCOPE_RESULT:-}" != success ]; then
	echo "::error::required-context-scope failed; refusing to report test success without a diff verdict"
	exit 1
	fi

	if [ "${RUN_TESTS:-}" != true ]; then
	echo "No v2 test paths changed; reporting required test context success without running shards."
	exit 0
	fi

	# Lanes with nothing selected still ran (and succeeded) — see the
	# Affected-package selection step in each shard — so the fail-closed
	# result check below is unchanged by selection.
	echo "Go test selection: ${SELECTION:-<missing>} (partial = affected packages only on this PR; push and scheduled runs are always full)"
	# Crux boundary filter (#13536): the lanes allowed to report success
	# without testing are exactly those whose run flag list-packages set
	# to false. Print that allowlist here so the verdict is auditable
	# from the aggregate log; a lane absent from it that did not test
	# would have failed its own selection step logic, and any shard
	# result other than success still fails this job below.
	if [ "${SELECTION:-}" = partial ]; then
	allow=()
	[ "${RUN_HUB:-true}" = false ] && allow+=(test-hub)
	[ "${RUN_AGENT:-true}" = false ] && allow+=(test-agent)
	[ "${RUN_DASHBOARD:-true}" = false ] && allow+=(test-dashboard-shuffle "test-sliced (dashboard)")
	[ "${RUN_SCHEDULER:-true}" = false ] && allow+=("test-sliced (scheduler)")
	echo "Crux scope: ${CRUX_SCOPE:-<missing>}; lanes allowed to succeed without testing: ${allow[*]:-<none>}"
	fi

	shard_names=(
	list-packages
	test-hub
	test-sliced
	test-agent
	test-rest
	test-dashboard-shuffle
	)
	shard_results=(
	"${LIST_PACKAGES_RESULT:-}"
	"${TEST_HUB_RESULT:-}"
	"${TEST_SLICED_RESULT:-}"
	"${TEST_AGENT_RESULT:-}"
	"${TEST_REST_RESULT:-}"
	"${TEST_DASHBOARD_SHUFFLE_RESULT:-}"
	)

	failures=()
	skipped=()
	cancelled=()
	unknown=()

	for i in "${!shard_names[@]}"; do
	name=${shard_names[$i]}
	result=${shard_results[$i]}
	printf '%-23s %s\n' "${name}:" "${result:-<missing>}"
	case "$result" in
	success) ;;
	failure) failures+=("$name") ;;
	skipped) skipped+=("$name") ;;
	cancelled) cancelled+=("$name") ;;
	*) unknown+=("$name=${result:-<missing>}") ;;
	esac
	done

	join_by_comma() {
	local IFS=', '
	echo "$*"
	}

	append_summary() {
	if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
	{
	echo "## v2 shard aggregate"
	echo ""
	echo "$1"
	} >> "$GITHUB_STEP_SUMMARY"
	fi
	}

	if [ "${#failures[@]}" -gt 0 ] \|\| [ "${#skipped[@]}" -gt 0 ] \|\| [ "${#unknown[@]}" -gt 0 ]; then
	if [ "${#failures[@]}" -gt 0 ]; then
	echo "::error::Failing shard(s): $(join_by_comma "${failures[@]}")"
	fi
	if [ "${#skipped[@]}" -gt 0 ]; then
	echo "::error::Skipped shard(s): $(join_by_comma "${skipped[@]}"). A skipped shard means tests may not have run."
	fi
	if [ "${#unknown[@]}" -gt 0 ]; then
	echo "::error::Unknown shard result(s): $(join_by_comma "${unknown[@]}")"
	fi
	if [ "${#cancelled[@]}" -gt 0 ]; then
	echo "Cancelled shard(s) were also present: $(join_by_comma "${cancelled[@]}"). They are not treated as infrastructure-only because another shard failed, skipped, or reported an unknown state."
	fi
	append_summary "❌ At least one shard failed, skipped, or reported an unknown state. Cancelled shards, if any, were not allowed to mask that result."
	exit 1
	fi

	if [ "${#cancelled[@]}" -gt 0 ]; then
	message="Infrastructure cancellation: shard(s) did not complete: $(join_by_comma "${cancelled[@]}"). No test failure was observed, but this run is not complete test evidence for those shard(s)."
	echo "::warning title=Infrastructure cancellation::$message"
	append_summary "⚠️ $message"
	exit 0
	fi

	append_summary "✅ All shards succeeded."
	echo "All shards succeeded."

	# Per-package coverage-floor gate (#4112): parse-only, consumes the shard
	# artifacts — it runs no tests, so PRs run the suite exactly once. PR-only:
	# on the hourly schedule the coverage-hourly workflow remains the
	# monitoring + issue-filing authority.
	#
	# Scoring is computed from the coverage PROFILES, not from the `ok ...
	# coverage: X%` log lines: the agent slices each emit a PARTIAL percentage
	# for crux/agent, so per-block counts from ALL shard profiles are merged
	# (summed) first and every package is scored on the union. For unsplit
	# packages this yields the identical number `go test` prints. The logs are
	# still parsed for the two states a profile cannot express: FAIL and
	# `? ... [no test files]` — both hard failures, as before.
	coverage:
	if: ${{ always() && (inputs.integration_candidate \|\| github.event_name == 'merge_group' \|\| (github.event_name == 'pull_request' && !startsWith(github.head_ref, 'release-gate/'))) }}
	# Required on v5: keep on the self-hosted pool so a GitHub-hosted runner
	# outage does not make otherwise-mergeable PRs impossible to merge.
	runs-on: ${{ !inputs.integration_candidate && (github.event_name != 'pull_request' \|\| github.event.pull_request.head.repo.full_name == github.repository) && fromJSON(vars.HIVE_RUNNER_LABELS \|\| '["ubuntu-latest"]') \|\| 'ubuntu-latest' }}
	timeout-minutes: 30
	needs: [required-context-scope, list-packages, test-hub, test-sliced, test-agent, test-rest]
	steps:
	- name: Report coverage not required for this diff
	if: ${{ needs.required-context-scope.result == 'success' && needs.required-context-scope.outputs.run-tests != 'true' }}
	run: echo "No v2 coverage paths changed; reporting required coverage context success."

	- name: Fail if scope failed or a required test shard did not complete
	if: ${{ needs.required-context-scope.result != 'success' \|\| (needs.required-context-scope.outputs.run-tests == 'true' && (needs.test-hub.result != 'success' \|\| needs.test-sliced.result != 'success' \|\| needs.test-agent.result != 'success' \|\| needs.test-rest.result != 'success')) }}
	run: \|
	if [ "${{ needs.required-context-scope.result }}" != success ]; then
	echo "::error::required-context-scope failed; refusing to report coverage success without a diff verdict"
	exit 1
	fi
	echo "::error::Coverage requires successful test shards (hub=${{ needs.test-hub.result }}, sliced=${{ needs.test-sliced.result }}, agent=${{ needs.test-agent.result }}, rest=${{ needs.test-rest.result }})"
	exit 1

	# Affected-package selection (#12454). On a partial PR run only the
	# selected packages produced profiles, so only they are scored — each
	# against its unchanged floor below. There is no repo-wide total floor
	# to weaken, and this job never writes a floor or baseline (floors are
	# code, ratcheted by hand; coverage-hourly.yml scores the full suite).
	# An empty selection (no Go package affected) has nothing to score.
	- name: Decide what coverage to score
	id: cov-scope
	if: ${{ needs.required-context-scope.outputs.run-tests == 'true' }}
	env:
	SELECTION: ${{ needs.list-packages.outputs.selection }}
	SELECTED: ${{ needs.list-packages.outputs.selected }}
	run: \|
	if [ "$SELECTION" = partial ] && [ -z "${SELECTED// /}" ]; then
	echo "Affected-package selection chose no Go package; no coverage to score. Push and scheduled runs score the full suite."
	echo "score=false" >> "$GITHUB_OUTPUT"
	else
	echo "Coverage scoring: ${SELECTION:-<missing>} selection."
	echo "score=true" >> "$GITHUB_OUTPUT"
	fi

	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	if: ${{ steps.cov-scope.outputs.score == 'true' }}
	with:
	persist-credentials: ${{ !inputs.integration_candidate }}

	# Populate the shared node-local Go tool cache under the runner pods'
	# node-wide lock before setup-go, so parallel jobs on a cold cache (a
	# src/go.mod bump) stop clobbering each other's rm -rf + copy (#11175,
	# #10234). Never fails the job.
	- name: Warm shared Go tool cache (serialize the cold-cache population)
	if: ${{ steps.cov-scope.outputs.score == 'true' }}
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-toolcache-guard.sh"
	- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
	if: ${{ steps.cov-scope.outputs.score == 'true' }}
	with:
	# This job only lists packages and scores uploaded profiles. Saving
	# the runners' shared multi-GB module/build caches adds a serial
	# post-step to the required coverage verdict with no compile reuse.
	cache: false
	go-version-file: src/go.mod
	# Fill GOMODCACHE up front with bounded retries so a transient
	# proxy.golang.org stream reset (INTERNAL_ERROR, #14976) cannot fail the
	# job's first module-fetching step; see .github/actions/go-mod-download-retry.
	- name: Download Go modules (retry transient proxy resets)
	if: ${{ steps.cov-scope.outputs.score == 'true' }}
	uses: ./.github/actions/go-mod-download-retry

	- name: List packages present in checkout
	if: ${{ steps.cov-scope.outputs.score == 'true' }}
	working-directory: src
	env:
	SELECTION: ${{ needs.list-packages.outputs.selection }}
	run: \|
	# The coverage scorer consumes shard artifacts, but the package set it
	# scores must still be bounded by the checked-out tree for this run.
	# This mirrors coverage-hourly.yml, whose `go test` command can only
	# emit packages that exist in the checkout.
	go list ./pkg/... ./cmd/... ./internal/... ./crux/... \| LC_ALL=C sort -u > ../existing-pkgs.txt
	if [ ! -s ../existing-pkgs.txt ]; then
	echo "::error::go list produced no packages for coverage scoring"
	exit 1
	fi
	# On a partial selection the shard logs only cover the selected
	# packages, so take the no-test-files gate from the whole tree
	# instead (cheap: go list only). Same definition as the `?` /
	# `coverage: 0.0%` lines a full run classifies below.
	: > ../no-test-pkgs.txt
	if [ "$SELECTION" = partial ]; then
	go list -f '{{if not (or .TestGoFiles .XTestGoFiles)}}{{.ImportPath}}{{end}}' ./pkg/... ./cmd/... ./crux/... \
	\| { grep . \|\| true; } \| LC_ALL=C sort -u > ../no-test-pkgs.txt
	fi

	- name: Download shard coverage artifacts
	if: ${{ steps.cov-scope.outputs.score == 'true' }}
	uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
	with:
	pattern: test-shard-*
	path: shards
	merge-multiple: true

	- name: Check per-package coverage
	if: ${{ steps.cov-scope.outputs.score == 'true' }}
	working-directory: shards
	env:
	SELECTION: ${{ needs.list-packages.outputs.selection }}
	SELECTED: ${{ needs.list-packages.outputs.selected }}
	run: \|
	THRESHOLD=90
	FAILED=""
	BROKEN=""
	REPORT=""

	# Per-package threshold overrides. A handful of packages are heavily
	# coupled to the live runtime (fixed /data,/proc,/var/run paths, real
	# github.com/platform.claude.com device-flow, a live tmux agent buffer,
	# an in-cluster k8s serviceaccount) and cannot be exercised hermetically
	# in CI without invasive production seams. They get a documented lower
	# floor so the gate still catches regressions below their real ceiling,
	# rather than blocking forever on untestable code. Raise these toward 90
	# as seams are added; do not add new entries without justification here.
	#
	# KEEP IN SYNC with coverage-hourly.yml, which applies the same floors
	# on its hourly scheduled run.
	declare -A PKG_THRESHOLD=(
	[dashboard]=84 # 86.2% on 2026-09-01; the residue is the inception tmux watcher + copilot/claude device-flow + /data file I/O + k8s
	[hub]=87 # kubectl-shelling + live-GitHub paths; see #2355. Was 89 against v2's 89.9%; v4 measures 87.6%.
	[crux/agent]=89 # pre-existing shortfall inherited from #2350, NOT a new regression. See #2355. Keyed crux/<pkg> since the module cut (#13536)

	# ── Floors recorded when the gate moved from v2 to v4 (#3903) ─────
	# A RATCHET, NOT A TARGET — see coverage-hourly.yml for the full
	# rationale. Raise these toward 90 as tests land; never raise one to
	# paper over a drop.
	#
	# ── Ratchet turned for #4570 ─────────────────────────────────────
	# sandbox/ioscan/pushbroker: tests added in the same PR, so the
	# floors move with the gain (67.6→98.5, 71.0→97.1, 72.8→100.0).
	# intent/tracing/dashboard: NO code change — these had drifted far
	# above floors recorded at d89121f5 and were no longer catching
	# anything. Re-measured on v4 at cd8aee5d and moved up to what
	# they actually hold, with ~1 point of headroom.
	# scheduler (90.4) and escalation (93.2) now clear the 90% default
	# and no longer need an entry at all.
	[intent]=87
	[tracing]=85
	[review]=80
	[retro]=82
	[github]=88 # 88.9% on 2026-10-05
	[hivectl]=88
	[knowledge]=89
	[crux/proxy]=89
	[ioscan]=96
	[crux/sandbox]=98
	[crux/pushbroker]=99
	[crux/kubejob]=90 # new in #6311; sandbox.Launcher over the in-cluster API, tested against httptest

	# ── Stale module-path parser repaired on v5 (#6066) ─────────────
	# These packages were pre-existing below-default floors that became
	# visible again when the v2-tests coverage classifier was corrected
	# from the old github.com/kubestellar/hive module path to
	# github.com/hivecommons/hive. Record their current ceilings so the
	# parser fix can gate new packages (pkg/effects, pkg/hub/spoke)
	# without turning unrelated backlog into a blocker.
	[convergence/mutation]=86 # 86.7% on PR #6076
	[convergence/audit]=92 # measured 95.6% on v5 after #9930 (in-package Run/reconcile/publish tests, #9928); was 6 (seed tests only, #9688)
	[dashboard/webstatic]=88 # 88.9% on PR #6076
	[github/automerge]=88 # 88.6% on PR #6076
	[mint]=89 # 89.1% on PR #6076
	[turn]=87 # 87.1% on PR #6076

	# ── Ratchet turned for #7146 ─────────────────────────────────────
	# spoke was unfloored and flapped across the 90 default (run
	# 35026215085 at 89.6%, green runs either side, no spoke change
	# between). #7130 covered the timing-dependent paths rather than
	# lowering the floor; spoke now holds a flat 96.3% with zero variance
	# over five runs. See coverage-hourly.yml for the full rationale.
	[spoke]=95 # 96.3% on 2026-09-16 after #7130; was unfloored/flapping

	# ── ./cmd/... floors (#4701) ─────────────────────────────────────
	# See coverage-hourly.yml for the rationale. cmd keys are prefixed
	# ("cmd/<name>") so cmd/hivectl can never collide with
	# pkg/hivectl's floor. Recorded just below what v4 measures after
	# #4700/#4719 so the ratchet can turn as CLI tests land.
	# ── Ratchet turned 2026-09-01 ────────────────────────────────────
	# cmd/bd, cmd/hive, cmd/hive-backup, dashboard, github: NO code
	# change — each had drifted 4–20 points above its recorded floor
	# (cmd/hive most of all: 18.8 → 37.5 as eval-cycle helpers grew
	# tests) and the gate was no longer catching a drop. Re-measured
	# on the 2026-09-01 hourly run and moved to ~2–4 points under
	# what they hold. cmd/apiproxy is unchanged at 13.9%.
	# ── Ratchet turned for #7407 ────────────────────────────────────
	# NO code change — re-measured on 2026-09-17 with `go test -cover
	# ./cmd/... -short -count=1` (three runs, zero variance) and moved
	# to ~2-3 points under the lower of the linux/darwin measurements.
	# See coverage-hourly.yml for the full rationale and both numbers.
	[cmd/bd]=92 # 95.7% on 2026-09-17; was 75
	[cmd/hive]=42 # 44.8% on 2026-09-17; bulk is main()/run-loop wiring; was 35
	[cmd/apiproxy]=92 # 95.6% on 2026-09-17; was 13
	[cmd/hive-backup]=85 # 90.4% on 2026-09-17; was 78
	[cmd/hivectl]=97 # 100.0% on 2026-09-17; was exempt as an untested main
	[cmd/flue-fixture]=75 # 80.0% on 2026-09-22 (#8361): main() is the single uncovered statement; run() and signalContext() are tested in-process and the binary is smoke-tested as a child process, which yields no profile
	[cmd/crux]=75 # 80.0% on 2026-10-23 (#13536): same shape as flue-fixture; main() is the one uncovered statement, run() is tested in-process. floor_key maps BOTH ./cmd/crux (root host) and ./crux/cmd/crux (the module's own binary, #14138) here; the module one measured 80.0% on 2026-10-25
	[crux/cli]=92 # 97.8% on 2026-10-23 (#13536): the crux CLI against a fake crux.Runtime plus a real unix control socket; the residue is Main() and write-error branches

	# ── Crux module (#13536, Stage 2 step 4) ─────────────────────────
	# Packages of the nested module at src/crux are keyed "crux/<path>"
	# (floor_key below): crux/agent, crux/proxy, crux/sandbox,
	# crux/pushbroker, crux/kubejob and crux/cli above carried their
	# pkg/<name> floors across the move unchanged. The module root
	# package is keyed "crux". Every other crux/<pkg> takes the
	# default.
	)

	# Thin flag-parsing mains with no test files, wrapping packages the
	# gate already floors. KEEP IN SYNC with coverage-hourly.yml. Any NEW
	# untested cmd package still hard-fails.
	#
	# Empty since #7407: cmd/hivectl, the only entry, gained tests in
	# a3f9842c, so no "[no test files]" line is emitted for it and the
	# exemption could never match. The mechanism itself is unchanged.
	UNTESTED_CMD_MAINS=''

	# Merge every shard profile: per-block counts are summed, so the N
	# partial crux/agent profiles combine into the package's true
	# coverage. Blocks are keyed by (location, statement count); package
	# = the directory of the block's file. Emits "pkg pct" lines plus a
	# TOTAL over ./pkg/... — the same statements-covered/statements
	# definition `go test -cover` prints, so floors are unchanged.
	#
	# Packages are selected by their "/pkg/" path segment rather than by
	# a hard-coded module prefix. The previous version matched the literal
	# module path, so the kubestellar -> hivecommons org rename silently
	# emptied every filter and disabled the whole gate. A segment match
	# survives any future rename, and only this repository's own packages
	# appear in these profiles, so it cannot over-match. The nested Crux
	# module's packages carry a "/crux" segment (github.com/<org>/hive/crux
	# and .../crux/<pkg>) and get a second, informational total.
	awk '
	FNR == 1 { next } # skip each file'\''s "mode:" line
	{
	file = $1; sub(/:[^:]*$/, "", file)
	pkg = file; sub(/\/[^\/]+$/, "", pkg)
	key = pkg "\|" $1 "\|" $2
	stmts[key] = $2; owner[key] = pkg
	count[key] += $3
	}
	END {
	for (k in count) {
	p = owner[k]
	tot[p] += stmts[k]
	if (count[k] > 0) cov[p] += stmts[k]
	}
	gt = 0; gc = 0; xt = 0; xc = 0
	for (p in tot) {
	printf "%s %.1f\n", p, cov[p] * 100 / tot[p]
	if (p ~ /\/pkg\//) {
	gt += tot[p]; gc += cov[p]
	} else if (p ~ /\/crux(\/\|$)/ && p !~ /\/cmd\//) {
	xt += tot[p]; xc += cov[p]
	}
	}
	if (gt > 0) printf "TOTAL %.1f\n", gc * 100 / gt
	if (xt > 0) printf "CRUX_TOTAL %.1f\n", xc * 100 / xt
	}
	' coverage-*.out > pkg-pcts.txt

	TOTAL=$(awk '$1 == "TOTAL" {print $2 "%"}' pkg-pcts.txt)
	CRUX_TOTAL=$(awk '$1 == "CRUX_TOTAL" {print $2 "%"}' pkg-pcts.txt)
	echo "Total coverage (./pkg/...): ${TOTAL:-unavailable}"
	echo "Total coverage (./crux/...): ${CRUX_TOTAL:-unavailable}"

	# Partial selection (#12454): every number here covers the selected
	# packages only. Per-package floors still apply unchanged to each of
	# them; the total is informational and not comparable to the
	# repo-wide figure, so it is labelled rather than compared.
	PARTIAL=false
	SELECTED_SCORED=""
	if [ "$SELECTION" = partial ]; then
	PARTIAL=true
	for p in $SELECTED; do
	case "$p" in /pkg/\|/cmd/\|/crux\|/crux/) ;; ) continue ;; esac
	grep -Fxq "$p" ../existing-pkgs.txt && SELECTED_SCORED="$SELECTED_SCORED $p"
	done
	echo "Partial selection: scoring only the $(wc -w <<<"$SELECTED_SCORED") selected ./pkg/..., ./cmd/... and ./crux/... packages; the totals above are over the selected packages and are not comparable to the repo-wide totals."
	if [ -z "$TOTAL" ] && ! grep -q '/pkg/' <<<"$SELECTED_SCORED"; then
	TOTAL="n/a (no ./pkg/... package selected)"
	fi
	fi

	# Fail closed. Every step below is a filter, and an empty filter
	# result looks exactly like "nothing was wrong". This gate silently
	# scored zero packages for the whole window in which it still matched
	# the pre-rename module path github.com/kubestellar/hive, while the
	# module had become github.com/hivecommons/hive: TOTAL was
	# "unavailable", pkg-states.txt was empty, the scoring loop never ran
	# once, and the job reported success on every PR. A gate that cannot
	# see anything must fail, not pass.
	if [ -z "$TOTAL" ]; then
	echo "COVERAGE GATE FAILED: no coverage data parsed from the shard artifacts."
	echo "coverage-*.out was present but produced no ./pkg/... totals."
	exit 1
	fi
	# Same guard for the Crux module on a full run: its shards (the
	# agent slices and the rest buckets holding ./crux/...) must have
	# produced profiles, or the module's floors are vacuously met.
	if [ "$SELECTION" != partial ] && [ -z "$CRUX_TOTAL" ]; then
	echo "COVERAGE GATE FAILED: no coverage data parsed for the Crux module (./crux/...)."
	exit 1
	fi

	# Classify every package exactly once from the combined logs. With
	# gotestsum reruns, a package can have an initial FAIL line followed
	# by a final ok line in the same successful shard; score the final
	# successful rerun rather than treating the surfaced flake as a
	# coverage-blocking broken package.
	# ./cmd/... is scored too (#4701): the shards have run cmd tests
	# since the partition change, but coverage floors previously applied
	# to ./pkg/... only, so a CLI package could lose all its tests
	# without the gate noticing. The TOTAL line remains ./pkg/...-only
	# for trend continuity.
	cat test-*.log > cover-run.log
	awk '
	NR == FNR {
	exists[$1] = 1
	next
	}
	$1 == "ok" \|\| $1 == "FAIL" \|\| $1 == "?" {
	if (!exists[$2]) next
	if ($2 ~ /\/(pkg\|cmd)\// \|\| $2 ~ /\/crux(\/\|$)/) {
	seen[$2] = 1
	if ($1 == "ok") ok[$2] = 1
	else if ($1 == "FAIL") fail[$2] = 1
	else if ($1 == "?") no_tests[$2] = 1
	}
	next
	}
	/^\t.*\t\tcoverage: 0\.0% of statements$/ {
	if (exists[$1] && ($1 ~ /\/(pkg\|cmd)\// \|\| $1 ~ /\/crux(\/\|$)/)) {
	seen[$1] = 1
	no_tests[$1] = 1
	}
	}
	END {
	for (pkg in seen) {
	if (ok[pkg]) print "ok", pkg
	else if (fail[pkg]) print "FAIL", pkg
	else if (no_tests[pkg]) print "?", pkg
	}
	}
	' ../existing-pkgs.txt cover-run.log \
	\| LC_ALL=C sort -u > pkg-states.txt

	# Second fail-closed guard, for the same reason as the TOTAL one:
	# an empty pkg-states.txt means the scoring loop never executes and
	# every floor is vacuously satisfied.
	if [ ! -s pkg-states.txt ] && { [ "$PARTIAL" != true ] \|\| [ -n "$SELECTED_SCORED" ]; }; then
	echo "COVERAGE GATE FAILED: no package pass/fail states parsed from test-*.log."
	echo "No package could be scored against its floor."
	exit 1
	fi

	# Partial selection: every selected package must have been run and
	# classified — a selected package missing here was never tested.
	# The no-test-files gate is taken from the whole tree (see "List
	# packages present in checkout") so an unselected untested package
	# still fails exactly as it would on a full run.
	if [ "$PARTIAL" = true ]; then
	MISSING=""
	for p in $SELECTED_SCORED; do
	awk -v p="$p" '$2 == p { f = 1 } END { exit !f }' pkg-states.txt \|\| MISSING="$MISSING $p"
	done
	if [ -n "$MISSING" ]; then
	echo "COVERAGE GATE FAILED: selected package(s) produced no test result:$MISSING"
	exit 1
	fi
	while read -r p; do
	[ -n "$p" ] \|\| continue
	awk -v p="$p" '$2 == p { f = 1 } END { exit !f }' pkg-states.txt \|\| echo "? $p" >> pkg-states.txt
	done < ../no-test-pkgs.txt
	fi

	# Floor-map key: cmd packages get a "cmd/" prefix so cmd/hivectl can
	# never resolve to pkg/hivectl's floor; pkg packages are keyed by
	# their full path below pkg/ so a nested package can never alias the
	# top-level one sharing its name (v5 has both pkg/spoke and
	# pkg/hub/spoke); the nested Crux module's packages are keyed
	# "crux/<path>" (the module root package is "crux") so crux/config
	# can never alias pkg/config. See coverage-hourly.yml for the full
	# rationale.
	floor_key() {
	case "$1" in
	/cmd/) echo "cmd/${1##*/}" ;;
	/pkg/) echo "${1##*/pkg/}" ;;
	/crux/) echo "crux/${1##*/crux/}" ;;
	*/crux) echo "crux" ;;
	) echo "${1##/}" ;;
	esac
	}

	while read -r state pkgpath; do
	pkg=$(floor_key "$pkgpath")
	case "$state" in
	ok)
	# Package states are collapsed above, so an ok means either
	# the package passed immediately or gotestsum reran its flakes
	# to a successful final result.
	pct=$(awk -v p="$pkgpath" '$1 == p {print $2}' pkg-pcts.txt)
	if [ -z "$pct" ]; then continue; fi
	int_pct=${pct%.*}
	pkg_threshold=${PKG_THRESHOLD[$pkg]:-$THRESHOLD}
	if [ "$int_pct" -lt "$pkg_threshold" ]; then
	FAILED="${FAILED}${pkg}: ${pct}% (floor ${pkg_threshold}%)\n"
	fi
	REPORT="${REPORT}${pkg}: ${pct}%\n"
	;;
	FAIL)
	BROKEN="${BROKEN}${pkg}: TESTS FAILING\n"
	REPORT="${REPORT}${pkg}: FAIL\n"
	;;
	\?)
	case " $UNTESTED_CMD_MAINS " in
	" $pkg ")
	REPORT="${REPORT}${pkg}: no tests (exempt thin main)\n"
	;;
	*)
	BROKEN="${BROKEN}${pkg}: NO TEST FILES\n"
	REPORT="${REPORT}${pkg}: no tests\n"
	;;
	esac
	;;
	esac
	done < pkg-states.txt

	{
	echo "## Per-package coverage"
	echo ""
	echo "Total (./pkg/...): ${TOTAL:-unavailable}"
	echo "Total (./crux/...): ${CRUX_TOTAL:-unavailable}"
	echo ""
	echo -e "${REPORT}"
	} >> "$GITHUB_STEP_SUMMARY"

	if [ -n "$BROKEN" ] \|\| [ -n "$FAILED" ]; then
	{
	[ -n "$BROKEN" ] && echo -e "### Packages that cannot be scored\n${BROKEN}"
	[ -n "$FAILED" ] && echo -e "### Packages below their floor\n${FAILED}"
	} >> "$GITHUB_STEP_SUMMARY"
	echo ""
	echo "COVERAGE GATE FAILED:"
	[ -n "$BROKEN" ] && echo -e "${BROKEN}"
	[ -n "$FAILED" ] && echo -e "${FAILED}"
	exit 1
	fi
	echo "All packages meet their coverage floor" >> "$GITHUB_STEP_SUMMARY"

	patch-coverage:
	if: ${{ !inputs.integration_candidate && github.event_name == 'pull_request' && needs.coverage.result == 'success' && needs.required-context-scope.outputs.run-tests == 'true' && !cancelled() }}
	needs: [coverage, required-context-scope]
	runs-on: ubuntu-latest
	timeout-minutes: 10
	permissions:
	contents: read
	issues: write
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	persist-credentials: false
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	ref: ${{ github.event.pull_request.base.sha }}
	path: .coverage-tools
	persist-credentials: false
	- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
	with:
	pattern: test-shard-*
	path: shards
	merge-multiple: true
	- name: Patch coverage on changed Go lines (advisory, trusted base tool)
	continue-on-error: true
	env:
	BASE_REF: ${{ github.base_ref }}
	PR_NUMBER: ${{ github.event.pull_request.number }}
	GH_TOKEN: ${{ github.token }}
	run: \|
	git fetch --no-tags --depth=1 origin "$BASE_REF" \|\| true
	if [ -d shards ]; then
	awk 'FNR == 1 { next } { key=$1 " " $2; hits[key]+=$3 }
	END { print "mode: atomic"; for (key in hits) print key, hits[key] }' \
	shards/coverage-*.out > shards/coverage-merged.out 2>/dev/null \|\| true
	fi
	bash .coverage-tools/.github/scripts/patch-coverage.sh \
	--base "origin/$BASE_REF" \
	--profile shards/coverage-merged.out \
	--min-percent 80 --min-uncovered 5 \
	--pr "$PR_NUMBER" \
	--advisory
	create-issue-on-failure:
	if: "!inputs.integration_candidate && (failure() \|\| cancelled()) && github.event_name == 'schedule'"
	runs-on: ${{ (github.event_name != 'pull_request' \|\| github.event.pull_request.head.repo.full_name == github.repository) && fromJSON(vars.HIVE_RUNNER_LABELS \|\| '["ubuntu-latest"]') \|\| 'ubuntu-latest' }}
	timeout-minutes: 10
	needs: [list-packages, test-hub, test-sliced, test-agent, test-rest, test-dashboard-shuffle]
	permissions:
	contents: read
	issues: write
	steps:
	- name: Create issue for test failure
	uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
	with:
	script: \|
	const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
	const sha = context.sha.substring(0, 7);
	// The scheduled run executes on the DEFAULT branch (v4) — name the
	// real branch instead of the workflow's historical "v2" label,
	// which sent operators to the retired v2 line (#4569).
	const branch = context.ref.replace('refs/heads/', '');
	const results = [
	'${{ needs.test-hub.result }}',
	'${{ needs.test-sliced.result }}',
	'${{ needs.test-agent.result }}',
	'${{ needs.test-rest.result }}',
	'${{ needs.test-dashboard-shuffle.result }}',
	'${{ needs.list-packages.result }}',
	];
	const wasCancelled = results.includes('cancelled');
	const kind = wasCancelled ? 'cancelled (likely timed out)' : 'failure';
	const title = `[Tests] ${branch} test ${wasCancelled ? 'cancelled' : 'failure'} on ${sha}`;

	const existing = await github.rest.issues.listForRepo({
	owner: context.repo.owner,
	repo: context.repo.repo,
	state: 'open',
	labels: 'ci-failure',
	per_page: 10,
	});
	const dup = existing.data.find(i => i.title.startsWith(`[Tests] ${branch} test`) \|\| i.title.startsWith('[Tests] v2 test'));
	if (dup) {
	await github.rest.issues.createComment({
	owner: context.repo.owner,
	repo: context.repo.repo,
	issue_number: dup.number,
	body: `Another ${kind} on \`${sha}\`: ${runUrl}`,
	});
	return;
	}

	await github.rest.issues.create({
	owner: context.repo.owner,
	repo: context.repo.repo,
	title,
	body: `## Test ${wasCancelled ? 'Cancelled' : 'Failure'}\n\n- Commit: \`${context.sha}\`\n- Run: ${runUrl}\n- Branch: ${branch}\n\nScheduled test run was ${kind}. See the [workflow run](${runUrl}) for details.`,
	labels: ['ci-failure', 'kind/bug'],
	});

	create-issues-on-flakes:
	if: "!inputs.integration_candidate && always() && github.event_name == 'push' && (github.ref == 'refs/heads/v5' \|\| github.ref == 'refs/heads/v6') && needs.test-hub.result == 'success' && needs.test-sliced.result == 'success' && needs.test-agent.result == 'success' && needs.test-rest.result == 'success' && needs.test-dashboard-shuffle.result == 'success'"
	runs-on: ${{ fromJSON(vars.HIVE_RUNNER_LABELS \|\| '["ubuntu-latest"]') }}
	timeout-minutes: 10
	needs: [test-hub, test-sliced, test-agent, test-rest, test-dashboard-shuffle]
	permissions:
	actions: read
	contents: read
	issues: write
	steps:
	- name: Download flaky rerun reports
	continue-on-error: true
	uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
	with:
	pattern: flaky-tests-*
	path: flaky-reports
	merge-multiple: false

	- name: Detect flaky rerun reports
	id: flake-reports
	run: \|
	if [ -d flaky-reports ] && find flaky-reports -type f -size +0c \| grep -q .; then
	echo "has-flakes=true" >> "$GITHUB_OUTPUT"
	else
	echo "has-flakes=false" >> "$GITHUB_OUTPUT"
	fi

	- name: Create or update flake issues
	if: steps.flake-reports.outputs.has-flakes == 'true'
	uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
	with:
	script: \|
	const fs = require('fs');
	const path = require('path');
	// Self-hosted runners lack the gh CLI (#8991); ensure the label via the API instead.
	const LABEL_ALREADY_EXISTS = 422;
	try {
	await github.rest.issues.createLabel({
	owner: context.repo.owner,
	repo: context.repo.repo,
	name: 'ci-flake',
	color: 'F9D0C4',
	description: 'Go test passed only after gotestsum reran a failed test',
	});
	} catch (e) {
	if (e.status !== LABEL_ALREADY_EXISTS) throw e;
	}

	const root = 'flaky-reports';
	if (!fs.existsSync(root)) {
	core.info('No flaky rerun report artifacts were downloaded.');
	return;
	}

	const reportFiles = [];
	for (const artifactDir of fs.readdirSync(root)) {
	const fullDir = path.join(root, artifactDir);
	if (!fs.statSync(fullDir).isDirectory()) continue;
	for (const entry of fs.readdirSync(fullDir)) {
	const fullPath = path.join(fullDir, entry);
	if (fs.statSync(fullPath).isFile()) reportFiles.push(fullPath);
	}
	}
	if (reportFiles.length === 0) {
	core.info('No flaky rerun report files were downloaded.');
	return;
	}

	const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
	const branch = context.ref.replace('refs/heads/', '');
	const seen = new Map();
	for (const file of reportFiles) {
	for (const rawLine of fs.readFileSync(file, 'utf8').split(/\r?\n/)) {
	const line = rawLine.trim();
	if (!line) continue;
	const subject = line.replace(/:\s+\d+\s+runs?,\s+\d+\s+failures?\s*$/, '');
	const dot = subject.lastIndexOf('.');
	const pkg = dot > -1 ? subject.slice(0, dot) : 'package unknown';
	const testName = dot > -1 ? subject.slice(dot + 1) : subject;
	const key = `${pkg}\t${testName}`;
	if (!seen.has(key)) seen.set(key, {pkg, testName, lines: []});
	seen.get(key).lines.push(line);
	}
	}

	if (seen.size === 0) {
	core.info('Downloaded reports were empty.');
	return;
	}

	const openFlakeIssues = await github.paginate(github.rest.issues.listForRepo, {
	owner: context.repo.owner,
	repo: context.repo.repo,
	state: 'open',
	labels: 'ci-flake',
	per_page: 100,
	});
	const tallyCommand = `bash src/scripts/ci-flake-tally.sh --repo ${context.repo.owner}/${context.repo.repo} --workflow v2-tests.yml --branch ${branch} -n 40`;
	const twentyFourHoursAgo = Date.now() - 24 * 60 * 60 * 1000;

	for (const flake of seen.values()) {
	const issueKey = `${flake.pkg}.${flake.testName}`;
	const title = `ci-flake: ${issueKey}`;
	const details = flake.lines.map(line => `- \`${line}\``).join('\n');
	const existing = openFlakeIssues.find(issue => issue.title === title);
	if (!existing) {
	await github.rest.issues.create({
	owner: context.repo.owner,
	repo: context.repo.repo,
	title,
	labels: ['ci-flake'],
	body: `## CI flake detected\n\n`
	+ `- Test: \`${flake.testName}\`\n`
	+ `- Package: \`${flake.pkg}\`\n`
	+ `- Run: ${runUrl}\n`
	+ `- Branch: \`${branch}\`\n\n`
	+ `### Rerun report\n\n${details}\n\n`
	+ `### Tally recent failures\n\n\`\`\`sh\n${tallyCommand}\n\`\`\`\n\n`
	+ `A gotestsum rerun is not a pass. Please fix the flaky test.`,
	});
	continue;
	}

	const comments = await github.paginate(github.rest.issues.listComments, {
	owner: context.repo.owner,
	repo: context.repo.repo,
	issue_number: existing.number,
	per_page: 100,
	});
	const lastComment = comments[comments.length - 1];
	if (lastComment && new Date(lastComment.created_at).getTime() > twentyFourHoursAgo) {
	core.info(`Skipping comment on #${existing.number}; last comment is less than 24h old.`);
	continue;
	}

	await github.rest.issues.createComment({
	owner: context.repo.owner,
	repo: context.repo.repo,
	issue_number: existing.number,
	body: `Flaked again on \`${branch}\`: ${runUrl}\n\n${details}\n\nTally with:\n\n\`\`\`sh\n${tallyCommand}\n\`\`\``,
	});
	}

	# Compiles the whole tree's test binaries (race) plus the plain build once
	# per v4 push and saves GOCACHE keyed by SHA. Every PR shard restores the
	# nearest ancestor's cache via the prefix restore-key above, so shard jobs
	# compile incrementally (seconds) instead of rebuilding ~2 minutes of -race
	# objects each. Push/schedule only: PR-scoped cache saves are invisible to
	# other branches and would only thrash the repo's 10GB cache budget.
	warm-build-cache:
	if: "!inputs.integration_candidate && github.event_name != 'pull_request' && !contains(vars.HIVE_RUNNER_LABELS, 'hive-runners')"
	runs-on: ${{ fromJSON(vars.HIVE_RUNNER_LABELS \|\| '["ubuntu-latest"]') }}
	timeout-minutes: 40
	defaults:
	run:
	working-directory: src
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

	# Populate the shared node-local Go tool cache under the runner pods'
	# node-wide lock before setup-go, so parallel jobs on a cold cache (a
	# src/go.mod bump) stop clobbering each other's rm -rf + copy (#11175,
	# #10234). Never fails the job.
	- name: Warm shared Go tool cache (serialize the cold-cache population)
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-toolcache-guard.sh"
	- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
	with:
	cache: false
	go-version-file: src/go.mod
	- name: Guard shared Go cache (fall back when the runner volume is full)
	run: bash "$GITHUB_WORKSPACE/.github/scripts/go-cache-guard.sh"
	# Fill GOMODCACHE up front with bounded retries so a transient
	# proxy.golang.org stream reset (INTERNAL_ERROR, #14976) cannot fail the
	# job's first module-fetching step; see .github/actions/go-mod-download-retry.
	- name: Download Go modules (retry transient proxy resets)
	uses: ./.github/actions/go-mod-download-retry

	- name: Restore warm Go build cache
	if: ${{ !contains(vars.HIVE_RUNNER_LABELS, 'hive-runners') }}
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: ~/.cache/go-build
	key: hive-go-build-${{ runner.os }}-${{ runner.arch }}-${{ github.sha }}
	restore-keys: \|
	hive-go-build-${{ runner.os }}-${{ runner.arch }}-

	- name: Restore cgo toolchain package cache
	id: restore-cgo-apt-cache
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2
	restore-keys: \|
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-
	hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-
	hive-cgo-race-apt-

	- name: Prepare cgo toolchain for race tests
	id: prepare-cgo-toolchain
	env:
	HIVE_CI_APT_CACHE_DIR: ${{ github.workspace }}/.ci/apt-cache/cgo-race
	HIVE_CI_MISSING_TOOLCHAIN_SOFT_FAIL: ${{ !inputs.integration_candidate && (github.event_name == 'schedule' \|\| github.event_name == 'workflow_dispatch') && '1' \|\| '0' }}
	run: \|
	bash "$GITHUB_WORKSPACE/.github/scripts/ci-install-tool.sh" \
	--require gcc --for '-race tests' \
	--apt 'gcc libc6-dev' --apk 'gcc musl-dev' --verify 'gcc --version'

	- name: Save cgo toolchain package cache
	if: always() && steps.restore-cgo-apt-cache.outputs.cache-hit != 'true' && steps.prepare-cgo-toolchain.outputs.apt-cache-populated == 'true'
	continue-on-error: true
	uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: .ci/apt-cache/cgo-race
	key: hive-cgo-race-apt-${{ runner.os }}-${{ runner.arch }}-gcc-libc6-dev-v2

	# -run '^$' compiles every package's test binary without running a test;
	# the plain build covers the non-race objects build-and-test needs.
	- name: Compile tree into the build cache
	if: steps.prepare-cgo-toolchain.outputs.toolchain-ready == 'true'
	run: \|
	go build ./... ./crux/...
	go test -race -count=1 -run '^$' ./pkg/... ./cmd/... ./internal/... ./crux/... > /dev/null

	- name: Save warm Go build cache
	continue-on-error: true
	uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: ~/.cache/go-build
	key: hive-go-build-${{ runner.os }}-${{ runner.arch }}-${{ github.sha }}

Navigation Menu

Repository navigation

🐛 fix(crux): /v1/health reports tmux and returns 503 when it is missing #24799

Workflow file

🐛 fix(crux): /v1/health reports tmux and returns 503 when it is missing #24799

Uh oh!

Workflow file for this run