Skip to content
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
8607271
chore(protection): retire the last exemption — the allow-list is now …
hyperpolymath Oct 24, 2026
37c45a4
docs: record the decision-layer fix everywhere a reader or agent woul…
hyperpolymath Oct 24, 2026
d9a78dd
fix(protection): the bypass removal missed three rulesets, one of the…
hyperpolymath Oct 24, 2026
da9073d
docs(ci): correct audit claims and clarify context exemption scope
coderabbitai[bot] Oct 24, 2026
8ed2f3a
Merge remote audit updates and preserve context review corrections
coderabbitai[bot] Oct 24, 2026
620060f
fix(protection): a failed read-back must not report success
hyperpolymath Oct 24, 2026
d4b3d28
fix(gate): a data-dependent rule blocks every PR and was invisible
hyperpolymath Oct 24, 2026
553bbd7
docs(gate): the failure message names both shapes of finding
hyperpolymath Oct 24, 2026
0aca273
docs(audit): the data-dependent rules are not repairable in-repo
hyperpolymath Oct 24, 2026
d49f8f7
fix(gate): a data-rule-only finding must fail strict mode
hyperpolymath Oct 24, 2026
88699f4
Update scripts/fix-required-context-name.sh
hyperpolymath Oct 25, 2026
0857173
docs(gate): clarify data-rule inspection and audit behavior
coderabbitai[bot] Oct 25, 2026
dc8caa8
fix(ci): Harden required-context audits and ruleset repair verification
coderabbitai[bot] Oct 25, 2026
477ff71
Merge remote branch updates and preserve required-context review fixes
coderabbitai[bot] Oct 25, 2026
97cd2e4
feat(ci): measure and publish real line coverage
hyperpolymath Oct 25, 2026
583101e
fix(ci): name the coverage context by slug, and publish the number
hyperpolymath Oct 25, 2026
57059c6
feat(ci): report the line counts, not just the percentage
hyperpolymath Oct 25, 2026
48fa3c9
docs(audit): correct the claim that coverage is unrepairable, with th…
hyperpolymath Oct 25, 2026
ffe1094
fix(gate): stop claiming nothing uploads while a workflow does
hyperpolymath Oct 25, 2026
faf2a1b
fix(ci): measure the commit the upload is attributed to
hyperpolymath Oct 25, 2026
64ada82
feat(gate): tell an unverified rule from an unsatisfiable one
hyperpolymath Oct 25, 2026
609deeb
Update scripts/check-required-contexts.sh
hyperpolymath Oct 25, 2026
0f433ca
Update scripts/check-required-contexts.sh
hyperpolymath Oct 25, 2026
98ebde4
fix(ci): Verify Code Quality evidence across all workflow pages
coderabbitai[bot] Oct 25, 2026
04acc6b
Merge remote head and preserve required-context evidence hardening
coderabbitai[bot] Oct 25, 2026
83afb5d
Update scripts/check-required-contexts.sh
hyperpolymath Oct 25, 2026
107d4c7
fix(ci): Match the correct GitHub Code Quality workflow path
coderabbitai[bot] Oct 25, 2026
c3e9fad
Merge commit '83afb5d2efb9b11d2b51d8d2706e2f21f34d52ca' into arena/94…
coderabbitai[bot] Oct 25, 2026
ddc6180
test(ci): Correct Code Quality workflow paths in required-context fix…
coderabbitai[bot] Oct 25, 2026
1d9ef56
Merge commit 'c3e9fad20d74964013bb91a29547a3bc60aa3579' into arena/94…
coderabbitai[bot] Oct 25, 2026
3dfb5fd
fix(gate): evidence only where a rule applies, and never a guessed br…
hyperpolymath Oct 25, 2026
95847d1
fix(gate): stop trusting a green coverage run that uploaded nothing
hyperpolymath Oct 25, 2026
d8d08ba
fix(ci): recognize exact canonical origins after standards owner rename
coderabbitai[bot] Oct 25, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/actions.lock
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,10 @@ workflows:
- 'haskell-actions/setup@0f8e8c99d88aeb3fbfd523f1ef2c6f762d10d64d'
- 'hyperpolymath/standards@d55304a01b5260b0511aae0d14ed8b8770b00293'
- 'oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6'
'.github/workflows/code-coverage.yml':
- 'actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1'
- 'actions/upload-code-coverage@2b21a77928be8d5168c2b9581a67f2adbebacc52'
- 'dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8'
'.github/workflows/codeql-reusable.yml':
- 'actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1'
- 'github/codeql-action@b96794f015dfd88f77b49b1c93e0fa7110f94c63'
Expand Down Expand Up @@ -235,6 +239,11 @@ dependencies:
repo_id: 496012378
uses:
- 'actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f'
'actions/upload-code-coverage@2b21a77928be8d5168c2b9581a67f2adbebacc52':
ref: 'v1.4.4'
commit: 'sha1-2b21a77928be8d5168c2b9581a67f2adbebacc52'
owner_id: 44036562
repo_id: 1220171870
'denoland/setup-deno@22d081ff2d3a40755e97629de92e3bcbfa7cf2ed':
ref: 'v2.0.5'
commit: 'sha1-22d081ff2d3a40755e97629de92e3bcbfa7cf2ed'
Expand Down
210 changes: 210 additions & 0 deletions .github/workflows/code-coverage.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,210 @@
# SPDX-License-Identifier: MPL-2.0
# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
#
# code-coverage.yml — publish real line-coverage data to GitHub Code Quality.
#
# WHY THIS EXISTS
# ---------------
# The main-gate ruleset carries a `code_coverage` rule (minimum line coverage).
# GitHub evaluates that rule against **uploaded** Cobertura reports; it runs
# nothing itself and it does not wait for an upload to arrive. A coverage rule
# with no uploader therefore blocks every pull request forever while looking,
# in the check list, like a rule rather than a fault. That is the same defect as
# a required status context no pull request can publish; see
# docs/audits/audit-ci-context-producibility-2026-09-20.adoc, which records five
# instances of it in a single day.
#
# The alternative — delete the rule — was offered and declined. So the number
# has to be real: this workflow measures what is measurable here and uploads it.
#
# WHAT IS MEASURED, AND WHAT IS NOT
# ---------------------------------
# Measured: line coverage of the six independent Rust crates in this tree
# (`cargo llvm-cov`, Cobertura output).
#
# NOT measured: the 245 shell scripts under scripts/ and the 756 AsciiDoc files
# that are the bulk of this repository. Shell coverage needs kcov, is materially
# more fragile, and belongs in its own change with its own review. Until that
# lands, the floor set in the ruleset describes the Rust crates only. Saying so
# is the point: a coverage number that quietly excludes most of the tree is
# worse than no number, because it reads as a guarantee.
#
# WHY push AND pull_request
# -------------------------
# Code Quality compares pull-request coverage against the **default branch**
# baseline. Without a `push` trigger on main there is no baseline, and the rule
# has nothing to compare against. Both triggers are load-bearing.
#
# WHY NO `paths:` FILTER
# ----------------------
# A path-filtered workflow never reports on pull requests that miss the filter.
# This job must be a required status check (see below), so a filter would leave
# those pull requests waiting for a check that never starts. That is deadlock,
# not optimisation — the same reasoning as lock-sync-gate.yml.
#
# WHY THE CHECK MUST BE REQUIRED
# ------------------------------
# The coverage rule evaluates "data that has already been uploaded". Merge
# before this job finishes and the rule can evaluate an empty set and pass.
# Requiring the job is what makes the rule mean anything, and it is also what
# makes a fork pull request — whose upload the action deliberately skips — wait
# for the same check as everything else instead of slipping through.
#
# ONE JOB PER CRATE, ONE LABEL PER UPLOAD
# ---------------------------------------
# GitHub aggregates the uploaded reports for a branch, so six labels are six
# contributions to one number rather than six competing numbers. A matrix keeps
# each crate's build isolated: these are independent crates with no workspace
# root, and one broken build must not mask another crate's coverage.
#
# SCOPE OF THE MATRIX
# -------------------
# Six crates live outside `9-archive/`. Eight more Cargo manifests live *inside*
# it. Archive contents are excluded on purpose: measuring code that is retained
# for the historical record puts a coverage floor on work nobody is meant to
# change. The exclusion is stated rather than silent, because the aggregate
# describes these six crates and nothing else.
#
# FIRST RUN, RECORDED (2026-10-25)
# --------------------------------
# All six legs measured successfully. All six uploads then failed with
# `Coverage upload failed (HTTP 404): Not Found` — the Code Quality ingestion
# endpoint does not exist until Code Quality is enabled for the repository, and
# that is a settings change, not a file. The job stayed red rather than
# reporting a passing upload that never happened, which is the behaviour this
# file intends. `fail-on-error` must therefore stay at its default until the
# setting is on; turning it off here would only hide the 404.
#
# The measured per-crate figures are recorded in
# docs/audits/audit-ci-context-producibility-2026-09-20.adoc under "Correction,
# 2026-10-25", including the aggregate the ruleset's floor has to sit at or
# below. They are kept there rather than duplicated here so the two cannot drift
# apart — this file states the mechanism, that document states the numbers.

name: Code coverage

on:
push:
branches: [main, master]
pull_request:
# Deliberately no workflow_dispatch: the upload action needs a pull request
# number for any non-default branch and skips uploads it cannot attribute, so
# a manual run would appear to work and publish nothing.

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

permissions:
contents: read

jobs:
coverage:
# The matrix suffix is generated by GitHub from the raw `include` values, so
# without this the required check would be named
# `Rust coverage (<crate path>, <slug>)` — the path baked into a context name
# that must stay stable for as long as the ruleset requires it. Naming the
# slug explicitly keeps the published context readable and short.
name: Rust coverage (${{ matrix.name }})
runs-on: ubuntu-24.04
timeout-minutes: 30
permissions:
contents: read
# Required to upload coverage data. Granted at job scope rather than
# workflow scope; nothing else here needs it.
code-quality: write
strategy:
fail-fast: false
matrix:
# `name` is a stable slug used in the upload label. Deriving the label
# from `strategy.job-index` would renumber every report whenever an
# entry is inserted, which silently invalidates history comparisons.
include:
- crate: .verisimdb/ecosystem-ingest
name: ecosystem-ingest
- crate: 1-formats/sub-specs/inline-annotations/extractor
name: inline-annotations-extractor
- crate: 2-protocols/k9-coordination/tools/k9-init
name: k9-init
- crate: hooks/playbook-to-recipe
name: playbook-to-recipe
- crate: tools/js-runtimes
name: js-runtimes
- crate: tools/uuid-v8
name: uuid-v8
steps:
- name: Check out
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
Comment thread
hyperpolymath marked this conversation as resolved.
# Measure the tree the upload is attributed to. On a `pull_request`
# event the default checkout is GitHub's synthetic merge commit, but
# the upload action attributes the report to
# `github.event.pull_request.head.sha`. When the head is behind the
# base, the two trees differ and base-branch changes land in coverage
# totals recorded against a commit that does not contain them. On
# `push` there is no pull request object, so this falls back to
# `github.sha` — the commit being pushed.
ref: ${{ github.event.pull_request.head.sha || github.sha }}

- name: Install Rust toolchain
# `toolchain:` is mandatory under a SHA pin.
uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # stable
with:
toolchain: stable
components: llvm-tools-preview

- name: Install cargo-llvm-cov
# Version-pinned: `cargo install` without a version silently picks up
# whatever crates.io serves that day, and a coverage tool that changes
# under you changes the number the ruleset is measuring.
run: cargo install cargo-llvm-cov --version 0.9.1 --locked

- name: Measure line coverage
working-directory: ${{ matrix.crate }}
run: |
set -euo pipefail
# Deliberately not `--locked`: four of these six crates do not commit a
# Cargo.lock, so `--locked` fails there rather than measuring them.
cargo llvm-cov --cobertura --output-path cobertura.xml
# Read the totals Cobertura puts on the root element. The rule compares
# a weighted aggregate across every uploaded report, so the raw counts
# matter as much as the ratio: a floor can only be chosen from data
# that shows how much of the aggregate each crate actually carries.
read -r PCT COVERED VALID <<< "$(awk -F'"' '/<coverage / {
for (i = 1; i < NF; i++) {
if ($i ~ /lines-valid=$/) valid = $(i + 1)
if ($i ~ /lines-covered=$/) covered = $(i + 1)
}
} END {
if (valid + 0 == 0) { print "0.00 0 0"; exit }
printf "%.2f %d %d", (covered + 0) * 100 / (valid + 0), covered + 0, valid + 0
}' cobertura.xml)"
{
echo "### ${{ matrix.name }}: ${PCT}%"
echo ""
echo "\`${COVERED}\` of \`${VALID}\` instrumented lines covered."
} >> "$GITHUB_STEP_SUMMARY"
# Also emit the number as an annotation. The step summary is only in the
# run's own page; an annotation shows up on the pull request and is
# readable through the API, which is what makes the figure auditable
# after the fact rather than "trust the green tick".
echo "::notice title=Line coverage: ${{ matrix.name }}::${PCT}% of instrumented lines are covered by tests (${COVERED}/${VALID})."
# A crate that instruments nothing would otherwise report a cheerful
# 0.00% or, worse, divide by zero. Both mean "this measured nothing",
# which is a different statement from "this is untested".
if [ "$VALID" -eq 0 ]; then
echo "::warning title=No instrumented lines: ${{ matrix.name }}::cargo llvm-cov reported no instrumented lines; this crate contributes nothing to the aggregate."
fi

- name: Upload coverage report
# Fork pull requests cannot receive the write permission this needs; the
# action detects that and skips rather than failing. `fail-on-error`
# stays at its default (true), so a genuine upload failure fails the
# job. A coverage job that goes green without uploading is the exact
# defect this file exists to prevent.
uses: actions/upload-code-coverage@2b21a77928be8d5168c2b9581a67f2adbebacc52 # v1.4.4
with:
file: ${{ matrix.crate }}/cobertura.xml
language: Rust
label: code-coverage/rust-${{ matrix.name }}
12 changes: 8 additions & 4 deletions .github/workflows/required-contexts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,11 @@ jobs:
fi
# Findings = every required context that cannot be satisfied on a pull
# request. `app-owned` and `producible` are not findings.
jq -r '[.results[] | select(.verdict == "PR-UNPRODUCIBLE" or .verdict == "UNSATISFIABLE") | .context] | .[]' \
# Three verdicts are findings, and the third is easy to forget: a
# data-dependent RULE (`code_coverage`, `code_quality`) never appears
# as a status context, so it cannot be caught by looking at contexts,
# yet it blocks every pull request exactly like an unproducible one.
jq -r '[.results[] | select(.verdict == "PR-UNPRODUCIBLE" or .verdict == "UNSATISFIABLE" or .verdict == "UNSATISFIED-RULE") | .context] | .[]' \
"$JSON" > "$RUNNER_TEMP/findings.txt"
awk '{ sub(/#.*/, ""); gsub(/^[ \t]+|[ \t]+$/, ""); if ($0 != "") print }' \
"$ALLOW" > "$RUNNER_TEMP/allow.txt"
Expand Down Expand Up @@ -146,9 +150,9 @@ jobs:
echo "::warning::$n_stale declared exemption(s) no longer match anything — remove them from $ALLOW"
fi
if [ "$n_new" -gt 0 ]; then
echo "::error::$n_new required context(s) cannot be published on a pull request and are not declared in $ALLOW"
echo "::error::$n_new requirement(s) cannot be satisfied on a pull request and are not declared in $ALLOW — see the findings above; a \`rule:\` entry is a rule whose data nothing uploads, a bare name is a status context nothing publishes"
sed 's/^/ /' "$RUNNER_TEMP/undeclared.txt"
echo "Every pull request waits for a check run that cannot happen. Either publish it on pull_request, require the name the PR-time workflow actually emits, or declare it in $ALLOW with a reason."
echo "For a status context: publish it on pull_request, or require the name the PR-time workflow actually emits. For a rule: upload the data it needs, or remove the rule — a coverage threshold on a repository that publishes no coverage blocks every pull request forever, and an undecidable gate is one people bypass. Declaring it in $ALLOW is the last resort, not the repair."
exit 1
fi
echo "All unsatisfiable required contexts are declared in $ALLOW ($n_find declared)."
echo "All unsatisfiable requirements are declared in $ALLOW ($n_find declared)."
78 changes: 50 additions & 28 deletions .machine_readable/required-contexts-allow.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@
#
# Every line is a debt with a name and a reason: `context`
#
# THE LIST IS EMPTY, AND THAT IS THE POINT
# ----------------------------------------
# This file's preamble has always said that removing the last entry is the
# goal and not an edge case. The one entry has now been removed, because the
# defect it recorded was repaired at the decision layer rather than declared
# here. Empty is the correct resting state; if an entry comes back, it is a
# regression with a review trail, not a configuration.
#
# WHY THIS FILE EXISTS INSTEAD OF `continue-on-error: true`
# ---------------------------------------------------------
# The alternative was an always-green advisory job, and this repository has
Expand All @@ -15,33 +23,47 @@
# was never wired, so it never disagreed with anything.)
#
# So the job FAILS on any unsatisfiable required context that is not listed
# here, and passes only for the ones below, which are reported on every run.
# A new entry cannot be added quietly: this file is the review surface.
#
# Removing the last entry is the point, not an edge case.
#
# The known entry is a branch RULE, not a missing workflow. The scan it names
# does run on every pull request — as `Hypatia verdict grid / Hypatia
# Neurosymbolic Analysis`, the fork-governance lane. The required context names
# the OTHER call site of the same reusable workflow: hypatia-scan.yml, whose
# `scan` job is deliberately push/schedule-only so the same findings are not
# uploaded to code scanning twice per pull request. So the defect is a context
# name pointing at the post-merge call site of a scan that is gated pre-merge
# under a different name.
#
# Consequence, measured on #1336: 65 checks green, reviewDecision clear,
# mergeable, and still BLOCKED — because the required check can never appear.
# Every pull request in this repository is in that state, and the only way work
# lands is the ruleset's `always` bypass, which discards the other nineteen
# required checks in the same action. That is the whole failure mode this
# repository keeps re-learning: the bypass is not an escape hatch used
# occasionally, it is the actual merge path, and it makes the nineteen
# decorative.
#
# Repair (decision layer, needs repository admin — the ruleset, not this file):
# replace the context below with `Hypatia verdict grid / Hypatia Neurosymbolic
# Analysis` in ruleset 23787415, leaving the other nineteen untouched.
# here. A new entry cannot be added quietly: this file is the review surface,
# and the gate also warns when a listed context stops appearing as a finding, so
# this file cannot quietly rot into a museum of old exemptions either.
#
Comment thread
coderabbitai[bot] marked this conversation as resolved.
# RECORD OF THE RETIRED ENTRY
# ---------------------------
# Context: `scan / Hypatia Neurosymbolic Analysis`
# Retired: 2026-10-24
#
# It named the second call site of hypatia-scan-reusable.yml. `hypatia-scan.yml`
# declares that job as push/schedule-only on purpose, so the same findings are
# not uploaded to code scanning twice per pull request; the pull-request call
# site is the fork-governance lane, which publishes
# `Hypatia verdict grid / Hypatia Neurosymbolic Analysis`. A required check is
# matched by name, and the names differ, so the required one could never be
# produced while a pull request was open.
#
# Measured consequence: #1336 reached 65 passing checks, a cleared
# reviewDecision and mergeable=MERGEABLE while mergeStateStatus stayed BLOCKED.
# #1337, which carried the repair, merged 79 minutes BEFORE the ruleset was
# corrected — so the fix provably did not enable its own merge. #1337's merge
# used the ruleset's `always` bypass, which discards every required check in the
# same action. This record does not establish that #1336 merged or used the bypass.
#
# Retired by the decision layer, as this file asked:
# ruleset 23787415, required_status_checks:
# scan / Hypatia Neurosymbolic Analysis
# -> Hypatia verdict grid / Hypatia Neurosymbolic Analysis
# bypass_actors: always on RepositoryRole:5 -> removed
# strict_required_status_checks_policy: false -> true
# + `Repo self-tests` required (21 contexts, all producible on a pull request)
#
# Doing it in this file instead would have required continuing to bypass, and a
# list of declared exceptions is only safer than a bypass while it stays short.
#
# Format: one context per line; `#` starts a comment; blank lines ignored.

scan / Hypatia Neurosymbolic Analysis
#
# An empty file (or one holding only comments) asserts that no repository-owned
# `PR-UNPRODUCIBLE` or `UNSATISFIABLE` finding is undeclared. It does
# NOT assert that every required context is satisfiable: the reconciliation step
# excludes `app-owned` contexts (those produced by a GitHub App other than
# Actions), and it only judges contexts that are findings. Both exclusions are
# deliberate — but they mean empty is a narrower claim than "all good", and
# this file should not be read as one.
Loading
Loading