[quality] Gate proposal: Go branches covered but never asserted (advisory mutation testing on changed packages)

Finding

The quality lane keeps finding the same class of defect in this repository's Go code: a branch or data-flow that existing tests execute (so it counts as covered) but whose effect no test asserts. Deleting the line leaves the suite green. The accepted patch-coverage gate (#14161) cannot see it — the lines are covered — and the per-package floors in v2-tests.yml / coverage-hourly.yml cannot either.

Prior instances of this class filed by this lane in hivecommons/hive:

🐛 fix(knowledge): FullFact only takes a vault body from the fact's own layer #16939 — src/pkg/knowledge/api.go:958-960 mergeFullFact Layer hunk: both Full/Layer arms ran under existing tests, neither outcome asserted.
🐛 fix(escalation): a red PR triaged as infrastructure is re-run once instead of sent back to fixers until it escalates #17002 (review) — src/pkg/relay/ledger two arms reached but unasserted; commented on the PR rather than filed.
🌱 test(crux/serve): pin request span naming and first-status-wins recording #17241 — src/crux/serve/metrics.go span.SetName(r.URL.Path) and attributes set on every request, never read back by any test.
Today (not filed per this rule) — src/crux/agent/watchdog_fleet.go:157,172 + markCLILaunchedLocked: the watchdog: PaneShowsLogin pages on Copilot CLI relaunch login flash #17320 relaunch boot grace depends on Observe copying lastCLILaunchAt into Observation.LastLaunchAt; TestWatchdogObserveCarriesStartedAt executes both lines, nothing asserts the value. Test PR opened separately (references this issue).

Three or more sites with one root cause means the gate is missing, not that another per-site issue is needed.

Recommendation

Add an advisory mutation-testing job to CI, scoped to packages changed by the PR, so "covered but unasserted" surfaces as a surviving mutant in the PR instead of a quality finding weeks later.

Decision taken by hive: gremlins (go-gremlins/gremlins) as the tool — it is Go-native, supports --diff-style package targeting, has a JSON report, and installs with go install (pinnable). Advisory first, mirroring the #14161 phasing:

Phase 1 (advisory): a mutation job in the Go PR workflow runs gremlins unleash against the packages the PR diff touches with a wall-clock budget (--timeout-coefficient, hard job timeout 20m) and posts/edits one  PR comment listing surviving mutants in changed files (_test.go, generated, testdata excluded). Never blocks.
Phase 2 (two weeks later, separate PR): fail the job when the efficacy for changed packages drops below a floor agreed from Phase 1 data, with a mutation/waived label as the override that must name a reason in the PR body.
src/crux gets the same job via the existing crux-sync CI mirror once Phase 1 is stable.

The workflow file lives under .github/workflows/, which this agent's tier cannot push; a human or an ISSUES_PRS_MERGE agent needs to land Phase 1. The lane will stop filing individual "covered but unasserted" findings in this repository while this proposal is open, and resumes only if it is declined.

Risk: governance

Priority

Impact: medium (the class has produced a live-incident fix shipped half-pinned at least once: watchdog: PaneShowsLogin pages on Copilot CLI relaunch login flash #17320's manager half)
Effort: medium (one workflow job + comment script; tool is go install-able)

Filed by quality agent (ACMM L4/L6 — full mode)

🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

What to reply

/hive approve: go with the recommendation
/hive decision <your instructions>: go ahead, following your instructions
/hive help: show these again

If what this issue waited on has been done (a merged fix, or an action taken outside this repository), a maintainer can reply /fixed to close it as completed; Hive clears its park labels.

Only a maintainer with write access can un-park; everything else is ignored. Full command reference: https://github.com/hivecommons/hive/blob/v5/src/docs/maintainer-commands.md.

task-list sweep: 0 of 3 items ticked. Not closing yet — outstanding boxes remain.

Outstanding items:

🔲 Phase 1 (advisory): a mutation job in the Go PR workflow runs gremlins unleash against the packages the PR diff touches with a wall-clock budget (--timeout-coefficient, hard job timeout 20m) and posts/edits one  PR comment listing surviving mutants in changed files (_test.go, generated, testdata excluded). Never blocks.
🔲 Phase 2 (two weeks later, separate PR): fail the job when the efficacy for changed packages drops below a floor agreed from Phase 1 data, with a mutation/waived label as the override that must name a reason in the PR body.
🔲 src/crux gets the same job via the existing crux-sync CI mirror once Phase 1 is stable.

Not yet touched: v2-tests.yml, coverage-hourly.yml, src/pkg/knowledge/api.go, src/crux/serve/metrics.go, _test.go

Merged PRs referencing this issue so far:

🌱 test(crux/agent): pin LastLaunchAt wiring from manager to watchdog observation #17350 — 🌱 test(crux/agent): pin LastLaunchAt wiring from manager to watchdog observation
🌱 ci: advisory gremlins mutation job for changed Go packages #17416 — 🌱 ci: advisory gremlins mutation job for changed Go packages

This comment is edited in place by the task-list sweep on every cycle; it is not duplicated.

PR #17350 has merged (#17350) and referenced this issue with a non-closing Refs #17349 rather than a closing keyword.

This issue remains open because the merged PR did not use Closes / Fixes / Resolves. If the fix should close it on merge, use a closing keyword in the PR body; use the opt-in hive: needs-confirmation marker only when the reporter genuinely must verify before closure.

/hive decision Approved for Phase 1 only (advisory, gremlins, changed packages, 20m hard timeout, never blocks). Phase 2 (blocking on an efficacy floor) is not approved yet - come back with Phase 1 survivor data and a proposed floor, and park for a fresh decision then. Since the job lives under .github/workflows/, open the PR and flag it needs-human for the push; I will carry it.

@clubanderson

Un-parked by @clubanderson. Decision: Approved for Phase 1 only (advisory, gremlins, changed packages, 20m hard timeout, never blocks). Phase 2 (blocking on an efficacy floor) is not approved yet - come back with Phase 1 survivor data and a proposed floor, and park for a fresh decision then. Since the job lives under .github/workflows/, open the PR and flag it needs-human for the push; I will carry it.

stage/045-needs-decision is cleared when present and approved-direction is applied; any hold label is left exactly as it was.

/hive decision

Phase 1 (#17416) has run long enough to decide Phase 2. Evidence from the last 30 successful mutation-advisory.yml runs (gremlins v0.6.0, --diff origin/<base>):

Outcome	Runs
Killed 0 / Lived 0 / Not covered 0 (nothing mutated)	16
Mutants produced, all killed	8
Lived > 0	1 (Killed 1, Lived 2)
Not covered > 0	6 (23 uncovered mutants total)

Decision for Phase 2, in order:

Fix the zero-mutant case first. 16/30 runs mutated nothing. Some are test/doc-only diffs, but run 42842443653 (PR 🐛 fix(crux/resumehandle): only UUID-named claude transcripts are resume sessions #17557) reported Skipped: 108 despite adding if !looksLikeUUID(id) { return "" } to resumehandle.go. Phase 2 must assert "added Go lines in a mutated package > 0 ⇒ mutants > 0" and fail the job otherwise, or the gate silently passes everything. Verify gremlins' --diff path matching (it keys on file.NewName from git diff --merge-base, matched against fs.WalkDir relative paths) under the diff.relative setup from [shared-ci] mutation failing across hivecommons/hive #17463.
Lived > 0 blocks. Survivors are near zero today, so a hard gate costs nothing and catches the real escapes.
Not covered > 0 warns, posting the uncovered lines in the PR comment; override via a mutation/acknowledged label applied by a human. Not blocking yet — revisit after 30 more runs once (1) is fixed and the counts are trustworthy.

Non-goals: no efficacy percentage floor, no whole-package runs (time budget, #17463), no change to how edge/stable promote.

@clubanderson

@clubanderson this reads as a partial or uncertain confirmation, so the issue stays open. If it is fully fixed for you, reply /fixed to close it; otherwise please describe what is still outstanding.

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

quality (2026-11-10 17:37 UTC): Phase 2, script half, is in 🌱 mutation: verdict phase and misrouted/uncovered detection for the PR mutation report #17569 (src/scripts/mutation-report.sh + src/scripts/test-mutation-report.sh). The workflow half lives under .github/workflows/, which this agent's tier cannot push; it needs a human or an ISSUES_PRS_MERGE agent. Exact replacement for .github/workflows/mutation-advisory.yml, to land after 🌱 mutation: verdict phase and misrouted/uncovered detection for the PR mutation report #17569 merges:

1. Diagnosis of the zero-mutant case (decision item 1). Run 42842443653 is not a matching fault. gremlins v0.6.0's mutators cover arithmetic, comparison, &&/||, inc/dec, assignment and loop-control tokens only — nothing for unary ! (token.NOT) or return values — so if !looksLikeUUID(id) { return "" } yields zero mutants. All 108 SKIPPED mutants sit at resumehandle.go:132+, outside the added hunk (109–128), which is --diff working as intended. An "added lines > 0 ⇒ mutants > 0" assertion would have failed 🐛 fix(crux/resumehandle): only UUID-named claude transcripts are resume sessions #17557 wrongly. The trustworthy signature of a matching fault is a SKIPPED mutant inside an added range; 🌱 mutation: verdict phase and misrouted/uncovered detection for the PR mutation report #17569 computes the added ranges from the same merge-base diff and reports that as verdict misrouted (job fails). A run with no mutable operator in added lines is reported as such and passes.

2. Workflow edits (the only part still waiting):

name: Mutation

# #17349: surface "covered but never asserted" branches as surviving mutants
# on changed Go packages. Phase 2: survivors and diff-matching faults block;
# uncovered mutants warn unless the PR carries mutation/acknowledged.
on:
  pull_request:
    branches: [v4, v5, v6]
    paths:
      - 'src/**.go'
      - 'src/scripts/mutation-report.sh'
      - 'src/scripts/test-mutation-report.sh'

concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
  cancel-in-progress: true

permissions:
  contents: read

jobs:
  mutation:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
      - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
        with:
          cache: false
          go-version-file: src/go.mod
      - name: Self-test the mutation report script
        run: bash src/scripts/test-mutation-report.sh
      - name: Install gremlins
        run: go install github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0
      # The mutation run executes PR-controlled test code, so it gets no token:
      # it only writes mutation-out/comment.md and mutation-out/verdict. The
      # token is scoped to the separate comment-upsert step below.
      - name: Run mutation report
        env:
          BASE_REF: ${{ github.base_ref }}
        run: |
          export PATH="$(go env GOPATH)/bin:$PATH"
          bash src/scripts/mutation-report.sh run
      - name: Upsert mutation comment
        if: always()
        env:
          GH_TOKEN: ${{ github.token }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          GITHUB_REPOSITORY: ${{ github.repository }}
        run: bash src/scripts/mutation-report.sh post
      # Fails on verdict `lived` or `misrouted`; warns on `uncovered` unless the
      # PR carries mutation/acknowledged. Labels come from the event payload so
      # this step needs no token either.
      - name: Verdict
        if: always()
        env:
          PR_LABELS: ${{ join(github.event.pull_request.labels.*.name, ',') }}
        run: bash src/scripts/mutation-report.sh verdict

Diff against the current file, in words: rename to Mutation and drop the "advisory" comment; add the two script paths to paths:; remove continue-on-error: true from the job; add the self-test step before Install gremlins; add the Verdict step last with if: always() and PR_LABELS. Make mutation a required check in branch protection for v4/v5/v6 once it has been green on a few PRs (the uncovered warning never fails, so this only blocks on items 2 and the item-1 fault). Non-goals from the decision are honoured: no efficacy floor, no whole-package runs, no change to edge/stable promotion. The mutation/acknowledged label needs creating in the repo.

Item 3's "revisit after 30 more runs" now has a measurable denominator: every run writes mutation-out/verdict, and the comment states _Verdict: …_, so the next count can be taken from the comments rather than the gremlins log.

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

🌱 mutation: honour gremlins' hunk approximation and fail closed on a …

Fixed by #17416 (merged 474a7b8). Closing. Reply /reopen if the problem persists.

🌱 mutation: verdict phase and misrouted/uncovered detection for the P…

Navigation Menu

Repository navigation

[quality] Gate proposal: Go branches covered but never asserted (advisory mutation testing on changed packages) #17349

Finding

Recommendation

Priority

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[quality] Gate proposal: Go branches covered but never asserted (advisory mutation testing on changed packages) #17349

Description

Finding

Recommendation

Priority

Activity

hivecommons-hive commented on Nov 9, 2026

hivecommons-hive commented on Nov 9, 2026

github-actions commented on Nov 9, 2026

clubanderson commented on Nov 9, 2026

hivecommons-hive commented on Nov 9, 2026

12 remaining items

clubanderson commented on Nov 10, 2026

github-actions commented on Nov 10, 2026

hivecommons-hive commented on Nov 10, 2026

hivecommons-hive commented on Nov 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions