🐛 [v6] contributor-first: a relay in a disabled tier or with no matching work does not cover a lane

changelog.d/fixed-disabled-tier-no-work-relay-not-cover.md

@@ -0,0 +1 @@
+- Contributor-first lanes are no longer left unworked by a relay the hub will not hand work to, or has no work for. A relay whose trust tier is in `hub.disabled_tiers` (or whose access is revoked) is refused every task, and a relay whose requests keep being answered `no_matching_work` (or `repo_unmintable`) is connected and idle, but both still counted as covering their role, so the governor kept the resident agent's kicks withheld while nobody worked the lane. Seen on hive-wild-mole on 2026-11-08: a relay covered the scanner lane while every request for work it sent, about every 30 seconds, was answered `no_matching_work`. The hub now leaves a revoked or disabled-tier relay out of coverage, using the same check that refuses it work, and leaves out a relay with no task that has been answered "no work for you" for two minutes. A task or fix-pr task handed to the relay counts it again at once.

src/docs/agent-configuration.md

@@ -1382,22 +1382,36 @@ the resident agent would flap on and off with every socket drop.
 
 A relay counts as covering the role only when all of these hold:
 
+- it is acting as the role: its effective role, after any owner "Acting as"
+  assignment, is that role;
 - its heartbeat is fresh;
-- it is not in a failure-streak pause;
+- it is signed in as a contributor with a GitHub login;
+- contributions are not suspended (`hub.contribute_suspended`);
 - it is not in standby mode (`just contribute-standby`). A standby relay never
   asks for ordinary work; it only takes what you dispatch to it by hand;
 - it has not reported a quota hold. A relay whose quota guard is holding stops
   asking for work but stays connected. A relay new enough to know about this
   tells the hub (`quota_hold_v1`, see
   [the contributor relay guide](contributor-relay.md#a-held-relay-tells-the-hub-quota_hold_v1));
   an older relay is still counted while held;
+- it is not in a failure-streak pause;
 - it has not used up its hourly or daily task allowance
   (`hub.tier_limits.<tier>.max_per_hour` / `max_per_day` for the
   contributor's trust tier). The hub refuses such a relay work
   (`hourly_limit` / `daily_limit`) until the window rolls, so it covers
   nothing in the meantime. Coverage and task assignment use the same check,
   so they always agree on which limits apply and how they are counted;
-- contributions are not suspended (`hub.contribute_suspended`).
+- its access is not revoked and its trust tier is not in
+  `hub.disabled_tiers`. The hub refuses such a relay work (`tier_disabled`),
+  using the same check as coverage, and covers again as soon as you re-enable
+  the tier;
+- it is not idle on an empty queue: a relay holding no task whose every
+  request for work has been answered `no_matching_work` (or `repo_unmintable`,
+  the same answer when the hub had to skip a repository it cannot get a token
+  for) for two minutes is connected but working nothing, so it stops covering.
+  An idle relay asks again about every 30 seconds, so one empty answer between
+  two tasks never drops it. A task or a fix-pr task handed to the relay ends
+  the idle stretch at once.
 
 A relay that is busy on a task still counts, including one holding its
 tier's `max_concurrent` tasks.

src/docs/design/agent-state-inventory.md

@@ -180,7 +180,7 @@ key generations (`/data/saas/hub-generations.json`).
 
 | # | State | Where (v5) | On a spoke (dashboard) pod roll | Verdict |
 |---|-------|------------|--------------------------------|---------|
-| 27 | Completed-task ledger + per-task cooldowns + PR URLs | `completedTasks` / `completedTaskCooldown` / `completedTaskPRURL` (`contribute_ws.go:965`), persisted to `/data/contributors/completed-tasks.json` | Survives; was moved to the PVC ledger through the #3792 → #3989 lineage after re-offer loops | **fixed** |
+| 27 | Completed-task ledger + per-task cooldowns + PR URLs | `completedTasks` / `completedTaskCooldown` / `completedTaskPRURL` (`contribute_ws.go:975`), persisted to `/data/contributors/completed-tasks.json` | Survives; was moved to the PVC ledger through the #3792 → #3989 lineage after re-offer loops | **fixed** |
 | 28 | Failure cooldowns + quarantine counters | `failedTasks` / `consecutiveFailures` (`:373`, `:381`), persisted to `/data/contributors/failed-tasks.json` | Survives (#2435 livelock fix, made durable with the ledger) | **fixed** |
 | 29 | No-PR completion streaks | `noPRStreaks` (`:394`), `/data/contributors/no-pr-streaks.json` | Survives (#3980; geometric backoff must outlive the completion entry it escalates) | **fixed** |
 | 30 | `no_work_needed` verdicts | `noWorkVerdicts` (`:416`), `/data/contributors/no-work-verdicts.json` | Survives (#3987/#3997) — "persisted in the same PVC-backed ledger dir as the cooldowns so a pod restart does not forget the verdict" | **fixed** |

src/pkg/dashboard/contribute_coverage.go

@@ -12,16 +12,19 @@ import (
 // agent role right now, keyed by role, for the governor's contributor-first
 // gate (`agents.<name>.contributor_first`; see pkg/governor/contributor_first.go).
 //
-// A relay counts as covering a role when:
+// A relay counts as covering a role when ALL of these hold:
 //
 //   - its EFFECTIVE role is that role — the owner's "Acting as" assignment
 //     after it has overridden whatever the relay requested, which is what
 //     c.role already holds once the hub admitted the claim;
 //   - its heartbeat is fresh, by the same wsHeartbeatTimeout rule
 //     FleetSnapshot uses to mark a relay stale, so a relay whose socket died
 //     silently cannot keep the resident agent idle;
-//   - it is not in a failure-streak pause: a relay whose runtime keeps dying
-//     is connected but is not doing the lane's work;
+//   - it is signed in as a contributor with a GitHub login;
+//   - the operator has not suspended contributions (hub.contribute_suspended).
+//     The suspend switch refuses work to every relay while their sockets stay
+//     up, so a connected relay is then covering nothing; reporting it as
+//     coverage would turn "pause contributors" into "pause the lane";
 //   - it has not declared standby (standby_declare, accepted or not). A
 //     standby relay stays connected but never asks for ordinary work; it only
 //     takes what an owner dispatches to it by hand, so it drains nothing;
@@ -31,24 +34,35 @@ import (
 //     Observed on 2026-11-08: a relay held at 15% of its weekly window kept
 //     the scanner lane "covered" while nobody worked it. A relay too old to
 //     send quota_hold is still counted, as before;
+//   - it is not in a failure-streak pause: a relay whose runtime keeps dying
+//     is connected but is not doing the lane's work;
 //   - its identity has not used up its hourly or daily task allowance
 //     (hub.tier_limits max_per_hour / max_per_day for its trust tier). Task
 //     selection refuses such a relay with hourly_limit / daily_limit until the
 //     window rolls, so it takes nothing in the meantime. The check is
 //     tierRateLimitReached, the same function selection refuses with, so the
 //     two can never disagree about which limits apply or how they are counted;
-//   - the operator has not suspended contributions (hub.contribute_suspended).
-//     The suspend switch refuses work to every relay while their sockets stay
-//     up, so a connected relay is then covering nothing; reporting it as
-//     coverage would turn "pause contributors" into "pause the lane".
+//   - its profile is not revoked and its trust tier is not in
+//     hub.disabled_tiers. The check is tierAccessGateReason, the rule a resume
+//     is refused with and whose disabled-tier half (tierDisabled) refuses a
+//     fresh assignment, so a relay the hub will not hand work to is never
+//     reported as working the lane;
+//   - it has not been answered no_matching_work (or repo_unmintable, its
+//     re-scan form) continuously for coverageNoWorkWindow while holding no
+//     task. Such a relay is connected and idle: the queue has nothing it may
+//     take. Observed on 2026-11-08: a relay covered the scanner lane while
+//     every ready it sent was answered no_matching_work. One empty answer
+//     between two tasks does not drop it (see coverageNoWorkWindow).
 //
 // A relay that is busy on a task still counts — it is draining the lane,
-// which is the point. Standby, quota hold and a used-up allowance do not wait
-// for the task to end: each means the relay will not take the lane's NEXT
+// which is the point. That includes a relay working only a fix-pr task, and a
+// relay whose last answer was no_matching_work but which has been handed work
+// since. Standby, quota hold, a used-up allowance and a closed tier do not
+// wait for the task to end: each means the relay will not take the lane's NEXT
 // item, which is what the governor is deciding whether to withhold. (Holding
-// max_concurrent tasks is different: that relay is busy, and counts.) Whether the relay is ALLOWED the role (trust tier,
-// grants, the delegatable list) was settled when the hub set c.role; nothing
-// is re-derived here.
+// max_concurrent tasks is different: that relay is busy, and counts.) Whether
+// the relay is ALLOWED the role (trust tier, grants, the delegatable list) was
+// settled when the hub set c.role; nothing is re-derived here.
 func (h *ContributeWSHub) RoleCoverage(now time.Time) map[string]governor.ContributorCoverage {
 	if h == nil || h.contributionsSuspended() {
 		return nil
@@ -71,8 +85,9 @@ func (h *ContributeWSHub) RoleCoverage(now time.Time) map[string]governor.Contri
 			tier = c.profile.TrustTier
 		}
 		notTakingWork := c.standbyDeclared || len(c.standby) > 0 || c.quotaHeld
+		queueEmpty := c.noWorkForWindowLocked(now)
 		c.mu.Unlock()
-		if role == "" || !fresh || login == "" || notTakingWork {
+		if role == "" || !fresh || login == "" || notTakingWork || queueEmpty {
 			continue
 		}
 		candidates = append(candidates, candidate{role: role, login: login, identity: identityOf(c), tier: tier})
@@ -83,9 +98,13 @@ func (h *ContributeWSHub) RoleCoverage(now time.Time) map[string]governor.Contri
 	// their own lock (failureStreakMu, rateMu); consult them with h.mu
 	// released so neither is ever nested inside it. The rate check is a scan
 	// of at most a day of assignment stamps for one identity, once per
-	// candidate per governor evaluation, and never takes selectMu.
+	// candidate per governor evaluation, and never takes selectMu. The tier
+	// gate reads only the config, from the tier copied out under c.mu above.
 	byRole := make(map[string]map[string]struct{})
 	for _, cand := range candidates {
+		if h.tierAccessGateReason(cand.tier) != "" {
+			continue
+		}
 		if paused, _, _ := h.contributorFailureStreakActive(cand.identity, now); paused {
 			continue
 		}
@@ -112,6 +131,65 @@ func (h *ContributeWSHub) RoleCoverage(now time.Time) map[string]governor.Contri
 	return out
 }
 
+// coverageNoWorkWindow is how long a relay holding no task must have been
+// answered nothing but no_matching_work / repo_unmintable before RoleCoverage
+// stops counting it.
+//
+// Two minutes, measured from the FIRST answer of the run. An idle relay asks
+// again 30 s after each task_unavailable (TASK_UNAVAILABLE_RETRY_MS in
+// bin/contributor-relay.js), so that is five empty answers in a row: one
+// empty answer between two tasks, with work arriving on the next ask, never
+// reaches it. It is a time, not an answer count, so a relay that asks less
+// often (or stops asking) is judged on how long it has been idle, not on how
+// many times it happened to ask. It does not need to be longer to stop the
+// resident agent flapping: RoleCoverage is sampled once per governor
+// evaluation (governor.eval_interval_s, 300 s by default, sooner on webhook
+// triggers), and a role that drops out of the report is still withheld for
+// contributor_first_grace (10 min by default) before the resident agent is
+// kicked, so a relay that finds work again inside the grace keeps the lane.
+const coverageNoWorkWindow = 2 * time.Minute
+
+// idleRunKey folds the task_unavailable reasons that mean "nothing in the queue
+// for this relay" into one run: the re-scan wrapper in selectTask renames
+// no_matching_work to repo_unmintable when it dropped an unmintable repo on the
+// way, so the two can alternate on consecutive readies with nothing changing.
+func idleRunKey(reason string) string {
+	if reason == taskUnavailableRepoUnmintable {
+		return taskUnavailableNoMatchingWork
+	}
+	return reason
+}
+
+// noteIdleReasonLocked records reason as the task_unavailable answer just sent
+// to c, starting a new run unless it continues the current one. The caller
+// holds c.mu.
+func (c *ContributorConnection) noteIdleReasonLocked(reason string, now time.Time) {
+	if c.lastIdleReasonSince.IsZero() || idleRunKey(reason) != idleRunKey(c.lastIdleReason) {
+		c.lastIdleReasonSince = now
+	}
+	c.lastIdleReason = reason
+}
+
+// clearIdleReasonLocked ends the idle run: c has been handed work. The caller
+// holds c.mu.
+func (c *ContributorConnection) clearIdleReasonLocked() {
+	c.lastIdleReason = ""
+	c.lastIdleReasonSince = time.Time{}
+}
+
+// noWorkForWindowLocked reports whether c holds no task (primary or fix-pr)
+// and has been answered no_matching_work / repo_unmintable continuously for at
+// least coverageNoWorkWindow at now. The caller holds c.mu.
+func (c *ContributorConnection) noWorkForWindowLocked(now time.Time) bool {
+	if c.currentTask != nil || len(c.fixTasks) > 0 {
+		return false
+	}
+	if c.lastIdleReasonSince.IsZero() || idleRunKey(c.lastIdleReason) != taskUnavailableNoMatchingWork {
+		return false
+	}
+	return now.Sub(c.lastIdleReasonSince) >= coverageNoWorkWindow
+}
+
 // contributionsSuspended mirrors the task-selection gate in
 // contribute_select.go: with hub.contribute_suspended set, no relay is handed
 // work, whatever role it holds.