Skip to content

Commit cfe210d

Browse files
Danatharclaude
andcommitted
🐛 [v6] contributor-first: a relay busy for another hive does not cover this hive's lane
A relay subscribed to several hives works one task at a time. While that task belonged to hive B, hive A still saw a connected relay holding its "Acting as" role with a fresh heartbeat and no refusal, so RoleCoverage counted it as contributor-first cover and A's resident agent stayed withheld while nobody worked A's lane. The relay now tells every other hub that advertises the new busy_elsewhere_v1 capability when it starts and finishes another hive's task (state busy/clear plus the task's start time; no hive, repo or issue). It restates on auth_ok and clears on complete, fail and revoke, since all of those end in writeRelayStatus. The hub records the flag per connection, clears it on the relay's next ready, and RoleCoverage skips a busy-elsewhere connection that holds no task or fix-pr task from this hub. No wall-clock expiry: the flag dies with the connection, and a stale flag can only fall back to the resident agent working the lane. Old relays and old hubs keep today's behaviour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
1 parent be24159 commit cfe210d

11 files changed

Lines changed: 371 additions & 5 deletions

‎bin/contributor-relay.js‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1776,6 +1776,10 @@ function makeHub(url, token, disabled = false, contributorId = '', repoScope = n
17761776
// (quota_hold, capability quota_hold_v1). Reset on every auth_ok: a fresh
17771777
// connection on the hub starts out not held.
17781778
quotaHoldSent: false,
1779+
// What this hub's CURRENT connection was last told about this relay
1780+
// working ANOTHER hive's task (busy_elsewhere, capability
1781+
// busy_elsewhere_v1). Reset on every auth_ok, like quotaHoldSent.
1782+
busyElsewhereSent: false,
17791783
// Set by handleTLSHostMismatch when the dialed host is not in the served
17801784
// certificate; parks the hub (no reconnect loop) until contributor.env
17811785
// changes. relocationAttempted bounds the hub-registry re-resolution to
@@ -1916,6 +1920,9 @@ function writeRelayStatus() {
19161920
// Every quota hold transition already ends here (it changes relayReadiness),
19171921
// so this is the one place the hub is told — see reportQuotaHold.
19181922
reportQuotaHoldToHubs();
1923+
// Every currentTask transition (assign, complete, fail, revoke) ends here
1924+
// too, so the same holds for busy_elsewhere — see reportBusyElsewhere.
1925+
reportBusyElsewhereToHubs();
19191926
}
19201927

19211928
// ── Quota hold report (quota_hold_v1) ────────────────────────────────────────
@@ -1950,6 +1957,39 @@ function reportQuotaHoldToHubs() {
19501957
for (const hub of hubs) reportQuotaHold(hub);
19511958
}
19521959

1960+
// ── Busy on another hive (busy_elsewhere_v1) ─────────────────────────────────
1961+
// A relay subscribed to several hives works one primary task at a time. While
1962+
// that task belongs to hive B, hive A still sees a connected relay that
1963+
// heartbeats, holds its "Acting as" role and has been refused nothing, so A's
1964+
// contributor-first gate (RoleCoverage) counted it as covering A's lane and
1965+
// withheld A's resident agent while nobody worked that lane. Tell every OTHER
1966+
// hub that advertises busy_elsewhere_v1 when such a task starts and when it
1967+
// ends; a hub without the token never sees the frame, and keeps today's
1968+
// reading. The frame carries only when the other task started — never which
1969+
// hive, repository or issue — so one hive learns nothing about another's work.
1970+
// The local review cycle belongs to no hub and is not reported.
1971+
function busyElsewhereFor(hub) {
1972+
if (!currentTask || isLocalOnlyTask(currentTask)) return false;
1973+
const owner = currentTask._hub;
1974+
return !!owner && owner !== hub;
1975+
}
1976+
1977+
function reportBusyElsewhere(hub) {
1978+
if (!hub || !hub.authenticated || hub.disabled) return;
1979+
if (!Array.isArray(hub.serverCapabilities) || !hub.serverCapabilities.includes('busy_elsewhere_v1')) return;
1980+
const busy = busyElsewhereFor(hub);
1981+
if (busy === hub.busyElsewhereSent) return;
1982+
if (!hub.ws || hub.ws.readyState !== WebSocket.OPEN) return;
1983+
hub.busyElsewhereSent = busy;
1984+
sendTo(hub, busy
1985+
? { type: 'busy_elsewhere', seq: nextSeq(), state: 'busy', since: currentTaskStartedAt || new Date().toISOString() }
1986+
: { type: 'busy_elsewhere', seq: nextSeq(), state: 'clear' });
1987+
}
1988+
1989+
function reportBusyElsewhereToHubs() {
1990+
for (const hub of hubs) reportBusyElsewhere(hub);
1991+
}
1992+
19531993
let progressInterval = null;
19541994
let verdictWatchInterval = null;
19551995
let verdictFastPathLine = null;
@@ -9810,6 +9850,10 @@ function handleMessage(data, hub) {
98109850
// before anything else so contributor-first cover is right from the start.
98119851
hub.quotaHoldSent = false;
98129852
reportQuotaHold(hub);
9853+
// Same for busy_elsewhere: a new connection starts out not busy on the
9854+
// hub, so restate another hive's running task to it.
9855+
hub.busyElsewhereSent = false;
9856+
reportBusyElsewhere(hub);
98139857
declareStandby(hub);
98149858
hub.scopeSent = false;
98159859
negotiateRepoScope(hub);

‎bin/contributor-relay.test.js‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17338,6 +17338,85 @@ test('#16463 standby acknowledgement does not outlive the connection it was gran
1733817338
}
1733917339
});
1734017340

17341+
// ---------------------------------------------------------------------------
17342+
// busy_elsewhere_v1: a relay working one hive's task tells its OTHER hives, so
17343+
// they stop counting it as contributor-first cover. Observed on 2026-11-09: a
17344+
// relay set "Acting as: scanner" on projectbluefin worked hivecommons tasks
17345+
// from 14:25Z to 14:32Z and sent projectbluefin no ready.
17346+
// ---------------------------------------------------------------------------
17347+
17348+
function busyElsewhereFrames(sent) {
17349+
return sent.filter(m => m.type === 'busy_elsewhere');
17350+
}
17351+
17352+
function busyAuth(relay, hub, capabilities = ['busy_elsewhere_v1']) {
17353+
relay.handleMessage(JSON.stringify({ type: 'auth_ok', contributor_id: 'c1', trust_tier: 'contributor', server_capabilities: capabilities }), hub);
17354+
}
17355+
17356+
// Rotate the polling slot to hive B (as a task_unavailable from A would) and
17357+
// take B's task through the real assignment handler.
17358+
function takeTaskFromHubB(relay, hubs, taskId) {
17359+
relay.sendReadyForNextTask('unavailable');
17360+
relay.handleMessage(JSON.stringify({ type: 'task_assign', task_id: taskId, task_gen: 1, kind: 'issue', repo: 'org/b', number: 7, title: 'work on hive B' }), hubs[1]);
17361+
assert.strictEqual(relay.getCurrentTask() && relay.getCurrentTask().task_id, taskId, 'precondition: the relay took hive B\'s task');
17362+
}
17363+
17364+
test('busy_elsewhere_v1: another hive\'s task is reported to the other hive, restated on reconnect, and cleared when it finishes', () => {
17365+
const relay = loadRelay({ env: MULTI_HUB_ENV });
17366+
const log = console.log; const warn = console.warn;
17367+
console.log = () => {}; console.warn = () => {};
17368+
try {
17369+
const { hubs, sentA, sentB } = attachHubSinks(relay);
17370+
busyAuth(relay, hubs[0]); busyAuth(relay, hubs[1]);
17371+
assert.deepStrictEqual(busyElsewhereFrames([...sentA, ...sentB]), [], 'an idle relay is busy nowhere');
17372+
17373+
takeTaskFromHubB(relay, hubs, 'b-1');
17374+
const frames = busyElsewhereFrames(sentA);
17375+
assert.deepStrictEqual(frames.map(m => m.state), ['busy'], 'hive A must hear about hive B\'s task once');
17376+
assert.deepStrictEqual(Object.keys(frames[0]).sort(), ['seq', 'since', 'state', 'type'],
17377+
'the frame names no hive, repository or task: one hive learns nothing about another\'s work');
17378+
assert.ok(!Number.isNaN(Date.parse(frames[0].since)), `since must be a timestamp: ${frames[0].since}`);
17379+
assert.deepStrictEqual(busyElsewhereFrames(sentB), [], 'the hive that owns the task is never told it is busy elsewhere');
17380+
17381+
// A reconnect to hive A is a new connection that knows nothing: restate.
17382+
busyAuth(relay, hubs[0]);
17383+
assert.deepStrictEqual(busyElsewhereFrames(sentA).map(m => m.state), ['busy', 'busy']);
17384+
17385+
relay.finishCurrentTask({ completionSignal: 'verdict', summary: 'done', tmuxLines: [], prURL: null, noWork: null });
17386+
assert.deepStrictEqual(busyElsewhereFrames(sentA).map(m => m.state), ['busy', 'busy', 'clear'],
17387+
'finishing hive B\'s task must clear the report on hive A');
17388+
assert.deepStrictEqual(busyElsewhereFrames(sentB), []);
17389+
} finally { console.log = log; console.warn = warn; relay.setCurrentTask(null); teardown(relay); }
17390+
});
17391+
17392+
test('busy_elsewhere_v1: a revoked task clears the report', () => {
17393+
const relay = loadRelay({ env: MULTI_HUB_ENV });
17394+
const log = console.log; const warn = console.warn;
17395+
console.log = () => {}; console.warn = () => {};
17396+
try {
17397+
const { hubs, sentA } = attachHubSinks(relay);
17398+
busyAuth(relay, hubs[0]); busyAuth(relay, hubs[1]);
17399+
takeTaskFromHubB(relay, hubs, 'b-2');
17400+
relay.handleMessage(JSON.stringify({ type: 'task_revoke', task_id: 'b-2', reason: 'operator stop' }), hubs[1]);
17401+
assert.strictEqual(relay.getCurrentTask(), null, 'precondition: the revoke ended the task');
17402+
assert.deepStrictEqual(busyElsewhereFrames(sentA).map(m => m.state), ['busy', 'clear']);
17403+
} finally { console.log = log; console.warn = warn; relay.setCurrentTask(null); teardown(relay); }
17404+
});
17405+
17406+
test('busy_elsewhere_v1: a hive that does not advertise the token never sees the frame', () => {
17407+
const relay = loadRelay({ env: MULTI_HUB_ENV });
17408+
const log = console.log; const warn = console.warn;
17409+
console.log = () => {}; console.warn = () => {};
17410+
try {
17411+
const { hubs, sentA, sentB } = attachHubSinks(relay);
17412+
busyAuth(relay, hubs[0], ['quota_hold_v1']); busyAuth(relay, hubs[1]);
17413+
takeTaskFromHubB(relay, hubs, 'b-3');
17414+
busyAuth(relay, hubs[0], ['quota_hold_v1']);
17415+
relay.finishCurrentTask({ completionSignal: 'verdict', summary: 'done', tmuxLines: [], prURL: null, noWork: null });
17416+
assert.deepStrictEqual(busyElsewhereFrames([...sentA, ...sentB]), [], 'an old hub must see exactly today\'s traffic');
17417+
} finally { console.log = log; console.warn = warn; relay.setCurrentTask(null); teardown(relay); }
17418+
});
17419+
1734117420
// ---------------------------------------------------------------------------
1734217421
// quota_hold_v1: a held relay tells the hub, so it stops counting as
1734317422
// contributor-first cover. Observed on 2026-11-08: a relay acting as scanner
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
- Contributor-first lanes are no longer left unworked by a relay that is busy on another hive. A relay subscribed to several hives works one task at a time, and while that task belonged to another hive it still looked to this one like a connected relay holding its role, so the governor kept the resident agent's kicks withheld while nobody worked the lane (seen on 2026-11-09: a relay set "Acting as: scanner" on one hive worked another hive's tasks for seven minutes and sent the first hive no request for work). Relays now tell every other hive that advertises the new `busy_elsewhere_v1` capability when they start and finish another hive's task, and the hub leaves such a relay out of coverage until the task ends or the relay asks it for work again. The frame names no hive, repository or issue. Older relays and older hubs behave as before.

‎src/docs/agent-configuration.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1394,6 +1394,14 @@ A relay counts as covering the role only when all of these hold:
13941394
tells the hub (`quota_hold_v1`, see
13951395
[the contributor relay guide](contributor-relay.md#a-held-relay-tells-the-hub-quota_hold_v1));
13961396
an older relay is still counted while held;
1397+
- it is not busy on another hive. A relay subscribed to several hives works
1398+
one task at a time; while that task belongs to another hive it stays
1399+
connected here but asks this hive for nothing. A relay new enough to know
1400+
about this tells every other hive it is subscribed to (`busy_elsewhere_v1`, see
1401+
[the contributor relay guide](contributor-relay.md#a-relay-busy-on-another-hive-tells-the-hub-busy_elsewhere_v1))
1402+
and clears it when that task ends; its next request for work here clears it
1403+
too. A relay that this hive has also handed a task or a fix-pr task still
1404+
counts, and an older relay is still counted while busy elsewhere;
13971405
- it is not in a failure-streak pause;
13981406
- it has not used up its hourly or daily task allowance
13991407
(`hub.tier_limits.<tier>.max_per_hour` / `max_per_day` for the

‎src/docs/contributor-relay.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2021,6 +2021,46 @@ about it:
20212021

20222022
The expired-CLI-login hold is not reported this way yet.
20232023

2024+
### A relay busy on another hive tells the hub (`busy_elsewhere_v1`)
2025+
2026+
A relay subscribed to several hives works one task at a time. While that task
2027+
belongs to hive B, hive A still sees a connected relay that answers heartbeats,
2028+
holds its "Acting as" role and has been refused nothing. Before this, a relay
2029+
acting as a role on A kept counting as
2030+
[contributor-first](agent-configuration.md#contributor-first-lanes-contributor_first)
2031+
cover there for as long as it worked B's tasks, so A's resident agent was held
2032+
back while nobody worked A's lane.
2033+
2034+
Every other hub that advertises `busy_elsewhere_v1` in
2035+
`auth_ok.server_capabilities` now hears about it:
2036+
2037+
```jsonc
2038+
{ "type": "busy_elsewhere", "seq": 61, "state": "busy", "since": "2026-11-09T14:25:03Z" }
2039+
{ "type": "busy_elsewhere", "seq": 62, "state": "clear" }
2040+
```
2041+
2042+
- The relay sends `busy` when it starts a task from another hive and `clear`
2043+
when that task completes, fails or is revoked. It sends each transition once
2044+
per connection, and restates a running task right after `auth_ok`, because a
2045+
new connection starts out not busy on the hub. The hive that owns the task
2046+
is never sent the frame.
2047+
- The frame says only when the other task started. It never names the other
2048+
hive, repository or issue.
2049+
- The hub records it on the connection and uses it only for contributor-first
2050+
coverage. It changes no lease, assignment or standby state. A relay this hub
2051+
has also handed a task or a fix-pr task still counts.
2052+
- There is no timer on the hub. The flag goes away with the connection (a
2053+
socket close or a relay restart), and the relay's next `ready` to this hub
2054+
clears it, since a relay asks for work only when it holds none. A timer
2055+
would start counting a relay that is still busy on a long task again; a flag
2056+
left set by mistake only means the resident agent works the lane, which is
2057+
what happens without `contributor_first`.
2058+
- A hub that does not advertise the token never receives the frame. A relay
2059+
that never sends it is read as not busy elsewhere. Either way the behaviour
2060+
is exactly what it was before.
2061+
2062+
The local PR-review cycle belongs to no hive and is not reported.
2063+
20242064
### Provider adapter sources
20252065

20262066
The `src/pkg/rotation` probers normalize each provider's usage into the reading

‎src/docs/design/agent-state-inventory.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -180,7 +180,7 @@ key generations (`/data/saas/hub-generations.json`).
180180

181181
| # | State | Where (v5) | On a spoke (dashboard) pod roll | Verdict |
182182
|---|-------|------------|--------------------------------|---------|
183-
| 27 | Completed-task ledger + per-task cooldowns + PR URLs | `completedTasks` / `completedTaskCooldown` / `completedTaskPRURL` (`contribute_ws.go:975`), persisted to `/data/contributors/completed-tasks.json` | Survives; was moved to the PVC ledger through the #3792 → #3989 lineage after re-offer loops | **fixed** |
183+
| 27 | Completed-task ledger + per-task cooldowns + PR URLs | `completedTasks` / `completedTaskCooldown` / `completedTaskPRURL` (`contribute_ws.go:985`), persisted to `/data/contributors/completed-tasks.json` | Survives; was moved to the PVC ledger through the #3792 → #3989 lineage after re-offer loops | **fixed** |
184184
| 28 | Failure cooldowns + quarantine counters | `failedTasks` / `consecutiveFailures` (`:373`, `:381`), persisted to `/data/contributors/failed-tasks.json` | Survives (#2435 livelock fix, made durable with the ledger) | **fixed** |
185185
| 29 | No-PR completion streaks | `noPRStreaks` (`:394`), `/data/contributors/no-pr-streaks.json` | Survives (#3980; geometric backoff must outlive the completion entry it escalates) | **fixed** |
186186
| 30 | `no_work_needed` verdicts | `noWorkVerdicts` (`:416`), `/data/contributors/no-work-verdicts.json` | Survives (#3987/#3997) — "persisted in the same PVC-backed ledger dir as the cooldowns so a pod restart does not forget the verdict" | **fixed** |

‎src/pkg/dashboard/contribute_coverage.go‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,15 @@ import (
3434
// Observed on 2026-11-08: a relay held at 15% of its weekly window kept
3535
// the scanner lane "covered" while nobody worked it. A relay too old to
3636
// send quota_hold is still counted, as before;
37+
// - it has not reported that it is busy on another hive (busy_elsewhere,
38+
// capability busy_elsewhere_v1) while holding no task from this one. A
39+
// relay subscribed to several hives works one task at a time; while that
40+
// task belongs to another hive it heartbeats and keeps its role here but
41+
// asks this hub for nothing. Observed on 2026-11-09: a relay set "Acting
42+
// as: scanner" on one hive spent the next seven minutes on another hive's
43+
// tasks and sent the first hive no ready. Its next ready here clears the
44+
// report (see busyElsewhereLocked). A relay too old to send busy_elsewhere
45+
// is still counted, as before;
3746
// - it is not in a failure-streak pause: a relay whose runtime keeps dying
3847
// is connected but is not doing the lane's work;
3948
// - its identity has not used up its hourly or daily task allowance
@@ -60,7 +69,9 @@ import (
6069
// since. Standby, quota hold, a used-up allowance and a closed tier do not
6170
// wait for the task to end: each means the relay will not take the lane's NEXT
6271
// item, which is what the governor is deciding whether to withhold. (Holding
63-
// max_concurrent tasks is different: that relay is busy, and counts.) Whether
72+
// max_concurrent tasks is different: that relay is busy, and counts. So is a
73+
// relay busy on another hive that this hub has also handed a task or a fix-pr
74+
// task: it is working this lane too.) Whether
6475
// the relay is ALLOWED the role (trust tier, grants, the delegatable list) was
6576
// settled when the hub set c.role; nothing is re-derived here.
6677
func (h *ContributeWSHub) RoleCoverage(now time.Time) map[string]governor.ContributorCoverage {
@@ -84,7 +95,7 @@ func (h *ContributeWSHub) RoleCoverage(now time.Time) map[string]governor.Contri
8495
login = strings.TrimSpace(c.profile.GitHubUsername)
8596
tier = c.profile.TrustTier
8697
}
87-
notTakingWork := c.standbyDeclared || len(c.standby) > 0 || c.quotaHeld
98+
notTakingWork := c.standbyDeclared || len(c.standby) > 0 || c.quotaHeld || c.busyElsewhereLocked()
8899
queueEmpty := c.noWorkForWindowLocked(now)
89100
c.mu.Unlock()
90101
if role == "" || !fresh || login == "" || notTakingWork || queueEmpty {
@@ -190,6 +201,21 @@ func (c *ContributorConnection) noWorkForWindowLocked(now time.Time) bool {
190201
return now.Sub(c.lastIdleReasonSince) >= coverageNoWorkWindow
191202
}
192203

204+
// busyElsewhereLocked reports whether c has said it is working another hive's
205+
// task (busy_elsewhere) and this hub has handed it no task of its own, primary
206+
// or fix-pr. The caller holds c.mu.
207+
//
208+
// There is no wall-clock expiry. The flag is per connection, so a socket close
209+
// or a relay restart drops it with the connection; the relay clears it when the
210+
// other task ends, and the relay's next ready here clears it too (handleReady).
211+
// A timeout would only re-count a relay that is still busy on a long task —
212+
// the bug this check exists to stop — while a flag left set by mistake errs
213+
// the safe way: the resident agent works the lane, as it would without
214+
// contributor_first.
215+
func (c *ContributorConnection) busyElsewhereLocked() bool {
216+
return c.busyElsewhere && c.currentTask == nil && len(c.fixTasks) == 0
217+
}
218+
193219
// contributionsSuspended mirrors the task-selection gate in
194220
// contribute_select.go: with hub.contribute_suspended set, no relay is handed
195221
// work, whatever role it holds.

0 commit comments

Comments
 (0)