Skip to content

fix(node): a refused round no longer ends the contributor's signing loop - #199

Open
RonTuretzky wants to merge 1 commit into
legacy-orchestratorfrom
RonTuretzky/contributor-survives-refusal
Open

RonTuretzky wants to merge 1 commit into
legacy-orchestratorfrom
RonTuretzky/contributor-survives-refusal

Conversation

@RonTuretzky

Copy link
Copy Markdown
Contributor

@bagelface — one-fix branch on the legacy-orchestrator lineage (1a9716c), which is what gas-killer/service still locks. main is not affected: the consensus automaton retries validation and then signs a skip digest.

The bug

node/src/contributor/handler.rs, on a round's Start:

let payload = validator.validate_and_return_expected_hash(&buf).await?;

A validation error returns out of Contributor::run. The embedding process (gas-killer's node/src/main.rs) logs Contributor error: … and carries on: p2p stays connected, /healthz and /readyz stay green, so neither Docker's restart policy nor a k8s probe notices — and the operator signs nothing again until someone restarts it. The peer-signature branch of the same loop already treats the identical error with let Ok(..) = .. else { continue }.

It is a liveness problem, not a safety one (no divergent signature results), and it is not new — any RPC error during analysis triggers it. It became urgent with gas-killer's UNBOUNDED_V3 work (gas-killer/service#451): operators now deliberately decline tasks whose consumer needs a guest program they have not installed. In a mixed fleet, the first such task permanently removes every operator without the program from all later rounds, for every consumer — a quorum that degrades once stays degraded.

The fix

Decline the round instead of ending the loop: log at warn, continue. The round is also removed from signed, so the router's rebroadcast of the same Start gets a fresh attempt — a transient RPC failure heals on the next rebroadcast, a deterministic refusal simply declines again.

Evidence

gas-killer/service node/tests/gkvm_abstain_quorum.rs drives the real Contributor::run for five operators over in-memory p2p. Against 1a9716c it printed signing loop EXITED for both refusing operators. Against this branch all five loops stay up, and the test now asserts it (RonTuretzky/gkvm-m6-e2e). cargo test -p commonware-avs-node, clippy -D warnings, fmt: clean.

gas-killer/service consumes this branch by name until it merges.

🤖 Generated with Claude Code

On a round's Start, a validation error was propagated with `?` out of
Contributor::run. The embedding process keeps running — p2p links, health and
readiness probes all stay up — but the operator never signs again until it is
restarted. The peer-signature branch of the same loop already handles the
identical error with `continue`.

Decline the round instead: log at warn, forget the round so a rebroadcast of
the same Start is retried (transient RPC errors heal), and keep listening.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant