fix(node): a refused round no longer ends the contributor's signing loop - #199
Open
RonTuretzky wants to merge 1 commit into
Open
RonTuretzky wants to merge 1 commit into
RonTuretzky wants to merge 1 commit into
Conversation
On a round's Start, a validation error was propagated with `?` out of Contributor::run. The embedding process keeps running — p2p links, health and readiness probes all stay up — but the operator never signs again until it is restarted. The peer-signature branch of the same loop already handles the identical error with `continue`. Decline the round instead: log at warn, forget the round so a rebroadcast of the same Start is retried (transient RPC errors heal), and keep listening.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@bagelface — one-fix branch on the
legacy-orchestratorlineage (1a9716c), which is what gas-killer/service still locks.mainis not affected: the consensus automaton retries validation and then signs a skip digest.The bug
node/src/contributor/handler.rs, on a round'sStart:A validation error returns out of
Contributor::run. The embedding process (gas-killer'snode/src/main.rs) logsContributor error: …and carries on: p2p stays connected,/healthzand/readyzstay green, so neither Docker's restart policy nor a k8s probe notices — and the operator signs nothing again until someone restarts it. The peer-signature branch of the same loop already treats the identical error withlet Ok(..) = .. else { continue }.It is a liveness problem, not a safety one (no divergent signature results), and it is not new — any RPC error during analysis triggers it. It became urgent with gas-killer's UNBOUNDED_V3 work (gas-killer/service#451): operators now deliberately decline tasks whose consumer needs a guest program they have not installed. In a mixed fleet, the first such task permanently removes every operator without the program from all later rounds, for every consumer — a quorum that degrades once stays degraded.
The fix
Decline the round instead of ending the loop: log at
warn,continue. The round is also removed fromsigned, so the router's rebroadcast of the sameStartgets a fresh attempt — a transient RPC failure heals on the next rebroadcast, a deterministic refusal simply declines again.Evidence
gas-killer/service
node/tests/gkvm_abstain_quorum.rsdrives the realContributor::runfor five operators over in-memory p2p. Against1a9716cit printedsigning loop EXITEDfor both refusing operators. Against this branch all five loops stay up, and the test now asserts it (RonTuretzky/gkvm-m6-e2e).cargo test -p commonware-avs-node, clippy-D warnings, fmt: clean.gas-killer/service consumes this branch by name until it merges.
🤖 Generated with Claude Code