Repository navigation
GO-7561 nodespace: retry creating the space's consensus log - #251
Merged
Merged
Conversation
Every node that stores a space creates its consensus log when it loads the space, usually at the same time as the others, and then watches it. A failed AddLog was only logged, and the watch that followed could miss a log that was not created yet. A failed AddLog is now retried a few times until the log is created or found existing; Forbidden and InvalidPayload are not retried. A failure is still only logged, so the space loads, and the warning now names the space.
Covers Init itself, so the retry cannot be bypassed there, and drops a wall-clock assertion from the cancellation test; the mock already checks the single attempt.
Coverage provided by https://github.com/seriousben/go-patch-cover-action |
The spaceId parameter was always s.Id(); the tests now use a mock space.
cheggaaa
approved these changes
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
nodeSpace.Initcreates the space's consensus log withAddLogand then watches it. Every node that stores the space does this, usually at the same time. A failedAddLogwas only logged, and the watch that followed could miss a log that another node had not committed yet, leaving the node without ACL pushes for that space. The warning also did not name the space.Fix
addLogretries a failedAddLogup to 3 times (100 ms, then 200 ms) until the log is created or found existing.ForbiddenandInvalidPayloadare not retried, and a done context stops the retries. A failure is still only logged, so the space loads; the warning now has the space id.Tests
TestNodeSpace_addLog: success, existing log, retry until the log exists, exhaustion, permanent error, cancelled context.TestNodeSpace_Init:AddLogis retried beforeWatch; the space still loads when every attempt fails.Deploy
A safety net: with anyproto/any-sync-consensusnode#115 the consensus node itself retries the conflicting transaction, and the losing node gets
ErrLogExists. This PR helps with older consensus nodes, if that retry is exhausted, or on a transport error. It does not re-watch: if every attempt fails, the watch still ends withErrLogNotFound, as before. Works with old and new consensus nodes; no deploy order.Part of GO-7561.