Repository navigation
fix: don't crash exec_all_mem_mutations on an empty batch - #2206
Merged
Merged
Conversation
List.last(blocks_mutations) returns nil when the batch has zero blocks
(e.g. a resume/extension check that turns out to have nothing new past
the already-synced height), and matching {{height, _mbi}, _block, _}
against nil raises MatchError, crashing the sync task.
Confirmed live on mainnet mdw1 (157.180.104.172): crashed 48 times in a
tight burst, hit @max_restarts, and AeMdw.Sync.Server went idle (no sync
progress, mdw_gens_per_minute: 0.0) until manually hot-patched. Live
patch verified sync resumed immediately after (512+ gens/min, no new
crashes).
Fix: treat an empty batch as a no-op (return state unchanged) instead of
crashing.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
exec_all_mem_mutations/2crashes withMatchError: no match of right hand side value: nilwhen the batch has zero blocks:List.last(blocks_mutations)returns
nilon an empty list, and matching{{height, _mbi}, _block, _mutations} = nilraises.Live incident (confirmed, not theoretical)
Hit this in production on mainnet shortly after
today's #2202/#2203 merges: crashed 48 times in a ~100ms burst, exhausted
AeMdw.Sync.Server's@max_restarts, and the sync GenStateMachine went idle(
mdw_gens_per_minute: 0.0, no forward progress) — matching the previouslydocumented "crashes N times then sleeps up to 1h" behavior. Confirmed via the
actual stacktrace in the deployed node's logs:
Fix
Treat an empty batch as a legitimate no-op (return
stateunchanged) insteadof crashing — this can happen when a resume/extension check computes a
from_heightthat turns out to have nothing new to sync past thealready-synced height.
Verification
today) to stop the active crash loop immediately: confirmed zero new
crashes post-patch,
mdw_gens_per_minutejumped from0.0to512+andheight caught up rapidly.
mix format --check-formatted && mix credo— clean.mix dialyzer --format github— passed successfully.mix test test/ae_mdw/sync/ test/ae_mdw/db/sync/— 55/55 passing.