Skip to content

fix: don't crash exec_all_mem_mutations on an empty batch - #2206

Merged
venimus merged 1 commit into
masterfrom
fix/exec-all-mem-mutations-empty-batch
Sep 8, 2026
Merged

venimus merged 1 commit into
masterfrom
fix/exec-all-mem-mutations-empty-batch

Conversation

@venimus

@venimus venimus commented Sep 8, 2026 •

Copy link
Copy Markdown
Member

What

exec_all_mem_mutations/2 crashes with MatchError: no match of right hand side value: nil when the batch has zero blocks: List.last(blocks_mutations)
returns nil on an empty list, and matching {{height, _mbi}, _block, _mutations} = nil raises.

Live incident (confirmed, not theoretical)

Hit this in production on mainnet shortly after
today's #2202/#2203 merges: crashed 48 times in a ~100ms burst, exhausted
AeMdw.Sync.Server's @max_restarts, and the sync GenStateMachine went idle
(mdw_gens_per_minute: 0.0, no forward progress) — matching the previously
documented "crashes N times then sleeps up to 1h" behavior. Confirmed via the
actual stacktrace in the deployed node's logs:

** (MatchError) no match of right hand side value: nil
    (ae_mdw 1.108.0) lib/ae_mdw/sync/server.ex:494: AeMdw.Sync.Server.exec_all_mem_mutations/2
    (ae_mdw 1.108.0) lib/ae_mdw/sync/server.ex:340: anonymous fn/3 in AeMdw.Sync.Server.spawn_mem_sync_from/3

Fix

Treat an empty batch as a legitimate no-op (return state unchanged) instead
of crashing — this can happen when a resume/extension check computes a
from_height that turns out to have nothing new to sync past the
already-synced height.

Verification

  • Live-patched mainnet mdw1 directly (hot-load, same procedure as earlier
    today) to stop the active crash loop immediately: confirmed zero new
    crashes post-patch, mdw_gens_per_minute jumped from 0.0 to 512+ and
    height caught up rapidly.
  • mix format --check-formatted && mix credo — clean.
  • mix dialyzer --format github — passed successfully.
  • mix test test/ae_mdw/sync/ test/ae_mdw/db/sync/ — 55/55 passing.

List.last(blocks_mutations) returns nil when the batch has zero blocks
(e.g. a resume/extension check that turns out to have nothing new past
the already-synced height), and matching {{height, _mbi}, _block, _}
against nil raises MatchError, crashing the sync task.

Confirmed live on mainnet mdw1 (157.180.104.172): crashed 48 times in a
tight burst, hit @max_restarts, and AeMdw.Sync.Server went idle (no sync
progress, mdw_gens_per_minute: 0.0) until manually hot-patched. Live
patch verified sync resumed immediately after (512+ gens/min, no new
crashes).

Fix: treat an empty batch as a no-op (return state unchanged) instead of
crashing.
@venimus
venimus merged commit 96c410a into master Sep 8, 2026
7 checks passed
@venimus
venimus deleted the fix/exec-all-mem-mutations-empty-batch branch September 8, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant