sov-db: add offline NOMT hashtable-bucket migration utility#2988
Open
Marcolist wants to merge 1 commit into
Open
sov-db: add offline NOMT hashtable-bucket migration utility#2988Marcolist wants to merge 1 commit into
Marcolist wants to merge 1 commit into
Conversation
Contributor
|
All contributors have signed the CLA ✍️ ✅ |
Author
|
I have read the CLA Document and I hereby sign the CLA |
The kernel state keyspace grows unboundedly over a chain's lifetime (per-block
maps such as chain-state's slot_number_history are never pruned), so a long-running
node eventually saturates its kernel NOMT page table. hashtable_buckets is fixed at
DB creation and nomt has no in-place resize, and the only documented remedy
('resync database') requires replaying DA — impossible for a non-archival node.
Add a migration utility that rebuilds a namespace's NOMT with a new hashtable_buckets
purely from the flat state store (which is independent of the bucket count): it
enumerates the namespace's flat column family, re-derives each authenticated leaf
(value hash + size, matching NodeLeaf::make_leaf), and commits them into a fresh
NOMT. The rebuilt root is asserted equal to the source root before anything is
persisted, so a faulty reconstruction aborts instead of corrupting state.
- sov-db: new pub module 'migration' with rebuild_kernel/rebuild_user
- sov-db: example 'migrate_nomt_buckets' (CLI), run with the node stopped
Marcolist
force-pushed
the
nomt-bucket-migration
branch
from
June 18, 2026 08:15
473987a to
72f4506
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an offline, root-verified utility to rebuild a namespace's NOMT page-table with a different
hashtable_buckets, without replaying DA — the missing escape hatch for thekernel_state"Occupancy rate for NOMT hashtable is too high. Please update buckets size and resync database" warning.Why this is needed. Several kernel state maps add one entry per block and are never pruned, so the authenticated kernel keyspace grows linearly forever and any fixed
kernel_hashtable_bucketseventually fills. The bucket count “Cannot be changed for the existing database” (nomt has no in-place resize) and the default is only 256k, so the suggested remedy ("resync database" = replay DA) is impossible for a node without archival DA. At ~100% occupancy the node can no longer commit.Evidence it's causal (not correlational). On a mainnet rollup at height 1,694,895, a prefix histogram of the kernel flat CF (
kernel_jmt_values, total 6,565,869 entries, one consistent pre-rebuild snapshot at height ≈1,688,645) shows three in-trie key families each holding exactly one entry per block (counts 1,688,644–1,688,645 == block height); the keyspace is ≈4× block-height. This matchesslot_number_history: StateMap<RollupHeight, VisibleSlotNumber>(a plain in-trieStateMap, written every block, never removed). On the arithmetic:hashtable_bucketssizes NOMT's page table (each bucket = a page holding many keys), so occupancy ≠ key count — the 6.5M keys occupied 231,877 / 256,000 page slots = 90.6%, rising ~0.5%/day.What this PR adds.
hashtable_bucketsis independent of the flat store, so the trie is rebuilt locally from it:SlotKey → SlotValue);value_hash ++ size, matchingNodeLeaf::make_leaf/combine_val_hash_and_size) into a fresh NOMT with the new bucket count;API:
sov_db::migration::{rebuild_kernel, rebuild_user}(generic over the state hasher) +examples/migrate_nomt_buckets.rs(CLI,Sha256). Run with the node stopped, then swap the output dir in and set the matching*_hashtable_bucketsbefore restarting (thenomt_basedstartup check validates by root hash, not nomt's internal version).Open question for maintainers (the deeper fix). This is a mitigation. The root cause is the unbounded in-trie footprint of these per-block maps. Sibling maps in the same struct —
accessory_slot_number_history/true_slot_number_history— are alreadyAccessoryStateMap(out of the trie). Does the authenticatedslot_number_historyneed the full history in-trie, or only the provable/pruning window with older entries served from a non-trie store? Happy to follow up on whatever shape you prefer.CHANGELOG.mdwith a new entry if my PR makes any breaking changes or fixes a bug. If my PR removes a feature or changes its behavior, I provide help for users on how to migrate to the new behavior.Cargo.tomlchanges before opening the PRs. (Are all new dependencies necessary? Is any module dependency leaked into the full-node (hint: it shouldn't)?)(No
Cargo.tomlchanges — no new dependencies; the example uses the existingsha2dev-dependency. Additive only: a newpub mod migration+ one example.)Linked Issues
Testing
new_root == old_rootand refuses to persist otherwise, so a wrong reconstruction cannot corrupt state.cargo fmt,cargo clippyandcargo checkare clean forsov-db(lib + example).Docs
migrationmodule and its public functions are documented (rustdoc), including the run procedure and the root-gate guarantee.migrate_nomt_bucketsexample carries CLI usage and the stop-node / swap / config steps in its header.