Skip to content

sov-db: add offline NOMT hashtable-bucket migration utility#2988

Open
Marcolist wants to merge 1 commit into
Sovereign-Labs:devfrom
Marcolist:nomt-bucket-migration
Open

sov-db: add offline NOMT hashtable-bucket migration utility#2988
Marcolist wants to merge 1 commit into
Sovereign-Labs:devfrom
Marcolist:nomt-bucket-migration

Conversation

@Marcolist

Copy link
Copy Markdown

Description

Adds an offline, root-verified utility to rebuild a namespace's NOMT page-table with a different hashtable_buckets, without replaying DA — the missing escape hatch for the kernel_state "Occupancy rate for NOMT hashtable is too high. Please update buckets size and resync database" warning.

Why this is needed. Several kernel state maps add one entry per block and are never pruned, so the authenticated kernel keyspace grows linearly forever and any fixed kernel_hashtable_buckets eventually fills. The bucket count “Cannot be changed for the existing database” (nomt has no in-place resize) and the default is only 256k, so the suggested remedy ("resync database" = replay DA) is impossible for a node without archival DA. At ~100% occupancy the node can no longer commit.

Evidence it's causal (not correlational). On a mainnet rollup at height 1,694,895, a prefix histogram of the kernel flat CF (kernel_jmt_values, total 6,565,869 entries, one consistent pre-rebuild snapshot at height ≈1,688,645) shows three in-trie key families each holding exactly one entry per block (counts 1,688,644–1,688,645 == block height); the keyspace is ≈4× block-height. This matches slot_number_history: StateMap<RollupHeight, VisibleSlotNumber> (a plain in-trie StateMap, written every block, never removed). On the arithmetic: hashtable_buckets sizes NOMT's page table (each bucket = a page holding many keys), so occupancy ≠ key count — the 6.5M keys occupied 231,877 / 256,000 page slots = 90.6%, rising ~0.5%/day.

What this PR adds. hashtable_buckets is independent of the flat store, so the trie is rebuilt locally from it:

  1. open the source NOMT, read its root;
  2. enumerate the namespace's flat CF (canonical SlotKey → SlotValue);
  3. re-derive each authenticated leaf (value_hash ++ size, matching NodeLeaf::make_leaf / combine_val_hash_and_size) into a fresh NOMT with the new bucket count;
  4. assert the rebuilt root equals the source root before persisting — a faulty reconstruction aborts instead of corrupting state.

API: sov_db::migration::{rebuild_kernel, rebuild_user} (generic over the state hasher) + examples/migrate_nomt_buckets.rs (CLI, Sha256). Run with the node stopped, then swap the output dir in and set the matching *_hashtable_buckets before restarting (the nomt_based startup check validates by root hash, not nomt's internal version).

Open question for maintainers (the deeper fix). This is a mitigation. The root cause is the unbounded in-trie footprint of these per-block maps. Sibling maps in the same struct — accessory_slot_number_history / true_slot_number_history — are already AccessoryStateMap (out of the trie). Does the authenticated slot_number_history need the full history in-trie, or only the provable/pruning window with older entries served from a non-trie store? Happy to follow up on whatever shape you prefer.

  • I have updated CHANGELOG.md with a new entry if my PR makes any breaking changes or fixes a bug. If my PR removes a feature or changes its behavior, I provide help for users on how to migrate to the new behavior.
  • I have carefully reviewed all my Cargo.toml changes before opening the PRs. (Are all new dependencies necessary? Is any module dependency leaked into the full-node (hint: it shouldn't)?)

(No Cargo.toml changes — no new dependencies; the example uses the existing sha2 dev-dependency. Additive only: a new pub mod migration + one example.)

Linked Issues

  • No separate issue — the full problem analysis is in the description above (kept the report and the fix in one place). Happy to split it into a tracking issue for the root-cause fix if you'd prefer.

Testing

  • Correctness gate: the rebuild asserts new_root == old_root and refuses to persist otherwise, so a wrong reconstruction cannot corrupt state.
  • Production: applied to a mainnet Celestia rollup whose kernel hashtable was at ~90.6% of 256k buckets — rebuilt at 4,000,000 buckets, the gate confirmed an identical root, the node restarted cleanly and resynced, page occupancy dropped to ~5.8%.
  • cargo fmt, cargo clippy and cargo check are clean for sov-db (lib + example).
  • Not yet added: a self-contained unit test (it needs a small fixture flat-DB + source NOMT). Glad to add one if you'd like it in this PR.

Docs

  • The migration module and its public functions are documented (rustdoc), including the run procedure and the root-gate guarantee.
  • The migrate_nomt_buckets example carries CLI usage and the stop-node / swap / config steps in its header.

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@Marcolist

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Jun 18, 2026
The kernel state keyspace grows unboundedly over a chain's lifetime (per-block
maps such as chain-state's slot_number_history are never pruned), so a long-running
node eventually saturates its kernel NOMT page table. hashtable_buckets is fixed at
DB creation and nomt has no in-place resize, and the only documented remedy
('resync database') requires replaying DA — impossible for a non-archival node.

Add a migration utility that rebuilds a namespace's NOMT with a new hashtable_buckets
purely from the flat state store (which is independent of the bucket count): it
enumerates the namespace's flat column family, re-derives each authenticated leaf
(value hash + size, matching NodeLeaf::make_leaf), and commits them into a fresh
NOMT. The rebuilt root is asserted equal to the source root before anything is
persisted, so a faulty reconstruction aborts instead of corrupting state.

- sov-db: new pub module 'migration' with rebuild_kernel/rebuild_user
- sov-db: example 'migrate_nomt_buckets' (CLI), run with the node stopped
@Marcolist
Marcolist force-pushed the nomt-bucket-migration branch from 473987a to 72f4506 Compare June 18, 2026 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant