feat(smt): Fjall storage backend for RocksDbStorage

[battlmonstr]

Overview

The goal is to implement alternative storage backends for merkle::smt::large::storage and compare them against each other. A Fjall database was chosen as a promising candidate alternative. A couple of others backends are tested but not included here.

RocksDbStorage (rocksdb.rs) used to have business logic to store a large SMT. That business logic was wired to RocksDB but in theory could be applied to any key-value storage mechanism.

Instead of copy-pasting RocksDbStorage for each backend an approach was chosen to refactor the RocksDbStorage to work with a new set of abstract KVDB traits. These traits can be implemented for multiple alternative backends. It saves time because instead of testing, updating and reviewing the whole 1700 SLOC per backend the new code has about 400 SLOC per backend.

Contents

The PR commits are organized in this sequence:

1. Preparational commits and refactorings.
2. RocksDbStorage refactoring onto KVDB traits: the traits and RocksKVDB implementation are introduced, then RocksDbStorage is ported method by method, and finalized with a code cleanup.
3. Generalizing RocksDbStorage to KVDBSmtStorage and introducing an smt-kvdb-rocks feature flag to prepare for alternative backends. Renamings after generalization.
4. Fjall KVDB implementation with smt-kvdb-fjall feature flag.

Each intermediate commit produces a valid state in which tests and benchmarks pass.
It is recommended to do code review sequentially commit by commit.

Tests

make test-large-smt
make test-large-smt-no-forest
make test-large-smt-fjall

Benchmarks

macOS 26, Mac Mini M4 Pro 10+4 cores

test	RocksDB	Fjall
test-large-smt-no-forest	0.4 sec	0.6 sec
bench-large-smt-rocksdb	11 sec	11.2 sec
bench-large-smt-rocksdb-open	3.5 sec	4 sec
bench-large-smt-codspeed-rocksdb	-	-46%..-5%
bench-large-smt-criterion-rocksdb	-	-34%..+22%

Fjall patch

Original Fjall runs criterion benchmarks and rocksdb_large_smt integration test much slower than RocksDB because it calls fsync() multiple times during normal operation while RocksDB doesn't. fjall library and its dependency lsm-tree are patched to disable this behavior and speed up the tests.

[github-actions]

Automated check (CONTRIBUTING.md)

Findings:

• Link an issue in the PR body (e.g., "Fixes feat: added enum to describe the meaning of the bit #123").

Recommendations:

• Consider adding a Test plan or clear review steps.

Next steps:

• Link a relevant issue (e.g., "Fixes feat: added enum to describe the meaning of the bit #123") and ensure it is assigned to you.
• See CONTRIBUTING.md for expectations.
• If this is a false positive, comment: /quality-review.

[battlmonstr]

/quality-review

[battlmonstr]

@sergerad, @krushimir, @huitseeker this is ready for review, please approve the CI workflows 🙏