MemoryKV is the address of a memory header: no pointer or count ceiling

[thedavidmeister]

Closes #13

Implements the plan in #13 (comment).

#123 is closed by #192, which named the packed layout's literals. This layout names every offset, width and mask as an internal constant the same way.

Layout

• MemoryKV stays uint256 and MEMORY_KV_EMPTY stays 0. No function signature changes.
• 0 is the empty store, which owns no memory. Any other value is the address of a 16-word header. The first insert allocates it at the free memory pointer and zeroes it with calldatacopy.
• Header words 0..14 are the list heads, each a full-width node address. Word 15 is meta: the word count in bits 16..255, bit 15 zero, and an occupancy mask in bits 0..14, where list s sets bit 14 - s.
• Nodes are unchanged: key, value, next.
• Nothing is packed into the handle, so there is no pointer ceiling and no count ceiling. MemoryKVOverflow and MemoryKVLengthOverflow are removed.
• get returns (0, 0) for kv == 0 without reading memory.
• toBytes32Array walks the mask lowest bit first and maps each bit to its list through SLOT_TABLE. That is lists 14 down to 0, each newest first, which is the order the bisect produced. The order stays unspecified. The bisect and its slither cyclomatic-complexity disable are gone.
• Aliasing changes. Copies of a non-empty handle are one store: inserts and updates show through every copy. Only the empty handle branches. The return of set must still be assigned, because the first insert's return is the only thing holding the header.
• The layout is documented once, in the MemoryKV NatSpec, and every offset, width and mask is a named internal constant.
• PAIR_COUNT_INCREMENT = 0x20000 is a literal. Written as shl(COUNT_BIT_OFFSET, 2), it was not folded and every insert paid for the shift.

Tests

• Deleted, because the behaviour they pin is gone: bisect.t.sol, errorAbi.t.sol, setSlotMask.t.sol, wordCountOverflow.t.sol. test/lib/LibMemoryKVCapacity.sol (EMPTY_FRAME_PAIRS, the packed 682-pair ceiling) is deleted with them.
• Added packedParity.t.sol: differential against test/lib/LibMemoryKVPacked.sol, kept as the oracle for export order and get/has. The oracle is main's packed LibMemoryKV at f50c40d, not current main: that version's code with the library and handle type renamed (LibMemoryKVPacked, PackedMemoryKV), the key and value types imported from src, and its own pragma and NatSpec. Main now names the same literal values as constants, so the oracle behaves as main does. The tests pin the exact meta word and export order for every occupancy mask from 0 to OCCUPANCY_MASK; SLOT_TABLE rebuilt from its definition; the header zeroed over dirty memory.
• Added exportOccupancy.t.sol: every list alone, both ends of the mask, adjacent and alternate lists, every occupancy combination, below and above 0x10000, and count bits never read as mask bits.
• test/lib/LibMemoryKVHandle.sol reads and crafts the header layout through src's constants. The rest of the tree moves to the new layout. types.t.sol pins how the header constants tile the header and that each head points at its list's node. storageParity.t.sol checks every fuzzed pair, with no per-store cap.
• setCapacity.t.sol: testTwoThousandPairsFromAboveSixteenBits puts the free pointer at 0x10000 and inserts 2000 distinct keys; every key reads back, and the count, the export and the allocation are exact. testAStoreStraddlingSixteenBitsReadsBackWhole fuzzes the start so the header and nodes straddle 0x10000.
• gasClaims.t.sol compares paths within one build and pins no gas figure, and the README states gas the same way. One pair exports at the same cost from every list. Each list a store leaves empty saves exactly one list visit. The walk costs less than LibMemoryKVSlow.toBytes32ArrayLinear at every occupancy from 0 to LIST_COUNT lists, and from one occupied list on the saving never grows as lists fill. The first insert into an empty store costs more than an insert into an empty list of a store that has its header. Each key ahead in a list makes an insert into it, and a get of the key furthest from its head, cost more.

QA

• CI on b0a6a20: rainix-sol static, legal and test pass; 27 suites, 160 tests passed, 0 failed (https://github.com/rainlanguage/rain.lib.memkv/actions/runs/35382810381).
• Adversarial review at 3f21e4a, through three lenses: semantics, src and tests (adversarial-semantics.md, adversarial-src.md, adversarial-tests.md in /home/gildlab/artifacts/memkv/issue13/). It found no src defect. Its two findings are fixed here: no test pinned the export at an unaligned free memory pointer, and this body called the packed oracle current main verbatim.
• Test equivalence with main (test-equivalence.md, same directory): of the 78 main test names this branch does not have, 49 are covered by a test here and 26 pin behaviour this layout removes. The other 3, main's wordCountOverflow inserts up to a full count and update at a full count, were a gap, because no test here drove set or the export above 4000 words. Fixed by the word-count tests below.
• Unaligned export (exportAlloc.t.sol): testExportAllocatesAtAnUnalignedFreePointer puts the free memory pointer 1 to 31 bytes past a word, above live bytes. The array is at the pointer and the live bytes are unchanged. testExportAfterEncodePackedLeavesTheBytesAlone runs the export after abi.encodePacked, and the packed bytes and their length are unchanged. Mutant E04 (array := and(mload(0x40), not(0x1f))), run against the whole suite at b0a6a20: killed, by these two tests and no other (baseline 160 passed).
• Word counts wider than sixteen bits (setCapacity.t.sol): writeList crafts a header with a count from 0x10000 up to the widest the field holds. An insert adds exactly 2 and keeps the mask (testInsertAddsOnePairToAWordCountWiderThanSixteenBits). An update leaves the meta word unchanged (testUpdateLeavesAWordCountWiderThanSixteenBitsAlone). The export takes its length and its allocation from the whole count (testExportSizesItsArrayFromAWordCountWiderThanSixteenBits). Two mutants narrow the count to 12 bits, each run against the whole suite at b0a6a20. W01, in set's insert, is killed by the insert test alone. W02, in the export's sizing, is killed by the export test alone. The update test is killed, by hand, by a mutant that sends every set down the insert path (switch iszero(pointer) to switch 1).
• src mutation probe: 63 mutants of src/lib/LibMemoryKV.sol (constants, get, set, the export walk, SLOT_TABLE, has), whole suite, at d8b5358 (the head b0a6a20 only adds tests on top of it; src is unchanged): 63/63 killed, 0 no-run, 0 harness errors. Run with --gas-limit 10000000000, so a mutant that links a list into a cycle (S13: the node allocation leaves the free pointer where it was) fails out of gas instead of timing out as a no-run. C04 (NODE_BYTES 0x60 -> 0x80) is killed by testSetInsertIntoTheEmptyStoreAllocatesAHeaderThenANode, which checks that the first insert's allocation ends after the node's next word.
• Helper mutation probe: 43 mutants, whole suite, at d8b5358 (the head b0a6a20 only adds tests on top of it; src is unchanged): 43/43 killed. B01–B31 cover the readers and crafting functions in test/lib/LibMemoryKVHandle.sol, including writeList's bounds and kept bits; test/lib/LibMemoryKVHandle.t.sol alone kills all 31. L02–L09 (LibMemoryKVSlow.toBytes32ArrayLinear) are killed by testRoundTripLinear in array.t.sol, and L01 (the empty-store guard) by testSlowLinearExportAllocatesWhatItWrites in LibMemoryKVSlow.t.sol. F01–F03 (setAtFreePointerInFrame) are killed by testSetInsertRecordsTheWholeAddress in setInsert.t.sol.
• Gas ordering, one build, the walk against LibMemoryKVSlow.toBytes32ArrayLinear: the walk costs less at every measured size, 0 to 15 occupied lists and 0 to 20 sequential keys. The saving is largest with one list occupied and falls with each list after it. It is smaller for the empty store than for one occupied list, because both exports skip everything there.
• Category check: support larger pointers #13 asks for pointers wider than 16 bits. The handle is a full-width pointer to the header, and the two setCapacity tests named under Tests put a store above and across 0x10000. No test pins an absolute gas figure: no assertApproxEq* or bound against a gas constant, and git grep '~[0-9]' over src and README.md finds nothing.

🤖 Generated with Claude Code

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 8f16a83a-9847-40fc-a615-09289242c9c9

📥 Commits

Reviewing files that changed from the base of the PR and between b33df1d and d8b5358.

📒 Files selected for processing (31)

• README.md
• src/lib/LibMemoryKV.sol
• test/lib/LibMemoryKVCapacity.sol
• test/lib/LibMemoryKVHandle.sol
• test/lib/LibMemoryKVHandle.t.sol
• test/lib/LibMemoryKVKeys.sol
• test/lib/LibMemoryKVPacked.sol
• test/lib/LibMemoryKVSlow.sol
• test/lib/SetAtFreePointer.sol
• test/src/lib/LibMemoryKV.array.t.sol
• test/src/lib/LibMemoryKV.bisect.t.sol
• test/src/lib/LibMemoryKV.crossFunction.t.sol
• test/src/lib/LibMemoryKV.errorAbi.t.sol
• test/src/lib/LibMemoryKV.exportAlloc.t.sol
• test/src/lib/LibMemoryKV.exportOccupancy.t.sol
• test/src/lib/LibMemoryKV.exportWalk.t.sol
• test/src/lib/LibMemoryKV.frameLocal.t.sol
• test/src/lib/LibMemoryKV.gasClaims.t.sol
• test/src/lib/LibMemoryKV.getMatch.t.sol
• test/src/lib/LibMemoryKV.getWalk.t.sol
• test/src/lib/LibMemoryKV.getset.t.sol
• test/src/lib/LibMemoryKV.handleAlias.t.sol
• test/src/lib/LibMemoryKV.packedParity.t.sol
• test/src/lib/LibMemoryKV.saturate.t.sol
• test/src/lib/LibMemoryKV.setCapacity.t.sol
• test/src/lib/LibMemoryKV.setInsert.t.sol
• test/src/lib/LibMemoryKV.setSlotMask.t.sol
• test/src/lib/LibMemoryKV.setUpdate.t.sol
• test/src/lib/LibMemoryKV.storageParity.t.sol
• test/src/lib/LibMemoryKV.types.t.sol
• test/src/lib/LibMemoryKV.wordCountOverflow.t.sol

💤 Files with no reviewable changes (5)

• test/lib/LibMemoryKVCapacity.sol
• test/src/lib/LibMemoryKV.bisect.t.sol
• test/src/lib/LibMemoryKV.wordCountOverflow.t.sol
• test/src/lib/LibMemoryKV.setSlotMask.t.sol
• test/src/lib/LibMemoryKV.errorAbi.t.sol

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

Walkthrough

Changes

MemoryKV header migration

Layer / File(s)	Summary
Header storage and metadata contract src/lib/LibMemoryKV.sol, test/lib/LibMemoryKVHandle.sol, test/src/lib/LibMemoryKV.types.t.sol	MemoryKV now refers to a memory header containing list heads, a count, and an occupancy mask. Accessors and layout tests use the header representation.
Shared-store mutation and handle behavior test/src/lib/LibMemoryKV.setInsert.t.sol, test/src/lib/LibMemoryKV.setUpdate.t.sol, test/src/lib/LibMemoryKV.handleAlias.t.sol, test/src/lib/LibMemoryKV.frameLocal.t.sol, test/src/lib/LibMemoryKV.crossFunction.t.sol	Empty stores allocate and zero headers. Non-empty handle copies share mutations. Inserts and updates use full-width memory addresses.
Occupancy export and reference validation src/lib/LibMemoryKV.sol, test/src/lib/LibMemoryKV.exportOccupancy.t.sol, test/src/lib/LibMemoryKV.exportWalk.t.sol, test/src/lib/LibMemoryKV.gasClaims.t.sol	Exports use the header count and occupancy mask to walk occupied lists. Tests cover allocation, occupancy combinations, boundary addresses, and gas measurements.
Full-width pointer and parity validation test/lib/LibMemoryKVPacked.sol, test/src/lib/LibMemoryKV.packedParity.t.sol, test/src/lib/LibMemoryKV.setCapacity.t.sol	A packed reference implementation preserves the previous bounded layout for parity checks. Capacity tests now verify operation above the sixteen-bit address boundary and remove pointer and count overflow expectations.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to d8b53

No actionable merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Issue #13 requires larger pointer capacity than the former 16-bit packed handle. The PR changes MemoryKV to store a full-width header address. The header stores full-width list heads, metadata store…
Out of Scope Changes check	✅ Passed	The changes stay within Issue #13. The README updates document the new header representation. The helper libraries, packed oracle, parity tests, allocation tests, export tests, alias tests, and layout…
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: MemoryKV now points to a full-width memory header without pointer or count ceilings.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}