Implementation plan: format-specific bin/ext/str decoding (#13)

[SeanTAllen]

Implementation plan for #13: adding format-specific decode methods for the str, bin, and ext format families. The encoder already has all format-specific methods; the decoder currently only has compact (auto-dispatching) methods for these families.

Context

MessagePackDecoder provides two styles of methods:

• Compact methods peek at the format byte and accept any wire format within a family. Convenient but slower due to the if chain. Examples: uint(), int(), str(), byte_array(), ext().
• Format-specific methods validate a single expected format byte, then read the data directly. Faster when the caller knows the exact wire format. Examples: u8(), u16(), fixstr(), array_16().

The decoder is missing format-specific methods for three families: str (except fixstr which exists), bin, and ext. The encoder has all of these already.

Methods to Add

14 methods on MessagePackDecoder:

str family (3 methods)

Method	Format byte	Wire format	Return type
str_8(b: Reader): String iso^ ?	0xD9	1 type + 1 len + data	String iso^
str_16(b: Reader): String iso^ ?	0xDA	1 type + 2 len + data	String iso^
str_32(b: Reader): String iso^ ?	0xDB	1 type + 4 len + data	String iso^

bin family (3 methods)

Method	Format byte	Wire format	Return type
bin_8(b: Reader): Array[U8] iso^ ?	0xC4	1 type + 1 len + data	Array[U8] iso^
bin_16(b: Reader): Array[U8] iso^ ?	0xC5	1 type + 2 len + data	Array[U8] iso^
bin_32(b: Reader): Array[U8] iso^ ?	0xC6	1 type + 4 len + data	Array[U8] iso^

fixext family (5 methods)

Method	Format byte	Wire format	Return type
fixext_1(b: Reader): (U8, Array[U8] val) ?	0xD4	1 type + 1 ext_type + 1 data	(U8, Array[U8] val)
fixext_2(b: Reader): (U8, Array[U8] val) ?	0xD5	1 type + 1 ext_type + 2 data	(U8, Array[U8] val)
fixext_4(b: Reader): (U8, Array[U8] val) ?	0xD6	1 type + 1 ext_type + 4 data	(U8, Array[U8] val)
fixext_8(b: Reader): (U8, Array[U8] val) ?	0xD7	1 type + 1 ext_type + 8 data	(U8, Array[U8] val)
fixext_16(b: Reader): (U8, Array[U8] val) ?	0xD8	1 type + 1 ext_type + 16 data	(U8, Array[U8] val)

ext family (3 methods)

Method	Format byte	Wire format	Return type
ext_8(b: Reader): (U8, Array[U8] val) ?	0xC7	1 type + 1 len + 1 ext_type + data	(U8, Array[U8] val)
ext_16(b: Reader): (U8, Array[U8] val) ?	0xC8	1 type + 2 len + 1 ext_type + data	(U8, Array[U8] val)
ext_32(b: Reader): (U8, Array[U8] val) ?	0xC9	1 type + 4 len + 1 ext_type + data	(U8, Array[U8] val)

Design Decisions

1. All format-specific methods validate the format byte

Each method calls _read_type(b)? and checks it matches the expected format byte, erroring on mismatch. This matches the pattern used by existing format-specific methods like u8(), array_16(), map_32().

The "fix" variants that embed data in the format byte (fixstr, fixarray, fixmap, positive_fixint) skip validation because consuming the format byte is reading the data. The new methods all have separate format bytes and data, so validation applies.

The cost is one comparison — negligible relative to the data read, and it catches misuse (calling str_8 on str_16-encoded data).

2. ext methods return (U8, Array[U8] val) — same as compact ext()

The ext wire format is: [format byte] [length] [ext type byte] [data bytes]. The ext type byte is user-defined metadata identifying what kind of extension this is. It's part of the value, not something to discard or split off.

Returning (ext_type, data) keeps the methods self-contained — one call decodes the complete ext value. This matches the existing compact ext() method, giving a consistent API regardless of which method the caller chooses.

The alternative (returning just the data array) would either lose the ext type byte or require the caller to read it separately, splitting the decode across two calls with no benefit.

3. Compact methods remain independent — no delegation

The compact methods (str(), byte_array(), ext()) cannot delegate to the new format-specific methods. The reason is structural: compact methods call _read_type(b)? to consume and inspect the format byte, then dispatch. But format-specific methods also call _read_type(b)? and validate it. By the time the compact method knows which format-specific method to call, the format byte is already consumed.

This differs from the encoder, where compact methods can delegate because the dispatch is based on value size (which doesn't consume the writer), not on reading a format byte.

Both API surfaces coexist as alternatives: callers who don't know the wire format use compact methods; callers who do use format-specific methods.

4. Streaming decoder updated to use format-specific methods

MessagePackStreamingDecoder already knows the exact wire format before calling into MessagePackDecoder (it peeks the format byte for dispatch and size checking). Currently it calls compact methods, which redundantly re-dispatch on the format byte. After this change, it will call format-specific methods instead, skipping the redundant if chain.

The coupling comments ("Note: MessagePackStreamingDecoder pre-validates total size before calling this method...") move from the compact methods to the format-specific methods, following the principle of documenting coupling at the point of breakage — a future maintainer changing str_8()'s read sequence needs to know the streaming decoder depends on it.

The compact methods' coupling comments are updated or removed since the streaming decoder no longer calls them for str/bin/ext.

5. Parameter style: Reader (implicit ref)

The decoder has an existing inconsistency: integer/compact methods use explicit Reader ref, while str/bin/ext/array/map format-specific methods use bare Reader (which defaults to ref for classes). The new methods go in the str/bin/ext sections, so they match those sections' existing style: Reader.

6. byte_array naming left as-is

The compact bin decoder is called byte_array() while the encoder's equivalent is bin(). This naming asymmetry predates this work. A separate issue should be filed to track renaming byte_array() to bin() for encoder/decoder symmetry. The new format-specific methods are named bin_8/bin_16/bin_32 (matching the encoder and the MessagePack spec), sitting alongside the existing byte_array().

7. Scope boundary — timestamps excluded

During planning we identified that timestamp_32/timestamp_64/timestamp_96 decode methods are also missing (the encoder has them, the decoder only has compact timestamp()). We kicked this out to a separate discussion (#61) because timestamps are a higher-level concept layered on ext wire formats, and the performance argument may differ.

Implementation Steps

Step 1: Add format-specific methods to MessagePackDecoder

File: msgpack/message_pack_decoder.pony

Add the 14 methods in their respective format-family sections:

• str family (after str()): str_8, str_16, str_32
• byte array family (after byte_array()): bin_8, bin_16, bin_32
• ext family (after ext(), before timestamp()): fixext_1 through fixext_16, then ext_8, ext_16, ext_32

Each method follows the same pattern:

fun str_8(b: Reader): String iso^ ? =>
  if _read_type(b)? != _FormatName.str_8() then error end
  let len = b.u8()?.usize()
  String.from_iso_array(b.block(len)?)

fun fixext_4(b: Reader): (U8, Array[U8] val) ? =>
  if _read_type(b)? != _FormatName.fixext_4() then error end
  (b.u8()?, b.block(4)?)

fun ext_16(b: Reader): (U8, Array[U8] val) ? =>
  if _read_type(b)? != _FormatName.ext_16() then error end
  let len = b.u16_be()?.usize()
  (b.u8()?, b.block(len)?)

Add docstrings to each method. Follow the existing docstring style from the encoder's format-specific methods — describe the wire format, note that the format byte is validated, and describe the return value. For ext methods, document the (ext_type, data) tuple. Example:

fun str_8(b: Reader): String iso^ ? =>
  """
  Decodes a MessagePack str 8 value. Validates that the format
  byte is str_8 (0xD9), then reads the 1-byte length prefix and
  data.

  Errors if the format byte does not match or if insufficient
  data is available. For a method that accepts any string wire
  format, use `str()` instead.
  """
  if _read_type(b)? != _FormatName.str_8() then error end
  let len = b.u8()?.usize()
  String.from_iso_array(b.block(len)?)

Step 2: Update coupling comments

In message_pack_decoder.pony:

• Remove the coupling comments on str(), byte_array(), and ext() that reference the streaming decoder. These comments document a coupling that no longer exists after the refactoring — the streaming decoder will call format-specific methods instead. Leaving stale coupling comments is worse than having none.
• Add coupling comments to the new format-specific methods that the streaming decoder calls. Example:

// Note: MessagePackStreamingDecoder pre-validates total size
// before calling this method. If the read sequence changes,
// update the size check in _decode_str.
fun str_8(b: Reader): String iso^ ? =>

Step 3: Refactor streaming decoder

File: msgpack/message_pack_streaming_decoder.pony

Update four methods to call format-specific methods instead of compact ones:

• _decode_str(fb): Replace MessagePackDecoder.str(_reader)? with a dispatch to MessagePackDecoder.str_8/str_16/str_32 based on fb.
• _decode_bin(fb): Replace MessagePackDecoder.byte_array(_reader)? with MessagePackDecoder.bin_8/bin_16/bin_32.
• _decode_fixext(fb): Replace MessagePackDecoder.ext(_reader)? with MessagePackDecoder.fixext_1/2/4/8/16. Note: this method has a timestamp guard that checks for ext_type 0xFF on fixext_4 and fixext_8 and routes to _decode_timestamp() before reaching the ext decode. The timestamp guard stays in place — only the ext decode after it changes.
• _decode_ext_variable(fb): Replace MessagePackDecoder.ext(_reader)? with MessagePackDecoder.ext_8/16/32. Note: same timestamp guard applies here — ext_8 with data_len=12 and ext_type 0xFF routes to _decode_timestamp() before reaching the ext decode. The guard stays in place.

The streaming decoder already dispatches on fb, so this adds a second dispatch within each method. This is intentional — it matches the existing pattern in _decode_uint and _decode_int, which already dispatch to format-specific methods (u8, u16, etc.) rather than calling compact uint().

Step 4: Update MessagePackDecoder docstring

Update the primitive-level docstring on MessagePackDecoder to mention that format-specific methods are now available for all format families (str, bin, ext), not just integers and arrays/maps.

Step 5: Add tests

File: msgpack/_test_decoder.pony

Existing test classes to update — test classes named _TestDecodeStr8, _TestDecodeStr16, _TestDecodeStr32, _TestDecodeBin8, _TestDecodeBin16, _TestDecodeBin32, _TestDecodeFixext1 through _TestDecodeFixext16, _TestDecodeExt8, _TestDecodeExt16, and _TestDecodeExt32 already exist. They currently encode with the format-specific encoder method and decode with the compact decoder method (e.g., MessagePackDecoder.str()). Update these 14 tests to decode with the new format-specific methods instead (e.g., MessagePackDecoder.str_8()). This is a simple method name substitution — the return types are identical (String iso^, Array[U8] iso^, and (U8, Array[U8] val) respectively), so the rest of each test body (assertions, destructuring) remains unchanged. Compact decode coverage for these wire formats is already provided by the _TestDecodeCompact* tests added in PR #60. These existing tests are already registered in _TestDecoder.tests(), so no registration changes needed for them.

Property-based roundtrip tests — one property per sub-family. All property tests must use the format-specific encoder method (e.g., MessagePackEncoder.str_8), not the compact encoder (MessagePackEncoder.str). The compact encoder selects the smallest format, so short strings would be encoded as fixstr, causing MessagePackDecoder.str_8() to error on the mismatched format byte.

• _PropertyStr8Roundtrip: generate strings 0–255 bytes, roundtrip through str_8 encode/decode.
• _PropertyStr16Roundtrip: generate strings 0–65535 bytes (biased toward smaller for speed).
• _PropertyBin8Roundtrip, _PropertyBin16Roundtrip: same pattern with Array[U8].
• _PropertyFixext1Roundtrip through _PropertyFixext16Roundtrip: generate ext_type (U8, filtered to exclude 0xFF to avoid timestamp interpretation) and data of exact fixed size.
• _PropertyExt8Roundtrip, _PropertyExt16Roundtrip: generate ext_type (U8, filtered to exclude 0xFF) and variable-length data.

For str_32, bin_32, and ext_32, property tests with realistic data sizes are impractical. Example-based roundtrip tests with small payloads are sufficient.

Rejection tests — verify each method errors on wrong format bytes. Every new method should appear in at least one rejection test:

• str_8 should error when given str_16 data, and vice versa.
• bin_8 should error when given bin_16 data.
• fixext_1 should error when given fixext_2 data.
• ext_8 should error when given ext_16 data.

These can be grouped into a small number of test classes that exercise cross-format rejection systematically.

Test registration — all new test classes (property-based and rejection) must be registered in _TestDecoder.tests(). Property tests use the test(Property1UnitTest[T](...)) pattern (see existing registrations at the top of _test_decoder.pony for examples). Rejection tests use the standard test(...) pattern.

Streaming decoder — existing tests should pass without modification (internal refactoring, same external behavior). No new streaming decoder tests needed unless the refactoring changes observable behavior, which it should not.

Build and run tests with: make test

Step 6: Update documentation

README.md: The existing line "Format-specific methods (uint_8, uint_32, fixstr, str_8, etc.) are available when you need explicit control over the wire format." is already accurate once the methods exist. Consider expanding with a short example showing format-specific decode usage — this helps users understand when and why they'd choose format-specific over compact methods. For example:

// When you know the exact wire format, use format-specific
// methods to skip the format-byte dispatch:
MessagePackEncoder.str_8(w, "hello")?
// ...
MessagePackDecoder.str_8(r)?  // "hello"

CLAUDE.md: Update the Architecture section's MessagePackDecoder description to reflect that format-specific methods are now available for str, bin, and ext families.

Step 7: Add release notes

Create a file in .release-notes/ (e.g., format-specific-decode.md):

## Add format-specific str, bin, and ext decoding methods

`MessagePackDecoder` now provides format-specific decoding methods for the
str, bin, and ext format families: `str_8`, `str_16`, `str_32`, `bin_8`,
`bin_16`, `bin_32`, `fixext_1`, `fixext_2`, `fixext_4`, `fixext_8`,
`fixext_16`, `ext_8`, `ext_16`, and `ext_32`.

These methods decode a single wire format directly, skipping the
format-byte dispatch used by the compact methods (`str()`,
`byte_array()`, `ext()`). Use them when you know the exact wire format
and want to avoid the overhead of format detection.

```pony
// Compact — accepts any string wire format:
MessagePackDecoder.str(reader)?

// Format-specific — decodes str_8 only, errors on other formats:
MessagePackDecoder.str_8(reader)?

## Consistency Checklist

Per the "consistency across repetitive structure" principle — the 14 methods follow the same pattern, but quality can taper across variants. During implementation and review, verify:

- Every method validates its format byte (don't skip for "obvious" ones)
- Every method has a docstring
- Coupling comments are present on every method the streaming decoder calls
- Every method has at least a roundtrip test
- Property-based tests cover all sub-families where practical
- Rejection tests cover representative cross-format cases
- Every new method appears in at least one rejection test
- Ext property test generators exclude ext_type `0xFF` (timestamp marker)

[SeanTAllen]

Implemented in #63