Skip: advancing past values without decoding

[SeanTAllen]

The problem

When decoding a msgpack map, the caller reads the header (element count), then must read every key-value pair sequentially to advance the reader to the correct position for whatever follows the map. If the caller encounters a key it doesn't recognize — because the sender is running a newer protocol version, or because the field is irrelevant to this consumer — it still must decode the value to advance past it. But it can't call the right decode method without knowing the value's type.

The caller is stuck: they can't decode the value (unknown type), they can't skip it (no such method exists), and they can't stop reading (the map header committed them to N pairs). The only options are to error out or to use the streaming decoder's next() method to decode into a MessagePackValue union — which works but allocates memory for data that will be immediately discarded.

This problem isn't limited to maps. An array might contain elements of mixed types where the caller only cares about some positions. Or a protocol might prepend metadata headers that some consumers should ignore.

Forward compatibility

Skip is the foundation for forward-compatible protocols. Without it, adding a field to a msgpack map is a breaking change — every consumer must be updated simultaneously to decode the new field's type. With skip, consumers ignore fields they don't understand and continue processing the rest of the message. This is the same principle that makes JSON-based protocols tolerant of new fields.

Example

A versioned protocol sends maps with known fields. A newer sender adds a priority field:

// Without skip — breaks on unknown fields:
let count = MessagePackDecoder.map(reader)?
var i: U32 = 0
while i < count do
  let key = MessagePackDecoder.str(reader)?
  match consume key
  | "action" => action = MessagePackDecoder.str(reader)?
  | "name" => name = MessagePackDecoder.str(reader)?
  else
    // Unknown key — what type is the value?
    // Can't decode it. Can't skip it. Stuck.
    error
  end
  i = i + 1
end

// With skip — handles unknown fields gracefully:
let count = MessagePackDecoder.map(reader)?
var i: U32 = 0
while i < count do
  let key = MessagePackDecoder.str(reader)?
  match consume key
  | "action" => action = MessagePackDecoder.str(reader)?
  | "name" => name = MessagePackDecoder.str(reader)?
  else
    MessagePackDecoder.skip(reader)?  // advance past unknown value
  end
  i = i + 1
end

How other libraries handle this

vmihailenco/msgpack (Go)

Decoder.Skip() is a single method call that handles any value type, including nested containers. It reads the format byte, determines the type, and either advances past fixed-size data or recursively skips container elements. Returns an error if the data is malformed.

MessagePack-CSharp

MessagePackReader.Skip() advances past one complete value. For containers, it recursively skips all contained elements. The reader is a struct, so skip operates on the same position-tracking state as normal reads.

msgpack-python

Unpacker.skip() skips the next value in the stream. It internally reads and discards the value. Some implementations optimize by not constructing intermediate objects.

msgpack-java (Zeebe variant)

MsgPackReader.skipValue() reads the format byte, determines the value's byte count (recursively for containers), and advances the reader position.

Common pattern

All implementations provide a single call that handles every format type including nested containers. The recursion is internal — callers never need to handle container skipping themselves. The format byte determines the skip strategy:

• Fixed-size scalars (nil, bool, ints, floats, fixint): known byte count from format byte alone.
• Variable-length flat values (str, bin, ext): read the length prefix, advance past header + payload.
• Fixed-size ext (fixext 1/2/4/8/16): known byte count from format byte alone.
• Containers (array, map): read the element count, then recursively skip that many elements (or count * 2 for maps).

Design

Byte counts by format

For skip to work, it must know the total byte count for every format. The msgpack spec makes this deterministic:

Format range	Type	Total bytes
0x00-0x7F	positive fixint	1
0x80-0x8F	fixmap	1 + recursive skip of (fb & 0x0F) * 2 values
0x90-0x9F	fixarray	1 + recursive skip of (fb & 0x0F) values
0xA0-0xBF	fixstr	1 + (fb & 0x1F)
0xC0	nil	1
0xC1	(invalid)	error
0xC2-0xC3	bool	1
0xC4	bin 8	2 + len (1-byte length prefix)
0xC5	bin 16	3 + len (2-byte length prefix)
0xC6	bin 32	5 + len (4-byte length prefix)
0xC7	ext 8	3 + len (1-byte length prefix + 1 ext type)
0xC8	ext 16	4 + len (2-byte length prefix + 1 ext type)
0xC9	ext 32	6 + len (4-byte length prefix + 1 ext type)
0xCA	float 32	5
0xCB	float 64	9
0xCC	uint 8	2
0xCD	uint 16	3
0xCE	uint 32	5
0xCF	uint 64	9
0xD0	int 8	2
0xD1	int 16	3
0xD2	int 32	5
0xD3	int 64	9
0xD4	fixext 1	3
0xD5	fixext 2	4
0xD6	fixext 4	6
0xD7	fixext 8	10
0xD8	fixext 16	18
0xD9	str 8	2 + len
0xDA	str 16	3 + len
0xDB	str 32	5 + len
0xDC	array 16	3 + recursive skip of count values
0xDD	array 32	5 + recursive skip of count values
0xDE	map 16	3 + recursive skip of count * 2 values
0xDF	map 32	5 + recursive skip of count * 2 values
0xE0-0xFF	negative fixint	1

For flat values, the total byte count is deterministic from the header alone. For containers, the total byte count requires examining every contained element — msgpack container headers store element counts, not byte counts.

Timestamps are transparently handled because they are wire-encoded as ext formats (timestamp_32 is fixext_4, timestamp_64 is fixext_8, timestamp_96 is ext_8). Skip operates at the wire format level and does not need to interpret ext semantics.

The container recursion challenge

Containers make skip fundamentally different from other decode operations. Skipping a flat value (string, binary, integer) is O(1) — peek at the header, compute the byte count, advance. Skipping a container is O(n) where n is the total number of values in the container tree, because each element's format byte must be examined to determine its size.

This is unavoidable — the msgpack format does not store total byte counts for containers. Every skip implementation in every library does this recursive walk.

Which APIs get skip

MessagePackDecoder: A skip method on the low-level decoder is the natural primitive. It takes a Reader ref and advances past one complete value, recursing for containers. Uses ? for errors (same as all other decoder methods). This is the building block.

MessagePackStreamingDecoder: A skip method on the streaming decoder provides the streaming-safe variant. It must confirm all bytes are available before consuming any, consistent with the decoder's contract that NotEnoughData means zero bytes consumed.

Streaming decoder: the all-or-nothing problem

The streaming decoder's contract is: either decode completely, or consume nothing and return NotEnoughData. For flat values, this is easy — peek at the header, check reader.size(), proceed or return. For containers, the total byte count isn't known until every element has been examined.

Two approaches:

Approach A: Peek-based pre-scan

Calculate the total byte count using only peek operations (no consuming), then check if enough data is available, then advance past all bytes at once using Reader.skip(n) (which advances the cursor without allocating).

The reader supports peek_u8(offset), peek_u16_be(offset), and peek_u32_be(offset) at arbitrary offsets. A recursive function walks the structure using peeks to compute the total byte count:

fun _skip_size(b: Reader ref, offset: USize): USize ? =>
  let fb = b.peek_u8(offset)?
  if fb <= 0x7F then
    1  // positive fixint
  elseif fb <= 0x8F then
    // fixmap: header + (count * 2) recursive values
    let count = (fb and 0x0F).usize()
    var size: USize = 1
    var i: USize = 0
    while i < (count * 2) do
      size = size + _skip_size(b, offset + size)?
      i = i + 1
    end
    size
  elseif fb <= 0x9F then
    // fixarray: header + count recursive values
    let count = (fb and 0x0F).usize()
    var size: USize = 1
    var i: USize = 0
    while i < count do
      size = size + _skip_size(b, offset + size)?
      i = i + 1
    end
    size
  elseif fb <= 0xBF then
    1 + (fb and 0x1F).usize()  // fixstr
  // ... remaining formats follow the byte count table ...
  else
    error  // invalid format byte
  end

The streaming decoder calls _skip_size(reader, 0)? to get the total byte count. If any peek fails (data not yet available), the function errors and the streaming decoder returns NotEnoughData. On success, it advances past the bytes with reader.skip(total_size)?.

This approach maintains the all-or-nothing contract cleanly. The _skip_size function is a pure calculation (peek-only, no consumption), while skip is the consumer. The same _skip_size helper could be reused by both MessagePackDecoder and MessagePackStreamingDecoder.

The cost is that the peek pass walks every element in the skipped value. For flat values this is negligible (one peek for the format byte, possibly one more for the length prefix). For large nested containers, the walk visits every contained element. This is unavoidable given the msgpack format.

Approach B: Delegate to MessagePackDecoder.skip

The streaming decoder already confirms data availability before delegating to MessagePackDecoder for all other operations. For flat values, this is straightforward — peek at the header to compute the byte count, check size, delegate.

For containers, the streaming decoder could delegate the recursive skip to MessagePackDecoder.skip(reader)?. But this breaks the all-or-nothing contract: if MessagePackDecoder.skip succeeds for the first 50 elements of an array but fails on element 51 (not enough data), 50 elements have already been consumed. The reader is now in a corrupted state — the same problem the streaming decoder was designed to prevent.

This approach only works for flat values. For containers, it would need to fall back to Approach A's pre-scan, making it a hybrid that's more complex without clear benefit.

Interaction with size limits (discussion #64)

If decoding size limits are implemented, skip needs a policy for how it interacts with them:

Option 1: Skip respects limits. If a value's declared size exceeds the configured limit, skip returns LimitExceeded instead of advancing. This prevents an attacker from using skip as a vector to force the decoder to walk past huge payloads. But it partially defeats skip's purpose — if a forward-compatible decoder skips unknown fields, a large unknown field would halt processing even though the data is being discarded.

Option 2: Skip ignores limits. Skip advances past any value regardless of size limits. The rationale is that skip doesn't allocate or return the data — it just advances the reader. The memory was already consumed at the buffering layer. Limits exist to prevent the caller from receiving oversized values, which skip doesn't do.

Option 3: Skip has its own limits. A separate max_skip_len or max_skip_depth limit controls how much skip is willing to traverse. This provides DoS protection specific to skip's threat model (recursive container amplification, walking past huge flat values) without coupling it to the decode limits.

The choice depends on the threat model. For flat values, skipping 4 GB of string data means the reader must have 4 GB buffered — the memory cost is the same whether you decode or skip. For containers, the threat is CPU cost: recursively walking a deeply nested or very wide container structure. Container depth limits (noted as future work in discussion #64) are relevant here.

Return type

For MessagePackDecoder:

fun skip(b: Reader ref) ? =>

Partial, consistent with all other decoder methods. Errors if insufficient data or invalid format byte.

For MessagePackStreamingDecoder, a dedicated return type avoids confusion with DecodeResult (which includes MessagePackValue variants that don't apply to skip):

type SkipResult is (None | NotEnoughData | InvalidData)

None means the value was successfully skipped. This is unambiguous because SkipResult is a separate type from DecodeResult — there's no confusion with "decoded a nil value."

If size limits apply to skip, add LimitExceeded:

type SkipResult is (None | NotEnoughData | InvalidData | LimitExceeded)

Scope

This proposal covers:

• A skip method on MessagePackDecoder for the non-streaming case
• A skip method on MessagePackStreamingDecoder for the streaming-safe case
• The design for handling containers recursively
• Interaction with the size limits proposed in discussion Decoding size limits for DoS protection #64

This proposal does not cover:

• Higher-level selective decoding APIs (e.g., "decode only these map keys")
• Seek/random-access patterns (msgpack is a sequential format)

[SeanTAllen]

Container depth limits have been implemented in PR #72. The streaming decoder now tracks nesting depth internally and enforces a configurable max_depth (default 512) via MessagePackDecodeLimits. This is relevant to skip's design — the "Interaction with size limits" section's Option 3 (separate max_skip_depth) can now build on the existing depth tracking infrastructure rather than implementing its own.

[SeanTAllen]

Review findings

Overall

The problem statement is well-motivated, the byte count table is accurate against the msgpack spec, and the two streaming approaches are analyzed correctly — Approach A (peek-based pre-scan) is the right choice, and Approach B is correctly rejected because partial consumption on failure violates the all-or-nothing contract.

The SkipResult return type is well-designed — separating it from DecodeResult avoids any confusion with "decoded a nil value" vs "skipped successfully."

Five findings, one significant.

1. Stack overflow in recursive _skip_size (significant)

The proposed _skip_size is recursive — each container element triggers a recursive call. A crafted payload with a 10,000-deep nested empty array causes 10,000 stack frames. Pony doesn't guarantee tail-call optimization, and this isn't tail-recursive anyway.

An iterative approach eliminates this entirely. Instead of recursion, maintain a counter of values remaining to examine:

fun _skip_size(b: Reader ref, start: USize): USize ? =>
  var offset = start
  var remaining: USize = 1
  while remaining > 0 do
    remaining = remaining - 1
    let fb = b.peek_u8(offset)?
    if fb <= 0x7F then
      offset = offset + 1
    elseif fb <= 0x8F then
      remaining = remaining + ((fb and 0x0F).usize() * 2)
      offset = offset + 1
    elseif fb <= 0x9F then
      remaining = remaining + (fb and 0x0F).usize()
      offset = offset + 1
    // ... remaining formats per the byte count table
    end
  end
  offset - start

This is flat, has no stack overflow risk, and remaining naturally tracks total work. A max_skip_values limit on the counter would bound CPU cost without needing depth tracking at all.

2. Depth tracking interaction with _stack

The discussion doesn't address how skip() integrates with the streaming decoder's container depth tracking (_stack). When skip is called inside a container, the skipped value counts as one element consumed from the parent. So skip() should call _decrement_parent() and _drain_completed() after success — the same path that _track takes for scalar values. Without this, the parent's remaining-element count won't decrement and the depth stack will desync.

3. Limits interaction — no recommendation

The three options for limits interaction are presented without a recommendation. Given the iterative approach from finding #1, I'd suggest a hybrid:

• Flat-value size limits (max_str_len, etc.): ignore them. Skip doesn't allocate the data. The bytes are already buffered. Respecting these limits defeats skip's forward-compatibility purpose — a legitimate unknown field that happens to be large would halt processing.
• Container work: limit it. Either reuse max_depth (if staying recursive) or add a max_skip_values limit on total elements traversed (if going iterative). This bounds CPU cost from container amplification attacks. A max_skip_values default could be derived from the existing limits — e.g., something proportional to max_array_len * max_depth.

4. 0xC1 handling

The byte count table correctly lists 0xC1 as "error." The pseudocode for _skip_size shows partial ranges and says "remaining formats follow the byte count table." In the actual implementation, 0xC1 must be an explicit error case — it falls between 0xC0 (nil) and 0xC2 (false), so a range-based elseif chain could accidentally include it if the boundaries aren't precise.

5. _skip_size reuse — coupling comment

The discussion says _skip_size could be shared between MessagePackDecoder and MessagePackStreamingDecoder. If _skip_size is shared, its comment should note that both callers depend on it — a change to the peek sequence would affect both. This parallels the existing coupling comments on methods like fixstr and str_8 in the decoder.

[SeanTAllen]

this was added via #73