Decoding size limits for DoS protection

[SeanTAllen]

The problem

The decoder has no limits on the size of data it will attempt to allocate when decoding variable-length values. An attacker who controls the input can craft a msgpack message with a str_32 header declaring 4 GB of string data, a map_32 header declaring 2^32 key-value pairs, or similar. The library will either attempt to allocate the declared size (for str/bin/ext) or return a count that naive callers will use to size a collection (for array/map).

This is not theoretical. msgpack-python added size limits as a security improvement after recognizing the risk, and now ships with conservative defaults (1 MB for strings, binary, and ext; 131,072 for arrays and maps). msgpack-javascript, msgpack-cxx, rmp-serde (Rust), and MessagePack-CSharp all provide configurable limits. The Pony library currently has none.

Specific attack vectors

Memory exhaustion via str/bin/ext: A str_32, bin_32, or ext_32 header can declare up to 2^32 - 1 bytes (4 GB) of payload. The decoder calls Reader.block(len) to extract that many bytes from the reader's internal buffer. If the attacker sends a large payload and the application buffers it (via append() on the streaming decoder, or by passing a pre-filled Reader to the low-level decoder), the decode proceeds and returns the full payload. MessagePackStreamingDecoder's existing size check only protects against reader corruption from partial reads — it confirms the data is present before consuming, but does nothing to prevent processing an oversized value.

Collection pre-allocation via array/map counts: The library returns array and map element counts to the caller. A map_32 header can claim 2^32 pairs. Callers who pre-allocate a Map or Array sized to this count will attempt a multi-billion-element allocation. The library doesn't allocate the collection itself, but it returns the unchecked count that drives the allocation.

Nested container amplification: A sequence of array headers each claiming large counts creates multiplicative depth. While the library doesn't track nesting, callers building recursive decoders on top of it are vulnerable to stack overflow or exponential work.

How other libraries handle this

msgpack-python

The Unpacker class accepts per-type limits as constructor arguments:

• max_str_len (default: 1,048,576 — 1 MB)
• max_bin_len (default: 1,048,576)
• max_array_len (default: 131,072)
• max_map_len (default: 131,072)
• max_ext_len (default: 1,048,576)
• max_buffer_size (default: 100 MB)

When a declared size exceeds the limit, the unpacker raises ValueError. Older versions had no limits and a max_buffer_size of 0 (unlimited); the security-conscious defaults were added in a later release.

msgpack-javascript (@msgpack/msgpack)

The Decoder constructor accepts an options object with:

• maxStrLength (default: 4_294_967_295 — effectively unlimited)
• maxBinLength, maxArrayLength, maxMapLength, maxExtLength (same default)

Exceeding a limit throws DecodeError. The defaults are permissive; security-conscious callers must opt in.

msgpack-cxx

The unpack() function accepts an unpack_limit object with array, map, str, bin, ext, and depth fields. All default to 0xFFFFFFFF except depth (512). Exceeding a limit throws msgpack::array_size_overflow or similar typed exceptions.

rmp-serde (Rust)

The deserializer supports a depth limit via Deserializer::set_max_depth(). Exceeding it returns DepthLimitExceeded. Per-type size limits are not provided at the rmp layer; Rust's serde ecosystem typically handles this at the type level.

Design

Which APIs get limits

MessagePackStreamingDecoder is the natural home. It's already stateful (a class, not a primitive), it's the recommended API for untrusted data (per the issue #14 warning), and it already peeks at declared sizes before consuming data — the limit check slots in between the peek and the consume.

MessagePackDecoder is a stateless primitive. Adding limits would require either method parameters (cluttering every call site) or wrapping it in a stateful class (which is what the streaming decoder already is). The recommendation should be: for untrusted data, use MessagePackStreamingDecoder. The low-level MessagePackDecoder is for trusted data or callers who've already validated sizes externally.

Limits configuration

Two options:

Option A: Constructor parameters

class MessagePackStreamingDecoder
  new create(
    max_str_len: USize = USize.max_value(),
    max_bin_len: USize = USize.max_value(),
    max_array_len: U32 = U32.max_value(),
    max_map_len: U32 = U32.max_value(),
    max_ext_len: USize = USize.max_value()
  ) =>

Simple, direct, minimal API surface. Adding a new limit later changes the constructor signature but doesn't break existing callers (new parameters get defaults).

Option B: Separate configuration class

class val MessagePackDecodeLimits
  let max_str_len: USize
  let max_bin_len: USize
  let max_array_len: U32
  let max_map_len: U32
  let max_ext_len: USize

  new val create(
    max_str_len': USize = USize.max_value(),
    max_bin_len': USize = USize.max_value(),
    max_array_len': U32 = U32.max_value(),
    max_map_len': U32 = U32.max_value(),
    max_ext_len': USize = USize.max_value()
  ) =>
    max_str_len = max_str_len'
    max_bin_len = max_bin_len'
    max_array_len = max_array_len'
    max_map_len = max_map_len'
    max_ext_len = max_ext_len'

  new val unlimited() =>
    max_str_len = USize.max_value()
    max_bin_len = USize.max_value()
    max_array_len = U32.max_value()
    max_map_len = U32.max_value()
    max_ext_len = USize.max_value()

class MessagePackStreamingDecoder
  new create(
    limits: MessagePackDecodeLimits =
      MessagePackDecodeLimits.unlimited()
  ) =>

More extensible — the limits object is reusable, the unlimited() factory communicates intent, and future limits (depth, buffer size) can be added to the class without changing MessagePackStreamingDecoder's constructor. If limits are later added to MessagePackDecoder (via a wrapper), the same type can be shared.

Default values

Two schools of thought, represented by the libraries surveyed:

Permissive defaults (msgpack-javascript, msgpack-cxx): All limits set to max values, effectively off. No behavioral change for existing users. Secure callers must opt in. This is the "no surprises" approach — existing code keeps working identically.

Conservative defaults (msgpack-python): Reasonable limits like 1 MB for payloads, 131K for container counts. Security by default. Existing code that legitimately decodes large values would need to raise limits explicitly.

For a beta library, permissive defaults avoid breaking existing users while making the capability available. The documentation and README should prominently recommend setting limits for untrusted data.

Result type for limit violations

The streaming decoder currently returns DecodeResult, which is (MessagePackValue | NotEnoughData | InvalidData). A limit violation is semantically distinct from both existing error cases:

• It's not NotEnoughData — the data may be fully present.
• It's not InvalidData — the data is valid msgpack; it just exceeds policy.

The caller's response is also distinct: log the violation and close the connection, rather than "wait for more data" or "stream is corrupt."

Option A: New LimitExceeded primitive

primitive LimitExceeded

type DecodeResult is
  ( MessagePackValue
  | NotEnoughData
  | InvalidData
  | LimitExceeded )

Minimal, consistent with NotEnoughData and InvalidData being primitives. The caller knows a limit was hit but not which one. Matches the pattern established by the existing result variants.

Option B: LimitExceeded class with details

class val LimitExceeded
  let limit_name: String val
  let declared_size: USize
  let max_size: USize

  new val create(
    limit_name': String val,
    declared_size': USize,
    max_size': USize
  ) =>
    limit_name = limit_name'
    declared_size = declared_size'
    max_size = max_size'

type DecodeResult is
  ( MessagePackValue
  | NotEnoughData
  | InvalidData
  | LimitExceeded )

Richer — the caller gets a diagnostic message without guessing. The result type has the same shape from the pattern-matching perspective (match on LimitExceeded), but the value carries context. This follows the precedent of MessagePackArray, MessagePackMap, MessagePackExt, and MessagePackTimestamp being classes in the result union rather than primitives.

Where the checks go

In MessagePackStreamingDecoder, the limit check slots into each variable-length decode method between the size peek and the delegation to MessagePackDecoder:

peek length from header
  -> check reader.size() >= required  (existing)
  -> check length <= configured limit  (new)
  -> delegate to MessagePackDecoder

Methods affected:

Method	Limit checked
_decode_fixstr	max_str_len
_decode_str	max_str_len
_decode_bin	max_bin_len
_decode_fixext	max_ext_len (always passes — max 16 bytes)
_decode_ext_variable	max_ext_len
_decode_fixarray	max_array_len
_decode_array	max_array_len
_decode_fixmap	max_map_len
_decode_map	max_map_len

The fixext methods (fixext_1 through fixext_16) have fixed data sizes (1, 2, 4, 8, or 16 bytes). Checking them against max_ext_len is technically correct but will never trigger for any reasonable limit. Including the check anyway provides defense-in-depth.

_decode_timestamp is not listed because it is always reached via the _decode_fixext or _decode_ext_variable paths, which check max_ext_len before the timestamp guard runs. The timestamp data sizes are fixed and small (4, 8, or 12 bytes), so no separate limit is needed.

Byte consumption on limit violation

When a limit is exceeded, the decoder must NOT consume the bytes. This is consistent with NotEnoughData behavior — the reader remains at the same position. The caller can't meaningfully retry (unlike NotEnoughData), but not consuming keeps the reader state clean for diagnostics or for callers who want to skip the offending value with a future skip() method.

Depth limits

Container depth tracking is a related but separate concern. The streaming decoder currently has no nesting awareness — it returns array/map headers and the caller manages nesting. Adding a depth limit would require the decoder to track an internal nesting counter, incremented when returning MessagePackArray/MessagePackMap and decremented... when? The decoder doesn't know when the caller has finished reading a container's elements.

Two approaches for depth limits:

1. Caller-side: Document a recommended pattern where the caller maintains a depth counter and checks it before recursing. No library changes needed.
2. Decoder-side: Add begin_container()/end_container() protocol methods that the caller must call, allowing the decoder to track depth. This changes the usage pattern significantly.

Depth limits are not addressed in this proposal. They're worth considering separately once the basic size limits are in place.

Scope

This proposal covers:

• A limits configuration mechanism for MessagePackStreamingDecoder
• Size limit checks for str, bin, ext, array, and map decoding
• A new result type variant for limit violations
• Documentation updates

This proposal does not cover:

• Limits for MessagePackDecoder (low-level API — use streaming decoder for untrusted data)
• Container depth limits (requires nesting awareness the decoder doesn't have)
• Total buffer size limits (the caller controls what they feed to append())

[SeanTAllen]

I prefer option B with conservative defaults.

And for sizing violations, I think starting with the primitive makes the most sense.

[SeanTAllen]

most of this has been addressed in #71

[SeanTAllen]

#72 addresses the rest of this.