Object serialization API — what other msgpack libraries provide and what it could look like in Pony

[SeanTAllen]

Current state

The library currently provides two levels of API:

1. Format-specific — uint_8, uint_32, fixstr, str_8, fixarray, array_16, etc. The caller chooses the exact wire format.
2. Compact / format-family (added in the compact encoding PR) — uint, int, str, array, map, ext, timestamp. These auto-select the smallest wire format within a format family, but the caller still thinks in terms of msgpack types.

Most other msgpack libraries provide a third level that we don't have yet.

What "high-level API" means in other libraries

The common pattern across languages: one call to serialize a native object into bytes, one call to deserialize bytes back. The user never thinks about msgpack format codes, array headers, or string length prefixes. The library maps language-native types to msgpack format families automatically.

Python (msgpack-python)

import msgpack

data = {
    "name": "Alice",
    "age": 30,
    "scores": [95, 87, 92]
}

# Serialize — one call, any nested structure
packed = msgpack.packb(data)

# Deserialize — one call, reconstructs the structure
unpacked = msgpack.unpackb(packed, raw=False)
# => {'name': 'Alice', 'age': 30, 'scores': [95, 87, 92]}

Python's dynamic typing makes this straightforward: dicts become maps, lists become arrays, strings/ints/floats/bools/None map directly.

Go (vmihailenco/msgpack)

import "github.com/vmihailenco/msgpack/v5"

type Person struct {
    Name   string `msgpack:"name"`
    Age    int    `msgpack:"age"`
    Scores []int  `msgpack:"scores"`
}

p := Person{Name: "Alice", Age: 30, Scores: []int{95, 87, 92}}

// Serialize
data, _ := msgpack.Marshal(p)

// Deserialize
var result Person
_ = msgpack.Unmarshal(data, &result)

Go uses runtime reflection plus struct tags to map struct fields to map keys. The Marshal/Unmarshal interface mirrors encoding/json.

Rust (rmp-serde)

use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, Debug)]
struct Person {
    name: String,
    age: u32,
    scores: Vec<u32>,
}

let p = Person {
    name: "Alice".to_string(),
    age: 30,
    scores: vec![95, 87, 92],
};

// Serialize
let data = rmp_serde::to_vec(&p).unwrap();

// Deserialize
let result: Person = rmp_serde::from_slice(&data).unwrap();

Rust uses serde's derive macros to generate serialization code at compile time — no runtime reflection needed. Any type that implements Serialize/Deserialize works automatically.

What this would look like in Pony

Pony has no runtime reflection. The two viable approaches are:

Approach 1: Trait-based

Define a trait that types implement to describe their serialization:

trait MessagePackSerializable
  fun encode_msgpack(w: Writer) ?
  // Deserialization is harder — would need a factory or
  // static-like construction pattern

trait MessagePackDeserializable
  new from_msgpack(r: Reader ref) ?

Usage would look something like:

class Person is (MessagePackSerializable & MessagePackDeserializable)
  let name: String
  let age: U64
  let scores: Array[U64] val

  new create(name': String, age': U64, scores': Array[U64] val) =>
    name = name'
    age = age'
    scores = scores'

  fun encode_msgpack(w: Writer) ? =>
    MessagePackEncoder.map(w, 3)
    MessagePackEncoder.str(w, "name")?
    MessagePackEncoder.str(w, name)?
    MessagePackEncoder.str(w, "age")?
    MessagePackEncoder.uint(w, age)
    MessagePackEncoder.str(w, "scores")?
    MessagePackEncoder.array(w, scores.size().u32())
    for s in scores.values() do
      MessagePackEncoder.uint(w, s)
    end

  new from_msgpack(r: Reader ref) ? =>
    let count = MessagePackDecoder.map(r)?
    var name': String = ""
    var age': U64 = 0
    var scores': Array[U64] iso = recover iso Array[U64] end
    var i: U32 = 0
    while i < count do
      match MessagePackDecoder.str(r)?
      | "name" => name' = MessagePackDecoder.str(r)?
      | "age" => age' = MessagePackDecoder.uint(r)?
      | "scores" =>
        let n = MessagePackDecoder.array(r)?
        var j: U32 = 0
        while j < n do
          scores'.push(MessagePackDecoder.uint(r)?)
          j = j + 1
        end
      end
      i = i + 1
    end
    name = name'
    age = age'
    scores = consume scores'

This is verbose. Each type manually implements its serialization. It's roughly equivalent to manually implementing serde::Serialize in Rust without the derive macro — nobody does that by choice.

Approach 2: Dynamic value type

Instead of serializing user types directly, provide a recursive value type that mirrors the msgpack data model:

type MsgPackValue is
  ( None
  | Bool
  | I64
  | U64
  | F64
  | String val
  | Array[MsgPackValue] val
  | Map[String val, MsgPackValue] val
  // possibly ext, timestamp, binary
  )

With pack/unpack functions:

// Serialize any MsgPackValue tree
MessagePackEncoder.pack(w, value)?

// Deserialize into a MsgPackValue tree
let value = MessagePackDecoder.unpack(r)?

Users would build and destructure MsgPackValue trees. Simpler to implement than trait-based serialization, but the user has to convert between their types and MsgPackValue manually.

Discussion points

1. Is there demand? The compact methods cover the "don't make me think about format sizes" use case. Is "don't make me think about msgpack types at all" a real need for Pony users, or is the current level of abstraction sufficient?

2. Trait-based vs. value type? The trait approach is more type-safe but very verbose without macros. The value type approach is simpler to implement and use, but loses static typing.

3. Reference capabilities complicate deserialization. Constructing complex Pony objects from a byte stream requires careful capability management (iso, val, ref). Serialization is easier (everything is readable). This asymmetry makes a general-purpose deserialization API harder to design in Pony than in most languages.

4. Scope. A full object serialization layer is a significant undertaking. It could be a separate package that depends on this one, rather than being built into the core library.