Preserve MySQL JSON output formatting in deep caches

[floze-the-genius]

Summary

• format replicated MySQL JSON with MySQL-compatible separators and key ordering
• apply the same representation during snapshots and binlog replication
• cover reverse-order, nested, numeric-string, and escaped keys/values
• bump the persistent-state format and regenerate its compatibility fixture because stored JSON key bytes change

Root cause

MySQL JSON values were parsed into serde_json::Value and converted back to DfValue. That reserialization stripped spaces and used serde_json object ordering, so deep-cache responses differed byte-for-byte from MySQL. Applying formatting to only one ingestion path would also break row matching, so both snapshot and binlog conversion now preserve the same canonical representation.

The branch is rebased onto current main ddc6f307. The snapshot-side conflict was adapted to the current mysql_value_to_noria_value helper while preserving its detailed UTF-8 errors and Sensitive(&val) diagnostics. PERSISTENT_STATE_VERSION is bumped from 7 to 8, and serialized-meta.json was regenerated with the repository's official generator.

Fixes #1638

Validation

• cargo fmt --all -- --check — passed
• cargo test -p replicators mysql_json_print --lib — 2 passed
• cargo test -p dataflow-state meta_serialization_backwards_compatibility --lib -- --nocapture — 1 passed
• cargo test -p dataflow-state meta_deserialization_backwards_compatibility --lib -- --nocapture — 1 passed
• cargo clippy -p replicators --lib -- -D warnings — passed
• MySQL integration target compiled successfully with --no-run
• git diff --check — passed

Before the rebase, the exact runtime integration test passed against MySQL 8.0.46 with ROW/FULL binlogging. After the rebase, it was compile-checked but not rerun because no MySQL server is listening on the local integration port 33367; this is the only unverified current-head item.

[CLAassistant]

All committers have signed the CLA.

[altmannmarcelo]

This is a good fix, and the root-cause analysis is exactly right — reserialization through serde_json does lose both MySQL's key order and its spacing, and doing it in only one ingestion path would have been worse than doing it in neither.

I validated the branch against a live MySQL 8.4.8 (binlog_format=ROW, binlog_row_image=FULL). I built a 28-row probe table where each row isolates one dimension of MySQL's canonicalization — key ordering, separators, escaping, scalars, empty containers, nesting depth, duplicate keys, and numbers across every magnitude — then compared MySQL's own bytes against what the replicator stores. It holds up well:

	main	this PR
object/array rows byte-exact vs MySQL	0 of 22	22 of 22
scalar rows correct	2 of 6	6 of 6

A few things I want to call out specifically, because they're the parts that are easy to get wrong and you got them right:

• Key ordering. MySQL sorts by byte length first, then bytewise. Your comparator handles the case that distinguishes bytewise from charwise: {"é":1,"ab":2,"zz":3} renders as {"ab": 2, "zz": 3, "é": 1}, because é is 0xC3 0xA9 and 0xC3 > 0x7a. A charwise sort would pass every test currently in the tree and be subtly wrong.
• Escaping. Matches MySQL exactly — \t \n \r \b \f, \uXXXX for other control bytes, / and non-ASCII left raw. I checked all of those against the server.

One blocking item, which is a trap in our codebase rather than anything you did: this needs a persistent-state version bump. Since on updates we match before row entirely, this will break each delete and update that we receive via binlog. Check public/dataflow-state/src/persistent_state/format_version.rs - PERSISTENT_STATE_VERSION

Beyond that it's one suggestion that makes the snapshot path both simpler and faster, some optional perf polish,

[floze-the-genius]

Updated the branch in e330ce3a7 to address the review:

• bumped PERSISTENT_STATE_VERSION from 7 to 8;
• snapshot ingestion now validates UTF-8 and stores the canonical JSON bytes returned by MySQL directly;
• mysql_json_print now returns ReadySetResult<String> and propagates serialization errors instead of panicking;
• added initial output capacity while keeping the optional deeper allocation refactor out of this patch.

Validation: the two mysql_json_print unit tests pass, and the mysql_replicate_json_field integration test target compiles successfully.

[floze-the-genius]

@altmannmarcelo Rebased onto current main at ddc6f307 and adapted the snapshot path to the new mysql_value_to_noria_value helper while preserving its detailed/redacted error handling. The persistent-state bump remains 7→8, and I regenerated the four metadata golden records with the repository generator. Current-head validation passes: JSON unit tests 2/2, both persistent-state compatibility tests 1/1, replicators clippy with -D warnings, formatting, and diff check. The MySQL integration target compiles; the runtime case was not rerun after rebase because no local server is listening on port 33367. Ready for another look.