Skip to content

feat(negotiation): switch the value format once every node can read it - #413

Open
cosmin-staicu wants to merge 1 commit into
perf/binary-event-formatterfrom
perf/format-negotiation
Open

cosmin-staicu wants to merge 1 commit into
perf/binary-event-formatterfrom
perf/format-negotiation

Conversation

@cosmin-staicu

@cosmin-staicu cosmin-staicu commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Summary

Sixth in the fast value serializer stack, on top of #412. A policy's non-JSON value format is written only once every live node is known to read it. The switch happens in one deploy, with no operator step.

every node with a non-JSON serializer registered
  every 15 s: HSET advertisement {formats its reads decode}, stamped with Redis TIME
  (a policy whose reads do not look for a header advertises json only, which blocks)
  (reader-only and Force nodes advertise too; a stopping node marks its ad as left)

one decision per registered serializer (registered, merged and call-site policies share it)
  every 15 s (own schedule): read the advertisements and the Streams consumer groups
  every live node advertises the format, no node joined for UpgradeHold     -> write it
  a node leaving, or crashing (its ad outlives its group), keeps the format
  a node that cannot read it, or a live group with no advertisement          -> JSON at once
  node list unreadable                     -> keep the decision for MaxViewAge, then JSON
  after an outage                       -> keep it while peers advertise again
  cannot see peers (broadcast off, Pub/Sub only)  -> JSON unless AssumeAllNodesCanRead
                                           (an ad lacking the format still blocks; hold applies)
  Mode = Force                          -> write the policy's format, as before
  raw or custom proxy policies          -> always write their own format (no lossless fallback)

hot path: FormatFor reads a cached decision, no Redis call

UpgradeHold defaults to 1 hour, and 0 is allowed. A rollback within the hold never wrote the new format, so old nodes see no misses. After the switch, a rollback makes old nodes miss and rewrite those keys as JSON. They never read wrong values.

An app with no serializer registered does nothing new. It writes no key, starts no timer and allocates nothing extra on the JSON path. The Pub/Sub node list and binary events follow in the next PR.

New options are under RedisCacheOptions.FormatNegotiation: Mode, AdvertiseInterval, ViewInterval, LivenessWindow, UpgradeHold, MaxViewAge and AssumeAllNodesCanRead. New metrics are cache.format.effective, cache.format.fallback, cache.format.nodes and cache.format.view.age.

Evidence

  • The JSON default path allocates what the base (edd0081) allocates. SerializerBenchmark with no registry, B:

    Call Small base / branch Medium base / branch
    Get 1104 / 1104 2136 / 2136
    GetOrAdd hit 1408 / 1408 2440 / 2440
    Set 776 / 776 1408 / 1408
  • Two nodes against Redis:

    • An old node blocks the switch.
    • The writer switches after the liveness window and the hold.
    • It goes back to JSON at once when the old node returns.
    • A hash with peer fields stays readable across the mixed versions.
  • Mutations caught:

    • an advertisement stamped with the local clock;
    • "any node" instead of "every node" advertising the format;
    • no hold;
    • the hold default set to 5 minutes;
    • liveness from a group existing instead of its idle time;
    • the node list read before advertising;
    • an extra Redis call in FormatFor;
    • the advertisement service registered for an app with no serializer;
    • Force ignored;
    • a reader-only node not advertising.
  • Integration tests that expect a headered write through the container set Mode = Force, since Auto writes JSON until the hold passes.

  • Full suite with Redis: 5168, 0 failed, 1 skipped (a pre-Redis-7 case on Redis 7.4). -c Release -warnaserror builds clean.

Merge Danger

Door: two-way

Apps with no serializer registered are unchanged. Apps that already write MessagePack now write JSON until every node advertises and the hold passes, or until they set Mode = Force. All 3.2 nodes read both formats, so either way is safe.

Blast Radius: storage

The value format of keys whose policy writes a non-JSON serializer, plus one advertisement hash per keyspace.

@github-actions github-actions Bot added the needs-cla-review A maintainer should assess whether a signed CLA is required (see CONTRIBUTING.md) label Nov 12, 2026
@github-actions

Copy link
Copy Markdown

🔎 Maintainer heads-up: automated triage flagged this PR as potentially material, so it may need a signed CLA in addition to the DCO sign-off.

Strong signals

  • adds public API surface (PublicAPI.Unshipped.txt in src/UiPath.Caching)

Other signals

  • large production change (+1016 lines under src/)

This is advisory only — the bot does not decide. Please judge against the CLA criteria (material, product-critical, patent-sensitive, corporate contributor, broad commercial use). Note that thresholds can be gamed by splitting PRs, so use your judgement.

  • If a CLA is needed → add the cla-required label (a contributor comment with signing steps is posted automatically).
  • If it is not needed → replace needs-cla-review with cla-not-required so later pushes don't re-flag it.

A policy's non-JSON value format is now written only once every live node is known to read it. The switch happens in one deploy with no operator step. Every node with a non-JSON serializer registered advertises, every 15 seconds, the formats its reads decode, stamped with Redis TIME in a hash. A policy whose reads do not look for a header advertises json only, which blocks the switch. Reader-only and Force nodes advertise too, and a stopping node marks its advertisement as left.

One decision is kept per registered serializer, refreshed every 15 seconds from the advertisements and the Streams consumer groups. The format is written when every live node advertises it and no node joined within UpgradeHold. A node that leaves or crashes keeps the format. A node that cannot read it, or a live group with no advertisement, sends writes back to JSON at once. An unreadable node list keeps the decision for MaxViewAge and then falls back to JSON. A node that cannot see its peers, as with broadcast off or Pub/Sub only, writes JSON unless AssumeAllNodesCanRead is set. Mode = Force writes the policy's format as before, and raw or custom proxy policies always write their own format.

UpgradeHold defaults to 1 hour and 0 is allowed. A rollback within the hold never wrote the new format, and after the switch it makes old nodes miss and rewrite those keys as JSON, never read wrong values. FormatFor reads a cached decision with no Redis call. An app with no serializer registered writes no key, starts no timer and allocates nothing extra.

New options live under RedisCacheOptions.FormatNegotiation: Mode, AdvertiseInterval, ViewInterval, LivenessWindow, UpgradeHold, MaxViewAge and AssumeAllNodesCanRead. New metrics are cache.format.effective, cache.format.fallback, cache.format.nodes and cache.format.view.age.

Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>
@cosmin-staicu
cosmin-staicu force-pushed the perf/format-negotiation branch from 3c4c09c to 19fd942 Compare November 14, 2026 07:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-cla-review A maintainer should assess whether a signed CLA is required (see CONTRIBUTING.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant