Skip to content

feat(negotiation): switch event formats once every subscriber can read them - #415

Open
cosmin-staicu wants to merge 1 commit into
perf/format-negotiationfrom
perf/event-format-negotiation
Open

cosmin-staicu wants to merge 1 commit into
perf/format-negotiationfrom
perf/event-format-negotiation

Conversation

@cosmin-staicu

@cosmin-staicu cosmin-staicu commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Summary

Seventh in the fast value serializer stack, on top of #413. Binary events switch the way values do, and Pub/Sub deployments can be seen, so they switch too.

AddMessagePackEvents(write: true)
  every node lists the event formats it decodes in its advertisement (er=json,02 with the decoder, er=json without; a node before this one lists nothing)
  every ad lists 02, no live node unaccounted for, UpgradeHold passed   -> binary events
  a node that lists only json, or an unaccounted node                  -> JSON at once
  write: false -> always JSON     Mode = Force -> binary at once, as before
  the decoder reads both formats always
  an old (3.0/3.1) node blocks binary events while it is live; its stale-L1 risk is the join window and a node judged
  dead that returns, bounded by its own LocalMaxExpiration (set a short one on old nodes before enabling write: true)
  no Redis connection, so no node list: JSON events unless AssumeAllNodesCanRead (hold applies) or Force

Streams transport: the consumer-group node list from #413, unchanged
Pub/Sub transport: no groups, so count subscribers
  live      = sum of PUBSUB NUMSUB over every server (primaries and replicas)
  accounted = sum of the subscriptions the live ads claim (sub=<channel>:1)
              a node that left lists what it still holds, and that claim is optional for its ad's lifetime
              (a slow shutdown, with the connection closing views after left=1, never takes the fleet to JSON)
  one round trip per server per view: PUBSUB NUMSUB for all channels (names carry the connection's ChannelPrefix)
  and PUBSUB NUMPAT; each channel is debounced on its own
  a server this node knows about but is not connected to fails the whole view (no partial sum): the debounce resets
  and the decision is kept until MaxViewAge
  patterns: NUMSUB leaves them out, NUMPAT counts the whole server; each node advertises psub=<n>, and more patterns
  than claimed (another app's included) is an unknown subscriber -> JSON unless AssumeAllNodesCanRead
  a node claims a subscription only once the server confirmed it, and advertises at once (subscribe, then advertise)
  live > accounted in two views in a row   -> unknown subscriber -> JSON for values and events
  live < accounted in two views in a row   -> a node went away (its ad outlives it) -> JSON too
                                              a departure, not a join: the hold is not restarted; when the ad expires and
                                              the counts match for two views, the earlier decision resumes
                                              (a deficit that clears with the claim still fresh is a join: the hold restarts)
  a one-view blip pauses a hold in progress; only a join, a confirmed surplus/deficit or a missing format restarts it
  NUMSUB refused, or a pattern/sharded channel strategy -> unknown, logged once
  upgrade needs two clean views over the same number of servers
  NUMSUB refused -> unknown -> JSON unless AssumeAllNodesCanRead

hot path: the event formatter reads one volatile reference and the injected clock, no Redis call

Values and events are decided apart. A node with the decoder and no value serializer advertises and decides too. An app that registers nothing is unchanged: no wrapper, no timer, no key.

No new options. New: the Caching.Stats.FormatBlockers metric, which names who blocks (node ids or Pub/Sub channels), only when the set changes, and not on the bounded cache.format.* metrics. cache.format.effective and cache.format.fallback gain a kind tag (value or event) and the reasons unaccounted_subscriber and subscriber_deficit; node ids never go into a tag. The "deploy 1 / deploy 2" text for events is gone from the docs.

Evidence

  • The JSON default path allocates what feat(negotiation): switch the value format once every node can read it #413 allocates. SerializerBenchmark, B, Small and Medium, Registered=False, the same for Registered=True:

    Call Small base / branch Medium base / branch
    Get 1104 / 1104 2136 / 2136
    GetOrAdd hit 1408 / 1408 2440 / 2440
    Set 776 / 776 1408 / 1408

    Large differs by under 0.03% in both directions (LOH arrays, short job).

  • With nothing registered the event formatter is the plain one, and encoding allocates what the default formatter allocates (a test). Through the negotiated wrapper it allocates what a fixed wrapper does.

  • Real Redis:

    • Two nodes on Pub/Sub, then a subscriber that advertises nothing joins: events go back to JSON within two views, the wire shows it, and they return to binary after it leaves.
    • The same on Streams with a group that advertises nothing.
    • Two connections on one channel sum to two, and a confirmed subscription is already counted by the server.
  • Mutations caught (50): events ignoring er; "any" instead of "every"; no hold for events; groups ignored for events; the composite deciding once; Force ignored; write: false asking the node list; NUMSUB on one server only; a surplus counted at the first view; an upgrade on one clean view; the server count ignored; a refused NUMSUB read as zero; a failed read keeping the debounce; claiming channels, not confirmed subscriptions; no immediate advertisement after a confirmation; a node that left still claiming; a deficit not blocking, counted at the first view, restarting the hold, or clearing as clean; a join during a deficit not restarting the hold; no node list writing binary, ignoring the attestation, or ignoring Force; a departing ad with no subscriptions; a leaving claim required or not optional; a blip restarting the hold; a deficit cleared by a fresh claim not a join; no reset after a failed or empty view; a refused NUMSUB not logged; advertisements not serialized, written after the departing one, or not waited for by a stop; more than one command per server; a pattern or sharded channel counted; the debounce not per channel; an empty policy tag; a disconnected server left out of the sum; patterns not counted, not claimed, not advertised or not read from the server; fewer patterns than claimed as a deficit; the channel prefix not applied.

  • Tests that expect binary events through the container set Mode = Force, since Auto writes JSON until the hold passes.

  • Full suite with Redis: 5253, 0 failed, 1 skipped (a pre-Redis-7 case on Redis 7.4). -c Release -warnaserror builds clean.

Merge Danger

Door: two-way

Apps that already write binary events now write JSON until every node advertises the decoder and the hold passes, or until they set Mode = Force; an app with no Redis connection writes JSON unless it sets AssumeAllNodesCanRead or Force. All 3.2 nodes read both formats, so either way is safe. A Pub/Sub app whose Redis refuses PUBSUB NUMSUB stays on JSON unless AssumeAllNodesCanRead is set.

Blast Radius: storage

The wire format of events on the topics of apps that set write: true, the value format on Pub/Sub deployments (now able to switch), and one advertisement per node now lists its subscriptions.

@github-actions github-actions Bot added the needs-cla-review A maintainer should assess whether a signed CLA is required (see CONTRIBUTING.md) label Nov 12, 2026
@github-actions

Copy link
Copy Markdown

🔎 Maintainer heads-up: automated triage flagged this PR as potentially material, so it may need a signed CLA in addition to the DCO sign-off.

Strong signals

  • adds public API surface (PublicAPI.Unshipped.txt in src/UiPath.Caching)

Other signals

  • large production change (+622 lines under src/)

This is advisory only — the bot does not decide. Please judge against the CLA criteria (material, product-critical, patent-sensitive, corporate contributor, broad commercial use). Note that thresholds can be gamed by splitting PRs, so use your judgement.

  • If a CLA is needed → add the cla-required label (a contributor comment with signing steps is posted automatically).
  • If it is not needed → replace needs-cla-review with cla-not-required so later pushes don't re-flag it.

@cosmin-staicu cosmin-staicu changed the title feat(negotiation): switch event formats and use Pub/Sub subscriber counts as the roster feat(negotiation): switch event formats once every subscriber can read them Nov 13, 2026
@cosmin-staicu
cosmin-staicu force-pushed the perf/format-negotiation branch from 252edb9 to d5d8a19 Compare November 13, 2026 13:39
@cosmin-staicu
cosmin-staicu force-pushed the perf/event-format-negotiation branch from 42ea5b0 to cb75a10 Compare November 13, 2026 13:39
@cosmin-staicu
cosmin-staicu force-pushed the perf/format-negotiation branch 2 times, most recently from 3c4c09c to 19fd942 Compare November 14, 2026 07:47
@cosmin-staicu
cosmin-staicu force-pushed the perf/event-format-negotiation branch from cb75a10 to 32d0840 Compare November 14, 2026 08:18
…d them

Binary events now switch the way values do, and Pub/Sub deployments can be seen, so they switch too. With AddMessagePackEvents(write: true), each node lists the event formats it decodes in its advertisement. Binary events are written once every advertisement lists 02, no live node is unaccounted for and UpgradeHold has passed. A node that lists only json, or an unaccounted node, sends events back to JSON at once. Write false always writes JSON, Mode = Force writes binary at once, the decoder always reads both formats, and with no Redis connection events stay JSON unless AssumeAllNodesCanRead or Force is set. Values and events are decided apart.

Pub/Sub has no consumer groups, so subscribers are counted. The live count is the sum of PUBSUB NUMSUB over every server, primaries and replicas, and the accounted count is the sum of the subscriptions that live advertisements claim. A node claims a subscription only after the server confirms it, and advertises at once. Patterns are counted with PUBSUB NUMPAT and advertised as psub, and more patterns than claimed is an unknown subscriber. More live than accounted for two views in a row is an unknown subscriber, and fewer is a departed node whose advertisement outlives it. Both fall back to JSON, and a departure does not restart the hold. A server the node knows about but is not connected to fails the whole view, and a refused NUMSUB is unknown, so the format stays JSON unless AssumeAllNodesCanRead is set.

An upgrade needs two clean views over the same number of servers, and a one-view blip only pauses a hold in progress. The event formatter reads one volatile reference and the injected clock, with no Redis call on the hot path, and an app that registers nothing gets no wrapper, timer or key.

There are no new options. The Caching.Stats.FormatBlockers metric names who blocks, as node ids or Pub/Sub channels, only when the set changes. cache.format.effective and cache.format.fallback gain a kind tag with value or event, plus the reasons unaccounted_subscriber and subscriber_deficit.

Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>
@cosmin-staicu
cosmin-staicu force-pushed the perf/event-format-negotiation branch from 32d0840 to 87600a7 Compare November 14, 2026 09:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-cla-review A maintainer should assess whether a signed CLA is required (see CONTRIBUTING.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant