Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)

### Changed

- **A policy's non-JSON `Serializer.Write` is written only while every node can read it.** With the new `RedisCacheOptions.FormatNegotiation` (`Mode` `Auto` by default), each node advertises the formats it reads in a Redis hash stamped with the server's clock and reads the node list, with the Streams consumer groups, on a 15-second timer, off the read and write paths. A policy writes its format once every fresh advertisement lists it, no live node is unaccounted for, and that has held for `UpgradeHold` (an hour by default; zero switches at once); a node that cannot read it stops the writes at once, and an unreadable node list keeps the decision for `MaxViewAge` (two minutes), then writes JSON. A node with a serializer registered advertises what it reads in either mode, even if none of its policies writes a non-JSON format, so a reader-only or `Force` node never blocks the writers (an app with no Streams transport neither advertises nor reads advertisements); in `Auto` it decides for every registered serializer, so a policy built at the call site is held to it too. An advertisement outlives the consumer group of a node that crashed, and a node that stops writes a departing advertisement, so a node that leaves never drops the fleet to JSON or restarts the hold, which a join or a restart does. After a Redis outage the decision is kept for two advertise intervals while peers advertise again. `AssumeAllNodesCanRead` covers only nodes that advertise nothing; with no Streams transport it switches after the hold with nothing that can stop it. An advertisement older than its group's last read does not account for the group, so a restart under the same id is seen at once, and a node list that fails to read restarts the hold. The advertisement follows what a node's reads really do: on a custom proxy a policy that writes no non-JSON format advertises JSON only, which keeps writers on JSON, and a policy on such a proxy always writes its configured format, since raw bytes have no JSON to fall back to and could read back as a header. Advertise and node list reads run on their own intervals, and a topic's stream is forgotten when the topic is removed. An app with no serializer registered does nothing new. An app that cannot see its peers (`BroadcastEnabled` off, or Pub/Sub only) writes JSON unless `AssumeAllNodesCanRead` is set. `Mode = Force` writes exactly what each policy configures, as before. New metrics `cache.format.effective`, `cache.format.fallback`, `cache.format.nodes` and `cache.format.view.age`. See [choosing when a format is written](docs/concepts.md#choosing-when-a-format-is-written).
- **An event in a known format that cannot be decoded is a topic loss.** The Streams reader acknowledges the entry and reports a loss, and counts such an entry as one it would have acted on when it checks what a new group skipped; the Pub/Sub reader expires through the coalesced dropped-event path, off the delivery thread, so it holds for caches that keep their tier over a subscription gap. The Streams reader reports once per batch. This applies to a body whose first byte is `0x02`, `0x03` or a consumer format id (`0x10`-`0x1E`, `0x1F`), such as a binary event on a node that cannot read it. Any other body that does not decode, another producer's first byte included, is still skipped as junk.
- **Cache event properties are read as typed values.** The built-in event formatters parse `_version_`, the expirations, the metadata and the peer fields straight into `long`, `DateTimeOffset`, string dictionaries and byte arrays instead of a `JsonElement` that was parsed again by the value serializer, which saves a clone, a box and a second parse per property of a peer-update event. The event bytes on the wire are unchanged. A custom `ISerializerProxy<byte[]>` no longer sees these properties in `TryDeserialize(object)` when the built-in formatters decode the event.
- **Decoded `ICacheEvent.Data.Properties` hold typed values.** From the built-in formatter and the CloudEvents fast path they are `long`, `DateTimeOffset`, `Dictionary<string, string?>` and byte-valued peer fields instead of `JsonElement`, so a custom subscriber that casts a property to `JsonElement` must handle those types.
Expand Down
38 changes: 36 additions & 2 deletions docs/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -516,9 +516,10 @@ the key's policy is not known to the peer, it does not apply the update and read

### Rolling a serializer out

Reading comes first, writing second. Every node from 3.2.0 reads every registered format; nothing writes a
With `FormatNegotiation:Mode` at `Force`, reading comes first, writing second. Every node from 3.2.0 reads every registered format; nothing writes a
new one until a policy sets `Serializer`, so set it only after every node runs 3.2.0 or later with the
serializer registered.
serializer registered. With the default `Auto`, a node that sees an older node keeps writing JSON by itself; see
[choosing when a format is written](#choosing-when-a-format-is-written).

A node that cannot read a format treats the value as a miss, and what happens to the key depends on why:

Expand All @@ -543,6 +544,39 @@ See [MessagePack events](how-to/extending.md#messagepack-events).
All of it shows up as factory load and churn during the deploy, never as a wrong value. Switching a policy back
to JSON is safe at any time: new writes are JSON and the older values still read.

### Choosing when a format is written

A policy's `Serializer.Write` names the format it *wants*. With `FormatNegotiation:Mode` at `Auto` (the default), a node writes it
only while every node that shares the Streams transport is known to read it, and writes JSON until then; this replaces the two-deploy
order for values, so a node that registers the serializer and sets `Write` can be deployed in one step. A policy that sets `Write = "json"`
is pinned to JSON, and `Mode = Force` writes exactly what each policy configures, as before.

How it works. An app with a serializer registered advertises what it reads, whether or not any policy of its own writes a non-JSON format and in either mode, so a node that only reads, or runs `Force`, never blocks the writers. In `Auto` it also reads the node list and decides, for every registered serializer, so a policy built at the call site that names one is held to the same decision as a registered policy. An app with no serializer registered does none of this: it advertises nothing, and so correctly counts as a node that cannot read.

The advertisement says what the node's reads actually do. With the default JSON proxy a read looks for a format header whatever the policy writes. With a custom proxy, a subclass of the default or `RawByteSerializerProxy`, a read looks for one only under a policy that itself writes a non-JSON format; for every other policy (and the default policy, when it is not one) the node advertises that it reads JSON only, and no writer upgrades while such a node is on the node list. A policy on such a proxy never falls back to JSON either, since its bytes are not JSON and could read back as a header: it always writes the format it names, as in `Force`.

- Each node writes an advertisement (the formats it reads, and the read list of each policy that sets one) to a hash in Redis every
15 seconds, stamped with the server's clock, so a node's own clock never matters. It advertises first when its hosted service starts, before it reads the
node list; the service is registered with the caching, so an app hosted service that touches a cache earlier can subscribe first and be briefly unaccounted
for. An advertisement counts for `LivenessWindow` plus two advertise intervals, so it outlives the consumer group of a node that crashed. A node that stops
writes a departing advertisement (`left=1`) instead of deleting its own: its group is then a node that left, neither a reader nor an unaccounted one.
- Every node also reads that hash and the consumer groups of its Streams topics. A group has the node's source id as its name; one whose
consumer read within `LivenessWindow` and that no fresh advertisement accounts for is a node that cannot say what it reads (a
node before 3.2, for one), and it keeps every writer on JSON. A node that went quiet for longer than `LivenessWindow` no longer counts.
- The format is written once every advertisement lists it, no live node is unaccounted for, and that has held for `UpgradeHold` (an hour by default) since a node
last joined or restarted (each process advertises its own id, so a restart under the same name is a join). A node that leaves never restarts the hold. A node that cannot read it, or an unaccounted node that appears, stops the writes at once. Redis being unreachable keeps the decision
for `MaxViewAge` since the last good read, then writes JSON, and for two advertise intervals after it returns, so peers can advertise again; an advertisement that lacks the format is never excused. All of this runs on a timer; a read or a write sends no extra command, only a read of the cached decision and its age.

The hold trades speed for rollback safety. After a rollback to a build that cannot read the format, nodes miss on the values written in that
format and the generator rewrites them as JSON, never a wrong value. With the default hour, an app rolled back within the hour never wrote the
format at all. A shorter `UpgradeHold` switches sooner and widens that rollback window; zero switches as soon as the node list agrees.

The node list sees only nodes on the Streams transport. An app that has `BroadcastEnabled` off, or that publishes events over Pub/Sub only, cannot see the others and writes JSON; set
`FormatNegotiation:AssumeAllNodesCanRead` to attest that every node that says nothing reads the desired formats. The attestation covers nodes that advertise nothing, not one whose advertisement lacks the format, and the hold still applies. Every node judges the others by its own `AdvertiseInterval` and `LivenessWindow`, so keep them alike across the fleet. Group
names that are not node ids (for example another app sharing the stream key) also count as live nodes, a safe false positive.
Telemetry (`cache.format.effective`, `cache.format.fallback`, `cache.format.nodes` and `cache.format.view.age`) is emitted only while this runs, and
a node that is blocking is named in a Warning log. The advertisement needs a host that runs hosted services: without one, nothing advertises and values stay JSON.

See also: [how-to/extending.md#value-serializers](how-to/extending.md#value-serializers).

---
Expand Down
2 changes: 1 addition & 1 deletion docs/how-to/extending.md
Original file line number Diff line number Diff line change
Expand Up @@ -546,7 +546,7 @@ The package depends on the exact `UiPath.Caching` version it was built with, so
The serializer owns format ids `0x02` (plain) and `0x03` (LZ4); the id stored with each value says which, so a node reads
both whatever its threshold.

Roll it out readers first. A node reads a format only once the serializer is registered on it, so register
By default (`FormatNegotiation:Mode` `Auto`) a policy that sets `Write = "messagepack"` writes it only while every node on the Streams transport is known to read it, and JSON until then, so this is safe to deploy in one step; see [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written). With `Force`, roll it out readers first. A node reads a format only once the serializer is registered on it, so register
`AddMessagePackSerializer` on every node and finish that deploy before any policy sets `Write = "messagepack"`; a node that
cannot read the value misses and regenerates it (see [rolling a serializer out](../concepts.md#rolling-a-serializer-out)).
Setting `Write` back to `json` keeps values already stored as MessagePack readable while the default JSON proxy is in use; with a custom `ISerializerProxy` registered, headers are only recognized under a policy that writes a non-JSON format, so under `Write = "json"` those values go to your proxy and read as misses.
Expand Down
7 changes: 7 additions & 0 deletions docs/reference/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,13 @@ Per-topic overrides: add entries to `Topics[]` under `Broadcast:RedisPubSub`. Ea
| `ReadScriptsEnabled` | `bool` | `false` | Per-provider | By default a value and its expiration are read with a pipelined `GET` and `PTTL` (one round trip, no transaction); the pair is not atomic, so a key that expires between the two reads as a value without a TTL. Set to `true` to read on Redis 7 and later with one `EVAL_RO` script instead (a batch read runs one script for all its keys; less allocation, but several times the server time of a plain read on Azure Managed Redis). The connection's user then needs `EVAL_RO`/`EVALSHA_RO` (ACL category `@scripting`); if the first script fails for any reason, or the server later refuses one (`NOPERM`, or a disabled or renamed command), reads switch to the `GET`/`PTTL` pair for good with one warning per cache instance, so a server or proxy that cannot run Lua never turns reads into misses. Before Redis 7 no script is sent. Command statistics then show `EVAL_RO` instead of `GET`/`PTTL`. |
| `ReadFrom` | `RedisReadFrom` | `Master` | Per-provider | `Master` sends reads to the primary (`PreferMaster`); `Replica` sends them to a replica when one exists (`PreferReplica`). On Azure Redis 6 a replica round trip measured about 3 times the primary's (1.6 ms against 0.55 ms), and a replica can still hold a key the primary has already removed, so a read right after a write or an invalidation could return, or refill the entry with, stale data. With `Replica`, a key this node wrote or removed (counted from when the write completes), or heard an invalidation for through a live local entry, within `MasterReadWindow` still reads from the primary, as does every read for that window after a lost-events clear on a node holding local entries and, if more than 10,000 keys were touched within one window, every read until the window passes. A key with no local entry is not guarded against another node's write. Turn it on only to take read load off the primary. Writes and refreshes always go to the primary. |
| `MasterReadWindow` | `TimeSpan` | `00:00:01` | Per-provider | With `ReadFrom` set to `Replica`, how long a key this node wrote, removed or heard invalidated keeps reading from the master. Must be positive, and must exceed the expected replica lag: a replica that is further behind can still return the old value once the window ends. A fire-and-forget TTL write (`AwaitRefresh` off) starts the window when it is queued, not when the server applies it. Measured on a monotonic clock. Ignored with `Master`. |
| `FormatNegotiation:Mode` | `FormatNegotiationMode` | `Auto` | Per-provider | `Auto` decides which formats this node writes: a registered serializer is written only while every node on the Streams transport is known to read it, whichever policy asks for it (a registered policy or one built at the call site), and JSON otherwise. `Force` writes exactly what each policy configures, with no node list and no decision. A node with a serializer registered advertises the formats it reads in either mode, so a `Force` node does not keep an `Auto` neighbour on JSON, unless the app has no Streams transport (see `AssumeAllNodesCanRead`), where a node neither advertises nor reads advertisements. A policy with `Write = "json"` is never affected, and neither is one on a custom `ISerializerProxy<byte[]>` (or a subclass of the default, or `RawByteSerializerProxy`): its bytes are not JSON, so there is nothing safe to fall back to, and it always writes the format the policy names, as `Force` does. See [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written). |
| `FormatNegotiation:AdvertiseInterval` | `TimeSpan` | `00:00:15` | Per-provider | How often this node rewrites its advertisement (the formats it reads) in Redis, stamped with the server's clock. Every node judges the others' advertisements by its own settings, so set `AdvertiseInterval`, `ViewInterval` and `LivenessWindow` alike across the fleet. An advertisement counts for `LivenessWindow` plus two of these, so it outlives the consumer group of a node that crashed. Must be positive. |
| `FormatNegotiation:ViewInterval` | `TimeSpan` | `00:00:15` | Per-provider | How often this node reads the advertisements and the Streams consumer groups; it advertises first, on the same tick. Must be positive. |
| `FormatNegotiation:LivenessWindow` | `TimeSpan` | `00:02:00` | Per-provider | A consumer group no advertisement accounts for (a node before 3.2) counts as a live node, and blocks the switch, while its consumer read within this window. A group no consumer ever read from counts as nobody. An advertisement accounts for its group only if it is at least as fresh as the group's last read (within two `AdvertiseInterval`s), so a node restarted under the same id is not hidden by its previous run's advertisement. Must be positive. |
| `FormatNegotiation:UpgradeHold` | `TimeSpan` | `01:00:00` | Per-provider | How long every node must have been accounted for before a format is written, counted from the last time a node joined or restarted (a node that leaves never restarts it). Zero switches at once; negative is rejected. A shorter hold switches sooner, but if the app is rolled back after it, nodes that cannot read the format miss and rewrite values as JSON, never a wrong value. A longer one means an app rolled back within it never wrote the format, so the older nodes see no misses. A node that cannot read the format stops the writes at once, whatever the hold. A node list that fails to read (other than a group that vanished mid-read) restarts the hold. |
| `FormatNegotiation:MaxViewAge` | `TimeSpan` | `00:02:00` | Per-provider | While Redis cannot be read, the last decision stands for this long since the last good read, then values are written as JSON; a decision older than this is never used, even if a read hangs. After an outage shorter than this, the decision is also kept for two `AdvertiseInterval`s of successful ticks, so peers can advertise again, unless an advertisement says a node cannot read the format. Must be positive. |
| `FormatNegotiation:AssumeAllNodesCanRead` | `bool` | `false` | Per-provider | Attests that the nodes that say nothing read the desired formats: a live Streams consumer group with no advertisement, and, for an app with `BroadcastEnabled` off or without a Streams topic (Pub/Sub only), the nodes it cannot see at all. For an app that can list its peers, an advertisement that lacks the format still stops the writes. Without a Streams transport the node neither advertises nor reads advertisements, so nothing can stop the switch once `UpgradeHold` has passed, counted from the first decision. Without the setting such an app writes JSON. |

*Code-only seams:* `EntryFactory`, `CacheKeyStrategy`, `RedisKeyStrategyFactory`.

Expand Down
Loading
Loading