Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CHANGELOG.md

Large diffs are not rendered by default.

81 changes: 69 additions & 12 deletions docs/concepts.md

Large diffs are not rendered by default.

23 changes: 14 additions & 9 deletions docs/how-to/extending.md
Original file line number Diff line number Diff line change
Expand Up @@ -546,7 +546,7 @@ The package depends on the exact `UiPath.Caching` version it was built with, so
The serializer owns format ids `0x02` (plain) and `0x03` (LZ4); the id stored with each value says which, so a node reads
both whatever its threshold.

By default (`FormatNegotiation:Mode` `Auto`) a policy that sets `Write = "messagepack"` writes it only while every node on the Streams transport is known to read it, and JSON until then, so this is safe to deploy in one step; see [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written). With `Force`, roll it out readers first. A node reads a format only once the serializer is registered on it, so register
By default (`FormatNegotiation:Mode` `Auto`) a policy that sets `Write = "messagepack"` writes it only while every node on the Streams or Pub/Sub transport is known to read it, and JSON until then, so this is safe to deploy in one step; see [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written). With `Force`, roll it out readers first. A node reads a format only once the serializer is registered on it, so register
`AddMessagePackSerializer` on every node and finish that deploy before any policy sets `Write = "messagepack"`; a node that
cannot read the value misses and regenerates it (see [rolling a serializer out](../concepts.md#rolling-a-serializer-out)).
Setting `Write` back to `json` keeps values already stored as MessagePack readable while the default JSON proxy is in use; with a custom `ISerializerProxy` registered, headers are only recognized under a policy that writes a non-JSON format, so under `Write = "json"` those values go to your proxy and read as misses.
Expand All @@ -565,15 +565,20 @@ JSON size and a peer-field event saves the 33% base64 adds. Properties it has no
and read back as `JsonElement`, as they do from a JSON event, and an expiry reads back in UTC.

```csharp
b.AddMessagePackEvents(write: false); // first deploy: reads both, still writes JSON
b.AddMessagePackEvents(write: true); // second deploy: writes binary
b.AddMessagePackEvents(write: true); // reads both; writes binary once every live node can read it
b.AddMessagePackEvents(write: false); // reads both; always writes JSON
```

Switch in two deploys, like a value serializer. Every node must run library version 3.2.0 or later, the first one that
has this formatter and the loss handling below, with `AddMessagePackEvents(write: false)` registered, before any node
uses `write: true`. A node on an older version acknowledges and skips a binary event, logging a warning for every 100th one it cannot decode and tracking telemetry, but never treating it as a loss, so it keeps what
it has until `LocalMaxExpiration`; one on 3.2.0 or later without the formatter treats the event as lost and drops its
whole local tier for that topic. Switching back to `write: false` is safe at any time. Never remove the JSON decoder:
There is no deploy order. With `write: true` a node writes JSON until the node list shows that every node reads binary events, for the
`UpgradeHold` (an hour by default) after the last node joined, and writes JSON again as soon as a node that cannot read them appears. Each node
advertises the event formats it decodes: one with this formatter registered lists binary, one without lists JSON only, and a node before
3.2.0 lists nothing and is treated as unable to read. See [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written),
including how a Pub/Sub deployment counts subscribers. A node on an older version acknowledges and skips a binary event, logging a warning for every 100th one it cannot decode and tracking telemetry, but never treating it as a loss, so it keeps
what it has until `LocalMaxExpiration`; one on 3.2.0 or later without the formatter treats the event as lost and drops its whole local tier for that
topic. An old node blocks binary events while it is live in the node list, so what is left is the window a join leaves open, about one view interval; a Pub/Sub node that advertises nothing and takes a freed subscription slot, which stays hidden for up to the advertisement lifetime plus two views (about 180 s by default), during which it skips binary events and may serve a stale local value; and a node judged dead that returns on the same source id: the
stale local entry it keeps is bounded by that old node's own `LocalMaxExpiration`. Set a short one, a few minutes, on 3.0 and 3.1 nodes before enabling `write: true`.

`FormatNegotiation:Mode` `Force` writes binary at once with `write: true`: then every node must read both first, as before. An app with no Redis connection has no node list to read, and `write: true` writes JSON events unless `FormatNegotiation:AssumeAllNodesCanRead` attests that every node reads binary or `Mode` is `Force`. Switching back to `write: false` is safe at any time. Never remove the JSON decoder:
a restarted node replays its stream backlog, and a JSON entry it could not read is skipped without counting as a loss, so the node would keep what that entry invalidated. The setting is
app-wide, not per topic.

Expand All @@ -592,7 +597,7 @@ read. A map's initial capacity is capped at 16 entries, so allocation follows th

The integer event type values on the wire (`CacheSet` 0, `CacheRefreshed` 1, `CacheRemoved` 2, `TopicInvalidated` 3)
are fixed and only ever appended to. A node that reads a type number it does not know refuses the event, which is a
loss, so a release that adds a type needs the same two deploys.
loss, so a release that adds a type needs every node on it before any node writes binary events: use `Force` only after that, and in `Auto` upgrade every node before the hold has passed.

## Swapping the default factories

Expand Down
4 changes: 2 additions & 2 deletions docs/reference/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,13 +215,13 @@ Per-topic overrides: add entries to `Topics[]` under `Broadcast:RedisPubSub`. Ea
| `ReadScriptsEnabled` | `bool` | `false` | Per-provider | By default a value and its expiration are read with a pipelined `GET` and `PTTL` (one round trip, no transaction); the pair is not atomic, so a key that expires between the two reads as a value without a TTL. Set to `true` to read on Redis 7 and later with one `EVAL_RO` script instead (a batch read runs one script for all its keys; less allocation, but several times the server time of a plain read on Azure Managed Redis). The connection's user then needs `EVAL_RO`/`EVALSHA_RO` (ACL category `@scripting`); if the first script fails for any reason, or the server later refuses one (`NOPERM`, or a disabled or renamed command), reads switch to the `GET`/`PTTL` pair for good with one warning per cache instance, so a server or proxy that cannot run Lua never turns reads into misses. Before Redis 7 no script is sent. Command statistics then show `EVAL_RO` instead of `GET`/`PTTL`. |
| `ReadFrom` | `RedisReadFrom` | `Master` | Per-provider | `Master` sends reads to the primary (`PreferMaster`); `Replica` sends them to a replica when one exists (`PreferReplica`). On Azure Redis 6 a replica round trip measured about 3 times the primary's (1.6 ms against 0.55 ms), and a replica can still hold a key the primary has already removed, so a read right after a write or an invalidation could return, or refill the entry with, stale data. With `Replica`, a key this node wrote or removed (counted from when the write completes), or heard an invalidation for through a live local entry, within `MasterReadWindow` still reads from the primary, as does every read for that window after a lost-events clear on a node holding local entries and, if more than 10,000 keys were touched within one window, every read until the window passes. A key with no local entry is not guarded against another node's write. Turn it on only to take read load off the primary. Writes and refreshes always go to the primary. |
| `MasterReadWindow` | `TimeSpan` | `00:00:01` | Per-provider | With `ReadFrom` set to `Replica`, how long a key this node wrote, removed or heard invalidated keeps reading from the master. Must be positive, and must exceed the expected replica lag: a replica that is further behind can still return the old value once the window ends. A fire-and-forget TTL write (`AwaitRefresh` off) starts the window when it is queued, not when the server applies it. Measured on a monotonic clock. Ignored with `Master`. |
| `FormatNegotiation:Mode` | `FormatNegotiationMode` | `Auto` | Per-provider | `Auto` decides which formats this node writes: a registered serializer is written only while every node on the Streams transport is known to read it, whichever policy asks for it (a registered policy or one built at the call site), and JSON otherwise. `Force` writes exactly what each policy configures, with no node list and no decision. A node with a serializer registered advertises the formats it reads in either mode, so a `Force` node does not keep an `Auto` neighbour on JSON, unless the app has no Streams transport (see `AssumeAllNodesCanRead`), where a node neither advertises nor reads advertisements. A policy with `Write = "json"` is never affected, and neither is one on a custom `ISerializerProxy<byte[]>` (or a subclass of the default, or `RawByteSerializerProxy`): its bytes are not JSON, so there is nothing safe to fall back to, and it always writes the format the policy names, as `Force` does. See [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written). |
| `FormatNegotiation:Mode` | `FormatNegotiationMode` | `Auto` | Per-provider | `Auto` decides which formats this node writes: a registered serializer, and binary events when `AddMessagePackEvents(write: true)` is registered, are written only while every node on the Streams or Pub/Sub transport is known to read them, whichever policy asks for a serializer (a registered policy or one built at the call site), and JSON otherwise. `Force` writes exactly what each policy configures, and binary events with `write: true`, with no node list and no decision. A node with a serializer or the binary event decoder registered advertises the formats it reads in either mode, so a `Force` node does not keep an `Auto` neighbour on JSON, unless the app has neither a Streams nor a Pub/Sub transport (see `AssumeAllNodesCanRead`), where a node neither advertises nor reads advertisements. A policy with `Write = "json"` is never affected, and neither is one on a custom `ISerializerProxy<byte[]>` (or a subclass of the default, or `RawByteSerializerProxy`): its bytes are not JSON, so there is nothing safe to fall back to, and it always writes the format the policy names, as `Force` does. See [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written). |
| `FormatNegotiation:AdvertiseInterval` | `TimeSpan` | `00:00:15` | Per-provider | How often this node rewrites its advertisement (the formats it reads) in Redis, stamped with the server's clock. Every node judges the others' advertisements by its own settings, so set `AdvertiseInterval`, `ViewInterval` and `LivenessWindow` alike across the fleet. An advertisement counts for `LivenessWindow` plus two of these, so it outlives the consumer group of a node that crashed. Must be positive. |
| `FormatNegotiation:ViewInterval` | `TimeSpan` | `00:00:15` | Per-provider | How often this node reads the advertisements and the Streams consumer groups; it advertises first, on the same tick. Must be positive. |
| `FormatNegotiation:LivenessWindow` | `TimeSpan` | `00:02:00` | Per-provider | A consumer group no advertisement accounts for (a node before 3.2) counts as a live node, and blocks the switch, while its consumer read within this window. A group no consumer ever read from counts as nobody. An advertisement accounts for its group only if it is at least as fresh as the group's last read (within two `AdvertiseInterval`s), so a node restarted under the same id is not hidden by its previous run's advertisement. Must be positive. |
| `FormatNegotiation:UpgradeHold` | `TimeSpan` | `01:00:00` | Per-provider | How long every node must have been accounted for before a format is written, counted from the last time a node joined or restarted (a node that leaves never restarts it). Zero switches at once; negative is rejected. A shorter hold switches sooner, but if the app is rolled back after it, nodes that cannot read the format miss and rewrite values as JSON, never a wrong value. A longer one means an app rolled back within it never wrote the format, so the older nodes see no misses. A node that cannot read the format stops the writes at once, whatever the hold. A node list that fails to read (other than a group that vanished mid-read) restarts the hold. |
| `FormatNegotiation:MaxViewAge` | `TimeSpan` | `00:02:00` | Per-provider | While Redis cannot be read, the last decision stands for this long since the last good read, then values are written as JSON; a decision older than this is never used, even if a read hangs. After an outage shorter than this, the decision is also kept for two `AdvertiseInterval`s of successful ticks, so peers can advertise again, unless an advertisement says a node cannot read the format. Must be positive. |
| `FormatNegotiation:AssumeAllNodesCanRead` | `bool` | `false` | Per-provider | Attests that the nodes that say nothing read the desired formats: a live Streams consumer group with no advertisement, and, for an app with `BroadcastEnabled` off or without a Streams topic (Pub/Sub only), the nodes it cannot see at all. For an app that can list its peers, an advertisement that lacks the format still stops the writes. Without a Streams transport the node neither advertises nor reads advertisements, so nothing can stop the switch once `UpgradeHold` has passed, counted from the first decision. Without the setting such an app writes JSON. |
| `FormatNegotiation:AssumeAllNodesCanRead` | `bool` | `false` | Per-provider | Attests that the nodes that say nothing read the desired formats: a live Streams consumer group with no advertisement, a Pub/Sub subscriber no advertisement accounts for (a pattern subscriber included, which `PUBSUB NUMPAT` counts for the whole server, other apps' too), a Pub/Sub deployment where `PUBSUB NUMSUB` is refused, and, for an app with `BroadcastEnabled` off, with neither a Streams nor a Pub/Sub topic, or with no Redis connection, the nodes it cannot see at all. For an app that can list its peers, an advertisement that lacks the format still stops the writes. Without a Streams or Pub/Sub transport the node neither advertises nor reads advertisements, so nothing can stop the switch once `UpgradeHold` has passed, counted from the first decision. Without the setting such an app writes JSON. |

*Code-only seams:* `EntryFactory`, `CacheKeyStrategy`, `RedisKeyStrategyFactory`.

Expand Down
4 changes: 2 additions & 2 deletions samples/UiPath.Caching.Sample/appsettings.all.json
Original file line number Diff line number Diff line change
Expand Up @@ -377,7 +377,7 @@
// MasterReadWindow: with ReadFrom Replica, how long a just-written or invalidated key keeps reading from the master; must exceed the replica lag
"MasterReadWindow": "0:00:01",
"FormatNegotiation": {
// Mode: Auto writes a registered serializer's format only while every node on the Streams transport is known to read it; Force writes exactly what each policy configures. Either way the node advertises what it reads
// Mode: Auto writes a registered serializer's format, and binary events with AddMessagePackEvents(write: true), only while every node on the Streams or Pub/Sub transport is known to read it; Force writes exactly what is configured. Either way the node advertises what it reads
"Mode": "Auto",
// AdvertiseInterval: how often this node refreshes its advertisement; an advertisement lives LivenessWindow plus two of these. Keep it alike across the fleet
"AdvertiseInterval": "0:00:15",
Expand All @@ -389,7 +389,7 @@
"UpgradeHold": "1:00:00",
// MaxViewAge: how long a decision survives a node list that cannot be read, before writes fall back to JSON
"MaxViewAge": "0:02:00",
// AssumeAllNodesCanRead: attest that nodes that advertise nothing (live Streams groups without an advertisement; broadcast off, Pub/Sub only) read the desired formats; an advertisement that lacks the format still blocks
// AssumeAllNodesCanRead: attest that nodes that advertise nothing (live Streams groups without an advertisement, Pub/Sub subscribers nobody advertises or a refused PUBSUB NUMSUB, broadcast off) read the desired formats; an advertisement that lacks the format still blocks
"AssumeAllNodesCanRead": false
}
},
Expand Down
Loading
Loading