diff --git a/CHANGELOG.md b/CHANGELOG.md index b6cdc763..f86aa864 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,11 +11,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) - **`UiPath.Caching.MessagePack`, a MessagePack value serializer.** `AddMessagePackSerializer()` registers it as `messagepack`, and a policy opts in per policy with `Serializer.Write = "messagepack"`; every other policy keeps writing JSON. It writes format id `0x02`, or `0x03` when the value's MessagePack bytes reach `CompressionThresholdBytes` (default 1024) and LZ4 compression makes them smaller, so an incompressible value stays `0x02`; it uses MessagePack's own LZ4 and adds no dependency. The default options are the contractless resolver with `MessagePackSecurity.UntrustedData`, and `SerializerOptions` replaces them. `MaxDecompressedBytes` defaults to 64 MiB: deserializing can allocate several times the decompressed size, so size it to available memory. Key types MessagePack has no collision-resistant comparer for (`TimeSpan`, `decimal`, tuples, custom structs) fall back to the default comparer. Register it on every node and finish that deploy before any policy writes it, since a node that cannot read the format misses and regenerates the value. Like JSON, contractless MessagePack writes property names, so renaming or retyping a property silently loses that data for values written before the change; `[MessagePackObject]` with `[Key]` avoids that. The package accepts MessagePack `[3.1.12, 4.0)`. See [MessagePack](docs/how-to/extending.md#messagepack). - **A seam for value serializers.** `ICacheValueSerializer` and `CacheValueFormats` define a buffer-based serializer whose output starts with a one-byte format id, `AddValueSerializer` registers one by name, and `CachePolicy.Serializer` (`Write`, `Read`, `MigrateOnRead`) names it per policy. `AddValueSerializer` throws at the call for a name used twice on one builder or for `json`. A policy that names an unregistered or blank serializer, has an empty `Read`, or writes a format its `Read` leaves out fails with an `OptionsValidationException`, as does a serializer whose ids are outside the consumer ranges; these are checked when the policy factory is first resolved, on first cache use rather than at host start. A custom `ICachePolicyFactory` whose policy names an unregistered serializer fails the same way when the Redis provider builds its cache. Serializers the container created are disposed with it, through `DisposeAsync` when they only implement `IAsyncDisposable`, so a factory must return a new instance rather than one resolved from DI. A policy that sets `Write` makes `RedisCache` and `RedisHashCache` write their values, hash fields and hash metadata with that serializer, and every read picks the serializer by the format id, whatever the policy writes. A value in a format no registered serializer owns is a miss, counted in `cache.serializer.unreadable` and removed only if the key still holds the bytes that were read, so a newer value written in between survives; a hash with such a field misses whole and is left in place. A value whose registered serializer throws is a logged miss and the key stays, like one that is not headered and fails to parse as JSON, because the bytes may be good for another version of the type during a rolling deploy; this departs from the design spec, which removed it, and an `OutOfMemoryException` from its payload is never swallowed, while one from a value with no header (JSON) is a logged miss as it is with no serializer registered. `RedisHashCache.TryDecodeFields` now treats any exception except `OutOfMemoryException` as a refresh, since a registered serializer can throw any type, where it caught four. A node with no serializer registered logs the `JsonException` and keeps the key. A policy passed to a single call that names an unregistered serializer throws `OptionsValidationException` before anything is written. The reader only looks for a header with the default JSON serializer or under a policy whose effective `Write` is a serializer other than `json` (`Read` does not count), so `byte[]` values stored through `RawByteSerializerProxy` or `AddDistributedCache` are never reinterpreted. A policy that sets no serializer, or only `Read`, takes the default policy's `Write`. The default stays JSON with no header, and `AddDistributedCache` ignores `Serializer`: its own policy always writes and reads JSON with no migration, so the caller's bytes never get a format header. A hash write that names a serializer leaves `_metadata_` out when the metadata is empty and nulls are not cached, and a `HashReplace` one deletes the field an earlier write left. Hash system field names are compared as bytes instead of strings. See [value serializers](docs/how-to/extending.md#value-serializers) and [rolling a serializer out](docs/concepts.md#rolling-a-serializer-out). -- **A binary event formatter for `UiPath.Caching.MessagePack`.** `AddMessagePackEvents(write)` reads JSON and binary events, told apart by the first byte (`0x02` for binary). `write: false` still writes JSON, byte for byte what the default formatter writes, and `write: true` writes a MessagePack map with integer keys, a binary GUID id, an integer event type and the peer fields as raw bytes. Switch in two deploys, read both first and write binary second, and never retire the JSON decoder; every node must be on 3.2.0 or later before any writes binary, since an older one skips binary events, logging a warning for every 100th and tracking telemetry, but never treats one as a loss. It wraps the JSON formatter that would otherwise be registered, `AddCloudEvents` included in either order, so the JSON wire is unchanged. See [MessagePack events](docs/how-to/extending.md#messagepack-events). +- **A binary event formatter for `UiPath.Caching.MessagePack`.** `AddMessagePackEvents(write)` reads JSON and binary events, told apart by the first byte (`0x02` for binary). `write: false` still writes JSON, byte for byte what the default formatter writes, and `write: true` writes a MessagePack map with integer keys, a binary GUID id, an integer event type and the peer fields as raw bytes, once every live node can read it (see the automatic switch below). Never retire the JSON decoder; an older node skips binary events, logging a warning for every 100th and tracking telemetry, but never treating one as a loss. It wraps the JSON formatter that would otherwise be registered, `AddCloudEvents` included in either order, so the JSON wire is unchanged. See [MessagePack events](docs/how-to/extending.md#messagepack-events). ### Changed -- **A policy's non-JSON `Serializer.Write` is written only while every node can read it.** With the new `RedisCacheOptions.FormatNegotiation` (`Mode` `Auto` by default), each node advertises the formats it reads in a Redis hash stamped with the server's clock and reads the node list, with the Streams consumer groups, on a 15-second timer, off the read and write paths. A policy writes its format once every fresh advertisement lists it, no live node is unaccounted for, and that has held for `UpgradeHold` (an hour by default; zero switches at once); a node that cannot read it stops the writes at once, and an unreadable node list keeps the decision for `MaxViewAge` (two minutes), then writes JSON. A node with a serializer registered advertises what it reads in either mode, even if none of its policies writes a non-JSON format, so a reader-only or `Force` node never blocks the writers (an app with no Streams transport neither advertises nor reads advertisements); in `Auto` it decides for every registered serializer, so a policy built at the call site is held to it too. An advertisement outlives the consumer group of a node that crashed, and a node that stops writes a departing advertisement, so a node that leaves never drops the fleet to JSON or restarts the hold, which a join or a restart does. After a Redis outage the decision is kept for two advertise intervals while peers advertise again. `AssumeAllNodesCanRead` covers only nodes that advertise nothing; with no Streams transport it switches after the hold with nothing that can stop it. An advertisement older than its group's last read does not account for the group, so a restart under the same id is seen at once, and a node list that fails to read restarts the hold. The advertisement follows what a node's reads really do: on a custom proxy a policy that writes no non-JSON format advertises JSON only, which keeps writers on JSON, and a policy on such a proxy always writes its configured format, since raw bytes have no JSON to fall back to and could read back as a header. Advertise and node list reads run on their own intervals, and a topic's stream is forgotten when the topic is removed. An app with no serializer registered does nothing new. An app that cannot see its peers (`BroadcastEnabled` off, or Pub/Sub only) writes JSON unless `AssumeAllNodesCanRead` is set. `Mode = Force` writes exactly what each policy configures, as before. New metrics `cache.format.effective`, `cache.format.fallback`, `cache.format.nodes` and `cache.format.view.age`. See [choosing when a format is written](docs/concepts.md#choosing-when-a-format-is-written). +- **A policy's non-JSON `Serializer.Write` is written only while every node can read it.** With the new `RedisCacheOptions.FormatNegotiation` (`Mode` `Auto` by default), each node advertises the formats it reads in a Redis hash stamped with the server's clock and reads the node list, with the Streams consumer groups, on a 15-second timer, off the read and write paths. A policy writes its format once every fresh advertisement lists it, no live node is unaccounted for, and that has held for `UpgradeHold` (an hour by default; zero switches at once); a node that cannot read it stops the writes at once, and an unreadable node list keeps the decision for `MaxViewAge` (two minutes), then writes JSON. A node with a serializer registered advertises what it reads in either mode, even if none of its policies writes a non-JSON format, so a reader-only or `Force` node never blocks the writers (an app with no Streams or Pub/Sub transport neither advertises nor reads advertisements); in `Auto` it decides for every registered serializer, so a policy built at the call site is held to it too. An advertisement outlives the consumer group of a node that crashed, and a node that stops writes a departing advertisement, so a node that leaves never drops the fleet to JSON or restarts the hold, which a join or a restart does. After a Redis outage the decision is kept for two advertise intervals while peers advertise again. `AssumeAllNodesCanRead` covers only nodes that advertise nothing; with no Streams or Pub/Sub transport it switches after the hold with nothing that can stop it. An advertisement older than its group's last read does not account for the group, so a restart under the same id is seen at once, and a node list that fails to read restarts the hold. The advertisement follows what a node's reads really do: on a custom proxy a policy that writes no non-JSON format advertises JSON only, which keeps writers on JSON, and a policy on such a proxy always writes its configured format, since raw bytes have no JSON to fall back to and could read back as a header. Advertise and node list reads run on their own intervals, and a topic's stream is forgotten when the topic is removed. An app with no serializer registered does nothing new. An app that cannot see its peers (`BroadcastEnabled` off, or no Streams or Pub/Sub topic) writes JSON unless `AssumeAllNodesCanRead` is set. `Mode = Force` writes exactly what each policy configures, as before. New metrics `cache.format.effective`, `cache.format.fallback`, `cache.format.nodes` and `cache.format.view.age`. See [choosing when a format is written](docs/concepts.md#choosing-when-a-format-is-written). +- **Binary events are written only while every live node can read them, over Streams and over Pub/Sub.** `AddMessagePackEvents(write: true)` now means that; `write: false` always writes JSON and `FormatNegotiation:Mode = Force` writes binary at once, as before. Each advertisement lists the event formats the node decodes (`er`): an app with the decoder lists JSON and binary, a node with a serializer and no decoder lists JSON, and a node before this version lists nothing and blocks through the node list. Events are decided apart from values, with the same rules: a fall back is immediate, an upgrade waits for `UpgradeHold`, the decoder always reads both formats, and the formatter reads the cached decision per event with no call to Redis and no allocation. An app with the decoder and no value serializer advertises and decides too. An old (3.0 or 3.1) node blocks binary events while it is live in the node list; its stale local entries are possible in the join window, when a Pub/Sub node that advertises nothing takes a freed subscription slot (hidden for up to the advertisement lifetime plus two views, about 180 s by default, during which it skips binary events), and when a node judged dead returns on the same source id, bounded by its own `LocalMaxExpiration`, so set a short one on old nodes before enabling `write: true`. **Pub/Sub deployments no longer always write JSON.** With no consumer groups to list, the node list sums `PUBSUB NUMSUB` of each channel over every server (primaries and replicas) and subtracts the subscriptions the advertisements claim (`sub`), which a node writes only once the server has confirmed its subscription, and writes again at once. A subscriber nobody claims keeps values and events on JSON; it takes two views in a row to count, and an upgrade needs two clean views over the same number of servers. Fewer subscribers than are claimed (a node that crashed, whose advertisement outlives it) is also unknown, so JSON after two views, and as a departure it never restarts the hold, a deficit confirmed in two views included: once the advertisement expires and the counts match for two views, the earlier decision resumes. An app with no Redis connection has no node list, and writes JSON events unless `AssumeAllNodesCanRead` or `Force`. A node that is stopping lists the subscriptions it still holds in its departing advertisement, which count as optional for its lifetime, so a slow shutdown never takes the fleet to JSON. A deficit that clears while the claim is still fresh is a join and restarts the hold; a one-view blip pauses a hold and no longer restarts it. A server the node knows about but is not connected to fails the whole view instead of being left out of the sum. The same round trip reads `PUBSUB NUMPAT`, nodes advertise the pattern subscriptions they hold (`psub`), and unclaimed patterns on the servers, another app's included, keep formats on JSON unless `AssumeAllNodesCanRead`. The channel names sent to `PUBSUB NUMSUB` carry the connection's `ChannelPrefix`. The node list sends one `PUBSUB NUMSUB` per server per view, logs a Warning once when it is refused, and treats a pattern or sharded channel strategy as uncountable, so JSON unless attested. Advertisements are written one at a time and none after the departing one. A server that refuses `PUBSUB NUMSUB` makes the count unknown, so JSON, unless `AssumeAllNodesCanRead`. New metric `Caching.Stats.FormatBlockers` carries who blocks, on its own and not on the bounded `cache.format.*` metrics; `cache.format.effective` and `cache.format.fallback` gain a `kind` tag (`value` or `event`) and the reasons `unaccounted_subscriber` and `subscriber_deficit`. See [choosing when a format is written](docs/concepts.md#choosing-when-a-format-is-written). - **An event in a known format that cannot be decoded is a topic loss.** The Streams reader acknowledges the entry and reports a loss, and counts such an entry as one it would have acted on when it checks what a new group skipped; the Pub/Sub reader expires through the coalesced dropped-event path, off the delivery thread, so it holds for caches that keep their tier over a subscription gap. The Streams reader reports once per batch. This applies to a body whose first byte is `0x02`, `0x03` or a consumer format id (`0x10`-`0x1E`, `0x1F`), such as a binary event on a node that cannot read it. Any other body that does not decode, another producer's first byte included, is still skipped as junk. - **Cache event properties are read as typed values.** The built-in event formatters parse `_version_`, the expirations, the metadata and the peer fields straight into `long`, `DateTimeOffset`, string dictionaries and byte arrays instead of a `JsonElement` that was parsed again by the value serializer, which saves a clone, a box and a second parse per property of a peer-update event. The event bytes on the wire are unchanged. A custom `ISerializerProxy` no longer sees these properties in `TryDeserialize(object)` when the built-in formatters decode the event. - **Decoded `ICacheEvent.Data.Properties` hold typed values.** From the built-in formatter and the CloudEvents fast path they are `long`, `DateTimeOffset`, `Dictionary` and byte-valued peer fields instead of `JsonElement`, so a custom subscriber that casts a property to `JsonElement` must handle those types. diff --git a/docs/concepts.md b/docs/concepts.md index ef8673ef..00d2a52e 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -536,10 +536,11 @@ A node that cannot read a format treats the value as a miss, and what happens to rolling deploy, and a serializer disposed at shutdown says nothing about the data. This departs from the design spec, which removed such a value. An `OutOfMemoryException` from a registered serializer's payload is never treated as a failed read: it propagates from `Get`, `GetOrAdd` and the hash reads instead of becoming a miss that runs the generator. A value with no header (JSON) behaves as it does with no serializer registered, so an `OutOfMemoryException` there is a logged miss. -Events follow the same order. A node that cannot decode an event with a format id in front treats it as a topic loss and -expires its local copies, so upgrade every node to 3.2.0 or later and register `AddMessagePackEvents(write: false)` -before any node uses `write: true`, and keep the JSON decoder for good: older nodes skip binary events silently. -See [MessagePack events](how-to/extending.md#messagepack-events). +Events switch by themselves too. `AddMessagePackEvents(write: true)` writes binary events only while every live node is known to read +them, and JSON before and after, so no deploy order is needed; with `FormatNegotiation:Mode` at `Force` it writes binary at once, and then the +nodes must read both first. A node that cannot decode an event with a format id in front treats it as a topic loss and expires its local copies, and +a node before 3.2.0 skips a binary event silently, so keep the JSON decoder for good. See +[choosing when a format is written](#choosing-when-a-format-is-written) and [MessagePack events](how-to/extending.md#messagepack-events). All of it shows up as factory load and churn during the deploy, never as a wrong value. Switching a policy back to JSON is safe at any time: new writes are JSON and the older values still read. @@ -547,11 +548,11 @@ to JSON is safe at any time: new writes are JSON and the older values still read ### Choosing when a format is written A policy's `Serializer.Write` names the format it *wants*. With `FormatNegotiation:Mode` at `Auto` (the default), a node writes it -only while every node that shares the Streams transport is known to read it, and writes JSON until then; this replaces the two-deploy +only while every node that shares the transport (Streams or Pub/Sub) is known to read it, and writes JSON until then; this replaces the two-deploy order for values, so a node that registers the serializer and sets `Write` can be deployed in one step. A policy that sets `Write = "json"` -is pinned to JSON, and `Mode = Force` writes exactly what each policy configures, as before. +is pinned to JSON, and `Mode = Force` writes exactly what each policy configures, as before. Binary events follow the same rule and are covered below. -How it works. An app with a serializer registered advertises what it reads, whether or not any policy of its own writes a non-JSON format and in either mode, so a node that only reads, or runs `Force`, never blocks the writers. In `Auto` it also reads the node list and decides, for every registered serializer, so a policy built at the call site that names one is held to the same decision as a registered policy. An app with no serializer registered does none of this: it advertises nothing, and so correctly counts as a node that cannot read. +How it works. An app with a serializer registered advertises what it reads, whether or not any policy of its own writes a non-JSON format and in either mode, so a node that only reads, or runs `Force`, never blocks the writers. In `Auto` it also reads the node list and decides, for every registered serializer, so a policy built at the call site that names one is held to the same decision as a registered policy. An app with no serializer registered and no binary event decoder does none of this: it advertises nothing, and so correctly counts as a node that cannot read. An app with only the decoder advertises the event formats it reads and decides events. The advertisement says what the node's reads actually do. With the default JSON proxy a read looks for a format header whatever the policy writes. With a custom proxy, a subclass of the default or `RawByteSerializerProxy`, a read looks for one only under a policy that itself writes a non-JSON format; for every other policy (and the default policy, when it is not one) the node advertises that it reads JSON only, and no writer upgrades while such a node is on the node list. A policy on such a proxy never falls back to JSON either, since its bytes are not JSON and could read back as a header: it always writes the format it names, as in `Force`. @@ -564,18 +565,74 @@ The advertisement says what the node's reads actually do. With the default JSON consumer read within `LivenessWindow` and that no fresh advertisement accounts for is a node that cannot say what it reads (a node before 3.2, for one), and it keeps every writer on JSON. A node that went quiet for longer than `LivenessWindow` no longer counts. - The format is written once every advertisement lists it, no live node is unaccounted for, and that has held for `UpgradeHold` (an hour by default) since a node - last joined or restarted (each process advertises its own id, so a restart under the same name is a join). A node that leaves never restarts the hold. A node that cannot read it, or an unaccounted node that appears, stops the writes at once. Redis being unreachable keeps the decision + last joined or restarted (each process advertises its own id, so a restart under the same name is a join). A node that leaves never restarts the hold. A node that cannot read it stops the writes at once, and so does an unaccounted node over Streams; over Pub/Sub an unaccounted node takes two views to count. Redis being unreachable keeps the decision for `MaxViewAge` since the last good read, then writes JSON, and for two advertise intervals after it returns, so peers can advertise again; an advertisement that lacks the format is never excused. All of this runs on a timer; a read or a write sends no extra command, only a read of the cached decision and its age. The hold trades speed for rollback safety. After a rollback to a build that cannot read the format, nodes miss on the values written in that format and the generator rewrites them as JSON, never a wrong value. With the default hour, an app rolled back within the hour never wrote the format at all. A shorter `UpgradeHold` switches sooner and widens that rollback window; zero switches as soon as the node list agrees. -The node list sees only nodes on the Streams transport. An app that has `BroadcastEnabled` off, or that publishes events over Pub/Sub only, cannot see the others and writes JSON; set -`FormatNegotiation:AssumeAllNodesCanRead` to attest that every node that says nothing reads the desired formats. The attestation covers nodes that advertise nothing, not one whose advertisement lacks the format, and the hold still applies. Every node judges the others by its own `AdvertiseInterval` and `LivenessWindow`, so keep them alike across the fleet. Group +An app that has `BroadcastEnabled` off, or that has neither a Streams nor a Pub/Sub topic, cannot see the others and writes JSON; set +`FormatNegotiation:AssumeAllNodesCanRead` to attest that every node that says nothing reads the desired formats. The attestation covers nodes that advertise nothing, and subscribers no advertisement accounts for, not a node whose advertisement lacks the format, and the hold still applies. Every node judges the others by its own `AdvertiseInterval` and `LivenessWindow`, so keep them alike across the fleet. Group names that are not node ids (for example another app sharing the stream key) also count as live nodes, a safe false positive. -Telemetry (`cache.format.effective`, `cache.format.fallback`, `cache.format.nodes` and `cache.format.view.age`) is emitted only while this runs, and -a node that is blocking is named in a Warning log. The advertisement needs a host that runs hosted services: without one, nothing advertises and values stay JSON. +Telemetry is emitted only while this runs; see [telemetry](#telemetry-of-the-switch). The advertisement needs a host that runs hosted services: without one, nothing advertises and values stay JSON, and events stay JSON unless `Mode` is `Force`. + +#### Events + +Events use the same node list and the same rules, decided apart from values. Each advertisement also lists the event formats the node decodes (`er`): an app with the +binary decoder registered (`AddMessagePackEvents`, either `write`) lists JSON and binary; a node with a serializer registered and no decoder lists JSON; a node before this version +lists nothing, and so counts as one that cannot say what it reads. An app with the decoder and no value serializer advertises and decides too, and lists JSON for values. + +- `AddMessagePackEvents(write: true)` means binary events once every advertisement lists binary, no live node is unaccounted for, and `UpgradeHold` has passed. The first node that + lists only JSON, or an unaccounted node, switches the events back to JSON at once. `write: false` always writes JSON and `Force` always writes binary. +- The decoder reads both formats always, so what was written before a fall back, or replayed from a stream backlog, still decodes. The formatter asks for the decision on every event + and sends no command to Redis; it costs a read of the cached decision. An app that registers nothing is unchanged, and has no wrapper at all. +- An old node (3.0 or 3.1) cannot read binary events: it skips them without a trace and keeps what it has until its own `LocalMaxExpiration`. While it is live in the node list it blocks binary events, + by the advertisement it does not write (on Streams, its consumer group; on Pub/Sub, its subscription). Its stale-L1 risk is therefore the join window, about one view interval from when it joins until the + others see it; over Pub/Sub, a node that advertises nothing and takes a freed subscription slot, which stays hidden for up to the advertisement lifetime (`LivenessWindow` plus two advertise intervals) plus two views, about 180 s by default, while binary reaches it and it skips those events, so its local tier can serve a stale value for that window; and when a node judged dead (idle past `LivenessWindow`) returns on the same source id and replays what was written meanwhile. All are bounded by that old node's own `LocalMaxExpiration`: set a short one + on old nodes, a few minutes, before enabling `write: true`. A node on this version never holds binary events back from itself; its local lifetimes are what you configured. + +#### Pub/Sub + +The Streams transport has a consumer group per node, which is how a node that advertises nothing is found. Pub/Sub has no membership, so over Pub/Sub the node list counts subscribers: + +- Each node counts the subscribers of each channel it listens on with `PUBSUB NUMSUB`, on every server of the deployment, primaries and replicas, and sums them, because a subscriber is local to the server it + connected to. Asking one server undercounts. A server this node knows about but is not connected to would hide its subscribers, so the whole view fails: it is not a partial sum, it resets the debounce + and keeps the last decision until `MaxViewAge`, then JSON. The channel names it asks for carry the connection's `ChannelPrefix`, since the server only knows the prefixed ones. +- `NUMSUB` leaves pattern subscriptions out, so the same round trip also reads `PUBSUB NUMPAT`. Each node advertises the pattern subscriptions it holds (`psub`, none unless a channel strategy makes patterns), + and more patterns on the servers than the advertisements claim is an unknown subscriber, under the same two-view rule as a surplus on a channel. `NUMPAT` counts the whole server and its unique patterns, not this app's + channels: another app on the same Redis that subscribes to a pattern (a keyspace listener, say) keeps binary events on JSON, and so does a tool left running. There is no deficit rule for patterns. Set + `AssumeAllNodesCanRead` to attest that none of them is a node that needs to read binary events. +- Each node advertises the subscriptions it holds (`sub`), once the server has confirmed them, and says so at once, not on the next interval. The order is deliberate: subscribe, then advertise. A claim made + first would hide a subscriber that says nothing. A node that is stopping writes its departing advertisement (`left=1`) with the subscriptions it still holds, and its connection closes a little later, maybe views later: + for as long as that advertisement lives, the count may include its subscription or not, either is clean, so a slow shutdown never takes the fleet to JSON. The node list sends one `PUBSUB NUMSUB` per server per view, for all + the channels in it. +- More subscribers than advertisements account for is a node that cannot say what it reads, and keeps values and events on JSON. A surplus counts only when two views in a row show it, so a node that has subscribed and not + yet advertised is absorbed. An upgrade needs two views in a row with no surplus and the same number of servers. An advertisement that lacks a format still stops the writes at once. +- Fewer subscribers than the advertisements claim is a count that does not add up too, and is unknown: JSON for values and events, after the same two views in a row. It is how a node that crashed shows, + since its advertisement outlives its connection and a claim with no subscriber behind it would hide a node that says nothing. It is a departure, not a join, so it never restarts the hold: once the + advertisement expires and the counts match again for two views, what was written before resumes at once. A node that joins meanwhile is a join, and waits for the hold, and so does a deficit that clears while the + claim is still fresh: the count rose, so something subscribed, and it advertised nothing. A blip, one view of surplus, deficit or a change in the number of servers, pauses a hold in progress and does not restart it; a + surplus confirmed in two views, or a node that lists no format, does; a deficit confirmed in two views never does, it holds the format back and keeps the hold it had earned. Each channel is judged on its own: a surplus on one channel and then on another is not two views of the same. +- When the server refuses `PUBSUB NUMSUB` (no permission, the command disabled or unknown), or a server that is up cannot answer, the count is unknown and the writes stay JSON, unless + `AssumeAllNodesCanRead` is set. The node logs a Warning once, naming the fallback, and again when it can count. A failed or partial read keeps the last decision for `MaxViewAge`, as for the Streams node list, and the two + views on either side of it are not consecutive. +- `PUBSUB NUMSUB` counts classic literal channels. A custom `IRedisChannelStrategy` that returns a pattern or a sharded channel cannot be counted, so the count is unknown for that node: JSON unless attested, logged + once with the channel names. + +Limits. A connection that half-closed keeps counting until Redis drops it, which only delays an upgrade. A node that crashed keeps the writes on JSON for as long as its advertisement lives (`LivenessWindow` plus +two advertise intervals) and for two views after it expires. A node that resubscribes after a reconnect is briefly short; two views absorb it. What remains hidden: an old node that subscribes to a channel in place of a +node that crashed, while that node's claim is still fresh, looks like the crashed node's subscription still held, and an old node that subscribes while a leaving node's advertisement is fresh looks like the leaving node still holding +its subscription. Either lasts at most the advertisement's lifetime (`LivenessWindow` plus two advertise intervals) plus two views, about 180 s by default; binary events reach that node, it skips them, and its local tier may serve a stale value until the window closes or its own `LocalMaxExpiration` passes. + +An app with no Redis connection has no node list at all, and `write: true` writes JSON events unless `FormatNegotiation:AssumeAllNodesCanRead` attests that every node reads binary (the hold applies) or `Mode` is `Force`. + +#### Telemetry of the switch + +`cache.format.effective` is 1 while a format is written and 0 while it is not, tagged `kind` (`value` or `event`) and `format`, and `policy` for a value. `cache.format.fallback` counts transitions to JSON, tagged +`reason`, `kind` and `policy`; the reasons are `missing_format`, `unaccounted_group`, `unaccounted_subscriber`, `subscriber_deficit`, `transport_unknown`, `no_view` and `view_stale`. `cache.format.nodes` and `cache.format.view.age` are the live and +blocking nodes and the age of the view. Who blocks is kept off these metrics, whose tags are bounded: the node ids, or the Pub/Sub channels (or `(patterns)`) with a surplus or a deficit, are in a Warning log and on the separate `Caching.Stats.FormatBlockers`, each time the set +changes. Those are as many values as there are nodes, so a backend that turns a metric's tags into dimensions will see them as one each: alert on the metric's value, not on its tags. A fallback of an event has no `policy` tag. See also: [how-to/extending.md#value-serializers](how-to/extending.md#value-serializers). diff --git a/docs/how-to/extending.md b/docs/how-to/extending.md index 0d1bd810..84f66ab0 100644 --- a/docs/how-to/extending.md +++ b/docs/how-to/extending.md @@ -546,7 +546,7 @@ The package depends on the exact `UiPath.Caching` version it was built with, so The serializer owns format ids `0x02` (plain) and `0x03` (LZ4); the id stored with each value says which, so a node reads both whatever its threshold. -By default (`FormatNegotiation:Mode` `Auto`) a policy that sets `Write = "messagepack"` writes it only while every node on the Streams transport is known to read it, and JSON until then, so this is safe to deploy in one step; see [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written). With `Force`, roll it out readers first. A node reads a format only once the serializer is registered on it, so register +By default (`FormatNegotiation:Mode` `Auto`) a policy that sets `Write = "messagepack"` writes it only while every node on the Streams or Pub/Sub transport is known to read it, and JSON until then, so this is safe to deploy in one step; see [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written). With `Force`, roll it out readers first. A node reads a format only once the serializer is registered on it, so register `AddMessagePackSerializer` on every node and finish that deploy before any policy sets `Write = "messagepack"`; a node that cannot read the value misses and regenerates it (see [rolling a serializer out](../concepts.md#rolling-a-serializer-out)). Setting `Write` back to `json` keeps values already stored as MessagePack readable while the default JSON proxy is in use; with a custom `ISerializerProxy` registered, headers are only recognized under a policy that writes a non-JSON format, so under `Write = "json"` those values go to your proxy and read as misses. @@ -565,15 +565,20 @@ JSON size and a peer-field event saves the 33% base64 adds. Properties it has no and read back as `JsonElement`, as they do from a JSON event, and an expiry reads back in UTC. ```csharp -b.AddMessagePackEvents(write: false); // first deploy: reads both, still writes JSON -b.AddMessagePackEvents(write: true); // second deploy: writes binary +b.AddMessagePackEvents(write: true); // reads both; writes binary once every live node can read it +b.AddMessagePackEvents(write: false); // reads both; always writes JSON ``` -Switch in two deploys, like a value serializer. Every node must run library version 3.2.0 or later, the first one that -has this formatter and the loss handling below, with `AddMessagePackEvents(write: false)` registered, before any node -uses `write: true`. A node on an older version acknowledges and skips a binary event, logging a warning for every 100th one it cannot decode and tracking telemetry, but never treating it as a loss, so it keeps what -it has until `LocalMaxExpiration`; one on 3.2.0 or later without the formatter treats the event as lost and drops its -whole local tier for that topic. Switching back to `write: false` is safe at any time. Never remove the JSON decoder: +There is no deploy order. With `write: true` a node writes JSON until the node list shows that every node reads binary events, for the +`UpgradeHold` (an hour by default) after the last node joined, and writes JSON again as soon as a node that cannot read them appears. Each node +advertises the event formats it decodes: one with this formatter registered lists binary, one without lists JSON only, and a node before +3.2.0 lists nothing and is treated as unable to read. See [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written), +including how a Pub/Sub deployment counts subscribers. A node on an older version acknowledges and skips a binary event, logging a warning for every 100th one it cannot decode and tracking telemetry, but never treating it as a loss, so it keeps +what it has until `LocalMaxExpiration`; one on 3.2.0 or later without the formatter treats the event as lost and drops its whole local tier for that +topic. An old node blocks binary events while it is live in the node list, so what is left is the window a join leaves open, about one view interval; a Pub/Sub node that advertises nothing and takes a freed subscription slot, which stays hidden for up to the advertisement lifetime plus two views (about 180 s by default), during which it skips binary events and may serve a stale local value; and a node judged dead that returns on the same source id: the +stale local entry it keeps is bounded by that old node's own `LocalMaxExpiration`. Set a short one, a few minutes, on 3.0 and 3.1 nodes before enabling `write: true`. + +`FormatNegotiation:Mode` `Force` writes binary at once with `write: true`: then every node must read both first, as before. An app with no Redis connection has no node list to read, and `write: true` writes JSON events unless `FormatNegotiation:AssumeAllNodesCanRead` attests that every node reads binary or `Mode` is `Force`. Switching back to `write: false` is safe at any time. Never remove the JSON decoder: a restarted node replays its stream backlog, and a JSON entry it could not read is skipped without counting as a loss, so the node would keep what that entry invalidated. The setting is app-wide, not per topic. @@ -592,7 +597,7 @@ read. A map's initial capacity is capped at 16 entries, so allocation follows th The integer event type values on the wire (`CacheSet` 0, `CacheRefreshed` 1, `CacheRemoved` 2, `TopicInvalidated` 3) are fixed and only ever appended to. A node that reads a type number it does not know refuses the event, which is a -loss, so a release that adds a type needs the same two deploys. +loss, so a release that adds a type needs every node on it before any node writes binary events: use `Force` only after that, and in `Auto` upgrade every node before the hold has passed. ## Swapping the default factories diff --git a/docs/reference/settings.md b/docs/reference/settings.md index de885079..246049c4 100644 --- a/docs/reference/settings.md +++ b/docs/reference/settings.md @@ -215,13 +215,13 @@ Per-topic overrides: add entries to `Topics[]` under `Broadcast:RedisPubSub`. Ea | `ReadScriptsEnabled` | `bool` | `false` | Per-provider | By default a value and its expiration are read with a pipelined `GET` and `PTTL` (one round trip, no transaction); the pair is not atomic, so a key that expires between the two reads as a value without a TTL. Set to `true` to read on Redis 7 and later with one `EVAL_RO` script instead (a batch read runs one script for all its keys; less allocation, but several times the server time of a plain read on Azure Managed Redis). The connection's user then needs `EVAL_RO`/`EVALSHA_RO` (ACL category `@scripting`); if the first script fails for any reason, or the server later refuses one (`NOPERM`, or a disabled or renamed command), reads switch to the `GET`/`PTTL` pair for good with one warning per cache instance, so a server or proxy that cannot run Lua never turns reads into misses. Before Redis 7 no script is sent. Command statistics then show `EVAL_RO` instead of `GET`/`PTTL`. | | `ReadFrom` | `RedisReadFrom` | `Master` | Per-provider | `Master` sends reads to the primary (`PreferMaster`); `Replica` sends them to a replica when one exists (`PreferReplica`). On Azure Redis 6 a replica round trip measured about 3 times the primary's (1.6 ms against 0.55 ms), and a replica can still hold a key the primary has already removed, so a read right after a write or an invalidation could return, or refill the entry with, stale data. With `Replica`, a key this node wrote or removed (counted from when the write completes), or heard an invalidation for through a live local entry, within `MasterReadWindow` still reads from the primary, as does every read for that window after a lost-events clear on a node holding local entries and, if more than 10,000 keys were touched within one window, every read until the window passes. A key with no local entry is not guarded against another node's write. Turn it on only to take read load off the primary. Writes and refreshes always go to the primary. | | `MasterReadWindow` | `TimeSpan` | `00:00:01` | Per-provider | With `ReadFrom` set to `Replica`, how long a key this node wrote, removed or heard invalidated keeps reading from the master. Must be positive, and must exceed the expected replica lag: a replica that is further behind can still return the old value once the window ends. A fire-and-forget TTL write (`AwaitRefresh` off) starts the window when it is queued, not when the server applies it. Measured on a monotonic clock. Ignored with `Master`. | -| `FormatNegotiation:Mode` | `FormatNegotiationMode` | `Auto` | Per-provider | `Auto` decides which formats this node writes: a registered serializer is written only while every node on the Streams transport is known to read it, whichever policy asks for it (a registered policy or one built at the call site), and JSON otherwise. `Force` writes exactly what each policy configures, with no node list and no decision. A node with a serializer registered advertises the formats it reads in either mode, so a `Force` node does not keep an `Auto` neighbour on JSON, unless the app has no Streams transport (see `AssumeAllNodesCanRead`), where a node neither advertises nor reads advertisements. A policy with `Write = "json"` is never affected, and neither is one on a custom `ISerializerProxy` (or a subclass of the default, or `RawByteSerializerProxy`): its bytes are not JSON, so there is nothing safe to fall back to, and it always writes the format the policy names, as `Force` does. See [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written). | +| `FormatNegotiation:Mode` | `FormatNegotiationMode` | `Auto` | Per-provider | `Auto` decides which formats this node writes: a registered serializer, and binary events when `AddMessagePackEvents(write: true)` is registered, are written only while every node on the Streams or Pub/Sub transport is known to read them, whichever policy asks for a serializer (a registered policy or one built at the call site), and JSON otherwise. `Force` writes exactly what each policy configures, and binary events with `write: true`, with no node list and no decision. A node with a serializer or the binary event decoder registered advertises the formats it reads in either mode, so a `Force` node does not keep an `Auto` neighbour on JSON, unless the app has neither a Streams nor a Pub/Sub transport (see `AssumeAllNodesCanRead`), where a node neither advertises nor reads advertisements. A policy with `Write = "json"` is never affected, and neither is one on a custom `ISerializerProxy` (or a subclass of the default, or `RawByteSerializerProxy`): its bytes are not JSON, so there is nothing safe to fall back to, and it always writes the format the policy names, as `Force` does. See [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written). | | `FormatNegotiation:AdvertiseInterval` | `TimeSpan` | `00:00:15` | Per-provider | How often this node rewrites its advertisement (the formats it reads) in Redis, stamped with the server's clock. Every node judges the others' advertisements by its own settings, so set `AdvertiseInterval`, `ViewInterval` and `LivenessWindow` alike across the fleet. An advertisement counts for `LivenessWindow` plus two of these, so it outlives the consumer group of a node that crashed. Must be positive. | | `FormatNegotiation:ViewInterval` | `TimeSpan` | `00:00:15` | Per-provider | How often this node reads the advertisements and the Streams consumer groups; it advertises first, on the same tick. Must be positive. | | `FormatNegotiation:LivenessWindow` | `TimeSpan` | `00:02:00` | Per-provider | A consumer group no advertisement accounts for (a node before 3.2) counts as a live node, and blocks the switch, while its consumer read within this window. A group no consumer ever read from counts as nobody. An advertisement accounts for its group only if it is at least as fresh as the group's last read (within two `AdvertiseInterval`s), so a node restarted under the same id is not hidden by its previous run's advertisement. Must be positive. | | `FormatNegotiation:UpgradeHold` | `TimeSpan` | `01:00:00` | Per-provider | How long every node must have been accounted for before a format is written, counted from the last time a node joined or restarted (a node that leaves never restarts it). Zero switches at once; negative is rejected. A shorter hold switches sooner, but if the app is rolled back after it, nodes that cannot read the format miss and rewrite values as JSON, never a wrong value. A longer one means an app rolled back within it never wrote the format, so the older nodes see no misses. A node that cannot read the format stops the writes at once, whatever the hold. A node list that fails to read (other than a group that vanished mid-read) restarts the hold. | | `FormatNegotiation:MaxViewAge` | `TimeSpan` | `00:02:00` | Per-provider | While Redis cannot be read, the last decision stands for this long since the last good read, then values are written as JSON; a decision older than this is never used, even if a read hangs. After an outage shorter than this, the decision is also kept for two `AdvertiseInterval`s of successful ticks, so peers can advertise again, unless an advertisement says a node cannot read the format. Must be positive. | -| `FormatNegotiation:AssumeAllNodesCanRead` | `bool` | `false` | Per-provider | Attests that the nodes that say nothing read the desired formats: a live Streams consumer group with no advertisement, and, for an app with `BroadcastEnabled` off or without a Streams topic (Pub/Sub only), the nodes it cannot see at all. For an app that can list its peers, an advertisement that lacks the format still stops the writes. Without a Streams transport the node neither advertises nor reads advertisements, so nothing can stop the switch once `UpgradeHold` has passed, counted from the first decision. Without the setting such an app writes JSON. | +| `FormatNegotiation:AssumeAllNodesCanRead` | `bool` | `false` | Per-provider | Attests that the nodes that say nothing read the desired formats: a live Streams consumer group with no advertisement, a Pub/Sub subscriber no advertisement accounts for (a pattern subscriber included, which `PUBSUB NUMPAT` counts for the whole server, other apps' too), a Pub/Sub deployment where `PUBSUB NUMSUB` is refused, and, for an app with `BroadcastEnabled` off, with neither a Streams nor a Pub/Sub topic, or with no Redis connection, the nodes it cannot see at all. For an app that can list its peers, an advertisement that lacks the format still stops the writes. Without a Streams or Pub/Sub transport the node neither advertises nor reads advertisements, so nothing can stop the switch once `UpgradeHold` has passed, counted from the first decision. Without the setting such an app writes JSON. | *Code-only seams:* `EntryFactory`, `CacheKeyStrategy`, `RedisKeyStrategyFactory`. diff --git a/samples/UiPath.Caching.Sample/appsettings.all.json b/samples/UiPath.Caching.Sample/appsettings.all.json index fb596f4a..94d88bb1 100644 --- a/samples/UiPath.Caching.Sample/appsettings.all.json +++ b/samples/UiPath.Caching.Sample/appsettings.all.json @@ -377,7 +377,7 @@ // MasterReadWindow: with ReadFrom Replica, how long a just-written or invalidated key keeps reading from the master; must exceed the replica lag "MasterReadWindow": "0:00:01", "FormatNegotiation": { - // Mode: Auto writes a registered serializer's format only while every node on the Streams transport is known to read it; Force writes exactly what each policy configures. Either way the node advertises what it reads + // Mode: Auto writes a registered serializer's format, and binary events with AddMessagePackEvents(write: true), only while every node on the Streams or Pub/Sub transport is known to read it; Force writes exactly what is configured. Either way the node advertises what it reads "Mode": "Auto", // AdvertiseInterval: how often this node refreshes its advertisement; an advertisement lives LivenessWindow plus two of these. Keep it alike across the fleet "AdvertiseInterval": "0:00:15", @@ -389,7 +389,7 @@ "UpgradeHold": "1:00:00", // MaxViewAge: how long a decision survives a node list that cannot be read, before writes fall back to JSON "MaxViewAge": "0:02:00", - // AssumeAllNodesCanRead: attest that nodes that advertise nothing (live Streams groups without an advertisement; broadcast off, Pub/Sub only) read the desired formats; an advertisement that lacks the format still blocks + // AssumeAllNodesCanRead: attest that nodes that advertise nothing (live Streams groups without an advertisement, Pub/Sub subscribers nobody advertises or a refused PUBSUB NUMSUB, broadcast off) read the desired formats; an advertisement that lacks the format still blocks "AssumeAllNodesCanRead": false } }, diff --git a/src/UiPath.Caching.MessagePack/MessagePackCachingBuilderExtensions.cs b/src/UiPath.Caching.MessagePack/MessagePackCachingBuilderExtensions.cs index abe94e29..27cb82b0 100644 --- a/src/UiPath.Caching.MessagePack/MessagePackCachingBuilderExtensions.cs +++ b/src/UiPath.Caching.MessagePack/MessagePackCachingBuilderExtensions.cs @@ -2,6 +2,7 @@ using Microsoft.Extensions.DependencyInjection.Extensions; using UiPath.Caching.Broadcast; using UiPath.Caching.Config; +using UiPath.Caching.Negotiation; namespace UiPath.Caching.MessagePack; @@ -24,29 +25,32 @@ public static ICachingBuilder AddMessagePackSerializer( return builder.AddLibraryValueSerializer(name, _ => new MessagePackValueSerializer(options)); } - /// Reads binary and JSON events; false still writes JSON (the first deploy), true writes binary (the second). + /// Reads binary and JSON events; true writes binary once every live node can read it, false always writes JSON. /// /// Wraps the JSON formatter that would otherwise be registered, whether AddCloudEvents is called before or after it, so the JSON /// wire is what it was. Keep the JSON decoder: a restarted node replays a stream backlog that may hold JSON. A second call replaces the first one's setting. + /// With write: true and FormatNegotiation.Mode Auto the node writes JSON until the node list shows that every node reads binary + /// events, and again as soon as one does not; Force writes binary at once. /// public static ICachingBuilder AddMessagePackEvents(this ICachingBuilder builder, bool write) { ArgumentNullException.ThrowIfNull(builder); - if (builder.Services.FirstOrDefault(d => d.ServiceType == typeof(EventSettings))?.ImplementationInstance is EventSettings existing) + if (builder.Services.FirstOrDefault(d => d.ServiceType == typeof(BinaryEventSettings))?.ImplementationInstance is BinaryEventSettings existing) { existing.Write = write; return builder; } - var settings = new EventSettings(write); + var settings = new BinaryEventSettings(write); builder.Services.AddSingleton(settings); + builder.Services.AddSingleton(sp => new LazyBinaryEvents(settings, sp.GetService)); // At completion, once every registration that supplies the JSON formatter has run, whichever order they were called in. - builder.RegisterOnCompleteCallback(typeof(MessagePackCachingBuilderExtensions), b => WrapJsonFormatter(b.Services, settings)); + builder.RegisterOnCompleteCallback(typeof(MessagePackCachingBuilderExtensions), b => WrapJsonFormatter(b.Services)); return builder; } - private static void WrapJsonFormatter(IServiceCollection services, EventSettings settings) + private static void WrapJsonFormatter(IServiceCollection services) { var json = services.LastOrDefault(d => d.ServiceType == typeof(IEventFormatterProxy) && !d.IsKeyedService); if (json is not null) @@ -59,7 +63,7 @@ private static void WrapJsonFormatter(IServiceCollection services, EventSettings services.AddSingleton>(sp => new CompositeEventFormatter( json is null ? new CacheEventFormatter() : sp.GetRequiredKeyedService>(InnerJsonKey), new MessagePackEventFormatter(), - settings.Write)); + sp.GetRequiredService())); } private static ServiceDescriptor Keyed(ServiceDescriptor descriptor) @@ -74,9 +78,4 @@ private static ServiceDescriptor Keyed(ServiceDescriptor descriptor) ? ServiceDescriptor.DescribeKeyed(type, InnerJsonKey, (sp, _) => factory(sp), ServiceLifetime.Singleton) : ServiceDescriptor.DescribeKeyed(type, InnerJsonKey, descriptor.ImplementationType!, ServiceLifetime.Singleton); } - - private sealed class EventSettings(bool write) - { - public bool Write { get; set; } = write; - } } diff --git a/src/UiPath.Caching/Broadcast/CompositeEventFormatter.cs b/src/UiPath.Caching/Broadcast/CompositeEventFormatter.cs index b4c46b28..13875a3b 100644 --- a/src/UiPath.Caching/Broadcast/CompositeEventFormatter.cs +++ b/src/UiPath.Caching/Broadcast/CompositeEventFormatter.cs @@ -1,16 +1,23 @@ +using UiPath.Caching.Negotiation; + namespace UiPath.Caching.Broadcast; -/// Reads a body by its first byte, the format id of a binary event or else JSON; writes binary only when told to. +/// Reads a body by its first byte, the format id of a binary event or else JSON; writes binary only while says so, asked on every event. internal sealed class CompositeEventFormatter( IEventFormatterProxy json, IEventFormatterProxy binary, - bool writeBinary) : IEventFormatterProxy, IEventSourceMatcherProvider + IBinaryEvents writes) : IEventFormatterProxy, IEventSourceMatcherProvider { + public CompositeEventFormatter(IEventFormatterProxy json, IEventFormatterProxy binary, bool writeBinary) + : this(json, binary, writeBinary ? FixedBinaryEvents.Always : FixedBinaryEvents.Never) + { + } + public ICacheEvent? Decode(ReadOnlyMemory body) => IsBinary(body) ? binary.Decode(body) : json.Decode(body); public ReadOnlyMemory Encode(ICacheEvent @event) => - writeBinary ? binary.Encode(@event) : json.Encode(@event); + writes.Writes ? binary.Encode(@event) : json.Encode(@event); /// Each side's matcher, picked by the first byte; a side without one leaves its bodies to be decoded and compared. public IEventSourceMatcher? CreateSourceMatcher(Uri source) diff --git a/src/UiPath.Caching/Broadcast/Redis/RedisPubSubSubjectWriter.cs b/src/UiPath.Caching/Broadcast/Redis/RedisPubSubSubjectWriter.cs index b5597b95..2cb4ca48 100644 --- a/src/UiPath.Caching/Broadcast/Redis/RedisPubSubSubjectWriter.cs +++ b/src/UiPath.Caching/Broadcast/Redis/RedisPubSubSubjectWriter.cs @@ -71,6 +71,21 @@ public RedisPubSubSubjectWriter( _subscribeTimer = new Timer(Subscribe, null, _timerDueTime, _timerPeriod); } + /// Raised once a subscription is in place, after the server has confirmed it. + internal event Action? Subscribed; + + /// Whether the server has confirmed the subscription and no subscription connection is known to be down. + internal bool IsSubscribed + { + get + { + lock (_gate) + { + return !_disposed && _activeGeneration == _generation && _subscriptionsDown.Count == 0 && !_unnamedFailure; + } + } + } + public void Dispose() { Action? unsubscribe; @@ -179,6 +194,8 @@ private void Subscribe(object? state) { _onSubscriptionGap(); } + + Subscribed?.Invoke(); } catch (Exception ex) { diff --git a/src/UiPath.Caching/Broadcast/Redis/RedisPubSubTopic.cs b/src/UiPath.Caching/Broadcast/Redis/RedisPubSubTopic.cs index a6d2865a..4ae65e6a 100644 --- a/src/UiPath.Caching/Broadcast/Redis/RedisPubSubTopic.cs +++ b/src/UiPath.Caching/Broadcast/Redis/RedisPubSubTopic.cs @@ -47,10 +47,24 @@ public RedisPubSubTopic( _dispatcher = new EventDispatcher(topicKey, channel, _subject, _logger, _stopTokenSource.Token); } + internal event Action? Subscribed + { + add => _subscriber.Subscribed += value; + remove => _subscriber.Subscribed -= value; + } + public TopicKey TopicKey { get; } public EventHandler? OnDisposed { get; set; } + internal string ChannelName => _redisChannel.ToString(); + + internal bool IsPattern => _redisChannel.IsPattern; + + internal bool Countable => !_redisChannel.IsPattern && !_redisChannel.IsSharded; + + internal bool IsSubscribed => _subscriber.IsSubscribed; + public IDisposable Subscribe(IObserver observer) { // Publications are missed while the Redis subscription is not in place, so the mark makes the next one that goes in diff --git a/src/UiPath.Caching/Broadcast/Redis/RedisPubSubTopicProvider.cs b/src/UiPath.Caching/Broadcast/Redis/RedisPubSubTopicProvider.cs index bdc36ba8..0ad94c34 100644 --- a/src/UiPath.Caching/Broadcast/Redis/RedisPubSubTopicProvider.cs +++ b/src/UiPath.Caching/Broadcast/Redis/RedisPubSubTopicProvider.cs @@ -1,9 +1,10 @@ +using UiPath.Caching.Negotiation; using UiPath.Caching.Policies; using UiPath.Caching.Telemetry; namespace UiPath.Caching.Broadcast.Redis; -public class RedisPubSubTopicProvider : RedisTopicProviderBase +public class RedisPubSubTopicProvider : RedisTopicProviderBase, IPubSubSubscriptions { private readonly RedisPubSubTopicOptions _options; private readonly IEventFormatterProxy _formatter; @@ -38,15 +39,32 @@ public RedisPubSubTopicProvider( Enabled = _options.Enabled; } + private event Action? SubscriptionConfirmed; + public override string Name => KnownTopicNames.RedisPubSub; public override bool Enabled { get; } + IReadOnlyList IPubSubSubscriptions.Channels() => + [.. Topics().Where(topic => topic.Countable).Select(topic => topic.ChannelName).Distinct(StringComparer.Ordinal)]; + + IReadOnlyList IPubSubSubscriptions.Uncountable() => + [.. Topics().Where(topic => !topic.Countable).Select(topic => topic.ChannelName).Distinct(StringComparer.Ordinal)]; + + // Each channel once: a connection holds one subscription to a channel, whatever the topics listening on it. + IReadOnlyDictionary IPubSubSubscriptions.Confirmed() => + Topics().Where(topic => topic.Countable && topic.IsSubscribed).Select(topic => topic.ChannelName).Distinct(StringComparer.Ordinal).ToDictionary(channel => channel, _ => 1, StringComparer.Ordinal); + + int IPubSubSubscriptions.Patterns() => + Topics().Where(topic => topic.IsPattern && topic.IsSubscribed).Select(topic => topic.ChannelName).Distinct(StringComparer.Ordinal).Count(); + + void IPubSubSubscriptions.WhenConfirmed(Action handler) => SubscriptionConfirmed += handler; + protected override ITopic CreateInternalTopic(TopicKey topicKey) { var resolved = ResolveOptions(topicKey) ?? _options; var strategy = resolved.RedisChannelStrategy ?? _defaultChannelStrategy; - return new RedisPubSubTopic( + var topic = new RedisPubSubTopic( topicKey, _sourceUri, ConnectionState, @@ -58,6 +76,8 @@ protected override ITopic CreateInternalTopic(TopicKey topicKey) resolved, LoggerFactory.Create>(), _stopTokenSource.Token); + topic.Subscribed += () => SubscriptionConfirmed?.Invoke(); + return topic; } protected override void Dispose(bool disposing) @@ -76,6 +96,8 @@ protected override void Dispose(bool disposing) base.Dispose(disposing); } + private IEnumerable> Topics() => CreatedTopics().OfType>(); + private RedisPubSubTopicOptions? ResolveOptions(TopicKey topicKey) => _registry.Resolve(topicKey, _options.Clone, _logger); } diff --git a/src/UiPath.Caching/Broadcast/Redis/RedisTopicProviderBase.cs b/src/UiPath.Caching/Broadcast/Redis/RedisTopicProviderBase.cs index d2f26487..cca08178 100644 --- a/src/UiPath.Caching/Broadcast/Redis/RedisTopicProviderBase.cs +++ b/src/UiPath.Caching/Broadcast/Redis/RedisTopicProviderBase.cs @@ -91,6 +91,10 @@ public void Dispose() GC.SuppressFinalize(this); } + /// The topics made so far and not removed, for the node node list. + internal IEnumerable> CreatedTopics() => + _topics.Values.Where(topic => topic.IsValueCreated).Select(topic => topic.Value); + protected virtual void Dispose(bool disposing) { if (!_disposed) diff --git a/src/UiPath.Caching/Config/CachingBuilder.cs b/src/UiPath.Caching/Config/CachingBuilder.cs index d0c769eb..8c2a25db 100644 --- a/src/UiPath.Caching/Config/CachingBuilder.cs +++ b/src/UiPath.Caching/Config/CachingBuilder.cs @@ -58,7 +58,7 @@ internal void Complete() ServiceDescriptor.Singleton, CacheKeyCasingSeeder>()); Services.TryAddSingleton>(sp => new SystemJsonByteSerializerProxy(sp.GetService())); Services.TryAddSingleton(ValueSerializerRegistration.BuildRegistry); - if (Services.Any(d => d.ServiceType == typeof(ValueSerializerRegistration))) + if (Services.Any(d => d.ServiceType == typeof(ValueSerializerRegistration) || d.ServiceType == typeof(BinaryEventSettings))) { Services.TryAddSingleton(FormatNegotiation.Create); Services.TryAddEnumerable(ServiceDescriptor.Singleton()); diff --git a/src/UiPath.Caching/Metrics.cs b/src/UiPath.Caching/Metrics.cs index 104d808e..f8827a07 100644 --- a/src/UiPath.Caching/Metrics.cs +++ b/src/UiPath.Caching/Metrics.cs @@ -19,6 +19,7 @@ public static class Metrics public const string FormatFallback = "cache.format.fallback"; public const string FormatViewAge = "cache.format.view.age"; public const string FormatNodes = "cache.format.nodes"; + public const string FormatBlockers = Prefix + "FormatBlockers"; public const string TopicName = nameof(TopicName); public const string SequenceNumber = nameof(SequenceNumber); diff --git a/src/UiPath.Caching/Negotiation/BinaryEventSettings.cs b/src/UiPath.Caching/Negotiation/BinaryEventSettings.cs new file mode 100644 index 00000000..42a8afee --- /dev/null +++ b/src/UiPath.Caching/Negotiation/BinaryEventSettings.cs @@ -0,0 +1,9 @@ +namespace UiPath.Caching.Negotiation; + +/// What the binary event formatter registered: this node decodes binary events, and whether it is set to write them. +internal sealed class BinaryEventSettings(bool write) +{ + public static string Token => NodeAdvertisement.Token(CacheValueFormats.MessagePack); + + public bool Write { get; set; } = write; +} diff --git a/src/UiPath.Caching/Negotiation/DesiredEvents.cs b/src/UiPath.Caching/Negotiation/DesiredEvents.cs new file mode 100644 index 00000000..20db49ae --- /dev/null +++ b/src/UiPath.Caching/Negotiation/DesiredEvents.cs @@ -0,0 +1,4 @@ +namespace UiPath.Caching.Negotiation; + +/// The event format a node is set to write, and the tokens a node must read to be sent it. +internal sealed record DesiredEvents(IReadOnlyList Tokens); diff --git a/src/UiPath.Caching/Negotiation/FixedBinaryEvents.cs b/src/UiPath.Caching/Negotiation/FixedBinaryEvents.cs new file mode 100644 index 00000000..005ea99e --- /dev/null +++ b/src/UiPath.Caching/Negotiation/FixedBinaryEvents.cs @@ -0,0 +1,13 @@ +namespace UiPath.Caching.Negotiation; + +/// An answer that does not depend on the node list: for a node that writes what it is configured to, for one set to write JSON. +internal sealed class FixedBinaryEvents : IBinaryEvents +{ + private FixedBinaryEvents(bool writes) => Writes = writes; + + public static FixedBinaryEvents Always { get; } = new(true); + + public static FixedBinaryEvents Never { get; } = new(false); + + public bool Writes { get; } +} diff --git a/src/UiPath.Caching/Negotiation/FormatNegotiation.cs b/src/UiPath.Caching/Negotiation/FormatNegotiation.cs index a1085b99..2e4c5cfe 100644 --- a/src/UiPath.Caching/Negotiation/FormatNegotiation.cs +++ b/src/UiPath.Caching/Negotiation/FormatNegotiation.cs @@ -24,6 +24,11 @@ internal sealed class FormatNegotiation /// The streams this node's topics read; when the app has no Streams transport, so the node list is blind. public Func>? StreamKeys { get; init; } + /// The channels this node's topics listen on, and which are confirmed; when the app has no Pub/Sub transport. + public IPubSubSubscriptions? PubSub { get; init; } + + public PubSubNodeList? Subscribers { get; init; } + public FormatNegotiationOptions Options { get; init; } = new(); public TimeProvider Clock { get; init; } = TimeProvider.System; @@ -32,16 +37,25 @@ internal sealed class FormatNegotiation public bool IsActive => Advertisement is not null; + /// The node reads the node list off a transport it can see; otherwise only the operator's word can switch a format. + public bool HasTransport => StreamKeys is not null || PubSub is not null; + public static FormatNegotiation Create(IServiceProvider services) { var redisOptions = services.GetRequiredService>().Value; var options = redisOptions.FormatNegotiation; var registry = services.GetRequiredService(); - if (registry.IsEmpty || services.GetService() is not { } connector) + var events = services.GetService(); + if (registry.IsEmpty && events is null) { return Inactive; } + if (services.GetService() is not { } connector) + { + return events is { Write: true } ? WithoutNodeList(services, options) : Inactive; + } + options.Validate(); var policies = services.GetRequiredService(); var cacheOptions = services.GetRequiredService>().Value; @@ -50,17 +64,53 @@ public static FormatNegotiation Create(IServiceProvider services) var streams = cacheOptions.BroadcastEnabled ? services.GetServices().OfType().FirstOrDefault(p => p.Enabled) : null; + var pubSub = cacheOptions.BroadcastEnabled + ? services.GetServices().OfType().FirstOrDefault(p => p.Enabled) + : null; var library = typeof(FormatNegotiation).Assembly.GetCustomAttribute()?.InformationalVersion ?? string.Empty; return new() { - Negotiator = options.Mode == FormatNegotiationMode.Force + // A node with nothing to decide, in Force or with no serializer and no binary events to write, only advertises. + Negotiator = options.Mode == FormatNegotiationMode.Force || (registry.IsEmpty && events is not { Write: true }) ? null - : new FormatNegotiator(NegotiationPlan.Desired(registry, policies), options, clock, services.GetRequiredService(), loggers.CreateLogger()), - Advertisement = NegotiationPlan.Advertisement(registry, policies, new ValueReader(services.GetRequiredService>(), registry), library), + : new FormatNegotiator( + NegotiationPlan.Desired(registry, policies), + options, + clock, + services.GetRequiredService(), + loggers.CreateLogger(), + events is { Write: true } ? new DesiredEvents([BinaryEventSettings.Token]) : null), + Advertisement = NegotiationPlan.Advertisement(registry, policies, new ValueReader(services.GetRequiredService>(), registry), library, readsBinaryEvents: events is not null), NodeId = (cacheOptions.SourceUri ?? CacheOptions.MachineUri).ToString().ToLowerInvariant(), Store = new NodeListStore(connector, NodeListStore.KeyFor(cacheOptions, redisOptions)), NodeList = new StreamNodeList(new RedisStreamGroupSource(connector), options), StreamKeys = streams is null ? null : streams.StreamKeys, + PubSub = pubSub, + Subscribers = pubSub is null ? null : new PubSubNodeList(new RedisSubscriberSource(connector)), + Options = options, + Clock = clock, + Logger = loggers.CreateLogger(), + }; + } + + /// + /// Whether this node writes binary events: never when it is not set to, as configured in , as the node list allows + /// in , and with no node list to ask only on the operator's word. + /// + public IBinaryEvents EventWrites(BinaryEventSettings settings) => + !settings.Write ? FixedBinaryEvents.Never : Negotiator is { } negotiator ? negotiator : IsActive || Options.Mode == FormatNegotiationMode.Force ? FixedBinaryEvents.Always : FixedBinaryEvents.Never; + + // No Redis connection, so no node list and nothing to advertise to: the node writes JSON events until the operator attests, or forces them. + private static FormatNegotiation WithoutNodeList(IServiceProvider services, FormatNegotiationOptions options) + { + options.Validate(); + var clock = services.GetRequiredService(); + var loggers = services.GetRequiredService(); + return new() + { + Negotiator = options.Mode == FormatNegotiationMode.Force + ? null + : new FormatNegotiator([], options, clock, services.GetRequiredService(), loggers.CreateLogger(), new DesiredEvents([BinaryEventSettings.Token])), Options = options, Clock = clock, Logger = loggers.CreateLogger(), diff --git a/src/UiPath.Caching/Negotiation/FormatNegotiationService.cs b/src/UiPath.Caching/Negotiation/FormatNegotiationService.cs index 0490a75b..e1d35a4e 100644 --- a/src/UiPath.Caching/Negotiation/FormatNegotiationService.cs +++ b/src/UiPath.Caching/Negotiation/FormatNegotiationService.cs @@ -5,29 +5,43 @@ namespace UiPath.Caching.Negotiation; /// /// Advertises this node and, where it decides what it writes, reads the node list, on timers off the read and write paths. It advertises first /// when it starts, and writes a departing advertisement when it stops, so the nodes that outlive it count it as gone and not as unaccounted for. +/// Over Pub/Sub the order is the other way round for a subscription: a node says it is subscribed only once the server has confirmed it, and says so +/// at once, because a claim made before the subscription would hide a subscriber that says nothing. Advertisements are written one at a time, each +/// from what the node holds when its turn comes, and none after the departing one. /// internal sealed class FormatNegotiationService(FormatNegotiation negotiation) : IHostedService { private static readonly TimeSpan Budget = TimeSpan.FromSeconds(2); private readonly CancellationTokenSource _stop = new(); + private readonly SemaphoreSlim _advertising = new(1, 1); private Task[] _loops = []; private int _stopped; + private int _started; + private int _left; public async Task StartAsync(CancellationToken cancellationToken) { if (!negotiation.IsActive) { + // An app with no Redis connection advertises nowhere, but may still decide its binary events on the operator's word. + if (negotiation.Negotiator is not null) + { + StartBlind(); + } + return; } var options = negotiation.Options; - if (negotiation.StreamKeys is null) + if (!negotiation.HasTransport) { StartBlind(); return; } + negotiation.PubSub?.WhenConfirmed(AdvertiseConfirmed); + using (var startup = new CancellationTokenSource(Budget, negotiation.Clock)) using (var linked = CancellationTokenSource.CreateLinkedTokenSource(startup.Token, cancellationToken)) { @@ -38,6 +52,7 @@ public async Task StartAsync(CancellationToken cancellationToken) _loops = negotiation.Negotiator is null ? [Task.Run(() => Loop(options.AdvertiseInterval, AdvertiseAsync), CancellationToken.None)] : [Task.Run(ScheduledLoop, CancellationToken.None)]; + Volatile.Write(ref _started, 1); } public async Task StopAsync(CancellationToken cancellationToken) @@ -58,17 +73,31 @@ public async Task StopAsync(CancellationToken cancellationToken) } _stop.Dispose(); - if (negotiation.IsActive && negotiation.StreamKeys is not null) + if (negotiation.IsActive && negotiation.HasTransport) { try { + // Behind any advertisement still being written, so a late one cannot land after the departing one and make this node look alive. using var timeout = new CancellationTokenSource(Budget, negotiation.Clock); - await negotiation.Store.WriteAsync(negotiation.NodeId, negotiation.Advertisement! with { Left = true }, timeout.Token).ConfigureAwait(false); + await _advertising.WaitAsync(timeout.Token).ConfigureAwait(false); + try + { + Volatile.Write(ref _left, 1); + await negotiation.Store.WriteAsync(negotiation.NodeId, Departing(), timeout.Token).ConfigureAwait(false); + } + finally + { + _advertising.Release(); + } } catch (Exception ex) when (ex is not OutOfMemoryException) { negotiation.Logger.LogDebug(ex, "This node's departing advertisement could not be written; its advertisement expires on its own."); } + finally + { + Volatile.Write(ref _left, 1); + } } } @@ -86,7 +115,23 @@ internal async Task AdvertiseAsync(CancellationToken cancellationToken) { try { - await negotiation.Store.WriteAsync(negotiation.NodeId, negotiation.Advertisement!, cancellationToken).ConfigureAwait(false); + await _advertising.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + if (Volatile.Read(ref _left) == 1) + { + return; + } + + // The subscriptions are read here, with the lock held, not when the service was built: what a node advertises is what it holds now, and a + // write that was started earlier cannot land over one started later. + var advertisement = negotiation.PubSub is { } pubSub ? negotiation.Advertisement! with { Subscriptions = pubSub.Confirmed(), PatternSubscriptions = pubSub.Patterns() } : negotiation.Advertisement!; + await negotiation.Store.WriteAsync(negotiation.NodeId, advertisement, cancellationToken).ConfigureAwait(false); + } + finally + { + _advertising.Release(); + } } catch (Exception ex) when (ex is not OutOfMemoryException && !Stopping()) { @@ -106,16 +151,22 @@ internal async Task RefreshAsync(CancellationToken cancellationToken) var all = await negotiation.Store.ReadAsync(cancellationToken).ConfigureAwait(false); var current = all.Where(a => a.Age <= fresh).ToList(); var readers = current.Where(a => !a.Advertisement.Left).ToList(); - var streams = negotiation.StreamKeys!(); + var streams = negotiation.StreamKeys?.Invoke() ?? []; + var channels = negotiation.PubSub?.Channels() ?? []; + var uncountable = negotiation.PubSub?.Uncountable() ?? []; var blockers = streams.Count == 0 ? [] : await negotiation.NodeList.BlockersAsync(streams, AgesByNode(current), cancellationToken).ConfigureAwait(false); - negotiator.Observe(new NodeListView(readers, blockers, Complete: streams.Count > 0)); + var subscribers = await ReadSubscribersAsync(channels, uncountable, readers, current.Where(a => a.Advertisement.Left), cancellationToken).ConfigureAwait(false); + negotiator.Observe(new NodeListView(readers, blockers, Complete: streams.Count > 0 || channels.Count > 0 || uncountable.Count > 0, subscribers)); await Prune(all.Where(a => a.Age > gone).ToList(), cancellationToken).ConfigureAwait(false); } catch (Exception ex) when (ex is not OutOfMemoryException && !Stopping()) { negotiation.Logger.LogWarning(ex, "The format node list could not be read; the last decision stands until it is too old."); + + // Whatever part of the view failed, the views on either side of it are not consecutive. + negotiation.Subscribers?.Reset(); negotiator.ObservationFailed(); } } @@ -123,7 +174,78 @@ internal async Task RefreshAsync(CancellationToken cancellationToken) private static Dictionary AgesByNode(IEnumerable ads) => ads.GroupBy(a => a.NodeId, StringComparer.OrdinalIgnoreCase).ToDictionary(g => g.Key, g => g.Min(a => a.Age), StringComparer.OrdinalIgnoreCase); - // No Streams topic, or broadcast off: nothing to read, so only the operator's word can switch, and the hold still applies. + // The subscriptions the nodes that read have confirmed, summed per channel: all the live subscribers they account for. For nodes that are leaving, the + // same sum is what they may still hold. + private static Dictionary Accounted(IEnumerable readers) + { + var accounted = new Dictionary(StringComparer.Ordinal); + foreach (var advertisement in readers.Select(r => r.Advertisement)) + { + foreach (var (channel, count) in advertisement.Subscriptions ?? new Dictionary()) + { + accounted[channel] = accounted.GetValueOrDefault(channel) + count; + } + + accounted[PubSubNodeList.PatternKey] = accounted.GetValueOrDefault(PubSubNodeList.PatternKey) + (advertisement.PatternSubscriptions ?? 0); + } + + return accounted; + } + + private async Task ReadSubscribersAsync( + IReadOnlyList channels, + IReadOnlyList uncountable, + IEnumerable readers, + IEnumerable leaving, + CancellationToken cancellationToken) + { + if (negotiation.Subscribers is not { } nodeList) + { + return null; + } + + if (uncountable.Count > 0) + { + // NUMSUB counts classic literal channels only, so a pattern or sharded one makes the whole count a guess. + nodeList.Reset(); + return SubscriberReading.Unknown with { Why = $"the channel strategy made patterns or sharded channels, which PUBSUB NUMSUB does not count: {string.Join(", ", uncountable)}" }; + } + + if (channels.Count == 0) + { + nodeList.Reset(); + return null; + } + + return await nodeList.ReadAsync(channels, Accounted(readers), Accounted(leaving), cancellationToken).ConfigureAwait(false); + } + + // Still lists the subscriptions it holds: the connection closes a moment after the node says it is leaving, and until then the server counts it. + private NodeAdvertisement Departing() => + negotiation.Advertisement! with { Left = true, Subscriptions = negotiation.PubSub?.Confirmed(), PatternSubscriptions = negotiation.PubSub?.Patterns() }; + + // Off the thread that confirmed the subscription, which is the topic's, and never failing it. + private void AdvertiseConfirmed() + { + if (Volatile.Read(ref _started) == 0 || Volatile.Read(ref _stopped) == 1) + { + return; + } + + _ = Task.Run(async () => + { + try + { + await AdvertiseAsync(_stop.Token).ConfigureAwait(false); + } + catch (Exception ex) when (ex is not OutOfMemoryException) + { + negotiation.Logger.LogDebug(ex, "The advertisement of a confirmed subscription was not written; the next one carries it."); + } + }); + } + + // No Streams or Pub/Sub topic, or broadcast off: nothing to read, so only the operator's word can switch, and the hold still applies. private void StartBlind() { if (negotiation.Negotiator is not { } negotiator) @@ -133,7 +255,7 @@ private void StartBlind() if (!negotiation.Options.AssumeAllNodesCanRead) { - negotiation.Logger.LogInformation("Value formats stay JSON: the app has no Streams transport, so the nodes cannot be listed. Set FormatNegotiation:AssumeAllNodesCanRead to attest that they all read the desired formats."); + negotiation.Logger.LogInformation("Value and event formats stay JSON: the app has no Streams or Pub/Sub transport, so the nodes cannot be listed. Set FormatNegotiation:AssumeAllNodesCanRead to attest that they all read the desired formats."); return; } diff --git a/src/UiPath.Caching/Negotiation/FormatNegotiator.cs b/src/UiPath.Caching/Negotiation/FormatNegotiator.cs index 56f08feb..21c02a9a 100644 --- a/src/UiPath.Caching/Negotiation/FormatNegotiator.cs +++ b/src/UiPath.Caching/Negotiation/FormatNegotiator.cs @@ -3,25 +3,33 @@ namespace UiPath.Caching.Negotiation; /// -/// Turns what the node list shows into which serializers are written now, whatever policy asks for them. A format is written only while every -/// advertisement lists it and no live node is unaccounted for, and only after that has held for -/// since the reading nodes last gained a member. The first node that cannot read it stops the writes at once; a node that leaves never does. +/// Turns what the node list shows into which serializers, and whether the binary event format, are written now, whatever policy asks for them. A format +/// is written only while every advertisement lists it and no live node is unaccounted for, and only after that has held for +/// since the reading nodes last gained a member. The first node that cannot read it stops the +/// writes at once; a node that leaves never does. /// -internal sealed class FormatNegotiator : IEffectiveFormat +internal sealed class FormatNegotiator : IEffectiveFormat, IBinaryEvents { public const string ReasonUnaccountedGroup = "unaccounted_group"; public const string ReasonMissingFormat = "missing_format"; public const string ReasonNoView = "no_view"; public const string ReasonViewStale = "view_stale"; public const string ReasonTransportUnknown = "transport_unknown"; + public const string ReasonUnaccountedSubscriber = "unaccounted_subscriber"; + public const string ReasonSubscriberDeficit = "subscriber_deficit"; - private readonly DesiredWrite[] _desired; + // Not a reason to fall back: the Pub/Sub count has not yet shown the same thing twice, so a format in use stays and one not in use waits. + private const string ReasonSettling = "settling"; + + private readonly Slot[] _slots; + private readonly int _eventSlot; private readonly FormatNegotiationOptions _options; private readonly TimeProvider _clock; private readonly ICachingTelemetryProvider _telemetry; private readonly ILogger _logger; private readonly Lock _gate = new(); private readonly long?[] _candidateSince; + private readonly bool[] _suspended; private readonly string?[] _reasons; private HashSet _readers = []; private volatile Snapshot _snapshot; @@ -29,24 +37,39 @@ internal sealed class FormatNegotiator : IEffectiveFormat private long? _graceStart; private bool _hasView; private bool _failed; + private string _blocking = string.Empty; + private string? _uncountable; public FormatNegotiator( IEnumerable desired, FormatNegotiationOptions options, TimeProvider clock, ICachingTelemetryProvider? telemetry = null, - ILogger? logger = null) + ILogger? logger = null, + DesiredEvents? events = null) { options.Validate(); - _desired = [.. desired]; + _slots = [.. desired.Select(d => new Slot(d.Writer, d.Tokens, d.Policies, d.Others)), .. events is null ? [] : new[] { new Slot(null, events.Tokens, [], []) }]; + _eventSlot = events is null ? -1 : _slots.Length - 1; _options = options; _clock = clock; _telemetry = telemetry ?? NullTelemetryProvider.Instance; _logger = logger ?? NullLogger.Instance; - _candidateSince = new long?[_desired.Length]; - _reasons = new string?[_desired.Length]; + _candidateSince = new long?[_slots.Length]; + _suspended = new bool[_slots.Length]; + _reasons = new string?[_slots.Length]; Array.Fill(_reasons, ReasonNoView); - _snapshot = Publish(new bool[_desired.Length], 0); + _snapshot = Publish(new bool[_slots.Length], 0); + } + + /// Whether binary events are written now; always for a node not set to write them. + public bool Writes + { + get + { + var snapshot = _snapshot; + return _eventSlot >= 0 && snapshot.Writers[_eventSlot].Allowed && _clock.GetElapsedTime(snapshot.Stamp) <= _options.MaxViewAge; + } } public bool Allows(ICacheValueSerializer writer) @@ -78,28 +101,58 @@ public void Observe(NodeListView view) } var grace = _graceStart is { } start && _clock.GetElapsedTime(start, now) < TimeSpan.FromTicks(_options.AdvertiseInterval.Ticks * 2); - var joined = view.Readers.Any(reader => !_readers.Contains(reader)); + // Something that subscribed to take the place of a claim that was still fresh is a join too, though it advertised nothing. + var joined = view.Readers.Any(reader => !_readers.Contains(reader)) || view.Subscribers is { Replaced: true }; _readers = [.. view.Readers]; _hasView = true; _lastView = now; - var allowed = new bool[_desired.Length]; + var allowed = new bool[_slots.Length]; var previous = _snapshot.Writers; - for (var i = 0; i < _desired.Length; i++) + for (var i = 0; i < _slots.Length; i++) { - _reasons[i] = Block(_desired[i], view); - if (_reasons[i] is ReasonNoView or ReasonUnaccountedGroup && grace && previous[i].Allowed) + // A node that joins ends what a departure held back: the format then waits for the hold like any other. + _suspended[i] &= !joined; + _reasons[i] = Block(_slots[i], view); + if (_reasons[i] is ReasonNoView or ReasonUnaccountedGroup or ReasonUnaccountedSubscriber && grace && previous[i].Allowed) { _reasons[i] = null; allowed[i] = true; } + else if (_reasons[i] is ReasonSettling) + { + // A blip pauses the hold, it does not restart it: only a join, or a format a node cannot read, does. + _reasons[i] = null; + if (joined) + { + _candidateSince[i] = null; + } + + allowed[i] = previous[i].Allowed; + } + else if (_reasons[i] is ReasonSubscriberDeficit) + { + // A departure, not a join: the format is not written, and the hold it had earned is kept for when the count adds up again. + _suspended[i] |= previous[i].Allowed && !joined; + if (joined) + { + _candidateSince[i] = null; + } + } else if (_reasons[i] is not null) { + _suspended[i] = false; _candidateSince[i] = null; } else if (previous[i].Allowed) { allowed[i] = true; } + else if (_suspended[i] && !joined) + { + // The count adds up again: what was written before the node went away resumes, without a new hold. + _suspended[i] = false; + allowed[i] = true; + } else { if (_candidateSince[i] is null || joined) @@ -137,19 +190,31 @@ public void ObservationFailed() } _hasView = false; + Array.Fill(_suspended, false); Array.Fill(_reasons, ReasonViewStale); - var allowed = new bool[_desired.Length]; + var allowed = new bool[_slots.Length]; Transition(_snapshot.Writers, allowed); _snapshot = Publish(allowed, _snapshot.Stamp); } } - private string? Block(DesiredWrite want, NodeListView view) + private string? Block(Slot want, NodeListView view) { // What an advertisement says it cannot read comes first: no grace after an outage and no attestation excuses it. foreach (var stored in view.Ads) { var ad = stored.Advertisement; + if (want.IsEvent) + { + // An advertisement that lists no event formats is from a build that did not know them, so it reads JSON at most. + if (ad.EventReads is null || !want.Tokens.All(ad.EventReads.Contains)) + { + return ReasonMissingFormat; + } + + continue; + } + if (!want.Tokens.All(ad.ValueReads.Contains)) { return ReasonMissingFormat; @@ -178,22 +243,43 @@ public void ObservationFailed() } // The attestation covers nodes that say nothing; a node that advertises without the format is not one of them. - return view.Blockers.Count > 0 && !attested ? ReasonUnaccountedGroup : null; + if (view.Blockers.Count > 0 && !attested) + { + return ReasonUnaccountedGroup; + } + + return view.Subscribers is { } subscribers && !attested + ? subscribers switch + { + { Known: false } => ReasonTransportUnknown, + { Unaccounted: true } => ReasonUnaccountedSubscriber, + { Short: true } => ReasonSubscriberDeficit, + { Settled: false } => ReasonSettling, + _ => null, + } + : null; } private void Transition(WriterState[] previous, bool[] allowed) { for (var i = 0; i < allowed.Length; i++) { - var policies = string.Join(',', _desired[i].Policies); + var slot = _slots[i]; + var kind = slot.IsEvent ? "event" : "value"; + var format = string.Join('+', slot.Tokens); if (previous[i].Allowed && !allowed[i]) { - _telemetry.TrackMetric(Metrics.FormatFallback, 1, [new("reason", _reasons[i] ?? ReasonNoView), new("policy", policies)]); - _logger.LogWarning("Value format {Format} falls back to JSON: {Reason}.", string.Join('+', _desired[i].Tokens), _reasons[i]); + _telemetry.TrackMetric( + Metrics.FormatFallback, + 1, + slot.IsEvent + ? [new("reason", _reasons[i] ?? ReasonNoView), new("kind", kind)] + : [new("reason", _reasons[i] ?? ReasonNoView), new("kind", kind), new("policy", string.Join(',', slot.Policies))]); + _logger.LogWarning("The {Kind} format {Format} falls back to JSON: {Reason}.", kind, format, _reasons[i]); } else if (!previous[i].Allowed && allowed[i]) { - _logger.LogInformation("Value format {Format} is now written.", string.Join('+', _desired[i].Tokens)); + _logger.LogInformation("The {Kind} format {Format} is now written.", kind, format); } } } @@ -202,34 +288,96 @@ private void Report(NodeListView view) { _telemetry.TrackMetric(Metrics.FormatViewAge, 0); _telemetry.TrackMetric(Metrics.FormatNodes, view.Ads.Count, [new("state", "live")]); - _telemetry.TrackMetric(Metrics.FormatNodes, view.Blockers.Count, [new("state", "blocking")]); - if (view.Blockers.Count > 0) - { - _logger.LogWarning("Nodes live on a stream without an advertisement, so not sent a binary format: {Nodes}.", string.Join(", ", view.Blockers)); - } - + var unknown = view.Subscribers?.Surplus ?? 0; + _telemetry.TrackMetric(Metrics.FormatNodes, view.Blockers.Count + unknown, [new("state", "blocking")]); + ReportBlocking(view); var current = _snapshot.Writers; - for (var i = 0; i < _desired.Length; i++) + for (var i = 0; i < _slots.Length; i++) { + var slot = _slots[i]; + var format = new KeyValuePair("format", string.Join('+', slot.Tokens)); _telemetry.TrackMetric( Metrics.FormatEffective, current[i].Allowed ? 1 : 0, - [new("kind", "value"), new("policy", string.Join(',', _desired[i].Policies)), new("format", string.Join('+', _desired[i].Tokens))]); + slot.IsEvent + ? [new("kind", "event"), format] + : [new("kind", "value"), new("policy", string.Join(',', slot.Policies)), format]); } } + // Who blocks goes to the log and to Caching.Stats.FormatBlockers when the set changes, and never to the cache.format.* metrics, whose tags are bounded: + // node ids are not, so a backend that turns a metric's tags into dimensions would see as many as there are nodes. + private void ReportBlocking(NodeListView view) + { + ReportUncountable(view.Subscribers); + var blocking = string.Join(", ", view.Blockers); + var channels = string.Join(", ", view.Subscribers?.Channels ?? []); + var missing = string.Join(", ", view.Subscribers?.ShortChannels ?? []); + var key = blocking + "|" + channels + "|" + missing; + if (key == _blocking) + { + return; + } + + _blocking = key; + if (blocking.Length > 0) + { + _logger.LogWarning("Nodes live on a stream without an advertisement, so not sent a binary format: {Nodes}.", blocking); + _telemetry.TrackMetric(Metrics.FormatBlockers, view.Blockers.Count, [new("nodes", blocking)]); + } + + if (channels.Length > 0) + { + _logger.LogWarning("Subscribers no advertisement accounts for on Pub/Sub channels or patterns, so not sent a binary format: {Channels}. Pattern subscriptions are counted for the whole server, other apps' included; set FormatNegotiation:AssumeAllNodesCanRead to attest that every node reads binary formats.", channels); + _telemetry.TrackMetric(Metrics.FormatBlockers, view.Subscribers!.Surplus, [new("channels", channels)]); + } + + if (missing.Length > 0) + { + _logger.LogWarning("Pub/Sub channels with fewer subscribers than the advertisements claim, so not sent a binary format until the claims expire: {Channels}.", missing); + _telemetry.TrackMetric(Metrics.FormatBlockers, view.Subscribers!.ShortChannels!.Count, [new("short_channels", missing)]); + } + } + + // Once per change: a node that cannot count Pub/Sub subscribers stays on JSON, and nothing else says why. + private void ReportUncountable(SubscriberReading? subscribers) + { + var why = subscribers is { Known: false } ? subscribers.Why ?? "the subscribers cannot be counted" : null; + if (why == _uncountable) + { + return; + } + + if (why is not null) + { + _logger.LogWarning("Pub/Sub subscribers cannot be counted ({Why}), so binary formats stay JSON. Set FormatNegotiation:AssumeAllNodesCanRead to attest that every node reads them.", why); + } + else + { + _logger.LogInformation("Pub/Sub subscribers can be counted again."); + } + + _uncountable = why; + } + private Snapshot Publish(bool[] allowed, long stamp) { - var writers = new WriterState[_desired.Length]; + var writers = new WriterState[_slots.Length]; for (var i = 0; i < writers.Length; i++) { - writers[i] = new(_desired[i].Writer, allowed[i]); + writers[i] = new(_slots[i].Writer, allowed[i]); } return new(writers, stamp); } - private sealed record WriterState(ICacheValueSerializer Writer, bool Allowed); + /// A serializer, or the binary event format when is , with what a node must read to be sent it. + private sealed record Slot(ICacheValueSerializer? Writer, IReadOnlyList Tokens, IReadOnlyCollection Policies, IReadOnlyCollection Others) + { + public bool IsEvent => Writer is null; + } + + private sealed record WriterState(ICacheValueSerializer? Writer, bool Allowed); private sealed record Snapshot(WriterState[] Writers, long Stamp); } diff --git a/src/UiPath.Caching/Negotiation/IBinaryEvents.cs b/src/UiPath.Caching/Negotiation/IBinaryEvents.cs new file mode 100644 index 00000000..c98ff6f7 --- /dev/null +++ b/src/UiPath.Caching/Negotiation/IBinaryEvents.cs @@ -0,0 +1,8 @@ +namespace UiPath.Caching.Negotiation; + +/// Whether this node writes binary events now; events are JSON otherwise. +internal interface IBinaryEvents +{ + /// Reads one volatile reference and the injected clock, with no allocation and no call to Redis. + bool Writes { get; } +} diff --git a/src/UiPath.Caching/Negotiation/IPubSubSubscriptions.cs b/src/UiPath.Caching/Negotiation/IPubSubSubscriptions.cs new file mode 100644 index 00000000..b251301e --- /dev/null +++ b/src/UiPath.Caching/Negotiation/IPubSubSubscriptions.cs @@ -0,0 +1,20 @@ +namespace UiPath.Caching.Negotiation; + +/// The Pub/Sub channels this node's topics listen on, and which of them the server has confirmed a subscription to. +internal interface IPubSubSubscriptions +{ + /// The channels whose subscribers PUBSUB NUMSUB can count. + IReadOnlyList Channels(); + + /// The channels it cannot count, because the strategy made them patterns or sharded channels: it counts classic literal channels only. + IReadOnlyList Uncountable(); + + /// Each channel whose subscription is in place, with how many subscriptions of this node it holds. + IReadOnlyDictionary Confirmed(); + + /// How many pattern subscriptions this node holds, which PUBSUB NUMSUB does not count and PUBSUB NUMPAT does. + int Patterns(); + + /// Calls each time a subscription is confirmed, so the node can say so at once. + void WhenConfirmed(Action handler); +} diff --git a/src/UiPath.Caching/Negotiation/ISubscriberSource.cs b/src/UiPath.Caching/Negotiation/ISubscriberSource.cs new file mode 100644 index 00000000..f69105cc --- /dev/null +++ b/src/UiPath.Caching/Negotiation/ISubscriberSource.cs @@ -0,0 +1,16 @@ +using System.Net; + +namespace UiPath.Caching.Negotiation; + +/// The servers of the Redis deployment, and how many subscribers each one counts on Pub/Sub channels and patterns. +internal interface ISubscriberSource +{ + /// + /// Every server the deployment has, primaries and replicas alike, since a subscriber is local to the server it connected to. It throws when it cannot + /// say that it has them all, a server that is not connected for one: a sum over some of the servers would hide the subscribers of the others. + /// + IReadOnlyList Endpoints(); + + /// The server's PUBSUB NUMSUB for all the channels, in the order asked, and its PUBSUB NUMPAT; when the server refuses either. + Task CountAsync(EndPoint endpoint, IReadOnlyList channels, CancellationToken cancellationToken); +} diff --git a/src/UiPath.Caching/Negotiation/LazyBinaryEvents.cs b/src/UiPath.Caching/Negotiation/LazyBinaryEvents.cs new file mode 100644 index 00000000..093934cf --- /dev/null +++ b/src/UiPath.Caching/Negotiation/LazyBinaryEvents.cs @@ -0,0 +1,18 @@ +namespace UiPath.Caching.Negotiation; + +/// +/// Asks the negotiation on first use, not when the formatter is built: the topic providers that need the formatter are what the negotiation is built from. +/// +internal sealed class LazyBinaryEvents(BinaryEventSettings settings, Func negotiation) : IBinaryEvents +{ + private IBinaryEvents? _inner; + + public bool Writes => (Volatile.Read(ref _inner) ?? Resolve()).Writes; + + private IBinaryEvents Resolve() + { + // A node set to write JSON has nothing to ask the node list, and one with no negotiation to ask has nothing that proves binary is safe. + var resolved = settings.Write && negotiation() is { } resolvedNegotiation ? resolvedNegotiation.EventWrites(settings) : FixedBinaryEvents.Never; + return Interlocked.CompareExchange(ref _inner, resolved, null) ?? resolved; + } +} diff --git a/src/UiPath.Caching/Negotiation/NegotiationPlan.cs b/src/UiPath.Caching/Negotiation/NegotiationPlan.cs index 38ea8407..8af846fe 100644 --- a/src/UiPath.Caching/Negotiation/NegotiationPlan.cs +++ b/src/UiPath.Caching/Negotiation/NegotiationPlan.cs @@ -24,7 +24,8 @@ [.. named.Where(p => ReferenceEquals(p.Policy.WriteSerializer, writer)).Select(p return desired; } - public static NodeAdvertisement Advertisement(ValueSerializerRegistry registry, ICachePolicyFactory policies, ValueReader reader, string library) + /// Whether the binary event decoder is registered; a node without it still lists the JSON events it reads. + public static NodeAdvertisement Advertisement(ValueSerializerRegistry registry, ICachePolicyFactory policies, ValueReader reader, string library, bool readsBinaryEvents = false) { var reads = new List { NodeAdvertisement.Json }; foreach (var (_, serializer, _) in registry.Entries) @@ -51,7 +52,7 @@ public static NodeAdvertisement Advertisement(ValueSerializerRegistry registry, strict[DefaultPolicyName] = [NodeAdvertisement.Json, NodeAdvertisement.NoSniff]; } - return new(library, Guid.NewGuid().ToString("N"), reads.Distinct().ToList(), strict); + return new(library, Guid.NewGuid().ToString("N"), reads.Distinct().ToList(), strict, EventReads: readsBinaryEvents ? [NodeAdvertisement.Json, BinaryEventSettings.Token] : [NodeAdvertisement.Json]); } private static IEnumerable Tokens(ValueSerializerRegistry registry, string format) => diff --git a/src/UiPath.Caching/Negotiation/NodeAdvertisement.cs b/src/UiPath.Caching/Negotiation/NodeAdvertisement.cs index bcf588e3..55b367b9 100644 --- a/src/UiPath.Caching/Negotiation/NodeAdvertisement.cs +++ b/src/UiPath.Caching/Negotiation/NodeAdvertisement.cs @@ -3,14 +3,20 @@ namespace UiPath.Caching.Negotiation; -/// What a node tells the others about itself: the value formats it reads, per-policy read lists, a field reserved for migration capabilities, and whether the node is leaving. +/// +/// What a node tells the others about itself: the value and event formats it reads, per-policy read lists, the Pub/Sub channels it has confirmed a subscription +/// to and how many pattern subscriptions it holds, a field reserved for migration capabilities, and whether the node is leaving. A node that lists no event formats reads none it can name. +/// internal sealed record NodeAdvertisement( string Library, string Instance, IReadOnlyList ValueReads, IReadOnlyDictionary> Strict, string? Migration = null, - bool Left = false) + bool Left = false, + IReadOnlyList? EventReads = null, + IReadOnlyDictionary? Subscriptions = null, + int? PatternSubscriptions = null) { public const string Json = "json"; @@ -34,6 +40,27 @@ public string Encode() } } + if (EventReads is not null) + { + text.Append(";er=").AppendJoin(',', EventReads); + } + + if (Subscriptions is { Count: > 0 }) + { + text.Append(";sub="); + var first = true; + foreach (var (channel, count) in Subscriptions.OrderBy(p => p.Key, StringComparer.Ordinal)) + { + text.Append(first ? string.Empty : ",").Append(Escape(channel)).Append(':').Append(count.ToString(CultureInfo.InvariantCulture)); + first = false; + } + } + + if (PatternSubscriptions is > 0) + { + text.Append(";psub=").Append(PatternSubscriptions.Value.ToString(CultureInfo.InvariantCulture)); + } + if (Migration is not null) { text.Append(";mig=").Append(Escape(Migration)); @@ -54,8 +81,10 @@ public static bool TryParse(string text, [NotNullWhen(true)] out NodeAdvertiseme string library = string.Empty, instance = string.Empty; string? migration = null; var left = false; - IReadOnlyList? reads = null; + IReadOnlyList? reads = null, eventReads = null; Dictionary> strict = []; + Dictionary subscriptions = []; + int? patterns = null; foreach (var field in text.Split(';')) { var separator = field.IndexOf('=', StringComparison.Ordinal); @@ -76,6 +105,24 @@ public static bool TryParse(string text, [NotNullWhen(true)] out NodeAdvertiseme case "vr": reads = value.Split(',', StringSplitOptions.RemoveEmptyEntries); break; + case "er": + eventReads = value.Split(',', StringSplitOptions.RemoveEmptyEntries); + break; + case "sub": + // An entry that does not parse is left out: the node then counts for fewer subscribers, which only holds writes back. + foreach (var entry in value.Split(',', StringSplitOptions.RemoveEmptyEntries)) + { + var colon = entry.LastIndexOf(':'); + if (colon > 0 && int.TryParse(entry[(colon + 1)..], NumberStyles.None, CultureInfo.InvariantCulture, out var count)) + { + subscriptions[Unescape(entry[..colon])] = count; + } + } + + break; + case "psub": + patterns = int.TryParse(value, NumberStyles.None, CultureInfo.InvariantCulture, out var pattern) ? pattern : null; + break; case "mig": migration = Unescape(value); break; @@ -103,7 +150,7 @@ public static bool TryParse(string text, [NotNullWhen(true)] out NodeAdvertiseme return false; } - advertisement = new NodeAdvertisement(library, instance, reads, strict, migration, left); + advertisement = new NodeAdvertisement(library, instance, reads, strict, migration, left, eventReads, subscriptions.Count > 0 ? subscriptions : null, patterns); return true; } diff --git a/src/UiPath.Caching/Negotiation/NodeListView.cs b/src/UiPath.Caching/Negotiation/NodeListView.cs index 15233828..b3dff46a 100644 --- a/src/UiPath.Caching/Negotiation/NodeListView.cs +++ b/src/UiPath.Caching/Negotiation/NodeListView.cs @@ -1,7 +1,10 @@ namespace UiPath.Caching.Negotiation; -/// What one node list read found: the advertisements of the nodes that read, the live nodes without one, and whether the node list can see every node. -internal sealed record NodeListView(IReadOnlyList Ads, IReadOnlyList Blockers, bool Complete) +/// +/// What one node list read found: the advertisements of the nodes that read, the live Streams nodes without one, whether the node list can see every node, and, +/// where events go over Pub/Sub, what the subscriber count shows. +/// +internal sealed record NodeListView(IReadOnlyList Ads, IReadOnlyList Blockers, bool Complete, SubscriberReading? Subscribers = null) { /// Each reading node and the process it advertised from, so a node that joins or restarts is told from one that merely stays. public IReadOnlyCollection Readers { get; } = [.. Ads.Select(a => a.NodeId + "/" + a.Advertisement.Instance)]; diff --git a/src/UiPath.Caching/Negotiation/PubSubNodeList.cs b/src/UiPath.Caching/Negotiation/PubSubNodeList.cs new file mode 100644 index 00000000..73a3b70b --- /dev/null +++ b/src/UiPath.Caching/Negotiation/PubSubNodeList.cs @@ -0,0 +1,134 @@ +using System.Net; + +namespace UiPath.Caching.Negotiation; + +/// +/// Finds the subscribers of a Pub/Sub channel that no advertisement accounts for. A subscriber is local to the server it connected to, so the live count +/// is the sum of PUBSUB NUMSUB over every server, one round trip per server for all the channels and for PUBSUB NUMPAT, which counts the +/// pattern subscriptions NUMSUB leaves out; what advertisements account for is the sum of the subscriptions they list, patterns included, and a node that is leaving may or may not still hold its own. More than that is a node that advertises nothing. Fewer is the +/// opposite: a node that went away, whose claim outlives it. A count that does not add up either way counts against a format only once two views running +/// show it on the same channel, so a node that has subscribed and not yet advertised, or one that is resubscribing, is absorbed, and a count counts for an +/// upgrade only once two views running show every channel clean over the same number of servers. +/// +internal sealed class PubSubNodeList(ISubscriberSource source) +{ + /// + /// Stands for the pattern subscriptions of the servers in the counts and the claims. PUBSUB NUMPAT counts the whole server, not the channels of this + /// node, so another app that subscribes to a pattern is in it too. + /// + public const string PatternKey = "(patterns)"; + + private readonly Lock _gate = new(); + private readonly Dictionary _confirmedShort = new(StringComparer.Ordinal); + private Dictionary? _previous; + private int _endpoints; + + public Task ReadAsync(IReadOnlyCollection channels, IReadOnlyDictionary accounted, CancellationToken cancellationToken) => + ReadAsync(channels, accounted, new Dictionary(), cancellationToken); + + /// The channels this node listens on. + /// The subscriptions the nodes that read have confirmed, per channel. + /// The subscriptions nodes that are leaving claimed, per channel: each may or may not still be held, until its advertisement expires. + /// Cancels the read. + public async Task ReadAsync(IReadOnlyCollection channels, IReadOnlyDictionary accounted, IReadOnlyDictionary leaving, CancellationToken cancellationToken) + { + try + { + var endpoints = source.Endpoints(); + if (endpoints.Count == 0) + { + Reset(); + return SubscriberReading.Unknown with { Why = "no server is connected" }; + } + + IReadOnlyList asked = [.. channels]; + var replies = await Task.WhenAll(endpoints.Select(endpoint => source.CountAsync(endpoint, asked, cancellationToken))).ConfigureAwait(false); + if (replies.Any(reply => reply is null)) + { + // One server that cannot say makes the sum a guess, and a guess is not an answer. + Reset(); + return SubscriberReading.Unknown with { Why = "a server refused PUBSUB NUMSUB" }; + } + + return Judge(asked, [.. replies.Select(reply => reply!)], endpoints.Count, accounted, leaving); + } + catch (Exception ex) when (ex is not OutOfMemoryException) + { + Reset(); + throw; + } + } + + /// Two views are consecutive only if nothing came between them: a view that failed, was partial or could not count starts the debounce again, but not a deficit already confirmed. + public void Reset() + { + lock (_gate) + { + _previous = null; + } + } + + private SubscriberReading Judge(IReadOnlyList asked, List replies, int endpoints, IReadOnlyDictionary accounted, IReadOnlyDictionary leaving) + { + // The patterns are one more thing to account for, with no deficit: NUMPAT counts the unique patterns, so claims may exceed it. + IReadOnlyList channels = [.. asked, PatternKey]; + var surplus = new SortedDictionary(StringComparer.Ordinal); + var missing = new SortedSet(StringComparer.Ordinal); + lock (_gate) + { + var previous = _previous; + var next = new Dictionary(channels.Count, StringComparer.Ordinal); + bool unaccounted = false, confirmedShort = false, replaced = false; + var settled = previous is not null && _endpoints == endpoints; + for (var i = 0; i < channels.Count; i++) + { + var channel = channels[i]; + var live = i < asked.Count ? replies.Sum(reply => reply.Channels[i]) : replies.Sum(reply => reply.Patterns); + var claimed = accounted.GetValueOrDefault(channel); + var over = live > claimed + leaving.GetValueOrDefault(channel); + var under = i < asked.Count && live < claimed; + ChannelState? before = previous is not null && previous.TryGetValue(channel, out var found) ? found : null; + if (over) + { + surplus[channel] = live - claimed - leaving.GetValueOrDefault(channel); + } + + if (under) + { + missing.Add(channel); + } + + var shortConfirmed = under && before is { Under: true }; + unaccounted |= over && before is { Over: true }; + confirmedShort |= shortConfirmed; + if (_confirmedShort.TryGetValue(channel, out var shortClaim) && !under) + { + replaced |= claimed >= shortClaim; + _confirmedShort.Remove(channel); + } + + if (shortConfirmed) + { + _confirmedShort[channel] = claimed; + } + + settled &= !over && !under && before is { Over: false, Under: false }; + next[channel] = new(over, under); + } + + _previous = next; + _endpoints = endpoints; + return new( + Known: true, + Unaccounted: unaccounted, + Settled: settled, + surplus.Values.Sum(), + [.. surplus.Keys], + Short: confirmedShort, + ShortChannels: [.. missing], + Replaced: replaced); + } + } + + private readonly record struct ChannelState(bool Over, bool Under); +} diff --git a/src/UiPath.Caching/Negotiation/RedisSubscriberSource.cs b/src/UiPath.Caching/Negotiation/RedisSubscriberSource.cs new file mode 100644 index 00000000..7e706f0f --- /dev/null +++ b/src/UiPath.Caching/Negotiation/RedisSubscriberSource.cs @@ -0,0 +1,54 @@ +using System.Net; + +namespace UiPath.Caching.Negotiation; + +internal sealed class RedisSubscriberSource(IRedisConnector redis) : ISubscriberSource +{ + public IReadOnlyList Endpoints() + { + var servers = redis.Database.Multiplexer.GetServers().Where(s => s.ServerType != ServerType.Sentinel).ToList(); + var down = servers.Where(s => !s.IsConnected).Select(s => s.EndPoint.ToString()).ToList(); + if (down.Count > 0) + { + throw new InvalidOperationException($"The servers {string.Join(", ", down)} are not connected, so their subscribers cannot be counted."); + } + + return [.. servers.Select(s => s.EndPoint)]; + } + + public async Task CountAsync(EndPoint endpoint, IReadOnlyList channels, CancellationToken cancellationToken) + { + await redis.ConnectAsync(cancellationToken).AsTask().WaitAsync(cancellationToken).ConfigureAwait(false); + var multiplexer = redis.Database.Multiplexer; + var server = multiplexer.GetServer(endpoint); + var prefix = ChannelPrefix(multiplexer); + List subscribers = ["NUMSUB", .. channels.Select(channel => prefix + channel)]; + var numsub = server.ExecuteAsync("PUBSUB", subscribers); + var numpat = server.ExecuteAsync("PUBSUB", new List { "NUMPAT" }); + try + { + await Task.WhenAll(numsub, numpat).WaitAsync(cancellationToken).ConfigureAwait(false); + } + catch (Exception ex) when (IsRefusal(ex)) + { + return null; + } + + // Name, count, name, count: in the order asked. + var reply = (RedisResult[])(await numsub.ConfigureAwait(false))!; + return new([.. Enumerable.Range(0, channels.Count).Select(i => (long)reply[(2 * i) + 1])], (long)(await numpat.ConfigureAwait(false))); + } + + /// A server that will not run the command, by permission, by a renamed or disabled command, or because it does not know it. + internal static bool IsRefusal(Exception ex) => + ex is RedisCommandException + || (ex is RedisServerException server + && (server.Message.StartsWith("NOPERM", StringComparison.OrdinalIgnoreCase) + || server.Message.Contains("unknown", StringComparison.OrdinalIgnoreCase) + || server.Message.Contains("disabled", StringComparison.OrdinalIgnoreCase))); + + // The multiplexer puts its channel prefix in front of every channel it subscribes to or publishes on, and the server only knows the result. + // The multiplexer's configuration string carries it, and a multiplexer that does not say has none. + private static string ChannelPrefix(IConnectionMultiplexer multiplexer) => + multiplexer.Configuration is { Length: > 0 } configuration ? (string?)ConfigurationOptions.Parse(configuration).ChannelPrefix ?? string.Empty : string.Empty; +} diff --git a/src/UiPath.Caching/Negotiation/ServerCount.cs b/src/UiPath.Caching/Negotiation/ServerCount.cs new file mode 100644 index 00000000..13eb868d --- /dev/null +++ b/src/UiPath.Caching/Negotiation/ServerCount.cs @@ -0,0 +1,4 @@ +namespace UiPath.Caching.Negotiation; + +/// What one server counts: subscribers of each channel asked, and the patterns subscribed to on the whole server. +internal sealed record ServerCount(IReadOnlyList Channels, long Patterns); diff --git a/src/UiPath.Caching/Negotiation/SubscriberReading.cs b/src/UiPath.Caching/Negotiation/SubscriberReading.cs new file mode 100644 index 00000000..36699117 --- /dev/null +++ b/src/UiPath.Caching/Negotiation/SubscriberReading.cs @@ -0,0 +1,23 @@ +namespace UiPath.Caching.Negotiation; + +/// +/// What the Pub/Sub node list found on its latest read, after the debounce: whether it could count at all, whether subscribers no advertisement accounts for +/// have shown in two views running, whether the count has been clean and unchanged for two, and what is surplus now. is the other +/// way round: fewer subscribers than the advertisements claim, in two views running, which is a node that went away without saying so, and so a count +/// that does not add up; names those channels. is a confirmed deficit that ended while the claims +/// behind it are still fresh, so the subscriber count rose: something subscribed, which is a join and not the claim expiring. says why +/// the count is unknown. +/// +internal sealed record SubscriberReading( + bool Known, + bool Unaccounted, + bool Settled, + long Surplus, + IReadOnlyList Channels, + bool Short = false, + IReadOnlyList? ShortChannels = null, + bool Replaced = false, + string? Why = null) +{ + public static SubscriberReading Unknown { get; } = new(false, false, false, 0, []); +} diff --git a/src/UiPath.Caching/PublicAPI.Unshipped.txt b/src/UiPath.Caching/PublicAPI.Unshipped.txt index bf65855f..632c6f2a 100644 --- a/src/UiPath.Caching/PublicAPI.Unshipped.txt +++ b/src/UiPath.Caching/PublicAPI.Unshipped.txt @@ -22,6 +22,7 @@ UiPath.Caching.Redis.FormatNegotiationOptions.ViewInterval.get -> System.TimeSpa UiPath.Caching.Redis.FormatNegotiationOptions.ViewInterval.set -> void UiPath.Caching.Redis.RedisCacheOptions.FormatNegotiation.get -> UiPath.Caching.Redis.FormatNegotiationOptions! UiPath.Caching.Redis.RedisCacheOptions.FormatNegotiation.set -> void +const UiPath.Caching.Metrics.FormatBlockers = "Caching.Stats.FormatBlockers" -> string! const UiPath.Caching.Metrics.FormatEffective = "cache.format.effective" -> string! const UiPath.Caching.Metrics.FormatFallback = "cache.format.fallback" -> string! const UiPath.Caching.Metrics.FormatNodes = "cache.format.nodes" -> string! diff --git a/tests/UiPath.Caching.Tests/Fakes/RecordedLogger.cs b/tests/UiPath.Caching.Tests/Fakes/RecordedLogger.cs new file mode 100644 index 00000000..678ad880 --- /dev/null +++ b/tests/UiPath.Caching.Tests/Fakes/RecordedLogger.cs @@ -0,0 +1,34 @@ +using Microsoft.Extensions.Logging; + +namespace UiPath.Caching.Tests.Fakes; + +/// A logger that keeps what it was told, at every level. +internal sealed class RecordedLogger : ILogger +{ + private readonly object _gate = new(); + private readonly List<(LogLevel Level, string Message)> _records = []; + + public IReadOnlyList<(LogLevel Level, string Message)> Records + { + get + { + lock (_gate) + { + return [.. _records]; + } + } + } + + public IDisposable? BeginScope(TState state) + where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) + { + lock (_gate) + { + _records.Add((logLevel, formatter(state, exception))); + } + } +} diff --git a/tests/UiPath.Caching.Tests/MessagePack/MessagePackEventFormatterTests.cs b/tests/UiPath.Caching.Tests/MessagePack/MessagePackEventFormatterTests.cs index f209c3e2..11d20129 100644 --- a/tests/UiPath.Caching.Tests/MessagePack/MessagePackEventFormatterTests.cs +++ b/tests/UiPath.Caching.Tests/MessagePack/MessagePackEventFormatterTests.cs @@ -830,7 +830,7 @@ private static ServiceProvider ResolveProvider(Action configure b => { b.AddRedisConnection(o => o.ConnectionString = "127.0.0.1:1"); - b.AddRedis(); + b.AddRedis(o => o.FormatNegotiation.Mode = FormatNegotiationMode.Force); b.AddBroadcast(); if (configureFirst) { @@ -843,7 +843,11 @@ private static ServiceProvider ResolveProvider(Action configure configure(b); } }, - o => o.SourceUri = Source) + o => + { + o.AppShortName = "mp"; + o.SourceUri = Source; + }) .BuildServiceProvider(); private class DisposableFormatter : IEventFormatterProxy, IDisposable, IAsyncDisposable diff --git a/tests/UiPath.Caching.Tests/MessagePack/MessagePackEventIntegrationTests.cs b/tests/UiPath.Caching.Tests/MessagePack/MessagePackEventIntegrationTests.cs index 37adc989..85db0768 100644 --- a/tests/UiPath.Caching.Tests/MessagePack/MessagePackEventIntegrationTests.cs +++ b/tests/UiPath.Caching.Tests/MessagePack/MessagePackEventIntegrationTests.cs @@ -197,7 +197,8 @@ private ServiceProvider Node(string source, Events events, bool pubSub = false, b => { b.AddRedisConnection(o => o.ConnectionString = fixture.ConnectionString); - b.AddRedis(_ => { }); + // These tests are about the wire; the automatic switch has its own. + b.AddRedis(o => o.FormatNegotiation.Mode = FormatNegotiationMode.Force); if (pubSub) { b.AddRedisPubSub(o => o.Enabled = true); diff --git a/tests/UiPath.Caching.Tests/Negotiation/EventFormatNegotiationIntegrationTests.cs b/tests/UiPath.Caching.Tests/Negotiation/EventFormatNegotiationIntegrationTests.cs new file mode 100644 index 00000000..c42d4398 --- /dev/null +++ b/tests/UiPath.Caching.Tests/Negotiation/EventFormatNegotiationIntegrationTests.cs @@ -0,0 +1,360 @@ +using System.Collections.Concurrent; +using System.Net; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using StackExchange.Redis; +using UiPath.Caching.Broadcast.Redis; +using UiPath.Caching.Config; +using UiPath.Caching.MessagePack; +using UiPath.Caching.Negotiation; +using UiPath.Caching.Tests.Redis; + +namespace UiPath.Caching.Tests.Negotiation; + +[Collection("RedisIntegration")] +[Trait("Category", "Integration")] +public class EventFormatNegotiationIntegrationTests(RedisContainerFixture fixture) +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task Sum_over_all_endpoints_counts_every_connection_that_subscribed() + { + Assert.SkipUnless(fixture.Enabled, "Set RUN_REDIS_INTEGRATION_TESTS=1 (Docker required) to run."); + await using var first = await ConnectionMultiplexer.ConnectAsync(fixture.ConnectionString); + await using var second = await ConnectionMultiplexer.ConnectAsync(fixture.ConnectionString); + await using var asking = await ConnectionMultiplexer.ConnectAsync(fixture.ConnectionString); + var channel = RedisChannel.Literal($"fe-{Guid.NewGuid():N}"); + await first.GetSubscriber().SubscribeAsync(channel, (_, _) => { }); + await second.GetSubscriber().SubscribeAsync(channel, (_, _) => { }); + var source = new RedisSubscriberSource(new Connector(asking)); + var nodeList = new PubSubNodeList(source); + + source.Endpoints().Should().NotBeEmpty(); + await nodeList.ReadAsync([channel.ToString()], new Dictionary { [channel.ToString()] = 1 }, Ct); + var reading = await nodeList.ReadAsync([channel.ToString()], new Dictionary { [channel.ToString()] = 1 }, Ct); + + reading.Known.Should().BeTrue(); + reading.Surplus.Should().Be(1, "two connections subscribed and one is accounted for"); + reading.Unaccounted.Should().BeTrue(); + var other = RedisChannel.Literal($"fe-{Guid.NewGuid():N}"); + await first.GetSubscriber().SubscribeAsync(other, (_, _) => { }); + (await source.CountAsync(source.Endpoints()[0], [channel.ToString(), other.ToString(), $"fe-nobody-{Guid.NewGuid():N}"], Ct))!.Channels.Should().Equal(new long[] { 2, 1, 0 }, "one command, every channel, in the order asked"); + } + + [Fact] + public async Task A_connection_with_a_channel_prefix_is_counted_under_the_names_it_really_subscribed_to() + { + Assert.SkipUnless(fixture.Enabled, "Set RUN_REDIS_INTEGRATION_TESTS=1 (Docker required) to run."); + var options = ConfigurationOptions.Parse(fixture.ConnectionString); + options.ChannelPrefix = RedisChannel.Literal($"pfx-{Guid.NewGuid():N}:"); + await using var prefixed = await ConnectionMultiplexer.ConnectAsync(options); + await using var another = await ConnectionMultiplexer.ConnectAsync(options); + var channel = $"fe-{Guid.NewGuid():N}"; + await prefixed.GetSubscriber().SubscribeAsync(RedisChannel.Literal(channel), (_, _) => { }); + await another.GetSubscriber().SubscribeAsync(RedisChannel.Literal(channel), (_, _) => { }); + var source = new RedisSubscriberSource(new Connector(prefixed)); + + var counted = await source.CountAsync(source.Endpoints()[0], [channel], Ct); + + // The server knows the channel only with the prefix in front. Asked as the bare name, there is nobody on it. + counted!.Channels.Should().Equal(new long[] { 2 }, "two connections subscribed under the prefix, and the count is of those"); + await using var bare = await ConnectionMultiplexer.ConnectAsync(fixture.ConnectionString); + (await new RedisSubscriberSource(new Connector(bare)).CountAsync(source.Endpoints()[0], [channel], Ct))!.Channels.Should().Equal(new long[] { 0 }); + } + + [Fact] + public async Task A_pattern_subscriber_is_counted_by_the_server_and_claims_account_for_it() + { + Assert.SkipUnless(fixture.Enabled, "Set RUN_REDIS_INTEGRATION_TESTS=1 (Docker required) to run."); + await using var holder = await ConnectionMultiplexer.ConnectAsync(fixture.ConnectionString); + await using var asking = await ConnectionMultiplexer.ConnectAsync(fixture.ConnectionString); + var pattern = RedisChannel.Pattern($"fe-{Guid.NewGuid():N}-*"); + var channel = $"fe-{Guid.NewGuid():N}"; + await asking.GetSubscriber().SubscribeAsync(RedisChannel.Literal(channel), (_, _) => { }); + var source = new RedisSubscriberSource(new Connector(asking)); + var before = (await source.CountAsync(source.Endpoints()[0], [channel], Ct))!.Patterns; + await holder.GetSubscriber().SubscribeAsync(pattern, (_, _) => { }); + var nodeList = new PubSubNodeList(source); + var accounted = new Dictionary { [channel] = 1, [PubSubNodeList.PatternKey] = before }; + + await nodeList.ReadAsync([channel], accounted, Ct); + var unadvertised = await nodeList.ReadAsync([channel], accounted, Ct); + accounted[PubSubNodeList.PatternKey] = before + 1; + await nodeList.ReadAsync([channel], accounted, Ct); + var advertised = await nodeList.ReadAsync([channel], accounted, Ct); + + unadvertised.Unaccounted.Should().BeTrue("NUMSUB cannot see a pattern subscriber, NUMPAT can"); + unadvertised.Channels.Should().Equal(PubSubNodeList.PatternKey); + advertised.Unaccounted.Should().BeFalse(); + advertised.Settled.Should().BeTrue(); + } + + [Fact] + public async Task A_confirmed_subscription_is_already_counted_by_the_server() + { + Assert.SkipUnless(fixture.Enabled, "Set RUN_REDIS_INTEGRATION_TESTS=1 (Docker required) to run."); + var app = App(); + await using var raw = await ConnectionMultiplexer.ConnectAsync(fixture.ConnectionString); + using var node = Node(app, "node-a", pubSub: true, write: true); + var provider = (RedisPubSubTopicProvider)node.GetRequiredService().Get(KnownTopicNames.RedisPubSub); + var subscriptions = (IPubSubSubscriptions)provider; + var channel = $"{app}:ps:string"; + provider.Create(new TopicKey("string")); + subscriptions.Channels().Should().Equal(channel); + subscriptions.Confirmed().Should().BeEmpty("the subscription goes in place on a timer, not when the topic is made"); + + // Whenever the node would claim the subscription, the server must already count it: the claim never comes first. + var until = DateTime.UtcNow.AddSeconds(30); + while (subscriptions.Confirmed().Count == 0) + { + Assert.True(DateTime.UtcNow < until, "The subscription was not confirmed in time."); + await Task.Delay(1, Ct); + } + + raw.GetServer(raw.GetEndPoints()[0]).SubscriptionSubscriberCount(RedisChannel.Literal(channel)).Should().Be(1); + subscriptions.Confirmed().Should().BeEquivalentTo(new Dictionary { [channel] = 1 }); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_pattern_or_sharded_channel_is_listed_as_uncountable_and_never_claimed(bool sharded) + { + Assert.SkipUnless(fixture.Enabled, "Set RUN_REDIS_INTEGRATION_TESTS=1 (Docker required) to run."); + var app = App(); + using var node = Node(app, "node-a", pubSub: true, write: true, new ChannelStrategy(app, sharded)); + var provider = (RedisPubSubTopicProvider)node.GetRequiredService().Get(KnownTopicNames.RedisPubSub); + var subscriptions = (IPubSubSubscriptions)provider; + + provider.Create(new TopicKey("string")); + + subscriptions.Channels().Should().BeEmpty("PUBSUB NUMSUB counts classic literal channels only"); + subscriptions.Uncountable().Should().Equal($"{app}:ps:string"); + subscriptions.Confirmed().Should().BeEmpty(); + } + + [Fact] + public async Task A_pub_sub_subscriber_that_advertises_nothing_keeps_events_json_until_it_leaves_and_returns_them_to_json_when_it_comes_back() + { + Assert.SkipUnless(fixture.Enabled, "Set RUN_REDIS_INTEGRATION_TESTS=1 (Docker required) to run."); + var app = App(); + var channel = RedisChannel.Literal($"{app}:ps:string"); + var topic = new TopicKey("string"); + await using var raw = await ConnectionMultiplexer.ConnectAsync(fixture.ConnectionString); + var db = raw.GetDatabase(); + using var a = Node(app, "node-a", pubSub: true, write: true); + using var b = Node(app, "node-b", pubSub: true, write: true); + var serviceA = Service(a); + var serviceB = Service(b); + var writes = a.GetRequiredService(); + var cache = a.GetRequiredService().CreateCache(KnownCacheProviderNames.InMemoryRedis); + var received = new ConcurrentQueue(); + var nodeListKey = NodeListStore.KeyFor(new CacheOptions { AppShortName = app }, new RedisCacheOptions()); + try + { + a.GetRequiredService().Get(KnownTopicNames.RedisPubSub).Create(topic); + b.GetRequiredService().Get(KnownTopicNames.RedisPubSub).Create(topic); + await serviceA.StartAsync(Ct); + await serviceB.StartAsync(Ct); + + // Both nodes subscribed and said so: nothing is unaccounted for, so after the hold the events are binary. + await WaitAsync(() => writes.Writes, TimeSpan.FromSeconds(40)); + await raw.GetSubscriber().SubscribeAsync(channel, (_, value) => received.Enqueue((byte[])value!)); + + // A subscriber nobody advertised: two views, and the writes are JSON again. + await WaitAsync(() => !writes.Writes, TimeSpan.FromSeconds(15)); + await cache.SetAsync("k1", "json-while-blocked", TimeSpan.FromMinutes(1), policy: null, Ct); + await WaitAsync(() => received.Any(body => body[0] == (byte)'{'), TimeSpan.FromSeconds(10)); + received.Should().NotContain(body => body[0] == CacheValueFormats.MessagePack); + + // It leaves: two clean views and the hold, and they are binary. + await raw.GetSubscriber().UnsubscribeAsync(channel); + await WaitAsync(() => writes.Writes, TimeSpan.FromSeconds(40)); + received.Clear(); + await raw.GetSubscriber().SubscribeAsync(channel, (_, value) => received.Enqueue((byte[])value!)); + await cache.SetAsync("k2", "binary-until-it-returns", TimeSpan.FromMinutes(1), policy: null, Ct); + await WaitAsync(() => !received.IsEmpty, TimeSpan.FromSeconds(10)); + received.Should().Contain(body => body[0] == CacheValueFormats.MessagePack, "the returning subscriber is not yet seen, and a node that reads both loses nothing"); + + // Seen in two views, it stops them again. + await WaitAsync(() => !writes.Writes, TimeSpan.FromSeconds(15)); + } + finally + { + await serviceA.StopAsync(CancellationToken.None); + await serviceB.StopAsync(CancellationToken.None); + await db.KeyDeleteAsync(nodeListKey); + } + } + + [Fact] + public async Task A_stream_group_that_advertises_nothing_keeps_events_json_until_it_goes_idle_and_a_returning_one_stops_the_binary_writes_at_once() + { + Assert.SkipUnless(fixture.Enabled, "Set RUN_REDIS_INTEGRATION_TESTS=1 (Docker required) to run."); + var app = App(); + var topic = new TopicKey("string"); + await using var raw = await ConnectionMultiplexer.ConnectAsync(fixture.ConnectionString); + var db = raw.GetDatabase(); + using var node = Node(app, "node-a", pubSub: false, write: true); + var service = Service(node); + var writes = node.GetRequiredService(); + var cache = node.GetRequiredService().CreateCache(KnownCacheProviderNames.InMemoryRedis); + var streamKey = new PrefixStrategy(RedisKeyspaces.Streams, new CacheOptions { AppShortName = app }).GetRedisKey(topic); + var nodeListKey = NodeListStore.KeyFor(new CacheOptions { AppShortName = app }, new RedisCacheOptions()); + try + { + await db.StreamAddAsync(streamKey, "f", "v"); + await db.StreamCreateConsumerGroupAsync(streamKey, "urn:node-old", "0-0"); + await db.StreamReadGroupAsync(streamKey, "urn:node-old", "urn:node-old", ">", 1); + node.GetRequiredService().Get(KnownTopicNames.RedisStreams).Create(topic); + await service.StartAsync(Ct); + + for (var i = 0; i < 8; i++) + { + await db.StreamReadGroupAsync(streamKey, "urn:node-old", "urn:node-old", ">", 1); + await Task.Delay(500, Ct); + } + + writes.Writes.Should().BeFalse("a live group without an advertisement may not read binary events"); + await cache.SetAsync("k1", "json-while-blocked", TimeSpan.FromMinutes(1), policy: null, Ct); + (await LastBodyAsync(db, streamKey))[0].Should().Be((byte)'{'); + + await WaitAsync(() => writes.Writes, TimeSpan.FromSeconds(30)); + await cache.SetAsync("k2", "binary", TimeSpan.FromMinutes(1), policy: null, Ct); + (await LastBodyAsync(db, streamKey))[0].Should().Be(CacheValueFormats.MessagePack); + + await db.StreamReadGroupAsync(streamKey, "urn:node-old", "urn:node-old", ">", 1); + await WaitAsync(() => !writes.Writes, TimeSpan.FromSeconds(5)); + await cache.SetAsync("k3", "json-again", TimeSpan.FromMinutes(1), policy: null, Ct); + (await LastBodyAsync(db, streamKey))[0].Should().Be((byte)'{'); + } + finally + { + await service.StopAsync(CancellationToken.None); + await db.KeyDeleteAsync([streamKey, nodeListKey]); + } + } + + private static async Task LastBodyAsync(IDatabase db, RedisKey stream) + { + // The topic publishes just after the call returns on a background thread; look until the entry is there. + for (var i = 0; i < 50; i++) + { + var last = await db.StreamRangeAsync(stream, count: 1, messageOrder: Order.Descending); + if (last.Length > 0 && last[0].Values.FirstOrDefault(v => v.Value.Length() > 1) is { Value.IsNull: false } body) + { + return (byte[])body.Value!; + } + + await Task.Delay(100, Ct); + } + + throw new TimeoutException("No event reached the stream."); + } + + private static async Task WaitAsync(Func condition, TimeSpan timeout) + { + var until = DateTime.UtcNow + timeout; + while (!condition()) + { + Assert.True(DateTime.UtcNow < until, "The condition did not hold in time."); + await Task.Delay(100, Ct); + } + } + + private static string App() => $"fe{Guid.NewGuid():N}"[..12]; + + private static FormatNegotiationService Service(ServiceProvider node) => + node.GetServices().OfType().Single(); + + private ServiceProvider Node(string app, string source, bool pubSub, bool write, IRedisChannelStrategy? strategy = null) => + new ServiceCollection() + .AddLogging() + .AddCaching( + b => + { + b.AddRedisConnection(o => o.ConnectionString = fixture.ConnectionString); + b.AddRedis(o => + { + o.FormatNegotiation.AdvertiseInterval = TimeSpan.FromSeconds(1); + o.FormatNegotiation.ViewInterval = TimeSpan.FromSeconds(1); + o.FormatNegotiation.LivenessWindow = TimeSpan.FromSeconds(3); + o.FormatNegotiation.UpgradeHold = TimeSpan.FromSeconds(2); + o.FormatNegotiation.MaxViewAge = TimeSpan.FromSeconds(10); + }); + if (pubSub) + { + b.AddRedisPubSub(o => + { + o.Enabled = true; + o.RedisChannelStrategy = strategy; + }); + b.AddRedisStreams(o => { o.Enabled = false; o.MaintainerEnabled = false; }); + } + else + { + b.AddRedisStreams(o => { o.Enabled = true; o.MaintainerEnabled = false; }); + } + + b.AddBroadcast(); + b.AddInMemoryRedis(o => o.ClearLocalOnReconnect = pubSub ? false : null); + b.AddMessagePackEvents(write); + }, + o => + { + o.AppShortName = app; + o.SourceUri = new Uri($"urn:{source}"); + if (pubSub) + { + o.DefaultTopic = KnownTopicNames.RedisPubSub; + } + }) + .BuildServiceProvider(); + + private sealed class ChannelStrategy(string app, bool sharded) : IRedisChannelStrategy + { + public RedisChannel GetRedisChannel(TopicKey topicKey) => sharded + ? RedisChannel.Sharded($"{app}:ps:{topicKey}") + : RedisChannel.Pattern($"{app}:ps:{topicKey}"); + } + + private sealed class Connector(IConnectionMultiplexer mux) : IRedisConnector + { + public event EventHandler? OnConnectionFailed + { + add { } + remove { } + } + + public event EventHandler? OnConnectionRestored + { + add { } + remove { } + } + + public event EventHandler? OnReconnected + { + add { } + remove { } + } + + public bool IsConnected => mux.IsConnected; + + public Version Version => mux.GetServer(mux.GetEndPoints()[0]).Version; + + public IDatabase Database => mux.GetDatabase(); + + public ISubscriber Subscriber => mux.GetSubscriber(); + + public void ForceReconnect() + { + } + + public EndPoint[] GetEndPoints(bool configuredOnly = false) => mux.GetEndPoints(configuredOnly); + + public void Dispose() + { + } + } +} diff --git a/tests/UiPath.Caching.Tests/Negotiation/EventFormatNegotiationTests.cs b/tests/UiPath.Caching.Tests/Negotiation/EventFormatNegotiationTests.cs new file mode 100644 index 00000000..1d7cb694 --- /dev/null +++ b/tests/UiPath.Caching.Tests/Negotiation/EventFormatNegotiationTests.cs @@ -0,0 +1,222 @@ +using UiPath.Caching.Broadcast; +using UiPath.Caching.MessagePack; +using UiPath.Caching.Negotiation; +using UiPath.Caching.Tests.Fakes; +using UiPath.Caching.Tests.Telemetry; + +namespace UiPath.Caching.Tests.Negotiation; + +public class EventFormatNegotiationTests +{ + private static readonly string[] Binary = ["02"]; + private static readonly string[] Json = ["json"]; + private static readonly string[] Both = ["json", "02"]; + private static readonly string[] Value = ["10"]; + + private readonly ManualTimeProvider _clock = new(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero)); + private readonly RecordingTelemetryProvider _telemetry = new(); + private readonly FakeValueSerializer _writer = new(0x10); + private readonly FormatNegotiationOptions _options = new(); + + [Fact] + public void Binary_only_when_all_can_read_events() + { + var sut = Negotiator(); + sut.Writes.Should().BeFalse("no node list has been read"); + + Settle(sut, View(Node("a", Both), Node("b", Both))); + sut.Writes.Should().BeTrue(); + + sut.Observe(View(Node("a", Both), Node("b", Both), Node("c", Json))); + sut.Writes.Should().BeFalse("one node that reads only JSON events"); + _telemetry.Metrics.Should().Contain(m => m.Name == Metrics.FormatFallback && m.Properties!["reason"] == FormatNegotiator.ReasonMissingFormat && m.Properties["kind"] == "event"); + _telemetry.Metrics.Where(m => m.Name == Metrics.FormatFallback && m.Properties!["kind"] == "event").Should().OnlyContain(m => !m.Properties!.ContainsKey("policy"), "an event has no policy, and an empty tag is a tag value all the same"); + } + + [Fact] + public void The_switch_waits_for_the_hold_and_the_fall_back_does_not() + { + var sut = Negotiator(); + var view = View(Node("a", Both)); + + sut.Observe(view); + sut.Writes.Should().BeFalse("the node list has only just agreed"); + _clock.Advance(_options.UpgradeHold - TimeSpan.FromSeconds(1)); + sut.Observe(view); + sut.Writes.Should().BeFalse(); + _clock.Advance(TimeSpan.FromSeconds(1)); + sut.Observe(view); + sut.Writes.Should().BeTrue(); + + sut.Observe(View(Node("a", Both), Node("old", Json))); + + sut.Writes.Should().BeFalse("the first node that cannot read them stops the writes at once"); + } + + [Fact] + public void Legacy_group_flips_event_writes_to_json() + { + var sut = Negotiator(); + Settle(sut, View(Node("a", Both))); + sut.Writes.Should().BeTrue(); + + sut.Observe(new NodeListView([Stored(Node("a", Both))], ["urn:legacy"], Complete: true)); + + sut.Writes.Should().BeFalse(); + _telemetry.Metrics.Should().Contain(m => m.Name == Metrics.FormatFallback && m.Properties!["reason"] == FormatNegotiator.ReasonUnaccountedGroup && m.Properties["kind"] == "event"); + } + + [Fact] + public void A_node_without_the_decoder_blocks_binary_events_and_not_values() + { + var sut = Negotiator(values: true); + var withoutDecoder = Node("plain", Json, valueReads: ["json", "10"]); + + Settle(sut, View(Node("a", Both, valueReads: ["json", "10"]), withoutDecoder)); + + sut.Writes.Should().BeFalse("it lists er=json"); + sut.Allows(_writer).Should().BeTrue("it reads the value format, and the two decisions are apart"); + } + + [Fact] + public void An_ad_that_lists_no_event_formats_blocks_binary_events() + { + var sut = Negotiator(); + var ad = new NodeAdvertisement("3.2.0", "pre", ["json"], new Dictionary>()); + + Settle(sut, View(Node("a", Both), ("pre", ad))); + + sut.Writes.Should().BeFalse("an advertisement that says nothing about events is a build that does not know them"); + } + + [Fact] + public void A_node_that_is_not_set_to_write_binary_events_never_does() + { + var sut = new FormatNegotiator([], _options, _clock, _telemetry); + + Settle(sut, View(Node("a", Both))); + + sut.Writes.Should().BeFalse(); + } + + [Fact] + public void A_view_that_stops_refreshing_ends_binary_events_after_the_max_view_age() + { + var sut = Negotiator(); + Settle(sut, View(Node("a", Both))); + sut.Writes.Should().BeTrue(); + + _clock.Advance(_options.MaxViewAge + TimeSpan.FromSeconds(1)); + + sut.Writes.Should().BeFalse(); + } + + [Fact] + public void The_effective_event_format_is_reported_with_its_kind_and_no_node_id() + { + var sut = Negotiator(); + Settle(sut, View(Node("node-id-that-must-not-be-a-tag", Both))); + + _telemetry.Metrics.Should().Contain(m => m.Name == Metrics.FormatEffective && m.Properties!["kind"] == "event" && m.Properties["format"] == "02" && m.Value == 1); + _telemetry.Metrics.Where(m => m.Name.StartsWith("cache.format.", StringComparison.Ordinal)) + .SelectMany(m => (IEnumerable?)m.Properties?.Values ?? []) + .Should().NotContain(value => value.Contains("node-id", StringComparison.Ordinal)); + } + + [Fact] + public void A_blocking_stream_node_is_named_on_the_blockers_metric_and_never_on_the_bounded_ones() + { + var sut = Negotiator(); + + sut.Observe(new NodeListView([Stored(Node("a", Both))], ["urn:legacy"], Complete: true)); + + _telemetry.Metrics.Should().Contain(m => m.Name == Metrics.FormatBlockers && m.Properties!["nodes"] == "urn:legacy"); + _telemetry.Metrics.Where(m => m.Name.StartsWith("cache.format.", StringComparison.Ordinal)) + .SelectMany(m => (IEnumerable?)m.Properties?.Values ?? []) + .Should().NotContain("urn:legacy"); + } + + [Fact] + public void Force_writes_binary_events_at_once_and_a_node_set_to_json_never_does() + { + var forced = new FormatNegotiation { Negotiator = null, Advertisement = new("3.2.0", "me", ["json"], new Dictionary>()) }; + var automatic = new FormatNegotiation { Negotiator = Negotiator(), Advertisement = forced.Advertisement }; + + forced.EventWrites(new BinaryEventSettings(true)).Writes.Should().BeTrue("Force writes what is configured, with no node list"); + automatic.EventWrites(new BinaryEventSettings(true)).Writes.Should().BeFalse("Auto waits for a node list"); + forced.EventWrites(new BinaryEventSettings(false)).Writes.Should().BeFalse(); + automatic.EventWrites(new BinaryEventSettings(false)).Writes.Should().BeFalse(); + FormatNegotiation.Inactive.EventWrites(new BinaryEventSettings(true)).Writes.Should().BeFalse("with no node list to ask, nothing proves binary is safe"); + new FormatNegotiation { Options = new() { Mode = FormatNegotiationMode.Force } }.EventWrites(new BinaryEventSettings(true)).Writes.Should().BeTrue("Force needs no node list"); + new FormatNegotiation { Options = new() { Mode = FormatNegotiationMode.Force } }.EventWrites(new BinaryEventSettings(false)).Writes.Should().BeFalse(); + } + + [Fact] + public void Decoder_reads_both_after_fallback() + { + var sut = Negotiator(); + var json = new CacheEventFormatter(); + var composite = new CompositeEventFormatter(json, new MessagePackEventFormatter(), sut); + var original = new CacheEvent { Id = Guid.NewGuid().ToString(), Type = KnownEventTypes.CacheSet, Source = new Uri("urn:node-a"), Data = new CacheEventData("the-key", new Dictionary { [KnownFieldNames.VersionKey] = 3L }) }; + Settle(sut, View(Node("a", Both))); + var binary = composite.Encode(original); + binary.Span[0].Should().Be(CacheValueFormats.MessagePack); + + sut.Observe(View(Node("a", Both), Node("old", Json))); + var fallback = composite.Encode(original); + + fallback.Span[0].Should().Be((byte)'{', "the writes fell back to JSON"); + composite.Decode(binary)!.Id.Should().Be(original.Id, "a backlog written before the fall back is still read"); + composite.Decode(fallback)!.Id.Should().Be(original.Id); + } + + [Fact] + public void Encoding_asks_the_decision_on_every_event_and_allocates_no_more_than_a_fixed_one() + { + var sut = Negotiator(); + Settle(sut, View(Node("a", Both))); + sut.Observe(View(Node("a", Both), Node("old", Json))); + var original = new CacheEvent { Id = Guid.NewGuid().ToString(), Type = KnownEventTypes.CacheSet, Source = new Uri("urn:node-a"), Data = new CacheEventData("the-key", null) }; + var negotiated = new CompositeEventFormatter(new CacheEventFormatter(), new MessagePackEventFormatter(), sut); + var fixedJson = new CompositeEventFormatter(new CacheEventFormatter(), new MessagePackEventFormatter(), FixedBinaryEvents.Never); + var plain = new CacheEventFormatter(); + + var viaNodeList = Allocated(() => negotiated.Encode(original)); + var viaFixed = Allocated(() => fixedJson.Encode(original)); + var direct = Allocated(() => plain.Encode(original)); + + viaNodeList.Should().Be(viaFixed, "reading the decision costs nothing"); + viaFixed.Should().Be(direct, "and neither does the wrapper"); + } + + private static long Allocated(Action action) + { + action(); + action(); + var before = GC.GetAllocatedBytesForCurrentThread(); + for (var i = 0; i < 100; i++) + { + action(); + } + + return GC.GetAllocatedBytesForCurrentThread() - before; + } + + private static (string Id, NodeAdvertisement Ad) Node(string id, string[] events, string[]? valueReads = null) => + (id, new("3.2.0", id, valueReads ?? Json, new Dictionary>(), EventReads: events)); + + private static StoredAdvertisement Stored((string Id, NodeAdvertisement Ad) node) => new(node.Id, node.Ad, TimeSpan.Zero); + + private static NodeListView View(params (string Id, NodeAdvertisement Ad)[] nodes) => + new([.. nodes.Select(Stored)], [], Complete: true); + + private FormatNegotiator Negotiator(bool values = false) => + new(values ? [new DesiredWrite(_writer, Value, ["default"], [])] : [], _options, _clock, _telemetry, events: new DesiredEvents(Binary)); + + private void Settle(FormatNegotiator sut, NodeListView view) + { + sut.Observe(view); + _clock.Advance(_options.UpgradeHold); + sut.Observe(view); + } +} diff --git a/tests/UiPath.Caching.Tests/Negotiation/EventNegotiationWiringTests.cs b/tests/UiPath.Caching.Tests/Negotiation/EventNegotiationWiringTests.cs new file mode 100644 index 00000000..8ea9ff58 --- /dev/null +++ b/tests/UiPath.Caching.Tests/Negotiation/EventNegotiationWiringTests.cs @@ -0,0 +1,161 @@ +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using UiPath.Caching.Broadcast; +using UiPath.Caching.Config; +using UiPath.Caching.MessagePack; +using UiPath.Caching.Negotiation; +using UiPath.Caching.Tests.Fakes; + +namespace UiPath.Caching.Tests.Negotiation; + +public class EventNegotiationWiringTests +{ + private static readonly Uri Source = new("urn:node-a"); + + [Fact] + public void A_node_with_the_binary_decoder_advertises_both_event_formats_and_one_without_advertises_json() + { + using var plain = Provider(b => b.AddValueSerializer("fast", _ => new FakeValueSerializer(0x10))); + using var decoder = Provider(b => + { + b.AddValueSerializer("fast", _ => new FakeValueSerializer(0x10)); + b.AddMessagePackEvents(write: false); + }); + + plain.GetRequiredService().Advertisement!.EventReads.Should().Equal("json"); + decoder.GetRequiredService().Advertisement!.EventReads.Should().Equal("json", "02"); + NodeAdvertisement.TryParse(plain.GetRequiredService().Advertisement!.Encode(), out var parsed).Should().BeTrue(); + parsed!.EventReads.Should().Equal("json"); + } + + [Fact] + public void An_app_with_only_binary_events_advertises_decides_and_starts_the_service() + { + using var writes = Provider(b => b.AddMessagePackEvents(write: true)); + using var reads = Provider(b => b.AddMessagePackEvents(write: false)); + + var deciding = writes.GetRequiredService(); + deciding.IsActive.Should().BeTrue(); + deciding.Advertisement!.ValueReads.Should().Equal("json"); + deciding.Advertisement.EventReads.Should().Equal("json", "02"); + deciding.Negotiator.Should().NotBeNull(); + writes.GetRequiredService().Writes.Should().BeFalse("Auto writes JSON until a node list says otherwise"); + writes.GetServices().Should().Contain(s => s is FormatNegotiationService); + + var reader = reads.GetRequiredService(); + reader.IsActive.Should().BeTrue(); + reader.Negotiator.Should().BeNull("a node set to write JSON has nothing to decide, so it only advertises"); + reads.GetRequiredService().Writes.Should().BeFalse(); + } + + [Fact] + public void Force_writes_binary_events_at_once() + { + using var forced = Provider(b => b.AddMessagePackEvents(write: true), force: true); + + forced.GetRequiredService().Writes.Should().BeTrue(); + forced.GetRequiredService().Negotiator.Should().BeNull(); + forced.GetRequiredService().IsActive.Should().BeTrue("it still tells the nodes that decide what it reads"); + forced.GetRequiredService>().Encode(Event()).Span[0].Should().Be(CacheValueFormats.MessagePack); + } + + [Fact] + public void With_no_redis_connection_there_is_no_node_list_and_binary_events_wait_for_the_operator() + { + using var automatic = NoRedis(write: true, mode: FormatNegotiationMode.Auto); + using var attested = NoRedis(write: true, mode: FormatNegotiationMode.Auto, assume: true); + using var forced = NoRedis(write: true, mode: FormatNegotiationMode.Force); + using var json = NoRedis(write: false, mode: FormatNegotiationMode.Auto, assume: true); + + automatic.GetRequiredService().Writes.Should().BeFalse("silent until proven"); + automatic.GetRequiredService>().Encode(Event()).Span[0].Should().Be((byte)'{'); + forced.GetRequiredService().Writes.Should().BeTrue(); + forced.GetRequiredService>().Encode(Event()).Span[0].Should().Be(CacheValueFormats.MessagePack); + json.GetRequiredService().Writes.Should().BeFalse(); + attested.GetRequiredService().Writes.Should().BeFalse("the operator's word counts once the service has started, and the hold has passed"); + } + + [Fact] + public async Task With_no_redis_connection_the_operators_word_switches_after_the_hold() + { + using var attested = NoRedis(write: true, mode: FormatNegotiationMode.Auto, assume: true); + var service = attested.GetServices().OfType().Single(); + + await service.StartAsync(TestContext.Current.CancellationToken); + + attested.GetRequiredService().Writes.Should().BeTrue("the hold is zero here"); + await service.StopAsync(TestContext.Current.CancellationToken); + } + + [Fact] + public void An_app_with_nothing_registered_adds_nothing_and_encodes_as_it_always_did() + { + using var plain = Provider(_ => { }); + var services = new ServiceCollection(); + var builder = new CachingBuilder(services); + builder.Complete(); + + services.Should().NotContain(d => d.ServiceType == typeof(FormatNegotiation) || d.ServiceType == typeof(IBinaryEvents) || d.ServiceType == typeof(BinaryEventSettings)); + var formatter = plain.GetRequiredService>(); + formatter.Should().BeOfType("no wrapper, so no decision to ask for"); + var direct = new CacheEventFormatter(); + var @event = Event(); + + var viaContainer = Allocated(() => formatter.Encode(@event)); + var baseline = Allocated(() => direct.Encode(@event)); + + viaContainer.Should().Be(baseline, "an app that registers nothing pays nothing for encoding an event"); + } + + private static long Allocated(Action action) + { + action(); + action(); + var before = GC.GetAllocatedBytesForCurrentThread(); + for (var i = 0; i < 100; i++) + { + action(); + } + + return GC.GetAllocatedBytesForCurrentThread() - before; + } + + private static CacheEvent Event() => new() + { + Id = Guid.NewGuid().ToString(), + Type = KnownEventTypes.CacheSet, + Source = Source, + Data = new CacheEventData("the-key", new Dictionary { [KnownFieldNames.VersionKey] = 3L }), + }; + + private static ServiceProvider NoRedis(bool write, FormatNegotiationMode mode, bool assume = false) => + new ServiceCollection() + .AddLogging() + .Configure(o => + { + o.FormatNegotiation.Mode = mode; + o.FormatNegotiation.AssumeAllNodesCanRead = assume; + o.FormatNegotiation.UpgradeHold = TimeSpan.Zero; + }) + .AddCaching(b => b.AddMessagePackEvents(write), o => o.AppShortName = "wiring") + .BuildServiceProvider(); + + private static ServiceProvider Provider(Action configure, bool force = false) => + new ServiceCollection() + .AddLogging() + .AddCaching( + b => + { + b.AddRedisConnection(o => o.ConnectionString = "127.0.0.1:1"); + b.AddRedis(o => o.FormatNegotiation.Mode = force ? FormatNegotiationMode.Force : FormatNegotiationMode.Auto); + b.AddBroadcast(); + b.AddInMemoryRedis(); + configure(b); + }, + o => + { + o.AppShortName = "wiring"; + o.SourceUri = Source; + }) + .BuildServiceProvider(); +} diff --git a/tests/UiPath.Caching.Tests/Negotiation/FakeNodeListServer.cs b/tests/UiPath.Caching.Tests/Negotiation/FakeNodeListServer.cs index ad808c13..dda54989 100644 --- a/tests/UiPath.Caching.Tests/Negotiation/FakeNodeListServer.cs +++ b/tests/UiPath.Caching.Tests/Negotiation/FakeNodeListServer.cs @@ -7,6 +7,7 @@ namespace UiPath.Caching.Tests.Negotiation; internal sealed class FakeNodeListServer { private readonly Dictionary _hash = []; + private int _inFlight; public FakeNodeListServer(bool scripts = true) { @@ -15,7 +16,10 @@ public FakeNodeListServer(bool scripts = true) Connector.Database.Returns(Database); Database.ScriptEvaluateAsync(Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any()) .Returns(call => Failure is { } failure ? Task.FromException(failure) : scripts - ? Task.FromResult(Script(call.ArgAt(0), call.ArgAt(2))) + ? (WriteDelay is { } delay && call.ArgAt(0).Contains("HSET", StringComparison.Ordinal) + ? DelayedScript(delay, call.ArgAt(0), call.ArgAt(2)) + : Task.FromResult(Script(call.ArgAt(0), call.ArgAt(2)))) + #pragma warning disable CS0618 // The only public constructor of a server error. : Task.FromException(new RedisServerException("ERR unknown command 'EVAL'"))); #pragma warning restore CS0618 @@ -39,6 +43,12 @@ public FakeNodeListServer(bool scripts = true) public IDatabase Database { get; } + /// When set, a write waits for it before it lands, as a slow Redis would hold it. + public Func? WriteDelay { get; set; } + + /// The most writes that were in flight at once. + public int MaxConcurrentWrites { get; private set; } + /// When set, every script fails with it, as an unreachable Redis does. public Exception? Failure { get; set; } @@ -55,6 +65,24 @@ public FakeNodeListServer(bool scripts = true) /// Stores a raw field, as another node or version would have written it. public void Put(string field, string value) => _hash[field] = value; + private async Task DelayedScript(Func delay, string script, RedisValue[] arguments) + { + var inFlight = Interlocked.Increment(ref _inFlight); + MaxConcurrentWrites = Math.Max(MaxConcurrentWrites, inFlight); + try + { + await delay().ConfigureAwait(false); + lock (_hash) + { + return Script(script, arguments); + } + } + finally + { + Interlocked.Decrement(ref _inFlight); + } + } + private RedisResult Script(string script, RedisValue[] arguments) { if (script.Contains("HDEL", StringComparison.Ordinal)) diff --git a/tests/UiPath.Caching.Tests/Negotiation/NodeAdvertisementTests.cs b/tests/UiPath.Caching.Tests/Negotiation/NodeAdvertisementTests.cs index 03d50457..6f93fe36 100644 --- a/tests/UiPath.Caching.Tests/Negotiation/NodeAdvertisementTests.cs +++ b/tests/UiPath.Caching.Tests/Negotiation/NodeAdvertisementTests.cs @@ -31,6 +31,56 @@ public void Ad_round_trips_formats_version_and_strict_lists() parsed.Left.Should().BeTrue(); } + [Fact] + public void Ad_round_trips_event_formats_and_confirmed_subscriptions() + { + var ad = new NodeAdvertisement( + "3.2.0", + "9f1c", + ["json"], + new Dictionary>(), + EventReads: ["json", "02"], + Subscriptions: new Dictionary { ["app:ps:orders"] = 1, ["a,b;c=d"] = 2 }); + + ad.Encode().Should().Contain(";er=json,02").And.Contain("sub="); + NodeAdvertisement.TryParse(ad.Encode(), out var parsed).Should().BeTrue(); + + parsed!.EventReads.Should().Equal("json", "02"); + parsed.Subscriptions.Should().BeEquivalentTo(new Dictionary { ["app:ps:orders"] = 1, ["a,b;c=d"] = 2 }); + } + + [Fact] + public void Ad_round_trips_the_pattern_subscriptions_and_omits_none() + { + var ad = new NodeAdvertisement("3.2.0", "i", ["json"], new Dictionary>(), PatternSubscriptions: 2); + + ad.Encode().Should().Contain(";psub=2"); + NodeAdvertisement.TryParse(ad.Encode(), out var parsed).Should().BeTrue(); + parsed!.PatternSubscriptions.Should().Be(2); + (ad with { PatternSubscriptions = 0 }).Encode().Should().NotContain("psub"); + NodeAdvertisement.TryParse("lib=3.2.0;inst=i;vr=json;psub=x", out var bad).Should().BeTrue(); + bad!.PatternSubscriptions.Should().BeNull("a count that does not parse is no claim"); + } + + [Fact] + public void An_ad_that_says_nothing_about_events_or_subscriptions_parses_as_none() + { + var ad = new NodeAdvertisement("3.2.0", "i", ["json"], new Dictionary>()); + + ad.Encode().Should().NotContain("er=").And.NotContain("sub="); + NodeAdvertisement.TryParse("lib=3.2.0;inst=i;vr=json", out var parsed).Should().BeTrue(); + parsed!.EventReads.Should().BeNull("a build that did not know events lists none, which is not the same as listing json"); + parsed.Subscriptions.Should().BeNull(); + } + + [Fact] + public void A_subscription_entry_that_does_not_parse_is_left_out_not_the_whole_ad() + { + NodeAdvertisement.TryParse("lib=3.2.0;inst=i;vr=json;sub=good:2,bad,worse:x,:3", out var parsed).Should().BeTrue(); + + parsed!.Subscriptions.Should().BeEquivalentTo(new Dictionary { ["good"] = 2 }); + } + [Fact] public void An_ad_that_is_not_leaving_does_not_say_so() { diff --git a/tests/UiPath.Caching.Tests/Negotiation/PubSubNegotiationServiceTests.cs b/tests/UiPath.Caching.Tests/Negotiation/PubSubNegotiationServiceTests.cs new file mode 100644 index 00000000..412845d7 --- /dev/null +++ b/tests/UiPath.Caching.Tests/Negotiation/PubSubNegotiationServiceTests.cs @@ -0,0 +1,504 @@ +using System.Net; +using Microsoft.Extensions.Logging; +using UiPath.Caching.Negotiation; +using UiPath.Caching.Tests.Fakes; + +namespace UiPath.Caching.Tests.Negotiation; + +public class PubSubNegotiationServiceTests +{ + private const string Channel = "app:ps:orders"; + + private readonly ManualTimeProvider _clock = new(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero)); + private readonly FakeNodeListServer _server = new(); + private readonly FormatNegotiationOptions _options = new() { UpgradeHold = TimeSpan.Zero }; + private readonly FakeSubscribers _subscribers; + private readonly FakePubSub _pubSub; + + public PubSubNegotiationServiceTests() + { + _subscribers = new FakeSubscribers(_server.Events); + _pubSub = new FakePubSub(_server.Events) { Channels = [Channel] }; + } + + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task Subscribe_then_advertise_order() + { + var sut = Service(out _); + + await sut.StartAsync(Ct); + _server.Hash["me"].Should().NotContain("sub=", "the subscription is not in place, so there is nothing to claim"); + _pubSub.Confirm(Channel); + await UntilAsync(() => _server.Hash["me"].Contains("sub=", StringComparison.Ordinal)); + + // The claim is written after the server confirmed the subscription, and at once, not at the next interval. + var subscribed = _server.Events.IndexOf("subscribed"); + subscribed.Should().BeGreaterThan(0); + _server.Events.Skip(subscribed + 1).Should().StartWith("write"); + NodeAdvertisement.TryParse(_server.Hash["me"].Split('|', 2)[1], out var ad).Should().BeTrue(); + ad!.Subscriptions.Should().BeEquivalentTo(new Dictionary { [Channel] = 1 }); + await sut.StopAsync(Ct); + } + + [Fact] + public async Task A_subscription_that_is_not_confirmed_is_not_claimed() + { + var sut = Service(out _); + + await sut.StartAsync(Ct); + await sut.AdvertiseAsync(Ct); + await sut.TickAsync(Ct); + + _server.Hash["me"].Should().NotContain("sub="); + await sut.StopAsync(Ct); + } + + [Fact] + public async Task A_subscriber_nobody_advertises_blocks_after_two_views_and_goes_when_it_does() + { + var sut = Service(out var negotiator); + _pubSub.Confirm(Channel); + _subscribers.Count(Channel, 1); + await sut.StartAsync(Ct); + await sut.AdvertiseAsync(Ct); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeTrue("this node is the only subscriber, and it says so"); + + _subscribers.Count(Channel, 2); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeTrue("one view is no proof"); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeFalse("a second subscriber that no advertisement claims"); + + _server.Put("urn:other", _server.NowMs + $"|lib=3.2;inst=o;vr=json,10;er=json,02;sub={Channel.Replace(":", "%3A", StringComparison.Ordinal)}:1"); + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeTrue("that node advertised its subscription"); + await sut.StopAsync(Ct); + } + + [Fact] + public async Task A_node_that_stops_keeps_the_fleet_on_the_format_whether_its_connection_closes_at_once_or_views_later() + { + _options.UpgradeHold = TimeSpan.FromHours(1); + var claim = Channel.Replace(":", "%3A", StringComparison.Ordinal); + var sut = Service(out var negotiator); + _pubSub.Confirm(Channel); + _subscribers.Count(Channel, 2); + _server.Put("urn:stopping", _server.NowMs + $"|lib=3.2;inst=s;vr=json;er=json,02;sub={claim}:1"); + + // Driven by hand: started, its timer would tick whenever the test moves the clock. + await sut.AdvertiseAsync(Ct); + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + _clock.Advance(TimeSpan.FromHours(1)); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeTrue(); + + // It writes left=1 and its connection stays open for three views, as a slow shutdown leaves it: the server still counts a subscriber no reader claims. + _server.Put("urn:stopping", _server.NowMs + $"|lib=3.2;inst=s;vr=json;er=json,02;sub={claim}:1;left=1"); + for (var view = 0; view < 3; view++) + { + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeTrue($"the leaving node may still hold its subscription, view {view + 1}"); + } + + // Then it closes, and the count falls to what the nodes that read claim. + _subscribers.Count(Channel, 1); + for (var view = 0; view < 3; view++) + { + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeTrue($"a leaving node's claim is optional, view {view + 1}"); + } + } + + [Fact] + public async Task A_departing_advertisement_lists_the_subscriptions_still_held() + { + var sut = Service(out _); + _pubSub.Confirm(Channel); + await sut.StartAsync(Ct); + + await sut.StopAsync(Ct); + + NodeAdvertisement.TryParse(_server.Hash["me"].Split('|', 2)[1], out var ad).Should().BeTrue(); + ad!.Left.Should().BeTrue(); + ad.Subscriptions.Should().BeEquivalentTo(new Dictionary { [Channel] = 1 }); + } + + [Fact] + public async Task A_subscriber_nobody_claims_after_a_node_left_is_still_a_surplus() + { + var sut = Service(out var negotiator); + _pubSub.Confirm(Channel); + _subscribers.Count(Channel, 3); + _server.Put("urn:gone", _server.NowMs + $"|lib=3.2;inst=g;vr=json;er=json,02;sub={Channel.Replace(":", "%3A", StringComparison.Ordinal)}:1;left=1"); + await sut.AdvertiseAsync(Ct); + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + + negotiator.Writes.Should().BeFalse("this node and the one that left account for two of the three"); + } + + [Fact] + public async Task No_advertisement_that_is_not_leaving_is_written_after_the_node_stopped() + { + var sut = Service(out _); + _pubSub.Confirm(Channel); + await sut.StartAsync(Ct); + await sut.StopAsync(Ct); + _server.Hash["me"].Should().Contain("left=1"); + + // The confirmation of a subscription, or a tick, that was already on its way. + await sut.AdvertiseAsync(Ct); + _pubSub.Confirm(Channel); + await Task.Delay(100, Ct); + + _server.Hash["me"].Should().Contain("left=1", "a node that said it was leaving must not look alive again"); + } + + [Fact] + public async Task Advertisements_are_written_one_at_a_time() + { + var sut = Service(out _); + _pubSub.Confirm(Channel); + _server.WriteDelay = () => Task.Delay(30); + + await Task.WhenAll(Enumerable.Range(0, 6).Select(_ => sut.AdvertiseAsync(Ct))); + + _server.MaxConcurrentWrites.Should().Be(1, "a write started earlier must not land over one started later"); + } + + [Fact] + public async Task A_stop_waits_for_an_advertisement_in_flight_so_it_cannot_land_after_the_departing_one() + { + var sut = Service(out _); + _pubSub.Confirm(Channel); + var release = new TaskCompletionSource(); + _server.WriteDelay = () => release.Task; + var inFlight = sut.AdvertiseAsync(Ct); + await Task.Delay(50, Ct); + _server.WriteDelay = null; + + var stopping = sut.StopAsync(Ct); + await Task.Delay(50, Ct); + stopping.IsCompleted.Should().BeFalse("it is behind the write in flight"); + release.SetResult(); + await Task.WhenAll(inFlight, stopping); + + _server.Hash["me"].Should().Contain("left=1"); + } + + [Fact] + public async Task A_failed_ad_read_breaks_the_run_of_views() + { + var sut = Service(out var negotiator); + _pubSub.Confirm(Channel); + _subscribers.Count(Channel, 1); + await sut.AdvertiseAsync(Ct); + await sut.RefreshAsync(Ct); + + _server.Failure = new TimeoutException(); + await sut.RefreshAsync(Ct); + _server.Failure = null; + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeFalse("the view before the failure does not count, so this is the first of two"); + await sut.RefreshAsync(Ct); + + negotiator.Writes.Should().BeTrue(); + } + + [Fact] + public async Task A_pattern_subscriber_nobody_advertises_blocks_and_one_that_is_advertised_does_not() + { + var logger = new RecordedLogger(); + var sut = Service(out var negotiator, logger); + _pubSub.Confirm(Channel); + _subscribers.Count(Channel, 1); + _subscribers.Patterns = 1; + await sut.AdvertiseAsync(Ct); + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeFalse("a pattern subscriber on the server, say another app's, that no node of this one claims"); + logger.Records.Should().ContainSingle(r => r.Level == LogLevel.Warning && r.Message.Contains("(patterns)", StringComparison.Ordinal)) + .Which.Message.Should().Contain("whole server").And.Contain("AssumeAllNodesCanRead"); + + _server.Put("urn:patterns", _server.NowMs + "|lib=3.2;inst=p;vr=json;er=json,02;psub=1"); + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + + negotiator.Writes.Should().BeTrue("a node that reads advertised the pattern it holds"); + } + + [Fact] + public async Task The_advertisement_lists_the_pattern_subscriptions_this_node_holds() + { + _pubSub.PatternCount = 2; + var sut = Service(out _); + + await sut.AdvertiseAsync(Ct); + + NodeAdvertisement.TryParse(_server.Hash["me"].Split('|', 2)[1], out var ad).Should().BeTrue(); + ad!.PatternSubscriptions.Should().Be(2); + } + + [Fact] + public async Task A_view_with_no_channel_to_count_breaks_the_run_of_views() + { + var sut = Service(out var negotiator); + _pubSub.Confirm(Channel); + _subscribers.Count(Channel, 1); + await sut.AdvertiseAsync(Ct); + await sut.RefreshAsync(Ct); + + _pubSub.Channels = []; + await sut.RefreshAsync(Ct); + _pubSub.Channels = [Channel]; + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeFalse("the view before the gap does not count"); + await sut.RefreshAsync(Ct); + + negotiator.Writes.Should().BeTrue(); + } + + [Fact] + public async Task One_numsub_command_per_server_per_view_whatever_the_channels() + { + _pubSub.Channels = ["one", "two", "three"]; + var sut = Service(out _); + await sut.AdvertiseAsync(Ct); + + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + + _server.Events.Count(e => e == "numsub").Should().Be(2, "one server, two views"); + } + + [Fact] + public async Task A_refused_numsub_is_logged_once_with_the_fallback_and_again_when_it_is_counted_again() + { + var logger = new RecordedLogger(); + var sut = Service(out var negotiator, logger); + _pubSub.Confirm(Channel); + _subscribers.Denied = true; + await sut.AdvertiseAsync(Ct); + + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + + negotiator.Writes.Should().BeFalse(); + logger.Records.Where(r => r.Level == LogLevel.Warning && r.Message.Contains("cannot be counted", StringComparison.Ordinal)).Should().ContainSingle() + .Which.Message.Should().Contain("refused PUBSUB NUMSUB").And.Contain("binary formats stay JSON").And.Contain("AssumeAllNodesCanRead"); + + _subscribers.Denied = false; + _subscribers.Count(Channel, 1); + await sut.RefreshAsync(Ct); + logger.Records.Should().Contain(r => r.Level == LogLevel.Information && r.Message.Contains("counted again", StringComparison.Ordinal)); + } + + [Fact] + public async Task A_pattern_or_sharded_channel_cannot_be_counted_so_the_node_stays_on_json_unless_attested() + { + var logger = new RecordedLogger(); + _pubSub.Uncountable = ["app:ps:orders:*"]; + _subscribers.Count(Channel, 1); + var sut = Service(out var negotiator, logger); + _pubSub.Confirm(Channel); + await sut.AdvertiseAsync(Ct); + + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + + negotiator.Writes.Should().BeFalse(); + logger.Records.Should().ContainSingle(r => r.Level == LogLevel.Warning && r.Message.Contains("app:ps:orders:*", StringComparison.Ordinal) && r.Message.Contains("patterns or sharded", StringComparison.Ordinal)); + _server.Events.Should().NotContain("numsub", "it does not ask for a count it cannot read"); + + _options.AssumeAllNodesCanRead = true; + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeTrue("the operator's word, with a hold of zero here"); + } + + [Fact] + public async Task A_node_that_crashed_leaves_a_deficit_until_its_advertisement_expires_and_then_the_decision_resumes_without_a_new_hold() + { + _options.UpgradeHold = TimeSpan.FromHours(1); + var claim = $"|lib=3.2;inst=c;vr=json;er=json,02;sub={Channel.Replace(":", "%3A", StringComparison.Ordinal)}:1"; + var sut = Service(out var negotiator); + _pubSub.Confirm(Channel); + _subscribers.Count(Channel, 2); + _server.Put("urn:crashing", _server.NowMs + claim); + + // Driven by hand: started, its timer would tick whenever the test moves the clock. + await sut.AdvertiseAsync(Ct); + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + _clock.Advance(TimeSpan.FromHours(1)); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeTrue(); + + // It crashes: its connection is gone, its advertisement is not. + _subscribers.Count(Channel, 1); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeTrue("one view is absorbed"); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeFalse("a claim with no subscriber behind it hides a subscriber that says nothing"); + + // The advertisement is older than it lives; this node's own is fresh. + _server.NowMs += 151_000; + await sut.AdvertiseAsync(Ct); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeFalse("one clean view is not enough"); + await sut.RefreshAsync(Ct); + + negotiator.Writes.Should().BeTrue("it was a departure, not a join, so the hour already waited stands"); + } + + [Fact] + public async Task An_app_with_only_a_pub_sub_transport_reads_the_node_list_and_is_not_blind() + { + var sut = Service(out var negotiator); + _pubSub.Confirm(Channel); + _subscribers.Count(Channel, 1); + + await sut.StartAsync(Ct); + await sut.RefreshAsync(Ct); + + _server.Events.Should().Contain("read").And.Contain("numsub"); + negotiator.Writes.Should().BeTrue(); + await sut.StopAsync(Ct); + } + + [Fact] + public async Task A_node_with_no_channel_yet_has_no_transport_to_read_and_writes_json() + { + _pubSub.Channels = []; + var sut = Service(out var negotiator); + + await sut.StartAsync(Ct); + await sut.RefreshAsync(Ct); + + negotiator.Writes.Should().BeFalse("the node list cannot see a transport yet"); + _server.Events.Should().NotContain("numsub"); + await sut.StopAsync(Ct); + } + + [Fact] + public async Task A_subscriber_count_that_cannot_be_read_keeps_the_decision_until_it_is_too_old() + { + var sut = Service(out var negotiator); + _pubSub.Confirm(Channel); + _subscribers.Count(Channel, 1); + await sut.StartAsync(Ct); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeTrue(); + + _subscribers.Failure = new TimeoutException(); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeTrue("a failed read is not an answer"); + _clock.Advance(_options.MaxViewAge + TimeSpan.FromSeconds(1)); + + negotiator.Writes.Should().BeFalse(); + await sut.StopAsync(Ct); + } + + private static async Task UntilAsync(Func condition) + { + var until = DateTime.UtcNow.AddSeconds(10); + while (!condition()) + { + Assert.True(DateTime.UtcNow < until, "The condition did not hold in time."); + await Task.Delay(10, Ct); + } + } + + private FormatNegotiationService Service(out FormatNegotiator negotiator, ILogger? logger = null) + { + negotiator = new FormatNegotiator([], _options, _clock, logger: logger, events: new DesiredEvents(["02"])); + return new FormatNegotiationService(new FormatNegotiation + { + Negotiator = negotiator, + Advertisement = new("3.2.0", "me", ["json"], new Dictionary>(), EventReads: ["json", "02"]), + NodeId = "me", + Store = new NodeListStore(_server.Connector, "ads"), + NodeList = new StreamNodeList(new NoGroups(), _options), + PubSub = _pubSub, + Subscribers = new PubSubNodeList(_subscribers), + Options = _options, + Clock = _clock, + }); + } + + private sealed class NoGroups : IStreamGroupSource + { + public Task> GroupsAsync(StackExchange.Redis.RedisKey stream, CancellationToken cancellationToken) => Task.FromResult>([]); + + public Task IdleAsync(StackExchange.Redis.RedisKey stream, string group, CancellationToken cancellationToken) => Task.FromResult(null); + } + + private sealed class FakePubSub(List events) : IPubSubSubscriptions + { + private readonly Dictionary _confirmed = []; + private Action? _handler; + + public IReadOnlyList Channels { get; set; } = []; + + public IReadOnlyList Uncountable { get; set; } = []; + + public int PatternCount { get; set; } + + public void Confirm(string channel) + { + lock (_confirmed) + { + _confirmed[channel] = 1; + } + + events.Add("subscribed"); + _handler?.Invoke(); + } + + IReadOnlyList IPubSubSubscriptions.Channels() => Channels; + + IReadOnlyList IPubSubSubscriptions.Uncountable() => Uncountable; + + int IPubSubSubscriptions.Patterns() => PatternCount; + + IReadOnlyDictionary IPubSubSubscriptions.Confirmed() + { + lock (_confirmed) + { + return new Dictionary(_confirmed); + } + } + + void IPubSubSubscriptions.WhenConfirmed(Action handler) => _handler = handler; + } + + private sealed class FakeSubscribers(List events) : ISubscriberSource + { + private readonly Dictionary _counts = []; + + public Exception? Failure { get; set; } + + public bool Denied { get; set; } + + public long Patterns { get; set; } + + public void Count(string channel, long count) => _counts[channel] = count; + + public IReadOnlyList Endpoints() => [new DnsEndPoint("a", 6379)]; + + public Task CountAsync(EndPoint endpoint, IReadOnlyList channels, CancellationToken cancellationToken) + { + events.Add("numsub"); + return Failure is { } failure + ? Task.FromException(failure) + : Task.FromResult(Denied ? null : new([.. channels.Select(channel => _counts.GetValueOrDefault(channel))], Patterns)); + } + } +} diff --git a/tests/UiPath.Caching.Tests/Negotiation/PubSubNodeListTests.cs b/tests/UiPath.Caching.Tests/Negotiation/PubSubNodeListTests.cs new file mode 100644 index 00000000..25562ba9 --- /dev/null +++ b/tests/UiPath.Caching.Tests/Negotiation/PubSubNodeListTests.cs @@ -0,0 +1,698 @@ +using System.Net; +using UiPath.Caching.Negotiation; +using UiPath.Caching.Tests.Fakes; +using UiPath.Caching.Tests.Telemetry; + +namespace UiPath.Caching.Tests.Negotiation; + +public class PubSubNodeListTests +{ + private const string Channel = "app:ps:orders"; + + private static readonly string[] Both = ["json", "02"]; + private static readonly FakeValueSerializer Writer = new(0x10); + + private readonly FakeSubscribers _source = new(); + private readonly ManualTimeProvider _clock = new(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero)); + private readonly RecordingTelemetryProvider _telemetry = new(); + private readonly FormatNegotiationOptions _options = new() { UpgradeHold = TimeSpan.Zero }; + + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task Sum_over_all_endpoints() + { + _source.Servers("a", "b", "c"); + _source.Count("a", Channel, 1); + _source.Count("b", Channel, 1); + _source.Count("c", Channel, 0); + var nodeList = new PubSubNodeList(_source); + + // Two subscribers, each on a different server, and one advertised: one is surplus. Asking one server would see one and miss it. + await nodeList.ReadAsync([Channel], Accounted(1), Ct); + var reading = await nodeList.ReadAsync([Channel], Accounted(1), Ct); + + reading.Known.Should().BeTrue(); + reading.Surplus.Should().Be(1); + reading.Channels.Should().Equal(Channel); + reading.Unaccounted.Should().BeTrue(); + _source.Asked.Should().BeEquivalentTo(["a", "b", "c", "a", "b", "c"], "every server, every view"); + } + + [Fact] + public async Task A_subscriber_that_is_accounted_for_is_no_surplus_and_no_deficit() + { + _source.Servers("a", "b"); + _source.Count("a", Channel, 1); + _source.Count("b", Channel, 1); + var nodeList = new PubSubNodeList(_source); + + var reading = await nodeList.ReadAsync([Channel], Accounted(2), Ct); + + reading.Surplus.Should().Be(0); + reading.Short.Should().BeFalse(); + reading.ShortChannels.Should().BeEmpty(); + } + + [Fact] + public async Task Fewer_subscribers_than_are_claimed_is_a_deficit_only_in_two_views_running() + { + _source.Servers("a", "b"); + _source.Count("a", Channel, 1); + _source.Count("b", Channel, 0); + var nodeList = new PubSubNodeList(_source); + + var first = await nodeList.ReadAsync([Channel], Accounted(2), Ct); + var second = await nodeList.ReadAsync([Channel], Accounted(2), Ct); + + first.Surplus.Should().Be(0, "a deficit is no surplus"); + first.Short.Should().BeFalse("one view is no proof: a node resubscribing is briefly missing"); + first.Settled.Should().BeFalse(); + second.Short.Should().BeTrue(); + second.Settled.Should().BeFalse(); + second.ShortChannels.Should().Equal(Channel); + second.Unaccounted.Should().BeFalse(); + } + + [Fact] + public async Task A_deficit_that_is_gone_for_one_view_has_not_cleared() + { + _source.Servers("a"); + _source.Count("a", Channel, 1); + var nodeList = new PubSubNodeList(_source); + await nodeList.ReadAsync([Channel], Accounted(2), Ct); + await nodeList.ReadAsync([Channel], Accounted(2), Ct); + + var firstClean = await nodeList.ReadAsync([Channel], Accounted(1), Ct); + var secondClean = await nodeList.ReadAsync([Channel], Accounted(1), Ct); + + firstClean.Short.Should().BeFalse(); + firstClean.Settled.Should().BeFalse("the view before it was short"); + secondClean.Settled.Should().BeTrue(); + } + + [Fact] + public async Task Deficit_blocks_and_recovery_does_not_restart_the_hold() + { + _options.UpgradeHold = TimeSpan.FromHours(1); + _source.Servers("a"); + _source.Count("a", Channel, 2); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + _clock.Advance(TimeSpan.FromHours(1)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + sut.Writes.Should().BeTrue(); + sut.Allows(Writer).Should().BeTrue(); + + // A node claimed a subscription and is gone: its claim outlives it. + _source.Count("a", Channel, 1); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + sut.Writes.Should().BeTrue("one view is absorbed"); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + sut.Writes.Should().BeFalse("the count does not add up, so it is unknown"); + sut.Allows(Writer).Should().BeFalse("values as well"); + _telemetry.Metrics.Should().Contain(m => m.Name == Metrics.FormatFallback && m.Properties!["reason"] == FormatNegotiator.ReasonSubscriberDeficit && m.Properties["kind"] == "event"); + + // Its claim expires, and the counts match again, for two views: what was written before resumes with no new hold. + _clock.Advance(TimeSpan.FromMinutes(3)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + sut.Writes.Should().BeFalse("the first clean view is not enough"); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + + sut.Writes.Should().BeTrue("a departure is not a join, so the hour it had already waited stands"); + sut.Allows(Writer).Should().BeTrue(); + } + + [Fact] + public async Task A_node_that_joins_while_the_count_is_short_restarts_the_hold() + { + _options.UpgradeHold = TimeSpan.FromHours(1); + _source.Servers("a"); + _source.Count("a", Channel, 2); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + _clock.Advance(TimeSpan.FromHours(1)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + _source.Count("a", Channel, 1); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + sut.Writes.Should().BeFalse(); + + // The count adds up again, with a node that was not there before. + sut.Observe(new NodeListView([Stored("a", Both), Stored("b", Both)], [], Complete: true, await nodeList.ReadAsync([Channel], Accounted(1), Ct))); + sut.Observe(new NodeListView([Stored("a", Both), Stored("b", Both)], [], Complete: true, await nodeList.ReadAsync([Channel], Accounted(1), Ct))); + sut.Writes.Should().BeFalse("a join waits for the hold"); + + _clock.Advance(TimeSpan.FromHours(1)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + sut.Writes.Should().BeTrue(); + } + + [Fact] + public async Task One_command_per_server_carries_every_channel() + { + _source.Servers("a", "b", "c"); + var nodeList = new PubSubNodeList(_source); + + await nodeList.ReadAsync(["one", "two", "three"], new Dictionary(), Ct); + await nodeList.ReadAsync(["one", "two", "three"], new Dictionary(), Ct); + + _source.Commands.Should().HaveCount(6, "three servers, two views, one command each"); + _source.Commands.Should().OnlyContain(command => command.SequenceEqual(new[] { "one", "two", "three" })); + } + + [Fact] + public async Task Surplus_on_one_channel_and_then_on_another_is_not_confirmed() + { + _source.Servers("a"); + var nodeList = new PubSubNodeList(_source); + _source.Count("a", "one", 1); + await nodeList.ReadAsync(["one", "two"], new Dictionary(), Ct); + _source.Count("a", "one", 0); + _source.Count("a", "two", 1); + + var moved = await nodeList.ReadAsync(["one", "two"], new Dictionary(), Ct); + var stayed = await nodeList.ReadAsync(["one", "two"], new Dictionary(), Ct); + + moved.Unaccounted.Should().BeFalse("the surplus is on another channel than the one before it"); + stayed.Unaccounted.Should().BeTrue("the same channel in two views running"); + stayed.Channels.Should().Equal("two"); + } + + [Fact] + public async Task A_deficit_on_one_channel_and_then_on_another_is_not_confirmed() + { + _source.Servers("a"); + var nodeList = new PubSubNodeList(_source); + var claims = new Dictionary { ["one"] = 1, ["two"] = 1 }; + _source.Count("a", "one", 0); + _source.Count("a", "two", 1); + await nodeList.ReadAsync(["one", "two"], claims, Ct); + _source.Count("a", "one", 1); + _source.Count("a", "two", 0); + + (await nodeList.ReadAsync(["one", "two"], claims, Ct)).Short.Should().BeFalse(); + (await nodeList.ReadAsync(["one", "two"], claims, Ct)).Short.Should().BeTrue(); + } + + [Fact] + public async Task A_leaving_node_may_or_may_not_still_hold_its_subscription() + { + _source.Servers("a"); + var nodeList = new PubSubNodeList(_source); + var claims = new Dictionary { [Channel] = 1 }; + var leaving = new Dictionary { [Channel] = 1 }; + + // Held, and closed: both are what a stop looks like, over the two or three views its connection takes to close. + _source.Count("a", Channel, 2); + await nodeList.ReadAsync([Channel], claims, leaving, Ct); + var held = await nodeList.ReadAsync([Channel], claims, leaving, Ct); + _source.Count("a", Channel, 1); + await nodeList.ReadAsync([Channel], claims, leaving, Ct); + var closed = await nodeList.ReadAsync([Channel], claims, leaving, Ct); + _source.Count("a", Channel, 3); + await nodeList.ReadAsync([Channel], claims, leaving, Ct); + var surplus = await nodeList.ReadAsync([Channel], claims, leaving, Ct); + + held.Surplus.Should().Be(0); + held.Unaccounted.Should().BeFalse(); + closed.Short.Should().BeFalse("a leaving node's claim is not one that has to be held"); + surplus.Surplus.Should().Be(1, "one more than the nodes that read and the one that is leaving can hold"); + surplus.Unaccounted.Should().BeTrue(); + } + + [Fact] + public async Task A_one_view_blip_late_in_the_hold_does_not_restart_it() + { + _options.UpgradeHold = TimeSpan.FromHours(1); + _source.Servers("a"); + _source.Count("a", Channel, 1); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + _clock.Advance(TimeSpan.FromMinutes(55)); + + // A replica reconnects for one view: a surplus that is gone by the next. + _source.Count("a", Channel, 2); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + _source.Count("a", Channel, 1); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + sut.Writes.Should().BeFalse("five minutes of the hour are left"); + _clock.Advance(TimeSpan.FromMinutes(5)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + + sut.Writes.Should().BeTrue("a blip pauses the hold, it does not start it again"); + } + + [Fact] + public async Task A_confirmed_surplus_does_restart_the_hold() + { + _options.UpgradeHold = TimeSpan.FromHours(1); + _source.Servers("a"); + _source.Count("a", Channel, 1); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + _clock.Advance(TimeSpan.FromMinutes(55)); + _source.Count("a", Channel, 2); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + _source.Count("a", Channel, 1); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + + _clock.Advance(TimeSpan.FromMinutes(5)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + + sut.Writes.Should().BeFalse("a subscriber nobody advertised was there for two views"); + } + + [Fact] + public async Task A_deficit_that_clears_while_the_claim_is_still_fresh_is_a_join_and_restarts_the_hold() + { + _options.UpgradeHold = TimeSpan.FromHours(1); + _source.Servers("a"); + _source.Count("a", Channel, 2); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + _clock.Advance(TimeSpan.FromHours(1)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + _source.Count("a", Channel, 1); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + sut.Writes.Should().BeFalse(); + + // Two claims, two subscribers again, and the claim that was behind the deficit has not expired: something subscribed. + _source.Count("a", Channel, 2); + var replaced = await nodeList.ReadAsync([Channel], Accounted(2), Ct); + Observe(sut, replaced); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + replaced.Replaced.Should().BeTrue(); + sut.Writes.Should().BeFalse("it waits for the hold like a join"); + + _clock.Advance(TimeSpan.FromHours(1)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + sut.Writes.Should().BeTrue(); + } + + [Fact] + public async Task A_deficit_that_clears_because_the_claim_expired_is_not_a_join() + { + _source.Servers("a"); + _source.Count("a", Channel, 1); + var nodeList = new PubSubNodeList(_source); + await nodeList.ReadAsync([Channel], Accounted(2), Ct); + await nodeList.ReadAsync([Channel], Accounted(2), Ct); + + var expired = await nodeList.ReadAsync([Channel], Accounted(1), Ct); + + expired.Replaced.Should().BeFalse("the claim went, and the subscriber count did not rise"); + } + + [Fact] + public async Task A_confirmed_deficit_survives_a_failed_view_and_the_replacement_still_waits_for_the_hold() + { + _options.UpgradeHold = TimeSpan.FromHours(1); + _source.Servers("a"); + _source.Count("a", Channel, 2); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + _clock.Advance(TimeSpan.FromHours(1)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + sut.Writes.Should().BeTrue(); + _source.Count("a", Channel, 1); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + sut.Writes.Should().BeFalse(); + + _source.Failure = new TimeoutException(); + await FluentActions.Awaiting(() => nodeList.ReadAsync([Channel], Accounted(2), Ct)).Should().ThrowAsync(); + _source.Failure = null; + _source.Count("a", Channel, 2); + var replaced = await nodeList.ReadAsync([Channel], Accounted(2), Ct); + Observe(sut, replaced); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + + replaced.Replaced.Should().BeTrue(); + sut.Writes.Should().BeFalse("a view that failed in between does not make the replacement a node that was always there"); + } + + [Fact] + public async Task An_unknown_node_that_takes_a_departing_nodes_slot_stays_hidden_until_the_ad_expires() + { + _source.Servers("a"); + _source.Count("a", Channel, 2); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + var claims = Accounted(2); + var none = new Dictionary(); + for (var i = 0; i < 3; i++) + { + Observe(sut, await nodeList.ReadAsync([Channel], claims, none, Ct)); + } + + sut.Writes.Should().BeTrue(); + + // The departing ad lists its subscription as optional, and a node that advertises nothing takes the slot: the count never moves. + var leaving = Accounted(1); + for (var i = 0; i < 4; i++) + { + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), leaving, Ct)); + sut.Writes.Should().BeTrue("the hidden node is covered by the departing ad's optional claim"); + } + + for (var i = 0; i < 2; i++) + { + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), none, Ct)); + } + + sut.Writes.Should().BeFalse("the ad expired, and the node behind the slot is unaccounted"); + } + + [Fact] + public async Task A_deficit_of_one_view_that_clears_is_not_a_join() + { + _source.Servers("a"); + _source.Count("a", Channel, 1); + var nodeList = new PubSubNodeList(_source); + await nodeList.ReadAsync([Channel], Accounted(2), Ct); + _source.Count("a", Channel, 2); + + (await nodeList.ReadAsync([Channel], Accounted(2), Ct)).Replaced.Should().BeFalse("a node that resubscribed was never confirmed missing"); + } + + [Fact] + public async Task An_unadvertised_pattern_subscriber_blocks() + { + _source.Servers("a", "b"); + _source.Count("a", Channel, 1); + _source.Patterns("b", 1); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + var reading = await nodeList.ReadAsync([Channel], Accounted(1), Ct); + Observe(sut, reading); + + reading.Unaccounted.Should().BeTrue("NUMSUB does not count a pattern subscriber, NUMPAT does, and nobody advertised it"); + reading.Channels.Should().Equal(PubSubNodeList.PatternKey); + sut.Writes.Should().BeFalse(); + _source.Commands.Should().HaveCount(6, "the pattern count comes with the channel counts, one round trip per server"); + } + + [Fact] + public async Task An_advertised_pattern_subscriber_does_not() + { + _source.Servers("a", "b"); + _source.Count("a", Channel, 1); + _source.Patterns("a", 1); + _source.Patterns("b", 1); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + var accounted = new Dictionary { [Channel] = 1, [PubSubNodeList.PatternKey] = 2 }; + + Observe(sut, await nodeList.ReadAsync([Channel], accounted, Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], accounted, Ct)); + + sut.Writes.Should().BeTrue("both pattern subscriptions are claimed"); + } + + [Fact] + public async Task Fewer_patterns_than_claimed_is_no_deficit_since_the_server_counts_unique_patterns() + { + _source.Servers("a"); + _source.Count("a", Channel, 1); + var nodeList = new PubSubNodeList(_source); + var accounted = new Dictionary { [Channel] = 1, [PubSubNodeList.PatternKey] = 3 }; + + await nodeList.ReadAsync([Channel], accounted, Ct); + var reading = await nodeList.ReadAsync([Channel], accounted, Ct); + + reading.Short.Should().BeFalse(); + reading.Settled.Should().BeTrue(); + } + + [Fact] + public async Task A_deficit_does_not_count_while_the_operator_attests() + { + _source.Servers("a"); + _source.Count("a", Channel, 0); + _options.AssumeAllNodesCanRead = true; + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + + sut.Writes.Should().BeTrue(); + } + + [Fact] + public async Task Each_channel_is_counted_on_its_own() + { + _source.Servers("a"); + _source.Count("a", "one", 1); + _source.Count("a", "two", 2); + var nodeList = new PubSubNodeList(_source); + + var reading = await nodeList.ReadAsync(["one", "two"], new Dictionary { ["one"] = 1, ["two"] = 1, ["elsewhere"] = 5 }, Ct); + + reading.Surplus.Should().Be(1); + reading.Channels.Should().Equal("two"); + } + + [Fact] + public async Task Numsub_denied_means_json() + { + _source.Servers("a", "b"); + _source.Count("a", Channel, 1); + _source.Count("b", Channel, null); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + + var reading = await nodeList.ReadAsync([Channel], Accounted(1), Ct); + reading.Known.Should().BeFalse("one server that refuses makes the sum a guess"); + Observe(sut, reading); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + + sut.Writes.Should().BeFalse(); + sut.Allows(Writer).Should().BeFalse(); + + // The operator says the nodes it cannot count all read it. + _options.AssumeAllNodesCanRead = true; + var attested = Negotiator(); + Observe(attested, reading); + attested.Writes.Should().BeTrue(); + } + + [Fact] + public async Task Numsub_denied_is_reported_as_transport_unknown() + { + _source.Servers("a"); + _source.Count("a", Channel, null); + _options.UpgradeHold = TimeSpan.Zero; + var sut = Negotiator(); + Observe(sut, new SubscriberReading(true, false, true, 0, [])); + sut.Writes.Should().BeTrue(); + + Observe(sut, await new PubSubNodeList(_source).ReadAsync([Channel], Accounted(1), Ct)); + + sut.Writes.Should().BeFalse(); + _telemetry.Metrics.Should().Contain(m => m.Name == Metrics.FormatFallback && m.Properties!["reason"] == FormatNegotiator.ReasonTransportUnknown && m.Properties["kind"] == "event"); + } + + [Fact] + public async Task Surplus_subscriber_blocks() + { + _source.Servers("a"); + _source.Count("a", Channel, 2); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + sut.Writes.Should().BeTrue("both subscribers are accounted for"); + _source.Count("a", Channel, 3); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + + sut.Writes.Should().BeFalse("a third subscriber that no advertisement claims"); + sut.Allows(Writer).Should().BeFalse(); + _telemetry.Metrics.Should().Contain(m => m.Name == Metrics.FormatFallback && m.Properties!["reason"] == FormatNegotiator.ReasonUnaccountedSubscriber); + _telemetry.Metrics.Should().Contain(m => m.Name == Metrics.FormatBlockers && m.Properties!["channels"] == Channel && m.Value == 1); + } + + [Fact] + public async Task Debounce_two_views_for_surplus() + { + _source.Servers("a"); + _source.Count("a", Channel, 1); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + sut.Writes.Should().BeTrue(); + + // A node that has subscribed and not yet advertised is surplus for one view only. + _source.Count("a", Channel, 2); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + sut.Writes.Should().BeTrue("one view is not enough to take the format away"); + _source.Count("a", Channel, 2); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(2), Ct)); + sut.Writes.Should().BeTrue("it advertised, and the surplus is gone"); + + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + sut.Writes.Should().BeFalse("two views in a row"); + } + + [Fact] + public async Task An_upgrade_needs_two_clean_views_over_the_same_servers() + { + _source.Servers("a", "b"); + _source.Count("a", Channel, 1); + _source.Count("b", Channel, 0); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + sut.Writes.Should().BeFalse("the hold is zero, but one view is a guess"); + _source.Servers("a", "b", "c"); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + sut.Writes.Should().BeFalse("the servers changed between the two views"); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + sut.Writes.Should().BeTrue(); + } + + [Fact] + public async Task A_surplus_between_two_clean_views_starts_the_upgrade_again() + { + _source.Servers("a"); + _source.Count("a", Channel, 1); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + _source.Count("a", Channel, 2); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + _source.Count("a", Channel, 1); + + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + sut.Writes.Should().BeFalse("the clean view after a surplus is the first of two"); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + + sut.Writes.Should().BeTrue(); + } + + [Fact] + public async Task An_ad_that_lacks_the_format_still_stops_the_writes_at_once() + { + _source.Servers("a"); + _source.Count("a", Channel, 1); + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + sut.Writes.Should().BeTrue(); + + sut.Observe(new NodeListView([Stored("a", Both), Stored("old", ["json"])], [], Complete: true, await nodeList.ReadAsync([Channel], Accounted(1), Ct))); + + sut.Writes.Should().BeFalse("no debounce for a node that says what it cannot read"); + } + + [Fact] + public async Task A_failed_read_is_not_a_view_and_starts_the_debounce_again() + { + _source.Servers("a"); + _source.Count("a", Channel, 1); + var nodeList = new PubSubNodeList(_source); + await nodeList.ReadAsync([Channel], Accounted(1), Ct); + _source.Failure = new TimeoutException(); + + await FluentActions.Awaiting(() => nodeList.ReadAsync([Channel], Accounted(1), Ct)).Should().ThrowAsync(); + _source.Failure = null; + + (await nodeList.ReadAsync([Channel], Accounted(1), Ct)).Settled.Should().BeFalse("the view before the failure does not count"); + (await nodeList.ReadAsync([Channel], Accounted(1), Ct)).Settled.Should().BeTrue(); + } + + [Fact] + public async Task No_connected_server_is_unknown_and_so_is_no_answer_from_one_that_cannot_count() + { + var nodeList = new PubSubNodeList(_source); + + (await nodeList.ReadAsync([Channel], Accounted(0), Ct)).Known.Should().BeFalse("nothing to ask"); + } + + [Fact] + public async Task A_surplus_does_not_count_while_the_operator_attests() + { + _source.Servers("a"); + _source.Count("a", Channel, 5); + _options.AssumeAllNodesCanRead = true; + var nodeList = new PubSubNodeList(_source); + var sut = Negotiator(); + + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + Observe(sut, await nodeList.ReadAsync([Channel], Accounted(1), Ct)); + + sut.Writes.Should().BeTrue(); + } + + private static StoredAdvertisement Stored(string id, string[] events, string[]? values = null) => + new(id, new("3.2.0", id, values ?? ["json", "10"], new Dictionary>(), EventReads: events), TimeSpan.Zero); + + private static Dictionary Accounted(long count) => new() { [Channel] = count }; + + private FormatNegotiator Negotiator() => + new([new DesiredWrite(Writer, ["10"], ["default"], [])], _options, _clock, _telemetry, events: new DesiredEvents(["02"])); + + private void Observe(FormatNegotiator sut, SubscriberReading reading) => + sut.Observe(new NodeListView([Stored("a", Both)], [], Complete: true, reading)); + + private sealed class FakeSubscribers : ISubscriberSource + { + private readonly Dictionary<(string Server, string Channel), long?> _counts = []; + private readonly Dictionary _patterns = []; + private List _endpoints = []; + + public Exception? Failure { get; set; } + + public List Asked { get; } = []; + + public List Commands { get; } = []; + + public void Servers(params string[] names) => _endpoints = [.. names.Select(n => (EndPoint)new DnsEndPoint(n, 6379))]; + + public void Count(string server, string channel, long? count) => _counts[(server, channel)] = count; + + public void Patterns(string server, long count) => _patterns[server] = count; + + public IReadOnlyList Endpoints() => _endpoints; + + public Task CountAsync(EndPoint endpoint, IReadOnlyList channels, CancellationToken cancellationToken) + { + var server = ((DnsEndPoint)endpoint).Host; + Asked.Add(server); + Commands.Add([.. channels]); + if (Failure is { } failure) + { + return Task.FromException(failure); + } + + var counts = channels.Select(channel => _counts.GetValueOrDefault((server, channel), 0L)).ToList(); + return Task.FromResult(counts.Any(count => count is null) ? null : new([.. counts.Select(count => count!.Value)], _patterns.GetValueOrDefault(server))); + } + } +} diff --git a/tests/UiPath.Caching.Tests/Negotiation/RedisSubscriberSourceTests.cs b/tests/UiPath.Caching.Tests/Negotiation/RedisSubscriberSourceTests.cs new file mode 100644 index 00000000..4380a624 --- /dev/null +++ b/tests/UiPath.Caching.Tests/Negotiation/RedisSubscriberSourceTests.cs @@ -0,0 +1,152 @@ +using System.Net; +using StackExchange.Redis; +using UiPath.Caching.Negotiation; +using UiPath.Caching.Tests.Fakes; + +namespace UiPath.Caching.Tests.Negotiation; + +public class RedisSubscriberSourceTests +{ + private const string Channel = "app:ps:orders"; + + private readonly IConnectionMultiplexer _multiplexer = Substitute.For(); + private readonly IRedisConnector _connector = Substitute.For(); + private readonly FakeNodeListServer _nodeList = new(); + private readonly List _numsub = []; + + public RedisSubscriberSourceTests() + { + _nodeList.Database.Multiplexer.Returns(_multiplexer); + _connector.Database.Returns(_nodeList.Database); + } + + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public void A_server_that_is_not_connected_makes_the_whole_list_fail() + { + Wire(Server("a", connected: true), Server("b", connected: false)); + + var endpoints = () => new RedisSubscriberSource(_connector).Endpoints(); + + endpoints.Should().Throw().WithMessage("*b*not connected*"); + } + + [Fact] + public void Every_connected_server_is_listed_and_a_sentinel_is_not_one() + { + Wire(Server("a", true), Server("b", true), Server("s", true, ServerType.Sentinel)); + + var endpoints = new RedisSubscriberSource(_connector).Endpoints(); + + endpoints.Select(e => ((DnsEndPoint)e).Host).Should().Equal("a", "b"); + } + + [Fact] + public async Task With_one_server_down_the_view_fails_and_binary_events_are_not_enabled() + { + Wire(Server("a", true, count: 1), Server("b", connected: false)); + var negotiator = new FormatNegotiator([], new FormatNegotiationOptions { UpgradeHold = TimeSpan.Zero }, TimeProvider.System, events: new DesiredEvents(["02"])); + var sut = new FormatNegotiationService(new FormatNegotiation + { + Negotiator = negotiator, + Advertisement = new("3.2.0", "me", ["json"], new Dictionary>(), EventReads: ["json", "02"]), + NodeId = "me", + Store = new NodeListStore(_nodeList.Connector, "ads"), + NodeList = new StreamNodeList(new NoGroups(), new FormatNegotiationOptions()), + PubSub = new Subscriptions(), + Subscribers = new PubSubNodeList(new RedisSubscriberSource(_connector)), + Options = new FormatNegotiationOptions { UpgradeHold = TimeSpan.Zero }, + }); + await sut.AdvertiseAsync(Ct); + + for (var view = 0; view < 4; view++) + { + await sut.RefreshAsync(Ct); + } + + negotiator.Writes.Should().BeFalse("the sum over the servers it can reach hides the subscribers of the one it cannot"); + + // The same servers, all connected: the count is whole, and the node that holds the only subscription may write. + Wire(Server("a", true, count: 1), Server("b", true)); + await sut.RefreshAsync(Ct); + await sut.RefreshAsync(Ct); + negotiator.Writes.Should().BeTrue(); + } + + [Fact] + public async Task The_channel_names_it_asks_for_carry_the_connections_channel_prefix() + { + var server = Server("a", true, count: 1); + Wire(server); + _multiplexer.Configuration.Returns("a:6379,channelPrefix=pfx:"); + var source = new RedisSubscriberSource(_connector); + + await source.CountAsync(source.Endpoints()[0], [Channel, "other"], Ct); + + _numsub.Should().ContainSingle().Which.Should().Equal($"pfx:{Channel}", "pfx:other"); + } + + [Fact] + public async Task Without_a_prefix_the_names_are_asked_as_they_are() + { + var server = Server("a", true, count: 1); + Wire(server); + _multiplexer.Configuration.Returns("a:6379"); + var source = new RedisSubscriberSource(_connector); + + await source.CountAsync(source.Endpoints()[0], [Channel], Ct); + + _numsub.Should().ContainSingle().Which.Should().Equal(Channel); + } + + private void Wire(params IServer[] servers) + { + _multiplexer.GetServers().Returns(servers); + _multiplexer.GetServer(Arg.Any()).Returns(call => servers.First(s => s.EndPoint.Equals(call.Arg()))); + } + + private IServer Server(string host, bool connected, ServerType type = ServerType.Standalone, long count = 0) + { + var server = Substitute.For(); + server.EndPoint.Returns(new DnsEndPoint(host, 6379)); + server.IsConnected.Returns(connected); + server.ServerType.Returns(type); + server.ExecuteAsync("PUBSUB", Arg.Any>(), Arg.Any()).Returns(call => + { + var arguments = call.ArgAt>(1); + var first = arguments.First(); + if (first.Equals("NUMSUB")) + { + _numsub.Add([.. arguments.Skip(1).Select(argument => (string)argument)]); + } + + return Task.FromResult(first.Equals("NUMPAT") + ? RedisResult.Create((RedisValue)0) + : RedisResult.Create([.. arguments.Skip(1).SelectMany(name => new[] { RedisResult.Create((RedisValue)(string)name, ResultType.BulkString), RedisResult.Create((RedisValue)count) })])); + }); + return server; + } + + private sealed class NoGroups : IStreamGroupSource + { + public Task> GroupsAsync(RedisKey stream, CancellationToken cancellationToken) => Task.FromResult>([]); + + public Task IdleAsync(RedisKey stream, string group, CancellationToken cancellationToken) => Task.FromResult(null); + } + + private sealed class Subscriptions : IPubSubSubscriptions + { + public IReadOnlyList Channels() => [Channel]; + + public IReadOnlyList Uncountable() => []; + + public IReadOnlyDictionary Confirmed() => new Dictionary { [Channel] = 1 }; + + public int Patterns() => 0; + + public void WhenConfirmed(Action handler) + { + } + } +}