Skip to content

Commit 42ea5b0

Browse files
committed
fix(negotiation): count every server, pattern subscribers and the channel prefix
A server this node knows about but is not connected to is not left out of the NUMSUB sum: it would hide that server's subscribers and let two matching views enable binary events. The whole view fails instead, which resets the debounce and keeps the last decision until MaxViewAge. PUBSUB NUMSUB leaves pattern subscriptions out. The same per-server round trip now reads PUBSUB NUMPAT, each node advertises the pattern subscriptions it holds (psub), and more patterns on the servers than the advertisements claim is an unknown subscriber under the same two-view rule as a channel surplus. NUMPAT counts the whole server, so another app's pattern subscriber blocks too; the docs and the Warning name AssumeAllNodesCanRead as the escape. The channel names sent to NUMSUB carry the connection's ChannelPrefix, which the server applies to every subscription made through it. Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>
1 parent 50d5b00 commit 42ea5b0

18 files changed

Lines changed: 407 additions & 37 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
1616
### Changed
1717

1818
- **A policy's non-JSON `Serializer.Write` is written only while every node can read it.** With the new `RedisCacheOptions.FormatNegotiation` (`Mode` `Auto` by default), each node advertises the formats it reads in a Redis hash stamped with the server's clock and reads the roster, with the Streams consumer groups, on a 15-second timer, off the read and write paths. A policy writes its format once every fresh advertisement lists it, no live node is unaccounted for, and that has held for `UpgradeHold` (an hour by default; zero switches at once); a node that cannot read it stops the writes at once, and an unreadable roster keeps the decision for `MaxViewAge` (two minutes), then writes JSON. A node with a serializer registered advertises what it reads in either mode, even if none of its policies writes a non-JSON format, so a reader-only or `Force` node never blocks the writers; in `Auto` it decides for every registered serializer, so a policy built at the call site is held to it too. An advertisement outlives the consumer group of a node that crashed, and a node that stops writes a departing advertisement, so a node that leaves never drops the fleet to JSON or restarts the hold, which a join or a restart does. After a Redis outage the decision is kept for two advertise intervals while peers advertise again. `AssumeAllNodesCanRead` covers only nodes that advertise nothing. The advertisement follows what a node's reads really do: on a custom proxy a policy that writes no non-JSON format advertises JSON only, which keeps writers on JSON, and a policy on such a proxy always writes its configured format, since raw bytes have no JSON to fall back to and could read back as a header. Advertise and roster reads run on their own intervals, and a topic's stream is forgotten when the topic is removed. An app with no serializer registered does nothing new. An app that cannot see its peers (`BroadcastEnabled` off, or no Streams or Pub/Sub topic) writes JSON unless `AssumeAllNodesCanRead` is set. `Mode = Force` writes exactly what each policy configures, as before. New metrics `cache.format.effective`, `cache.format.fallback`, `cache.format.nodes` and `cache.format.view.age`. See [choosing when a format is written](docs/concepts.md#choosing-when-a-format-is-written).
19-
- **Binary events are written only while every live node can read them, over Streams and over Pub/Sub.** `AddMessagePackEvents(write: true)` now means that; `write: false` always writes JSON and `FormatNegotiation:Mode = Force` writes binary at once, as before. Each advertisement lists the event formats the node decodes (`er`): an app with the decoder lists JSON and binary, a node with a serializer and no decoder lists JSON, and a node before this version lists nothing and blocks through the roster. Events are decided apart from values, with the same rules: a fall back is immediate, an upgrade waits for `UpgradeHold`, the decoder always reads both formats, and the formatter reads the cached decision per event with no call to Redis and no allocation. An app with the decoder and no value serializer advertises and decides too. An old (3.0 or 3.1) node blocks binary events while it is live in the roster; its stale local entries are possible only in the join window and when a node judged dead returns on the same source id, bounded by its own `LocalMaxExpiration`, so set a short one on old nodes before enabling `write: true`. **Pub/Sub deployments no longer always write JSON.** With no consumer groups to list, the roster sums `PUBSUB NUMSUB` of each channel over every server (primaries and replicas) and subtracts the subscriptions the advertisements claim (`sub`), which a node writes only once the server has confirmed its subscription, and writes again at once. A subscriber nobody claims keeps values and events on JSON; it takes two views in a row to count, and an upgrade needs two clean views over the same number of servers. Fewer subscribers than are claimed (a node that crashed, whose advertisement outlives it) is also unknown, so JSON after two views, and as a departure it never restarts the hold: once the advertisement expires and the counts match for two views, the earlier decision resumes. A node that left claims nothing. An app with no Redis connection has no roster, and writes JSON events unless `AssumeAllNodesCanRead` or `Force`. A node that is stopping lists the subscriptions it still holds in its departing advertisement, which count as optional for its lifetime, so a slow shutdown never takes the fleet to JSON. A deficit that clears while the claim is still fresh is a join and restarts the hold; a one-view blip pauses a hold and no longer restarts it. The roster sends one `PUBSUB NUMSUB` per server per view, logs a Warning once when it is refused, and treats a pattern or sharded channel strategy as uncountable, so JSON unless attested. Advertisements are written one at a time and none after the departing one. A server that refuses `PUBSUB NUMSUB` makes the count unknown, so JSON, unless `AssumeAllNodesCanRead`. New metric `Caching.Stats.FormatBlockers` carries who blocks, on its own and not on the bounded `cache.format.*` metrics; `cache.format.effective` and `cache.format.fallback` gain a `kind` tag (`value` or `event`) and the reasons `unaccounted_subscriber` and `subscriber_deficit`. See [choosing when a format is written](docs/concepts.md#choosing-when-a-format-is-written).
19+
- **Binary events are written only while every live node can read them, over Streams and over Pub/Sub.** `AddMessagePackEvents(write: true)` now means that; `write: false` always writes JSON and `FormatNegotiation:Mode = Force` writes binary at once, as before. Each advertisement lists the event formats the node decodes (`er`): an app with the decoder lists JSON and binary, a node with a serializer and no decoder lists JSON, and a node before this version lists nothing and blocks through the roster. Events are decided apart from values, with the same rules: a fall back is immediate, an upgrade waits for `UpgradeHold`, the decoder always reads both formats, and the formatter reads the cached decision per event with no call to Redis and no allocation. An app with the decoder and no value serializer advertises and decides too. An old (3.0 or 3.1) node blocks binary events while it is live in the roster; its stale local entries are possible only in the join window and when a node judged dead returns on the same source id, bounded by its own `LocalMaxExpiration`, so set a short one on old nodes before enabling `write: true`. **Pub/Sub deployments no longer always write JSON.** With no consumer groups to list, the roster sums `PUBSUB NUMSUB` of each channel over every server (primaries and replicas) and subtracts the subscriptions the advertisements claim (`sub`), which a node writes only once the server has confirmed its subscription, and writes again at once. A subscriber nobody claims keeps values and events on JSON; it takes two views in a row to count, and an upgrade needs two clean views over the same number of servers. Fewer subscribers than are claimed (a node that crashed, whose advertisement outlives it) is also unknown, so JSON after two views, and as a departure it never restarts the hold: once the advertisement expires and the counts match for two views, the earlier decision resumes. A node that left claims nothing. An app with no Redis connection has no roster, and writes JSON events unless `AssumeAllNodesCanRead` or `Force`. A node that is stopping lists the subscriptions it still holds in its departing advertisement, which count as optional for its lifetime, so a slow shutdown never takes the fleet to JSON. A deficit that clears while the claim is still fresh is a join and restarts the hold; a one-view blip pauses a hold and no longer restarts it. A server the node knows about but is not connected to fails the whole view instead of being left out of the sum. The same round trip reads `PUBSUB NUMPAT`, nodes advertise the pattern subscriptions they hold (`psub`), and unclaimed patterns on the servers, another app's included, keep formats on JSON unless `AssumeAllNodesCanRead`. The channel names sent to `PUBSUB NUMSUB` carry the connection's `ChannelPrefix`. The roster sends one `PUBSUB NUMSUB` per server per view, logs a Warning once when it is refused, and treats a pattern or sharded channel strategy as uncountable, so JSON unless attested. Advertisements are written one at a time and none after the departing one. A server that refuses `PUBSUB NUMSUB` makes the count unknown, so JSON, unless `AssumeAllNodesCanRead`. New metric `Caching.Stats.FormatBlockers` carries who blocks, on its own and not on the bounded `cache.format.*` metrics; `cache.format.effective` and `cache.format.fallback` gain a `kind` tag (`value` or `event`) and the reasons `unaccounted_subscriber` and `subscriber_deficit`. See [choosing when a format is written](docs/concepts.md#choosing-when-a-format-is-written).
2020
- **An event in a known format that cannot be decoded is a topic loss.** The Streams reader acknowledges the entry and reports a loss, and counts such an entry as one it would have acted on when it checks what a new group skipped; the Pub/Sub reader expires through the coalesced dropped-event path, off the delivery thread, so it holds for caches that keep their tier over a subscription gap. The Streams reader reports once per batch. This applies to a body whose first byte is `0x02`, `0x03` or a consumer format id (`0x10`-`0x1E`, `0x1F`), such as a binary event on a node that cannot read it. Any other body that does not decode, another producer's first byte included, is still skipped as junk.
2121
- **Cache event properties are read as typed values.** The built-in event formatters parse `_version_`, the expirations, the metadata and the peer fields straight into `long`, `DateTimeOffset`, string dictionaries and byte arrays instead of a `JsonElement` that was parsed again by the value serializer, which saves a clone, a box and a second parse per property of a peer-update event. The event bytes on the wire are unchanged. A custom `ISerializerProxy<byte[]>` no longer sees these properties in `TryDeserialize(object)` when the built-in formatters decode the event.
2222

‎docs/concepts.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -596,8 +596,13 @@ lists nothing, and so counts as one that cannot say what it reads. An app with t
596596

597597
The Streams transport has a consumer group per node, which is how a node that advertises nothing is found. Pub/Sub has no membership, so over Pub/Sub the roster counts subscribers:
598598

599-
- Each node counts the subscribers of each channel it listens on with `PUBSUB NUMSUB`, on every server it is connected to, primaries and replicas, and sums them, because a subscriber is local to the server it
600-
connected to. Asking one server undercounts.
599+
- Each node counts the subscribers of each channel it listens on with `PUBSUB NUMSUB`, on every server of the deployment, primaries and replicas, and sums them, because a subscriber is local to the server it
600+
connected to. Asking one server undercounts. A server this node knows about but is not connected to would hide its subscribers, so the whole view fails: it is not a partial sum, it resets the debounce
601+
and keeps the last decision until `MaxViewAge`, then JSON. The channel names it asks for carry the connection's `ChannelPrefix`, since the server only knows the prefixed ones.
602+
- `NUMSUB` leaves pattern subscriptions out, so the same round trip also reads `PUBSUB NUMPAT`. Each node advertises the pattern subscriptions it holds (`psub`, none unless a channel strategy makes patterns),
603+
and more patterns on the servers than the advertisements claim is an unknown subscriber, under the same two-view rule as a surplus on a channel. `NUMPAT` counts the whole server and its unique patterns, not this app's
604+
channels: another app on the same Redis that subscribes to a pattern (a keyspace listener, say) keeps binary events on JSON, and so does a tool left running. There is no deficit rule for patterns. Set
605+
`AssumeAllNodesCanRead` to attest that none of them is a node that needs to read binary events.
601606
- Each node advertises the subscriptions it holds (`sub`), once the server has confirmed them, and says so at once, not on the next interval. The order is deliberate: subscribe, then advertise. A claim made
602607
first would hide a subscriber that says nothing. A node that is stopping writes its departing advertisement (`left=1`) with the subscriptions it still holds, and its connection closes a little later, maybe views later:
603608
for as long as that advertisement lives, the count may include its subscription or not, either is clean, so a slow shutdown never takes the fleet to JSON. The roster sends one `PUBSUB NUMSUB` per server per view, for all
@@ -626,7 +631,7 @@ An app with no Redis connection has no roster at all, and `write: true` writes J
626631

627632
`cache.format.effective` is 1 while a format is written and 0 while it is not, tagged `kind` (`value` or `event`) and `format`, and `policy` for a value. `cache.format.fallback` counts transitions to JSON, tagged
628633
`reason`, `kind` and `policy`; the reasons are `missing_format`, `unaccounted_group`, `unaccounted_subscriber`, `subscriber_deficit`, `transport_unknown`, `no_view` and `view_stale`. `cache.format.nodes` and `cache.format.view.age` are the live and
629-
blocking nodes and the age of the view. Who blocks is kept off these metrics, whose tags are bounded: the node ids, or the Pub/Sub channels with a surplus or a deficit, are in a Warning log and on the separate `Caching.Stats.FormatBlockers`, each time the set
634+
blocking nodes and the age of the view. Who blocks is kept off these metrics, whose tags are bounded: the node ids, or the Pub/Sub channels (or `(patterns)`) with a surplus or a deficit, are in a Warning log and on the separate `Caching.Stats.FormatBlockers`, each time the set
630635
changes. Those are as many values as there are nodes, so a backend that turns a metric's tags into dimensions will see them as one each: alert on the metric's value, not on its tags. A fallback of an event has no `policy` tag.
631636

632637
See also: [how-to/extending.md#value-serializers](how-to/extending.md#value-serializers).

‎docs/reference/settings.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -221,7 +221,7 @@ Per-topic overrides: add entries to `Topics[]` under `Broadcast:RedisPubSub`. Ea
221221
| `FormatNegotiation:LivenessWindow` | `TimeSpan` | `00:02:00` | Per-provider | A consumer group no advertisement accounts for (a node before 3.2) counts as a live node, and blocks the switch, while its consumer read within this window. A group no consumer ever read from counts as nobody. Must be positive. |
222222
| `FormatNegotiation:UpgradeHold` | `TimeSpan` | `01:00:00` | Per-provider | How long every node must have been accounted for before a format is written, counted from the last time a node joined or restarted (a node that leaves never restarts it). Zero switches at once; negative is rejected. A shorter hold switches sooner, but if the app is rolled back after it, nodes that cannot read the format miss and rewrite values as JSON, never a wrong value. A longer one means an app rolled back within it never wrote the format, so the older nodes see no misses. A node that cannot read the format stops the writes at once, whatever the hold. |
223223
| `FormatNegotiation:MaxViewAge` | `TimeSpan` | `00:02:00` | Per-provider | While Redis cannot be read, the last decision stands for this long since the last good read, then values are written as JSON; a decision older than this is never used, even if a read hangs. After an outage shorter than this, the decision is also kept for two `AdvertiseInterval`s of successful ticks, so peers can advertise again, unless an advertisement says a node cannot read the format. Must be positive. |
224-
| `FormatNegotiation:AssumeAllNodesCanRead` | `bool` | `false` | Per-provider | Attests that the nodes that say nothing read the desired formats: a live Streams consumer group with no advertisement, a Pub/Sub subscriber no advertisement accounts for, a Pub/Sub deployment where `PUBSUB NUMSUB` is refused, and, for an app with `BroadcastEnabled` off, with neither a Streams nor a Pub/Sub topic, or with no Redis connection, the nodes it cannot see at all. An advertisement that lacks the format still stops the writes, and `UpgradeHold` still applies, counted from the first decision. Without it such an app writes JSON. |
224+
| `FormatNegotiation:AssumeAllNodesCanRead` | `bool` | `false` | Per-provider | Attests that the nodes that say nothing read the desired formats: a live Streams consumer group with no advertisement, a Pub/Sub subscriber no advertisement accounts for (a pattern subscriber included, which `PUBSUB NUMPAT` counts for the whole server, other apps' too), a Pub/Sub deployment where `PUBSUB NUMSUB` is refused, and, for an app with `BroadcastEnabled` off, with neither a Streams nor a Pub/Sub topic, or with no Redis connection, the nodes it cannot see at all. An advertisement that lacks the format still stops the writes, and `UpgradeHold` still applies, counted from the first decision. Without it such an app writes JSON. |
225225

226226
*Code-only seams:* `EntryFactory`, `CacheKeyStrategy`, `RedisKeyStrategyFactory`.
227227

‎src/UiPath.Caching/Broadcast/Redis/RedisPubSubTopic.cs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,8 @@ internal event Action? Subscribed
5959

6060
internal string ChannelName => _redisChannel.ToString();
6161

62+
internal bool IsPattern => _redisChannel.IsPattern;
63+
6264
internal bool Countable => !_redisChannel.IsPattern && !_redisChannel.IsSharded;
6365

6466
internal bool IsSubscribed => _subscriber.IsSubscribed;

‎src/UiPath.Caching/Broadcast/Redis/RedisPubSubTopicProvider.cs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,9 @@ IReadOnlyList<string> IPubSubSubscriptions.Uncountable() =>
5555
IReadOnlyDictionary<string, int> IPubSubSubscriptions.Confirmed() =>
5656
Topics().Where(topic => topic.Countable && topic.IsSubscribed).Select(topic => topic.ChannelName).Distinct(StringComparer.Ordinal).ToDictionary(channel => channel, _ => 1, StringComparer.Ordinal);
5757

58+
int IPubSubSubscriptions.Patterns() =>
59+
Topics().Where(topic => topic.IsPattern && topic.IsSubscribed).Select(topic => topic.ChannelName).Distinct(StringComparer.Ordinal).Count();
60+
5861
void IPubSubSubscriptions.WhenConfirmed(Action handler) => SubscriptionConfirmed += handler;
5962

6063
protected override ITopic<ICacheEvent> CreateInternalTopic(TopicKey topicKey)

0 commit comments

Comments
 (0)