Skip to content

Commit 252edb9

Browse files
committed
fix(negotiation): never write bytes that read back differently, and advertise what reads do
A policy on a custom or raw byte proxy has no JSON to fall back to, and what the proxy writes can read back as a format header, so such a policy always writes the format it names, as Force does. The advertisement follows the sniff gate of the node's own reads: a policy whose reads do not look for a header advertises JSON only, marked nosniff, and no writer upgrades while it is on the roster. What an advertisement says it cannot read is judged before an unaccounted group, so the grace after an outage never excuses a known incompatible peer. The advertisement and the roster read run on their own due times, a read that is due advertising first. A topic's stream is forgotten when the topic is disposed. Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>
1 parent e5f0a7d commit 252edb9

16 files changed

Lines changed: 303 additions & 39 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
1515

1616
### Changed
1717

18-
- **A policy's non-JSON `Serializer.Write` is written only while every node can read it.** With the new `RedisCacheOptions.FormatNegotiation` (`Mode` `Auto` by default), each node advertises the formats it reads in a Redis hash stamped with the server's clock and reads the roster, with the Streams consumer groups, on a 15-second timer, off the read and write paths. A policy writes its format once every fresh advertisement lists it, no live node is unaccounted for, and that has held for `UpgradeHold` (an hour by default; zero switches at once); a node that cannot read it stops the writes at once, and an unreadable roster keeps the decision for `MaxViewAge` (two minutes), then writes JSON. A node with a serializer registered advertises what it reads in either mode, even if none of its policies writes a non-JSON format, so a reader-only or `Force` node never blocks the writers; in `Auto` it decides for every registered serializer, so a policy built at the call site is held to it too. An advertisement outlives the consumer group of a node that crashed, and a node that stops writes a departing advertisement, so a node that leaves never drops the fleet to JSON or restarts the hold, which a join or a restart does. After a Redis outage the decision is kept for two advertise intervals while peers advertise again. `AssumeAllNodesCanRead` covers only nodes that advertise nothing. An app with no serializer registered does nothing new. An app that cannot see its peers (`BroadcastEnabled` off, or Pub/Sub only) writes JSON unless `AssumeAllNodesCanRead` is set. `Mode = Force` writes exactly what each policy configures, as before. New metrics `cache.format.effective`, `cache.format.fallback`, `cache.format.nodes` and `cache.format.view.age`. See [choosing when a format is written](docs/concepts.md#choosing-when-a-format-is-written).
18+
- **A policy's non-JSON `Serializer.Write` is written only while every node can read it.** With the new `RedisCacheOptions.FormatNegotiation` (`Mode` `Auto` by default), each node advertises the formats it reads in a Redis hash stamped with the server's clock and reads the roster, with the Streams consumer groups, on a 15-second timer, off the read and write paths. A policy writes its format once every fresh advertisement lists it, no live node is unaccounted for, and that has held for `UpgradeHold` (an hour by default; zero switches at once); a node that cannot read it stops the writes at once, and an unreadable roster keeps the decision for `MaxViewAge` (two minutes), then writes JSON. A node with a serializer registered advertises what it reads in either mode, even if none of its policies writes a non-JSON format, so a reader-only or `Force` node never blocks the writers; in `Auto` it decides for every registered serializer, so a policy built at the call site is held to it too. An advertisement outlives the consumer group of a node that crashed, and a node that stops writes a departing advertisement, so a node that leaves never drops the fleet to JSON or restarts the hold, which a join or a restart does. After a Redis outage the decision is kept for two advertise intervals while peers advertise again. `AssumeAllNodesCanRead` covers only nodes that advertise nothing. The advertisement follows what a node's reads really do: on a custom proxy a policy that writes no non-JSON format advertises JSON only, which keeps writers on JSON, and a policy on such a proxy always writes its configured format, since raw bytes have no JSON to fall back to and could read back as a header. Advertise and roster reads run on their own intervals, and a topic's stream is forgotten when the topic is removed. An app with no serializer registered does nothing new. An app that cannot see its peers (`BroadcastEnabled` off, or Pub/Sub only) writes JSON unless `AssumeAllNodesCanRead` is set. `Mode = Force` writes exactly what each policy configures, as before. New metrics `cache.format.effective`, `cache.format.fallback`, `cache.format.nodes` and `cache.format.view.age`. See [choosing when a format is written](docs/concepts.md#choosing-when-a-format-is-written).
1919
- **An event in a known format that cannot be decoded is a topic loss.** The Streams reader acknowledges the entry and reports a loss, and counts such an entry as one it would have acted on when it checks what a new group skipped; the Pub/Sub reader expires through the coalesced dropped-event path, off the delivery thread, so it holds for caches that keep their tier over a subscription gap. The Streams reader reports once per batch. This applies to a body whose first byte is `0x02`, `0x03` or a consumer format id (`0x10`-`0x1E`, `0x1F`), such as a binary event on a node that cannot read it. Any other body that does not decode, another producer's first byte included, is still skipped as junk.
2020
- **Cache event properties are read as typed values.** The built-in event formatters parse `_version_`, the expirations, the metadata and the peer fields straight into `long`, `DateTimeOffset`, string dictionaries and byte arrays instead of a `JsonElement` that was parsed again by the value serializer, which saves a clone, a box and a second parse per property of a peer-update event. The event bytes on the wire are unchanged. A custom `ISerializerProxy<byte[]>` no longer sees these properties in `TryDeserialize(object)` when the built-in formatters decode the event.
2121

‎docs/concepts.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -553,6 +553,8 @@ is pinned to JSON, and `Mode = Force` writes exactly what each policy configures
553553

554554
How it works. An app with a serializer registered advertises what it reads, whether or not any policy of its own writes a non-JSON format and in either mode, so a node that only reads, or runs `Force`, never blocks the writers. In `Auto` it also reads the roster and decides, for every registered serializer, so a policy built at the call site that names one is held to the same decision as a registered policy. An app with no serializer registered does none of this: it advertises nothing, and so correctly counts as a node that cannot read.
555555

556+
The advertisement says what the node's reads actually do. With the default JSON proxy a read looks for a format header whatever the policy writes. With a custom proxy, a subclass of the default or `RawByteSerializerProxy`, a read looks for one only under a policy that itself writes a non-JSON format; for every other policy (and the default policy, when it is not one) the node advertises that it reads JSON only, and no writer upgrades while such a node is on the roster. A policy on such a proxy never falls back to JSON either, since its bytes are not JSON and could read back as a header: it always writes the format it names, as in `Force`.
557+
556558
- Each node writes an advertisement (the formats it reads, and the read list of each policy that sets one) to a hash in Redis every
557559
15 seconds, stamped with the server's clock, so a node's own clock never matters. It advertises first when its hosted service starts, before it reads the
558560
roster; the service is registered with the caching, so an app hosted service that touches a cache earlier can subscribe first and be briefly unaccounted

‎docs/reference/settings.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -215,7 +215,7 @@ Per-topic overrides: add entries to `Topics[]` under `Broadcast:RedisPubSub`. Ea
215215
| `ReadScriptsEnabled` | `bool` | `false` | Per-provider | By default a value and its expiration are read with a pipelined `GET` and `PTTL` (one round trip, no transaction); the pair is not atomic, so a key that expires between the two reads as a value without a TTL. Set to `true` to read on Redis 7 and later with one `EVAL_RO` script instead (a batch read runs one script for all its keys; less allocation, but several times the server time of a plain read on Azure Managed Redis). The connection's user then needs `EVAL_RO`/`EVALSHA_RO` (ACL category `@scripting`); if the first script fails for any reason, or the server later refuses one (`NOPERM`, or a disabled or renamed command), reads switch to the `GET`/`PTTL` pair for good with one warning per cache instance, so a server or proxy that cannot run Lua never turns reads into misses. Before Redis 7 no script is sent. Command statistics then show `EVAL_RO` instead of `GET`/`PTTL`. |
216216
| `ReadFrom` | `RedisReadFrom` | `Master` | Per-provider | `Master` sends reads to the primary (`PreferMaster`); `Replica` sends them to a replica when one exists (`PreferReplica`). On Azure Redis 6 a replica round trip measured about 3 times the primary's (1.6 ms against 0.55 ms), and a replica can still hold a key the primary has already removed, so a read right after a write or an invalidation could return, or refill the entry with, stale data. With `Replica`, a key this node wrote or removed (counted from when the write completes), or heard an invalidation for through a live local entry, within `MasterReadWindow` still reads from the primary, as does every read for that window after a lost-events clear on a node holding local entries and, if more than 10,000 keys were touched within one window, every read until the window passes. A key with no local entry is not guarded against another node's write. Turn it on only to take read load off the primary. Writes and refreshes always go to the primary. |
217217
| `MasterReadWindow` | `TimeSpan` | `00:00:01` | Per-provider | With `ReadFrom` set to `Replica`, how long a key this node wrote, removed or heard invalidated keeps reading from the master. Must be positive, and must exceed the expected replica lag: a replica that is further behind can still return the old value once the window ends. A fire-and-forget TTL write (`AwaitRefresh` off) starts the window when it is queued, not when the server applies it. Measured on a monotonic clock. Ignored with `Master`. |
218-
| `FormatNegotiation:Mode` | `FormatNegotiationMode` | `Auto` | Per-provider | `Auto` decides which formats this node writes: a registered serializer is written only while every node on the Streams transport is known to read it, whichever policy asks for it (a registered policy or one built at the call site), and JSON otherwise. `Force` writes exactly what each policy configures, with no roster and no decision. In both modes a node with a serializer registered advertises the formats it reads, so a `Force` node does not keep an `Auto` neighbour on JSON. A policy with `Write = "json"` is never affected. See [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written). |
218+
| `FormatNegotiation:Mode` | `FormatNegotiationMode` | `Auto` | Per-provider | `Auto` decides which formats this node writes: a registered serializer is written only while every node on the Streams transport is known to read it, whichever policy asks for it (a registered policy or one built at the call site), and JSON otherwise. `Force` writes exactly what each policy configures, with no roster and no decision. In both modes a node with a serializer registered advertises the formats it reads, so a `Force` node does not keep an `Auto` neighbour on JSON. A policy with `Write = "json"` is never affected, and neither is one on a custom `ISerializerProxy<byte[]>` (or a subclass of the default, or `RawByteSerializerProxy`): its bytes are not JSON, so there is nothing safe to fall back to, and it always writes the format the policy names, as `Force` does. See [choosing when a format is written](../concepts.md#choosing-when-a-format-is-written). |
219219
| `FormatNegotiation:AdvertiseInterval` | `TimeSpan` | `00:00:15` | Per-provider | How often this node rewrites its advertisement (the formats it reads) in Redis, stamped with the server's clock. Every node judges the others' advertisements by its own settings, so set `AdvertiseInterval`, `ViewInterval` and `LivenessWindow` alike across the fleet. An advertisement counts for `LivenessWindow` plus two of these, so it outlives the consumer group of a node that crashed. Must be positive. |
220220
| `FormatNegotiation:ViewInterval` | `TimeSpan` | `00:00:15` | Per-provider | How often this node reads the advertisements and the Streams consumer groups; it advertises first, on the same tick. Must be positive. |
221221
| `FormatNegotiation:LivenessWindow` | `TimeSpan` | `00:02:00` | Per-provider | A consumer group no advertisement accounts for (a node before 3.2) counts as a live node, and blocks the switch, while its consumer read within this window. A group no consumer ever read from counts as nobody. Must be positive. |

‎src/UiPath.Caching/Broadcast/Redis/RedisStreamsTopicProvider.cs‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,8 @@ public RedisStreamsTopicProvider(
4141

4242
public override bool Enabled { get; }
4343

44+
internal int StreamKeyCount => _streamKeys.Count;
45+
4446
/// <summary>The stream of each topic this provider has made, for the node roster.</summary>
4547
internal IReadOnlyList<RedisKey> StreamKeys()
4648
{
@@ -60,7 +62,7 @@ protected override ITopic<ICacheEvent> CreateInternalTopic(TopicKey topicKey)
6062
{
6163
var resolved = ResolveOptions(topicKey) ?? _redisStreamsTopicOptions;
6264
_streamKeys[topicKey] = (resolved.RedisStreamKeyStrategy ?? new PrefixStrategy(RedisKeyspaces.Streams, _cacheOptions)).GetRedisKey(topicKey);
63-
return new RedisStreamsTopic<ICacheEvent>(
65+
var topic = new RedisStreamsTopic<ICacheEvent>(
6466
topicKey,
6567
ConnectionState,
6668
Redis,
@@ -73,6 +75,10 @@ protected override ITopic<ICacheEvent> CreateInternalTopic(TopicKey topicKey)
7375
Telemetry,
7476
Profiler,
7577
_stopTokenSource.Token);
78+
79+
// The topic's stream goes with it, so a provider that makes and drops many topics keeps no record of the dropped ones.
80+
topic.OnDisposed += (_, _) => _streamKeys.TryRemove(topicKey, out _);
81+
return topic;
7682
}
7783

7884
protected override void Dispose(bool disposing)

‎src/UiPath.Caching/Negotiation/FormatNegotiation.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ public static FormatNegotiation Create(IServiceProvider services)
5656
Negotiator = options.Mode == FormatNegotiationMode.Force
5757
? null
5858
: new FormatNegotiator(NegotiationPlan.Desired(registry, policies), options, clock, services.GetRequiredService<ICachingTelemetryProvider>(), loggers.CreateLogger<FormatNegotiator>()),
59-
Advertisement = NegotiationPlan.Advertisement(registry, policies, library),
59+
Advertisement = NegotiationPlan.Advertisement(registry, policies, new ValueReader(services.GetRequiredService<ISerializerProxy<byte[]>>(), registry), library),
6060
NodeId = (cacheOptions.SourceUri ?? CacheOptions.MachineUri).ToString().ToLowerInvariant(),
6161
Store = new NodeRosterStore(connector, NodeRosterStore.KeyFor(cacheOptions, redisOptions)),
6262
Roster = new StreamRoster(new RedisStreamGroupSource(connector), options),

‎src/UiPath.Caching/Negotiation/FormatNegotiationService.cs‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ public async Task StartAsync(CancellationToken cancellationToken)
3737
// A node that only reads has no decision to make, so it never reads the roster.
3838
_loops = negotiation.Negotiator is null
3939
? [Task.Run(() => Loop(options.AdvertiseInterval, AdvertiseAsync), CancellationToken.None)]
40-
: [Task.Run(() => Loop(TimeSpan.FromTicks(Math.Min(options.AdvertiseInterval.Ticks, options.ViewInterval.Ticks)), TickAsync), CancellationToken.None)];
40+
: [Task.Run(ScheduledLoop, CancellationToken.None)];
4141
}
4242

4343
public async Task StopAsync(CancellationToken cancellationToken)
@@ -154,6 +154,38 @@ private async Task Prune(IReadOnlyCollection<StoredAdvertisement> gone, Cancella
154154
}
155155
}
156156

157+
// The advertisement and the roster read keep their own cadences; a read that is due advertises first.
158+
private async Task ScheduledLoop()
159+
{
160+
var options = negotiation.Options;
161+
var schedule = new TickSchedule(options.AdvertiseInterval, options.ViewInterval);
162+
var started = negotiation.Clock.GetTimestamp();
163+
while (!_stop.IsCancellationRequested)
164+
{
165+
try
166+
{
167+
await Task.Delay(schedule.Wait(negotiation.Clock.GetElapsedTime(started)), negotiation.Clock, _stop.Token).ConfigureAwait(false);
168+
var (advertise, refresh) = schedule.Take(negotiation.Clock.GetElapsedTime(started));
169+
if (refresh)
170+
{
171+
await TickAsync(_stop.Token).ConfigureAwait(false);
172+
}
173+
else if (advertise)
174+
{
175+
await AdvertiseAsync(_stop.Token).ConfigureAwait(false);
176+
}
177+
}
178+
catch (OperationCanceledException) when (_stop.IsCancellationRequested)
179+
{
180+
return;
181+
}
182+
catch (Exception ex) when (ex is not OutOfMemoryException)
183+
{
184+
negotiation.Logger.LogWarning(ex, "A format negotiation step failed; it runs again when it is next due.");
185+
}
186+
}
187+
}
188+
157189
private async Task Loop(TimeSpan interval, Func<CancellationToken, Task> step)
158190
{
159191
while (!_stop.IsCancellationRequested)

‎src/UiPath.Caching/Negotiation/FormatNegotiator.cs‎

Lines changed: 17 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -146,25 +146,7 @@ public void ObservationFailed()
146146

147147
private string? Block(DesiredWrite want, RosterView view)
148148
{
149-
var attested = _options.AssumeAllNodesCanRead;
150-
if (!view.Complete)
151-
{
152-
if (!attested)
153-
{
154-
return ReasonTransportUnknown;
155-
}
156-
}
157-
else if (view.Ads.Count == 0)
158-
{
159-
return ReasonNoView;
160-
}
161-
162-
// The attestation covers nodes that say nothing; a node that advertises without the format is not one of them.
163-
if (view.Blockers.Count > 0 && !attested)
164-
{
165-
return ReasonUnaccountedGroup;
166-
}
167-
149+
// What an advertisement says it cannot read comes first: no grace after an outage and no attestation excuses it.
168150
foreach (var stored in view.Ads)
169151
{
170152
var ad = stored.Advertisement;
@@ -175,14 +157,28 @@ public void ObservationFailed()
175157

176158
foreach (var (policy, list) in ad.Strict)
177159
{
178-
if (!want.Others.Contains(policy) && !want.Tokens.All(list.Contains))
160+
if (list.Contains(NodeAdvertisement.NoSniff) || (!want.Others.Contains(policy) && !want.Tokens.All(list.Contains)))
179161
{
180162
return ReasonMissingFormat;
181163
}
182164
}
183165
}
184166

185-
return null;
167+
var attested = _options.AssumeAllNodesCanRead;
168+
if (!view.Complete)
169+
{
170+
if (!attested)
171+
{
172+
return ReasonTransportUnknown;
173+
}
174+
}
175+
else if (view.Ads.Count == 0)
176+
{
177+
return ReasonNoView;
178+
}
179+
180+
// The attestation covers nodes that say nothing; a node that advertises without the format is not one of them.
181+
return view.Blockers.Count > 0 && !attested ? ReasonUnaccountedGroup : null;
186182
}
187183

188184
private void Transition(WriterState[] previous, bool[] allowed)

0 commit comments

Comments
 (0)