diff --git a/sdk/cosmos/.cspell.json b/sdk/cosmos/.cspell.json index 06a6e92503..a68c74911a 100644 --- a/sdk/cosmos/.cspell.json +++ b/sdk/cosmos/.cspell.json @@ -123,6 +123,7 @@ "failback", "failovers", "fanout", + "fanouts", "Fatalf", "fieldless", "FILETIME", @@ -351,6 +352,7 @@ "unpadded", "uncontended", "undecoded", + "undercount", "underspecified", "undrained", "unemitted", diff --git a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md index 154e208cba..e680b7fea8 100644 --- a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md @@ -12,12 +12,14 @@ - Added an SDK-generated `x-ms-client-id` header that remains stable for each `CosmosClient`. ([#4844](https://github.com/Azure/azure-sdk-for-rust/pull/4844)) - Added opt-in Cosmos binary JSON encoding for item operations (`create`/`read`/`replace`/`upsert`). Enable it via `CosmosClientBuilder::with_binary_encoding_options` (or the `AZURE_COSMOS_BINARY_ENCODING_ENABLED` environment-variable fallback). Off by default; when disabled, requests and responses are byte-for-byte unchanged. ([#4671](https://github.com/Azure/azure-sdk-for-rust/pull/4671)) - Added `FeedOptions::max_fan_out` (and `FeedOptions::with_max_fan_out`) to cap how many physical partitions a cross-partition query or change feed may fan out to. Applies to `ContainerClient::query_items` and `ContainerClient::query_change_feed`. The cap is enforced only at initial query setup; a partition that splits mid-execution and pushes the fan-out higher does not abort the operation. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855)) +- Surfaced the Hedging Detection API through `DiagnosticsContext`: `hedging_started()`, `requested_regions()`, `responded_regions()`, and the exact-count `total_requested_regions()` / `total_responded_regions()`, and re-exported the `RequestedRegion` / `RequestedRegionReason` types (mirroring the existing `DiagnosticsContext` re-export). ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Added resumable cross-partition streaming `ORDER BY` query support. ([#4800](https://github.com/Azure/azure-sdk-for-rust/pull/4800)) - Added a pluggable client-side diagnostics emission layer — the `DiagnosticsHandler` trait and ordered `DiagnosticsHandlerChain` (registered via `CosmosClientBuilder::with_diagnostics_handler`) — invoked once per operation (singleton and paginated, on success and failure) with the completed `DiagnosticsContext` plus an SDK-supplied `CosmosOperationContext`; the empty default chain is a zero-overhead no-op. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added the `metrics`-gated `CosmosMetricsHandler` (with `MetricsOptions`), emitting the stable `db.client.operation.duration` histogram plus per-signal opt-in metrics (`with_request_charge_metric`, `with_returned_rows_metric`) and an opt-in extended attribute set (`with_extended_attributes`); a no-op when no meter provider is registered. Each histogram declares explicit bucket boundaries (`attributes::BUCKETS_OPERATION_DURATION_SECONDS`, `BUCKETS_REQUEST_CHARGE_RU`, `BUCKETS_RETURNED_ROWS`) rather than inheriting OpenTelemetry's millisecond-scaled defaults, which would collapse every real observation of the seconds-valued duration metric into a single bucket and make latency percentiles constant. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added `MetricsOptions::with_active_instance_metric`, an opt-in `azure.cosmosdb.client.active_instance.count` up-down counter reporting the number of live `CosmosClient` instances per account endpoint, keyed on `server.address` (plus `server.port` for a non-default port). ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) - Added `DiagnosticsHandler::on_client_created`, a defaulted hook that lets a handler observe client construction (`CosmosClientInfo`) and return a `ClientLifetimeToken` dropped with the client, for handlers that need to track client lifetime. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) - Added composable tail-sampled emission handlers — a `TracingLogHandler` leaf that writes a compact `tracing` line and a `SamplingLogHandler` wrapper (holding an `Arc`) that applies the sampling gate plus a shared per-window rate limit, defaulting to wrap a `TracingLogHandler` — and the `distributed_tracing`-gated `CosmosTracingHandler` (backdated span tree), also rate-limited so an error storm can't overwhelm exporters. All emit only for operations which fail or breach a configurable `DiagnosticsThresholds`, and stamp *why* they were sampled (a failure, or which threshold) on the emitted line and span. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) +- Surfaced hedging through the observability handlers, reusing the Hedging Detection API. When a cross-region hedge fans out, `CosmosTracingHandler` adds `azure.cosmosdb.operation.{hedging_started,hedge_region,hedge_terminal_state}` plus `requested_regions`/`responded_regions` (`string[]`, bounded by the driver's `max_request_diagnostics`, with a `*_total` companion attribute emitted only when a retry storm truncated the history) to the sampled operation span and tags the hedge-leg child span (`azure.cosmosdb.request.hedge`); `SamplingLogHandler` adds `hedging_started` / `hedge_region` / `hedge_terminal_state` fields to the sampled log line; and `CosmosMetricsHandler` gains an opt-in `azure.cosmosdb.client.operation.hedged` counter (`MetricsOptions::with_hedged_metric`) carrying the low-cardinality `hedge_terminal_state`, with the higher-cardinality `hedge_region` dimension added only under `with_extended_attributes`. All three surfaces decide fan-out from `hedging_started()`, so a hedge whose race ended both-transient and was then resolved by a failover attempt is still reported; the per-outcome `hedge_region` / `hedge_terminal_state` fields are omitted there (the counter's dimension carries the `unresolved` sentinel so its attribute schema stays uniform). ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Added the non-default `control_plane` feature that gates the control-plane APIs (database and container CRUD, and throughput/offer management). It is intentionally independent of `key_auth` so these APIs are not tied to key-based authentication. ([#4854](https://github.com/Azure/azure-sdk-for-rust/pull/4854)) ### Breaking Changes @@ -26,11 +28,13 @@ - `DatabaseClient::id()` now returns `&ResourceIdentity` instead of `&str`. ([#4687](https://github.com/Azure/azure-sdk-for-rust/pull/4687)) - Control-plane APIs are now gated behind the new `control_plane` feature, which is **not** enabled by default. Code using database or container management (`CosmosClient::create_database`/`query_databases`, `DatabaseClient::read`/`create_container`/`query_containers`/`delete`, `ContainerClient::replace`/`delete`), throughput management (`read_throughput`/`begin_replace_throughput`, `ThroughputPoller`), or the associated model and options types (`DatabaseProperties`, `ThroughputProperties`, and the container create/replace/delete/query, database, and throughput option types) must now enable the `control_plane` feature. Reading container properties via `ContainerClient::read()` — along with `ContainerProperties`, `IndexingPolicy`, `ResourceResponse`, and `ReadContainerOptions` — remains available without the feature, since it works with Entra ID authentication and mirrors the metadata read the SDK already performs internally. ([#4854](https://github.com/Azure/azure-sdk-for-rust/pull/4854)) - Fresh cross-partition queries and change feed reads now fail with a `BadRequest` error if they would fan out to more than 100 physical partitions. Raise `FeedOptions::max_fan_out` to run a broader operation. The limit is checked only at initial query setup — resuming from a continuation token is unaffected, and a partition split that raises the fan-out mid-execution does not abort the operation. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855)) +- Serialized `DiagnosticsContext` output (the diagnostics JSON surfaced to consumers, and the sampled diagnostics log line) now serializes driver-generated operation retries with `execution_context` = `"operation_retry"` instead of `"retry"`. The driver's `ExecutionContext::Retry` variant has been removed entirely; use `ExecutionContext::OperationRetry` instead. Telemetry/log parsers that match the literal `"retry"` execution context must update. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) ### Bugs Fixed - `DatabaseClient::read_throughput` and `begin_replace_throughput` now reject a non-database RID (for example a container RID) with `CLIENT_INVALID_RESOURCE_ID` instead of silently reading or replacing that resource's throughput offer. Throughput offers are keyed only by `offerResourceId`, so a `DatabaseClient` addressed by a container RID would otherwise operate on the container's offer. ([#4640](https://github.com/Azure/azure-sdk-for-rust/pull/4640)) - The Cosmos tracing span's operation label now prefers the caller-facing `CosmosOperationContext` identity over the driver-recorded name, matching how the `db.operation.name` metric attribute is resolved. Previously an aggregate whose surfaced sub-operation differed from the caller's operation — such as a PATCH that fails during its internal read — could label the span `read_item` while the metric reported `patch_item`. Attempt spans now carry the operation that issued them, so a PATCH's attempts report `db.operation.name` of `patch_read_item` / `patch_replace_item` while its operation span and metric stay `patch_item`; attempts of every other operation continue to inherit the operation's own name. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) +- Fixed hedged operations under-reporting diagnostics for the leg that lost the race. Cancelling the loser previously discarded every attempt it had already completed, so a leg that received a `429` and was retrying when it lost contributed nothing to the surfaced `DiagnosticsContext` — omitting its region from `regions_contacted()`/`responded_regions()` and under-reporting `total_request_charge()` against RU the account was actually billed. Attempts observed by a cancelled leg are now retained. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) ### Other Changes diff --git a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md new file mode 100644 index 0000000000..2efdd87712 --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md @@ -0,0 +1,307 @@ +# Hedging Detection API — Spec + +**Status:** Implemented on `main`. +**Cross-SDK contract:** The Azure Cosmos DB SDKs are converging on a "Hedging +Detection" capability exposed on each SDK's per-operation diagnostics surface. +This document specifies how the Rust SDK satisfies that contract. + +--- + +## 1. Summary + +The Rust Cosmos SDK surfaces hedging and per-region dispatch/response history +from a single [`DiagnosticsContext`]. This capability is realized by three +inherent accessors on the driver's `DiagnosticsContext` plus two small public +value types, all re-exported from `azure_data_cosmos`: + +| Member | Signature | Semantics | +| --- | --- | --- | +| `DiagnosticsContext::hedging_started` | `-> bool` | `true` iff a hedge arm was actually dispatched (fan-out happened). | +| `DiagnosticsContext::requested_regions` | `-> Vec` | Regions dispatched to, in **dispatch order**, duplicates allowed, each tagged with a reason. Bounded (§4.5). | +| `DiagnosticsContext::responded_regions` | `-> Vec<&Region>` | Regions that produced a **service reply**, in **completion order**, duplicates allowed. Bounded (§4.5). | +| `DiagnosticsContext::total_requested_regions` | `-> usize` | Exact dispatch count, including entries elided by the bound. | +| `DiagnosticsContext::total_responded_regions` | `-> usize` | Exact reply count, including entries elided by the bound. | +| `RequestedRegion` | `{ region, reason }` | A dispatched region paired with the reason it was chosen. | +| `RequestedRegionReason` | enum | Why the SDK dispatched to a region; `#[non_exhaustive]`. | + +These build on the existing per-operation building blocks and coexist with the +Rust-native `HedgeDiagnostics` surface (see §5); the two are complementary. + +--- + +## 2. Building blocks (already public on `main`) + +| Item | Signature | Notes | +| --- | --- | --- | +| `DiagnosticsContext` | re-exported as `azure_data_cosmos::DiagnosticsContext` | The per-operation diagnostics handle. | +| `DiagnosticsContext::requests` | `-> Arc>` | Retained per-attempt records in dispatch order — **not** a guaranteed-complete append-only history: under a `429`/`410` retry storm the list is bounded/compacted (see `max_request_diagnostics`, which can drop or reorder entries). A structurally-dropped hedge loser leg *is* represented for every attempt it completed (rescued through the hedge journal, §4.3), but an attempt it left in flight is absent. Cloning the `Arc` is a cheap atomic increment. | +| `DiagnosticsContext::hedge_diagnostics` | `-> Option<&HedgeDiagnostics>` | An **optional retained race outcome**, not a configuration probe. `Some` when a hedge race recorded a terminal outcome (including primary-wins-under-threshold). `None` when hedging was not selected for the operation, when a configured strategy found the operation ineligible, **and** on the both-transient→failover path, where a terminal outcome is deliberately left unset so a later successful retry does not carry a misleading `BothTransient` state. | +| `DiagnosticsContext::regions_contacted` | `-> Vec` | Distinct regions **deduplicated in first-contact (failover) order — not sorted**, captured from the full attempt list before compaction. | +| `RequestDiagnostics::region` | `-> Option<&Region>` | `None` for pre-region-selection failures. | +| `RequestDiagnostics::execution_context` | `-> ExecutionContext` | Why this attempt was dispatched (see §3). | +| `RequestDiagnostics::completed_at` | `-> Option` | Set by `complete()`, `timeout()`, **and** `fail_transport()` — so "completed" alone is not "responded" (see §4.3). | +| `RequestDiagnostics::timed_out` | `-> bool` | `true` for a client-side end-to-end timeout. | +| `RequestDiagnostics::error` | `-> Option<&str>` | `Some` for a transport-level failure with no service reply. | +| `HedgeDiagnostics::primary_region` | `-> &Region` | The primary leg's region (unknown-region sentinel for global-endpoint accounts). | +| `HedgeDiagnostics::alternate_region` | `-> Option<&Region>` | `Some` exactly when the orchestrator dispatched an alternate hedge leg (fan-out happened). | +| `HedgeDiagnostics::response_region` | `-> Option<&Region>` | The single winning region, when a leg produced a final response. | +| `HedgeDiagnostics::terminal_state` | `-> HedgeTerminalState` | Authoritative race outcome. | + +Because these are inherent methods on the driver's `DiagnosticsContext` and the +SDK depends on the driver (never the reverse), the diagnostics model is +driver-owned and re-exported by `azure_data_cosmos`, exactly like +`DiagnosticsContext` itself. + +The hedging orchestrator/dispatch is **landed** on `main`: it emits +`ExecutionContext::Hedging` for alternate legs and populates `HedgeDiagnostics` +(design: [`HEDGING_SPEC.md`](https://github.com/Azure/azure-sdk-for-rust/blob/main/sdk/cosmos/azure_data_cosmos_driver/docs/HEDGING_SPEC.md)). + +--- + +## 3. The `Retry → OperationRetry` rename + +`ExecutionContext` is the per-request "why" returned by +`RequestDiagnostics::execution_context()`: + +```rust +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize)] +#[serde(rename_all = "snake_case")] +#[non_exhaustive] +pub enum ExecutionContext { + Initial, + OperationRetry, // was: Retry + TransportRetry, + Hedging, + RegionFailover, + CircuitBreakerProbe, +} +``` + +`Retry` has been removed and replaced by `OperationRetry` so the operation-level +retry reason is clearly distinct from the transport-level `TransportRetry`. The +hand-written `ExecutionContext::as_str()` and every dispatch site +(`operation_pipeline.rs`, `transport_pipeline.rs`, `cosmos_driver.rs`) are updated +accordingly. Telemetry parsers that matched the literal `"retry"` execution context +must update; serialized output now emits `"operation_retry"` instead. + +--- + +## 4. Detection recipes → API + +### 4.1 Reason mapping — `RequestedRegionReason` + +`requested_regions()` tags each dispatched region with a `RequestedRegionReason`, +projected from the driver-internal `ExecutionContext` via a **total** +`From` mapping: + +| `ExecutionContext` | `RequestedRegionReason` | +| --- | --- | +| `Initial` | `Initial` | +| `OperationRetry` | `OperationRetry` | +| `TransportRetry` | `TransportRetry` | +| `Hedging` | `Hedging` | +| `RegionFailover` | `RegionFailover` | +| `CircuitBreakerProbe` | `CircuitBreakerProbe` | + +The mapping is total (no wildcard arm) so it fails to compile if a new +`ExecutionContext` variant is added without a corresponding reason. + +### 4.2 Did fan-out happen? — `hedging_started()` + +`true` iff at least one hedge arm was actually dispatched. This is `false` — not +an error — when the primary returns before the hedging threshold elapses, even +though a hedging strategy was active. + +There is **no** accessor for "was a strategy configured?", and +`hedge_diagnostics().is_some()` is not one: it is `None` for a configured but +ineligible operation, and on the both-transient→failover path (§5). Reading it +as a configuration probe would make consumers conclude hedging was disabled when +it was not. + +Like the two region accessors, this is **materialized at finalization** from the +dispatch-time fan-out log (§4.3), so a race whose attempts were later compacted +away — or whose representative `HedgeDiagnostics` was dropped during +sub-operation aggregation — still reports `true`. + +### 4.3 Regions dispatched to, with reason — `requested_regions()` + +The complete dispatch history in dispatch order, duplicates preserved (a region +dispatched twice appears twice), entries with no resolved region skipped. The +initial attempt is included and tagged `RequestedRegionReason::Initial`. + +**Materialized, not derived.** All three accessors are computed once in +`DiagnosticsContextBuilder::complete()` from the **full, pre-compaction** attempt +list plus a dispatch-time hedge fan-out log, then stored as fields — the same +pattern `regions_contacted()` already used. Reading them is a field read. This +matters because the retained `requests()` list is *not* the dispatch history: + +- a clean hedge race structurally drops the losing leg's sub-builder, so only the + attempts rescued through the hedge journal survive, and any attempt the loser + left in flight is genuinely gone (see §5 and `HedgeDiagnostics`); +- a `429`/`410` retry storm compacts `requests()` down to + `max_request_diagnostics`, dropping whole buckets; +- `aggregate_sub_operations` keeps only **one** representative + `HedgeDiagnostics` for a multi-round-trip operation. + +**Hedge journal.** A hedge leg records into its own private sub-builder, and the +race structurally drops the loser's future — so the loser's records must be +rescued out-of-band. Every attempt a leg *completes* is mirrored, at the moment +it reaches a terminal state, into an operation-scoped hedge journal shared by the +parent and all legs; the winner's copies are discarded when its sub-builder is +merged, and `complete()` folds the remaining copies back in and stable-sorts the +union by dispatch instant. The result is exactly one record per attempt in true +global dispatch order, no matter which leg won. Consequently a dropped leg that +had already received a reply (a `429` it was backing off from, say) still +contributes its region, its status and its RU charge to `requested_regions()`, +`responded_regions()` and the charge totals. + +An attempt that was still **in flight** when its leg was cancelled observed no +reply, so it is deliberately *not* recovered — reporting it would invent a +response that never arrived. + +**Hedge fan-out.** Each fan-out is additionally recorded on the *parent* builder +at dispatch time, before the race runs. This is a fallback for the one case the +journal cannot cover: `select` polls the primary first, so a primary that is +already `Ready` causes the alternate to be dropped **without ever being polled**, +meaning it never reaches `start_request` and has no attempt of its own to +mirror. Such a leg is spliced in as a synthetic entry, positioned by its dispatch +instant, so both legs always appear: the primary leg tagged with the reason it +was **actually** dispatched under (`Initial` for a first attempt, or the +failover/session reason when a hedge upgraded a retry), and the alternate leg +tagged `Hedging`. A leg that did dispatch describes itself through its own +(surviving) attempts and is never double-counted. A leg that never produced a +service reply has no `responded_regions()` entry. + +For an aggregated operation (e.g. `PATCH`) stitched from multiple +sub-operations, every sub-operation's fan-out is preserved: the aggregated list +is the concatenation of each sub-operation's own materialized list, in +sub-operation order. + +This is distinct from `regions_contacted()`, which is *deduplicated* in +first-contact order and so answers "which distinct regions did we touch?" rather +than "what did we dispatch, in what order, and why?". + +### 4.4 Regions that responded — `responded_regions()` + +A region "responded" only if a service reply actually arrived. `completed_at` is +**not** a sufficient filter: the driver also sets it for client-side timeouts +(`timeout()`) and transport failures (`fail_transport()`). The internal +`RequestDiagnostics::responded_with_service_reply()` predicate excludes those two +cases: + +```rust +self.region.is_some() + && self.completed_at.is_some() + && !self.timed_out + && self.error.is_none() +``` + +A non-2xx HTTP status (404/429/503 from the service) still counts as a response. +Results are in arrival order (stable sort by `completed_at`, preserving dispatch +order among ties); duplicates are preserved. To deduplicate, collect into a +`BTreeSet`. Like `requested_regions()`, the list is materialized from the full +pre-compaction attempt list, so a response whose attempt was later compacted +away is still reported. + +### 4.5 Both histories are bounded + +Materializing from the *pre-compaction* attempt list is what makes a dropped +hedge leg survive, but taken alone it would make the two histories grow with +attempt count — a 410/429 retry storm would produce an O(attempts) artifact even +though the retained `requests()` list stays capped. That violates the bounded-size +guarantee in the driver's `DIAGNOSTICS-CONTRACT.md` §8, which requires every +materialized representation to have an upper bound independent of attempt count, +and it would flow straight into span attributes. + +Both histories are therefore capped at `max_request_diagnostics` (default 512, +minimum 16) at finalization, and re-capped in `aggregate_sub_operations` — the +aggregate is an independent unbounded path, since a PATCH conflict loop adds a +sub-operation per retry. + +The elision keeps the **head and tail** of the history and drops the repetitive +middle, mirroring the "first and last of each run" policy the contract already +applies to attempt compaction. The head preserves the initial dispatch and any +early hedge fan-out; the tail preserves where the operation finally landed. + +Because the cap is applied both per sub-operation and again to the concatenated +aggregate, a long enough PATCH conflict loop is bounded **twice** — the aggregate +keeps the head and tail of a list whose own entries are already head/tail +extracts. This compounding is deliberate: it costs some middle detail that was +already elided once, and in exchange it preserves the two properties consumers +actually assert on — the operation's first dispatch and where it finally landed — +under a hard bound that holds no matter how many sub-operations run. Raising the +cap would not remove the compounding, only move the threshold at which it starts, +while making the worst-case artifact proportionally larger. + +Truncation is never silent. `total_requested_regions()` / `total_responded_regions()` +report the exact pre-truncation counts, so `requested_regions().len() < total_requested_regions()` +detects an elision. On the span, the matching `*_total` attribute is emitted only +when the history was truncated, so the normal path carries no redundant integer. + +--- + +## 5. Reconciliation with `HedgeDiagnostics` + +The Hedging Detection API and the Rust-native `HedgeDiagnostics` +([PR #4330](https://github.com/Azure/azure-sdk-for-rust/pull/4330) design, +[#4432](https://github.com/Azure/azure-sdk-for-rust/pull/4432) implementation) +coexist on the same `DiagnosticsContext` and serve different audiences. + +| Question | Hedging Detection API | Rust-native `HedgeDiagnostics` | +| --- | --- | --- | +| Did fan-out happen? | `hedging_started()` — from the fan-out log | `alternate_region().is_some()` — equivalent only when a terminal outcome was retained | +| Was a strategy configured? | *(not derived)* | *(not derived — see below)* | +| Regions tried | `requested_regions()` (every dispatch, with reason) | `primary_region()` + `alternate_region()` (one race's legs only) | +| Regions that responded | `responded_regions()` (full list, completion order) | `response_region()` (single winner) | +| Race outcome | *(not derived)* | `terminal_state()` (authoritative, when retained) | + +Neither surface answers "was hedging configured?". `hedge_diagnostics()` is an +optional *retained race outcome*: it is `None` for a configured-but-ineligible +operation, and it is deliberately left unset when a both-transient race +continues to a successful failover, so that a later successful retry does not +carry a misleading `BothTransient` state. On that path a dispatched hedge leg +still makes `hedging_started()` `true` while `hedge_diagnostics()` is `None` — +the two are not interchangeable. + +The Detection API deliberately does **not** read `hedge_diagnostics` to answer +its three questions. Aggregation keeps only one representative `HedgeDiagnostics` +for a multi-round-trip operation, so deriving from it would silently under-report +every other sub-operation's fan-out — and the both-transient path would +under-report fan-out entirely. The fan-out log is per-builder, is written at +dispatch time, and survives aggregation intact. + +`main`'s `HedgeDiagnostics` classifies the race via `terminal_state` / +`alternate_region` (there is no `total_requests_launched` counter), so "fan-out +happened" is `alternate_region().is_some()` and "the alternate won" is +`matches!(terminal_state(), HedgeTerminalState::AlternateWon)`. Consult +`terminal_state()` for hedge win-rate; do not infer an alternate win from the +presence of `alternate_region()` alone (several terminal states still record an +alternate region). + +### 5.1 Consequence for the observability surfaces + +All three SDK emission surfaces — the sampled root span, the sampled log line, +and the opt-in `azure.cosmosdb.client.operation.hedged` counter — decide +"did this operation hedge?" from `hedging_started()`, never from +`hedge_diagnostics().is_some()`. Gating on the latter would silently undercount +every both-transient race that was subsequently resolved by a failover attempt, +which is exactly the population an operator most wants to see. + +The per-outcome fields (`hedge_region`, `hedge_terminal_state`) still require +`hedge_diagnostics`, and are simply omitted when it is absent rather than +emitted as empty strings. The counter is the one exception: its +`hedge_terminal_state` dimension carries the `unresolved` sentinel instead of +being dropped, so every data point on the counter has the same attribute set +and `group by hedge_terminal_state` never fragments the time series. + +--- + +## 6. Future work + +The three accessors return owned/borrowed collections cloned from fields +materialized once at finalization, so a read never re-walks the attempt list. +If `ExecutionContext` becomes a prominent part of the public detection surface +it could be renamed to something friendlier (e.g., `RequestPurpose` / +`RequestIntent`); that rename is out of scope here. + +[`DiagnosticsContext`]: https://docs.rs/azure_data_cosmos/latest/azure_data_cosmos/ diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/attributes.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/attributes.rs index 0e54f25785..a188539381 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/attributes.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/attributes.rs @@ -58,6 +58,43 @@ pub(crate) const CONNECTION_MODE: &str = "azure.cosmosdb.connection.mode"; /// `azure.cosmosdb.operation.contacted_regions` — regions contacted (ordered `string[]`). pub(crate) const CONTACTED_REGIONS: &str = "azure.cosmosdb.operation.contacted_regions"; +/// `azure.cosmosdb.operation.hedging_started` — `true` when the operation +/// dispatched at least one cross-region hedge (fan-out occurred). +pub(crate) const HEDGING_STARTED: &str = "azure.cosmosdb.operation.hedging_started"; + +/// `azure.cosmosdb.operation.hedge_region` — the alternate region the hedge was +/// dispatched to, when a hedge fan-out occurred. +pub(crate) const HEDGE_REGION: &str = "azure.cosmosdb.operation.hedge_region"; + +/// `azure.cosmosdb.operation.hedge_terminal_state` — how the hedging race ended +/// (see `HedgeTerminalState::as_str`). +pub(crate) const HEDGE_TERMINAL_STATE: &str = "azure.cosmosdb.operation.hedge_terminal_state"; + +/// `azure.cosmosdb.operation.requested_regions` — regions dispatched to, in +/// dispatch order (`string[]`). High-signal for hedge fan-out. +/// +/// Bounded by `max_request_diagnostics`; see [`REQUESTED_REGIONS_TOTAL`]. +pub(crate) const REQUESTED_REGIONS: &str = "azure.cosmosdb.operation.requested_regions"; + +/// `azure.cosmosdb.operation.requested_regions_total` — exact dispatch count, +/// emitted only when [`REQUESTED_REGIONS`] was truncated under a retry storm so +/// the elision is explicit rather than silent. +pub(crate) const REQUESTED_REGIONS_TOTAL: &str = "azure.cosmosdb.operation.requested_regions_total"; + +/// `azure.cosmosdb.operation.responded_regions` — regions that returned a +/// service reply, in arrival order (`string[]`). +/// +/// Bounded by `max_request_diagnostics`; see [`RESPONDED_REGIONS_TOTAL`]. +pub(crate) const RESPONDED_REGIONS: &str = "azure.cosmosdb.operation.responded_regions"; + +/// `azure.cosmosdb.operation.responded_regions_total` — exact reply count, +/// emitted only when [`RESPONDED_REGIONS`] was truncated under a retry storm. +pub(crate) const RESPONDED_REGIONS_TOTAL: &str = "azure.cosmosdb.operation.responded_regions_total"; + +/// `azure.cosmosdb.request.hedge` — `true` on the per-attempt (child) span for a +/// speculative hedge leg dispatched to an alternate region. +pub(crate) const HEDGE_LEG: &str = "azure.cosmosdb.request.hedge"; + /// `azure.cosmosdb.response.sub_status_code` — the Cosmos sub-status code. pub(crate) const SUB_STATUS_CODE: &str = "azure.cosmosdb.response.sub_status_code"; diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs index 101c48e26f..32b8970346 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs @@ -77,7 +77,41 @@ impl DiagnosticsHandler for TracingLogHandler { // only evaluated when a subscriber is actually listening for the event // (the `tracing` macros run an "is enabled" check before evaluating // field expressions). - if diagnostics.is_failure() { + // + // When a cross-region hedge fanned out, surface the hedging signal as + // dedicated fields (in addition to the JSON blob) — high-signal for + // exactly the failed / threshold-breaching operations this handler emits. + // + // Fan-out is decided by `hedging_started()`, consistently with the hedged + // metric counter and the root span. The per-outcome `hedge_region` / + // `hedge_terminal_state` fields additionally require `hedge_diagnostics` + // (with a fanned-out alternate): a both-transient hedge that is later + // resolved by a failover attempt has no recorded terminal outcome + // (`finalize_both_transient` deliberately does not stamp one on the + // non-terminal path). `tracing` field sets are fixed per call site, so + // that case gets its own arm carrying just `hedging_started` rather than + // emitting misleading empty strings — or, worse, dropping the signal. + let hedge = diagnostics + .hedge_diagnostics() + .filter(|hedge| hedge.alternate_region().is_some()); + if let Some(hedge) = hedge { + let hedge_region = hedge + .alternate_region() + .map(|region| region.as_str()) + .unwrap_or_default(); + let hedge_terminal_state = hedge.terminal_state().as_str(); + if diagnostics.is_failure() { + tracing::warn!(target: SAMPLED_TARGET, reason, hedging_started = true, hedge_region, hedge_terminal_state, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); + } else { + tracing::info!(target: SAMPLED_TARGET, reason, hedging_started = true, hedge_region, hedge_terminal_state, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); + } + } else if diagnostics.hedging_started() { + if diagnostics.is_failure() { + tracing::warn!(target: SAMPLED_TARGET, reason, hedging_started = true, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); + } else { + tracing::info!(target: SAMPLED_TARGET, reason, hedging_started = true, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); + } + } else if diagnostics.is_failure() { tracing::warn!(target: SAMPLED_TARGET, reason, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); } else { tracing::info!(target: SAMPLED_TARGET, reason, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs index 98908f7867..55b052b071 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs @@ -20,12 +20,15 @@ pub use handler::{SamplingLogHandler, TracingLogHandler}; #[cfg(test)] mod tests { + use std::collections::HashMap; use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::Arc; use std::time::{Duration, Instant}; use azure_core::http::{Context, StatusCode}; - use azure_data_cosmos_driver::diagnostics::{DiagnosticsContext, RequestDiagnostics}; + use azure_data_cosmos_driver::diagnostics::{ + DiagnosticsContext, HedgeDiagnostics, HedgeTerminalState, RequestDiagnostics, + }; use azure_data_cosmos_driver::models::{ActivityId, RequestCharge}; use azure_data_cosmos_driver::options::Region; use azure_data_cosmos_driver::{CosmosStatus, DiagnosticsThresholds}; @@ -275,4 +278,164 @@ mod tests { assert_eq!(captured.lock().unwrap().as_deref(), Some("request_charge")); }); } + + /// Captures every field of the most recent sampled log event into a map, so + /// tests can assert on the presence and value of the hedging fields. + struct FieldCapture(Arc>>); + + impl Layer for FieldCapture { + fn on_event(&self, event: &tracing::Event<'_>, _cx: LayerContext<'_, S>) { + if event.metadata().target() != "azure_data_cosmos::diagnostics::sampled" { + return; + } + struct Visitor<'a>(&'a mut HashMap); + impl tracing::field::Visit for Visitor<'_> { + fn record_bool(&mut self, field: &tracing::field::Field, value: bool) { + self.0.insert(field.name().to_string(), value.to_string()); + } + fn record_str(&mut self, field: &tracing::field::Field, value: &str) { + self.0.insert(field.name().to_string(), value.to_string()); + } + fn record_debug( + &mut self, + field: &tracing::field::Field, + value: &dyn std::fmt::Debug, + ) { + self.0 + .entry(field.name().to_string()) + .or_insert_with(|| format!("{value:?}")); + } + } + let mut map = self.0.lock().unwrap(); + event.record(&mut Visitor(&mut map)); + } + } + + /// A failed operation that fanned out an `AlternateWon` hedge to West US 2. + fn hedged_context() -> DiagnosticsContext { + let now = Instant::now(); + let primary = RequestDiagnostics::for_testing( + "https://acct-eastus.documents.azure.com:443/", + Some(Region::EAST_US), + CosmosStatus::new(StatusCode::TooManyRequests), + RequestCharge::new(2.0), + now - Duration::from_millis(20), + now, + ); + let hedge = HedgeDiagnostics::for_testing( + Region::EAST_US, + Some(Region::WEST_US_2), + Some(Region::WEST_US_2), + HedgeTerminalState::AlternateWon, + ); + DiagnosticsContext::for_testing_with_hedge( + ActivityId::new_uuid(), + Duration::from_millis(20), + Some(CosmosStatus::new(StatusCode::TooManyRequests)), + Some("read_item"), + vec![primary], + Some(hedge), + ) + } + + #[test] + fn sampled_line_carries_hedging_fields_when_hedging_occurred() { + let captured = Arc::new(std::sync::Mutex::new(HashMap::new())); + let layer = FieldCapture(Arc::clone(&captured)); + let subscriber = tracing_subscriber::registry().with(layer); + let handler = SamplingLogHandler::new(); + + tracing::subscriber::with_default(subscriber, || { + handler.handle(&hedged_context(), &Context::new()); + }); + + let map = captured.lock().unwrap(); + assert_eq!(map.get("hedging_started").map(String::as_str), Some("true")); + assert_eq!(map.get("hedge_region").map(String::as_str), Some("westus2")); + assert_eq!( + map.get("hedge_terminal_state").map(String::as_str), + Some("alternate_won") + ); + } + + #[test] + fn sampled_line_omits_hedging_fields_without_hedging() { + let captured = Arc::new(std::sync::Mutex::new(HashMap::new())); + let layer = FieldCapture(Arc::clone(&captured)); + let subscriber = tracing_subscriber::registry().with(layer); + let handler = SamplingLogHandler::new(); + + // A plain failure — no hedge occurred. + let failed = context( + Duration::from_millis(5), + CosmosStatus::new(StatusCode::TooManyRequests), + 2.0, + ); + tracing::subscriber::with_default(subscriber, || { + handler.handle(&failed, &Context::new()); + }); + + let map = captured.lock().unwrap(); + // The line was emitted (reason present) but carries no hedging fields. + assert!(map.contains_key("reason")); + assert!(!map.contains_key("hedging_started")); + assert!(!map.contains_key("hedge_region")); + assert!(!map.contains_key("hedge_terminal_state")); + } + + #[test] + fn sampled_line_reports_fanout_without_terminal_outcome() { + // A hedge fanned out both-transient and was then resolved by a later + // failover attempt: the fan-out log makes `hedging_started()` true, but + // the pipeline deliberately leaves `hedge_diagnostics()` None (no + // terminal outcome). The line must still report `hedging_started` — the + // operation really did hedge — while omitting the per-outcome + // hedge_region / hedge_terminal_state fields rather than emitting empty + // strings for them. + use azure_data_cosmos_driver::diagnostics::ExecutionContext; + + let captured = Arc::new(std::sync::Mutex::new(HashMap::new())); + let layer = FieldCapture(Arc::clone(&captured)); + let subscriber = tracing_subscriber::registry().with(layer); + let handler = SamplingLogHandler::new(); + + let now = Instant::now(); + let hedge_leg = RequestDiagnostics::for_testing( + "https://acct-westus2.documents.azure.com:443/", + Some(Region::WEST_US_2), + CosmosStatus::new(StatusCode::TooManyRequests), + RequestCharge::new(2.0), + now - Duration::from_millis(20), + now, + ) + .with_execution_context_for_testing(ExecutionContext::Hedging); + let ctx = DiagnosticsContext::for_testing_with_hedge( + ActivityId::new_uuid(), + Duration::from_millis(20), + Some(CosmosStatus::new(StatusCode::TooManyRequests)), + Some("read_item"), + vec![hedge_leg], + None, + ); + assert!( + ctx.hedging_started(), + "a retained Hedging request makes hedging_started() true" + ); + + tracing::subscriber::with_default(subscriber, || { + handler.handle(&ctx, &Context::new()); + }); + + let map = captured.lock().unwrap(); + // The line is emitted (it is a failure) and reports the fan-out, but + // carries no empty-string hedge_region / hedge_terminal_state. + assert!(map.contains_key("reason")); + assert_eq!( + map.get("hedging_started").map(String::as_str), + Some("true"), + "a fanned-out hedge is reported even without a terminal outcome" + ); + assert!(!map.contains_key("hedge_region")); + assert!(!map.contains_key("hedge_terminal_state")); + } } diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs index a220a800b5..665930aa8f 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs @@ -32,6 +32,10 @@ pub const METRIC_OPERATION_REQUEST_CHARGE: &str = "azure.cosmosdb.client.operati /// Optional histogram (rows): number of rows/items returned by an operation. pub const METRIC_RESPONSE_RETURNED_ROWS: &str = "db.client.response.returned_rows"; +/// Optional counter (operations): number of operations that dispatched a +/// cross-region hedge fan-out. +pub const METRIC_OPERATION_HEDGED: &str = "azure.cosmosdb.client.operation.hedged"; + /// Optional up-down counter (instances): number of live /// [`CosmosMetricsHandler`](super::CosmosMetricsHandler) instances (one per /// instrumented client, under the intended one-handler-per-client registration). @@ -54,6 +58,9 @@ pub const UNIT_REQUEST_UNIT: &str = "{request_unit}"; /// Unit for [`METRIC_RESPONSE_RETURNED_ROWS`] — rows. pub const UNIT_ROW: &str = "{row}"; +/// Unit for [`METRIC_OPERATION_HEDGED`] — operations. +pub const UNIT_OPERATION: &str = "{operation}"; + /// Unit for [`METRIC_ACTIVE_INSTANCE_COUNT`] — client instances. pub const UNIT_INSTANCE: &str = "{instance}"; @@ -166,3 +173,12 @@ pub const ATTR_SUB_STATUS_CODE: &str = attributes::SUB_STATUS_CODE; /// `azure.cosmosdb.connection.mode` — gateway vs. direct connection mode. pub const ATTR_CONNECTION_MODE: &str = attributes::CONNECTION_MODE; + +/// `azure.cosmosdb.operation.hedge_terminal_state` — how the hedging race ended. +/// Low cardinality (one value per terminal state), attached to the hedged +/// counter unconditionally. +pub const ATTR_HEDGE_TERMINAL_STATE: &str = attributes::HEDGE_TERMINAL_STATE; + +/// `azure.cosmosdb.operation.hedge_region` — the alternate hedge region. Higher +/// cardinality, so attached only under the extended-attribute opt-in. +pub const ATTR_HEDGE_REGION: &str = attributes::HEDGE_REGION; diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs index ddb776d0f1..89048d85ef 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs @@ -245,6 +245,57 @@ impl CosmosMetricsHandler { } } +impl CosmosMetricsHandler { + /// Value of the `hedge_terminal_state` dimension when a hedge demonstrably + /// fanned out but the race retained no terminal outcome. + /// + /// The both-transient→failover path deliberately leaves `hedge_diagnostics` + /// unset so a later successful retry does not carry a misleading + /// `BothTransient` state. Those operations really did hedge, so they must be + /// counted; this sentinel keeps the counter's attribute schema uniform + /// (every data point carries the dimension, so `group by + /// hedge_terminal_state` never fragments) while staying distinguishable + /// from every real `HedgeTerminalState` value. + const HEDGE_TERMINAL_STATE_UNRESOLVED: &'static str = "unresolved"; + + /// Records the hedged-operation counter for an operation that fanned out a + /// cross-region hedge. + /// + /// The low-cardinality `hedge_terminal_state` dimension is always attached; + /// the higher-cardinality `hedge_region` dimension is added only under the + /// extended-attributes opt-in (mirroring how contacted regions are gated on + /// the duration metric). + /// + /// Fan-out is decided by [`DiagnosticsContext::hedging_started`], which is + /// materialized from the dispatch-time fan-out log and is therefore the + /// authoritative signal — gating on `hedge_diagnostics` instead would + /// silently undercount, because a both-transient race that later succeeds + /// through failover retains no terminal outcome. An aggregated operation + /// (e.g. PATCH) whose sub-op hedged is counted once for the whole operation. + fn record_hedged(&self, diagnostics: &DiagnosticsContext, base_attrs: &[KeyValue]) { + if !diagnostics.hedging_started() { + return; + } + let hedge = diagnostics.hedge_diagnostics(); + let mut attrs = base_attrs.to_vec(); + attrs.push(KeyValue::new( + attributes::ATTR_HEDGE_TERMINAL_STATE, + hedge.map_or(Self::HEDGE_TERMINAL_STATE_UNRESOLVED, |hedge| { + hedge.terminal_state().as_str() + }), + )); + if self.options.extended_attributes_enabled() { + if let Some(alternate) = hedge.and_then(|hedge| hedge.alternate_region()) { + attrs.push(KeyValue::new( + attributes::ATTR_HEDGE_REGION, + alternate.as_str().to_string(), + )); + } + } + self.instruments.hedged.add(1, &attrs); + } +} + impl Default for CosmosMetricsHandler { fn default() -> Self { Self::new() @@ -284,6 +335,13 @@ impl DiagnosticsHandler for CosmosMetricsHandler { self.instruments.returned_rows.record(rows, &attributes); } } + + // Hedging counter: emitted only when opted in and a hedge actually + // fanned out. `record_hedged` re-checks fan-out so the invariant holds + // regardless of call site. + if self.options.hedged_metric_enabled() { + self.record_hedged(diagnostics, &attributes); + } } fn on_client_created(&self, client: &CosmosClientInfo) -> Option { @@ -316,7 +374,9 @@ mod tests { use super::*; use crate::CosmosStatus; use azure_core::http::StatusCode; + use azure_data_cosmos_driver::diagnostics::{HedgeDiagnostics, HedgeTerminalState}; use azure_data_cosmos_driver::models::ActivityId; + use azure_data_cosmos_driver::options::Region; use opentelemetry::metrics::MeterProvider as _; use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData, ResourceMetrics}; use opentelemetry_sdk::metrics::{InMemoryMetricExporter, PeriodicReader, SdkMeterProvider}; @@ -360,6 +420,25 @@ mod tests { ) } + /// Completed context whose operation fanned out an `AlternateWon` hedge to + /// West US 2. + fn hedged_completed(status_code: u16) -> DiagnosticsContext { + let hedge = HedgeDiagnostics::for_testing( + Region::EAST_US, + Some(Region::WEST_US_2), + Some(Region::WEST_US_2), + HedgeTerminalState::AlternateWon, + ); + DiagnosticsContext::for_testing_with_hedge( + ActivityId::new_uuid(), + Duration::from_millis(42), + Some(CosmosStatus::new(StatusCode::from(status_code))), + Some("read_item"), + Vec::new(), + Some(hedge), + ) + } + fn operation_context() -> CosmosOperationContext { CosmosOperationContext::new() .with_operation_name("read_item") @@ -793,4 +872,171 @@ mod tests { ); assert_eq!(host_of("not a url"), None); } + + /// Returns the attributes (as a string map) and summed value of the + /// `azure.cosmosdb.client.operation.hedged` counter, if present. + fn hedged_point(metrics: &[ResourceMetrics]) -> Option<(HashMap, u64)> { + for rm in metrics { + for sm in rm.scope_metrics() { + for m in sm.metrics() { + if m.name() != attributes::METRIC_OPERATION_HEDGED { + continue; + } + if let AggregatedMetrics::U64(MetricData::Sum(sum)) = m.data() { + if let Some(point) = sum.data_points().next() { + let attrs = point + .attributes() + .map(|kv| { + (kv.key.as_str().to_string(), kv.value.as_str().into_owned()) + }) + .collect(); + return Some((attrs, point.value())); + } + } + } + } + } + None + } + + #[test] + fn hedged_metric_off_by_default_even_for_hedged_operation() { + let harness = test_meter(); + let handler = CosmosMetricsHandler::with_meter(harness.meter.clone()); + + let cx = Context::new().with_value(operation_context()); + handler.handle(&hedged_completed(200), &cx); + + // The stable duration metric is emitted, but the opt-in hedged counter is not. + let names = metric_names(&harness.collect()); + assert!(names + .iter() + .any(|n| n == attributes::METRIC_OPERATION_DURATION)); + assert!(!names + .iter() + .any(|n| n == attributes::METRIC_OPERATION_HEDGED)); + } + + #[test] + fn hedged_metric_emitted_for_hedged_operation_when_enabled() { + let harness = test_meter(); + let options = MetricsOptions::default().with_hedged_metric(true); + let handler = CosmosMetricsHandler::with_meter_and_options(harness.meter.clone(), options); + + let cx = Context::new().with_value(operation_context()); + handler.handle(&hedged_completed(200), &cx); + + let metrics = harness.collect(); + let (attrs, value) = hedged_point(&metrics).expect("hedged counter should be emitted"); + assert_eq!(value, 1); + + // Operation identity carries through, plus the low-cardinality terminal state. + assert_eq!( + attrs + .get(attributes::ATTR_DB_OPERATION_NAME) + .map(String::as_str), + Some("read_item") + ); + assert_eq!( + attrs + .get(attributes::ATTR_HEDGE_TERMINAL_STATE) + .map(String::as_str), + Some("alternate_won") + ); + // The high-cardinality region dimension stays off without extended attributes. + assert!(!attrs.contains_key(attributes::ATTR_HEDGE_REGION)); + } + + #[test] + fn hedged_metric_adds_region_dimension_under_extended_attributes() { + let harness = test_meter(); + let options = MetricsOptions::default() + .with_hedged_metric(true) + .with_extended_attributes(true); + let handler = CosmosMetricsHandler::with_meter_and_options(harness.meter.clone(), options); + + let cx = Context::new().with_value(operation_context()); + handler.handle(&hedged_completed(200), &cx); + + let metrics = harness.collect(); + let (attrs, _) = hedged_point(&metrics).expect("hedged counter should be emitted"); + assert_eq!( + attrs.get(attributes::ATTR_HEDGE_REGION).map(String::as_str), + Some("westus2") + ); + } + + #[test] + fn hedged_metric_not_emitted_for_non_hedged_operation() { + let harness = test_meter(); + let options = MetricsOptions::default().with_hedged_metric(true); + let handler = CosmosMetricsHandler::with_meter_and_options(harness.meter.clone(), options); + + // Enabled, but this operation never hedged: no counter data point. + let cx = Context::new().with_value(operation_context()); + handler.handle(&completed(200), &cx); + + let metrics = harness.collect(); + assert!( + hedged_point(&metrics).is_none(), + "a non-hedged operation must not increment the hedged counter" + ); + } + + #[test] + fn hedged_metric_counts_hedge_without_terminal_state() { + // A hedge that fanned out both-transient and was then resolved by a later + // failover attempt leaves a retained `Hedging` request (so + // `hedging_started()` is true) but no recorded terminal outcome + // (`finalize_both_transient` deliberately does not stamp `hedge_diagnostics` + // on the non-terminal path — see operation_pipeline.rs). That operation + // really did hedge, so it must be counted; the dimension carries the + // `unresolved` sentinel rather than being omitted, which keeps the + // counter's attribute schema uniform for `group by hedge_terminal_state`. + use azure_data_cosmos_driver::diagnostics::{ExecutionContext, RequestDiagnostics}; + use azure_data_cosmos_driver::models::RequestCharge; + use std::time::Instant; + + let harness = test_meter(); + let options = MetricsOptions::default().with_hedged_metric(true); + let handler = CosmosMetricsHandler::with_meter_and_options(harness.meter.clone(), options); + + let now = Instant::now(); + let hedge_leg = RequestDiagnostics::for_testing( + "https://acct-westus2.documents.azure.com:443/", + Some(Region::WEST_US_2), + CosmosStatus::new(StatusCode::Ok), + RequestCharge::new(1.0), + now - Duration::from_millis(50), + now, + ) + .with_execution_context_for_testing(ExecutionContext::Hedging); + let ctx = DiagnosticsContext::for_testing_with_hedge( + ActivityId::new_uuid(), + Duration::from_millis(42), + Some(CosmosStatus::new(StatusCode::Ok)), + Some("read_item"), + vec![hedge_leg], + None, + ); + assert!( + ctx.hedging_started(), + "a Hedging-tagged request makes hedging_started() true" + ); + + let cx = Context::new().with_value(operation_context()); + handler.handle(&ctx, &cx); + + let metrics = harness.collect(); + let (attrs, value) = hedged_point(&metrics) + .expect("a hedge that fanned out is counted even without a terminal outcome"); + assert_eq!(value, 1); + assert_eq!( + attrs + .get(attributes::ATTR_HEDGE_TERMINAL_STATE) + .map(String::as_str), + Some("unresolved"), + "the dimension is always present so the counter's schema stays uniform" + ); + } } diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs index fc3224c2ad..86923f7f0c 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs @@ -8,7 +8,7 @@ //! reference-counted). Building them eagerly keeps the per-operation hot path to //! just `record`/`add` calls with no allocation of instrument state. -use opentelemetry::metrics::{Histogram, Meter, UpDownCounter}; +use opentelemetry::metrics::{Counter, Histogram, Meter, UpDownCounter}; use crate::diagnostics::metrics::attributes; @@ -18,9 +18,9 @@ use crate::diagnostics::metrics::attributes; /// per-signal instruments are recorded only when the matching /// [`MetricsOptions`](super::MetricsOptions) toggle /// (`request_charge_metric_enabled` / `returned_rows_metric_enabled` / -/// `active_instance_metric_enabled`) is set. -/// They are still created unconditionally because instrument creation is cheap -/// and idempotent, and doing so keeps the handler's record path branch-free per +/// `hedged_metric_enabled` / `active_instance_metric_enabled`) is set. They are +/// still created unconditionally because instrument creation is cheap and +/// idempotent, and doing so keeps the handler's record path branch-free per /// instrument. #[derive(Clone)] pub(crate) struct Instruments { @@ -33,6 +33,10 @@ pub(crate) struct Instruments { /// Development: `db.client.response.returned_rows` (rows). pub(crate) returned_rows: Histogram, + /// Development: `azure.cosmosdb.client.operation.hedged` (operations that + /// dispatched a cross-region hedge fan-out). + pub(crate) hedged: Counter, + /// Development: `azure.cosmosdb.client.active_instance.count` (instances). /// /// An up-down counter incremented when a @@ -66,6 +70,14 @@ impl Instruments { .with_boundaries(attributes::BUCKETS_RETURNED_ROWS.to_vec()) .build(); + let hedged = meter + .u64_counter(attributes::METRIC_OPERATION_HEDGED) + .with_unit(attributes::UNIT_OPERATION) + .with_description( + "Number of Cosmos DB operations that dispatched a cross-region hedge.", + ) + .build(); + let active_instance = meter .i64_up_down_counter(attributes::METRIC_ACTIVE_INSTANCE_COUNT) .with_unit(attributes::UNIT_INSTANCE) @@ -76,6 +88,7 @@ impl Instruments { operation_duration, request_charge, returned_rows, + hedged, active_instance, } } diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/options.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/options.rs index 4bb37221fa..146f9c6de4 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/options.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/options.rs @@ -22,6 +22,7 @@ /// let stable = MetricsOptions::default(); /// assert!(!stable.request_charge_metric_enabled()); /// assert!(!stable.returned_rows_metric_enabled()); +/// assert!(!stable.hedged_metric_enabled()); /// assert!(!stable.extended_attributes_enabled()); /// /// // Opt into just the request-charge metric. @@ -33,10 +34,12 @@ /// let full = MetricsOptions::default() /// .with_request_charge_metric(true) /// .with_returned_rows_metric(true) +/// .with_hedged_metric(true) /// .with_active_instance_metric(true) /// .with_extended_attributes(true); /// assert!(full.request_charge_metric_enabled()); /// assert!(full.returned_rows_metric_enabled()); +/// assert!(full.hedged_metric_enabled()); /// assert!(full.active_instance_metric_enabled()); /// assert!(full.extended_attributes_enabled()); /// ``` @@ -44,6 +47,7 @@ pub struct MetricsOptions { request_charge_metric: bool, returned_rows_metric: bool, + hedged_metric: bool, active_instance_metric: bool, extended_attributes: bool, } @@ -71,6 +75,20 @@ impl MetricsOptions { self } + /// Enables (or disables) the `azure.cosmosdb.client.operation.hedged` counter + /// (operations that dispatched a cross-region hedge fan-out). Off by default. + /// + /// The counter increments only for operations where hedging actually fanned + /// out, so it is near-zero cardinality; it always carries the low-cardinality + /// `hedge_terminal_state` dimension. The higher-cardinality hedge-region + /// dimension is added only when [`with_extended_attributes`](Self::with_extended_attributes) + /// is also enabled. + #[must_use] + pub fn with_hedged_metric(mut self, enabled: bool) -> Self { + self.hedged_metric = enabled; + self + } + /// Enables (or disables) the /// `azure.cosmosdb.client.active_instance.count` up-down counter, which /// tracks the number of live [`CosmosClient`](crate::CosmosClient) @@ -112,6 +130,11 @@ impl MetricsOptions { self.returned_rows_metric } + /// Whether the hedged-operation counter is emitted. + pub fn hedged_metric_enabled(&self) -> bool { + self.hedged_metric + } + /// Whether the active-instance up-down counter is emitted. pub fn active_instance_metric_enabled(&self) -> bool { self.active_instance_metric diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs index fe754d1313..9f46bc8047 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs @@ -52,6 +52,7 @@ pub use handler::{ pub use logging::{SamplingLogHandler, TracingLogHandler}; pub use operation_context::CosmosOperationContext; pub use rate_limiter::RateLimiterConfig; +pub use region::{RequestedRegion, RequestedRegionReason}; #[cfg(feature = "distributed_tracing")] pub use tracing::CosmosTracingHandler; @@ -72,6 +73,7 @@ mod handler; mod logging; mod operation_context; mod reason; +mod region; // Count-per-interval rate limiter shared by the sampling handlers (logging and, // when enabled, tracing) so they can bound emission under an error storm. pub(crate) mod rate_limiter; diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs new file mode 100644 index 0000000000..2af2204d74 --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs @@ -0,0 +1,144 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +//! SDK-owned region-request types for the Hedging Detection API. +//! +//! These types mirror the cross-SDK Hedging Detection API's `RequestedRegion` +//! and `RequestedRegionReason` while remaining wholly owned by +//! `azure_data_cosmos`. They are projected from the driver equivalents at +//! diagnostics-context finalization, which lets the driver evolve its internal +//! model without forcing an SDK major-version bump. + +use crate::options::Region; + +/// The reason the SDK dispatched a request to a particular region. +/// +/// Carried by each [`RequestedRegion`] entry returned from +/// [`DiagnosticsContext::requested_regions`](crate::diagnostics::DiagnosticsContext::requested_regions). +/// +/// The enum is `#[non_exhaustive]`; always include a wildcard arm in `match` +/// expressions. +/// +/// # Example +/// +/// ```rust,no_run +/// # use azure_data_cosmos::diagnostics::{RequestedRegion, RequestedRegionReason}; +/// fn describe(r: &RequestedRegion) -> &'static str { +/// match r.reason { +/// RequestedRegionReason::Initial => "initial dispatch", +/// RequestedRegionReason::OperationRetry => "SDK-level retry", +/// RequestedRegionReason::TransportRetry => "transport-level retry", +/// RequestedRegionReason::Hedging => "speculative hedge", +/// RequestedRegionReason::RegionFailover => "region-failover retry", +/// RequestedRegionReason::CircuitBreakerProbe => "circuit-breaker probe", +/// _ => "unknown", +/// } +/// } +/// ``` +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum RequestedRegionReason { + /// The first dispatch of the operation. + Initial, + /// An operation-level retry decided by the SDK's client-retry policy. + OperationRetry, + /// A transport-level retry inside the per-region transport stack. + TransportRetry, + /// A speculative cross-region hedge fan-out dispatch. + Hedging, + /// An endpoint-failure-driven retry to a different region. + RegionFailover, + /// A probe dispatch to a previously circuit-broken region. + CircuitBreakerProbe, +} + +impl From for RequestedRegionReason { + fn from( + driver: azure_data_cosmos_driver::diagnostics::ExecutionContext, + ) -> RequestedRegionReason { + match driver { + azure_data_cosmos_driver::diagnostics::ExecutionContext::Initial => { + RequestedRegionReason::Initial + } + azure_data_cosmos_driver::diagnostics::ExecutionContext::OperationRetry => { + RequestedRegionReason::OperationRetry + } + azure_data_cosmos_driver::diagnostics::ExecutionContext::TransportRetry => { + RequestedRegionReason::TransportRetry + } + azure_data_cosmos_driver::diagnostics::ExecutionContext::Hedging => { + RequestedRegionReason::Hedging + } + azure_data_cosmos_driver::diagnostics::ExecutionContext::RegionFailover => { + RequestedRegionReason::RegionFailover + } + azure_data_cosmos_driver::diagnostics::ExecutionContext::CircuitBreakerProbe => { + RequestedRegionReason::CircuitBreakerProbe + } + // The driver enum is #[non_exhaustive]; map any future variants to + // the closest known reason rather than panicking. + _ => RequestedRegionReason::Initial, + } + } +} + +/// A single region the SDK dispatched a request to, tagged with the reason the +/// orchestrator chose to send it. +/// +/// Realizes the cross-SDK Hedging Detection API's `RequestedRegion` value type. +/// Returned by +/// [`DiagnosticsContext::requested_regions`](crate::diagnostics::DiagnosticsContext::requested_regions). +/// +/// The struct is `#[non_exhaustive]`; construct via the public fields only in +/// owned contexts, and pattern-match with `..` to remain forward-compatible. +#[derive(Clone, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct RequestedRegion { + /// The region the SDK dispatched to. + pub region: Region, + /// The reason the SDK chose this region for this dispatch attempt. + pub reason: RequestedRegionReason, +} + +impl From for RequestedRegion { + fn from(driver: azure_data_cosmos_driver::diagnostics::RequestedRegion) -> RequestedRegion { + RequestedRegion { + region: driver.region, + reason: driver.reason.into(), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use azure_data_cosmos_driver::diagnostics::ExecutionContext as DriverCtx; + + #[test] + fn reason_from_driver_all_variants() { + assert_eq!( + RequestedRegionReason::from(DriverCtx::Initial), + RequestedRegionReason::Initial + ); + assert_eq!( + RequestedRegionReason::from(DriverCtx::OperationRetry), + RequestedRegionReason::OperationRetry + ); + assert_eq!( + RequestedRegionReason::from(DriverCtx::TransportRetry), + RequestedRegionReason::TransportRetry + ); + assert_eq!( + RequestedRegionReason::from(DriverCtx::Hedging), + RequestedRegionReason::Hedging + ); + assert_eq!( + RequestedRegionReason::from(DriverCtx::RegionFailover), + RequestedRegionReason::RegionFailover + ); + assert_eq!( + RequestedRegionReason::from(DriverCtx::CircuitBreakerProbe), + RequestedRegionReason::CircuitBreakerProbe + ); + } +} diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs index 1e47437f22..5987062ef5 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs @@ -19,11 +19,15 @@ mod tests { use std::time::{Duration, Instant, SystemTime}; use azure_core::http::StatusCode; - use azure_data_cosmos_driver::diagnostics::{DiagnosticsContext, RequestDiagnostics}; + use azure_data_cosmos_driver::diagnostics::{ + DiagnosticsContext, ExecutionContext, HedgeDiagnostics, HedgeTerminalState, + RequestDiagnostics, + }; use azure_data_cosmos_driver::models::{ActivityId, RequestCharge}; use azure_data_cosmos_driver::options::Region; use azure_data_cosmos_driver::{CosmosStatus, DiagnosticsThresholds}; use opentelemetry::trace::{SpanId, TracerProvider}; + use opentelemetry::{Array, Value}; use opentelemetry_sdk::trace::{in_memory_exporter::InMemorySpanExporter, SdkTracerProvider}; use super::handler::should_emit_span; @@ -492,4 +496,249 @@ mod tests { "root must start no later than its earliest child" ); } + + /// Builds a hedged operation in its **production `AlternateWon` shape**: the + /// primary East US leg lost the race and was structurally dropped (so it has + /// no retained per-request record), leaving only the winning speculative + /// hedge leg to West US 2 (200) tagged `ExecutionContext::Hedging`, with + /// `AlternateWon` hedge diagnostics naming both regions. The dropped primary + /// region is recovered for `requested_regions()` from the hedge diagnostics. + fn hedged_context(anchor: Instant) -> DiagnosticsContext { + let hedge_leg = RequestDiagnostics::for_testing( + "https://acct-westus2.documents.azure.com:443/", + Some(Region::WEST_US_2), + CosmosStatus::new(StatusCode::Ok), + RequestCharge::new(2.0), + anchor - Duration::from_millis(150), + anchor - Duration::from_millis(50), + ) + .with_execution_context_for_testing(ExecutionContext::Hedging); + let hedge = HedgeDiagnostics::for_testing( + Region::EAST_US, + Some(Region::WEST_US_2), + Some(Region::WEST_US_2), + HedgeTerminalState::AlternateWon, + ); + DiagnosticsContext::for_testing_with_hedge( + ActivityId::new_uuid(), + Duration::from_millis(250), + Some(CosmosStatus::new(StatusCode::Ok)), + Some("read_item"), + vec![hedge_leg], + Some(hedge), + ) + } + + /// Returns the string members of a `string[]` span attribute, if present. + fn string_array_attr( + span: &opentelemetry_sdk::trace::SpanData, + key: &str, + ) -> Option> { + span.attributes.iter().find_map(|kv| { + if kv.key.as_str() != key { + return None; + } + match &kv.value { + Value::Array(Array::String(values)) => { + Some(values.iter().map(|v| v.as_str().to_string()).collect()) + } + _ => None, + } + }) + } + + #[test] + fn hedged_operation_surfaces_hedging_span_attributes() { + let (provider, exporter) = exportable(); + let tracer = provider.tracer("test"); + let now_instant = Instant::now(); + let now_system = SystemTime::now(); + + let ctx = hedged_context(now_instant); + emit_backdated_span_tree(&tracer, &ctx, None, None, now_instant, now_system); + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + let root = spans + .iter() + .find(|s| s.name == "read_item") + .expect("root span present"); + + // hedging_started = true (bool). + assert!( + root.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGING_STARTED + && matches!(kv.value, Value::Bool(true)) + }), + "root span must carry hedging_started = true" + ); + // hedge region = the alternate region the hedge fanned out to. + assert!(root.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGE_REGION && kv.value.as_str() == "westus2" + })); + // terminal state = alternate_won. + assert!(root.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGE_TERMINAL_STATE + && kv.value.as_str() == "alternate_won" + })); + // requested_regions carries both dispatched regions — the winning hedge + // leg (westus2) plus the structurally-dropped primary (eastus) recovered + // from the hedge diagnostics. + let requested = string_array_attr(root, attributes::REQUESTED_REGIONS) + .expect("requested_regions string[] present"); + assert!(requested.iter().any(|r| r == "eastus")); + assert!(requested.iter().any(|r| r == "westus2")); + // responded_regions carries only the winning hedge region: the dropped + // primary leg never produced a service reply. + let responded = string_array_attr(root, attributes::RESPONDED_REGIONS) + .expect("responded_regions string[] present"); + assert!(responded.iter().any(|r| r == "westus2")); + assert!( + !responded.iter().any(|r| r == "eastus"), + "the structurally-dropped primary leg must not appear in responded_regions" + ); + // Nothing was truncated, so the exact-count attributes stay off the span + // entirely — they exist only to make an elision explicit, and emitting + // them unconditionally would put a redundant integer on every hedged + // span. + for key in [ + attributes::REQUESTED_REGIONS_TOTAL, + attributes::RESPONDED_REGIONS_TOTAL, + ] { + assert!( + !root.attributes.iter().any(|kv| kv.key.as_str() == key), + "{key} must be absent when the region history was not truncated" + ); + } + + // The speculative hedge leg's child span is tagged. + let tagged = spans + .iter() + .filter(|s| s.name == "cosmosdb.request") + .any(|s| { + s.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGE_LEG + && matches!(kv.value, Value::Bool(true)) + }) + }); + assert!(tagged, "the hedge leg child span must carry the hedge tag"); + } + + /// Builds a hedged operation in its **production `PrimaryWonAfterHedge` + /// shape**: the primary East US leg won after the threshold, so the + /// speculative West US 2 hedge leg was structurally cancelled and has no + /// retained record. Only the primary (Initial, 200) survives; the hedge + /// region is recovered from the hedge diagnostics. + fn primary_won_after_hedge_context(anchor: Instant) -> DiagnosticsContext { + let primary = RequestDiagnostics::for_testing( + "https://acct-eastus.documents.azure.com:443/", + Some(Region::EAST_US), + CosmosStatus::new(StatusCode::Ok), + RequestCharge::new(2.0), + anchor - Duration::from_millis(300), + anchor - Duration::from_millis(180), + ); + let hedge = HedgeDiagnostics::for_testing( + Region::EAST_US, + Some(Region::WEST_US_2), + Some(Region::EAST_US), + HedgeTerminalState::PrimaryWonAfterHedge, + ); + DiagnosticsContext::for_testing_with_hedge( + ActivityId::new_uuid(), + Duration::from_millis(200), + Some(CosmosStatus::new(StatusCode::Ok)), + Some("read_item"), + vec![primary], + Some(hedge), + ) + } + + #[test] + fn primary_won_after_hedge_recovers_hedge_region_without_child_span() { + // When the primary wins cleanly the hedge leg is dropped, so there is no + // child span to tag. The authoritative hedge signal must still be on the + // root span, and the dropped hedge region recovered into requested_regions. + let (provider, exporter) = exportable(); + let tracer = provider.tracer("test"); + let now_instant = Instant::now(); + let now_system = SystemTime::now(); + + let ctx = primary_won_after_hedge_context(now_instant); + emit_backdated_span_tree(&tracer, &ctx, None, None, now_instant, now_system); + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + let root = spans + .iter() + .find(|s| s.name == "read_item") + .expect("root span present"); + + assert!(root.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGING_STARTED && matches!(kv.value, Value::Bool(true)) + })); + assert!(root.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGE_REGION && kv.value.as_str() == "westus2" + })); + assert!(root.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGE_TERMINAL_STATE + && kv.value.as_str() == "primary_won_after_hedge" + })); + // The dropped hedge leg's region is recovered into requested_regions... + let requested = string_array_attr(root, attributes::REQUESTED_REGIONS) + .expect("requested_regions string[] present"); + assert!(requested.iter().any(|r| r == "eastus")); + assert!(requested.iter().any(|r| r == "westus2")); + // ...but only the winning primary produced a response. + let responded = string_array_attr(root, attributes::RESPONDED_REGIONS) + .expect("responded_regions string[] present"); + assert_eq!(responded, vec!["eastus".to_string()]); + // The hedge leg was structurally dropped, so no child span carries the + // hedge tag; the root span's hedge attributes carry the signal instead. + let any_hedge_child = spans + .iter() + .filter(|s| s.name == "cosmosdb.request") + .any(|s| { + s.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGE_LEG + && matches!(kv.value, Value::Bool(true)) + }) + }); + assert!( + !any_hedge_child, + "a structurally-dropped hedge leg must not produce a tagged child span" + ); + } + + #[test] + fn non_hedged_operation_omits_hedging_span_attributes() { + let (provider, exporter) = exportable(); + let tracer = provider.tracer("test"); + let now_instant = Instant::now(); + let now_system = SystemTime::now(); + + // A plain failed operation — no hedge diagnostics, all Initial legs. + let ctx = context( + Duration::from_millis(5), + Some(CosmosStatus::new(StatusCode::TooManyRequests)), + Some("read_item"), + &[(5, 5, CosmosStatus::new(StatusCode::TooManyRequests))], + now_instant, + ); + emit_backdated_span_tree(&tracer, &ctx, None, None, now_instant, now_system); + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + for span in &spans { + for kv in span.attributes.iter() { + let key = kv.key.as_str(); + assert_ne!(key, attributes::HEDGING_STARTED); + assert_ne!(key, attributes::HEDGE_REGION); + assert_ne!(key, attributes::HEDGE_TERMINAL_STATE); + assert_ne!(key, attributes::REQUESTED_REGIONS); + assert_ne!(key, attributes::RESPONDED_REGIONS); + assert_ne!(key, attributes::HEDGE_LEG); + } + } + } } diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs index aeac38709d..a6e53b8f00 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs @@ -22,7 +22,9 @@ use opentelemetry::{ Array, Context, KeyValue, StringValue, Value, }; -use azure_data_cosmos_driver::diagnostics::{DiagnosticsContext, RequestDiagnostics}; +use azure_data_cosmos_driver::diagnostics::{ + DiagnosticsContext, ExecutionContext, RequestDiagnostics, +}; use crate::diagnostics::attributes; use crate::diagnostics::CosmosOperationContext; @@ -33,6 +35,19 @@ const DEFAULT_OPERATION_SPAN_NAME: &str = "cosmosdb.operation"; /// Span name for each per-attempt ("child") span. const REQUEST_SPAN_NAME: &str = "cosmosdb.request"; +/// Builds an OpenTelemetry ordered `string[]` [`Value`] from region names. +/// +/// The Cosmos semantic conventions model region lists (contacted, requested, +/// responded) as ordered `string[]`; this emits an array value rather than a +/// joined scalar. +fn region_string_array<'a>(regions: impl IntoIterator) -> Value { + let values: Vec = regions + .into_iter() + .map(|region| StringValue::from(region.to_string())) + .collect(); + Value::Array(Array::String(values)) +} + /// Emits a backdated operation span with one child span per retained attempt. /// /// * `tracer` — the OpenTelemetry tracer to build spans on. Generic so tests can @@ -168,6 +183,67 @@ pub(crate) fn emit_backdated_span_tree( Value::Array(Array::String(values)), )); } + // Hedging surfacing: only when a cross-region hedge actually fanned out. + // These attributes stay off the common (non-hedged) sampled span entirely. + // + // Fan-out is decided by `hedging_started()` (materialized from the + // dispatch-time fan-out log), consistently with the hedged metric counter + // and the sampled log line. The per-outcome `HEDGE_REGION` / + // `HEDGE_TERMINAL_STATE` fields are additionally gated on + // `hedge_diagnostics()`, which is `None` when a both-transient race was + // resolved by a later failover attempt — so those fields are simply absent + // there rather than carrying empty or placeholder values. The region + // history is still emitted on that path, which is exactly where it is most + // useful. + // + // Both region arrays come from the driver already bounded by + // `max_request_diagnostics`, so a retry storm cannot produce an unbounded + // span attribute. When the driver elided the middle of a history, the + // matching `*_total` count is emitted alongside so the truncation is + // explicit in the telemetry rather than silent. + if diagnostics.hedging_started() { + root_attrs.push(KeyValue::new(attributes::HEDGING_STARTED, true)); + if let Some(hedge) = diagnostics.hedge_diagnostics() { + if let Some(alternate) = hedge.alternate_region() { + root_attrs.push(KeyValue::new( + attributes::HEDGE_REGION, + alternate.as_str().to_string(), + )); + } + root_attrs.push(KeyValue::new( + attributes::HEDGE_TERMINAL_STATE, + hedge.terminal_state().as_str(), + )); + } + let requested = diagnostics.requested_regions(); + if !requested.is_empty() { + root_attrs.push(KeyValue::new( + attributes::REQUESTED_REGIONS, + region_string_array(requested.iter().map(|r| r.region.as_str())), + )); + let total = diagnostics.total_requested_regions(); + if total > requested.len() { + root_attrs.push(KeyValue::new( + attributes::REQUESTED_REGIONS_TOTAL, + total as i64, + )); + } + } + let responded = diagnostics.responded_regions(); + if !responded.is_empty() { + root_attrs.push(KeyValue::new( + attributes::RESPONDED_REGIONS, + region_string_array(responded.iter().map(|region| region.as_str())), + )); + let total = diagnostics.total_responded_regions(); + if total > responded.len() { + root_attrs.push(KeyValue::new( + attributes::RESPONDED_REGIONS_TOTAL, + total as i64, + )); + } + } + } // Prefer the caller-supplied server-address override (mirroring the metrics // handler) before falling back to the host of the first contacted endpoint, // so an override changes both the metric and the root span consistently. @@ -263,6 +339,18 @@ pub(crate) fn emit_backdated_span_tree( )])), )); } + // Tag the speculative hedge leg so the child span is attributable to the + // hedge fan-out rather than an initial/retry dispatch. This tag is + // present only for a *retained* hedge-leg record: when the primary wins + // a clean race the alternate leg is structurally cancelled before it + // produces a per-request record, so no child span exists for it. In that + // case the authoritative hedge signal lives on the root span + // (`hedge_started` / `hedge_region` / `hedge_terminal_state`, plus the + // alternate region in `requested_regions`), so the fan-out is still + // attributable even without a tagged child. + if matches!(req.execution_context(), ExecutionContext::Hedging) { + child_attrs.push(KeyValue::new(attributes::HEDGE_LEG, true)); + } if let Some(addr) = server_address(req) { child_attrs.push(KeyValue::new(attributes::SERVER_ADDRESS, addr)); } diff --git a/sdk/cosmos/azure_data_cosmos_driver/ARCHITECTURE.md b/sdk/cosmos/azure_data_cosmos_driver/ARCHITECTURE.md index 5566d2216e..0425784dff 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/ARCHITECTURE.md +++ b/sdk/cosmos/azure_data_cosmos_driver/ARCHITECTURE.md @@ -296,7 +296,7 @@ The `DiagnosticsContext` provides comprehensive visibility into operation execut flowchart TD DC["DiagnosticsContext (Immutable, per-operation)
• activity_id: ActivityId — unique identifier for the operation
• duration: Duration — total operation time
• status_code: StatusCode — final HTTP status after retries
• sub_status_code: SubStatusCode — Cosmos-specific error classification
• requests: Arc<Vec<RequestDiagnostics>>"] RD["RequestDiagnostics (per-HTTP-request details)
• region: Region
• endpoint: String
• status_code: StatusCode
• sub_status_code: Option<SubStatusCode>
• request_charge: f64
• duration_ms: u64"] - EC["execution_context: ExecutionContext
• Initial — first attempt
• Retry — retry after 429/503/etc.
• Hedging — speculative request
• RegionFailover — cross-region retry
• CircuitBreakerProbe — recovery check"] + EC["execution_context: ExecutionContext
• Initial — first attempt
• OperationRetry — retry after 429/503/etc.
• Hedging — speculative request
• RegionFailover — cross-region retry
• CircuitBreakerProbe — recovery check"] RSS["request_sent: RequestSentStatus
• Sent — definitely transmitted
• NotSent — definitely NOT transmitted
• Unknown — cannot determine"] RE["events: Vec<RequestEvent>
• timestamp: Instant
• duration_ms: Option<u64>
• details: Option<String>"] RET["event_type: RequestEventType
• TransportStart
• ResponseHeadersReceived
• TransportComplete
• TransportFailed"] @@ -474,7 +474,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 45 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -483,7 +483,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 52 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -492,7 +492,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 78 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -501,7 +501,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 120 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -510,7 +510,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 189 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -519,7 +519,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 312 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -528,7 +528,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 456 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -537,7 +537,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 623 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -546,7 +546,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 780 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -555,7 +555,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 890 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 200, @@ -591,7 +591,7 @@ Same operation with deduplication applied: "duration_ms": 45 }, "last": { - "execution_context": "retry", + "execution_context": "operation_retry", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 200, "request_charge": 1.0, @@ -602,7 +602,7 @@ Same operation with deduplication applied: "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, "sub_status_code": 3200, - "execution_context": "retry", + "execution_context": "operation_retry", "count": 9, "total_request_charge": 9.0, "min_duration_ms": 52, diff --git a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md index cb02891fca..f3ba091b5b 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md @@ -11,6 +11,8 @@ - Added `PlanOptions` (with `DEFAULT_MAX_FAN_OUT`) to `CosmosDriver::plan_operation`, enforcing a maximum fan-out on fresh cross-partition plans. A fresh plan spanning more leaf request nodes than `PlanOptions::max_fan_out` (default 100) is rejected with the new `CosmosStatus::CLIENT_CROSS_PARTITION_FAN_OUT_EXCEEDED` (HTTP 400). The limit is enforced only at initial plan time: resuming from a continuation token skips the check, and a partition split that raises the fan-out mid-execution does not abort the operation. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855)) - Added `CosmosOperation::db_operation_name`, returning the canonical OpenTelemetry `db.operation.name` (e.g. `read_item`, `query_items`, `execute_batch`, and `read_all_items_of_logical_partition` for a read feed scoped to one logical partition) for an operation. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) - Added `RequestDiagnostics::operation_name`, naming the operation that issued an individual attempt. It is set only where one `DiagnosticsContext` aggregates attempts from more than one operation — today a PATCH, whose attempts report `patch_read_item` / `patch_replace_item` while the context reports `patch_item` — and is `None` otherwise, meaning the attempt shares the context's operation name. `CosmosOperation::is_patch_sub_operation` reports the same distinction on the operation itself. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) +- Added the Hedging Detection API on `DiagnosticsContext`: `hedging_started()` (whether a hedge arm actually fanned out), `requested_regions()` (regions dispatched to, in dispatch order with duplicates, each tagged with a reason), and `responded_regions()` (regions that produced an actual service reply, in completion order). All three are materialized at finalization from the full pre-compaction attempt list — including attempts rescued from a structurally-dropped hedge leg — plus a dispatch-time hedge fan-out log, so a cancelled hedge leg, a retry storm that compacts `requests()`, and multi-round-trip aggregation all report a complete history. Both region histories are bounded by `max_request_diagnostics` (keeping head and tail, dropping the repetitive middle) so a retry storm cannot produce an unbounded diagnostics artifact; `total_requested_regions()` / `total_responded_regions()` report the exact pre-truncation counts, so truncation is detectable rather than silent. Adds the public `RequestedRegion` struct and the new `ExecutionContext::OperationRetry` variant. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Added `HedgeTerminalState::as_str()` (and a matching `Display`) returning a stable, low-cardinality `snake_case` identifier for the hedging race outcome, for use as an observability attribute / log-field value. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Extended cross-region hedging to the container and partition-key-range metadata reads, so a slow (but not failed) region no longer stalls a client's first operation against a container. Metadata hedges use a fixed 1.5s threshold and never let a hedged region override a definitive primary result. ([#4896](https://github.com/Azure/azure-sdk-for-rust/pull/4896)) - Added `HedgingOptions` (via `DriverOptionsBuilder::with_hedging_options`), bounding how many metadata operations may make simultaneous cross-region attempts. Defaults to 32; `0` disables metadata hedging. An operation refused a slot follows the ordinary sequential failover path instead of queueing. Data-plane hedging is not budgeted — tracked by [#4916](https://github.com/Azure/azure-sdk-for-rust/issues/4916). ([#4896](https://github.com/Azure/azure-sdk-for-rust/pull/4896)) - Added `CosmosResponse::serving_region`, returning the region that produced a response — the hedge winner when the operation raced, otherwise the region of the final attempt. ([#4896](https://github.com/Azure/azure-sdk-for-rust/pull/4896)) @@ -20,6 +22,7 @@ - Resource-reference accessors now return `Option` to account for RID-addressed references that have no name. `DatabaseReference::name_based_path`, `ContainerReference::database_name`, and `ContainerReference::name_based_path` return `None` when the reference is addressed by RID (previously they returned `&str`/`String` and assumed a name was always present). Use the new `ContainerReference::base_path` to obtain the addressing-appropriate path (RID-based or name-based) when building request URLs. ([#4640](https://github.com/Azure/azure-sdk-for-rust/pull/4640)) - `AccountReference`, `DatabaseReference`, `ContainerReference`, and `ItemReference` are now tuple structs wrapping private shared state, so wildcard struct patterns such as `AccountReference { .. }` no longer compile. All accessors are unchanged. `DatabaseReference::into_account` was removed; use `DatabaseReference::account` and clone. ([#4908](https://github.com/Azure/azure-sdk-for-rust/pull/4908)) +- Removed `ExecutionContext::Retry`. Use `ExecutionContext::OperationRetry` to represent an operation-level retry decided by the SDK's client-retry policy; `TransportRetry` continues to represent a transport-level retry inside the per-region transport stack. The `RequestedRegion::reason` field is now typed as `ExecutionContext` directly (the `RequestedRegionReason` type has also been removed from the driver). The customer-visible wire-format change is that driver-generated operation retries now serialize as `"operation_retry"` instead of `"retry"`; telemetry parsers that match the literal `"retry"` execution context must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - `CosmosDriver::plan_operation` now takes an additional `plan_options: &PlanOptions` argument (after `continuation`). The continuation token remains its own argument. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855)) ### Bugs Fixed @@ -32,6 +35,9 @@ - Unified `403/3` and `403/1008` topology retries under a 5-second cumulative delay budget so a persistent topology error surfaces promptly instead of hanging. Multi-write `403/3` and all `403/1008` come down from ~120 seconds of fixed 1-second retries; single-write `403/3` moves up from three immediate generic retries onto the same topology policy. The first retry is always immediate; later retries use exponential backoff with jitter. ([#4740](https://github.com/Azure/azure-sdk-for-rust/pull/4740)) - `DiagnosticsContext::operation_name()` is now populated in production (previously always `None`): the operation pipeline sets it from `CosmosOperation::db_operation_name`, so tail-sampling classification and the tracing span have an operation name even when no SDK-supplied `CosmosOperationContext` is present. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) - PATCH operations now report `patch_item` rather than the underlying Replace, on both the aggregated success path and every error path (including read, deserialize, patch-evaluation, serialize, and non-412 replace failures). The two internal sub-operations report `patch_read_item` and `patch_replace_item` on their own attempt diagnostics, so the read-modify-write decomposition stays visible underneath the caller-facing operation instead of being flattened to a single name. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) +- Fixed the primary leg of a hedged request being recorded with `ExecutionContext::Initial` even when the hedge was dispatched from a retry. The primary leg now carries the execution context computed from the live retry state, so a hedge that upgraded a session retry or a region failover is no longer misreported as a first attempt in diagnostics. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Fixed a hedged operation losing the diagnostics of the leg that lost the race. Each leg records into its own builder, and the race cancels the loser by dropping its future, which also dropped every attempt that leg had already completed — so a leg that received a `429` and was retrying when it lost contributed nothing to `request_count()`, `regions_contacted()`, or `total_request_charge()`, under-reporting the RU the account was actually billed. Completed attempts are now mirrored into an operation-scoped hedge journal and folded back in at finalization. An attempt still in flight when its leg was cancelled observed no reply and remains unreported. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Fixed the finalized attempt list interleaving hedge legs by leg rather than by dispatch time. Attempts from both legs plus the hedge fan-out are now ordered by their actual dispatch instant, so a primary-leg retry that was dispatched before the alternate fanned out is reported before it instead of after. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Fixed a partition-key-range refresh that could remain permanently pinned to an unreachable region. A refresh resuming a region-affine change-feed continuation is routed back to the region that served it, with hedging suppressed; if that region then became unavailable, every subsequent forced refresh repeated the same failing request forever. Such a refresh now retries once from cold, which clears both the continuation and the region pin together. The same clearing now also applies when an incremental routing-map merge falls back to a full refresh that comes back empty, which previously left the continuation in place after the pin protecting it had already been released. ([#4896](https://github.com/Azure/azure-sdk-for-rust/pull/4896)) - Fixed session-token parsing rejecting the version sentinel `-1` (as in `0:-1#42`). Merging such a token now succeeds and round-trips `-1` verbatim. ([#4800](https://github.com/Azure/azure-sdk-for-rust/pull/4800)) diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/TRANSPORT_PIPELINE_SPEC.md b/sdk/cosmos/azure_data_cosmos_driver/docs/TRANSPORT_PIPELINE_SPEC.md index cbbbe1e63e..adf671d6fb 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/TRANSPORT_PIPELINE_SPEC.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/TRANSPORT_PIPELINE_SPEC.md @@ -728,12 +728,25 @@ async fn execute_hedged( (preferring recent measurements for fast reaction). Hard-coded safety gates clamp the effective threshold to the range **50 ms – 4000 ms**; configurable min/max bounds can tighten this range further. -- The primary (first) attempt always uses `ExecutionContext::Initial` so that - diagnostics can clearly distinguish whether hedging occurred. A hedged attempt - uses `ExecutionContext::Hedging`. This makes it easy to identify concurrent - execution caused by the original attempt not producing a terminal result - within the configured threshold. -- Both attempts are tracked in `DiagnosticsContext` +- The primary (first) leg of the race carries the `ExecutionContext` the + *non-hedged* path would have used for that same attempt, so the reason a + hedged operation was already retrying is never lost. Concretely: + - **STAGE 2b** (a fresh operation that crosses the latency threshold) — the + retry counters are still zero, so the primary leg is `Initial`. + - **STAGE 7** (an already-retrying operation whose next attempt is upgraded + into a hedge) — the primary leg keeps the retry's own context, e.g. + `OperationRetry` or `RegionFailover`. + + The *hedged* leg is always `ExecutionContext::Hedging`. Hedging is therefore + identified by the presence of the `Hedging` leg, not by the primary being + `Initial`, which keeps concurrent execution unambiguous without erasing why + the operation was retrying in the first place. +- Both legs of the race are tracked in `DiagnosticsContext`, including the leg + that loses and is cancelled: each leg records into a private builder, but + every attempt it *completes* is mirrored into an operation-scoped hedge + journal, so a reply already received by the loser is still reported (region + history, RU charge, status). An attempt still in flight when its leg is + cancelled observed nothing and is deliberately not reported. - The hedged attempt's RU charge is always reported regardless of which wins - **Deadline enforcement**: The hedged attempt shares the original e2e deadline. By the time the hedged attempt starts, at least `hedging_threshold` has diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs index bf544bc2a2..2fe6f94e9c 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs @@ -17,7 +17,7 @@ use azure_core::http::StatusCode; use serde::Serialize; use std::{ collections::HashMap, - sync::{Arc, OnceLock}, + sync::{Arc, Mutex, OnceLock}, time::{Duration, Instant}, }; @@ -58,8 +58,11 @@ pub enum ThresholdBreach { pub enum ExecutionContext { /// Initial request attempt (first try). Initial, - /// Retry due to transient error (e.g., 429, 503). - Retry, + /// An operation-level retry decided by the SDK's client-retry policy. + /// + /// Distinguishes user-visible operation retries from transport-layer + /// retries ([`ExecutionContext::TransportRetry`]). + OperationRetry, /// Transport-level shard retry within the same region. /// /// The initial attempt failed with a connectivity error and the transport @@ -79,7 +82,7 @@ impl ExecutionContext { pub fn as_str(&self) -> &'static str { match self { ExecutionContext::Initial => "initial", - ExecutionContext::Retry => "retry", + ExecutionContext::OperationRetry => "operation_retry", ExecutionContext::TransportRetry => "transport_retry", ExecutionContext::Hedging => "hedging", ExecutionContext::RegionFailover => "region_failover", @@ -88,6 +91,21 @@ impl ExecutionContext { } } +/// A single region the SDK dispatched a request to, tagged with the reason the +/// orchestrator chose to send it. +/// +/// Realizes the cross-SDK Hedging Detection API's `RequestedRegion` value type. +/// Returned by [`DiagnosticsContext::requested_regions`]. Region equality is +/// delegated to [`Region`]'s own `PartialEq`. +#[derive(Clone, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct RequestedRegion { + /// The region the SDK dispatched to. + pub region: Region, + /// The reason the SDK chose this region for this dispatch attempt. + pub reason: ExecutionContext, +} + impl AsRef for ExecutionContext { fn as_ref(&self) -> &str { self.as_str() @@ -584,6 +602,24 @@ impl RequestDiagnostics { } } + /// **Internal test helper — do not call.** + /// + /// Overrides the [`ExecutionContext`] of a test-constructed request so + /// emission-layer tests can synthesize a speculative hedge leg (or a retry + /// / failover dispatch). Gated behind the + /// `__internal_test_diagnostics_construction` Cargo feature and + /// `#[doc(hidden)]`, mirroring [`for_testing`](Self::for_testing). + #[cfg(feature = "__internal_test_diagnostics_construction")] + #[doc(hidden)] + #[must_use] + pub fn with_execution_context_for_testing( + mut self, + execution_context: ExecutionContext, + ) -> Self { + self.execution_context = execution_context; + self + } + /// **Internal test helper — do not call.** /// /// Stamps this attempt with the sub-operation that issued it, mirroring what @@ -716,6 +752,21 @@ impl RequestDiagnostics { self.completed_at.is_some() } + /// Returns whether this request received an actual service response. + /// + /// `completed_at` alone is insufficient: the driver also sets it for + /// client-side end-to-end timeouts ([`timeout`](Self::timeout)) and + /// transport-level failures ([`fail_transport`](Self::fail_transport)). + /// This predicate excludes those two cases so that only requests that + /// produced a service reply (any HTTP status, including non-2xx) are + /// counted. Used by [`DiagnosticsContext::responded_regions`]. + pub(crate) fn responded_with_service_reply(&self) -> bool { + self.region.is_some() + && self.completed_at.is_some() + && !self.timed_out + && self.error.is_none() + } + // Public getters for read-only access to fields /// Returns the execution context describing why this request was made. @@ -1387,6 +1438,79 @@ impl SystemUsageSnapshot { } } +/// Operation-scoped state shared by the parent [`DiagnosticsContextBuilder`] and +/// every hedge leg cloned from it. +/// +/// A hedge race resolves by *structurally dropping* the losing future. Without a +/// shared sink the loser's private builder — including service replies it had +/// already observed, their RU charge, and their timings — disappears with it, so +/// `responded_regions()` would omit real replies and the "exact totals" +/// guarantee would silently not hold. Every leg therefore mirrors each attempt +/// here the moment that attempt reaches a terminal state, tagged with the leg's +/// id; [`DiagnosticsContextBuilder::merge_hedge_attempt`] removes the winning +/// leg's copies (the winner's originals are merged directly), so exactly one +/// copy of every attempt survives into [`DiagnosticsContextBuilder::complete`]. +#[derive(Debug, Default)] +struct HedgeJournal { + /// Next leg id to hand out. Leg ids start at 1; `0` is reserved for the + /// parent builder, which is never a race participant. + next_leg_id: u64, + /// Legs that reached [`DiagnosticsContextBuilder::start_request`] at least + /// once, so materialization can tell "this leg described itself" from "this + /// leg was dropped before it dispatched anything". + dispatched_legs: Vec, + /// Terminal attempts mirrored by each leg, tagged with the leg id. What + /// remains at finalization is exactly the set of attempts that belonged to + /// legs the race dropped. + orphaned_attempts: Vec<(u64, RequestDiagnostics)>, +} + +/// One leg of a cross-region hedge race, as described to +/// [`DiagnosticsContextBuilder::record_hedge_fanout`]. +#[derive(Clone, Debug)] +pub(crate) struct HedgeLegDispatch { + /// The region the leg was routed to. `None` when the routed endpoint + /// carries no named region (global-endpoint accounts). + pub(crate) region: Option, + /// Why the orchestrator dispatched this leg to that region. + pub(crate) reason: ExecutionContext, + /// The leg builder's journal id, used to detect whether the leg ever + /// dispatched a request of its own. + leg_id: u64, + /// When the leg was launched, on the same clock as + /// [`RequestDiagnostics::started_at`]. + dispatched_at: Instant, +} + +impl HedgeLegDispatch { + /// Materializes this leg as a requested-region entry, if it was routed to a + /// named region. Used only to reconstruct a leg that never dispatched. + fn requested_region(&self) -> Option { + self.region.clone().map(|region| RequestedRegion { + region, + reason: self.reason, + }) + } +} + +/// One cross-region hedge fan-out, recorded on the parent builder at the moment +/// the race is dispatched. +/// +/// Legs describe themselves through their own [`RequestDiagnostics`], which the +/// [`HedgeJournal`] keeps alive even for the leg the race drops. This record is +/// the fallback for the one case that leaves no attempt behind: a leg cancelled +/// before it dispatched anything at all — most commonly the alternate, which +/// `select` never polls when the primary is already resolved. It also marks the +/// operation as having fanned out at all, independently of which leg won, of +/// retry-storm compaction, and of sub-operation aggregation. +#[derive(Clone, Debug)] +pub(crate) struct HedgeFanout { + /// The primary leg of the race. + primary: HedgeLegDispatch, + /// The speculative alternate leg of the race. + alternate: HedgeLegDispatch, +} + /// Internal mutable builder for constructing a [`DiagnosticsContext`]. /// /// This type is used during operation execution to collect diagnostic data. @@ -1434,6 +1558,27 @@ pub(crate) struct DiagnosticsContextBuilder { /// `None` when hedging was not selected for this operation. hedge_diagnostics: Option, + /// Every hedge fan-out dispatched by this operation, in dispatch order. + /// + /// Recorded by the hedging orchestrator on the *parent* builder before the + /// race starts, so a leg dropped before it dispatched anything is still + /// described. Empty for the overwhelming majority of operations (no + /// hedging), which keeps the common path allocation-free. + hedge_fanouts: Vec, + + /// Operation-scoped state shared with every hedge leg cloned from this + /// builder, or `None` when the operation never hedged. + /// + /// Allocated lazily by the first + /// [`clone_for_hedge_attempt`](Self::clone_for_hedge_attempt), so the + /// non-hedged path — the overwhelming majority of operations — never pays + /// for the `Arc`/`Mutex`. + hedge_journal: Option>>, + + /// This builder's id within [`Self::hedge_journal`]. `0` on the parent + /// builder, which never races. + hedge_leg_id: u64, + /// Test-only override for system usage snapshot, bypassing the CPU monitor. #[cfg(test)] test_system_usage: Option, @@ -1454,6 +1599,9 @@ impl DiagnosticsContextBuilder { #[cfg(feature = "fault_injection")] fault_injection_enabled: false, hedge_diagnostics: None, + hedge_fanouts: Vec::new(), + hedge_journal: None, + hedge_leg_id: 0, #[cfg(test)] test_system_usage: None, } @@ -1481,6 +1629,45 @@ impl DiagnosticsContextBuilder { self.hedge_diagnostics = Some(diagnostics); } + /// Records a cross-region hedge fan-out at the moment the race is dispatched. + /// + /// Called by the hedging orchestrator on the *parent* builder once both leg + /// builders exist. Each leg normally describes itself through its own + /// attempts — the [`HedgeJournal`] keeps those alive even for the leg the + /// race drops — so this record only materializes a + /// [`RequestedRegion`](crate::diagnostics::RequestedRegion) for a leg that + /// was cancelled before it dispatched anything. Each leg's `region` is + /// `None` when its routed endpoint carries no named region (global-endpoint + /// accounts), in which case that leg contributes nothing to + /// [`requested_regions`](DiagnosticsContext::requested_regions) — matching + /// how a region-less attempt is skipped. + pub(crate) fn record_hedge_fanout( + &mut self, + primary: HedgeLegDispatch, + alternate: HedgeLegDispatch, + ) { + self.hedge_fanouts.push(HedgeFanout { primary, alternate }); + } + + /// Describes this (leg) builder as one side of a hedge race, for + /// [`record_hedge_fanout`](Self::record_hedge_fanout). + /// + /// The leg's `started_at` is its launch instant — `clone_for_hedge_attempt` + /// stamps it fresh — so it sits on the same clock as every + /// [`RequestDiagnostics::started_at`] and orders correctly against them. + pub(crate) fn leg_dispatch( + &self, + region: Option, + reason: ExecutionContext, + ) -> HedgeLegDispatch { + HedgeLegDispatch { + region, + reason, + leg_id: self.hedge_leg_id, + dispatched_at: self.started_at, + } + } + /// Creates a fresh builder for a single hedge attempt. /// /// Shares operation-level context (`activity_id`, `options`, @@ -1488,8 +1675,18 @@ impl DiagnosticsContextBuilder { /// but starts with an empty request list and a fresh `started_at`, so /// per-attempt durations measure from launch rather than from /// operation start. The winning attempt's requests are merged back - /// via [`merge_hedge_attempt`](Self::merge_hedge_attempt). - pub(crate) fn clone_for_hedge_attempt(&self) -> Self { + /// via [`merge_hedge_attempt`](Self::merge_hedge_attempt); the losing + /// attempt's are recovered from the shared [`HedgeJournal`], which is + /// allocated here on the first call. + pub(crate) fn clone_for_hedge_attempt(&mut self) -> Self { + let journal = self + .hedge_journal + .get_or_insert_with(|| Arc::new(Mutex::new(HedgeJournal::default()))); + let leg_id = { + let mut guard = journal.lock().unwrap_or_else(|e| e.into_inner()); + guard.next_leg_id += 1; + guard.next_leg_id + }; Self { activity_id: self.activity_id.clone(), started_at: Instant::now(), @@ -1502,6 +1699,12 @@ impl DiagnosticsContextBuilder { #[cfg(feature = "fault_injection")] fault_injection_enabled: self.fault_injection_enabled, hedge_diagnostics: None, + // Fan-out records live on the parent builder only: a leg's own + // builder is either merged back (winner) or dropped (loser), so + // recording here would be lost exactly when it matters. + hedge_fanouts: Vec::new(), + hedge_journal: Some(Arc::clone(journal)), + hedge_leg_id: leg_id, #[cfg(test)] test_system_usage: self.test_system_usage.clone(), } @@ -1513,7 +1716,17 @@ impl DiagnosticsContextBuilder { /// Only the request list is moved; the attempt's `status` and /// `hedge_diagnostics` are discarded because those operation-level /// fields are written directly on the parent. + /// + /// The merged leg's mirrored copies are dropped from the shared + /// [`HedgeJournal`] at the same time: the originals are now owned by the + /// parent, so leaving the mirror in place would double-count the winner. pub(crate) fn merge_hedge_attempt(&mut self, attempt: Self) { + if let Some(journal) = self.hedge_journal.as_ref() { + let mut guard = journal.lock().unwrap_or_else(|e| e.into_inner()); + guard + .orphaned_attempts + .retain(|(leg_id, _)| *leg_id != attempt.hedge_leg_id); + } if self.requests.is_empty() { self.requests = attempt.requests; } else { @@ -1585,9 +1798,55 @@ impl DiagnosticsContextBuilder { request.with_activity_id(self.activity_id.clone()); let handle = RequestHandle(self.requests.len()); self.requests.push(request); + self.journal_dispatch(); handle } + /// Records that this hedge leg has dispatched at least one request. + /// + /// Lets materialization tell a leg that described itself through its own + /// attempts from one the race cancelled before it dispatched anything — + /// only the latter needs a synthetic entry from the fan-out record. + fn journal_dispatch(&mut self) { + if self.hedge_leg_id == 0 { + return; + } + let Some(journal) = self.hedge_journal.as_ref() else { + return; + }; + let mut guard = journal.lock().unwrap_or_else(|e| e.into_inner()); + if !guard.dispatched_legs.contains(&self.hedge_leg_id) { + guard.dispatched_legs.push(self.hedge_leg_id); + } + } + + /// Mirrors a now-terminal attempt into the shared [`HedgeJournal`]. + /// + /// Attempts are immutable once terminal (`update_request` rejects a + /// completed handle), so the mirrored copy is final. If this leg goes on to + /// win the race its copies are dropped again by + /// [`merge_hedge_attempt`](Self::merge_hedge_attempt); if the race drops + /// this leg instead, the copy is the only surviving record of a reply the + /// service really sent. + /// + /// An attempt still in flight when the leg is cancelled is genuinely + /// unobserved — no reply arrived — and is intentionally not recovered. + fn journal_terminal_attempt(&mut self, handle: RequestHandle) { + if self.hedge_leg_id == 0 { + return; + } + let Some(journal) = self.hedge_journal.as_ref() else { + return; + }; + let Some(request) = self.requests.get(handle.0) else { + return; + }; + let mut guard = journal.lock().unwrap_or_else(|e| e.into_inner()); + guard + .orphaned_attempts + .push((self.hedge_leg_id, request.clone())); + } + /// Records the response headers and completion of a request in one shot. /// /// Convenience wrapper around [`update_request`](Self::update_request) + @@ -1642,6 +1901,7 @@ impl DiagnosticsContextBuilder { if let Some(request) = self.requests.get_mut(handle.0) { request.complete(status_code, sub_status); } + self.journal_terminal_attempt(handle); } /// Records end-to-end timeout of a request. @@ -1657,6 +1917,7 @@ impl DiagnosticsContextBuilder { if let Some(request) = self.requests.get_mut(handle.0) { request.timeout(); } + self.journal_terminal_attempt(handle); } /// Records a transport-level failure for a request that received no Cosmos response. @@ -1670,6 +1931,7 @@ impl DiagnosticsContextBuilder { if let Some(request) = self.requests.get_mut(handle.0) { request.fail_transport(error, request_sent, status); } + self.journal_terminal_attempt(handle); } /// Updates a request's diagnostics with additional data. @@ -1746,9 +2008,38 @@ impl DiagnosticsContextBuilder { /// This consumes the builder and creates a finalized diagnostics context /// with all data frozen. The `DiagnosticsContext` can then be safely /// shared via `Arc` without any locking overhead. - pub(crate) fn complete(self) -> DiagnosticsContext { + pub(crate) fn complete(mut self) -> DiagnosticsContext { let duration = self.started_at.elapsed(); + // Recover the attempts of any hedge leg the race dropped before it + // could be merged. Those legs may have observed real service replies + // (a 429 answered just before the partner won, say); losing them would + // under-report charge, drop a genuine entry from `responded_regions`, + // and break the exactness of the `total_*_regions` counters. + // + // `started_at` is a single operation-wide clock, and every builder + // appends in start order, so re-sorting the union restores true global + // dispatch order across the parent and both legs — something no + // parent-relative index could express while the legs ran concurrently. + // Handles are never used after `complete` consumes the builder, so + // reordering here cannot invalidate one. + let (orphaned, dispatched_legs) = self + .hedge_journal + .take() + .map(|journal| { + let mut guard = journal.lock().unwrap_or_else(|e| e.into_inner()); + ( + std::mem::take(&mut guard.orphaned_attempts), + std::mem::take(&mut guard.dispatched_legs), + ) + }) + .unwrap_or_default(); + if !orphaned.is_empty() { + self.requests + .extend(orphaned.into_iter().map(|(_, request)| request)); + self.requests.sort_by_key(RequestDiagnostics::started_at); + } + // Exact operation-level total charge, summed from the FULL attempt list // before any compaction so it stays exact even when the retained list is // bounded under a retry storm. @@ -1760,6 +2051,33 @@ impl DiagnosticsContextBuilder { // only attempts are elided must still surface at the operation level. let regions_contacted = ordered_unique_regions(&self.requests); + // Materialize the Hedging Detection API's region history from the FULL + // attempt list plus the dispatch-time hedge fan-out log, for the same + // reason: neither the retained (possibly compacted) attempt list nor the + // winning leg alone is a complete dispatch history. + // + // Both histories are then bounded by the same `max_request_diagnostics` + // cap as the attempt list, so a retry storm cannot grow them without + // limit (DIAGNOSTICS-CONTRACT.md §8). The pre-truncation lengths are + // retained so the truncation is explicit rather than silent. + let cap = self.options.max_request_diagnostics(); + let requested_regions = + requested_regions_from(&self.requests, &self.hedge_fanouts, &dispatched_legs); + let responded_regions = responded_regions_from(&self.requests); + let total_requested_regions = requested_regions.len(); + let total_responded_regions = responded_regions.len(); + let requested_regions = bound_region_history(requested_regions, cap); + let responded_regions = bound_region_history(responded_regions, cap); + let hedging_started = !self.hedge_fanouts.is_empty() + || self + .hedge_diagnostics + .as_ref() + .is_some_and(|hd| hd.alternate_region().is_some()) + || self + .requests + .iter() + .any(|r| matches!(r.execution_context(), ExecutionContext::Hedging)); + // Bound the finalized per-attempt list under a retry storm. // // Common path (attempts <= cap): the list is retained verbatim, no @@ -1776,7 +2094,6 @@ impl DiagnosticsContextBuilder { // is on the finalized serialized artifact, not on live mid-operation // memory: `self.requests` still grows one entry per attempt while the // operation is in flight. - let cap = self.options.max_request_diagnostics(); let original_count = self.requests.len(); let (requests, compaction) = if original_count > cap { let compacted = compact_requests(self.requests, cap); @@ -1801,6 +2118,11 @@ impl DiagnosticsContextBuilder { requests: Arc::new(requests), total_request_charge, regions_contacted, + requested_regions, + responded_regions, + total_requested_regions, + total_responded_regions, + hedging_started, status: self.status, options: self.options, cpu_monitor: self.cpu_monitor, @@ -1888,6 +2210,45 @@ pub struct DiagnosticsContext { /// Cosmos semantic conventions require (it conveys failover order). regions_contacted: Vec, + /// Regions this operation dispatched a request to, in dispatch order, each + /// tagged with the reason the SDK chose it. + /// + /// Materialized at finalization from the **full** attempt list — before any + /// retry-storm compaction — with every hedge fan-out spliced in at the point + /// it was dispatched, so a structurally-dropped hedge loser leg and a + /// compacted-away retry are both still reported. Unlike `regions_contacted` + /// this list is *not* deduplicated: it is a dispatch log, so repeat + /// dispatches to the same region each contribute an entry. + /// + /// Bounded by `max_request_diagnostics` (head + tail retained, repetitive + /// middle elided) so a retry storm cannot grow it without limit; + /// `total_requested_regions` records the pre-truncation length. + requested_regions: Vec, + + /// Regions that produced an actual service reply, in arrival (completion) + /// order. + /// + /// Materialized at finalization from the **full** attempt list, for the same + /// reason as `requested_regions`, and bounded the same way. Client-side + /// timeouts and transport failures are excluded; a non-2xx service response + /// still counts. + responded_regions: Vec, + + /// Exact number of dispatches recorded before `requested_regions` was + /// bounded. Equal to `requested_regions.len()` when no truncation occurred. + total_requested_regions: usize, + + /// Exact number of service replies recorded before `responded_regions` was + /// bounded. Equal to `responded_regions.len()` when no truncation occurred. + total_responded_regions: usize, + + /// Whether this operation actually fanned out at least one hedge request. + /// + /// Materialized at finalization so it survives compaction dropping the + /// hedge attempts and aggregation retaining only one representative + /// `hedge_diagnostics`. + hedging_started: bool, + /// Operation-level combined HTTP status and sub-status (final status after retries). status: Option, @@ -2060,12 +2421,26 @@ impl DiagnosticsContext { .sum::(), ); let regions_contacted = ordered_unique_regions(&requests); + let requested_regions = requested_regions_from(&requests, &[], &[]); + let responded_regions = responded_regions_from(&requests); + // The helper builds a fixed, hand-supplied attempt list, so the history + // is never over the cap and the totals are simply its length. + let total_requested_regions = requested_regions.len(); + let total_responded_regions = responded_regions.len(); + let hedging_started = requests + .iter() + .any(|r| matches!(r.execution_context(), ExecutionContext::Hedging)); DiagnosticsContext { activity_id, duration, requests: Arc::new(requests), total_request_charge, regions_contacted, + requested_regions, + responded_regions, + total_requested_regions, + total_responded_regions, + hedging_started, status, options: Arc::new(DiagnosticsOptions::default()), cpu_monitor: None, @@ -2081,8 +2456,105 @@ impl DiagnosticsContext { } } - /// Concatenates the per-request diagnostics from a sequence of - /// sub-operation contexts into a single aggregated [`DiagnosticsContext`]. + /// **Internal test helper — do not call.** + /// + /// Like [`for_testing_with_requests`](Self::for_testing_with_requests), but + /// also attaches `hedge_diagnostics` so the wrapper SDK's emission-layer + /// tests can exercise the hedging-surfacing paths (tracing / metrics / + /// logging). When the supplied diagnostics describe a fan-out (an alternate + /// region is present), the dispatch-time fan-out record the driver pipeline + /// would have written is synthesized too, so the materialized region history + /// matches a real hedged operation. Gated behind the + /// `__internal_test_diagnostics_construction` Cargo feature and + /// `#[doc(hidden)]`, mirroring [`for_testing`](Self::for_testing). + #[cfg(feature = "__internal_test_diagnostics_construction")] + #[doc(hidden)] + pub fn for_testing_with_hedge( + activity_id: ActivityId, + duration: Duration, + status: Option, + operation_name: Option<&str>, + requests: Vec, + hedge_diagnostics: Option, + ) -> Self { + // Reconstruct the fan-out the pipeline records at race dispatch, so the + // materialized region history is built the same way it is in production. + // The sentinel stands in for a global-endpoint account with no named + // region, which contributes no requested-region entry. + let not_sentinel = |region: &Region| { + (region.as_str() != HedgeDiagnostics::UNKNOWN_REGION_SENTINEL).then(|| region.clone()) + }; + // Both legs are stamped at (or before) the first supplied attempt, so a + // leg that has to be reconstructed lands at the head of the history — + // the same position the pipeline would produce for a race dispatched + // before any attempt was recorded. + let dispatched_at = requests + .iter() + .map(RequestDiagnostics::started_at) + .min() + .unwrap_or_else(Instant::now); + const PRIMARY_LEG: u64 = 1; + const ALTERNATE_LEG: u64 = 2; + let fanouts: Vec = hedge_diagnostics + .as_ref() + .and_then(|hedge| { + let alternate = hedge.alternate_region()?; + Some(HedgeFanout { + primary: HedgeLegDispatch { + region: not_sentinel(hedge.primary_region()), + reason: ExecutionContext::Initial, + leg_id: PRIMARY_LEG, + dispatched_at, + }, + alternate: HedgeLegDispatch { + region: not_sentinel(alternate), + reason: ExecutionContext::Hedging, + leg_id: ALTERNATE_LEG, + dispatched_at, + }, + }) + }) + .into_iter() + .collect(); + // A leg "dispatched" when the supplied attempt list already describes + // it; only a leg with no attempt of its own needs reconstructing. + let dispatched_legs: Vec = fanouts + .first() + .map(|fanout| { + [&fanout.primary, &fanout.alternate] + .into_iter() + .filter(|leg| { + leg.region.as_ref().is_some_and(|region| { + requests.iter().any(|request| { + request.region() == Some(region) + && request.execution_context() == leg.reason + }) + }) + }) + .map(|leg| leg.leg_id) + .collect() + }) + .unwrap_or_default(); + + let mut context = Self::for_testing_with_requests( + activity_id, + duration, + status, + operation_name, + requests, + ); + if !fanouts.is_empty() { + context.requested_regions = + requested_regions_from(&context.requests, &fanouts, &dispatched_legs); + context.total_requested_regions = context.requested_regions.len(); + context.hedging_started = true; + } + context.hedge_diagnostics = hedge_diagnostics; + context + } + + /// Aggregates the finalized per-sub-operation contexts of a multi-round-trip + /// operation into a single aggregated [`DiagnosticsContext`]. /// /// Used by the PATCH handler to surface **one operation = one /// [`DiagnosticsContext`]** even though the handler internally executes @@ -2209,19 +2681,101 @@ impl DiagnosticsContext { } } + // Concatenate the sub-ops' materialized dispatch histories in sub-op + // order. Each source already captured its own exact history from its + // full attempt list plus its own fan-out records, so every sub-op's + // hedge fan-out is preserved — unlike the single representative + // `hedge_diagnostics` below, which can only describe one of them. + // + // Concatenation is an independent unbounded path — a PATCH conflict loop + // adds a sub-op per retry, so the aggregate grows with sub-op count even + // when each source is individually bounded. Re-bound the result under + // the same cap and carry the summed pre-truncation totals, so the + // aggregate honours the same guarantee as a single operation. + // + // Bounding the concatenation of already-bounded sources compresses the + // middle twice: a sub-op in the middle can have had its own middle + // elided, and then be dropped wholesale here. That is intentional, not + // an oversight, and a proportionally larger cap is deliberately *not* + // used: + // + // - The contract is a flat "the finalized artifact holds at most `cap` + // region entries", the same promise a single operation makes. Scaling + // the cap with sub-op count would make an aggregate's size grow with + // the length of a retry loop — exactly the unbounded growth the bound + // exists to prevent. + // - Nothing exact is lost to it: `total_requested_regions` and + // `total_responded_regions` are summed from the sources' + // pre-truncation totals, `total_request_charge` and `request_count()` + // stay exact via the compaction marker, and `hedging_started` is a + // disjunction over the sources, so a fan-out elided from the ordered + // view is still reported. + // - The interesting entries in a conflict loop are the head (how the + // operation started) and the tail (where it finally landed), which + // head/tail bounding keeps verbatim by construction. + let total_requested_regions: usize = + sources.iter().map(|c| c.total_requested_regions).sum(); + let total_responded_regions: usize = + sources.iter().map(|c| c.total_responded_regions).sum(); + let requested_regions = bound_region_history( + sources + .iter() + .flat_map(|c| c.requested_regions.iter().cloned()) + .collect(), + cap, + ); + let responded_regions = bound_region_history( + sources + .iter() + .flat_map(|c| c.responded_regions.iter().cloned()) + .collect(), + cap, + ); + let hedging_started = sources.iter().any(|c| c.hedging_started); + Some(DiagnosticsContext { activity_id: last.activity_id.clone(), duration: aggregated_duration, requests: Arc::new(requests), total_request_charge, regions_contacted, + requested_regions, + responded_regions, + total_requested_regions, + total_responded_regions, + hedging_started, status: last.status, options: Arc::clone(&last.options), cpu_monitor: last.cpu_monitor.clone(), machine_id: last.machine_id.clone(), operation_name: last.operation_name.clone(), fault_injection_enabled: sources.iter().any(|c| c.fault_injection_enabled), - hedge_diagnostics: None, + // Propagate a representative hedge diagnostics so an aggregated + // operation (e.g. PATCH, whose internal Read sub-op can itself + // hedge) still reports a hedge terminal outcome: the hedged metric's + // `hedge_terminal_state` dimension and the tracing/log hedge fields + // must not silently drop just because the operation was aggregated. + // Prefer a sub-op that actually fanned out (an alternate region is + // present); otherwise fall back to any attached hedge diagnostics. + // Taking the last match mirrors how the aggregate inherits its + // operation-level fields from the last sub-op. Only one + // representative terminal outcome survives — but the Hedging + // Detection API does not depend on it, since `hedging_started` and + // `requested_regions` are stitched from every sub-op above. + hedge_diagnostics: sources + .iter() + .rev() + .find_map(|c| { + c.hedge_diagnostics + .clone() + .filter(|hd| hd.alternate_region().is_some()) + }) + .or_else(|| { + sources + .iter() + .rev() + .find_map(|c| c.hedge_diagnostics.clone()) + }), compaction, #[cfg(test)] test_system_usage: last.test_system_usage.clone(), @@ -2319,6 +2873,144 @@ impl DiagnosticsContext { self.regions_contacted.clone() } + /// Returns the regions to which this operation dispatched a request, each + /// tagged with the reason the SDK chose it. + /// + /// The list is materialized at finalization from the **full** attempt list — + /// before any retry-storm compaction — plus the dispatch-time hedge fan-out + /// log, so it is a complete dispatch history regardless of what survives in + /// [`requests`](Self::requests). + /// + /// Each dispatched attempt with a resolved region contributes one entry. + /// Duplicates are allowed: the same region may appear more than once if it + /// was dispatched multiple times (e.g., a retry to the same region, or a + /// hedge request to a region that was also the primary). The initial attempt + /// is included and tagged [`ExecutionContext::Initial`]. + /// + /// Entries with no resolved region (pre-region-selection failures, and + /// global-endpoint accounts that carry no named region) are skipped, so this + /// returns an empty `Vec` when an operation failed before any region was + /// selected. + /// + /// **Hedge fan-out.** When a hedge race resolves as a clean win (the primary + /// wins after the threshold, or the alternate wins outright), the losing + /// leg's future — and its per-request [`RequestDiagnostics`] — is + /// structurally dropped before it can be merged, so [`requests`](Self::requests) + /// holds only the winning leg. Every fan-out is therefore recorded on the + /// parent at dispatch time, so **both** legs always appear here: the primary + /// leg tagged with the reason it was actually dispatched under (`Initial` for + /// a first attempt, or the failover/session reason when the hedge upgraded a + /// retry), and the alternate leg tagged [`ExecutionContext::Hedging`]. + /// A dropped leg has no corresponding [`responded_regions`](Self::responded_regions) + /// entry, since it never produced a service reply. + /// + /// For an aggregated operation (e.g. PATCH) stitched from multiple + /// sub-operations, every sub-operation's fan-out is preserved: the + /// aggregated list is the concatenation of each sub-operation's own + /// materialized list, in sub-operation order. + /// + /// Order is dispatch order: each fan-out's two legs are spliced in at the + /// point the race was dispatched, primary before alternate. + /// + /// # Bounded under a retry storm + /// + /// The list is capped at `DiagnosticsOptions::max_request_diagnostics` + /// (default 512). Past that, the head and tail are kept verbatim and the + /// repetitive middle is elided, so the initial dispatch, any early hedge + /// fan-out, and the final landing region all survive. Truncation is never + /// silent: [`total_requested_regions`](Self::total_requested_regions) + /// reports the exact pre-truncation count, so + /// `requested_regions().len() < total_requested_regions()` detects it. + pub fn requested_regions(&self) -> Vec { + self.requested_regions.clone() + } + + /// Returns the exact number of dispatches recorded for this operation, + /// including any elided by the bound on + /// [`requested_regions`](Self::requested_regions). + /// + /// Equal to `requested_regions().len()` unless the history was truncated. + pub fn total_requested_regions(&self) -> usize { + self.total_requested_regions + } + + /// Returns the regions from which this operation received a response, in + /// arrival (completion) order. + /// + /// The list is materialized at finalization from the **full** attempt list — + /// before any retry-storm compaction — so a response whose attempt was later + /// elided from [`requests`](Self::requests) is still reported here. + /// + /// Each request that produced a service reply contributes one entry. + /// Duplicates are allowed: the same region may appear more than once if + /// multiple completed responses arrived from it (e.g., a late hedge + /// response after the hedge winner). `responded_regions().len() > 1` does + /// NOT imply more than one distinct region responded. + /// + /// Only requests that received an actual service response are included; + /// client-side timeouts and transport failures are excluded (via the + /// internal `responded_with_service_reply` predicate on each request). + /// A non-2xx HTTP status (e.g., 404/429) still counts — it is a response + /// from the region. + /// + /// Unlike [`requested_regions`](Self::requested_regions), this accessor does + /// **not** include a structurally-dropped hedge loser leg: on a clean hedge + /// win the losing leg is cancelled before it produces a service reply, so it + /// correctly does not appear here. A clean hedge win therefore lists only the + /// winning region even though `requested_regions()` lists both fan-out legs. + /// + /// To deduplicate, callers can collect into a set, for example: + /// `ctx.responded_regions().into_iter().collect::>()`. + /// + /// # Bounded under a retry storm + /// + /// Capped the same way as [`requested_regions`](Self::requested_regions); + /// [`total_responded_regions`](Self::total_responded_regions) reports the + /// exact pre-truncation count. + pub fn responded_regions(&self) -> Vec<&Region> { + self.responded_regions.iter().collect() + } + + /// Returns the exact number of service replies recorded for this operation, + /// including any elided by the bound on + /// [`responded_regions`](Self::responded_regions). + /// + /// Equal to `responded_regions().len()` unless the history was truncated. + pub fn total_responded_regions(&self) -> usize { + self.total_responded_regions + } + + /// Returns `true` iff this operation actually dispatched at least one hedge + /// request (i.e., fan-out occurred), and `false` otherwise. + /// + /// Like the two region accessors, this is materialized at finalization from + /// the full pre-compaction attempt list plus the dispatch-time fan-out log, + /// so a hedge race whose attempts were later compacted away still reports + /// `true`. + /// + /// `false` does NOT mean hedging was disabled or misconfigured; it means no + /// fan-out occurred. In particular, when the primary returns before the + /// hedging threshold elapses, this returns `false` even though a hedging + /// strategy was active. + /// + /// [`hedge_diagnostics`](Self::hedge_diagnostics) is a related but distinct + /// surface: it is `Some` when the hedge race recorded a terminal outcome — + /// including the primary-wins-under-threshold case where no fan-out happened + /// — but it is `None` when hedging was configured yet ineligible, and also + /// for a both-transient hedge that was subsequently resolved by a failover + /// attempt (the non-terminal both-transient path deliberately records no + /// terminal outcome). So `hedge_diagnostics().is_some()` is not a reliable + /// "was hedging configured" probe, and it can disagree with + /// `hedging_started()` on that both-transient→failover path (where the + /// recorded fan-out keeps this accessor `true`). This accessor is the + /// authoritative fan-out signal: the SDK's hedged metric counter, sampled + /// log line, and root-span hedging attributes all gate on it, so an + /// operation that demonstrably fanned out is reported on every surface even + /// when no terminal outcome was retained. + pub fn hedging_started(&self) -> bool { + self.hedging_started + } + /// Returns a shared reference to all request diagnostics. /// /// This returns an `Arc>`, enabling efficient @@ -2338,14 +3030,20 @@ impl DiagnosticsContext { Arc::clone(&self.requests) } - /// Returns the hedging diagnostics for this operation, if hedging was - /// selected. + /// Returns the hedging diagnostics for this operation, if the hedge race + /// recorded a terminal outcome. /// - /// This is `Some(_)` if and only - /// if `should_hedge()` returned `true` and `execute_hedged()` was - /// entered — even when the primary won before the threshold elapsed. - /// `None` means hedging was not selected for this operation (no - /// strategy resolved, strategy `Disabled`, or eligibility check failed). + /// This is `Some(_)` when the hedge race recorded a terminal outcome — + /// including the primary-wins-under-threshold case where `execute_hedged()` + /// ran but no alternate leg fanned out. It is `None` when hedging was not + /// selected for this operation (no strategy resolved, strategy `Disabled`, + /// or eligibility check failed), **and also** on the both-transient→failover + /// path: when both hedge legs return transient failures but the deadline has + /// not elapsed and failover budget remains, `finalize_both_transient` + /// deliberately does not stamp a terminal outcome (a later successful retry + /// would otherwise carry a misleading `BothTransient` state), so a retained + /// `ExecutionContext::Hedging` request can leave `hedging_started()` `true` + /// while this returns `None`. pub fn hedge_diagnostics(&self) -> Option<&HedgeDiagnostics> { self.hedge_diagnostics.as_ref() } @@ -2455,6 +3153,11 @@ impl DiagnosticsContext { ), total_request_charge: self.total_request_charge, regions_contacted: self.regions_contacted.clone(), + requested_regions: self.requested_regions.clone(), + responded_regions: self.responded_regions.clone(), + total_requested_regions: self.total_requested_regions, + total_responded_regions: self.total_responded_regions, + hedging_started: self.hedging_started, status: self.status, options: Arc::clone(&self.options), cpu_monitor: self.cpu_monitor.clone(), @@ -2694,6 +3397,11 @@ impl Clone for DiagnosticsContext { requests: Arc::clone(&self.requests), total_request_charge: self.total_request_charge, regions_contacted: self.regions_contacted.clone(), + requested_regions: self.requested_regions.clone(), + responded_regions: self.responded_regions.clone(), + total_requested_regions: self.total_requested_regions, + total_responded_regions: self.total_responded_regions, + hedging_started: self.hedging_started, status: self.status, options: Arc::clone(&self.options), cpu_monitor: self.cpu_monitor.clone(), @@ -2734,6 +3442,16 @@ impl PartialEq for DiagnosticsContext { && self.requests == other.requests && self.total_request_charge == other.total_request_charge && self.regions_contacted == other.regions_contacted + && self.requested_regions == other.requested_regions + && self.responded_regions == other.responded_regions + // Compared for the same reason as `total_request_charge`: after the + // region histories are bounded these are no longer derivable from + // the retained vectors, so excluding them would let two contexts + // with different public `total_requested_regions()` / + // `total_responded_regions()` results compare equal. + && self.total_requested_regions == other.total_requested_regions + && self.total_responded_regions == other.total_responded_regions + && self.hedging_started == other.hedging_started && self.status == other.status && self.options == other.options && self.operation_name == other.operation_name @@ -2795,6 +3513,139 @@ fn ordered_unique_regions(requests: &[RequestDiagnostics]) -> Vec { regions } +/// Bounds a materialized region history to `cap` entries, independently of +/// attempt count, per the bounded-size guarantee in `DIAGNOSTICS-CONTRACT.md` +/// §8. +/// +/// Under a `410`/`429` retry storm the dispatch history grows one entry per +/// attempt, so it needs its own bound — the per-attempt list's compaction does +/// not apply to it. The repetitive middle of a storm is elided while the head +/// (the initial dispatch and any early hedge fan-out) and the tail (where the +/// operation finally landed) are kept verbatim, mirroring the "first + last of +/// each run" policy `compact_requests` already uses. +/// +/// # Deliberately atomicity-oblivious +/// +/// This helper takes no stride or grouping parameter, and none is planned. A +/// hedge fan-out contributes two adjacent entries that straddle the head/tail +/// boundary at exactly `cap`; one leg is kept and the other drained, so the +/// bounded view can show a "half" race. That is accepted, because: +/// +/// - It only occurs on an operation whose history already exceeds `cap`, i.e. +/// one that has been failing over or storming long enough that a single race +/// in the middle is not the interesting signal. +/// - Nothing exact is lost. +/// [`total_requested_regions`](DiagnosticsContext::total_requested_regions) +/// and [`total_responded_regions`](DiagnosticsContext::total_responded_regions) +/// are computed pre-truncation and stay exact, and +/// [`hedging_started`](DiagnosticsContext::hedging_started) is derived from +/// the fan-out records, not from this list — so "did this operation hedge?" +/// is still answered correctly even if both legs were elided. +/// - Keeping pairs intact would make the output length depend on where the +/// boundary falls, breaking the flat `<= cap` bound this exists to provide. +/// +/// The two histories are also *not* index-paired with each other: +/// `requested_regions` is dispatch-ordered and `responded_regions` is +/// arrival-ordered over the subset that actually replied, so they routinely +/// differ in both length and order. Bounding them independently therefore +/// cannot break a correspondence that never existed. +/// +/// Truncation is never silent: the caller records the pre-truncation length, +/// which [`DiagnosticsContext::total_requested_regions`] and +/// [`DiagnosticsContext::total_responded_regions`] expose. +fn bound_region_history(mut history: Vec, cap: usize) -> Vec { + if history.len() <= cap { + return history; + } + let head = cap.div_ceil(2); + let tail = cap - head; + history.drain(head..history.len() - tail); + history +} + +/// Builds the dispatch-ordered requested-region history from the **full** +/// (pre-compaction) attempt list. +/// +/// `requests` is already in global dispatch order — every builder appends in +/// start order and [`DiagnosticsContextBuilder::complete`] re-sorts the union of +/// the parent's attempts and any recovered hedge-leg attempts by `started_at` — +/// so attempts are emitted verbatim in the order given. +/// +/// A hedge leg normally describes itself through its own attempts, including +/// when the race drops it (the [`HedgeJournal`] preserves them). The one leg +/// that leaves nothing behind is one cancelled before it dispatched anything — +/// most commonly the alternate, which `select` never polls when the primary is +/// already resolved. Only such a leg gets a synthetic entry here, positioned by +/// its dispatch instant, which is why `dispatched_legs` is required: it is the +/// set of legs that reached `start_request` at least once. +fn requested_regions_from( + requests: &[RequestDiagnostics], + fanouts: &[HedgeFanout], + dispatched_legs: &[u64], +) -> Vec { + // Legs that never dispatched anything, in dispatch order. Nothing to + // reconstruct on the overwhelmingly common non-hedged path. + let mut silent_legs: Vec<&HedgeLegDispatch> = if fanouts.is_empty() { + Vec::new() + } else { + fanouts + .iter() + .flat_map(|fanout| [&fanout.primary, &fanout.alternate]) + .filter(|leg| leg.region.is_some() && !dispatched_legs.contains(&leg.leg_id)) + .collect() + }; + silent_legs.sort_by_key(|leg| leg.dispatched_at); + + let mut regions: Vec = Vec::with_capacity(requests.len()); + let mut next_silent_leg = 0usize; + + for request in requests { + // Emit any never-dispatched leg launched at or before this attempt, so + // the reconstructed entry lands in true dispatch order. + while let Some(leg) = silent_legs.get(next_silent_leg) { + if leg.dispatched_at > request.started_at() { + break; + } + regions.extend(leg.requested_region()); + next_silent_leg += 1; + } + + if let Some(region) = request.region() { + regions.push(RequestedRegion { + region: region.clone(), + reason: request.execution_context(), + }); + } + } + + for leg in &silent_legs[next_silent_leg.min(silent_legs.len())..] { + regions.extend(leg.requested_region()); + } + + regions +} + +/// Builds the arrival-ordered responded-region history from the **full** +/// (pre-compaction) attempt list. +/// +/// Only attempts that received an actual service reply contribute; a +/// structurally-dropped hedge leg never does, which is why this list can be +/// shorter than [`requested_regions_from`]'s. +fn responded_regions_from(requests: &[RequestDiagnostics]) -> Vec { + let mut responded: Vec<&RequestDiagnostics> = requests + .iter() + .filter(|r| r.responded_with_service_reply()) + .collect(); + // Stable sort by completion time to yield arrival order while preserving + // dispatch order among ties. + responded.sort_by_key(|r| r.completed_at()); + responded + .iter() + .filter_map(|r| r.region()) + .cloned() + .collect() +} + /// Builds a summary for requests in a single region. fn build_region_summary( region: Option, @@ -3104,7 +3955,7 @@ mod tests { builder.update_request(h1, |req| req.request_charge = RequestCharge::new(3.0)); let h2 = builder.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::WEST_US_2), "https://test.documents.azure.com", ); @@ -3123,7 +3974,7 @@ mod tests { "https://test.westus2.documents.azure.com", ); builder.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::WEST_US_2), "https://test.westus2.documents.azure.com", ); @@ -3388,7 +4239,7 @@ mod tests { ); record_run( &mut read, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", "https://east/", CosmosStatus::new(StatusCode::TooManyRequests), @@ -3408,7 +4259,7 @@ mod tests { ); record_run( &mut replace, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "West US", "https://west/", CosmosStatus::new(StatusCode::Gone), @@ -3676,7 +4527,7 @@ mod tests { // Add several requests to trigger deduplication for i in 0..5 { let handle = builder.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::WEST_US_2), "https://test.documents.azure.com", ); @@ -3703,7 +4554,7 @@ mod tests { "request_count": 5, "total_request_charge": 10.0, "first": { - "execution_context": "retry", + "execution_context": "operation_retry", "endpoint": "https://test.documents.azure.com/", "status": "429/3200 (RUBudgetExceeded)", "request_charge": 0.0, @@ -3711,7 +4562,7 @@ mod tests { "timed_out": false }, "last": { - "execution_context": "retry", + "execution_context": "operation_retry", "endpoint": "https://test.documents.azure.com/", "status": "429/3200 (RUBudgetExceeded)", "request_charge": 4.0, @@ -3721,7 +4572,7 @@ mod tests { "deduplicated_groups": [{ "endpoint": "https://test.documents.azure.com/", "status": "429/3200 (RUBudgetExceeded)", - "execution_context": "retry", + "execution_context": "operation_retry", "count": 3, "total_request_charge": 6.0, @@ -4044,7 +4895,7 @@ mod tests { ); let succeeded = builder.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::WEST_US_2), "https://test.documents.azure.com", ); @@ -4138,7 +4989,10 @@ mod tests { #[test] fn execution_context_display() { assert_eq!(ExecutionContext::Initial.to_string(), "initial"); - assert_eq!(ExecutionContext::Retry.to_string(), "retry"); + assert_eq!( + ExecutionContext::OperationRetry.to_string(), + "operation_retry" + ); assert_eq!( ExecutionContext::TransportRetry.to_string(), "transport_retry" @@ -4154,44 +5008,1039 @@ mod tests { ); } - // ========================================================================= - // Pipeline/Transport/RequestSentStatus tests (merged from request_diagnostics.rs) - // ========================================================================= - #[test] - fn pipeline_type_classification() { - assert!(PipelineType::Metadata.is_metadata()); - assert!(!PipelineType::Metadata.is_data_plane()); - assert!(PipelineType::DataPlane.is_data_plane()); - assert!(!PipelineType::DataPlane.is_metadata()); - } + fn requested_regions_preserves_dispatch_order_and_reason() { + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + builder.start_test_request( + ExecutionContext::Initial, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.start_test_request( + ExecutionContext::RegionFailover, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + }); - #[test] - fn transport_security_classification() { - assert!(TransportSecurity::Secure.is_secure()); - assert!(!TransportSecurity::Secure.is_emulator()); - assert!(TransportSecurity::EmulatorWithInsecureCertificates.is_emulator()); - assert!(!TransportSecurity::EmulatorWithInsecureCertificates.is_secure()); + let requested = ctx.requested_regions(); + assert_eq!(requested.len(), 3); + // Dispatch order preserved, duplicates kept. + assert_eq!(requested[0].region, Region::WEST_US_2); + assert_eq!(requested[0].reason, ExecutionContext::Initial); + assert_eq!(requested[1].region, Region::WEST_US_2); + assert_eq!(requested[1].reason, ExecutionContext::OperationRetry); + assert_eq!(requested[2].region, Region::EAST_US_2); + assert_eq!(requested[2].reason, ExecutionContext::RegionFailover); } #[test] - fn transport_kind_classification() { - assert!(TransportKind::Gateway.is_gateway()); - assert!(!TransportKind::Gateway.is_gateway_v2()); - assert!(TransportKind::GatewayV2.is_gateway_v2()); - assert!(!TransportKind::GatewayV2.is_gateway()); - } + fn requested_regions_empty_without_region() { + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + builder.start_test_request( + ExecutionContext::Initial, + None, + "https://test.documents.azure.com", + ); + }); - #[test] - fn transport_http_version_classification() { - assert!(TransportHttpVersion::Http11.is_http11()); - assert!(!TransportHttpVersion::Http11.is_http2()); - assert!(TransportHttpVersion::Http2.is_http2()); - assert!(!TransportHttpVersion::Http2.is_http11()); + assert!(ctx.requested_regions().is_empty()); } #[test] - fn transport_security_default() { + fn responded_regions_excludes_timeouts_and_transport_failures() { + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + // A real service reply. + let h1 = builder.start_test_request( + ExecutionContext::Initial, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.complete_request(h1, StatusCode::Ok, None); + + // A client-side timeout (completed_at set, but no service reply). + let h2 = builder.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.timeout_request(h2); + + // A non-2xx response still counts as a reply from the region. + let h3 = builder.start_test_request( + ExecutionContext::RegionFailover, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h3, StatusCode::NotFound, None); + }); + + let responded = ctx.responded_regions(); + assert_eq!(responded, vec![&Region::WEST_US_2, &Region::EAST_US_2]); + } + + #[test] + fn hedging_started_false_without_hedge_dispatch() { + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let h = builder.start_test_request( + ExecutionContext::Initial, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Ok, None); + }); + + assert!(!ctx.hedging_started()); + } + + #[test] + fn hedging_started_true_when_hedge_dispatched() { + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + builder.start_test_request( + ExecutionContext::Initial, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.start_test_request( + ExecutionContext::Hedging, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + }); + + assert!(ctx.hedging_started()); + } + + fn hedge_config() -> crate::driver::pipeline::hedging_diagnostics::HedgingStrategyConfig { + crate::driver::pipeline::hedging_diagnostics::HedgingStrategyConfig::new( + crate::options::HedgeThreshold::new(std::time::Duration::from_millis(500)) + .expect("500ms is a valid hedge threshold"), + ) + } + + /// Mirrors the orchestrator's STAGE 1: spawn the primary leg's builder and + /// capture the dispatch record the parent keeps for it. + /// + /// Returned separately from [`spawn_alternate_leg`] so a test can dispatch + /// on the primary *between* the two, exactly as the pipeline does — the + /// alternate is only built once the hedge threshold has elapsed. + fn spawn_primary_leg( + parent: &mut DiagnosticsContextBuilder, + region: Option, + reason: ExecutionContext, + ) -> (DiagnosticsContextBuilder, HedgeLegDispatch) { + let leg = parent.clone_for_hedge_attempt(); + let dispatch = leg.leg_dispatch(region, reason); + (leg, dispatch) + } + + /// Mirrors the orchestrator's STAGE 3: spawn the alternate leg's builder + /// once the threshold has elapsed and record the fan-out on the parent. + fn spawn_alternate_leg( + parent: &mut DiagnosticsContextBuilder, + primary: HedgeLegDispatch, + region: Option, + ) -> DiagnosticsContextBuilder { + let leg = parent.clone_for_hedge_attempt(); + let dispatch = leg.leg_dispatch(region, ExecutionContext::Hedging); + parent.record_hedge_fanout(primary, dispatch); + leg + } + + #[test] + fn requested_regions_keeps_dropped_hedge_leg_on_primary_win() { + // PrimaryWonAfterHedge: the alternate leg is cancelled before it + // dispatches anything, so it leaves no attempt behind. Both legs must + // still appear, in dispatch order, from the fan-out the orchestrator + // recorded on the parent. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); + let h = primary.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(h, StatusCode::Ok, None); + // Threshold elapsed, so the alternate launched — and was then + // dropped, unpolled, when the primary won the race. + drop(spawn_alternate_leg( + builder, + primary_dispatch, + Some(Region::WEST_US_2), + )); + builder.merge_hedge_attempt(primary); + builder.set_hedge_diagnostics(HedgeDiagnostics::primary_won_after_hedge( + hedge_config(), + Region::EAST_US_2, + Region::WEST_US_2, + )); + }); + + assert!(ctx.hedging_started()); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::Initial, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: ExecutionContext::Hedging, + }, + ] + ); + // The dropped alternate leg never produced a service reply, so only the + // winning primary appears in responded_regions. + assert_eq!(ctx.responded_regions(), vec![&Region::EAST_US_2]); + } + + #[test] + fn requested_regions_keeps_dropped_primary_leg_on_alternate_win() { + // AlternateWon while the primary was still connecting: the primary leg + // is dropped without having dispatched, so the Initial primary must + // still be listed first (dispatch order) from the fan-out record. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let h = alternate.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.complete_request(h, StatusCode::Ok, None); + drop(primary); + builder.merge_hedge_attempt(alternate); + builder.set_hedge_diagnostics(HedgeDiagnostics::hedge_won( + hedge_config(), + Region::EAST_US_2, + Region::WEST_US_2, + )); + }); + + assert!(ctx.hedging_started()); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::Initial, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: ExecutionContext::Hedging, + }, + ] + ); + // Only the winning alternate produced a service reply. + assert_eq!(ctx.responded_regions(), vec![&Region::WEST_US_2]); + } + + #[test] + fn requested_regions_tags_primary_leg_with_upgrade_reason() { + // A hedge upgraded after a failover retry (STAGE 7) dispatches its + // primary leg as a failover, not as a first attempt — the recorded + // fan-out must carry that reason rather than defaulting to `Initial`. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let h = builder.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::ServiceUnavailable, None); + let (mut primary, primary_dispatch) = spawn_primary_leg( + builder, + Some(Region::WEST_US_2), + ExecutionContext::RegionFailover, + ); + let h = primary.start_test_request( + ExecutionContext::RegionFailover, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + primary.complete_request(h, StatusCode::Ok, None); + drop(spawn_alternate_leg( + builder, + primary_dispatch, + Some(Region::CENTRAL_US), + )); + builder.merge_hedge_attempt(primary); + }); + + assert!(ctx.hedging_started()); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::Initial, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: ExecutionContext::RegionFailover, + }, + RequestedRegion { + region: Region::CENTRAL_US, + reason: ExecutionContext::Hedging, + }, + ] + ); + } + + #[test] + fn requested_regions_bounded_but_exact_under_retry_storm() { + // The dispatch history is materialized from the FULL attempt list, so a + // retry storm that compacts `requests` down to the cap must not lose the + // hedge fan-out recorded mid-storm. But the history itself is also + // bounded (DIAGNOSTICS-CONTRACT.md §8) — the elision keeps head and tail + // and the exact count stays available. + let cap = 16; + let options = Arc::new( + DiagnosticsOptions::builder() + .with_max_request_diagnostics(cap) + .build() + .expect("valid options"), + ); + let mut builder = DiagnosticsContextBuilder::new(ActivityId::new_uuid(), options); + for _ in 0..40 { + let h = builder.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::TooManyRequests, None); + } + let (primary, primary_dispatch) = spawn_primary_leg( + &mut builder, + Some(Region::EAST_US_2), + ExecutionContext::OperationRetry, + ); + let mut alternate = + spawn_alternate_leg(&mut builder, primary_dispatch, Some(Region::WEST_US_2)); + let h = alternate.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.complete_request(h, StatusCode::Ok, None); + drop(primary); + builder.merge_hedge_attempt(alternate); + let ctx = builder.complete(); + + // The retained attempt list really was bounded... + assert!(ctx.requests().len() <= cap); + assert!(ctx.compaction().is_some()); + // ...and so is the dispatch history, independently of attempt count + // (DIAGNOSTICS-CONTRACT.md §8) — but the elision is explicit, not + // silent: the exact count is still reported. 40 retries + the hedge's + // reconstructed primary leg + the alternate's own attempt. + assert_eq!(ctx.total_requested_regions(), 42); + assert_eq!(ctx.requested_regions().len(), cap); + assert_eq!(ctx.total_responded_regions(), 41); + assert_eq!(ctx.responded_regions().len(), cap); + assert!(ctx.hedging_started()); + + // Head and tail survive: the storm's opening dispatch and the hedge + // fan-out that ended it are both still visible. + let requested = ctx.requested_regions(); + assert_eq!(requested[0].region, Region::EAST_US_2); + assert_eq!(requested[0].reason, ExecutionContext::OperationRetry); + assert_eq!( + requested.last().expect("non-empty").reason, + ExecutionContext::Hedging + ); + assert_eq!( + requested.last().expect("non-empty").region, + Region::WEST_US_2 + ); + } + + #[test] + fn region_history_is_bounded_without_a_hedge() { + // The bound is a property of the history itself, not of hedging: a plain + // retry storm must not produce an unbounded artifact either. + let cap = 16; + let options = Arc::new( + DiagnosticsOptions::builder() + .with_max_request_diagnostics(cap) + .build() + .expect("valid options"), + ); + let mut builder = DiagnosticsContextBuilder::new(ActivityId::new_uuid(), options); + for _ in 0..500 { + let h = builder.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::TooManyRequests, None); + } + let ctx = builder.complete(); + + assert_eq!(ctx.total_requested_regions(), 500); + assert_eq!(ctx.requested_regions().len(), cap); + assert_eq!(ctx.total_responded_regions(), 500); + assert_eq!(ctx.responded_regions().len(), cap); + assert!(!ctx.hedging_started()); + } + + #[test] + fn region_history_is_verbatim_under_the_cap() { + // The common path must be untouched: at or below the cap the history is + // exact and the reported total agrees with it. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + for region in [Region::EAST_US_2, Region::WEST_US_2] { + let h = builder.start_test_request( + ExecutionContext::RegionFailover, + Some(region), + "https://test.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Ok, None); + } + }); + + assert_eq!(ctx.requested_regions().len(), 2); + assert_eq!(ctx.total_requested_regions(), 2); + assert_eq!(ctx.responded_regions().len(), 2); + assert_eq!(ctx.total_responded_regions(), 2); + } + + #[test] + fn aggregated_region_history_is_rebounded() { + // Aggregation concatenates each sub-op's already-bounded history, which + // is an independent unbounded path: a PATCH conflict loop adds a sub-op + // per retry. The aggregate must be re-bounded and must report the summed + // exact totals. + let cap = 16; + let options = Arc::new( + DiagnosticsOptions::builder() + .with_max_request_diagnostics(cap) + .build() + .expect("valid options"), + ); + let sub_ops: Vec> = (0..20) + .map(|_| { + let mut builder = + DiagnosticsContextBuilder::new(ActivityId::new_uuid(), Arc::clone(&options)); + for _ in 0..5 { + let h = builder.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Conflict, None); + } + Arc::new(builder.complete()) + }) + .collect(); + + // Each sub-op is individually under the cap, so none is truncated. + assert_eq!(sub_ops[0].requested_regions().len(), 5); + assert_eq!(sub_ops[0].total_requested_regions(), 5); + + let aggregated = + DiagnosticsContext::aggregate_sub_operations(&sub_ops).expect("non-empty sources"); + + // 20 sub-ops x 5 dispatches = 100, bounded back down to the cap. + assert_eq!(aggregated.total_requested_regions(), 100); + assert_eq!(aggregated.requested_regions().len(), cap); + assert_eq!(aggregated.total_responded_regions(), 100); + assert_eq!(aggregated.responded_regions().len(), cap); + } + + #[test] + fn bound_region_history_keeps_head_and_tail() { + // The elision targets the repetitive middle, so both ends survive. + let bounded = bound_region_history((0..100).collect::>(), 10); + assert_eq!(bounded, vec![0, 1, 2, 3, 4, 95, 96, 97, 98, 99]); + + // An odd cap favors the head by one. + let bounded = bound_region_history((0..100).collect::>(), 5); + assert_eq!(bounded, vec![0, 1, 2, 98, 99]); + + // At or under the cap the input is returned verbatim. + let bounded = bound_region_history(vec![1, 2, 3], 10); + assert_eq!(bounded, vec![1, 2, 3]); + let bounded = bound_region_history(vec![1, 2, 3], 3); + assert_eq!(bounded, vec![1, 2, 3]); + } + + #[test] + fn requested_regions_preserves_repeat_dispatches_around_a_fanout() { + // Duplicates are meaningful: each leg's own attempts are listed as they + // were dispatched, and a genuine repeat dispatch to the same region + // under the same reason is never collapsed into a neighbor. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + for _ in 0..2 { + let h = builder.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::TooManyRequests, None); + } + // Both legs dispatch and are harvested back into the parent, so + // both describe themselves and neither is reconstructed... + let (mut primary, primary_dispatch) = spawn_primary_leg( + builder, + Some(Region::EAST_US_2), + ExecutionContext::OperationRetry, + ); + let h = primary.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(h, StatusCode::Ok, None); + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let h = alternate.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.complete_request(h, StatusCode::Ok, None); + builder.merge_hedge_attempt(primary); + builder.merge_hedge_attempt(alternate); + // ...and a later retry to the same region is still its own entry. + let late = builder.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(late, StatusCode::Ok, None); + }); + + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::OperationRetry, + }, + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::OperationRetry, + }, + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::OperationRetry, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: ExecutionContext::Hedging, + }, + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::OperationRetry, + }, + ] + ); + } + + #[test] + fn requested_regions_skips_fanout_without_named_regions() { + // Global-endpoint accounts route to endpoints with no named region; a + // fan-out there contributes no requested-region entries, but still + // counts as a fan-out. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (primary, primary_dispatch) = + spawn_primary_leg(builder, None, ExecutionContext::Initial); + drop(spawn_alternate_leg(builder, primary_dispatch, None)); + drop(primary); + }); + + assert!(ctx.hedging_started()); + assert!(ctx.requested_regions().is_empty()); + } + + #[test] + fn requested_regions_no_recovery_when_no_fanout() { + // PrimaryWonPreThreshold: a strategy was active but no alternate fanned + // out (alternate_region = None), so nothing is recovered and + // hedging_started() is false. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let h = builder.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Ok, None); + builder.set_hedge_diagnostics(HedgeDiagnostics::primary_only( + hedge_config(), + Region::EAST_US_2, + )); + }); + + assert!(!ctx.hedging_started()); + assert_eq!( + ctx.requested_regions(), + vec![RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::Initial, + }] + ); + } + + #[test] + fn hedge_loser_keeps_the_service_reply_it_already_observed() { + // The losing leg is not always empty. A leg that gets a 429, enters the + // transport pipeline's throttle-retry backoff, and only then loses the + // race has *already observed a real service reply*. Dropping its + // builder with the race would erase that reply, under-reporting charge + // and omitting a genuine entry from responded_regions. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); + // Primary is throttled and sleeps before its next attempt. + let throttled = primary.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.update_request(throttled, |req| { + req.request_charge = RequestCharge::new(1.5) + }); + primary.complete_request(throttled, StatusCode::TooManyRequests, None); + + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let won = alternate.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.update_request(won, |req| req.request_charge = RequestCharge::new(2.5)); + alternate.complete_request(won, StatusCode::Ok, None); + + // The alternate wins; `select` drops the primary mid-backoff. + drop(primary); + builder.merge_hedge_attempt(alternate); + builder.set_hedge_diagnostics(HedgeDiagnostics::hedge_won( + hedge_config(), + Region::EAST_US_2, + Region::WEST_US_2, + )); + }); + + // Both attempts survive, in true dispatch order. + assert_eq!(ctx.request_count(), 2); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::Initial, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: ExecutionContext::Hedging, + }, + ] + ); + // The dropped leg's 429 is a real reply from East US 2, so it belongs + // in the arrival-ordered responded history — and both totals are exact. + assert_eq!( + ctx.responded_regions(), + vec![&Region::EAST_US_2, &Region::WEST_US_2] + ); + assert_eq!(ctx.total_requested_regions(), 2); + assert_eq!(ctx.total_responded_regions(), 2); + assert_eq!( + ctx.regions_contacted(), + vec![Region::EAST_US_2, Region::WEST_US_2] + ); + // The throttled attempt's RU is billed even though its leg lost. + assert_eq!(ctx.total_request_charge(), RequestCharge::new(4.0)); + } + + #[test] + fn requested_regions_orders_primary_retry_before_the_alternate() { + // A hedge leg can retry *before* the threshold elapses, so the alternate + // is dispatched after the primary's second attempt. The history must + // report that true order, not group both legs' fan-out ahead of the + // primary's own attempts. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); + let first = primary.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(first, StatusCode::TooManyRequests, None); + let retry = primary.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(retry, StatusCode::Ok, None); + + // Threshold elapses only now, so the alternate is genuinely last. + drop(spawn_alternate_leg( + builder, + primary_dispatch, + Some(Region::WEST_US_2), + )); + builder.merge_hedge_attempt(primary); + }); + + assert!(ctx.hedging_started()); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::Initial, + }, + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::OperationRetry, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: ExecutionContext::Hedging, + }, + ] + ); + } + + #[test] + fn interleaved_leg_retries_report_true_global_dispatch_order() { + // Both legs retry concurrently and the loser is dropped. The finalized + // list must interleave the two legs by real dispatch time rather than + // concatenating one leg after the other. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); + let p1 = primary.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(p1, StatusCode::TooManyRequests, None); + + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let a1 = alternate.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.complete_request(a1, StatusCode::TooManyRequests, None); + + let p2 = primary.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(p2, StatusCode::TooManyRequests, None); + + let a2 = alternate.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.complete_request(a2, StatusCode::Ok, None); + + drop(primary); + builder.merge_hedge_attempt(alternate); + }); + + assert_eq!(ctx.request_count(), 4); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::Initial, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: ExecutionContext::Hedging, + }, + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::OperationRetry, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: ExecutionContext::OperationRetry, + }, + ] + ); + assert_eq!(ctx.total_responded_regions(), 4); + } + + #[test] + fn both_transient_legs_keep_every_observed_reply() { + // BothTransient: neither leg wins, so neither is merged and the parent + // continues into the failover loop. Everything both legs observed must + // still reach the finalized context. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); + let p = primary.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(p, StatusCode::ServiceUnavailable, None); + + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let a = alternate.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.complete_request(a, StatusCode::ServiceUnavailable, None); + + // Both transient: the race returns to the failover loop and neither + // leg builder is merged. + drop(primary); + drop(alternate); + + let failover = builder.start_test_request( + ExecutionContext::RegionFailover, + Some(Region::CENTRAL_US), + "https://test.centralus.documents.azure.com", + ); + builder.complete_request(failover, StatusCode::Ok, None); + }); + + assert_eq!(ctx.request_count(), 3); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::Initial, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: ExecutionContext::Hedging, + }, + RequestedRegion { + region: Region::CENTRAL_US, + reason: ExecutionContext::RegionFailover, + }, + ] + ); + assert_eq!( + ctx.responded_regions(), + vec![&Region::EAST_US_2, &Region::WEST_US_2, &Region::CENTRAL_US] + ); + assert_eq!(ctx.total_responded_regions(), 3); + } + + #[test] + fn hedge_leg_dropped_before_completing_is_not_recovered() { + // Only *observed* completions are recovered. An attempt still in flight + // when its leg is cancelled saw no reply, so it is intentionally absent + // from both histories — reporting it would invent a response. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); + // Dispatched, never completed. + let _in_flight = primary.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let won = alternate.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.complete_request(won, StatusCode::Ok, None); + drop(primary); + builder.merge_hedge_attempt(alternate); + }); + + assert_eq!(ctx.request_count(), 1); + // The primary leg dispatched, so it is not reconstructed from the + // fan-out record either — that fallback is only for a leg that never + // reached the wire at all. + assert_eq!( + ctx.requested_regions(), + vec![RequestedRegion { + region: Region::WEST_US_2, + reason: ExecutionContext::Hedging, + }] + ); + assert_eq!(ctx.responded_regions(), vec![&Region::WEST_US_2]); + } + + #[test] + fn aggregate_sub_operations_propagates_hedge_diagnostics() { + // An aggregated operation (e.g. PATCH) whose sub-op hedged must still + // report hedging so the metric/log/span surfaces stay consistent, even + // though the aggregate is stitched from multiple sub-op contexts. + let read = make_context_with(ActivityId::new_uuid(), |builder| { + let h = builder.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Ok, None); + }); + let patch = make_context_with(ActivityId::new_uuid(), |builder| { + let h = builder.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Ok, None); + builder.set_hedge_diagnostics(HedgeDiagnostics::hedge_won( + hedge_config(), + Region::EAST_US_2, + Region::WEST_US_2, + )); + }); + + let aggregate = + DiagnosticsContext::aggregate_sub_operations(&[Arc::new(read), Arc::new(patch)]) + .expect("non-empty sources"); + + assert!(aggregate.hedging_started()); + let hedge = aggregate + .hedge_diagnostics() + .expect("aggregate must inherit a representative hedge diagnostics"); + assert_eq!( + hedge.terminal_state(), + crate::driver::pipeline::hedging_diagnostics::HedgeTerminalState::AlternateWon, + ); + } + + #[test] + fn aggregate_sub_operations_keeps_every_sub_op_fanout() { + // Only one representative `hedge_diagnostics` survives aggregation, so + // the Hedging Detection API must not depend on it: each sub-op's own + // materialized dispatch history is concatenated in sub-op order. + let read = make_context_with(ActivityId::new_uuid(), |builder| { + let (primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let h = alternate.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.complete_request(h, StatusCode::Ok, None); + drop(primary); + builder.merge_hedge_attempt(alternate); + builder.set_hedge_diagnostics(HedgeDiagnostics::hedge_won( + hedge_config(), + Region::EAST_US_2, + Region::WEST_US_2, + )); + }); + let replace = make_context_with(ActivityId::new_uuid(), |builder| { + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); + let h = primary.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(h, StatusCode::Ok, None); + drop(spawn_alternate_leg( + builder, + primary_dispatch, + Some(Region::CENTRAL_US), + )); + builder.merge_hedge_attempt(primary); + builder.set_hedge_diagnostics(HedgeDiagnostics::primary_won_after_hedge( + hedge_config(), + Region::EAST_US_2, + Region::CENTRAL_US, + )); + }); + + let aggregate = + DiagnosticsContext::aggregate_sub_operations(&[Arc::new(read), Arc::new(replace)]) + .expect("non-empty sources"); + + assert!(aggregate.hedging_started()); + // Both fan-outs survive, including the non-representative one. + assert_eq!( + aggregate.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::Initial, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: ExecutionContext::Hedging, + }, + RequestedRegion { + region: Region::EAST_US_2, + reason: ExecutionContext::Initial, + }, + RequestedRegion { + region: Region::CENTRAL_US, + reason: ExecutionContext::Hedging, + }, + ] + ); + assert_eq!( + aggregate.responded_regions(), + vec![&Region::WEST_US_2, &Region::EAST_US_2] + ); + } + + // ========================================================================= + // Pipeline/Transport/RequestSentStatus tests (merged from request_diagnostics.rs) + // ========================================================================= + + #[test] + fn pipeline_type_classification() { + assert!(PipelineType::Metadata.is_metadata()); + assert!(!PipelineType::Metadata.is_data_plane()); + assert!(PipelineType::DataPlane.is_data_plane()); + assert!(!PipelineType::DataPlane.is_metadata()); + } + + #[test] + fn transport_security_classification() { + assert!(TransportSecurity::Secure.is_secure()); + assert!(!TransportSecurity::Secure.is_emulator()); + assert!(TransportSecurity::EmulatorWithInsecureCertificates.is_emulator()); + assert!(!TransportSecurity::EmulatorWithInsecureCertificates.is_secure()); + } + + #[test] + fn transport_kind_classification() { + assert!(TransportKind::Gateway.is_gateway()); + assert!(!TransportKind::Gateway.is_gateway_v2()); + assert!(TransportKind::GatewayV2.is_gateway_v2()); + assert!(!TransportKind::GatewayV2.is_gateway()); + } + + #[test] + fn transport_http_version_classification() { + assert!(TransportHttpVersion::Http11.is_http11()); + assert!(!TransportHttpVersion::Http11.is_http2()); + assert!(TransportHttpVersion::Http2.is_http2()); + assert!(!TransportHttpVersion::Http2.is_http11()); + } + + #[test] + fn transport_security_default() { assert_eq!(TransportSecurity::default(), TransportSecurity::Secure); } @@ -4587,7 +6436,7 @@ mod tests { ); record_run( &mut read_b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", "https://east/", CosmosStatus::new(StatusCode::Ok), @@ -4608,7 +6457,7 @@ mod tests { ); record_run( &mut replace_b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", "https://east/", CosmosStatus::new(StatusCode::Ok), @@ -4659,7 +6508,7 @@ mod tests { ); record_run( &mut b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", "https://east/", CosmosStatus::new(StatusCode::TooManyRequests), @@ -4726,7 +6575,7 @@ mod tests { ); record_run( &mut b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", "https://east/", CosmosStatus::new(StatusCode::TooManyRequests), @@ -4735,7 +6584,7 @@ mod tests { ); record_run( &mut b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", "https://east/", CosmosStatus::new(StatusCode::Gone).with_sub_status(1002), @@ -4744,7 +6593,7 @@ mod tests { ); record_run( &mut b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "West US", "https://west/", CosmosStatus::new(StatusCode::Ok), @@ -4797,13 +6646,13 @@ mod tests { ); for _ in 0..200 { let he = b.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::new("East US")), "https://east/", ); b.complete_request(he, StatusCode::ServiceUnavailable, None); let hw = b.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::new("West US")), "https://west/", ); @@ -4848,7 +6697,7 @@ mod tests { for i in 0..distinct { record_run( &mut b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", &format!("https://pkrange-{i}/"), CosmosStatus::new(StatusCode::Gone).with_sub_status(1002), @@ -4912,7 +6761,7 @@ mod tests { for i in 0..cold { let h = b.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::new("East US")), &format!("https://cold-{i}/"), ); @@ -4922,7 +6771,7 @@ mod tests { for _round in 0..hot_retries { for j in 0..hot { let h = b.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::new("West US")), &format!("https://hot-{j}/"), ); @@ -5002,7 +6851,7 @@ mod tests { let ctx = make_context_with(ActivityId::from_string("normal".to_string()), |b| { for _ in 0..3 { let h = b.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::new("East US")), "https://east/", ); @@ -5131,7 +6980,7 @@ mod tests { "https://test.westus2.documents.azure.com", ); builder.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::WEST_US_2), "https://test.westus2.documents.azure.com", ); diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs index 8581941402..f671ce376a 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs @@ -27,7 +27,7 @@ pub(crate) use diagnostics_context::DiagnosticsContextBuilder; pub use diagnostics_context::{ DiagnosticsContext, ExecutionContext, FailedTransportShardDiagnostics, PipelineType, RequestDiagnostics, RequestEvent, RequestEventType, RequestHandle, RequestSentStatus, - ThresholdBreach, TransportHttpVersion, TransportKind, TransportSecurity, + RequestedRegion, ThresholdBreach, TransportHttpVersion, TransportKind, TransportSecurity, TransportShardDiagnostics, }; pub use proxy_configuration::ProxyConfiguration; diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/cosmos_driver.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/cosmos_driver.rs index 4a1ddc4694..e9659531ad 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/cosmos_driver.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/cosmos_driver.rs @@ -874,7 +874,7 @@ impl CosmosDriver { .unwrap_or(azure_core::time::Duration::ZERO), ) .await; - execution_context = ExecutionContext::Retry; + execution_context = ExecutionContext::OperationRetry; continue; } } diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/hedging_diagnostics.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/hedging_diagnostics.rs index 0d2310d18c..f0f49bb72c 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/hedging_diagnostics.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/hedging_diagnostics.rs @@ -91,6 +91,31 @@ pub enum HedgeTerminalState { CancelledAwaitingPartner, } +impl HedgeTerminalState { + /// Returns a stable, low-cardinality `snake_case` identifier for this + /// terminal state, suitable as an observability attribute / log-field value. + /// + /// The [`BothTransient`](Self::BothTransient) variant collapses to + /// `"both_transient"` regardless of its `deadline_elapsed` payload, keeping + /// the emitted value set bounded to one string per variant. + pub fn as_str(&self) -> &'static str { + match self { + HedgeTerminalState::PrimaryWonPreThreshold => "primary_won_pre_threshold", + HedgeTerminalState::DeadlineExceededPreThreshold => "deadline_exceeded_pre_threshold", + HedgeTerminalState::PrimaryWonAfterHedge => "primary_won_after_hedge", + HedgeTerminalState::AlternateWon => "alternate_won", + HedgeTerminalState::BothTransient { .. } => "both_transient", + HedgeTerminalState::CancelledAwaitingPartner => "cancelled_awaiting_partner", + } + } +} + +impl std::fmt::Display for HedgeTerminalState { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(self.as_str()) + } +} + /// Diagnostic information about a hedging execution, attached to the /// winning response when a hedging strategy was active for the operation. /// @@ -294,6 +319,35 @@ impl HedgeDiagnostics { pub fn terminal_state(&self) -> HedgeTerminalState { self.terminal_state } + + /// **Internal test helper — do not call.** + /// + /// Builds a [`HedgeDiagnostics`] from explicit parts so the wrapper SDK + /// (`azure_data_cosmos`) can exercise the observability layer's hedging + /// surfacing (tracing / metrics / logging) against a realistic hedged + /// context without standing up the hedging pipeline. The strategy config is + /// filled with a fixed non-zero threshold since the emission layer does not + /// read it. Gated behind the `__internal_test_diagnostics_construction` + /// Cargo feature and `#[doc(hidden)]`, so it never appears on the public + /// surface. + #[cfg(feature = "__internal_test_diagnostics_construction")] + #[doc(hidden)] + pub fn for_testing( + primary_region: Region, + alternate_region: Option, + response_region: Option, + terminal_state: HedgeTerminalState, + ) -> Self { + let threshold = HedgeThreshold::new(std::time::Duration::from_millis(500)) + .expect("500ms is a valid non-zero hedge threshold"); + Self { + strategy_config: HedgingStrategyConfig::new(threshold), + primary_region, + alternate_region, + response_region, + terminal_state, + } + } } #[cfg(test)] @@ -484,6 +538,50 @@ mod tests { )); } + #[test] + fn terminal_state_as_str_is_stable_for_every_variant() { + // These strings are an observability contract (metric/log/span values), + // so pin all six. `BothTransient` collapses to a single value regardless + // of its `deadline_elapsed` payload to keep the emitted value set bounded. + assert_eq!( + HedgeTerminalState::PrimaryWonPreThreshold.as_str(), + "primary_won_pre_threshold" + ); + assert_eq!( + HedgeTerminalState::DeadlineExceededPreThreshold.as_str(), + "deadline_exceeded_pre_threshold" + ); + assert_eq!( + HedgeTerminalState::PrimaryWonAfterHedge.as_str(), + "primary_won_after_hedge" + ); + assert_eq!(HedgeTerminalState::AlternateWon.as_str(), "alternate_won"); + assert_eq!( + HedgeTerminalState::CancelledAwaitingPartner.as_str(), + "cancelled_awaiting_partner" + ); + assert_eq!( + HedgeTerminalState::BothTransient { + deadline_elapsed: true + } + .as_str(), + "both_transient" + ); + assert_eq!( + HedgeTerminalState::BothTransient { + deadline_elapsed: false + } + .as_str(), + "both_transient", + "both_transient must collapse regardless of deadline_elapsed" + ); + // Display delegates to as_str(). + assert_eq!( + HedgeTerminalState::AlternateWon.to_string(), + "alternate_won" + ); + } + #[test] fn debug_clone_round_trip() { let diag = HedgeDiagnostics::hedge_won(config(), Region::EAST_US, Region::WEST_US_2); diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs index 9df48a4bbb..253fb42dff 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs @@ -2043,7 +2043,7 @@ fn should_capture_session_token_from_status( /// transport pipeline expects for diagnostics annotation. /// /// - First attempt (no failover, no session retry) → `Initial` -/// - Any session retry in progress → `Retry` +/// - Any session retry in progress → `OperationRetry` /// - Otherwise (a failover retry) → `RegionFailover` /// /// Session-retry takes precedence over failover-retry because in the rare @@ -2054,7 +2054,7 @@ fn compute_execution_context(retry_state: &OperationRetryState) -> ExecutionCont if retry_state.failover_retry_count == 0 && retry_state.session_token_retry_count == 0 { ExecutionContext::Initial } else if retry_state.session_token_retry_count > 0 { - ExecutionContext::Retry + ExecutionContext::OperationRetry } else { ExecutionContext::RegionFailover } @@ -3315,7 +3315,20 @@ async fn execute_hedged( // ── Stage 1: Build the primary future ───────────────────────────── // The diag clone is owned by the future and returned alongside the // result, so the borrow checker can reclaim it after `select` resolves. + // + // The primary leg inherits the execution context the *non-hedged* path + // would have used for this same attempt. A hedge upgraded at STAGE 7 + // (after `advance_to_next_attempt`) is a failover or session retry, so + // hard-coding `Initial` here would misreport it in diagnostics; on the + // STAGE 2b path the retry counters are still zero and this yields + // `Initial` exactly as before. + let primary_execution_context = compute_execution_context(retry_state_snapshot); let primary_diag = parent_diagnostics.clone_for_hedge_attempt(); + // Describe the primary as a race participant before its builder is moved + // into the future. `leg_dispatch` reads the leg's launch instant, so the + // fan-out record orders correctly against every attempt in the operation. + let primary_dispatch = + primary_diag.leg_dispatch(primary_region.clone(), primary_execution_context); let primary_attempt = Box::pin(async move { let mut diag = primary_diag; // Primary is launched before Stage 2 elapses, so no shared @@ -3325,7 +3338,7 @@ async fn execute_hedged( let result = perform_single_attempt( ctx, primary_routing, - ExecutionContext::Initial, + primary_execution_context, None, &mut diag, ) @@ -3513,7 +3526,21 @@ async fn execute_hedged( ) .then(|| Arc::new(AtomicBool::new(ctx.hub_region_processing_only_initial))); let secondary_shared_latch = shared_hub_region_latch.clone(); + // Record the fan-out on the *parent* before the race runs, so the operation + // is known to have hedged regardless of which leg won. Each leg's own + // attempts survive the race via the diagnostics hedge journal, so this + // record only has to reconstruct a leg cancelled before it dispatched + // anything — most commonly the alternate, which `select` never polls when + // the primary is already resolved. The parent builder reaches every exit + // path (`finalize_hedge_attempt` on Terminal, `diagnostics: + // parent_diagnostics` on BothTransient), so the record always survives to + // `complete()`. let secondary_diag = parent_diagnostics.clone_for_hedge_attempt(); + let secondary_dispatch = secondary_diag.leg_dispatch( + secondary_region.clone(), + crate::diagnostics::ExecutionContext::Hedging, + ); + parent_diagnostics.record_hedge_fanout(primary_dispatch, secondary_dispatch); let secondary_attempt = Box::pin(async move { let mut diag = secondary_diag; let result = perform_single_attempt( @@ -4500,7 +4527,7 @@ mod tests { let ctx = TransportRequestContext { routing: &routing, activity_id: &activity_id, - execution_context: ExecutionContext::Retry, + execution_context: ExecutionContext::OperationRetry, deadline: Some(std::time::Instant::now() + Duration::from_secs(5)), effective_consistency: DefaultConsistencyLevel::Session, read_consistency_strategy: crate::options::ReadConsistencyStrategy::Default, @@ -4533,7 +4560,7 @@ mod tests { let ctx = TransportRequestContext { routing: &routing, activity_id: &activity_id, - execution_context: ExecutionContext::Retry, + execution_context: ExecutionContext::OperationRetry, deadline: Some(std::time::Instant::now() + Duration::from_secs(5)), effective_consistency: DefaultConsistencyLevel::Session, read_consistency_strategy: crate::options::ReadConsistencyStrategy::Default, @@ -4595,7 +4622,7 @@ mod tests { let ctx = TransportRequestContext { routing: &routing, activity_id: &activity_id, - execution_context: ExecutionContext::Retry, + execution_context: ExecutionContext::OperationRetry, deadline: Some(std::time::Instant::now() + Duration::from_secs(5)), effective_consistency: DefaultConsistencyLevel::Session, read_consistency_strategy: crate::options::ReadConsistencyStrategy::Default, @@ -4645,7 +4672,7 @@ mod tests { let ctx = TransportRequestContext { routing: &routing, activity_id: &activity_id, - execution_context: ExecutionContext::Retry, + execution_context: ExecutionContext::OperationRetry, deadline: Some(std::time::Instant::now() + Duration::from_secs(5)), effective_consistency: DefaultConsistencyLevel::Session, read_consistency_strategy: crate::options::ReadConsistencyStrategy::Default, @@ -8446,17 +8473,17 @@ mod tests { fn execution_context_retry_when_session_retry_active() { // Session-retry takes precedence over failover-retry: when both // counters are non-zero, the most recent advance was the session - // retry, so the attempt is annotated as a `Retry`. + // retry, so the attempt is annotated as an `OperationRetry`. let state = retry_state_with_counts(1, 1); assert!(matches!( super::compute_execution_context(&state), - ExecutionContext::Retry + ExecutionContext::OperationRetry )); let state = retry_state_with_counts(0, 1); assert!(matches!( super::compute_execution_context(&state), - ExecutionContext::Retry + ExecutionContext::OperationRetry )); } @@ -9137,7 +9164,7 @@ mod tests { #[test] fn diagnostics_clone_for_hedge_attempt_starts_empty() { - let parent = test_diagnostics(); + let mut parent = test_diagnostics(); let child = parent.clone_for_hedge_attempt(); // A fresh sub-builder must not carry the parent's request list, // status, or accumulated hedge diagnostics. diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/transport/transport_pipeline.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/transport/transport_pipeline.rs index 5de5b84afa..796d8230d8 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/transport/transport_pipeline.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/transport/transport_pipeline.rs @@ -232,7 +232,7 @@ pub(crate) async fn execute_transport_pipeline( } else if throttle_state.attempt_count == 0 { request.execution_context } else { - ExecutionContext::Retry + ExecutionContext::OperationRetry }; let request_handle = diagnostics.start_request(