Add Cosmos hedging detection API + observability surfacing

sdk/cosmos/.cspell.json

@@ -116,6 +116,7 @@
     "failback",
     "failovers",
     "fanout",
+    "fanouts",
     "Fatalf",
     "fieldless",
     "FILETIME",
@@ -330,6 +331,7 @@
     "uncollapsed",
     "uncontended",
     "undecoded",
+    "undercount",
     "underspecified",
     "undrained",
     "unfaulted",

sdk/cosmos/azure_data_cosmos/CHANGELOG.md

@@ -7,21 +7,25 @@
 - Added an SDK-generated `x-ms-client-id` header that remains stable for each `CosmosClient`. ([#4844](https://github.com/Azure/azure-sdk-for-rust/pull/4844))
 - Added opt-in Cosmos binary JSON encoding for item operations (`create`/`read`/`replace`/`upsert`). Enable it via `CosmosClientBuilder::with_binary_encoding_options` (or the `AZURE_COSMOS_BINARY_ENCODING_ENABLED` environment-variable fallback). Off by default; when disabled, requests and responses are byte-for-byte unchanged. ([#4671](https://github.com/Azure/azure-sdk-for-rust/pull/4671))
 - Added `FeedOptions::max_fan_out` (and `FeedOptions::with_max_fan_out`) to cap how many physical partitions a cross-partition query or change feed may fan out to. Applies to `ContainerClient::query_items` and `ContainerClient::query_change_feed`. The cap is enforced only at initial query setup; a partition that splits mid-execution and pushes the fan-out higher does not abort the operation. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855))
+- Surfaced the Hedging Detection API through `DiagnosticsContext`: `hedging_started()`, `requested_regions()`, `responded_regions()`, and the exact-count `total_requested_regions()` / `total_responded_regions()`, and re-exported the `RequestedRegion` / `RequestedRegionReason` types (mirroring the existing `DiagnosticsContext` re-export). ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871))
 - Added a pluggable client-side diagnostics emission layer — the `DiagnosticsHandler` trait and ordered `DiagnosticsHandlerChain` (registered via `CosmosClientBuilder::with_diagnostics_handler`) — invoked once per operation (singleton and paginated, on success and failure) with the completed `DiagnosticsContext` plus an SDK-supplied `CosmosOperationContext`; the empty default chain is a zero-overhead no-op. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789))
 - Added the `metrics`-gated `CosmosMetricsHandler` (with `MetricsOptions`), emitting the stable `db.client.operation.duration` histogram plus per-signal opt-in metrics (`with_request_charge_metric`, `with_returned_rows_metric`) and an opt-in extended attribute set (`with_extended_attributes`); a no-op when no meter provider is registered. Each histogram declares explicit bucket boundaries (`attributes::BUCKETS_OPERATION_DURATION_SECONDS`, `BUCKETS_REQUEST_CHARGE_RU`, `BUCKETS_RETURNED_ROWS`) rather than inheriting OpenTelemetry's millisecond-scaled defaults, which would collapse every real observation of the seconds-valued duration metric into a single bucket and make latency percentiles constant. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789))
 - Added `MetricsOptions::with_active_instance_metric`, an opt-in `azure.cosmosdb.client.active_instance.count` up-down counter reporting the number of live `CosmosClient` instances per account endpoint, keyed on `server.address` (plus `server.port` for a non-default port). ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874))
 - Added `DiagnosticsHandler::on_client_created`, a defaulted hook that lets a handler observe client construction (`CosmosClientInfo`) and return a `ClientLifetimeToken` dropped with the client, for handlers that need to track client lifetime. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874))
 - Added composable tail-sampled emission handlers — a `TracingLogHandler` leaf that writes a compact `tracing` line and a `SamplingLogHandler` wrapper (holding an `Arc<dyn DiagnosticsHandler>`) that applies the sampling gate plus a shared per-window rate limit, defaulting to wrap a `TracingLogHandler` — and the `distributed_tracing`-gated `CosmosTracingHandler` (backdated span tree), also rate-limited so an error storm can't overwhelm exporters. All emit only for operations which fail or breach a configurable `DiagnosticsThresholds`, and stamp *why* they were sampled (a failure, or which threshold) on the emitted line and span. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789))
+- Surfaced hedging through the observability handlers, reusing the Hedging Detection API. When a cross-region hedge fans out, `CosmosTracingHandler` adds `azure.cosmosdb.operation.{hedging_started,hedge_region,hedge_terminal_state}` plus `requested_regions`/`responded_regions` (`string[]`, bounded by the driver's `max_request_diagnostics`, with a `*_total` companion attribute emitted only when a retry storm truncated the history) to the sampled operation span and tags the hedge-leg child span (`azure.cosmosdb.request.hedge`); `SamplingLogHandler` adds `hedging_started` / `hedge_region` / `hedge_terminal_state` fields to the sampled log line; and `CosmosMetricsHandler` gains an opt-in `azure.cosmosdb.client.operation.hedged` counter (`MetricsOptions::with_hedged_metric`) carrying the low-cardinality `hedge_terminal_state`, with the higher-cardinality `hedge_region` dimension added only under `with_extended_attributes`. All three surfaces decide fan-out from `hedging_started()`, so a hedge whose race ended both-transient and was then resolved by a failover attempt is still reported; the per-outcome `hedge_region` / `hedge_terminal_state` fields are omitted there (the counter's dimension carries the `unresolved` sentinel so its attribute schema stays uniform). ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871))
 - Added the non-default `control_plane` feature that gates the control-plane APIs (database and container CRUD, and throughput/offer management). It is intentionally independent of `key_auth` so these APIs are not tied to key-based authentication. ([#4854](https://github.com/Azure/azure-sdk-for-rust/pull/4854))
 
 ### Breaking Changes
 
+- Serialized `DiagnosticsContext` output (the diagnostics JSON surfaced to consumers, and the sampled diagnostics log line) now serializes driver-generated operation retries with `execution_context` = `"operation_retry"` instead of `"retry"`. The driver's `ExecutionContext::Retry` variant has been removed entirely; use `ExecutionContext::OperationRetry` instead. Telemetry/log parsers that match the literal `"retry"` execution context must update. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871))
 - Control-plane APIs are now gated behind the new `control_plane` feature, which is **not** enabled by default. Code using database or container management (`CosmosClient::create_database`/`query_databases`, `DatabaseClient::read`/`create_container`/`query_containers`/`delete`, `ContainerClient::replace`/`delete`), throughput management (`read_throughput`/`begin_replace_throughput`, `ThroughputPoller`), or the associated model and options types (`DatabaseProperties`, `ThroughputProperties`, and the container create/replace/delete/query, database, and throughput option types) must now enable the `control_plane` feature. Reading container properties via `ContainerClient::read()` — along with `ContainerProperties`, `IndexingPolicy`, `ResourceResponse`, and `ReadContainerOptions` — remains available without the feature, since it works with Entra ID authentication and mirrors the metadata read the SDK already performs internally. ([#4854](https://github.com/Azure/azure-sdk-for-rust/pull/4854))
 - Fresh cross-partition queries and change feed reads now fail with a `BadRequest` error if they would fan out to more than 100 physical partitions. Raise `FeedOptions::max_fan_out` to run a broader operation. The limit is checked only at initial query setup — resuming from a continuation token is unaffected, and a partition split that raises the fan-out mid-execution does not abort the operation. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855))
 
 ### Bugs Fixed
 
 - The Cosmos tracing span's operation label now prefers the caller-facing `CosmosOperationContext` identity over the driver-recorded name, matching how the `db.operation.name` metric attribute is resolved. Previously an aggregate whose surfaced sub-operation differed from the caller's operation — such as a PATCH that fails during its internal read — could label the span `read_item` while the metric reported `patch_item`. Attempt spans now carry the operation that issued them, so a PATCH's attempts report `db.operation.name` of `patch_read_item` / `patch_replace_item` while its operation span and metric stay `patch_item`; attempts of every other operation continue to inherit the operation's own name. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874))
+- Fixed hedged operations under-reporting diagnostics for the leg that lost the race. Cancelling the loser previously discarded every attempt it had already completed, so a leg that received a `429` and was retrying when it lost contributed nothing to the surfaced `DiagnosticsContext` — omitting its region from `regions_contacted()`/`responded_regions()` and under-reporting `total_request_charge()` against RU the account was actually billed. Attempts observed by a cancelled leg are now retained. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871))
 
 ### Other Changes