From d158482f73a0c4770400b6a61774663680f8d7c8 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Mon, 27 Jul 2026 09:50:33 -0700 Subject: [PATCH 01/21] Add Cosmos hedging detection API to driver Re-implement the driver-side Hedging Detection API (H1) on current main, which already carries the merged observability layer and landed hedging implementation (#4432). Refs #4410. Driver (azure_data_cosmos_driver): - Rename `ExecutionContext::Retry` -> `OperationRetry`; keep `Retry` for one release as a `#[deprecated]` alias. Serialized form changes "retry" -> "operation_retry"; update `as_str()` and all dispatch sites in the operation/transport pipelines and driver retry loop. - Add public `RequestedRegion` struct and `RequestedRegionReason` enum (both `#[non_exhaustive]`) plus a total `From` mapping. - Add `DiagnosticsContext::requested_regions()` (dispatch order, duplicates, per-region reason), `responded_regions()` (completion order, service replies only via a new `RequestDiagnostics::responded_with_service_reply()` predicate), and `hedging_started()` (alternate-region / Hedging predicate). - Export the two new types from the diagnostics module. - Add unit tests for the accessors, mapping totality, and the rename; refresh ARCHITECTURE.md and the "retry" JSON expectations. SDK (azure_data_cosmos): - Re-export `RequestedRegion` / `RequestedRegionReason`, mirroring the existing `DiagnosticsContext` re-export. Docs/CHANGELOGs: - Add `docs/HEDGING_DETECTION_API_SPEC.md` reflecting main's reality and the landed API; add CHANGELOG entries to both crates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- sdk/cosmos/azure_data_cosmos/CHANGELOG.md | 1 + .../docs/HEDGING_DETECTION_API_SPEC.md | 191 ++++++++++ .../azure_data_cosmos/src/diagnostics/mod.rs | 2 +- .../azure_data_cosmos_driver/ARCHITECTURE.md | 26 +- .../azure_data_cosmos_driver/CHANGELOG.md | 4 + .../src/diagnostics/diagnostics_context.rs | 356 +++++++++++++++++- .../src/diagnostics/mod.rs | 4 +- .../src/driver/cosmos_driver.rs | 2 +- .../src/driver/pipeline/operation_pipeline.rs | 16 +- .../driver/transport/transport_pipeline.rs | 2 +- 10 files changed, 558 insertions(+), 46 deletions(-) create mode 100644 sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md diff --git a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md index 9f7282ce222..78b45a7e580 100644 --- a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md @@ -4,6 +4,7 @@ ### Features Added +- Surfaced the Hedging Detection API through `DiagnosticsContext`: `hedging_started()`, `requested_regions()`, and `responded_regions()`, and re-exported the `RequestedRegion` / `RequestedRegionReason` types (mirroring the existing `DiagnosticsContext` re-export). ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) - Added a pluggable client-side diagnostics emission layer — the `DiagnosticsHandler` trait and ordered `DiagnosticsHandlerChain` (registered via `CosmosClientBuilder::with_diagnostics_handler`) — invoked once per operation (singleton and paginated, on success and failure) with the completed `DiagnosticsContext` plus an SDK-supplied `CosmosOperationContext`; the empty default chain is a zero-overhead no-op. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added the `metrics`-gated `CosmosMetricsHandler` (with `MetricsOptions`), emitting the stable `db.client.operation.duration` histogram plus per-signal opt-in metrics (`with_request_charge_metric`, `with_returned_rows_metric`) and an opt-in extended attribute set (`with_extended_attributes`); a no-op when no meter provider is registered. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added composable tail-sampled emission handlers — a `TracingLogHandler` leaf that writes a compact `tracing` line and a `SamplingLogHandler` wrapper (holding an `Arc`) that applies the sampling gate plus a shared per-window rate limit, defaulting to wrap a `TracingLogHandler` — and the `distributed_tracing`-gated `CosmosTracingHandler` (backdated span tree), also rate-limited so an error storm can't overwhelm exporters. All emit only for operations which fail or breach a configurable `DiagnosticsThresholds`, and stamp *why* they were sampled (a failure, or which threshold) on the emitted line and span. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) diff --git a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md new file mode 100644 index 00000000000..77339c8016f --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md @@ -0,0 +1,191 @@ +# Hedging Detection API — Spec + +**Status:** Implemented on `main`. +**Tracking issue:** [Azure/azure-sdk-for-rust#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410) +**Cross-SDK contract:** The Azure Cosmos DB SDKs are converging on a "Hedging +Detection" capability exposed on each SDK's per-operation diagnostics surface. +This document specifies how the Rust SDK satisfies that contract. + +--- + +## 1. Summary + +The Rust Cosmos SDK surfaces hedging and per-region dispatch/response history +from a single [`DiagnosticsContext`]. This capability is realized by three +inherent accessors on the driver's `DiagnosticsContext` plus two small public +value types, all re-exported from `azure_data_cosmos`: + +| Member | Signature | Semantics | +| --- | --- | --- | +| `DiagnosticsContext::hedging_started` | `-> bool` | `true` iff a hedge arm was actually dispatched (fan-out happened). | +| `DiagnosticsContext::requested_regions` | `-> Vec` | Regions dispatched to, in **dispatch order**, duplicates allowed, each tagged with a reason. | +| `DiagnosticsContext::responded_regions` | `-> Vec<&Region>` | Regions that produced a **service reply**, in **completion order**, duplicates allowed. | +| `RequestedRegion` | `{ region, reason }` | A dispatched region paired with the reason it was chosen. | +| `RequestedRegionReason` | enum | Why the SDK dispatched to a region; `#[non_exhaustive]`. | + +These build on the existing per-operation building blocks and coexist with the +Rust-native `HedgeDiagnostics` surface (see §5); the two are complementary. + +--- + +## 2. Building blocks (already public on `main`) + +| Item | Signature | Notes | +| --- | --- | --- | +| `DiagnosticsContext` | re-exported as `azure_data_cosmos::DiagnosticsContext` | The per-operation diagnostics handle. | +| `DiagnosticsContext::requests` | `-> Arc>` | All dispatched attempts, in **dispatch order** (append-only). Cloning the `Arc` is a cheap atomic increment. | +| `DiagnosticsContext::hedge_diagnostics` | `-> Option<&HedgeDiagnostics>` | `Some` whenever a hedging strategy was active for the operation (including primary-wins-under-threshold). | +| `DiagnosticsContext::regions_contacted` | `-> Vec` | **Sorted and deduplicated** distinct regions — not dispatch order. | +| `RequestDiagnostics::region` | `-> Option<&Region>` | `None` for pre-region-selection failures. | +| `RequestDiagnostics::execution_context` | `-> ExecutionContext` | Why this attempt was dispatched (see §3). | +| `RequestDiagnostics::completed_at` | `-> Option` | Set by `complete()`, `timeout()`, **and** `fail_transport()` — so "completed" alone is not "responded" (see §4.3). | +| `RequestDiagnostics::timed_out` | `-> bool` | `true` for a client-side end-to-end timeout. | +| `RequestDiagnostics::error` | `-> Option<&str>` | `Some` for a transport-level failure with no service reply. | +| `HedgeDiagnostics::primary_region` | `-> &Region` | The primary leg's region (unknown-region sentinel for global-endpoint accounts). | +| `HedgeDiagnostics::alternate_region` | `-> Option<&Region>` | `Some` exactly when the orchestrator dispatched an alternate hedge leg (fan-out happened). | +| `HedgeDiagnostics::response_region` | `-> Option<&Region>` | The single winning region, when a leg produced a final response. | +| `HedgeDiagnostics::terminal_state` | `-> HedgeTerminalState` | Authoritative race outcome. | + +Because these are inherent methods on the driver's `DiagnosticsContext` and the +SDK depends on the driver (never the reverse), the diagnostics model is +driver-owned and re-exported by `azure_data_cosmos`, exactly like +`DiagnosticsContext` itself. + +The hedging orchestrator/dispatch is **landed** on `main` +([#4432](https://github.com/Azure/azure-sdk-for-rust/pull/4432)): it emits +`ExecutionContext::Hedging` for alternate legs and populates `HedgeDiagnostics` +(design: [`HEDGING_SPEC.md`](../../azure_data_cosmos_driver/docs/HEDGING_SPEC.md), +[PR #4330](https://github.com/Azure/azure-sdk-for-rust/pull/4330)). + +--- + +## 3. The `Retry → OperationRetry` rename + +`ExecutionContext` is the per-request "why" returned by +`RequestDiagnostics::execution_context()`: + +```rust +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize)] +#[serde(rename_all = "snake_case")] +#[non_exhaustive] +pub enum ExecutionContext { + Initial, + #[deprecated(since = "0.7.0", note = "use `ExecutionContext::OperationRetry`")] + Retry, + OperationRetry, // was: Retry + TransportRetry, + Hedging, + RegionFailover, + CircuitBreakerProbe, +} +``` + +`Retry` is renamed to `OperationRetry` so the operation-level retry reason is +clearly distinct from the transport-level `TransportRetry`. The hand-written +`ExecutionContext::as_str()` and every dispatch site (`operation_pipeline.rs`, +`transport_pipeline.rs`, `cosmos_driver.rs`) are updated accordingly. + +**Compatibility.** The old `Retry` variant is retained for one release as a +`#[deprecated]` alias so existing source keeps compiling. The **serialized form +changes from `"retry"` to `"operation_retry"`**: telemetry parsers that match on +the literal `"retry"` must update. (`ExecutionContext` derives `Serialize` only, +not `Deserialize`, so no `#[serde(alias)]` is needed.) + +--- + +## 4. Detection recipes → API + +### 4.1 Reason mapping — `RequestedRegionReason` + +`requested_regions()` tags each dispatched region with a `RequestedRegionReason`, +projected from the driver-internal `ExecutionContext` via a **total** +`From` mapping: + +| `ExecutionContext` | `RequestedRegionReason` | +| --- | --- | +| `Initial` | `Initial` | +| `Retry` (deprecated) / `OperationRetry` | `OperationRetry` | +| `TransportRetry` | `TransportRetry` | +| `Hedging` | `Hedging` | +| `RegionFailover` | `RegionFailover` | +| `CircuitBreakerProbe` | `CircuitBreakerProbe` | + +The mapping is total (no wildcard arm) so it fails to compile if a new +`ExecutionContext` variant is added without a corresponding reason. + +### 4.2 Did fan-out happen? — `hedging_started()` + +`true` iff at least one hedge arm was actually dispatched. This is `false` — not +an error — when the primary returns before the hedging threshold elapses, even +though a hedging strategy was active. To check whether a strategy was merely +*configured*, use `ctx.hedge_diagnostics().is_some()` (a superset that includes +primary-wins-under-threshold). + +The result is the disjunction of two equivalent fan-out signals — +`HedgeDiagnostics::alternate_region().is_some()` and any request tagged +`ExecutionContext::Hedging`. Either alone is sufficient; the disjunction stays +correct if a future change ever drifts one signal. + +### 4.3 Regions dispatched to, with reason — `requested_regions()` + +Dispatch order, duplicates preserved (a region dispatched twice appears twice), +entries with no resolved region skipped. The initial attempt is included and +tagged `RequestedRegionReason::Initial`. This is distinct from +`regions_contacted()`, which is sorted and deduplicated. + +### 4.4 Regions that responded — `responded_regions()` + +A region "responded" only if a service reply actually arrived. `completed_at` is +**not** a sufficient filter: the driver also sets it for client-side timeouts +(`timeout()`) and transport failures (`fail_transport()`). The internal +`RequestDiagnostics::responded_with_service_reply()` predicate excludes those two +cases: + +```rust +self.region.is_some() + && self.completed_at.is_some() + && !self.timed_out + && self.error.is_none() +``` + +A non-2xx HTTP status (404/429/503 from the service) still counts as a response. +Results are in arrival order (stable sort by `completed_at`, preserving dispatch +order among ties); duplicates are preserved. To deduplicate, collect into a +`BTreeSet`. + +--- + +## 5. Reconciliation with `HedgeDiagnostics` + +The Hedging Detection API and the Rust-native `HedgeDiagnostics` +([PR #4330](https://github.com/Azure/azure-sdk-for-rust/pull/4330) design, +[#4432](https://github.com/Azure/azure-sdk-for-rust/pull/4432) implementation) +coexist on the same `DiagnosticsContext` and serve different audiences. + +| Question | Hedging Detection API | Rust-native `HedgeDiagnostics` | +| --- | --- | --- | +| Did fan-out happen? | `hedging_started()` | `alternate_region().is_some()` — equivalent | +| Was a strategy active? | *(not derived)* | `hedge_diagnostics().is_some()` — superset of fan-out | +| Regions tried | `requested_regions()` (every attempt, with reason) | `primary_region()` + `alternate_region()` (hedge legs only) | +| Regions that responded | `responded_regions()` (full list, completion order) | `response_region()` (single winner) | +| Race outcome | *(not derived)* | `terminal_state()` (authoritative) | + +`main`'s `HedgeDiagnostics` classifies the race via `terminal_state` / +`alternate_region` (there is no `total_requests_launched` counter), so "fan-out +happened" is `alternate_region().is_some()` and "the alternate won" is +`matches!(terminal_state(), HedgeTerminalState::AlternateWon)`. Consult +`terminal_state()` for hedge win-rate; do not infer an alternate win from the +presence of `alternate_region()` alone (several terminal states still record an +alternate region). + +--- + +## 6. Future work + +The three accessors return owned/borrowed collections computed on demand from +the append-only attempt list, so callers allocate only when they read a derived +collection. If `ExecutionContext` becomes a prominent part of the public +detection surface it could be renamed to something friendlier (e.g., +`RequestPurpose` / `RequestIntent`); that rename is out of scope here. + +[`DiagnosticsContext`]: https://docs.rs/azure_data_cosmos/latest/azure_data_cosmos/struct.DiagnosticsContext.html diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs index e785d31cfdc..4a43af5821a 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs @@ -42,7 +42,7 @@ #[doc(inline)] pub use azure_data_cosmos_driver::diagnostics::{ - DiagnosticsContext, ThresholdBreach, TransportKind, + DiagnosticsContext, RequestedRegion, RequestedRegionReason, ThresholdBreach, TransportKind, }; #[doc(inline)] pub use azure_data_cosmos_driver::DiagnosticsThresholds; diff --git a/sdk/cosmos/azure_data_cosmos_driver/ARCHITECTURE.md b/sdk/cosmos/azure_data_cosmos_driver/ARCHITECTURE.md index 5566d2216e5..0425784dff9 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/ARCHITECTURE.md +++ b/sdk/cosmos/azure_data_cosmos_driver/ARCHITECTURE.md @@ -296,7 +296,7 @@ The `DiagnosticsContext` provides comprehensive visibility into operation execut flowchart TD DC["DiagnosticsContext (Immutable, per-operation)
• activity_id: ActivityId — unique identifier for the operation
• duration: Duration — total operation time
• status_code: StatusCode — final HTTP status after retries
• sub_status_code: SubStatusCode — Cosmos-specific error classification
• requests: Arc<Vec<RequestDiagnostics>>"] RD["RequestDiagnostics (per-HTTP-request details)
• region: Region
• endpoint: String
• status_code: StatusCode
• sub_status_code: Option<SubStatusCode>
• request_charge: f64
• duration_ms: u64"] - EC["execution_context: ExecutionContext
• Initial — first attempt
• Retry — retry after 429/503/etc.
• Hedging — speculative request
• RegionFailover — cross-region retry
• CircuitBreakerProbe — recovery check"] + EC["execution_context: ExecutionContext
• Initial — first attempt
• OperationRetry — retry after 429/503/etc.
• Hedging — speculative request
• RegionFailover — cross-region retry
• CircuitBreakerProbe — recovery check"] RSS["request_sent: RequestSentStatus
• Sent — definitely transmitted
• NotSent — definitely NOT transmitted
• Unknown — cannot determine"] RE["events: Vec<RequestEvent>
• timestamp: Instant
• duration_ms: Option<u64>
• details: Option<String>"] RET["event_type: RequestEventType
• TransportStart
• ResponseHeadersReceived
• TransportComplete
• TransportFailed"] @@ -474,7 +474,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 45 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -483,7 +483,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 52 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -492,7 +492,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 78 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -501,7 +501,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 120 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -510,7 +510,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 189 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -519,7 +519,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 312 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -528,7 +528,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 456 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -537,7 +537,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 623 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -546,7 +546,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 780 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, @@ -555,7 +555,7 @@ Scenario: Request throttled 10 times (429/3200) before succeeding on the 11th at "duration_ms": 890 }, { - "execution_context": "retry", + "execution_context": "operation_retry", "region": "West US 2", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 200, @@ -591,7 +591,7 @@ Same operation with deduplication applied: "duration_ms": 45 }, "last": { - "execution_context": "retry", + "execution_context": "operation_retry", "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 200, "request_charge": 1.0, @@ -602,7 +602,7 @@ Same operation with deduplication applied: "endpoint": "https://myaccount-westus2.documents.azure.com:443/dbs/myDatabase/colls/myContainer/docs/doc_001", "status_code": 429, "sub_status_code": 3200, - "execution_context": "retry", + "execution_context": "operation_retry", "count": 9, "total_request_charge": 9.0, "min_duration_ms": 52, diff --git a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md index e85982803d9..aca49b571bc 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md @@ -4,8 +4,12 @@ ### Features Added +- Added the Hedging Detection API on `DiagnosticsContext`: `hedging_started()` (whether a hedge arm actually fanned out), `requested_regions()` (regions dispatched to, in dispatch order with duplicates, each tagged with a reason), and `responded_regions()` (regions that produced an actual service reply, in completion order). Adds the public `RequestedRegion` struct and `RequestedRegionReason` enum (both `#[non_exhaustive]`, with a total `From` mapping) and the new `ExecutionContext::OperationRetry` variant. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) + ### Breaking Changes +- Renamed `ExecutionContext::Retry` to `ExecutionContext::OperationRetry` to distinguish operation-level retries from transport-level `TransportRetry`. The old `Retry` variant remains for one release as a `#[deprecated]` alias, but its serialized form changes from `"retry"` to `"operation_retry"`; telemetry parsers that match the literal `"retry"` must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) + ### Bugs Fixed ### Other Changes diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs index 94cf02a77e8..ff3ae827070 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs @@ -59,7 +59,18 @@ pub enum ExecutionContext { /// Initial request attempt (first try). Initial, /// Retry due to transient error (e.g., 429, 503). + /// + /// Renamed to [`ExecutionContext::OperationRetry`] to align with the + /// cross-SDK reason taxonomy. Retained for one release for source + /// compatibility; the serialized form also changes from `"retry"` to + /// `"operation_retry"`. + #[deprecated(since = "0.7.0", note = "use `ExecutionContext::OperationRetry`")] Retry, + /// An operation-level retry decided by the SDK's client-retry policy. + /// + /// Distinguishes user-visible operation retries from transport-layer + /// retries ([`ExecutionContext::TransportRetry`]). + OperationRetry, /// Transport-level shard retry within the same region. /// /// The initial attempt failed with a connectivity error and the transport @@ -77,9 +88,11 @@ pub enum ExecutionContext { impl ExecutionContext { /// Returns the string representation of this execution context. pub fn as_str(&self) -> &'static str { + #[allow(deprecated)] match self { ExecutionContext::Initial => "initial", ExecutionContext::Retry => "retry", + ExecutionContext::OperationRetry => "operation_retry", ExecutionContext::TransportRetry => "transport_retry", ExecutionContext::Hedging => "hedging", ExecutionContext::RegionFailover => "region_failover", @@ -88,6 +101,63 @@ impl ExecutionContext { } } +/// Reason the SDK chose to dispatch a request to a particular region. +/// +/// Realizes the cross-SDK Hedging Detection API's `RequestedRegionReason`. Each +/// entry returned by [`DiagnosticsContext::requested_regions`] carries one of +/// these, projected from the driver-internal [`ExecutionContext`] via the +/// [`From`] mapping below. +/// +/// The enum is `#[non_exhaustive]`; callers that `match` on it MUST include a +/// wildcard arm. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum RequestedRegionReason { + /// The first dispatch of the operation. + Initial, + /// An operation-level retry decided by the SDK's client-retry policy. + OperationRetry, + /// A transport-level retry inside the per-region transport stack. + TransportRetry, + /// A speculative cross-region hedge fan-out dispatch. + Hedging, + /// An endpoint-failure-driven retry to a different region. + RegionFailover, + /// A probe dispatch to a previously circuit-broken region. + CircuitBreakerProbe, +} + +impl From for RequestedRegionReason { + fn from(ctx: ExecutionContext) -> Self { + #[allow(deprecated)] + match ctx { + ExecutionContext::Initial => RequestedRegionReason::Initial, + ExecutionContext::Retry | ExecutionContext::OperationRetry => { + RequestedRegionReason::OperationRetry + } + ExecutionContext::TransportRetry => RequestedRegionReason::TransportRetry, + ExecutionContext::Hedging => RequestedRegionReason::Hedging, + ExecutionContext::RegionFailover => RequestedRegionReason::RegionFailover, + ExecutionContext::CircuitBreakerProbe => RequestedRegionReason::CircuitBreakerProbe, + } + } +} + +/// A single region the SDK dispatched a request to, tagged with the reason the +/// orchestrator chose to send it. +/// +/// Realizes the cross-SDK Hedging Detection API's `RequestedRegion` value type. +/// Returned by [`DiagnosticsContext::requested_regions`]. Region equality is +/// delegated to [`Region`]'s own `PartialEq`. +#[derive(Clone, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct RequestedRegion { + /// The region the SDK dispatched to. + pub region: Region, + /// The reason the SDK chose this region for this dispatch attempt. + pub reason: RequestedRegionReason, +} + impl AsRef for ExecutionContext { fn as_ref(&self) -> &str { self.as_str() @@ -686,6 +756,21 @@ impl RequestDiagnostics { self.completed_at.is_some() } + /// Returns whether this request received an actual service response. + /// + /// `completed_at` alone is insufficient: the driver also sets it for + /// client-side end-to-end timeouts ([`timeout`](Self::timeout)) and + /// transport-level failures ([`fail_transport`](Self::fail_transport)). + /// This predicate excludes those two cases so that only requests that + /// produced a service reply (any HTTP status, including non-2xx) are + /// counted. Used by [`DiagnosticsContext::responded_regions`]. + pub(crate) fn responded_with_service_reply(&self) -> bool { + self.region.is_some() + && self.completed_at.is_some() + && !self.timed_out + && self.error.is_none() + } + // Public getters for read-only access to fields /// Returns the execution context describing why this request was made. @@ -2177,6 +2262,92 @@ impl DiagnosticsContext { self.regions_contacted.clone() } + /// Returns the regions to which this operation dispatched a request, in + /// dispatch order, each tagged with the reason the SDK chose it. + /// + /// Each dispatched attempt with a resolved region contributes one entry. + /// Duplicates are allowed: the same region may appear more than once if it + /// was dispatched multiple times (e.g., a retry to the same region, or a + /// hedge request to a region that was also the primary). The initial + /// attempt is included and tagged [`RequestedRegionReason::Initial`]. + /// + /// Entries with no resolved region (pre-region-selection failures) are + /// skipped, so this returns an empty `Vec` when an operation failed before + /// any region was selected. + /// + /// Order matches [`RequestDiagnostics`] insertion order, which is dispatch + /// order. This is distinct from [`regions_contacted`](Self::regions_contacted), + /// which is sorted and deduplicated. + pub fn requested_regions(&self) -> Vec { + self.requests + .iter() + .filter_map(|r| { + r.region().map(|region| RequestedRegion { + region: region.clone(), + reason: RequestedRegionReason::from(r.execution_context()), + }) + }) + .collect() + } + + /// Returns the regions from which this operation received a response, in + /// arrival (completion) order. + /// + /// Each request that produced a service reply contributes one entry. + /// Duplicates are allowed: the same region may appear more than once if + /// multiple completed responses arrived from it (e.g., a late hedge + /// response after the hedge winner). `responded_regions().len() > 1` does + /// NOT imply more than one distinct region responded. + /// + /// Only requests that received an actual service response are included; + /// client-side timeouts and transport failures are excluded (via the + /// internal `responded_with_service_reply` predicate on each request). + /// A non-2xx HTTP status (e.g., 404/429) still counts — it is a response + /// from the region. + /// + /// To deduplicate, callers can collect into a set, for example: + /// `ctx.responded_regions().into_iter().collect::>()`. + pub fn responded_regions(&self) -> Vec<&Region> { + let mut responded: Vec<&RequestDiagnostics> = self + .requests + .iter() + .filter(|r| r.responded_with_service_reply()) + .collect(); + // Stable sort by completion time to yield arrival order while + // preserving dispatch order among ties. + responded.sort_by_key(|r| r.completed_at()); + responded.iter().filter_map(|r| r.region()).collect() + } + + /// Returns `true` iff this operation actually dispatched at least one hedge + /// request (i.e., fan-out occurred), and `false` otherwise. + /// + /// `false` does NOT mean hedging was disabled or misconfigured; it means no + /// fan-out occurred. In particular, when the primary returns before the + /// hedging threshold elapses, this returns `false` even though a hedging + /// strategy was active. + /// + /// To check whether a hedging strategy was *configured*, inspect + /// [`hedge_diagnostics`](Self::hedge_diagnostics) instead — it is `Some` + /// whenever hedging was active for the operation, including the + /// primary-wins-under-threshold case where no fan-out happened. + /// + /// The result is a disjunction of two independent fan-out signals: + /// [`HedgeDiagnostics::alternate_region`] being `Some` (the orchestrator + /// dispatched an alternate leg) and any [`RequestDiagnostics`] tagged + /// [`ExecutionContext::Hedging`]. Either alone is sufficient; the + /// disjunction stays correct if a future change ever drifts one signal. + pub fn hedging_started(&self) -> bool { + self.hedge_diagnostics + .as_ref() + .map(|hd| hd.alternate_region().is_some()) + .unwrap_or(false) + || self + .requests + .iter() + .any(|r| matches!(r.execution_context(), ExecutionContext::Hedging)) + } + /// Returns a shared reference to all request diagnostics. /// /// This returns an `Arc>`, enabling efficient @@ -2862,7 +3033,7 @@ mod tests { builder.update_request(h1, |req| req.request_charge = RequestCharge::new(3.0)); let h2 = builder.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::WEST_US_2), "https://test.documents.azure.com", ); @@ -2881,7 +3052,7 @@ mod tests { "https://test.westus2.documents.azure.com", ); builder.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::WEST_US_2), "https://test.westus2.documents.azure.com", ); @@ -3146,7 +3317,7 @@ mod tests { ); record_run( &mut read, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", "https://east/", CosmosStatus::new(StatusCode::TooManyRequests), @@ -3166,7 +3337,7 @@ mod tests { ); record_run( &mut replace, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "West US", "https://west/", CosmosStatus::new(StatusCode::Gone), @@ -3359,7 +3530,7 @@ mod tests { // Add several requests to trigger deduplication for i in 0..5 { let handle = builder.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::WEST_US_2), "https://test.documents.azure.com", ); @@ -3386,7 +3557,7 @@ mod tests { "request_count": 5, "total_request_charge": 10.0, "first": { - "execution_context": "retry", + "execution_context": "operation_retry", "endpoint": "https://test.documents.azure.com/", "status": "429/3200 (RUBudgetExceeded)", "request_charge": 0.0, @@ -3394,7 +3565,7 @@ mod tests { "timed_out": false }, "last": { - "execution_context": "retry", + "execution_context": "operation_retry", "endpoint": "https://test.documents.azure.com/", "status": "429/3200 (RUBudgetExceeded)", "request_charge": 4.0, @@ -3404,7 +3575,7 @@ mod tests { "deduplicated_groups": [{ "endpoint": "https://test.documents.azure.com/", "status": "429/3200 (RUBudgetExceeded)", - "execution_context": "retry", + "execution_context": "operation_retry", "count": 3, "total_request_charge": 6.0, @@ -3727,7 +3898,7 @@ mod tests { ); let succeeded = builder.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::WEST_US_2), "https://test.documents.azure.com", ); @@ -3821,7 +3992,14 @@ mod tests { #[test] fn execution_context_display() { assert_eq!(ExecutionContext::Initial.to_string(), "initial"); - assert_eq!(ExecutionContext::Retry.to_string(), "retry"); + #[allow(deprecated)] + { + assert_eq!(ExecutionContext::Retry.to_string(), "retry"); + } + assert_eq!( + ExecutionContext::OperationRetry.to_string(), + "operation_retry" + ); assert_eq!( ExecutionContext::TransportRetry.to_string(), "transport_retry" @@ -3837,6 +4015,144 @@ mod tests { ); } + #[test] + fn requested_regions_preserves_dispatch_order_and_reason() { + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + builder.start_test_request( + ExecutionContext::Initial, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.start_test_request( + ExecutionContext::RegionFailover, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + }); + + let requested = ctx.requested_regions(); + assert_eq!(requested.len(), 3); + // Dispatch order preserved, duplicates kept. + assert_eq!(requested[0].region, Region::WEST_US_2); + assert_eq!(requested[0].reason, RequestedRegionReason::Initial); + assert_eq!(requested[1].region, Region::WEST_US_2); + assert_eq!(requested[1].reason, RequestedRegionReason::OperationRetry); + assert_eq!(requested[2].region, Region::EAST_US_2); + assert_eq!(requested[2].reason, RequestedRegionReason::RegionFailover); + } + + #[test] + fn requested_regions_empty_without_region() { + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + builder.start_test_request( + ExecutionContext::Initial, + None, + "https://test.documents.azure.com", + ); + }); + + assert!(ctx.requested_regions().is_empty()); + } + + #[test] + fn responded_regions_excludes_timeouts_and_transport_failures() { + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + // A real service reply. + let h1 = builder.start_test_request( + ExecutionContext::Initial, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.complete_request(h1, StatusCode::Ok, None); + + // A client-side timeout (completed_at set, but no service reply). + let h2 = builder.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.timeout_request(h2); + + // A non-2xx response still counts as a reply from the region. + let h3 = builder.start_test_request( + ExecutionContext::RegionFailover, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h3, StatusCode::NotFound, None); + }); + + let responded = ctx.responded_regions(); + assert_eq!(responded, vec![&Region::WEST_US_2, &Region::EAST_US_2]); + } + + #[test] + fn hedging_started_false_without_hedge_dispatch() { + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let h = builder.start_test_request( + ExecutionContext::Initial, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Ok, None); + }); + + assert!(!ctx.hedging_started()); + } + + #[test] + fn hedging_started_true_when_hedge_dispatched() { + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + builder.start_test_request( + ExecutionContext::Initial, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.start_test_request( + ExecutionContext::Hedging, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + }); + + assert!(ctx.hedging_started()); + } + + #[test] + fn requested_region_reason_mapping_is_total() { + // A wildcard-free match forces this to stay total as variants are added. + #[allow(deprecated)] + let all = [ + ExecutionContext::Initial, + ExecutionContext::Retry, + ExecutionContext::OperationRetry, + ExecutionContext::TransportRetry, + ExecutionContext::Hedging, + ExecutionContext::RegionFailover, + ExecutionContext::CircuitBreakerProbe, + ]; + for ctx in all { + let reason = RequestedRegionReason::from(ctx); + #[allow(deprecated)] + let expected = match ctx { + ExecutionContext::Initial => RequestedRegionReason::Initial, + ExecutionContext::Retry | ExecutionContext::OperationRetry => { + RequestedRegionReason::OperationRetry + } + ExecutionContext::TransportRetry => RequestedRegionReason::TransportRetry, + ExecutionContext::Hedging => RequestedRegionReason::Hedging, + ExecutionContext::RegionFailover => RequestedRegionReason::RegionFailover, + ExecutionContext::CircuitBreakerProbe => RequestedRegionReason::CircuitBreakerProbe, + }; + assert_eq!(reason, expected); + } + } + // ========================================================================= // Pipeline/Transport/RequestSentStatus tests (merged from request_diagnostics.rs) // ========================================================================= @@ -4184,7 +4500,7 @@ mod tests { ); record_run( &mut b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", "https://east/", CosmosStatus::new(StatusCode::TooManyRequests), @@ -4251,7 +4567,7 @@ mod tests { ); record_run( &mut b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", "https://east/", CosmosStatus::new(StatusCode::TooManyRequests), @@ -4260,7 +4576,7 @@ mod tests { ); record_run( &mut b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", "https://east/", CosmosStatus::new(StatusCode::Gone).with_sub_status(1002), @@ -4269,7 +4585,7 @@ mod tests { ); record_run( &mut b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "West US", "https://west/", CosmosStatus::new(StatusCode::Ok), @@ -4322,13 +4638,13 @@ mod tests { ); for _ in 0..200 { let he = b.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::new("East US")), "https://east/", ); b.complete_request(he, StatusCode::ServiceUnavailable, None); let hw = b.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::new("West US")), "https://west/", ); @@ -4373,7 +4689,7 @@ mod tests { for i in 0..distinct { record_run( &mut b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", &format!("https://pkrange-{i}/"), CosmosStatus::new(StatusCode::Gone).with_sub_status(1002), @@ -4437,7 +4753,7 @@ mod tests { for i in 0..cold { let h = b.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::new("East US")), &format!("https://cold-{i}/"), ); @@ -4447,7 +4763,7 @@ mod tests { for _round in 0..hot_retries { for j in 0..hot { let h = b.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::new("West US")), &format!("https://hot-{j}/"), ); @@ -4527,7 +4843,7 @@ mod tests { let ctx = make_context_with(ActivityId::from_string("normal".to_string()), |b| { for _ in 0..3 { let h = b.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::new("East US")), "https://east/", ); diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs index 8581941402d..e64cb1b223c 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs @@ -27,8 +27,8 @@ pub(crate) use diagnostics_context::DiagnosticsContextBuilder; pub use diagnostics_context::{ DiagnosticsContext, ExecutionContext, FailedTransportShardDiagnostics, PipelineType, RequestDiagnostics, RequestEvent, RequestEventType, RequestHandle, RequestSentStatus, - ThresholdBreach, TransportHttpVersion, TransportKind, TransportSecurity, - TransportShardDiagnostics, + RequestedRegion, RequestedRegionReason, ThresholdBreach, TransportHttpVersion, TransportKind, + TransportSecurity, TransportShardDiagnostics, }; pub use proxy_configuration::ProxyConfiguration; diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/cosmos_driver.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/cosmos_driver.rs index 3414323aceb..d319b79177b 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/cosmos_driver.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/cosmos_driver.rs @@ -821,7 +821,7 @@ impl CosmosDriver { .unwrap_or(azure_core::time::Duration::ZERO), ) .await; - execution_context = ExecutionContext::Retry; + execution_context = ExecutionContext::OperationRetry; continue; } } diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs index b02fdb947ed..94f4abaafde 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs @@ -1891,7 +1891,7 @@ fn compute_execution_context(retry_state: &OperationRetryState) -> ExecutionCont if retry_state.failover_retry_count == 0 && retry_state.session_token_retry_count == 0 { ExecutionContext::Initial } else if retry_state.session_token_retry_count > 0 { - ExecutionContext::Retry + ExecutionContext::OperationRetry } else { ExecutionContext::RegionFailover } @@ -4187,7 +4187,7 @@ mod tests { let ctx = TransportRequestContext { routing: &routing, activity_id: &activity_id, - execution_context: ExecutionContext::Retry, + execution_context: ExecutionContext::OperationRetry, deadline: Some(std::time::Instant::now() + Duration::from_secs(5)), effective_consistency: DefaultConsistencyLevel::Session, read_consistency_strategy: crate::options::ReadConsistencyStrategy::Default, @@ -4220,7 +4220,7 @@ mod tests { let ctx = TransportRequestContext { routing: &routing, activity_id: &activity_id, - execution_context: ExecutionContext::Retry, + execution_context: ExecutionContext::OperationRetry, deadline: Some(std::time::Instant::now() + Duration::from_secs(5)), effective_consistency: DefaultConsistencyLevel::Session, read_consistency_strategy: crate::options::ReadConsistencyStrategy::Default, @@ -4282,7 +4282,7 @@ mod tests { let ctx = TransportRequestContext { routing: &routing, activity_id: &activity_id, - execution_context: ExecutionContext::Retry, + execution_context: ExecutionContext::OperationRetry, deadline: Some(std::time::Instant::now() + Duration::from_secs(5)), effective_consistency: DefaultConsistencyLevel::Session, read_consistency_strategy: crate::options::ReadConsistencyStrategy::Default, @@ -4332,7 +4332,7 @@ mod tests { let ctx = TransportRequestContext { routing: &routing, activity_id: &activity_id, - execution_context: ExecutionContext::Retry, + execution_context: ExecutionContext::OperationRetry, deadline: Some(std::time::Instant::now() + Duration::from_secs(5)), effective_consistency: DefaultConsistencyLevel::Session, read_consistency_strategy: crate::options::ReadConsistencyStrategy::Default, @@ -7996,17 +7996,17 @@ mod tests { fn execution_context_retry_when_session_retry_active() { // Session-retry takes precedence over failover-retry: when both // counters are non-zero, the most recent advance was the session - // retry, so the attempt is annotated as a `Retry`. + // retry, so the attempt is annotated as an `OperationRetry`. let state = retry_state_with_counts(1, 1); assert!(matches!( super::compute_execution_context(&state), - ExecutionContext::Retry + ExecutionContext::OperationRetry )); let state = retry_state_with_counts(0, 1); assert!(matches!( super::compute_execution_context(&state), - ExecutionContext::Retry + ExecutionContext::OperationRetry )); } diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/transport/transport_pipeline.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/transport/transport_pipeline.rs index 9189a8435ce..721745f674d 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/transport/transport_pipeline.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/transport/transport_pipeline.rs @@ -231,7 +231,7 @@ pub(crate) async fn execute_transport_pipeline( } else if throttle_state.attempt_count == 0 { request.execution_context } else { - ExecutionContext::Retry + ExecutionContext::OperationRetry }; let request_handle = diagnostics.start_request( From a33d7dffcd21590dc3d73deb5495401ff045c7d0 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Mon, 27 Jul 2026 10:18:14 -0700 Subject: [PATCH 02/21] Surface hedging through Cosmos observability handlers Build on H1's Hedging Detection API to surface hedging in each completed-operation signal, additive and opt-in where high-cardinality. Driver (azure_data_cosmos_driver): - Add `HedgeTerminalState::as_str()` + `Display` for a stable, low-cardinality snake_case terminal-state value (single source of truth for the attribute / log-field value). - Add `__internal_test_diagnostics_construction`-gated, `#[doc(hidden)]` test seams so the wrapper SDK can build a hedged context: `HedgeDiagnostics::for_testing`, `DiagnosticsContext::for_testing_with_hedge`, and `RequestDiagnostics::with_execution_context_for_testing`. SDK (azure_data_cosmos): - Tracing: when `hedging_started()`, add `hedging_started`, `hedge_region`, `hedge_terminal_state`, and `requested_regions`/`responded_regions` (`string[]`, like `contacted_regions`) to the sampled operation span, and tag the hedge-leg child span (`azure.cosmosdb.request.hedge`). - Metrics: add the opt-in `azure.cosmosdb.client.operation.hedged` counter (`MetricsOptions::with_hedged_metric`), emitted only when hedging fanned out; low-cardinality `hedge_terminal_state` dim always, high-cardinality `hedge_region` dim only under the existing extended-attributes gate. - Logging: add `hedging_started` / `hedge_region` / `hedge_terminal_state` fields to the compact sampled diagnostics line when hedging occurred. - Add shared attribute-name constants and unit tests (hedged vs non-hedged) for each signal using in-memory OTel exporters / a tracing capture layer. Refs #4410. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- sdk/cosmos/azure_data_cosmos/CHANGELOG.md | 1 + .../src/diagnostics/attributes.rs | 24 +++ .../src/diagnostics/logging/handler.rs | 20 ++- .../src/diagnostics/logging/mod.rs | 109 +++++++++++- .../src/diagnostics/metrics/attributes.rs | 16 ++ .../src/diagnostics/metrics/handler.rs | 165 ++++++++++++++++++ .../src/diagnostics/metrics/instruments.rs | 21 ++- .../src/diagnostics/metrics/options.rs | 20 +++ .../src/diagnostics/tracing/mod.rs | 151 +++++++++++++++- .../src/diagnostics/tracing/span_builder.rs | 53 +++++- .../azure_data_cosmos_driver/CHANGELOG.md | 1 + .../src/diagnostics/diagnostics_context.rs | 46 ++++- .../driver/pipeline/hedging_diagnostics.rs | 54 ++++++ 13 files changed, 671 insertions(+), 10 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md index 78b45a7e580..54a9dfd5626 100644 --- a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md @@ -8,6 +8,7 @@ - Added a pluggable client-side diagnostics emission layer — the `DiagnosticsHandler` trait and ordered `DiagnosticsHandlerChain` (registered via `CosmosClientBuilder::with_diagnostics_handler`) — invoked once per operation (singleton and paginated, on success and failure) with the completed `DiagnosticsContext` plus an SDK-supplied `CosmosOperationContext`; the empty default chain is a zero-overhead no-op. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added the `metrics`-gated `CosmosMetricsHandler` (with `MetricsOptions`), emitting the stable `db.client.operation.duration` histogram plus per-signal opt-in metrics (`with_request_charge_metric`, `with_returned_rows_metric`) and an opt-in extended attribute set (`with_extended_attributes`); a no-op when no meter provider is registered. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added composable tail-sampled emission handlers — a `TracingLogHandler` leaf that writes a compact `tracing` line and a `SamplingLogHandler` wrapper (holding an `Arc`) that applies the sampling gate plus a shared per-window rate limit, defaulting to wrap a `TracingLogHandler` — and the `distributed_tracing`-gated `CosmosTracingHandler` (backdated span tree), also rate-limited so an error storm can't overwhelm exporters. All emit only for operations which fail or breach a configurable `DiagnosticsThresholds`, and stamp *why* they were sampled (a failure, or which threshold) on the emitted line and span. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) +- Surfaced hedging through the observability handlers, reusing the Hedging Detection API. When a cross-region hedge fans out, `CosmosTracingHandler` adds `azure.cosmosdb.operation.{hedging_started,hedge_region,hedge_terminal_state}` plus `requested_regions`/`responded_regions` (`string[]`) to the sampled operation span and tags the hedge-leg child span (`azure.cosmosdb.request.hedge`); `SamplingLogHandler` adds `hedging_started` / `hedge_region` / `hedge_terminal_state` fields to the sampled log line; and `CosmosMetricsHandler` gains an opt-in `azure.cosmosdb.client.operation.hedged` counter (`MetricsOptions::with_hedged_metric`) carrying the low-cardinality `hedge_terminal_state`, with the higher-cardinality `hedge_region` dimension added only under `with_extended_attributes`. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) ### Breaking Changes diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/attributes.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/attributes.rs index b2f8c789716..06dc66ea95f 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/attributes.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/attributes.rs @@ -52,6 +52,30 @@ pub(crate) const CONNECTION_MODE: &str = "azure.cosmosdb.connection.mode"; /// `azure.cosmosdb.operation.contacted_regions` — regions contacted (ordered `string[]`). pub(crate) const CONTACTED_REGIONS: &str = "azure.cosmosdb.operation.contacted_regions"; +/// `azure.cosmosdb.operation.hedging_started` — `true` when the operation +/// dispatched at least one cross-region hedge (fan-out occurred). +pub(crate) const HEDGING_STARTED: &str = "azure.cosmosdb.operation.hedging_started"; + +/// `azure.cosmosdb.operation.hedge_region` — the alternate region the hedge was +/// dispatched to, when a hedge fan-out occurred. +pub(crate) const HEDGE_REGION: &str = "azure.cosmosdb.operation.hedge_region"; + +/// `azure.cosmosdb.operation.hedge_terminal_state` — how the hedging race ended +/// (see `HedgeTerminalState::as_str`). +pub(crate) const HEDGE_TERMINAL_STATE: &str = "azure.cosmosdb.operation.hedge_terminal_state"; + +/// `azure.cosmosdb.operation.requested_regions` — regions dispatched to, in +/// dispatch order (`string[]`). High-signal for hedge fan-out. +pub(crate) const REQUESTED_REGIONS: &str = "azure.cosmosdb.operation.requested_regions"; + +/// `azure.cosmosdb.operation.responded_regions` — regions that returned a +/// service reply, in arrival order (`string[]`). +pub(crate) const RESPONDED_REGIONS: &str = "azure.cosmosdb.operation.responded_regions"; + +/// `azure.cosmosdb.request.hedge` — `true` on the per-attempt (child) span for a +/// speculative hedge leg dispatched to an alternate region. +pub(crate) const HEDGE_LEG: &str = "azure.cosmosdb.request.hedge"; + /// `azure.cosmosdb.response.sub_status_code` — the Cosmos sub-status code. pub(crate) const SUB_STATUS_CODE: &str = "azure.cosmosdb.response.sub_status_code"; diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs index 101c48e26fe..cf3aa5ad76a 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs @@ -77,7 +77,25 @@ impl DiagnosticsHandler for TracingLogHandler { // only evaluated when a subscriber is actually listening for the event // (the `tracing` macros run an "is enabled" check before evaluating // field expressions). - if diagnostics.is_failure() { + // + // When a cross-region hedge fanned out, surface the hedging signal as + // dedicated fields (in addition to the JSON blob) — it is high-signal for + // exactly the failed / threshold-breaching operations this handler emits. + if diagnostics.hedging_started() { + let hedge = diagnostics.hedge_diagnostics(); + let hedge_region = hedge + .and_then(|h| h.alternate_region()) + .map(|region| region.as_str()) + .unwrap_or_default(); + let hedge_terminal_state = hedge + .map(|h| h.terminal_state().as_str()) + .unwrap_or_default(); + if diagnostics.is_failure() { + tracing::warn!(target: SAMPLED_TARGET, reason, hedging_started = true, hedge_region, hedge_terminal_state, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); + } else { + tracing::info!(target: SAMPLED_TARGET, reason, hedging_started = true, hedge_region, hedge_terminal_state, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); + } + } else if diagnostics.is_failure() { tracing::warn!(target: SAMPLED_TARGET, reason, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); } else { tracing::info!(target: SAMPLED_TARGET, reason, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs index 98908f7867c..8e311a25ba7 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs @@ -20,12 +20,15 @@ pub use handler::{SamplingLogHandler, TracingLogHandler}; #[cfg(test)] mod tests { + use std::collections::HashMap; use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::Arc; use std::time::{Duration, Instant}; use azure_core::http::{Context, StatusCode}; - use azure_data_cosmos_driver::diagnostics::{DiagnosticsContext, RequestDiagnostics}; + use azure_data_cosmos_driver::diagnostics::{ + DiagnosticsContext, HedgeDiagnostics, HedgeTerminalState, RequestDiagnostics, + }; use azure_data_cosmos_driver::models::{ActivityId, RequestCharge}; use azure_data_cosmos_driver::options::Region; use azure_data_cosmos_driver::{CosmosStatus, DiagnosticsThresholds}; @@ -275,4 +278,108 @@ mod tests { assert_eq!(captured.lock().unwrap().as_deref(), Some("request_charge")); }); } + + /// Captures every field of the most recent sampled log event into a map, so + /// tests can assert on the presence and value of the hedging fields. + struct FieldCapture(Arc>>); + + impl Layer for FieldCapture { + fn on_event(&self, event: &tracing::Event<'_>, _cx: LayerContext<'_, S>) { + if event.metadata().target() != "azure_data_cosmos::diagnostics::sampled" { + return; + } + struct Visitor<'a>(&'a mut HashMap); + impl tracing::field::Visit for Visitor<'_> { + fn record_bool(&mut self, field: &tracing::field::Field, value: bool) { + self.0.insert(field.name().to_string(), value.to_string()); + } + fn record_str(&mut self, field: &tracing::field::Field, value: &str) { + self.0.insert(field.name().to_string(), value.to_string()); + } + fn record_debug( + &mut self, + field: &tracing::field::Field, + value: &dyn std::fmt::Debug, + ) { + self.0 + .entry(field.name().to_string()) + .or_insert_with(|| format!("{value:?}")); + } + } + let mut map = self.0.lock().unwrap(); + event.record(&mut Visitor(&mut map)); + } + } + + /// A failed operation that fanned out an `AlternateWon` hedge to West US 2. + fn hedged_context() -> DiagnosticsContext { + let now = Instant::now(); + let primary = RequestDiagnostics::for_testing( + "https://acct-eastus.documents.azure.com:443/", + Some(Region::EAST_US), + CosmosStatus::new(StatusCode::TooManyRequests), + RequestCharge::new(2.0), + now - Duration::from_millis(20), + now, + ); + let hedge = HedgeDiagnostics::for_testing( + Region::EAST_US, + Some(Region::WEST_US_2), + Some(Region::WEST_US_2), + HedgeTerminalState::AlternateWon, + ); + DiagnosticsContext::for_testing_with_hedge( + ActivityId::new_uuid(), + Duration::from_millis(20), + Some(CosmosStatus::new(StatusCode::TooManyRequests)), + Some("read_item"), + vec![primary], + Some(hedge), + ) + } + + #[test] + fn sampled_line_carries_hedging_fields_when_hedging_occurred() { + let captured = Arc::new(std::sync::Mutex::new(HashMap::new())); + let layer = FieldCapture(Arc::clone(&captured)); + let subscriber = tracing_subscriber::registry().with(layer); + let handler = SamplingLogHandler::new(); + + tracing::subscriber::with_default(subscriber, || { + handler.handle(&hedged_context(), &Context::new()); + }); + + let map = captured.lock().unwrap(); + assert_eq!(map.get("hedging_started").map(String::as_str), Some("true")); + assert_eq!(map.get("hedge_region").map(String::as_str), Some("westus2")); + assert_eq!( + map.get("hedge_terminal_state").map(String::as_str), + Some("alternate_won") + ); + } + + #[test] + fn sampled_line_omits_hedging_fields_without_hedging() { + let captured = Arc::new(std::sync::Mutex::new(HashMap::new())); + let layer = FieldCapture(Arc::clone(&captured)); + let subscriber = tracing_subscriber::registry().with(layer); + let handler = SamplingLogHandler::new(); + + // A plain failure — no hedge occurred. + let failed = context( + Duration::from_millis(5), + CosmosStatus::new(StatusCode::TooManyRequests), + 2.0, + ); + tracing::subscriber::with_default(subscriber, || { + handler.handle(&failed, &Context::new()); + }); + + let map = captured.lock().unwrap(); + // The line was emitted (reason present) but carries no hedging fields. + assert!(map.contains_key("reason")); + assert!(!map.contains_key("hedging_started")); + assert!(!map.contains_key("hedge_region")); + assert!(!map.contains_key("hedge_terminal_state")); + } } diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs index 6286e098e92..8e44f1d905d 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs @@ -32,6 +32,10 @@ pub const METRIC_OPERATION_REQUEST_CHARGE: &str = "azure.cosmosdb.client.operati /// Optional histogram (rows): number of rows/items returned by an operation. pub const METRIC_RESPONSE_RETURNED_ROWS: &str = "db.client.response.returned_rows"; +/// Optional counter (operations): number of operations that dispatched a +/// cross-region hedge fan-out. +pub const METRIC_OPERATION_HEDGED: &str = "azure.cosmosdb.client.operation.hedged"; + // ========================================================================= // Instrument units // ========================================================================= @@ -49,6 +53,9 @@ pub const UNIT_REQUEST_UNIT: &str = "{request_unit}"; /// Unit for [`METRIC_RESPONSE_RETURNED_ROWS`] — rows. pub const UNIT_ROW: &str = "{row}"; +/// Unit for [`METRIC_OPERATION_HEDGED`] — operations. +pub const UNIT_OPERATION: &str = "{operation}"; + // ========================================================================= // Stable attributes (always emitted; operation scope, low cardinality) // @@ -99,3 +106,12 @@ pub const ATTR_SUB_STATUS_CODE: &str = attributes::SUB_STATUS_CODE; /// `azure.cosmosdb.connection.mode` — gateway vs. direct connection mode. pub const ATTR_CONNECTION_MODE: &str = attributes::CONNECTION_MODE; + +/// `azure.cosmosdb.operation.hedge_terminal_state` — how the hedging race ended. +/// Low cardinality (one value per terminal state), attached to the hedged +/// counter unconditionally. +pub const ATTR_HEDGE_TERMINAL_STATE: &str = attributes::HEDGE_TERMINAL_STATE; + +/// `azure.cosmosdb.operation.hedge_region` — the alternate hedge region. Higher +/// cardinality, so attached only under the extended-attribute opt-in. +pub const ATTR_HEDGE_REGION: &str = attributes::HEDGE_REGION; diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs index 1198782d289..0d8c5c935aa 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs @@ -209,6 +209,34 @@ impl CosmosMetricsHandler { } } +impl CosmosMetricsHandler { + /// Records the hedged-operation counter for an operation that fanned out a + /// cross-region hedge. + /// + /// The low-cardinality `hedge_terminal_state` dimension is always attached; + /// the higher-cardinality `hedge_region` dimension is added only under the + /// extended-attributes opt-in (mirroring how contacted regions are gated on + /// the duration metric). + fn record_hedged(&self, diagnostics: &DiagnosticsContext, base_attrs: &[KeyValue]) { + let mut attrs = base_attrs.to_vec(); + if let Some(hedge) = diagnostics.hedge_diagnostics() { + attrs.push(KeyValue::new( + attributes::ATTR_HEDGE_TERMINAL_STATE, + hedge.terminal_state().as_str(), + )); + if self.options.extended_attributes_enabled() { + if let Some(alternate) = hedge.alternate_region() { + attrs.push(KeyValue::new( + attributes::ATTR_HEDGE_REGION, + alternate.as_str().to_string(), + )); + } + } + } + self.instruments.hedged.add(1, &attrs); + } +} + impl Default for CosmosMetricsHandler { fn default() -> Self { Self::new() @@ -248,6 +276,12 @@ impl DiagnosticsHandler for CosmosMetricsHandler { self.instruments.returned_rows.record(rows, &attributes); } } + + // Hedging counter: emitted only when opted in and a hedge actually + // fanned out. Reuses H1's hedging_started() detection. + if self.options.hedged_metric_enabled() && diagnostics.hedging_started() { + self.record_hedged(diagnostics, &attributes); + } } } @@ -263,7 +297,9 @@ mod tests { use super::*; use crate::CosmosStatus; use azure_core::http::StatusCode; + use azure_data_cosmos_driver::diagnostics::{HedgeDiagnostics, HedgeTerminalState}; use azure_data_cosmos_driver::models::ActivityId; + use azure_data_cosmos_driver::options::Region; use opentelemetry::metrics::MeterProvider as _; use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData, ResourceMetrics}; use opentelemetry_sdk::metrics::{InMemoryMetricExporter, PeriodicReader, SdkMeterProvider}; @@ -307,6 +343,25 @@ mod tests { ) } + /// Completed context whose operation fanned out an `AlternateWon` hedge to + /// West US 2. + fn hedged_completed(status_code: u16) -> DiagnosticsContext { + let hedge = HedgeDiagnostics::for_testing( + Region::EAST_US, + Some(Region::WEST_US_2), + Some(Region::WEST_US_2), + HedgeTerminalState::AlternateWon, + ); + DiagnosticsContext::for_testing_with_hedge( + ActivityId::new_uuid(), + Duration::from_millis(42), + Some(CosmosStatus::new(StatusCode::from(status_code))), + Some("read_item"), + Vec::new(), + Some(hedge), + ) + } + fn operation_context() -> CosmosOperationContext { CosmosOperationContext::new() .with_operation_name("read_item") @@ -538,4 +593,114 @@ mod tests { ); assert_eq!(host_of("not a url"), None); } + + /// Returns the attributes (as a string map) and summed value of the + /// `azure.cosmosdb.client.operation.hedged` counter, if present. + fn hedged_point(metrics: &[ResourceMetrics]) -> Option<(HashMap, u64)> { + for rm in metrics { + for sm in rm.scope_metrics() { + for m in sm.metrics() { + if m.name() != attributes::METRIC_OPERATION_HEDGED { + continue; + } + if let AggregatedMetrics::U64(MetricData::Sum(sum)) = m.data() { + if let Some(point) = sum.data_points().next() { + let attrs = point + .attributes() + .map(|kv| { + (kv.key.as_str().to_string(), kv.value.as_str().into_owned()) + }) + .collect(); + return Some((attrs, point.value())); + } + } + } + } + } + None + } + + #[test] + fn hedged_metric_off_by_default_even_for_hedged_operation() { + let harness = test_meter(); + let handler = CosmosMetricsHandler::with_meter(harness.meter.clone()); + + let cx = Context::new().with_value(operation_context()); + handler.handle(&hedged_completed(200), &cx); + + // The stable duration metric is emitted, but the opt-in hedged counter is not. + let names = metric_names(&harness.collect()); + assert!(names + .iter() + .any(|n| n == attributes::METRIC_OPERATION_DURATION)); + assert!(!names + .iter() + .any(|n| n == attributes::METRIC_OPERATION_HEDGED)); + } + + #[test] + fn hedged_metric_emitted_for_hedged_operation_when_enabled() { + let harness = test_meter(); + let options = MetricsOptions::default().with_hedged_metric(true); + let handler = CosmosMetricsHandler::with_meter_and_options(harness.meter.clone(), options); + + let cx = Context::new().with_value(operation_context()); + handler.handle(&hedged_completed(200), &cx); + + let metrics = harness.collect(); + let (attrs, value) = hedged_point(&metrics).expect("hedged counter should be emitted"); + assert_eq!(value, 1); + + // Operation identity carries through, plus the low-cardinality terminal state. + assert_eq!( + attrs + .get(attributes::ATTR_DB_OPERATION_NAME) + .map(String::as_str), + Some("read_item") + ); + assert_eq!( + attrs + .get(attributes::ATTR_HEDGE_TERMINAL_STATE) + .map(String::as_str), + Some("alternate_won") + ); + // The high-cardinality region dimension stays off without extended attributes. + assert!(!attrs.contains_key(attributes::ATTR_HEDGE_REGION)); + } + + #[test] + fn hedged_metric_adds_region_dimension_under_extended_attributes() { + let harness = test_meter(); + let options = MetricsOptions::default() + .with_hedged_metric(true) + .with_extended_attributes(true); + let handler = CosmosMetricsHandler::with_meter_and_options(harness.meter.clone(), options); + + let cx = Context::new().with_value(operation_context()); + handler.handle(&hedged_completed(200), &cx); + + let metrics = harness.collect(); + let (attrs, _) = hedged_point(&metrics).expect("hedged counter should be emitted"); + assert_eq!( + attrs.get(attributes::ATTR_HEDGE_REGION).map(String::as_str), + Some("westus2") + ); + } + + #[test] + fn hedged_metric_not_emitted_for_non_hedged_operation() { + let harness = test_meter(); + let options = MetricsOptions::default().with_hedged_metric(true); + let handler = CosmosMetricsHandler::with_meter_and_options(harness.meter.clone(), options); + + // Enabled, but this operation never hedged: no counter data point. + let cx = Context::new().with_value(operation_context()); + handler.handle(&completed(200), &cx); + + let metrics = harness.collect(); + assert!( + hedged_point(&metrics).is_none(), + "a non-hedged operation must not increment the hedged counter" + ); + } } diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs index 78e60c598e0..2a5ccc42e5e 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs @@ -8,7 +8,7 @@ //! reference-counted). Building them eagerly keeps the per-operation hot path to //! just `record`/`add` calls with no allocation of instrument state. -use opentelemetry::metrics::{Histogram, Meter}; +use opentelemetry::metrics::{Counter, Histogram, Meter}; use crate::diagnostics::metrics::attributes; @@ -17,10 +17,10 @@ use crate::diagnostics::metrics::attributes; /// The stable operation-duration histogram is always recorded; the remaining /// per-signal instruments are recorded only when the matching /// [`MetricsOptions`](super::MetricsOptions) toggle -/// (`request_charge_metric_enabled` / `returned_rows_metric_enabled`) is set. -/// They are still created unconditionally because instrument creation is cheap -/// and idempotent, and doing so keeps the handler's record path branch-free per -/// instrument. +/// (`request_charge_metric_enabled` / `returned_rows_metric_enabled` / +/// `hedged_metric_enabled`) is set. They are still created unconditionally +/// because instrument creation is cheap and idempotent, and doing so keeps the +/// handler's record path branch-free per instrument. #[derive(Clone)] pub(crate) struct Instruments { /// Stable: `db.client.operation.duration` (seconds). @@ -31,6 +31,10 @@ pub(crate) struct Instruments { /// Development: `db.client.response.returned_rows` (rows). pub(crate) returned_rows: Histogram, + + /// Development: `azure.cosmosdb.client.operation.hedged` (operations that + /// dispatched a cross-region hedge fan-out). + pub(crate) hedged: Counter, } impl Instruments { @@ -54,10 +58,17 @@ impl Instruments { .with_description("Number of rows/items returned by a Cosmos DB operation.") .build(); + let hedged = meter + .u64_counter(attributes::METRIC_OPERATION_HEDGED) + .with_unit(attributes::UNIT_OPERATION) + .with_description("Cosmos DB operations that dispatched a cross-region hedge fan-out.") + .build(); + Self { operation_duration, request_charge, returned_rows, + hedged, } } } diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/options.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/options.rs index 2fddee6fc6a..973a1c0e909 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/options.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/options.rs @@ -42,6 +42,7 @@ pub struct MetricsOptions { request_charge_metric: bool, returned_rows_metric: bool, + hedged_metric: bool, extended_attributes: bool, } @@ -68,6 +69,20 @@ impl MetricsOptions { self } + /// Enables (or disables) the `azure.cosmosdb.client.operation.hedged` counter + /// (operations that dispatched a cross-region hedge fan-out). Off by default. + /// + /// The counter increments only for operations where hedging actually fanned + /// out, so it is near-zero cardinality; it always carries the low-cardinality + /// `hedge_terminal_state` dimension. The higher-cardinality hedge-region + /// dimension is added only when [`with_extended_attributes`](Self::with_extended_attributes) + /// is also enabled. + #[must_use] + pub fn with_hedged_metric(mut self, enabled: bool) -> Self { + self.hedged_metric = enabled; + self + } + /// Enables (or disables) the extended attribute set on every emitted metric: /// consistency level, contacted regions, sub-status code, and connection /// mode. These can be higher cardinality, so they are opt-in and off by @@ -88,6 +103,11 @@ impl MetricsOptions { self.returned_rows_metric } + /// Whether the hedged-operation counter is emitted. + pub fn hedged_metric_enabled(&self) -> bool { + self.hedged_metric + } + /// Whether the extended attribute set is attached to emitted metrics. pub fn extended_attributes_enabled(&self) -> bool { self.extended_attributes diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs index f678868c2d9..a48af6607eb 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs @@ -19,11 +19,15 @@ mod tests { use std::time::{Duration, Instant, SystemTime}; use azure_core::http::StatusCode; - use azure_data_cosmos_driver::diagnostics::{DiagnosticsContext, RequestDiagnostics}; + use azure_data_cosmos_driver::diagnostics::{ + DiagnosticsContext, ExecutionContext, HedgeDiagnostics, HedgeTerminalState, + RequestDiagnostics, + }; use azure_data_cosmos_driver::models::{ActivityId, RequestCharge}; use azure_data_cosmos_driver::options::Region; use azure_data_cosmos_driver::{CosmosStatus, DiagnosticsThresholds}; use opentelemetry::trace::{SpanId, TracerProvider}; + use opentelemetry::{Array, Value}; use opentelemetry_sdk::trace::{in_memory_exporter::InMemorySpanExporter, SdkTracerProvider}; use super::handler::should_emit_span; @@ -338,4 +342,149 @@ mod tests { "root must start no later than its earliest child" ); } + + /// Builds a hedged, failed operation: an initial East US leg (429) plus a + /// speculative hedge leg to West US 2 (200) tagged `ExecutionContext::Hedging`, + /// with `AlternateWon` hedge diagnostics. + fn hedged_context(anchor: Instant) -> DiagnosticsContext { + let primary = RequestDiagnostics::for_testing( + "https://acct-eastus.documents.azure.com:443/", + Some(Region::EAST_US), + CosmosStatus::new(StatusCode::TooManyRequests), + RequestCharge::new(2.0), + anchor - Duration::from_millis(300), + anchor - Duration::from_millis(200), + ); + let hedge_leg = RequestDiagnostics::for_testing( + "https://acct-westus2.documents.azure.com:443/", + Some(Region::WEST_US_2), + CosmosStatus::new(StatusCode::Ok), + RequestCharge::new(2.0), + anchor - Duration::from_millis(150), + anchor - Duration::from_millis(50), + ) + .with_execution_context_for_testing(ExecutionContext::Hedging); + let hedge = HedgeDiagnostics::for_testing( + Region::EAST_US, + Some(Region::WEST_US_2), + Some(Region::WEST_US_2), + HedgeTerminalState::AlternateWon, + ); + DiagnosticsContext::for_testing_with_hedge( + ActivityId::new_uuid(), + Duration::from_millis(250), + Some(CosmosStatus::new(StatusCode::TooManyRequests)), + Some("read_item"), + vec![primary, hedge_leg], + Some(hedge), + ) + } + + /// Returns the string members of a `string[]` span attribute, if present. + fn string_array_attr( + span: &opentelemetry_sdk::trace::SpanData, + key: &str, + ) -> Option> { + span.attributes.iter().find_map(|kv| { + if kv.key.as_str() != key { + return None; + } + match &kv.value { + Value::Array(Array::String(values)) => { + Some(values.iter().map(|v| v.as_str().to_string()).collect()) + } + _ => None, + } + }) + } + + #[test] + fn hedged_operation_surfaces_hedging_span_attributes() { + let (provider, exporter) = exportable(); + let tracer = provider.tracer("test"); + let now_instant = Instant::now(); + let now_system = SystemTime::now(); + + let ctx = hedged_context(now_instant); + emit_backdated_span_tree(&tracer, &ctx, None, None, now_instant, now_system); + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + let root = spans + .iter() + .find(|s| s.name == "read_item") + .expect("root span present"); + + // hedging_started = true (bool). + assert!( + root.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGING_STARTED + && matches!(kv.value, Value::Bool(true)) + }), + "root span must carry hedging_started = true" + ); + // hedge region = the alternate region the hedge fanned out to. + assert!(root.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGE_REGION && kv.value.as_str() == "westus2" + })); + // terminal state = alternate_won. + assert!(root.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGE_TERMINAL_STATE + && kv.value.as_str() == "alternate_won" + })); + // requested_regions is an ordered string[] carrying both dispatched regions. + let requested = string_array_attr(root, attributes::REQUESTED_REGIONS) + .expect("requested_regions string[] present"); + assert!(requested.iter().any(|r| r == "eastus")); + assert!(requested.iter().any(|r| r == "westus2")); + // responded_regions is an ordered string[] carrying both responders. + let responded = string_array_attr(root, attributes::RESPONDED_REGIONS) + .expect("responded_regions string[] present"); + assert!(responded.iter().any(|r| r == "eastus")); + assert!(responded.iter().any(|r| r == "westus2")); + + // The speculative hedge leg's child span is tagged. + let tagged = spans + .iter() + .filter(|s| s.name == "cosmosdb.request") + .any(|s| { + s.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGE_LEG + && matches!(kv.value, Value::Bool(true)) + }) + }); + assert!(tagged, "the hedge leg child span must carry the hedge tag"); + } + + #[test] + fn non_hedged_operation_omits_hedging_span_attributes() { + let (provider, exporter) = exportable(); + let tracer = provider.tracer("test"); + let now_instant = Instant::now(); + let now_system = SystemTime::now(); + + // A plain failed operation — no hedge diagnostics, all Initial legs. + let ctx = context( + Duration::from_millis(5), + Some(CosmosStatus::new(StatusCode::TooManyRequests)), + Some("read_item"), + &[(5, 5, CosmosStatus::new(StatusCode::TooManyRequests))], + now_instant, + ); + emit_backdated_span_tree(&tracer, &ctx, None, None, now_instant, now_system); + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + for span in &spans { + for kv in span.attributes.iter() { + let key = kv.key.as_str(); + assert_ne!(key, attributes::HEDGING_STARTED); + assert_ne!(key, attributes::HEDGE_REGION); + assert_ne!(key, attributes::HEDGE_TERMINAL_STATE); + assert_ne!(key, attributes::REQUESTED_REGIONS); + assert_ne!(key, attributes::RESPONDED_REGIONS); + assert_ne!(key, attributes::HEDGE_LEG); + } + } + } } diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs index f8152ec7a30..2b3b4632175 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs @@ -22,7 +22,9 @@ use opentelemetry::{ Array, Context, KeyValue, StringValue, Value, }; -use azure_data_cosmos_driver::diagnostics::{DiagnosticsContext, RequestDiagnostics}; +use azure_data_cosmos_driver::diagnostics::{ + DiagnosticsContext, ExecutionContext, RequestDiagnostics, +}; use crate::diagnostics::attributes; use crate::diagnostics::CosmosOperationContext; @@ -33,6 +35,19 @@ const DEFAULT_OPERATION_SPAN_NAME: &str = "cosmosdb.operation"; /// Span name for each per-attempt ("child") span. const REQUEST_SPAN_NAME: &str = "cosmosdb.request"; +/// Builds an OpenTelemetry ordered `string[]` [`Value`] from region names. +/// +/// The Cosmos semantic conventions model region lists (contacted, requested, +/// responded) as ordered `string[]`; this emits an array value rather than a +/// joined scalar. +fn region_string_array<'a>(regions: impl IntoIterator) -> Value { + let values: Vec = regions + .into_iter() + .map(|region| StringValue::from(region.to_string())) + .collect(); + Value::Array(Array::String(values)) +} + /// Emits a backdated operation span with one child span per retained attempt. /// /// * `tracer` — the OpenTelemetry tracer to build spans on. Generic so tests can @@ -161,6 +176,37 @@ pub(crate) fn emit_backdated_span_tree( Value::Array(Array::String(values)), )); } + // Hedging surfacing: only when a cross-region hedge actually fanned out. + // These attributes stay off the common (non-hedged) sampled span entirely. + if diagnostics.hedging_started() { + root_attrs.push(KeyValue::new(attributes::HEDGING_STARTED, true)); + if let Some(hedge) = diagnostics.hedge_diagnostics() { + if let Some(alternate) = hedge.alternate_region() { + root_attrs.push(KeyValue::new( + attributes::HEDGE_REGION, + alternate.as_str().to_string(), + )); + } + root_attrs.push(KeyValue::new( + attributes::HEDGE_TERMINAL_STATE, + hedge.terminal_state().as_str(), + )); + } + let requested = diagnostics.requested_regions(); + if !requested.is_empty() { + root_attrs.push(KeyValue::new( + attributes::REQUESTED_REGIONS, + region_string_array(requested.iter().map(|r| r.region.as_str())), + )); + } + let responded = diagnostics.responded_regions(); + if !responded.is_empty() { + root_attrs.push(KeyValue::new( + attributes::RESPONDED_REGIONS, + region_string_array(responded.iter().map(|region| region.as_str())), + )); + } + } // Prefer the caller-supplied server-address override (mirroring the metrics // handler) before falling back to the host of the first contacted endpoint, // so an override changes both the metric and the root span consistently. @@ -251,6 +297,11 @@ pub(crate) fn emit_backdated_span_tree( )])), )); } + // Tag the speculative hedge leg so the child span is attributable to the + // hedge fan-out rather than an initial/retry dispatch. + if matches!(req.execution_context(), ExecutionContext::Hedging) { + child_attrs.push(KeyValue::new(attributes::HEDGE_LEG, true)); + } if let Some(addr) = server_address(req) { child_attrs.push(KeyValue::new(attributes::SERVER_ADDRESS, addr)); } diff --git a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md index aca49b571bc..971e94544f4 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md @@ -5,6 +5,7 @@ ### Features Added - Added the Hedging Detection API on `DiagnosticsContext`: `hedging_started()` (whether a hedge arm actually fanned out), `requested_regions()` (regions dispatched to, in dispatch order with duplicates, each tagged with a reason), and `responded_regions()` (regions that produced an actual service reply, in completion order). Adds the public `RequestedRegion` struct and `RequestedRegionReason` enum (both `#[non_exhaustive]`, with a total `From` mapping) and the new `ExecutionContext::OperationRetry` variant. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) +- Added `HedgeTerminalState::as_str()` (and a matching `Display`) returning a stable, low-cardinality `snake_case` identifier for the hedging race outcome, for use as an observability attribute / log-field value. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) ### Breaking Changes diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs index ff3ae827070..89acc964bc4 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs @@ -638,6 +638,24 @@ impl RequestDiagnostics { } } + /// **Internal test helper — do not call.** + /// + /// Overrides the [`ExecutionContext`] of a test-constructed request so + /// emission-layer tests can synthesize a speculative hedge leg (or a retry + /// / failover dispatch). Gated behind the + /// `__internal_test_diagnostics_construction` Cargo feature and + /// `#[doc(hidden)]`, mirroring [`for_testing`](Self::for_testing). + #[cfg(feature = "__internal_test_diagnostics_construction")] + #[doc(hidden)] + #[must_use] + pub fn with_execution_context_for_testing( + mut self, + execution_context: ExecutionContext, + ) -> Self { + self.execution_context = execution_context; + self + } + /// Records completion of this request. /// /// Since we received a response, the request was definitely sent. @@ -2047,7 +2065,33 @@ impl DiagnosticsContext { } } - /// Concatenates the per-request diagnostics from a sequence of + /// **Internal test helper — do not call.** + /// + /// Like [`for_testing_with_requests`](Self::for_testing_with_requests), but + /// also attaches `hedge_diagnostics` so the wrapper SDK's emission-layer + /// tests can exercise the hedging-surfacing paths (tracing / metrics / + /// logging). Gated behind the `__internal_test_diagnostics_construction` + /// Cargo feature and `#[doc(hidden)]`, mirroring [`for_testing`](Self::for_testing). + #[cfg(feature = "__internal_test_diagnostics_construction")] + #[doc(hidden)] + pub fn for_testing_with_hedge( + activity_id: ActivityId, + duration: Duration, + status: Option, + operation_name: Option<&str>, + requests: Vec, + hedge_diagnostics: Option, + ) -> Self { + let mut context = Self::for_testing_with_requests( + activity_id, + duration, + status, + operation_name, + requests, + ); + context.hedge_diagnostics = hedge_diagnostics; + context + } /// sub-operation contexts into a single aggregated [`DiagnosticsContext`]. /// /// Used by the PATCH handler to surface **one operation = one diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/hedging_diagnostics.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/hedging_diagnostics.rs index 0d2310d18cd..d09b348091c 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/hedging_diagnostics.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/hedging_diagnostics.rs @@ -91,6 +91,31 @@ pub enum HedgeTerminalState { CancelledAwaitingPartner, } +impl HedgeTerminalState { + /// Returns a stable, low-cardinality `snake_case` identifier for this + /// terminal state, suitable as an observability attribute / log-field value. + /// + /// The [`BothTransient`](Self::BothTransient) variant collapses to + /// `"both_transient"` regardless of its `deadline_elapsed` payload, keeping + /// the emitted value set bounded to one string per variant. + pub fn as_str(&self) -> &'static str { + match self { + HedgeTerminalState::PrimaryWonPreThreshold => "primary_won_pre_threshold", + HedgeTerminalState::DeadlineExceededPreThreshold => "deadline_exceeded_pre_threshold", + HedgeTerminalState::PrimaryWonAfterHedge => "primary_won_after_hedge", + HedgeTerminalState::AlternateWon => "alternate_won", + HedgeTerminalState::BothTransient { .. } => "both_transient", + HedgeTerminalState::CancelledAwaitingPartner => "cancelled_awaiting_partner", + } + } +} + +impl std::fmt::Display for HedgeTerminalState { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(self.as_str()) + } +} + /// Diagnostic information about a hedging execution, attached to the /// winning response when a hedging strategy was active for the operation. /// @@ -294,6 +319,35 @@ impl HedgeDiagnostics { pub fn terminal_state(&self) -> HedgeTerminalState { self.terminal_state } + + /// **Internal test helper — do not call.** + /// + /// Builds a [`HedgeDiagnostics`] from explicit parts so the wrapper SDK + /// (`azure_data_cosmos`) can exercise the observability layer's hedging + /// surfacing (tracing / metrics / logging) against a realistic hedged + /// context without standing up the hedging pipeline. The strategy config is + /// filled with a fixed non-zero threshold since the emission layer does not + /// read it. Gated behind the `__internal_test_diagnostics_construction` + /// Cargo feature and `#[doc(hidden)]`, so it never appears on the public + /// surface. + #[cfg(feature = "__internal_test_diagnostics_construction")] + #[doc(hidden)] + pub fn for_testing( + primary_region: Region, + alternate_region: Option, + response_region: Option, + terminal_state: HedgeTerminalState, + ) -> Self { + let threshold = HedgeThreshold::new(std::time::Duration::from_millis(500)) + .expect("500ms is a valid non-zero hedge threshold"); + Self { + strategy_config: HedgingStrategyConfig::new(threshold), + primary_region, + alternate_region, + response_region, + terminal_state, + } + } } #[cfg(test)] From f861e9978c6a32ce2c2565b646eeb98e86a0090f Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Mon, 27 Jul 2026 12:23:19 -0700 Subject: [PATCH 03/21] Fix hedge-race region attribution + observability consistency Address review feedback on the hedging detection API: - requested_regions(): recover a structurally-dropped hedge fan-out leg from the authoritative hedge_diagnostics (primary->Initial, alternate-> Hedging, in dispatch order) so a clean primary/alternate win no longer omits a dispatched region or contradicts the hedge_region attribute. - aggregate_sub_operations(): propagate a representative hedge_diagnostics (prefer a fanned-out sub-op) so aggregated ops (PATCH) report hedging consistently instead of dropping it. - metrics record_hedged(): guard on hedge_diagnostics so the hedged counter never emits a data point missing the hedge_terminal_state dimension. - Docs/changelogs/spec: clarify that deprecated ExecutionContext::Retry is a distinct variant still serializing "retry" (the wire change comes from dispatch sites emitting OperationRetry); add the SDK breaking-change entry; note that requests is not a guaranteed-complete append-only list and regions_contacted is first-contact order (not sorted); document the recovered-leg behavior and that a dropped hedge leg has no HEDGE_LEG child span (the root span carries the authoritative signal). - Tests: replace impossible both-legs synthetic shapes with production shapes (PrimaryWonAfterHedge, AlternateWon single-leg) and add recovery, PATCH aggregation, PrimaryWonPreThreshold, metric-guard, and HedgeTerminalState::as_str all-variant coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- sdk/cosmos/azure_data_cosmos/CHANGELOG.md | 2 + .../docs/HEDGING_DETECTION_API_SPEC.md | 35 ++- .../src/diagnostics/metrics/handler.rs | 81 ++++- .../src/diagnostics/tracing/mod.rs | 119 +++++++- .../src/diagnostics/tracing/span_builder.rs | 9 +- .../azure_data_cosmos_driver/CHANGELOG.md | 2 +- .../src/diagnostics/diagnostics_context.rs | 280 +++++++++++++++++- .../driver/pipeline/hedging_diagnostics.rs | 44 +++ 8 files changed, 516 insertions(+), 56 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md index 54a9dfd5626..ad8e46641ba 100644 --- a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md @@ -12,6 +12,8 @@ ### Breaking Changes +- Serialized `DiagnosticsContext` output (the diagnostics JSON surfaced to consumers, and the sampled diagnostics log line) now serializes driver-generated operation retries with `execution_context` = `"operation_retry"` instead of `"retry"`, following the driver's `ExecutionContext::Retry` → `OperationRetry` rename. This is additive to the enum (the deprecated `Retry` variant still serializes as `"retry"`), but the wire value emitted for operation retries changes; telemetry/log parsers that match the literal `"retry"` execution context must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) + ### Bugs Fixed ### Other Changes diff --git a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md index 77339c8016f..8031725c29f 100644 --- a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md +++ b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md @@ -33,9 +33,9 @@ Rust-native `HedgeDiagnostics` surface (see §5); the two are complementary. | Item | Signature | Notes | | --- | --- | --- | | `DiagnosticsContext` | re-exported as `azure_data_cosmos::DiagnosticsContext` | The per-operation diagnostics handle. | -| `DiagnosticsContext::requests` | `-> Arc>` | All dispatched attempts, in **dispatch order** (append-only). Cloning the `Arc` is a cheap atomic increment. | +| `DiagnosticsContext::requests` | `-> Arc>` | Retained per-attempt records in dispatch order — **not** a guaranteed-complete append-only history: under a `429`/`410` retry storm the list is bounded/compacted (see `max_request_diagnostics`, which can drop or reorder entries), and a structurally-dropped hedge loser leg is absent. Cloning the `Arc` is a cheap atomic increment. | | `DiagnosticsContext::hedge_diagnostics` | `-> Option<&HedgeDiagnostics>` | `Some` whenever a hedging strategy was active for the operation (including primary-wins-under-threshold). | -| `DiagnosticsContext::regions_contacted` | `-> Vec` | **Sorted and deduplicated** distinct regions — not dispatch order. | +| `DiagnosticsContext::regions_contacted` | `-> Vec` | Distinct regions **deduplicated in first-contact (failover) order — not sorted**, captured from the full attempt list before compaction. | | `RequestDiagnostics::region` | `-> Option<&Region>` | `None` for pre-region-selection failures. | | `RequestDiagnostics::execution_context` | `-> ExecutionContext` | Why this attempt was dispatched (see §3). | | `RequestDiagnostics::completed_at` | `-> Option` | Set by `complete()`, `timeout()`, **and** `fail_transport()` — so "completed" alone is not "responded" (see §4.3). | @@ -86,10 +86,13 @@ clearly distinct from the transport-level `TransportRetry`. The hand-written `transport_pipeline.rs`, `cosmos_driver.rs`) are updated accordingly. **Compatibility.** The old `Retry` variant is retained for one release as a -`#[deprecated]` alias so existing source keeps compiling. The **serialized form -changes from `"retry"` to `"operation_retry"`**: telemetry parsers that match on -the literal `"retry"` must update. (`ExecutionContext` derives `Serialize` only, -not `Deserialize`, so no `#[serde(alias)]` is needed.) +distinct `#[deprecated]` variant (**not** a serde alias) so existing source keeps +compiling; a `Retry` value still serializes as `"retry"`. The customer-visible +wire-format change is that the dispatch sites now emit `OperationRetry` for +driver-generated operation retries, so those attempts serialize as +`"operation_retry"` instead of `"retry"`; telemetry parsers that match the +literal `"retry"` execution context must update. (`ExecutionContext` derives +`Serialize` only, not `Deserialize`, so no `#[serde(alias)]` is needed.) --- @@ -128,10 +131,22 @@ correct if a future change ever drifts one signal. ### 4.3 Regions dispatched to, with reason — `requested_regions()` -Dispatch order, duplicates preserved (a region dispatched twice appears twice), -entries with no resolved region skipped. The initial attempt is included and -tagged `RequestedRegionReason::Initial`. This is distinct from -`regions_contacted()`, which is sorted and deduplicated. +Retained attempts in dispatch order, duplicates preserved (a region dispatched +twice appears twice), entries with no resolved region skipped. The initial +attempt is included and tagged `RequestedRegionReason::Initial`. + +When a hedge fanned out and the race resolved as a **clean win** (the primary +wins after the threshold, or the alternate wins outright), the losing leg's +per-request record is structurally dropped before it can be merged (see §5 and +`HedgeDiagnostics`). This accessor recovers the missing fan-out leg(s) from the +authoritative `hedge_diagnostics` — the primary tagged `Initial` and the +alternate tagged `Hedging`, in dispatch order — so both dispatched regions are +always represented and consistent with the `hedge_region` telemetry attribute. A +recovered leg has **no** `responded_regions()` entry (a dropped leg never +produced a service reply). This is distinct from `regions_contacted()`, which is +deduplicated in first-contact order; under a retry storm the retained attempt +list may be compacted, so `regions_contacted()` (captured pre-compaction) is the +complete distinct-region set. ### 4.4 Regions that responded — `responded_regions()` diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs index 0d8c5c935aa..93a22bbb0c0 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs @@ -217,20 +217,28 @@ impl CosmosMetricsHandler { /// the higher-cardinality `hedge_region` dimension is added only under the /// extended-attributes opt-in (mirroring how contacted regions are gated on /// the duration metric). + /// + /// The counter is emitted only when `hedge_diagnostics` is present, so a + /// data point can never be recorded without its `hedge_terminal_state` + /// dimension — a mixed attribute schema on the same counter would fragment + /// its time series and break `group by hedge_terminal_state`. An aggregated + /// operation (e.g. PATCH) whose sub-op hedged propagates a representative + /// `hedge_diagnostics`, so this stays consistent with non-aggregated ops. fn record_hedged(&self, diagnostics: &DiagnosticsContext, base_attrs: &[KeyValue]) { + let Some(hedge) = diagnostics.hedge_diagnostics() else { + return; + }; let mut attrs = base_attrs.to_vec(); - if let Some(hedge) = diagnostics.hedge_diagnostics() { - attrs.push(KeyValue::new( - attributes::ATTR_HEDGE_TERMINAL_STATE, - hedge.terminal_state().as_str(), - )); - if self.options.extended_attributes_enabled() { - if let Some(alternate) = hedge.alternate_region() { - attrs.push(KeyValue::new( - attributes::ATTR_HEDGE_REGION, - alternate.as_str().to_string(), - )); - } + attrs.push(KeyValue::new( + attributes::ATTR_HEDGE_TERMINAL_STATE, + hedge.terminal_state().as_str(), + )); + if self.options.extended_attributes_enabled() { + if let Some(alternate) = hedge.alternate_region() { + attrs.push(KeyValue::new( + attributes::ATTR_HEDGE_REGION, + alternate.as_str().to_string(), + )); } } self.instruments.hedged.add(1, &attrs); @@ -703,4 +711,53 @@ mod tests { "a non-hedged operation must not increment the hedged counter" ); } + + #[test] + fn hedged_metric_skips_when_terminal_state_unavailable() { + // Defensive: if an operation reads as hedged (a `Hedging`-tagged request) + // but carries no `hedge_diagnostics`, the counter is skipped rather than + // emitted without its `hedge_terminal_state` dimension — a missing + // dimension would fragment the counter's time series. Real aggregated + // operations (e.g. PATCH) propagate a representative `hedge_diagnostics`, + // so this is a belt-and-suspenders guard. + use azure_data_cosmos_driver::diagnostics::{ExecutionContext, RequestDiagnostics}; + use azure_data_cosmos_driver::models::RequestCharge; + use std::time::Instant; + + let harness = test_meter(); + let options = MetricsOptions::default().with_hedged_metric(true); + let handler = CosmosMetricsHandler::with_meter_and_options(harness.meter.clone(), options); + + let now = Instant::now(); + let hedge_leg = RequestDiagnostics::for_testing( + "https://acct-westus2.documents.azure.com:443/", + Some(Region::WEST_US_2), + CosmosStatus::new(StatusCode::Ok), + RequestCharge::new(1.0), + now - Duration::from_millis(50), + now, + ) + .with_execution_context_for_testing(ExecutionContext::Hedging); + let ctx = DiagnosticsContext::for_testing_with_hedge( + ActivityId::new_uuid(), + Duration::from_millis(42), + Some(CosmosStatus::new(StatusCode::Ok)), + Some("read_item"), + vec![hedge_leg], + None, + ); + assert!( + ctx.hedging_started(), + "a Hedging-tagged request makes hedging_started() true" + ); + + let cx = Context::new().with_value(operation_context()); + handler.handle(&ctx, &cx); + + let metrics = harness.collect(); + assert!( + hedged_point(&metrics).is_none(), + "the counter must not emit a data point missing the hedge_terminal_state dimension" + ); + } } diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs index a48af6607eb..2f5ad822b0e 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs @@ -343,18 +343,13 @@ mod tests { ); } - /// Builds a hedged, failed operation: an initial East US leg (429) plus a - /// speculative hedge leg to West US 2 (200) tagged `ExecutionContext::Hedging`, - /// with `AlternateWon` hedge diagnostics. + /// Builds a hedged operation in its **production `AlternateWon` shape**: the + /// primary East US leg lost the race and was structurally dropped (so it has + /// no retained per-request record), leaving only the winning speculative + /// hedge leg to West US 2 (200) tagged `ExecutionContext::Hedging`, with + /// `AlternateWon` hedge diagnostics naming both regions. The dropped primary + /// region is recovered for `requested_regions()` from the hedge diagnostics. fn hedged_context(anchor: Instant) -> DiagnosticsContext { - let primary = RequestDiagnostics::for_testing( - "https://acct-eastus.documents.azure.com:443/", - Some(Region::EAST_US), - CosmosStatus::new(StatusCode::TooManyRequests), - RequestCharge::new(2.0), - anchor - Duration::from_millis(300), - anchor - Duration::from_millis(200), - ); let hedge_leg = RequestDiagnostics::for_testing( "https://acct-westus2.documents.azure.com:443/", Some(Region::WEST_US_2), @@ -373,9 +368,9 @@ mod tests { DiagnosticsContext::for_testing_with_hedge( ActivityId::new_uuid(), Duration::from_millis(250), - Some(CosmosStatus::new(StatusCode::TooManyRequests)), + Some(CosmosStatus::new(StatusCode::Ok)), Some("read_item"), - vec![primary, hedge_leg], + vec![hedge_leg], Some(hedge), ) } @@ -432,16 +427,22 @@ mod tests { kv.key.as_str() == attributes::HEDGE_TERMINAL_STATE && kv.value.as_str() == "alternate_won" })); - // requested_regions is an ordered string[] carrying both dispatched regions. + // requested_regions carries both dispatched regions — the winning hedge + // leg (westus2) plus the structurally-dropped primary (eastus) recovered + // from the hedge diagnostics. let requested = string_array_attr(root, attributes::REQUESTED_REGIONS) .expect("requested_regions string[] present"); assert!(requested.iter().any(|r| r == "eastus")); assert!(requested.iter().any(|r| r == "westus2")); - // responded_regions is an ordered string[] carrying both responders. + // responded_regions carries only the winning hedge region: the dropped + // primary leg never produced a service reply. let responded = string_array_attr(root, attributes::RESPONDED_REGIONS) .expect("responded_regions string[] present"); - assert!(responded.iter().any(|r| r == "eastus")); assert!(responded.iter().any(|r| r == "westus2")); + assert!( + !responded.iter().any(|r| r == "eastus"), + "the structurally-dropped primary leg must not appear in responded_regions" + ); // The speculative hedge leg's child span is tagged. let tagged = spans @@ -456,6 +457,92 @@ mod tests { assert!(tagged, "the hedge leg child span must carry the hedge tag"); } + /// Builds a hedged operation in its **production `PrimaryWonAfterHedge` + /// shape**: the primary East US leg won after the threshold, so the + /// speculative West US 2 hedge leg was structurally cancelled and has no + /// retained record. Only the primary (Initial, 200) survives; the hedge + /// region is recovered from the hedge diagnostics. + fn primary_won_after_hedge_context(anchor: Instant) -> DiagnosticsContext { + let primary = RequestDiagnostics::for_testing( + "https://acct-eastus.documents.azure.com:443/", + Some(Region::EAST_US), + CosmosStatus::new(StatusCode::Ok), + RequestCharge::new(2.0), + anchor - Duration::from_millis(300), + anchor - Duration::from_millis(180), + ); + let hedge = HedgeDiagnostics::for_testing( + Region::EAST_US, + Some(Region::WEST_US_2), + Some(Region::EAST_US), + HedgeTerminalState::PrimaryWonAfterHedge, + ); + DiagnosticsContext::for_testing_with_hedge( + ActivityId::new_uuid(), + Duration::from_millis(200), + Some(CosmosStatus::new(StatusCode::Ok)), + Some("read_item"), + vec![primary], + Some(hedge), + ) + } + + #[test] + fn primary_won_after_hedge_recovers_hedge_region_without_child_span() { + // When the primary wins cleanly the hedge leg is dropped, so there is no + // child span to tag. The authoritative hedge signal must still be on the + // root span, and the dropped hedge region recovered into requested_regions. + let (provider, exporter) = exportable(); + let tracer = provider.tracer("test"); + let now_instant = Instant::now(); + let now_system = SystemTime::now(); + + let ctx = primary_won_after_hedge_context(now_instant); + emit_backdated_span_tree(&tracer, &ctx, None, None, now_instant, now_system); + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + let root = spans + .iter() + .find(|s| s.name == "read_item") + .expect("root span present"); + + assert!(root.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGING_STARTED && matches!(kv.value, Value::Bool(true)) + })); + assert!(root.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGE_REGION && kv.value.as_str() == "westus2" + })); + assert!(root.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGE_TERMINAL_STATE + && kv.value.as_str() == "primary_won_after_hedge" + })); + // The dropped hedge leg's region is recovered into requested_regions... + let requested = string_array_attr(root, attributes::REQUESTED_REGIONS) + .expect("requested_regions string[] present"); + assert!(requested.iter().any(|r| r == "eastus")); + assert!(requested.iter().any(|r| r == "westus2")); + // ...but only the winning primary produced a response. + let responded = string_array_attr(root, attributes::RESPONDED_REGIONS) + .expect("responded_regions string[] present"); + assert_eq!(responded, vec!["eastus".to_string()]); + // The hedge leg was structurally dropped, so no child span carries the + // hedge tag; the root span's hedge attributes carry the signal instead. + let any_hedge_child = spans + .iter() + .filter(|s| s.name == "cosmosdb.request") + .any(|s| { + s.attributes.iter().any(|kv| { + kv.key.as_str() == attributes::HEDGE_LEG + && matches!(kv.value, Value::Bool(true)) + }) + }); + assert!( + !any_hedge_child, + "a structurally-dropped hedge leg must not produce a tagged child span" + ); + } + #[test] fn non_hedged_operation_omits_hedging_span_attributes() { let (provider, exporter) = exportable(); diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs index 2b3b4632175..d94f34fb252 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs @@ -298,7 +298,14 @@ pub(crate) fn emit_backdated_span_tree( )); } // Tag the speculative hedge leg so the child span is attributable to the - // hedge fan-out rather than an initial/retry dispatch. + // hedge fan-out rather than an initial/retry dispatch. This tag is + // present only for a *retained* hedge-leg record: when the primary wins + // a clean race the alternate leg is structurally cancelled before it + // produces a per-request record, so no child span exists for it. In that + // case the authoritative hedge signal lives on the root span + // (`hedge_started` / `hedge_region` / `hedge_terminal_state`, plus the + // alternate region in `requested_regions`), so the fan-out is still + // attributable even without a tagged child. if matches!(req.execution_context(), ExecutionContext::Hedging) { child_attrs.push(KeyValue::new(attributes::HEDGE_LEG, true)); } diff --git a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md index 971e94544f4..e7607d8fa5f 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md @@ -9,7 +9,7 @@ ### Breaking Changes -- Renamed `ExecutionContext::Retry` to `ExecutionContext::OperationRetry` to distinguish operation-level retries from transport-level `TransportRetry`. The old `Retry` variant remains for one release as a `#[deprecated]` alias, but its serialized form changes from `"retry"` to `"operation_retry"`; telemetry parsers that match the literal `"retry"` must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) +- Renamed `ExecutionContext::Retry` to `ExecutionContext::OperationRetry` to distinguish operation-level retries from transport-level `TransportRetry`. The old `Retry` remains for one release as a distinct `#[deprecated]` variant (**not** a serde alias); a `Retry` value still serializes as `"retry"`. The customer-visible wire-format change is that driver-generated operation retries now serialize as `"operation_retry"` instead of `"retry"`, because the dispatch sites emit `OperationRetry`; telemetry parsers that match the literal `"retry"` execution context must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) ### Bugs Fixed diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs index 89acc964bc4..81ff5a46850 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs @@ -60,10 +60,14 @@ pub enum ExecutionContext { Initial, /// Retry due to transient error (e.g., 429, 503). /// - /// Renamed to [`ExecutionContext::OperationRetry`] to align with the - /// cross-SDK reason taxonomy. Retained for one release for source - /// compatibility; the serialized form also changes from `"retry"` to - /// `"operation_retry"`. + /// **Deprecated:** superseded by [`ExecutionContext::OperationRetry`], which + /// aligns with the cross-SDK reason taxonomy and is distinct from the + /// transport-level [`ExecutionContext::TransportRetry`]. This is a distinct, + /// still-constructible variant — **not** a serde alias: a `Retry` value + /// continues to serialize as `"retry"`. The wire-format change to + /// `"operation_retry"` comes from the dispatch sites now emitting + /// `OperationRetry` instead of `Retry`, not from any change to this variant's + /// own serialization. Retained for one release for source compatibility. #[deprecated(since = "0.7.0", note = "use `ExecutionContext::OperationRetry`")] Retry, /// An operation-level retry decided by the SDK's client-retry policy. @@ -2208,7 +2212,30 @@ impl DiagnosticsContext { machine_id: last.machine_id.clone(), operation_name: last.operation_name.clone(), fault_injection_enabled: sources.iter().any(|c| c.fault_injection_enabled), - hedge_diagnostics: None, + // Propagate a representative hedge diagnostics so an aggregated + // operation (e.g. PATCH, whose internal Read sub-op can itself + // hedge) still reports hedging consistently: `hedging_started()`, + // the hedged metric's `hedge_terminal_state` dimension, and the + // tracing/log hedge fields must not silently drop just because the + // operation was aggregated. Prefer a sub-op that actually fanned + // out (an alternate region is present); otherwise fall back to any + // attached hedge diagnostics. Taking the last match mirrors how the + // aggregate inherits its operation-level fields from the last + // sub-op. + hedge_diagnostics: sources + .iter() + .rev() + .find_map(|c| { + c.hedge_diagnostics + .clone() + .filter(|hd| hd.alternate_region().is_some()) + }) + .or_else(|| { + sources + .iter() + .rev() + .find_map(|c| c.hedge_diagnostics.clone()) + }), compaction, #[cfg(test)] test_system_usage: last.test_system_usage.clone(), @@ -2306,24 +2333,44 @@ impl DiagnosticsContext { self.regions_contacted.clone() } - /// Returns the regions to which this operation dispatched a request, in - /// dispatch order, each tagged with the reason the SDK chose it. + /// Returns the regions to which this operation dispatched a request, each + /// tagged with the reason the SDK chose it. /// - /// Each dispatched attempt with a resolved region contributes one entry. - /// Duplicates are allowed: the same region may appear more than once if it - /// was dispatched multiple times (e.g., a retry to the same region, or a - /// hedge request to a region that was also the primary). The initial + /// Each retained dispatched attempt with a resolved region contributes one + /// entry. Duplicates are allowed: the same region may appear more than once + /// if it was dispatched multiple times (e.g., a retry to the same region, + /// or a hedge request to a region that was also the primary). The initial /// attempt is included and tagged [`RequestedRegionReason::Initial`]. /// /// Entries with no resolved region (pre-region-selection failures) are /// skipped, so this returns an empty `Vec` when an operation failed before /// any region was selected. /// - /// Order matches [`RequestDiagnostics`] insertion order, which is dispatch - /// order. This is distinct from [`regions_contacted`](Self::regions_contacted), - /// which is sorted and deduplicated. + /// **Hedge fan-out recovery.** When a hedge race resolves as a clean win + /// (the primary wins after the threshold, or the alternate wins outright), + /// the losing leg's future — and its per-request [`RequestDiagnostics`] — is + /// structurally dropped before it can be merged, so [`requests`](Self::requests) + /// holds only the winning leg. The fan-out regions are still recorded + /// authoritatively on [`hedge_diagnostics`](Self::hedge_diagnostics), so + /// when a fan-out occurred this accessor guarantees both fan-out legs are + /// represented: the primary leg tagged [`RequestedRegionReason::Initial`] + /// and the alternate leg tagged [`RequestedRegionReason::Hedging`]. A + /// recovered leg has no corresponding [`responded_regions`](Self::responded_regions) + /// entry (a structurally-dropped leg never produced a service reply). This + /// keeps the accessor consistent with the `hedge_region` observability + /// attribute, which is also sourced from `hedge_diagnostics`. + /// + /// Order is the retained attempts' dispatch (insertion) order, with the two + /// fan-out legs placed in dispatch order (the `Initial` primary before the + /// `Hedging` alternate). Under a `429`/`410` retry storm the retained list + /// may be bounded by [`DiagnosticsOptions::max_request_diagnostics`], so a + /// dropped attempt can be absent here even though it is still counted by + /// [`regions_contacted`](Self::regions_contacted) (which is captured from + /// the full attempt list before compaction and is deduplicated in + /// first-contact order). pub fn requested_regions(&self) -> Vec { - self.requests + let mut regions: Vec = self + .requests .iter() .filter_map(|r| { r.region().map(|region| RequestedRegion { @@ -2331,7 +2378,50 @@ impl DiagnosticsContext { reason: RequestedRegionReason::from(r.execution_context()), }) }) - .collect() + .collect(); + + // Recover a structurally-dropped hedge fan-out leg (see the doc comment). + // `hedge_diagnostics.alternate_region()` is `Some` exactly when a hedge + // arm fanned out; the primary leg is always dispatched as `Initial` and + // the alternate as `Hedging`. + if let Some(hedge) = self.hedge_diagnostics.as_ref() { + if let Some(alternate) = hedge.alternate_region() { + let is_sentinel = + |region: &Region| region.as_str() == HedgeDiagnostics::UNKNOWN_REGION_SENTINEL; + + let has_hedge_leg = regions + .iter() + .any(|r| r.reason == RequestedRegionReason::Hedging && &r.region == alternate); + if !has_hedge_leg && !is_sentinel(alternate) { + regions.push(RequestedRegion { + region: alternate.clone(), + reason: RequestedRegionReason::Hedging, + }); + } + + let primary = hedge.primary_region(); + let has_initial_leg = regions + .iter() + .any(|r| r.reason == RequestedRegionReason::Initial && &r.region == primary); + if !has_initial_leg && !is_sentinel(primary) { + // The primary leg is always launched first; place it before + // the first hedge leg so the fan-out reads in dispatch order. + let insert_at = regions + .iter() + .position(|r| r.reason == RequestedRegionReason::Hedging) + .unwrap_or(regions.len()); + regions.insert( + insert_at, + RequestedRegion { + region: primary.clone(), + reason: RequestedRegionReason::Initial, + }, + ); + } + } + } + + regions } /// Returns the regions from which this operation received a response, in @@ -2349,6 +2439,12 @@ impl DiagnosticsContext { /// A non-2xx HTTP status (e.g., 404/429) still counts — it is a response /// from the region. /// + /// Unlike [`requested_regions`](Self::requested_regions), this accessor does + /// **not** recover a structurally-dropped hedge loser leg: on a clean hedge + /// win the losing leg is cancelled before it produces a service reply, so it + /// correctly does not appear here. A clean hedge win therefore lists only the + /// winning region even though `requested_regions()` lists both fan-out legs. + /// /// To deduplicate, callers can collect into a set, for example: /// `ctx.responded_regions().into_iter().collect::>()`. pub fn responded_regions(&self) -> Vec<&Region> { @@ -4167,6 +4263,158 @@ mod tests { assert!(ctx.hedging_started()); } + fn hedge_config() -> crate::driver::pipeline::hedging_diagnostics::HedgingStrategyConfig { + crate::driver::pipeline::hedging_diagnostics::HedgingStrategyConfig::new( + crate::options::HedgeThreshold::new(std::time::Duration::from_millis(500)) + .expect("500ms is a valid hedge threshold"), + ) + } + + #[test] + fn requested_regions_recovers_dropped_hedge_leg_on_primary_win() { + // PrimaryWonAfterHedge: the alternate leg is structurally dropped, so + // `requests` holds only the primary (Initial) record. The alternate + // region must still be recovered from `hedge_diagnostics` and appended + // in dispatch order (after the Initial primary). + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let h = builder.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Ok, None); + builder.set_hedge_diagnostics(HedgeDiagnostics::primary_won_after_hedge( + hedge_config(), + Region::EAST_US_2, + Region::WEST_US_2, + )); + }); + + assert!(ctx.hedging_started()); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: RequestedRegionReason::Initial, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: RequestedRegionReason::Hedging, + }, + ] + ); + // The dropped alternate leg never produced a service reply, so only the + // winning primary appears in responded_regions. + assert_eq!(ctx.responded_regions(), vec![&Region::EAST_US_2]); + } + + #[test] + fn requested_regions_recovers_dropped_primary_leg_on_alternate_win() { + // AlternateWon: the primary leg is structurally dropped, so `requests` + // holds only the alternate (Hedging) record. The Initial primary region + // must be recovered and placed first (dispatch order). + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let h = builder.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Ok, None); + builder.set_hedge_diagnostics(HedgeDiagnostics::hedge_won( + hedge_config(), + Region::EAST_US_2, + Region::WEST_US_2, + )); + }); + + assert!(ctx.hedging_started()); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: RequestedRegionReason::Initial, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: RequestedRegionReason::Hedging, + }, + ] + ); + // Only the winning alternate produced a service reply. + assert_eq!(ctx.responded_regions(), vec![&Region::WEST_US_2]); + } + + #[test] + fn requested_regions_no_recovery_when_no_fanout() { + // PrimaryWonPreThreshold: a strategy was active but no alternate fanned + // out (alternate_region = None), so nothing is recovered and + // hedging_started() is false. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let h = builder.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Ok, None); + builder.set_hedge_diagnostics(HedgeDiagnostics::primary_only( + hedge_config(), + Region::EAST_US_2, + )); + }); + + assert!(!ctx.hedging_started()); + assert_eq!( + ctx.requested_regions(), + vec![RequestedRegion { + region: Region::EAST_US_2, + reason: RequestedRegionReason::Initial, + }] + ); + } + + #[test] + fn aggregate_sub_operations_propagates_hedge_diagnostics() { + // An aggregated operation (e.g. PATCH) whose sub-op hedged must still + // report hedging so the metric/log/span surfaces stay consistent, even + // though the aggregate is stitched from multiple sub-op contexts. + let read = make_context_with(ActivityId::new_uuid(), |builder| { + let h = builder.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Ok, None); + }); + let patch = make_context_with(ActivityId::new_uuid(), |builder| { + let h = builder.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Ok, None); + builder.set_hedge_diagnostics(HedgeDiagnostics::hedge_won( + hedge_config(), + Region::EAST_US_2, + Region::WEST_US_2, + )); + }); + + let aggregate = + DiagnosticsContext::aggregate_sub_operations(&[Arc::new(read), Arc::new(patch)]) + .expect("non-empty sources"); + + assert!(aggregate.hedging_started()); + let hedge = aggregate + .hedge_diagnostics() + .expect("aggregate must inherit a representative hedge diagnostics"); + assert_eq!( + hedge.terminal_state(), + crate::driver::pipeline::hedging_diagnostics::HedgeTerminalState::AlternateWon, + ); + } + #[test] fn requested_region_reason_mapping_is_total() { // A wildcard-free match forces this to stay total as variants are added. diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/hedging_diagnostics.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/hedging_diagnostics.rs index d09b348091c..f0f49bb72c2 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/hedging_diagnostics.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/hedging_diagnostics.rs @@ -538,6 +538,50 @@ mod tests { )); } + #[test] + fn terminal_state_as_str_is_stable_for_every_variant() { + // These strings are an observability contract (metric/log/span values), + // so pin all six. `BothTransient` collapses to a single value regardless + // of its `deadline_elapsed` payload to keep the emitted value set bounded. + assert_eq!( + HedgeTerminalState::PrimaryWonPreThreshold.as_str(), + "primary_won_pre_threshold" + ); + assert_eq!( + HedgeTerminalState::DeadlineExceededPreThreshold.as_str(), + "deadline_exceeded_pre_threshold" + ); + assert_eq!( + HedgeTerminalState::PrimaryWonAfterHedge.as_str(), + "primary_won_after_hedge" + ); + assert_eq!(HedgeTerminalState::AlternateWon.as_str(), "alternate_won"); + assert_eq!( + HedgeTerminalState::CancelledAwaitingPartner.as_str(), + "cancelled_awaiting_partner" + ); + assert_eq!( + HedgeTerminalState::BothTransient { + deadline_elapsed: true + } + .as_str(), + "both_transient" + ); + assert_eq!( + HedgeTerminalState::BothTransient { + deadline_elapsed: false + } + .as_str(), + "both_transient", + "both_transient must collapse regardless of deadline_elapsed" + ); + // Display delegates to as_str(). + assert_eq!( + HedgeTerminalState::AlternateWon.to_string(), + "alternate_won" + ); + } + #[test] fn debug_clone_round_trip() { let diag = HedgeDiagnostics::hedge_won(config(), Region::EAST_US, Region::WEST_US_2); From a47bc8641e1e19e9daeb9e9c376a0ae1da3d5395 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Mon, 27 Jul 2026 12:50:37 -0700 Subject: [PATCH 04/21] =?UTF-8?q?Make=20hedge=20observability=20surfaces?= =?UTF-8?q?=20consistent=20for=20both-transient=E2=86=92failover?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-2 review follow-up. The both-transient→failover-resolved path leaves a retained Hedging request (so hedging_started() is true) but no recorded hedge terminal outcome (finalize_both_transient deliberately does not stamp hedge_diagnostics on the non-terminal path). The metrics guard already skipped this case, but the logging handler still emitted empty-string hedge_region / hedge_terminal_state fields via unwrap_or_default(). - logging handler: gate the dedicated hedge fields on hedge_diagnostics (with a fanned-out alternate) instead of hedging_started(), so no misleading empty strings are emitted; consistent with the metrics counter. Add a regression test for the both-transient→failover shape. - metrics: correct the guard test's comment — the None case is production- reachable (both-transient then failover success), not an aggregate-only belt-and-suspenders case; document that the counter measures hedges with a resolved terminal outcome. - docs: note that requested_regions() on an aggregated operation recovers only the single representative fan-out's dropped leg (multi-hedge PATCH caveat). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../src/diagnostics/logging/handler.rs | 29 +++++++---- .../src/diagnostics/logging/mod.rs | 51 +++++++++++++++++++ .../src/diagnostics/metrics/handler.rs | 16 +++--- .../src/diagnostics/diagnostics_context.rs | 9 +++- 4 files changed, 89 insertions(+), 16 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs index cf3aa5ad76a..da6e6929fc6 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs @@ -78,18 +78,29 @@ impl DiagnosticsHandler for TracingLogHandler { // (the `tracing` macros run an "is enabled" check before evaluating // field expressions). // - // When a cross-region hedge fanned out, surface the hedging signal as - // dedicated fields (in addition to the JSON blob) — it is high-signal for - // exactly the failed / threshold-breaching operations this handler emits. - if diagnostics.hedging_started() { - let hedge = diagnostics.hedge_diagnostics(); + // When a cross-region hedge fanned out AND produced a recorded terminal + // outcome, surface the hedging signal as dedicated fields (in addition to + // the JSON blob) — high-signal for exactly the failed / threshold-breaching + // operations this handler emits. + // + // Gate on `hedge_diagnostics` (with a fanned-out alternate) rather than + // `hedging_started()`: a both-transient hedge that is subsequently resolved + // by a failover attempt leaves a retained `Hedging` request (so + // `hedging_started()` is true) but no recorded terminal outcome + // (`hedge_diagnostics()` is `None` — `finalize_both_transient` deliberately + // does not stamp one on the non-terminal path). Emitting empty + // `hedge_region` / `hedge_terminal_state` strings there would be misleading; + // this gate keeps the log line consistent with the metrics counter, which is + // likewise only recorded when a terminal hedge outcome exists. + if let Some(hedge) = diagnostics + .hedge_diagnostics() + .filter(|hedge| hedge.alternate_region().is_some()) + { let hedge_region = hedge - .and_then(|h| h.alternate_region()) + .alternate_region() .map(|region| region.as_str()) .unwrap_or_default(); - let hedge_terminal_state = hedge - .map(|h| h.terminal_state().as_str()) - .unwrap_or_default(); + let hedge_terminal_state = hedge.terminal_state().as_str(); if diagnostics.is_failure() { tracing::warn!(target: SAMPLED_TARGET, reason, hedging_started = true, hedge_region, hedge_terminal_state, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); } else { diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs index 8e311a25ba7..e83ae22cd8d 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs @@ -382,4 +382,55 @@ mod tests { assert!(!map.contains_key("hedge_region")); assert!(!map.contains_key("hedge_terminal_state")); } + + #[test] + fn sampled_line_omits_hedging_fields_when_no_terminal_outcome() { + // A hedge fanned out both-transient and was then resolved by a later + // failover attempt: the retained `Hedging` request makes + // `hedging_started()` true, but the pipeline deliberately leaves + // `hedge_diagnostics()` None (no terminal outcome). The log line must + // still be emitted, but must NOT carry empty-string hedge_region / + // hedge_terminal_state fields — consistent with the metrics counter. + use azure_data_cosmos_driver::diagnostics::ExecutionContext; + + let captured = Arc::new(std::sync::Mutex::new(HashMap::new())); + let layer = FieldCapture(Arc::clone(&captured)); + let subscriber = tracing_subscriber::registry().with(layer); + let handler = SamplingLogHandler::new(); + + let now = Instant::now(); + let hedge_leg = RequestDiagnostics::for_testing( + "https://acct-westus2.documents.azure.com:443/", + Some(Region::WEST_US_2), + CosmosStatus::new(StatusCode::TooManyRequests), + RequestCharge::new(2.0), + now - Duration::from_millis(20), + now, + ) + .with_execution_context_for_testing(ExecutionContext::Hedging); + let ctx = DiagnosticsContext::for_testing_with_hedge( + ActivityId::new_uuid(), + Duration::from_millis(20), + Some(CosmosStatus::new(StatusCode::TooManyRequests)), + Some("read_item"), + vec![hedge_leg], + None, + ); + assert!( + ctx.hedging_started(), + "a retained Hedging request makes hedging_started() true" + ); + + tracing::subscriber::with_default(subscriber, || { + handler.handle(&ctx, &Context::new()); + }); + + let map = captured.lock().unwrap(); + // The line is emitted (it is a failure) but carries no hedge fields — + // crucially, no empty-string hedge_region / hedge_terminal_state. + assert!(map.contains_key("reason")); + assert!(!map.contains_key("hedging_started")); + assert!(!map.contains_key("hedge_region")); + assert!(!map.contains_key("hedge_terminal_state")); + } } diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs index 93a22bbb0c0..792f941a765 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs @@ -714,12 +714,16 @@ mod tests { #[test] fn hedged_metric_skips_when_terminal_state_unavailable() { - // Defensive: if an operation reads as hedged (a `Hedging`-tagged request) - // but carries no `hedge_diagnostics`, the counter is skipped rather than - // emitted without its `hedge_terminal_state` dimension — a missing - // dimension would fragment the counter's time series. Real aggregated - // operations (e.g. PATCH) propagate a representative `hedge_diagnostics`, - // so this is a belt-and-suspenders guard. + // A hedge that fanned out both-transient and was then resolved by a later + // failover attempt leaves a retained `Hedging` request (so + // `hedging_started()` is true) but no recorded terminal outcome + // (`finalize_both_transient` deliberately does not stamp `hedge_diagnostics` + // on the non-terminal path — see operation_pipeline.rs). The counter is + // intentionally skipped there rather than emitted without its + // `hedge_terminal_state` dimension, which would fragment the counter's time + // series. The counter therefore measures hedges with a resolved terminal + // outcome; a both-transient hedge whose winning response ultimately came + // from a failover attempt is not counted here. use azure_data_cosmos_driver::diagnostics::{ExecutionContext, RequestDiagnostics}; use azure_data_cosmos_driver::models::RequestCharge; use std::time::Instant; diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs index 81ff5a46850..2570ecdf838 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs @@ -2221,7 +2221,9 @@ impl DiagnosticsContext { // out (an alternate region is present); otherwise fall back to any // attached hedge diagnostics. Taking the last match mirrors how the // aggregate inherits its operation-level fields from the last - // sub-op. + // sub-op. Limitation: only one representative survives, so if two + // sub-ops both fanned out to different alternates, `requested_regions` + // recovers only the representative one's dropped leg. hedge_diagnostics: sources .iter() .rev() @@ -2360,6 +2362,11 @@ impl DiagnosticsContext { /// keeps the accessor consistent with the `hedge_region` observability /// attribute, which is also sourced from `hedge_diagnostics`. /// + /// For an aggregated operation (e.g. PATCH) stitched from multiple + /// sub-operations, recovery uses the single representative `hedge_diagnostics` + /// retained by `aggregate_sub_operations`, so if more than one sub-operation + /// fanned out, only the representative fan-out's dropped leg is recovered. + /// /// Order is the retained attempts' dispatch (insertion) order, with the two /// fan-out legs placed in dispatch order (the `Initial` primary before the /// `Hedging` alternate). Under a `429`/`410` retry storm the retained list From d9f18811e154834797e547d1d2378733d475e1f6 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Mon, 27 Jul 2026 15:05:23 -0700 Subject: [PATCH 05/21] Fix broken links in hedging detection API spec The Verify Links / Analyze CI checks failed on two links in HEDGING_DETECTION_API_SPEC.md: - The docs.rs deep link to struct.DiagnosticsContext.html 404s because that page is not yet on the published 'latest' crate docs; point it at the crate root instead, matching the README convention. - The relative link to HEDGING_SPEC.md violated the link guidance (relative links disallowed); switch to the absolute GitHub blob/main URL. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md index 8031725c29f..af36274b320 100644 --- a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md +++ b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md @@ -54,7 +54,7 @@ driver-owned and re-exported by `azure_data_cosmos`, exactly like The hedging orchestrator/dispatch is **landed** on `main` ([#4432](https://github.com/Azure/azure-sdk-for-rust/pull/4432)): it emits `ExecutionContext::Hedging` for alternate legs and populates `HedgeDiagnostics` -(design: [`HEDGING_SPEC.md`](../../azure_data_cosmos_driver/docs/HEDGING_SPEC.md), +(design: [`HEDGING_SPEC.md`](https://github.com/Azure/azure-sdk-for-rust/blob/main/sdk/cosmos/azure_data_cosmos_driver/docs/HEDGING_SPEC.md), [PR #4330](https://github.com/Azure/azure-sdk-for-rust/pull/4330)). --- @@ -203,4 +203,4 @@ collection. If `ExecutionContext` becomes a prominent part of the public detection surface it could be renamed to something friendlier (e.g., `RequestPurpose` / `RequestIntent`); that rename is out of scope here. -[`DiagnosticsContext`]: https://docs.rs/azure_data_cosmos/latest/azure_data_cosmos/struct.DiagnosticsContext.html +[`DiagnosticsContext`]: https://docs.rs/azure_data_cosmos/latest/azure_data_cosmos/ From eb79114769b7b6b02fb0528eeacd2d471cbf39ef Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Mon, 27 Jul 2026 16:13:43 -0700 Subject: [PATCH 06/21] Documentation cleanups from round-3 review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Non-blocking doc precision fixes surfaced by the round-3 reviewers: - operation_pipeline.rs: correct the compute_execution_context doc bullet ("session retry in progress -> OperationRetry", was the stale "Retry"). - diagnostics_context.rs: reframe the hedging_started() doc — hedge_diagnostics() is a resolved-terminal-outcome surface, not a "was hedging configured" probe; document that it is None (while hedging_started() stays true) on the both-transient -> failover path, and that the metric/log surfaces key off it. - span_builder.rs: document the intentional asymmetry — the tracing span gates hedge attributes on hedging_started() (surfacing region history for any fan-out, no empty/placeholder values) while the metric counter and log hedge fields key off hedge_diagnostics (a resolved terminal outcome). - CHANGELOGs: add the PR link (#4871) alongside the tracking issue (#4410) on the hedging feature / surfacing / breaking-change entries, per changelog convention. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- sdk/cosmos/azure_data_cosmos/CHANGELOG.md | 6 +++--- .../src/diagnostics/tracing/span_builder.rs | 13 +++++++++++++ .../azure_data_cosmos_driver/CHANGELOG.md | 6 +++--- .../src/diagnostics/diagnostics_context.rs | 17 +++++++++++++---- .../src/driver/pipeline/operation_pipeline.rs | 2 +- 5 files changed, 33 insertions(+), 11 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md index ad8e46641ba..86f0f619e61 100644 --- a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md @@ -4,15 +4,15 @@ ### Features Added -- Surfaced the Hedging Detection API through `DiagnosticsContext`: `hedging_started()`, `requested_regions()`, and `responded_regions()`, and re-exported the `RequestedRegion` / `RequestedRegionReason` types (mirroring the existing `DiagnosticsContext` re-export). ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) +- Surfaced the Hedging Detection API through `DiagnosticsContext`: `hedging_started()`, `requested_regions()`, and `responded_regions()`, and re-exported the `RequestedRegion` / `RequestedRegionReason` types (mirroring the existing `DiagnosticsContext` re-export). ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Added a pluggable client-side diagnostics emission layer — the `DiagnosticsHandler` trait and ordered `DiagnosticsHandlerChain` (registered via `CosmosClientBuilder::with_diagnostics_handler`) — invoked once per operation (singleton and paginated, on success and failure) with the completed `DiagnosticsContext` plus an SDK-supplied `CosmosOperationContext`; the empty default chain is a zero-overhead no-op. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added the `metrics`-gated `CosmosMetricsHandler` (with `MetricsOptions`), emitting the stable `db.client.operation.duration` histogram plus per-signal opt-in metrics (`with_request_charge_metric`, `with_returned_rows_metric`) and an opt-in extended attribute set (`with_extended_attributes`); a no-op when no meter provider is registered. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added composable tail-sampled emission handlers — a `TracingLogHandler` leaf that writes a compact `tracing` line and a `SamplingLogHandler` wrapper (holding an `Arc`) that applies the sampling gate plus a shared per-window rate limit, defaulting to wrap a `TracingLogHandler` — and the `distributed_tracing`-gated `CosmosTracingHandler` (backdated span tree), also rate-limited so an error storm can't overwhelm exporters. All emit only for operations which fail or breach a configurable `DiagnosticsThresholds`, and stamp *why* they were sampled (a failure, or which threshold) on the emitted line and span. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) -- Surfaced hedging through the observability handlers, reusing the Hedging Detection API. When a cross-region hedge fans out, `CosmosTracingHandler` adds `azure.cosmosdb.operation.{hedging_started,hedge_region,hedge_terminal_state}` plus `requested_regions`/`responded_regions` (`string[]`) to the sampled operation span and tags the hedge-leg child span (`azure.cosmosdb.request.hedge`); `SamplingLogHandler` adds `hedging_started` / `hedge_region` / `hedge_terminal_state` fields to the sampled log line; and `CosmosMetricsHandler` gains an opt-in `azure.cosmosdb.client.operation.hedged` counter (`MetricsOptions::with_hedged_metric`) carrying the low-cardinality `hedge_terminal_state`, with the higher-cardinality `hedge_region` dimension added only under `with_extended_attributes`. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) +- Surfaced hedging through the observability handlers, reusing the Hedging Detection API. When a cross-region hedge fans out, `CosmosTracingHandler` adds `azure.cosmosdb.operation.{hedging_started,hedge_region,hedge_terminal_state}` plus `requested_regions`/`responded_regions` (`string[]`) to the sampled operation span and tags the hedge-leg child span (`azure.cosmosdb.request.hedge`); `SamplingLogHandler` adds `hedging_started` / `hedge_region` / `hedge_terminal_state` fields to the sampled log line; and `CosmosMetricsHandler` gains an opt-in `azure.cosmosdb.client.operation.hedged` counter (`MetricsOptions::with_hedged_metric`) carrying the low-cardinality `hedge_terminal_state`, with the higher-cardinality `hedge_region` dimension added only under `with_extended_attributes`. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) ### Breaking Changes -- Serialized `DiagnosticsContext` output (the diagnostics JSON surfaced to consumers, and the sampled diagnostics log line) now serializes driver-generated operation retries with `execution_context` = `"operation_retry"` instead of `"retry"`, following the driver's `ExecutionContext::Retry` → `OperationRetry` rename. This is additive to the enum (the deprecated `Retry` variant still serializes as `"retry"`), but the wire value emitted for operation retries changes; telemetry/log parsers that match the literal `"retry"` execution context must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) +- Serialized `DiagnosticsContext` output (the diagnostics JSON surfaced to consumers, and the sampled diagnostics log line) now serializes driver-generated operation retries with `execution_context` = `"operation_retry"` instead of `"retry"`, following the driver's `ExecutionContext::Retry` → `OperationRetry` rename. This is additive to the enum (the deprecated `Retry` variant still serializes as `"retry"`), but the wire value emitted for operation retries changes; telemetry/log parsers that match the literal `"retry"` execution context must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) ### Bugs Fixed diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs index d94f34fb252..a89f3853c80 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs @@ -178,6 +178,19 @@ pub(crate) fn emit_backdated_span_tree( } // Hedging surfacing: only when a cross-region hedge actually fanned out. // These attributes stay off the common (non-hedged) sampled span entirely. + // + // The span intentionally gates on `hedging_started()` (any fan-out) rather + // than on `hedge_diagnostics()` (a resolved terminal outcome), because the + // span is a rich per-operation diagnostic: the region history + // (`requested_regions`/`responded_regions`) is valuable exactly for a + // both-transient hedge that was then resolved by failover, and + // `HEDGING_STARTED=true` there is factually correct. The per-outcome + // `HEDGE_REGION`/`HEDGE_TERMINAL_STATE` fields are still gated on + // `hedge_diagnostics()` below, so no empty/placeholder values are emitted. + // This is a deliberate, documented asymmetry vs. the hedged metric counter + // and the log hedge fields, which key off `hedge_diagnostics` (a resolved + // terminal outcome) so every counter data point carries the + // `hedge_terminal_state` dimension — see `DiagnosticsContext::hedging_started`. if diagnostics.hedging_started() { root_attrs.push(KeyValue::new(attributes::HEDGING_STARTED, true)); if let Some(hedge) = diagnostics.hedge_diagnostics() { diff --git a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md index e7607d8fa5f..eba2bf73ce8 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md @@ -4,12 +4,12 @@ ### Features Added -- Added the Hedging Detection API on `DiagnosticsContext`: `hedging_started()` (whether a hedge arm actually fanned out), `requested_regions()` (regions dispatched to, in dispatch order with duplicates, each tagged with a reason), and `responded_regions()` (regions that produced an actual service reply, in completion order). Adds the public `RequestedRegion` struct and `RequestedRegionReason` enum (both `#[non_exhaustive]`, with a total `From` mapping) and the new `ExecutionContext::OperationRetry` variant. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) -- Added `HedgeTerminalState::as_str()` (and a matching `Display`) returning a stable, low-cardinality `snake_case` identifier for the hedging race outcome, for use as an observability attribute / log-field value. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) +- Added the Hedging Detection API on `DiagnosticsContext`: `hedging_started()` (whether a hedge arm actually fanned out), `requested_regions()` (regions dispatched to, in dispatch order with duplicates, each tagged with a reason), and `responded_regions()` (regions that produced an actual service reply, in completion order). Adds the public `RequestedRegion` struct and `RequestedRegionReason` enum (both `#[non_exhaustive]`, with a total `From` mapping) and the new `ExecutionContext::OperationRetry` variant. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Added `HedgeTerminalState::as_str()` (and a matching `Display`) returning a stable, low-cardinality `snake_case` identifier for the hedging race outcome, for use as an observability attribute / log-field value. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) ### Breaking Changes -- Renamed `ExecutionContext::Retry` to `ExecutionContext::OperationRetry` to distinguish operation-level retries from transport-level `TransportRetry`. The old `Retry` remains for one release as a distinct `#[deprecated]` variant (**not** a serde alias); a `Retry` value still serializes as `"retry"`. The customer-visible wire-format change is that driver-generated operation retries now serialize as `"operation_retry"` instead of `"retry"`, because the dispatch sites emit `OperationRetry`; telemetry parsers that match the literal `"retry"` execution context must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410)) +- Renamed `ExecutionContext::Retry` to `ExecutionContext::OperationRetry` to distinguish operation-level retries from transport-level `TransportRetry`. The old `Retry` remains for one release as a distinct `#[deprecated]` variant (**not** a serde alias); a `Retry` value still serializes as `"retry"`. The customer-visible wire-format change is that driver-generated operation retries now serialize as `"operation_retry"` instead of `"retry"`, because the dispatch sites emit `OperationRetry`; telemetry parsers that match the literal `"retry"` execution context must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) ### Bugs Fixed diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs index 2570ecdf838..67ac0fbb1a1 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs @@ -2474,10 +2474,19 @@ impl DiagnosticsContext { /// hedging threshold elapses, this returns `false` even though a hedging /// strategy was active. /// - /// To check whether a hedging strategy was *configured*, inspect - /// [`hedge_diagnostics`](Self::hedge_diagnostics) instead — it is `Some` - /// whenever hedging was active for the operation, including the - /// primary-wins-under-threshold case where no fan-out happened. + /// [`hedge_diagnostics`](Self::hedge_diagnostics) is a related but distinct + /// surface: it is `Some` when the hedge race recorded a terminal outcome — + /// including the primary-wins-under-threshold case where no fan-out happened + /// — but it is `None` when hedging was configured yet ineligible, and also + /// for a both-transient hedge that was subsequently resolved by a failover + /// attempt (the non-terminal both-transient path deliberately records no + /// terminal outcome). So `hedge_diagnostics().is_some()` is not a reliable + /// "was hedging configured" probe, and it can disagree with + /// `hedging_started()` on that both-transient→failover path (where a + /// retained `Hedging` request keeps this accessor `true`). The SDK's hedged + /// metric counter and log hedge fields key off `hedge_diagnostics` (a + /// resolved terminal outcome), so they intentionally do not surface that + /// path even though `hedging_started()` is `true`. /// /// The result is a disjunction of two independent fan-out signals: /// [`HedgeDiagnostics::alternate_region`] being `Some` (the orchestrator diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs index 94f4abaafde..2936ec93700 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs @@ -1880,7 +1880,7 @@ fn should_capture_session_token_from_status( /// transport pipeline expects for diagnostics annotation. /// /// - First attempt (no failover, no session retry) → `Initial` -/// - Any session retry in progress → `Retry` +/// - Any session retry in progress → `OperationRetry` /// - Otherwise (a failover retry) → `RegionFailover` /// /// Session-retry takes precedence over failover-retry because in the rare From d9ab7b8d742d1bc7d0ed680d4a76f96487ca883d Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Mon, 27 Jul 2026 16:29:28 -0700 Subject: [PATCH 07/21] Align hedge_diagnostics() docstring with hedging_started() The hedge_diagnostics() method doc still claimed Some iff execute_hedged() was entered, which contradicts the updated hedging_started() doc: on the both-transient->failover path execute_hedged() runs but hedge_diagnostics is deliberately left None. Reword to describe it as a resolved-terminal-outcome surface and enumerate the both-transient->failover None case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../src/diagnostics/diagnostics_context.rs | 20 ++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs index 67ac0fbb1a1..e52b0209b7a 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs @@ -2523,14 +2523,20 @@ impl DiagnosticsContext { Arc::clone(&self.requests) } - /// Returns the hedging diagnostics for this operation, if hedging was - /// selected. + /// Returns the hedging diagnostics for this operation, if the hedge race + /// recorded a terminal outcome. /// - /// This is `Some(_)` if and only - /// if `should_hedge()` returned `true` and `execute_hedged()` was - /// entered — even when the primary won before the threshold elapsed. - /// `None` means hedging was not selected for this operation (no - /// strategy resolved, strategy `Disabled`, or eligibility check failed). + /// This is `Some(_)` when the hedge race recorded a terminal outcome — + /// including the primary-wins-under-threshold case where `execute_hedged()` + /// ran but no alternate leg fanned out. It is `None` when hedging was not + /// selected for this operation (no strategy resolved, strategy `Disabled`, + /// or eligibility check failed), **and also** on the both-transient→failover + /// path: when both hedge legs return transient failures but the deadline has + /// not elapsed and failover budget remains, `finalize_both_transient` + /// deliberately does not stamp a terminal outcome (a later successful retry + /// would otherwise carry a misleading `BothTransient` state), so a retained + /// `ExecutionContext::Hedging` request can leave `hedging_started()` `true` + /// while this returns `None`. pub fn hedge_diagnostics(&self) -> Option<&HedgeDiagnostics> { self.hedge_diagnostics.as_ref() } From b81bf39e4aae6ddb7097e9cc5d3d027ad997a616 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Tue, 28 Jul 2026 12:03:28 -0700 Subject: [PATCH 08/21] Report hedge fan-out consistently across observability MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The hedged counter, the sampled log line and the root span each answered "did this operation hedge?" differently. The counter and the log gated on `hedge_diagnostics()` — a *retained terminal outcome* — while the span gated on `hedging_started()`. That difference was documented as a deliberate asymmetry, but it is really an undercount: a race that ends both-transient and is then resolved by a failover attempt deliberately retains no terminal outcome, so an operation that demonstrably fanned out was dropped from the counter entirely and lost its `hedging_started` field on the log line. Now that `hedging_started()` is materialized from the dispatch-time fan-out log it is the authoritative signal, so all three surfaces gate on it: - `metrics/handler.rs`: `record_hedged` gates on `hedging_started()` and keeps the counter's attribute schema uniform by falling back to an `unresolved` sentinel for `hedge_terminal_state`, so `group by hedge_terminal_state` still never fragments. - `logging/handler.rs`: adds an arm for the fanned-out-without-outcome case. `tracing` field sets are fixed per call site, so this reports `hedging_started` while omitting the per-outcome fields rather than emitting empty strings for them. - `tracing/span_builder.rs`: comment only — the asymmetry it described no longer exists. Both tests that asserted the old drop-it behavior now assert the signal is reported, and the spec gains §5.1 covering the shared rule. Also corrects §5's comparison table, which still described `hedge_diagnostics()` as answering "was a strategy active?". It does not: it is an optional retained race outcome, absent both for a configured-but-ineligible operation and on the both-transient path. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 37e127ae-4bbe-4eb4-a533-c6ddd54e884e --- sdk/cosmos/azure_data_cosmos/CHANGELOG.md | 2 +- .../docs/HEDGING_DETECTION_API_SPEC.md | 47 ++++++++++--- .../src/diagnostics/logging/handler.rs | 37 ++++++----- .../src/diagnostics/logging/mod.rs | 23 ++++--- .../src/diagnostics/metrics/handler.rs | 66 ++++++++++++------- .../src/diagnostics/tracing/span_builder.rs | 21 +++--- 6 files changed, 127 insertions(+), 69 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md index e5b1e3f092c..605032c6d54 100644 --- a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md @@ -9,7 +9,7 @@ - Added a pluggable client-side diagnostics emission layer — the `DiagnosticsHandler` trait and ordered `DiagnosticsHandlerChain` (registered via `CosmosClientBuilder::with_diagnostics_handler`) — invoked once per operation (singleton and paginated, on success and failure) with the completed `DiagnosticsContext` plus an SDK-supplied `CosmosOperationContext`; the empty default chain is a zero-overhead no-op. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added the `metrics`-gated `CosmosMetricsHandler` (with `MetricsOptions`), emitting the stable `db.client.operation.duration` histogram plus per-signal opt-in metrics (`with_request_charge_metric`, `with_returned_rows_metric`) and an opt-in extended attribute set (`with_extended_attributes`); a no-op when no meter provider is registered. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added composable tail-sampled emission handlers — a `TracingLogHandler` leaf that writes a compact `tracing` line and a `SamplingLogHandler` wrapper (holding an `Arc`) that applies the sampling gate plus a shared per-window rate limit, defaulting to wrap a `TracingLogHandler` — and the `distributed_tracing`-gated `CosmosTracingHandler` (backdated span tree), also rate-limited so an error storm can't overwhelm exporters. All emit only for operations which fail or breach a configurable `DiagnosticsThresholds`, and stamp *why* they were sampled (a failure, or which threshold) on the emitted line and span. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) -- Surfaced hedging through the observability handlers, reusing the Hedging Detection API. When a cross-region hedge fans out, `CosmosTracingHandler` adds `azure.cosmosdb.operation.{hedging_started,hedge_region,hedge_terminal_state}` plus `requested_regions`/`responded_regions` (`string[]`) to the sampled operation span and tags the hedge-leg child span (`azure.cosmosdb.request.hedge`); `SamplingLogHandler` adds `hedging_started` / `hedge_region` / `hedge_terminal_state` fields to the sampled log line; and `CosmosMetricsHandler` gains an opt-in `azure.cosmosdb.client.operation.hedged` counter (`MetricsOptions::with_hedged_metric`) carrying the low-cardinality `hedge_terminal_state`, with the higher-cardinality `hedge_region` dimension added only under `with_extended_attributes`. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Surfaced hedging through the observability handlers, reusing the Hedging Detection API. When a cross-region hedge fans out, `CosmosTracingHandler` adds `azure.cosmosdb.operation.{hedging_started,hedge_region,hedge_terminal_state}` plus `requested_regions`/`responded_regions` (`string[]`) to the sampled operation span and tags the hedge-leg child span (`azure.cosmosdb.request.hedge`); `SamplingLogHandler` adds `hedging_started` / `hedge_region` / `hedge_terminal_state` fields to the sampled log line; and `CosmosMetricsHandler` gains an opt-in `azure.cosmosdb.client.operation.hedged` counter (`MetricsOptions::with_hedged_metric`) carrying the low-cardinality `hedge_terminal_state`, with the higher-cardinality `hedge_region` dimension added only under `with_extended_attributes`. All three surfaces decide fan-out from `hedging_started()`, so a hedge whose race ended both-transient and was then resolved by a failover attempt is still reported; the per-outcome `hedge_region` / `hedge_terminal_state` fields are omitted there (the counter's dimension carries the `unresolved` sentinel so its attribute schema stays uniform). ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) ### Breaking Changes diff --git a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md index cf83da49862..a0d99647dd1 100644 --- a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md +++ b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md @@ -34,7 +34,7 @@ Rust-native `HedgeDiagnostics` surface (see §5); the two are complementary. | --- | --- | --- | | `DiagnosticsContext` | re-exported as `azure_data_cosmos::DiagnosticsContext` | The per-operation diagnostics handle. | | `DiagnosticsContext::requests` | `-> Arc>` | Retained per-attempt records in dispatch order — **not** a guaranteed-complete append-only history: under a `429`/`410` retry storm the list is bounded/compacted (see `max_request_diagnostics`, which can drop or reorder entries), and a structurally-dropped hedge loser leg is absent. Cloning the `Arc` is a cheap atomic increment. | -| `DiagnosticsContext::hedge_diagnostics` | `-> Option<&HedgeDiagnostics>` | `Some` whenever a hedging strategy was active for the operation (including primary-wins-under-threshold). | +| `DiagnosticsContext::hedge_diagnostics` | `-> Option<&HedgeDiagnostics>` | An **optional retained race outcome**, not a configuration probe. `Some` when a hedge race recorded a terminal outcome (including primary-wins-under-threshold). `None` when hedging was not selected for the operation, when a configured strategy found the operation ineligible, **and** on the both-transient→failover path, where a terminal outcome is deliberately left unset so a later successful retry does not carry a misleading `BothTransient` state. | | `DiagnosticsContext::regions_contacted` | `-> Vec` | Distinct regions **deduplicated in first-contact (failover) order — not sorted**, captured from the full attempt list before compaction. | | `RequestDiagnostics::region` | `-> Option<&Region>` | `None` for pre-region-selection failures. | | `RequestDiagnostics::execution_context` | `-> ExecutionContext` | Why this attempt was dispatched (see §3). | @@ -120,9 +120,13 @@ The mapping is total (no wildcard arm) so it fails to compile if a new `true` iff at least one hedge arm was actually dispatched. This is `false` — not an error — when the primary returns before the hedging threshold elapses, even -though a hedging strategy was active. To check whether a strategy was merely -*configured*, use `ctx.hedge_diagnostics().is_some()` (a superset that includes -primary-wins-under-threshold). +though a hedging strategy was active. + +There is **no** accessor for "was a strategy configured?", and +`hedge_diagnostics().is_some()` is not one: it is `None` for a configured but +ineligible operation, and on the both-transient→failover path (§5). Reading it +as a configuration probe would make consumers conclude hedging was disabled when +it was not. Like the two region accessors, this is **materialized at finalization** from the dispatch-time fan-out log (§4.3), so a race whose attempts were later compacted @@ -200,17 +204,26 @@ coexist on the same `DiagnosticsContext` and serve different audiences. | Question | Hedging Detection API | Rust-native `HedgeDiagnostics` | | --- | --- | --- | -| Did fan-out happen? | `hedging_started()` — from the fan-out log | `alternate_region().is_some()` — equivalent for a single operation | -| Was a strategy active? | *(not derived)* | `hedge_diagnostics().is_some()` — superset of fan-out | +| Did fan-out happen? | `hedging_started()` — from the fan-out log | `alternate_region().is_some()` — equivalent only when a terminal outcome was retained | +| Was a strategy configured? | *(not derived)* | *(not derived — see below)* | | Regions tried | `requested_regions()` (every dispatch, with reason) | `primary_region()` + `alternate_region()` (one race's legs only) | | Regions that responded | `responded_regions()` (full list, completion order) | `response_region()` (single winner) | -| Race outcome | *(not derived)* | `terminal_state()` (authoritative) | +| Race outcome | *(not derived)* | `terminal_state()` (authoritative, when retained) | + +Neither surface answers "was hedging configured?". `hedge_diagnostics()` is an +optional *retained race outcome*: it is `None` for a configured-but-ineligible +operation, and it is deliberately left unset when a both-transient race +continues to a successful failover, so that a later successful retry does not +carry a misleading `BothTransient` state. On that path a dispatched hedge leg +still makes `hedging_started()` `true` while `hedge_diagnostics()` is `None` — +the two are not interchangeable. The Detection API deliberately does **not** read `hedge_diagnostics` to answer its three questions. Aggregation keeps only one representative `HedgeDiagnostics` for a multi-round-trip operation, so deriving from it would silently under-report -every other sub-operation's fan-out. The fan-out log is per-builder and survives -aggregation intact. +every other sub-operation's fan-out — and the both-transient path would +under-report fan-out entirely. The fan-out log is per-builder, is written at +dispatch time, and survives aggregation intact. `main`'s `HedgeDiagnostics` classifies the race via `terminal_state` / `alternate_region` (there is no `total_requests_launched` counter), so "fan-out @@ -220,6 +233,22 @@ happened" is `alternate_region().is_some()` and "the alternate won" is presence of `alternate_region()` alone (several terminal states still record an alternate region). +### 5.1 Consequence for the observability surfaces + +All three SDK emission surfaces — the sampled root span, the sampled log line, +and the opt-in `azure.cosmosdb.client.operation.hedged` counter — decide +"did this operation hedge?" from `hedging_started()`, never from +`hedge_diagnostics().is_some()`. Gating on the latter would silently undercount +every both-transient race that was subsequently resolved by a failover attempt, +which is exactly the population an operator most wants to see. + +The per-outcome fields (`hedge_region`, `hedge_terminal_state`) still require +`hedge_diagnostics`, and are simply omitted when it is absent rather than +emitted as empty strings. The counter is the one exception: its +`hedge_terminal_state` dimension carries the `unresolved` sentinel instead of +being dropped, so every data point on the counter has the same attribute set +and `group by hedge_terminal_state` never fragments the time series. + --- ## 6. Future work diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs index da6e6929fc6..32b8970346f 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs @@ -78,24 +78,23 @@ impl DiagnosticsHandler for TracingLogHandler { // (the `tracing` macros run an "is enabled" check before evaluating // field expressions). // - // When a cross-region hedge fanned out AND produced a recorded terminal - // outcome, surface the hedging signal as dedicated fields (in addition to - // the JSON blob) — high-signal for exactly the failed / threshold-breaching - // operations this handler emits. + // When a cross-region hedge fanned out, surface the hedging signal as + // dedicated fields (in addition to the JSON blob) — high-signal for + // exactly the failed / threshold-breaching operations this handler emits. // - // Gate on `hedge_diagnostics` (with a fanned-out alternate) rather than - // `hedging_started()`: a both-transient hedge that is subsequently resolved - // by a failover attempt leaves a retained `Hedging` request (so - // `hedging_started()` is true) but no recorded terminal outcome - // (`hedge_diagnostics()` is `None` — `finalize_both_transient` deliberately - // does not stamp one on the non-terminal path). Emitting empty - // `hedge_region` / `hedge_terminal_state` strings there would be misleading; - // this gate keeps the log line consistent with the metrics counter, which is - // likewise only recorded when a terminal hedge outcome exists. - if let Some(hedge) = diagnostics + // Fan-out is decided by `hedging_started()`, consistently with the hedged + // metric counter and the root span. The per-outcome `hedge_region` / + // `hedge_terminal_state` fields additionally require `hedge_diagnostics` + // (with a fanned-out alternate): a both-transient hedge that is later + // resolved by a failover attempt has no recorded terminal outcome + // (`finalize_both_transient` deliberately does not stamp one on the + // non-terminal path). `tracing` field sets are fixed per call site, so + // that case gets its own arm carrying just `hedging_started` rather than + // emitting misleading empty strings — or, worse, dropping the signal. + let hedge = diagnostics .hedge_diagnostics() - .filter(|hedge| hedge.alternate_region().is_some()) - { + .filter(|hedge| hedge.alternate_region().is_some()); + if let Some(hedge) = hedge { let hedge_region = hedge .alternate_region() .map(|region| region.as_str()) @@ -106,6 +105,12 @@ impl DiagnosticsHandler for TracingLogHandler { } else { tracing::info!(target: SAMPLED_TARGET, reason, hedging_started = true, hedge_region, hedge_terminal_state, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); } + } else if diagnostics.hedging_started() { + if diagnostics.is_failure() { + tracing::warn!(target: SAMPLED_TARGET, reason, hedging_started = true, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); + } else { + tracing::info!(target: SAMPLED_TARGET, reason, hedging_started = true, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); + } } else if diagnostics.is_failure() { tracing::warn!(target: SAMPLED_TARGET, reason, diagnostics = %diagnostics.to_json_string(Some(DiagnosticsVerbosity::Summary)), "cosmos operation diagnostics"); } else { diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs index e83ae22cd8d..55b052b0710 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs @@ -384,13 +384,14 @@ mod tests { } #[test] - fn sampled_line_omits_hedging_fields_when_no_terminal_outcome() { + fn sampled_line_reports_fanout_without_terminal_outcome() { // A hedge fanned out both-transient and was then resolved by a later - // failover attempt: the retained `Hedging` request makes - // `hedging_started()` true, but the pipeline deliberately leaves - // `hedge_diagnostics()` None (no terminal outcome). The log line must - // still be emitted, but must NOT carry empty-string hedge_region / - // hedge_terminal_state fields — consistent with the metrics counter. + // failover attempt: the fan-out log makes `hedging_started()` true, but + // the pipeline deliberately leaves `hedge_diagnostics()` None (no + // terminal outcome). The line must still report `hedging_started` — the + // operation really did hedge — while omitting the per-outcome + // hedge_region / hedge_terminal_state fields rather than emitting empty + // strings for them. use azure_data_cosmos_driver::diagnostics::ExecutionContext; let captured = Arc::new(std::sync::Mutex::new(HashMap::new())); @@ -426,10 +427,14 @@ mod tests { }); let map = captured.lock().unwrap(); - // The line is emitted (it is a failure) but carries no hedge fields — - // crucially, no empty-string hedge_region / hedge_terminal_state. + // The line is emitted (it is a failure) and reports the fan-out, but + // carries no empty-string hedge_region / hedge_terminal_state. assert!(map.contains_key("reason")); - assert!(!map.contains_key("hedging_started")); + assert_eq!( + map.get("hedging_started").map(String::as_str), + Some("true"), + "a fanned-out hedge is reported even without a terminal outcome" + ); assert!(!map.contains_key("hedge_region")); assert!(!map.contains_key("hedge_terminal_state")); } diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs index 792f941a765..ed73dfd0564 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs @@ -210,6 +210,20 @@ impl CosmosMetricsHandler { } impl CosmosMetricsHandler { + /// Value of the `hedge_terminal_state` dimension when a hedge demonstrably + /// fanned out but the race retained no terminal outcome. + /// + /// The both-transient→failover path deliberately leaves `hedge_diagnostics` + /// unset so a later successful retry does not carry a misleading + /// `BothTransient` state. Those operations really did hedge, so they must be + /// counted; this sentinel keeps the counter's attribute schema uniform + /// (every data point carries the dimension, so `group by + /// hedge_terminal_state` never fragments) while staying distinguishable + /// from every real [`HedgeTerminalState`] value. + /// + /// [`HedgeTerminalState`]: azure_data_cosmos_driver::HedgeTerminalState + const HEDGE_TERMINAL_STATE_UNRESOLVED: &'static str = "unresolved"; + /// Records the hedged-operation counter for an operation that fanned out a /// cross-region hedge. /// @@ -218,23 +232,26 @@ impl CosmosMetricsHandler { /// extended-attributes opt-in (mirroring how contacted regions are gated on /// the duration metric). /// - /// The counter is emitted only when `hedge_diagnostics` is present, so a - /// data point can never be recorded without its `hedge_terminal_state` - /// dimension — a mixed attribute schema on the same counter would fragment - /// its time series and break `group by hedge_terminal_state`. An aggregated - /// operation (e.g. PATCH) whose sub-op hedged propagates a representative - /// `hedge_diagnostics`, so this stays consistent with non-aggregated ops. + /// Fan-out is decided by [`DiagnosticsContext::hedging_started`], which is + /// materialized from the dispatch-time fan-out log and is therefore the + /// authoritative signal — gating on `hedge_diagnostics` instead would + /// silently undercount, because a both-transient race that later succeeds + /// through failover retains no terminal outcome. An aggregated operation + /// (e.g. PATCH) whose sub-op hedged is counted once for the whole operation. fn record_hedged(&self, diagnostics: &DiagnosticsContext, base_attrs: &[KeyValue]) { - let Some(hedge) = diagnostics.hedge_diagnostics() else { + if !diagnostics.hedging_started() { return; - }; + } + let hedge = diagnostics.hedge_diagnostics(); let mut attrs = base_attrs.to_vec(); attrs.push(KeyValue::new( attributes::ATTR_HEDGE_TERMINAL_STATE, - hedge.terminal_state().as_str(), + hedge.map_or(Self::HEDGE_TERMINAL_STATE_UNRESOLVED, |hedge| { + hedge.terminal_state().as_str() + }), )); if self.options.extended_attributes_enabled() { - if let Some(alternate) = hedge.alternate_region() { + if let Some(alternate) = hedge.and_then(|hedge| hedge.alternate_region()) { attrs.push(KeyValue::new( attributes::ATTR_HEDGE_REGION, alternate.as_str().to_string(), @@ -286,8 +303,9 @@ impl DiagnosticsHandler for CosmosMetricsHandler { } // Hedging counter: emitted only when opted in and a hedge actually - // fanned out. Reuses H1's hedging_started() detection. - if self.options.hedged_metric_enabled() && diagnostics.hedging_started() { + // fanned out. `record_hedged` re-checks fan-out so the invariant holds + // regardless of call site. + if self.options.hedged_metric_enabled() { self.record_hedged(diagnostics, &attributes); } } @@ -713,17 +731,15 @@ mod tests { } #[test] - fn hedged_metric_skips_when_terminal_state_unavailable() { + fn hedged_metric_counts_hedge_without_terminal_state() { // A hedge that fanned out both-transient and was then resolved by a later // failover attempt leaves a retained `Hedging` request (so // `hedging_started()` is true) but no recorded terminal outcome // (`finalize_both_transient` deliberately does not stamp `hedge_diagnostics` - // on the non-terminal path — see operation_pipeline.rs). The counter is - // intentionally skipped there rather than emitted without its - // `hedge_terminal_state` dimension, which would fragment the counter's time - // series. The counter therefore measures hedges with a resolved terminal - // outcome; a both-transient hedge whose winning response ultimately came - // from a failover attempt is not counted here. + // on the non-terminal path — see operation_pipeline.rs). That operation + // really did hedge, so it must be counted; the dimension carries the + // `unresolved` sentinel rather than being omitted, which keeps the + // counter's attribute schema uniform for `group by hedge_terminal_state`. use azure_data_cosmos_driver::diagnostics::{ExecutionContext, RequestDiagnostics}; use azure_data_cosmos_driver::models::RequestCharge; use std::time::Instant; @@ -759,9 +775,15 @@ mod tests { handler.handle(&ctx, &cx); let metrics = harness.collect(); - assert!( - hedged_point(&metrics).is_none(), - "the counter must not emit a data point missing the hedge_terminal_state dimension" + let (attrs, value) = hedged_point(&metrics) + .expect("a hedge that fanned out is counted even without a terminal outcome"); + assert_eq!(value, 1); + assert_eq!( + attrs + .get(attributes::ATTR_HEDGE_TERMINAL_STATE) + .map(String::as_str), + Some("unresolved"), + "the dimension is always present so the counter's schema stays uniform" ); } } diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs index a89f3853c80..c9aacc95459 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs @@ -179,18 +179,15 @@ pub(crate) fn emit_backdated_span_tree( // Hedging surfacing: only when a cross-region hedge actually fanned out. // These attributes stay off the common (non-hedged) sampled span entirely. // - // The span intentionally gates on `hedging_started()` (any fan-out) rather - // than on `hedge_diagnostics()` (a resolved terminal outcome), because the - // span is a rich per-operation diagnostic: the region history - // (`requested_regions`/`responded_regions`) is valuable exactly for a - // both-transient hedge that was then resolved by failover, and - // `HEDGING_STARTED=true` there is factually correct. The per-outcome - // `HEDGE_REGION`/`HEDGE_TERMINAL_STATE` fields are still gated on - // `hedge_diagnostics()` below, so no empty/placeholder values are emitted. - // This is a deliberate, documented asymmetry vs. the hedged metric counter - // and the log hedge fields, which key off `hedge_diagnostics` (a resolved - // terminal outcome) so every counter data point carries the - // `hedge_terminal_state` dimension — see `DiagnosticsContext::hedging_started`. + // Fan-out is decided by `hedging_started()` (materialized from the + // dispatch-time fan-out log), consistently with the hedged metric counter + // and the sampled log line. The per-outcome `HEDGE_REGION` / + // `HEDGE_TERMINAL_STATE` fields are additionally gated on + // `hedge_diagnostics()`, which is `None` when a both-transient race was + // resolved by a later failover attempt — so those fields are simply absent + // there rather than carrying empty or placeholder values. The region + // history is still emitted on that path, which is exactly where it is most + // useful. if diagnostics.hedging_started() { root_attrs.push(KeyValue::new(attributes::HEDGING_STARTED, true)); if let Some(hedge) = diagnostics.hedge_diagnostics() { From 87e305a0f3d330a03a6e9873c3eeb7997b83c47a Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Tue, 28 Jul 2026 12:13:43 -0700 Subject: [PATCH 09/21] Fix broken intra-doc link in metrics handler `HedgeTerminalState` lives at `azure_data_cosmos_driver::diagnostics::`, not at the crate root, so the reference link on the new `HEDGE_TERMINAL_STATE_UNRESOLVED` sentinel failed to resolve and `cargo doc` with `-D warnings` errored. Switched to plain backticks, matching how `diagnostics/attributes.rs` already refers to the same type. The const is private, so a rustdoc hyperlink adds no navigational value. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 37e127ae-4bbe-4eb4-a533-c6ddd54e884e --- .../azure_data_cosmos/src/diagnostics/metrics/handler.rs | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs index ed73dfd0564..ee7d68402e3 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs @@ -219,9 +219,7 @@ impl CosmosMetricsHandler { /// counted; this sentinel keeps the counter's attribute schema uniform /// (every data point carries the dimension, so `group by /// hedge_terminal_state` never fragments) while staying distinguishable - /// from every real [`HedgeTerminalState`] value. - /// - /// [`HedgeTerminalState`]: azure_data_cosmos_driver::HedgeTerminalState + /// from every real `HedgeTerminalState` value. const HEDGE_TERMINAL_STATE_UNRESOLVED: &'static str = "unresolved"; /// Records the hedged-operation counter for an operation that fanned out a From 9bad8036a94d48bfb55bbca19ab22416bcf6a1a3 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Tue, 28 Jul 2026 12:42:07 -0700 Subject: [PATCH 10/21] Add fanouts and undercount to cosmos cspell dictionary The Build Analyze CI job runs cSpell over every changed file and failed on two words introduced by this PR. `fanouts` comes from the `hedge_fanouts` builder field; cSpell splits the identifier on `_`, and the dictionary already carried the singular `fanout`. `undercount` is standard English but absent from cSpell's base dictionary. Both are correct as written, so they are ignored rather than reworded. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 37e127ae-4bbe-4eb4-a533-c6ddd54e884e --- sdk/cosmos/.cspell.json | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sdk/cosmos/.cspell.json b/sdk/cosmos/.cspell.json index 26ee084a7f3..f352010605c 100644 --- a/sdk/cosmos/.cspell.json +++ b/sdk/cosmos/.cspell.json @@ -115,6 +115,7 @@ "failback", "failovers", "fanout", + "fanouts", "Fatalf", "fieldless", "FILETIME", @@ -326,6 +327,7 @@ "uncollapsed", "uncontended", "undecoded", + "undercount", "underspecified", "undrained", "unfaulted", From 42e63bd8700a5f2cb3f5104aa05538cb505bbb44 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Tue, 28 Jul 2026 13:46:46 -0700 Subject: [PATCH 11/21] bound materialized hedge region histories MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Materializing `requested_regions` / `responded_regions` from the full pre-compaction attempt list is what lets a structurally-dropped hedge leg survive, but on its own it made both lists grow one entry per attempt while `requests()` stayed capped. A 410/429 retry storm therefore produced an O(attempt-count) diagnostics artifact, contradicting the bounded-size guarantee in DIAGNOSTICS-CONTRACT.md section 8 and flowing straight into root-span attributes (section "Span tree"). There were three separate unbounded paths: `complete()`, the concatenation in `aggregate_sub_operations` (a PATCH conflict loop adds a sub-operation per retry, so re-applying the cap there is not redundant), and the span builder consuming both lists. Both histories are now capped at `max_request_diagnostics`, keeping head and tail and eliding the repetitive middle — the same shape `compact_requests` already uses, at whole-list granularity. The head preserves the initial dispatch and any early hedge fan-out; the tail preserves where the operation landed. Truncation is explicit, not silent: `total_requested_regions()` / `total_responded_regions()` report the exact pre-truncation counts, and the span emits a matching `*_total` attribute only when the history was actually truncated, so the normal path carries no redundant integer. Both totals are compared in `PartialEq` for the same reason `total_request_charge` is: after truncation they are no longer derivable from the retained vectors. Also corrects the `hedging_started()` doc, which still described the metric counter and log fields as keying off `hedge_diagnostics` after b81bf39e4 moved them onto `hedging_started()`. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 37e127ae-4bbe-4eb4-a533-c6ddd54e884e --- sdk/cosmos/azure_data_cosmos/CHANGELOG.md | 4 +- .../docs/HEDGING_DETECTION_API_SPEC.md | 31 +- .../src/diagnostics/attributes.rs | 13 + .../src/diagnostics/tracing/mod.rs | 13 + .../src/diagnostics/tracing/span_builder.rs | 20 ++ .../azure_data_cosmos_driver/CHANGELOG.md | 2 +- .../src/diagnostics/diagnostics_context.rs | 289 ++++++++++++++++-- 7 files changed, 345 insertions(+), 27 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md index 605032c6d54..48d1374419b 100644 --- a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md @@ -5,11 +5,11 @@ ### Features Added - Added opt-in Cosmos binary JSON encoding for item operations (`create`/`read`/`replace`/`upsert`). Enable it via `CosmosClientBuilder::with_binary_encoding_options` (or the `AZURE_COSMOS_BINARY_ENCODING_ENABLED` environment-variable fallback). Off by default; when disabled, requests and responses are byte-for-byte unchanged. ([#4671](https://github.com/Azure/azure-sdk-for-rust/pull/4671)) -- Surfaced the Hedging Detection API through `DiagnosticsContext`: `hedging_started()`, `requested_regions()`, and `responded_regions()`, and re-exported the `RequestedRegion` / `RequestedRegionReason` types (mirroring the existing `DiagnosticsContext` re-export). ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Surfaced the Hedging Detection API through `DiagnosticsContext`: `hedging_started()`, `requested_regions()`, `responded_regions()`, and the exact-count `total_requested_regions()` / `total_responded_regions()`, and re-exported the `RequestedRegion` / `RequestedRegionReason` types (mirroring the existing `DiagnosticsContext` re-export). ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Added a pluggable client-side diagnostics emission layer — the `DiagnosticsHandler` trait and ordered `DiagnosticsHandlerChain` (registered via `CosmosClientBuilder::with_diagnostics_handler`) — invoked once per operation (singleton and paginated, on success and failure) with the completed `DiagnosticsContext` plus an SDK-supplied `CosmosOperationContext`; the empty default chain is a zero-overhead no-op. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added the `metrics`-gated `CosmosMetricsHandler` (with `MetricsOptions`), emitting the stable `db.client.operation.duration` histogram plus per-signal opt-in metrics (`with_request_charge_metric`, `with_returned_rows_metric`) and an opt-in extended attribute set (`with_extended_attributes`); a no-op when no meter provider is registered. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added composable tail-sampled emission handlers — a `TracingLogHandler` leaf that writes a compact `tracing` line and a `SamplingLogHandler` wrapper (holding an `Arc`) that applies the sampling gate plus a shared per-window rate limit, defaulting to wrap a `TracingLogHandler` — and the `distributed_tracing`-gated `CosmosTracingHandler` (backdated span tree), also rate-limited so an error storm can't overwhelm exporters. All emit only for operations which fail or breach a configurable `DiagnosticsThresholds`, and stamp *why* they were sampled (a failure, or which threshold) on the emitted line and span. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) -- Surfaced hedging through the observability handlers, reusing the Hedging Detection API. When a cross-region hedge fans out, `CosmosTracingHandler` adds `azure.cosmosdb.operation.{hedging_started,hedge_region,hedge_terminal_state}` plus `requested_regions`/`responded_regions` (`string[]`) to the sampled operation span and tags the hedge-leg child span (`azure.cosmosdb.request.hedge`); `SamplingLogHandler` adds `hedging_started` / `hedge_region` / `hedge_terminal_state` fields to the sampled log line; and `CosmosMetricsHandler` gains an opt-in `azure.cosmosdb.client.operation.hedged` counter (`MetricsOptions::with_hedged_metric`) carrying the low-cardinality `hedge_terminal_state`, with the higher-cardinality `hedge_region` dimension added only under `with_extended_attributes`. All three surfaces decide fan-out from `hedging_started()`, so a hedge whose race ended both-transient and was then resolved by a failover attempt is still reported; the per-outcome `hedge_region` / `hedge_terminal_state` fields are omitted there (the counter's dimension carries the `unresolved` sentinel so its attribute schema stays uniform). ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Surfaced hedging through the observability handlers, reusing the Hedging Detection API. When a cross-region hedge fans out, `CosmosTracingHandler` adds `azure.cosmosdb.operation.{hedging_started,hedge_region,hedge_terminal_state}` plus `requested_regions`/`responded_regions` (`string[]`, bounded by the driver's `max_request_diagnostics`, with a `*_total` companion attribute emitted only when a retry storm truncated the history) to the sampled operation span and tags the hedge-leg child span (`azure.cosmosdb.request.hedge`); `SamplingLogHandler` adds `hedging_started` / `hedge_region` / `hedge_terminal_state` fields to the sampled log line; and `CosmosMetricsHandler` gains an opt-in `azure.cosmosdb.client.operation.hedged` counter (`MetricsOptions::with_hedged_metric`) carrying the low-cardinality `hedge_terminal_state`, with the higher-cardinality `hedge_region` dimension added only under `with_extended_attributes`. All three surfaces decide fan-out from `hedging_started()`, so a hedge whose race ended both-transient and was then resolved by a failover attempt is still reported; the per-outcome `hedge_region` / `hedge_terminal_state` fields are omitted there (the counter's dimension carries the `unresolved` sentinel so its attribute schema stays uniform). ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) ### Breaking Changes diff --git a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md index a0d99647dd1..1981376dbde 100644 --- a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md +++ b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md @@ -18,8 +18,10 @@ value types, all re-exported from `azure_data_cosmos`: | Member | Signature | Semantics | | --- | --- | --- | | `DiagnosticsContext::hedging_started` | `-> bool` | `true` iff a hedge arm was actually dispatched (fan-out happened). | -| `DiagnosticsContext::requested_regions` | `-> Vec` | Regions dispatched to, in **dispatch order**, duplicates allowed, each tagged with a reason. | -| `DiagnosticsContext::responded_regions` | `-> Vec<&Region>` | Regions that produced a **service reply**, in **completion order**, duplicates allowed. | +| `DiagnosticsContext::requested_regions` | `-> Vec` | Regions dispatched to, in **dispatch order**, duplicates allowed, each tagged with a reason. Bounded (§4.5). | +| `DiagnosticsContext::responded_regions` | `-> Vec<&Region>` | Regions that produced a **service reply**, in **completion order**, duplicates allowed. Bounded (§4.5). | +| `DiagnosticsContext::total_requested_regions` | `-> usize` | Exact dispatch count, including entries elided by the bound. | +| `DiagnosticsContext::total_responded_regions` | `-> usize` | Exact reply count, including entries elided by the bound. | | `RequestedRegion` | `{ region, reason }` | A dispatched region paired with the reason it was chosen. | | `RequestedRegionReason` | enum | Why the SDK dispatched to a region; `#[non_exhaustive]`. | @@ -193,6 +195,31 @@ order among ties); duplicates are preserved. To deduplicate, collect into a pre-compaction attempt list, so a response whose attempt was later compacted away is still reported. +### 4.5 Both histories are bounded + +Materializing from the *pre-compaction* attempt list is what makes a dropped +hedge leg survive, but taken alone it would make the two histories grow with +attempt count — a 410/429 retry storm would produce an O(attempts) artifact even +though the retained `requests()` list stays capped. That violates the bounded-size +guarantee in the driver's `DIAGNOSTICS-CONTRACT.md` §8, which requires every +materialized representation to have an upper bound independent of attempt count, +and it would flow straight into span attributes. + +Both histories are therefore capped at `max_request_diagnostics` (default 512, +minimum 16) at finalization, and re-capped in `aggregate_sub_operations` — the +aggregate is an independent unbounded path, since a PATCH conflict loop adds a +sub-operation per retry. + +The elision keeps the **head and tail** of the history and drops the repetitive +middle, mirroring the "first and last of each run" policy the contract already +applies to attempt compaction. The head preserves the initial dispatch and any +early hedge fan-out; the tail preserves where the operation finally landed. + +Truncation is never silent. `total_requested_regions()` / `total_responded_regions()` +report the exact pre-truncation counts, so `requested_regions().len() < total_requested_regions()` +detects an elision. On the span, the matching `*_total` attribute is emitted only +when the history was truncated, so the normal path carries no redundant integer. + --- ## 5. Reconciliation with `HedgeDiagnostics` diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/attributes.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/attributes.rs index 06dc66ea95f..c0eb0c406c7 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/attributes.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/attributes.rs @@ -66,12 +66,25 @@ pub(crate) const HEDGE_TERMINAL_STATE: &str = "azure.cosmosdb.operation.hedge_te /// `azure.cosmosdb.operation.requested_regions` — regions dispatched to, in /// dispatch order (`string[]`). High-signal for hedge fan-out. +/// +/// Bounded by `max_request_diagnostics`; see [`REQUESTED_REGIONS_TOTAL`]. pub(crate) const REQUESTED_REGIONS: &str = "azure.cosmosdb.operation.requested_regions"; +/// `azure.cosmosdb.operation.requested_regions_total` — exact dispatch count, +/// emitted only when [`REQUESTED_REGIONS`] was truncated under a retry storm so +/// the elision is explicit rather than silent. +pub(crate) const REQUESTED_REGIONS_TOTAL: &str = "azure.cosmosdb.operation.requested_regions_total"; + /// `azure.cosmosdb.operation.responded_regions` — regions that returned a /// service reply, in arrival order (`string[]`). +/// +/// Bounded by `max_request_diagnostics`; see [`RESPONDED_REGIONS_TOTAL`]. pub(crate) const RESPONDED_REGIONS: &str = "azure.cosmosdb.operation.responded_regions"; +/// `azure.cosmosdb.operation.responded_regions_total` — exact reply count, +/// emitted only when [`RESPONDED_REGIONS`] was truncated under a retry storm. +pub(crate) const RESPONDED_REGIONS_TOTAL: &str = "azure.cosmosdb.operation.responded_regions_total"; + /// `azure.cosmosdb.request.hedge` — `true` on the per-attempt (child) span for a /// speculative hedge leg dispatched to an alternate region. pub(crate) const HEDGE_LEG: &str = "azure.cosmosdb.request.hedge"; diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs index 2f5ad822b0e..c36738e262c 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs @@ -443,6 +443,19 @@ mod tests { !responded.iter().any(|r| r == "eastus"), "the structurally-dropped primary leg must not appear in responded_regions" ); + // Nothing was truncated, so the exact-count attributes stay off the span + // entirely — they exist only to make an elision explicit, and emitting + // them unconditionally would put a redundant integer on every hedged + // span. + for key in [ + attributes::REQUESTED_REGIONS_TOTAL, + attributes::RESPONDED_REGIONS_TOTAL, + ] { + assert!( + !root.attributes.iter().any(|kv| kv.key.as_str() == key), + "{key} must be absent when the region history was not truncated" + ); + } // The speculative hedge leg's child span is tagged. let tagged = spans diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs index c9aacc95459..83182086086 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs @@ -188,6 +188,12 @@ pub(crate) fn emit_backdated_span_tree( // there rather than carrying empty or placeholder values. The region // history is still emitted on that path, which is exactly where it is most // useful. + // + // Both region arrays come from the driver already bounded by + // `max_request_diagnostics`, so a retry storm cannot produce an unbounded + // span attribute. When the driver elided the middle of a history, the + // matching `*_total` count is emitted alongside so the truncation is + // explicit in the telemetry rather than silent. if diagnostics.hedging_started() { root_attrs.push(KeyValue::new(attributes::HEDGING_STARTED, true)); if let Some(hedge) = diagnostics.hedge_diagnostics() { @@ -208,6 +214,13 @@ pub(crate) fn emit_backdated_span_tree( attributes::REQUESTED_REGIONS, region_string_array(requested.iter().map(|r| r.region.as_str())), )); + let total = diagnostics.total_requested_regions(); + if total > requested.len() { + root_attrs.push(KeyValue::new( + attributes::REQUESTED_REGIONS_TOTAL, + total as i64, + )); + } } let responded = diagnostics.responded_regions(); if !responded.is_empty() { @@ -215,6 +228,13 @@ pub(crate) fn emit_backdated_span_tree( attributes::RESPONDED_REGIONS, region_string_array(responded.iter().map(|region| region.as_str())), )); + let total = diagnostics.total_responded_regions(); + if total > responded.len() { + root_attrs.push(KeyValue::new( + attributes::RESPONDED_REGIONS_TOTAL, + total as i64, + )); + } } } // Prefer the caller-supplied server-address override (mirroring the metrics diff --git a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md index 75b4852c6fb..0e00937619c 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md @@ -5,7 +5,7 @@ ### Features Added - Added a schema-agnostic Cosmos binary JSON codec (`binary_json`) and driver-side binary encoding via `OperationOptions.binary_encoding` (`BinaryEncodingOptions`). When enabled, the driver transcodes item request/response bodies between text and Cosmos binary JSON and negotiates the wire format; it is honored only for point `Document` item operations. Off by default and inert on the wire when unset. ([#4671](https://github.com/Azure/azure-sdk-for-rust/pull/4671)) -- Added the Hedging Detection API on `DiagnosticsContext`: `hedging_started()` (whether a hedge arm actually fanned out), `requested_regions()` (regions dispatched to, in dispatch order with duplicates, each tagged with a reason), and `responded_regions()` (regions that produced an actual service reply, in completion order). All three are materialized at finalization from the full pre-compaction attempt list plus a dispatch-time hedge fan-out log, so a structurally-dropped hedge leg, a retry storm that compacts `requests()`, and multi-round-trip aggregation all report a complete history. Adds the public `RequestedRegion` struct and `RequestedRegionReason` enum (both `#[non_exhaustive]`, with a total `From` mapping) and the new `ExecutionContext::OperationRetry` variant. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Added the Hedging Detection API on `DiagnosticsContext`: `hedging_started()` (whether a hedge arm actually fanned out), `requested_regions()` (regions dispatched to, in dispatch order with duplicates, each tagged with a reason), and `responded_regions()` (regions that produced an actual service reply, in completion order). All three are materialized at finalization from the full pre-compaction attempt list plus a dispatch-time hedge fan-out log, so a structurally-dropped hedge leg, a retry storm that compacts `requests()`, and multi-round-trip aggregation all report a complete history. Both region histories are bounded by `max_request_diagnostics` (keeping head and tail, dropping the repetitive middle) so a retry storm cannot produce an unbounded diagnostics artifact; `total_requested_regions()` / `total_responded_regions()` report the exact pre-truncation counts, so truncation is detectable rather than silent. Adds the public `RequestedRegion` struct and `RequestedRegionReason` enum (both `#[non_exhaustive]`, with a total `From` mapping) and the new `ExecutionContext::OperationRetry` variant. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Added `HedgeTerminalState::as_str()` (and a matching `Display`) returning a stable, low-cardinality `snake_case` identifier for the hedging race outcome, for use as an observability attribute / log-field value. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) ### Breaking Changes diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs index a39b2c40b54..82c4e776861 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs @@ -1860,8 +1860,18 @@ impl DiagnosticsContextBuilder { // attempt list plus the dispatch-time hedge fan-out log, for the same // reason: neither the retained (possibly compacted) attempt list nor the // winning leg alone is a complete dispatch history. + // + // Both histories are then bounded by the same `max_request_diagnostics` + // cap as the attempt list, so a retry storm cannot grow them without + // limit (DIAGNOSTICS-CONTRACT.md §8). The pre-truncation lengths are + // retained so the truncation is explicit rather than silent. + let cap = self.options.max_request_diagnostics(); let requested_regions = requested_regions_from(&self.requests, &self.hedge_fanouts); let responded_regions = responded_regions_from(&self.requests); + let total_requested_regions = requested_regions.len(); + let total_responded_regions = responded_regions.len(); + let requested_regions = bound_region_history(requested_regions, cap); + let responded_regions = bound_region_history(responded_regions, cap); let hedging_started = !self.hedge_fanouts.is_empty() || self .hedge_diagnostics @@ -1888,7 +1898,6 @@ impl DiagnosticsContextBuilder { // is on the finalized serialized artifact, not on live mid-operation // memory: `self.requests` still grows one entry per attempt while the // operation is in flight. - let cap = self.options.max_request_diagnostics(); let original_count = self.requests.len(); let (requests, compaction) = if original_count > cap { let compacted = compact_requests(self.requests, cap); @@ -1915,6 +1924,8 @@ impl DiagnosticsContextBuilder { regions_contacted, requested_regions, responded_regions, + total_requested_regions, + total_responded_regions, hedging_started, status: self.status, options: self.options, @@ -2012,16 +2023,29 @@ pub struct DiagnosticsContext { /// compacted-away retry are both still reported. Unlike `regions_contacted` /// this list is *not* deduplicated: it is a dispatch log, so repeat /// dispatches to the same region each contribute an entry. + /// + /// Bounded by `max_request_diagnostics` (head + tail retained, repetitive + /// middle elided) so a retry storm cannot grow it without limit; + /// `total_requested_regions` records the pre-truncation length. requested_regions: Vec, /// Regions that produced an actual service reply, in arrival (completion) /// order. /// /// Materialized at finalization from the **full** attempt list, for the same - /// reason as `requested_regions`. Client-side timeouts and transport - /// failures are excluded; a non-2xx service response still counts. + /// reason as `requested_regions`, and bounded the same way. Client-side + /// timeouts and transport failures are excluded; a non-2xx service response + /// still counts. responded_regions: Vec, + /// Exact number of dispatches recorded before `requested_regions` was + /// bounded. Equal to `requested_regions.len()` when no truncation occurred. + total_requested_regions: usize, + + /// Exact number of service replies recorded before `responded_regions` was + /// bounded. Equal to `responded_regions.len()` when no truncation occurred. + total_responded_regions: usize, + /// Whether this operation actually fanned out at least one hedge request. /// /// Materialized at finalization so it survives compaction dropping the @@ -2160,6 +2184,10 @@ impl DiagnosticsContext { let regions_contacted = ordered_unique_regions(&requests); let requested_regions = requested_regions_from(&requests, &[]); let responded_regions = responded_regions_from(&requests); + // The helper builds a fixed, hand-supplied attempt list, so the history + // is never over the cap and the totals are simply its length. + let total_requested_regions = requested_regions.len(); + let total_responded_regions = responded_regions.len(); let hedging_started = requests .iter() .any(|r| matches!(r.execution_context(), ExecutionContext::Hedging)); @@ -2171,6 +2199,8 @@ impl DiagnosticsContext { regions_contacted, requested_regions, responded_regions, + total_requested_regions, + total_responded_regions, hedging_started, status, options: Arc::new(DiagnosticsOptions::default()), @@ -2359,14 +2389,30 @@ impl DiagnosticsContext { // full attempt list plus its own fan-out records, so every sub-op's // hedge fan-out is preserved — unlike the single representative // `hedge_diagnostics` below, which can only describe one of them. - let requested_regions: Vec = sources - .iter() - .flat_map(|c| c.requested_regions.iter().cloned()) - .collect(); - let responded_regions: Vec = sources - .iter() - .flat_map(|c| c.responded_regions.iter().cloned()) - .collect(); + // + // Concatenation is an independent unbounded path — a PATCH conflict loop + // adds a sub-op per retry, so the aggregate grows with sub-op count even + // when each source is individually bounded. Re-bound the result under + // the same cap and carry the summed pre-truncation totals, so the + // aggregate honours the same guarantee as a single operation. + let total_requested_regions: usize = + sources.iter().map(|c| c.total_requested_regions).sum(); + let total_responded_regions: usize = + sources.iter().map(|c| c.total_responded_regions).sum(); + let requested_regions = bound_region_history( + sources + .iter() + .flat_map(|c| c.requested_regions.iter().cloned()) + .collect(), + cap, + ); + let responded_regions = bound_region_history( + sources + .iter() + .flat_map(|c| c.responded_regions.iter().cloned()) + .collect(), + cap, + ); let hedging_started = sources.iter().any(|c| c.hedging_started); Some(DiagnosticsContext { @@ -2377,6 +2423,8 @@ impl DiagnosticsContext { regions_contacted, requested_regions, responded_regions, + total_requested_regions, + total_responded_regions, hedging_started, status: last.status, options: Arc::clone(&last.options), @@ -2545,10 +2593,29 @@ impl DiagnosticsContext { /// /// Order is dispatch order: each fan-out's two legs are spliced in at the /// point the race was dispatched, primary before alternate. + /// + /// # Bounded under a retry storm + /// + /// The list is capped at `DiagnosticsOptions::max_request_diagnostics` + /// (default 512). Past that, the head and tail are kept verbatim and the + /// repetitive middle is elided, so the initial dispatch, any early hedge + /// fan-out, and the final landing region all survive. Truncation is never + /// silent: [`total_requested_regions`](Self::total_requested_regions) + /// reports the exact pre-truncation count, so + /// `requested_regions().len() < total_requested_regions()` detects it. pub fn requested_regions(&self) -> Vec { self.requested_regions.clone() } + /// Returns the exact number of dispatches recorded for this operation, + /// including any elided by the bound on + /// [`requested_regions`](Self::requested_regions). + /// + /// Equal to `requested_regions().len()` unless the history was truncated. + pub fn total_requested_regions(&self) -> usize { + self.total_requested_regions + } + /// Returns the regions from which this operation received a response, in /// arrival (completion) order. /// @@ -2576,10 +2643,25 @@ impl DiagnosticsContext { /// /// To deduplicate, callers can collect into a set, for example: /// `ctx.responded_regions().into_iter().collect::>()`. + /// + /// # Bounded under a retry storm + /// + /// Capped the same way as [`requested_regions`](Self::requested_regions); + /// [`total_responded_regions`](Self::total_responded_regions) reports the + /// exact pre-truncation count. pub fn responded_regions(&self) -> Vec<&Region> { self.responded_regions.iter().collect() } + /// Returns the exact number of service replies recorded for this operation, + /// including any elided by the bound on + /// [`responded_regions`](Self::responded_regions). + /// + /// Equal to `responded_regions().len()` unless the history was truncated. + pub fn total_responded_regions(&self) -> usize { + self.total_responded_regions + } + /// Returns `true` iff this operation actually dispatched at least one hedge /// request (i.e., fan-out occurred), and `false` otherwise. /// @@ -2602,10 +2684,11 @@ impl DiagnosticsContext { /// terminal outcome). So `hedge_diagnostics().is_some()` is not a reliable /// "was hedging configured" probe, and it can disagree with /// `hedging_started()` on that both-transient→failover path (where the - /// recorded fan-out keeps this accessor `true`). The SDK's hedged metric - /// counter and log hedge fields key off `hedge_diagnostics` (a resolved - /// terminal outcome), so they intentionally do not surface that path even - /// though `hedging_started()` is `true`. + /// recorded fan-out keeps this accessor `true`). This accessor is the + /// authoritative fan-out signal: the SDK's hedged metric counter, sampled + /// log line, and root-span hedging attributes all gate on it, so an + /// operation that demonstrably fanned out is reported on every surface even + /// when no terminal outcome was retained. pub fn hedging_started(&self) -> bool { self.hedging_started } @@ -2893,6 +2976,8 @@ impl Clone for DiagnosticsContext { regions_contacted: self.regions_contacted.clone(), requested_regions: self.requested_regions.clone(), responded_regions: self.responded_regions.clone(), + total_requested_regions: self.total_requested_regions, + total_responded_regions: self.total_responded_regions, hedging_started: self.hedging_started, status: self.status, options: Arc::clone(&self.options), @@ -2936,6 +3021,13 @@ impl PartialEq for DiagnosticsContext { && self.regions_contacted == other.regions_contacted && self.requested_regions == other.requested_regions && self.responded_regions == other.responded_regions + // Compared for the same reason as `total_request_charge`: after the + // region histories are bounded these are no longer derivable from + // the retained vectors, so excluding them would let two contexts + // with different public `total_requested_regions()` / + // `total_responded_regions()` results compare equal. + && self.total_requested_regions == other.total_requested_regions + && self.total_responded_regions == other.total_responded_regions && self.hedging_started == other.hedging_started && self.status == other.status && self.options == other.options @@ -2998,6 +3090,30 @@ fn ordered_unique_regions(requests: &[RequestDiagnostics]) -> Vec { regions } +/// Bounds a materialized region history to `cap` entries, independently of +/// attempt count, per the bounded-size guarantee in `DIAGNOSTICS-CONTRACT.md` +/// §8. +/// +/// Under a `410`/`429` retry storm the dispatch history grows one entry per +/// attempt, so it needs its own bound — the per-attempt list's compaction does +/// not apply to it. The repetitive middle of a storm is elided while the head +/// (the initial dispatch and any early hedge fan-out) and the tail (where the +/// operation finally landed) are kept verbatim, mirroring the "first + last of +/// each run" policy `compact_requests` already uses. +/// +/// Truncation is never silent: the caller records the pre-truncation length, +/// which [`DiagnosticsContext::total_requested_regions`] and +/// [`DiagnosticsContext::total_responded_regions`] expose. +fn bound_region_history(mut history: Vec, cap: usize) -> Vec { + if history.len() <= cap { + return history; + } + let head = cap.div_ceil(2); + let tail = cap - head; + history.drain(head..history.len() - tail); + history +} + /// Builds the dispatch-ordered requested-region history from the **full** /// (pre-compaction) attempt list, splicing in each hedge fan-out at the point it /// was dispatched. @@ -4625,11 +4741,12 @@ mod tests { } #[test] - fn requested_regions_survives_retry_storm_compaction() { + fn requested_regions_bounded_but_exact_under_retry_storm() { // The dispatch history is materialized from the FULL attempt list, so a - // retry storm that compacts `requests` down to the cap must not shrink - // it — including the hedge fan-out recorded mid-storm. Repeat dispatches - // to one region must survive as distinct entries. + // retry storm that compacts `requests` down to the cap must not lose the + // hedge fan-out recorded mid-storm. But the history itself is also + // bounded (DIAGNOSTICS-CONTRACT.md §8) — the elision keeps head and tail + // and the exact count stays available. let cap = 16; let options = Arc::new( DiagnosticsOptions::builder() @@ -4662,12 +4779,140 @@ mod tests { // The retained attempt list really was bounded... assert!(ctx.requests().len() <= cap); assert!(ctx.compaction().is_some()); - // ...but the dispatch history is complete: 40 retries + both fan-out + // ...and so is the dispatch history, independently of attempt count + // (DIAGNOSTICS-CONTRACT.md §8) — but the elision is explicit, not + // silent: the exact count is still reported. 40 retries + both fan-out // legs (the winning alternate's merged attempt is absorbed by the // fan-out entry it repeats). - assert_eq!(ctx.requested_regions().len(), 42); + assert_eq!(ctx.total_requested_regions(), 42); + assert_eq!(ctx.requested_regions().len(), cap); + assert_eq!(ctx.total_responded_regions(), 41); + assert_eq!(ctx.responded_regions().len(), cap); assert!(ctx.hedging_started()); - assert_eq!(ctx.responded_regions().len(), 41); + + // Head and tail survive: the storm's opening dispatch and the hedge + // fan-out that ended it are both still visible. + let requested = ctx.requested_regions(); + assert_eq!(requested[0].region, Region::EAST_US_2); + assert_eq!(requested[0].reason, RequestedRegionReason::OperationRetry); + assert_eq!( + requested.last().expect("non-empty").reason, + RequestedRegionReason::Hedging + ); + assert_eq!( + requested.last().expect("non-empty").region, + Region::WEST_US_2 + ); + } + + #[test] + fn region_history_is_bounded_without_a_hedge() { + // The bound is a property of the history itself, not of hedging: a plain + // retry storm must not produce an unbounded artifact either. + let cap = 16; + let options = Arc::new( + DiagnosticsOptions::builder() + .with_max_request_diagnostics(cap) + .build() + .expect("valid options"), + ); + let mut builder = DiagnosticsContextBuilder::new(ActivityId::new_uuid(), options); + for _ in 0..500 { + let h = builder.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::TooManyRequests, None); + } + let ctx = builder.complete(); + + assert_eq!(ctx.total_requested_regions(), 500); + assert_eq!(ctx.requested_regions().len(), cap); + assert_eq!(ctx.total_responded_regions(), 500); + assert_eq!(ctx.responded_regions().len(), cap); + assert!(!ctx.hedging_started()); + } + + #[test] + fn region_history_is_verbatim_under_the_cap() { + // The common path must be untouched: at or below the cap the history is + // exact and the reported total agrees with it. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + for region in [Region::EAST_US_2, Region::WEST_US_2] { + let h = builder.start_test_request( + ExecutionContext::RegionFailover, + Some(region), + "https://test.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Ok, None); + } + }); + + assert_eq!(ctx.requested_regions().len(), 2); + assert_eq!(ctx.total_requested_regions(), 2); + assert_eq!(ctx.responded_regions().len(), 2); + assert_eq!(ctx.total_responded_regions(), 2); + } + + #[test] + fn aggregated_region_history_is_rebounded() { + // Aggregation concatenates each sub-op's already-bounded history, which + // is an independent unbounded path: a PATCH conflict loop adds a sub-op + // per retry. The aggregate must be re-bounded and must report the summed + // exact totals. + let cap = 16; + let options = Arc::new( + DiagnosticsOptions::builder() + .with_max_request_diagnostics(cap) + .build() + .expect("valid options"), + ); + let sub_ops: Vec> = (0..20) + .map(|_| { + let mut builder = + DiagnosticsContextBuilder::new(ActivityId::new_uuid(), Arc::clone(&options)); + for _ in 0..5 { + let h = builder.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + builder.complete_request(h, StatusCode::Conflict, None); + } + Arc::new(builder.complete()) + }) + .collect(); + + // Each sub-op is individually under the cap, so none is truncated. + assert_eq!(sub_ops[0].requested_regions().len(), 5); + assert_eq!(sub_ops[0].total_requested_regions(), 5); + + let aggregated = + DiagnosticsContext::aggregate_sub_operations(&sub_ops).expect("non-empty sources"); + + // 20 sub-ops x 5 dispatches = 100, bounded back down to the cap. + assert_eq!(aggregated.total_requested_regions(), 100); + assert_eq!(aggregated.requested_regions().len(), cap); + assert_eq!(aggregated.total_responded_regions(), 100); + assert_eq!(aggregated.responded_regions().len(), cap); + } + + #[test] + fn bound_region_history_keeps_head_and_tail() { + // The elision targets the repetitive middle, so both ends survive. + let bounded = bound_region_history((0..100).collect::>(), 10); + assert_eq!(bounded, vec![0, 1, 2, 3, 4, 95, 96, 97, 98, 99]); + + // An odd cap favors the head by one. + let bounded = bound_region_history((0..100).collect::>(), 5); + assert_eq!(bounded, vec![0, 1, 2, 98, 99]); + + // At or under the cap the input is returned verbatim. + let bounded = bound_region_history(vec![1, 2, 3], 10); + assert_eq!(bounded, vec![1, 2, 3]); + let bounded = bound_region_history(vec![1, 2, 3], 3); + assert_eq!(bounded, vec![1, 2, 3]); } #[test] From 10249e5c78ac11fac90cf9a74e85b4d9c0f0ea15 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Fri, 31 Jul 2026 11:47:21 -0700 Subject: [PATCH 12/21] Preserve hedge-loser diagnostics and true dispatch order Addresses the remaining review feedback on the hedging detection API. Two related defects made hedged operations misreport diagnostics. Each hedge leg records into a private builder created by `clone_for_hedge_attempt`, and the race cancels the loser by dropping its future -- which also dropped every attempt that leg had *already completed*. A leg that received a `429`, entered the transport pipeline's throttle-retry backoff, and only then lost the race contributed nothing at all: its region vanished from the histories and its RU charge went unbilled in `total_request_charge()`. Separately, `HedgeFanout` recorded its position as `at: parent.requests.len()`, which cannot describe where a fan-out sits relative to attempts held in a concurrently running child builder, so a primary-leg retry dispatched before the alternate fanned out was reported after it. Both share one root cause -- per-leg state that the parent cannot see -- so they get one fix: - `HedgeJournal`, an operation-scoped `Arc>` shared by the parent and every leg, allocated lazily on the first `clone_for_hedge_attempt` so the non-hedged path pays nothing. Legs mirror each attempt into it as the attempt reaches a terminal state; `merge_hedge_attempt` retains out the winner's copies, and `complete()` folds the remainder back in. Exactly one record per attempt survives regardless of which leg won. An attempt still in flight when its leg was cancelled observed no reply and is deliberately not recovered. - Ordering moves from list indices to `RequestDiagnostics::started_at`, a free operation-wide monotonic clock. `complete()` sorts the union by it, restoring true global dispatch order across both legs. - `HedgeFanout` becomes a pair of `HedgeLegDispatch` records (region, reason, leg id, launch instant) and is now only a fallback: a leg that dispatched describes itself through its own surviving attempts, so a synthetic entry is emitted solely for a leg cancelled before it ever reached the wire. That case is still real -- `select` polls the primary first, so an already-resolved primary drops the alternate without ever polling it. Documentation: - `bound_region_history` now documents that it is deliberately atomicity-oblivious, and corrects the premise that the two histories are index-paired: `requested_regions` is dispatch-ordered while `responded_regions` is arrival-ordered over the subset that replied, so they routinely differ in both length and order. - `aggregate_sub_operations` documents why bounding per sub-operation and again on the concatenation is intentional rather than a missed cap-sizing decision. - `TRANSPORT_PIPELINE_SPEC.md` no longer claims the primary leg always uses `ExecutionContext::Initial`; it now distinguishes the STAGE 2b race from a STAGE 7 retry-upgraded hedge, and describes the journal. - `HEDGING_DETECTION_API_SPEC.md` replaces the "a dropped leg cannot be lost because the fan-out is recorded up front" reasoning with the journal semantics, and states the compound-bounding rationale. Tests: - Hedging tests now build legs through the real `clone_for_hedge_attempt` /`record_hedge_fanout`/`merge_hedge_attempt` flow via new `spawn_primary_leg`/`spawn_alternate_leg` helpers, so they exercise the production path instead of hand-assembled fan-out records. - New behaviour tests cover a loser that already observed a `429`, a primary retry dispatched before fan-out, interleaved retries on both legs, the both-transient path, and a leg cancelled mid-flight. - A serde insurance test pins the deprecated `ExecutionContext::Retry` to `"retry"` so it cannot silently become an alias for `"operation_retry"` during the deprecation window. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 421e5ae3-c3da-4c9e-bc34-75feaccd0604 --- sdk/cosmos/azure_data_cosmos/CHANGELOG.md | 2 + .../docs/HEDGING_DETECTION_API_SPEC.md | 54 +- .../azure_data_cosmos_driver/CHANGELOG.md | 4 +- .../docs/TRANSPORT_PIPELINE_SPEC.md | 25 +- .../src/diagnostics/diagnostics_context.rs | 943 +++++++++++++++--- .../src/driver/pipeline/operation_pipeline.rs | 30 +- 6 files changed, 883 insertions(+), 175 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md index 5b1d9ef8a9c..8aebd84bcb9 100644 --- a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md @@ -22,6 +22,8 @@ ### Bugs Fixed +- Fixed hedged operations under-reporting diagnostics for the leg that lost the race. Cancelling the loser previously discarded every attempt it had already completed, so a leg that received a `429` and was retrying when it lost contributed nothing to the surfaced `DiagnosticsContext` — omitting its region from `regions_contacted()`/`responded_regions()` and under-reporting `total_request_charge()` against RU the account was actually billed. Attempts observed by a cancelled leg are now retained. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) + ### Other Changes - Existing `ResponseHeaders` accessors now return Gateway 2.0 backend duration, quota, item-count, and local-LSN response metadata. ([#4797](https://github.com/Azure/azure-sdk-for-rust/pull/4797)) diff --git a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md index 1981376dbde..e1c9b501e92 100644 --- a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md +++ b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md @@ -35,7 +35,7 @@ Rust-native `HedgeDiagnostics` surface (see §5); the two are complementary. | Item | Signature | Notes | | --- | --- | --- | | `DiagnosticsContext` | re-exported as `azure_data_cosmos::DiagnosticsContext` | The per-operation diagnostics handle. | -| `DiagnosticsContext::requests` | `-> Arc>` | Retained per-attempt records in dispatch order — **not** a guaranteed-complete append-only history: under a `429`/`410` retry storm the list is bounded/compacted (see `max_request_diagnostics`, which can drop or reorder entries), and a structurally-dropped hedge loser leg is absent. Cloning the `Arc` is a cheap atomic increment. | +| `DiagnosticsContext::requests` | `-> Arc>` | Retained per-attempt records in dispatch order — **not** a guaranteed-complete append-only history: under a `429`/`410` retry storm the list is bounded/compacted (see `max_request_diagnostics`, which can drop or reorder entries). A structurally-dropped hedge loser leg *is* represented for every attempt it completed (rescued through the hedge journal, §4.3), but an attempt it left in flight is absent. Cloning the `Arc` is a cheap atomic increment. | | `DiagnosticsContext::hedge_diagnostics` | `-> Option<&HedgeDiagnostics>` | An **optional retained race outcome**, not a configuration probe. `Some` when a hedge race recorded a terminal outcome (including primary-wins-under-threshold). `None` when hedging was not selected for the operation, when a configured strategy found the operation ineligible, **and** on the both-transient→failover path, where a terminal outcome is deliberately left unset so a later successful retry does not carry a misleading `BothTransient` state. | | `DiagnosticsContext::regions_contacted` | `-> Vec` | Distinct regions **deduplicated in first-contact (failover) order — not sorted**, captured from the full attempt list before compaction. | | `RequestDiagnostics::region` | `-> Option<&Region>` | `None` for pre-region-selection failures. | @@ -147,22 +147,42 @@ list plus a dispatch-time hedge fan-out log, then stored as fields — the same pattern `regions_contacted()` already used. Reading them is a field read. This matters because the retained `requests()` list is *not* the dispatch history: -- a clean hedge race structurally drops the losing leg's sub-builder before it - can be merged (see §5 and `HedgeDiagnostics`); +- a clean hedge race structurally drops the losing leg's sub-builder, so only the + attempts rescued through the hedge journal survive, and any attempt the loser + left in flight is genuinely gone (see §5 and `HedgeDiagnostics`); - a `429`/`410` retry storm compacts `requests()` down to `max_request_diagnostics`, dropping whole buckets; - `aggregate_sub_operations` keeps only **one** representative `HedgeDiagnostics` for a multi-round-trip operation. -**Hedge fan-out.** Every fan-out is recorded on the *parent* builder at dispatch -time, before the race runs, so a dropped leg cannot be lost. Both legs therefore -always appear, spliced in at the point the race was dispatched, primary before -alternate: the primary leg tagged with the reason it was **actually** dispatched -under (`Initial` for a first attempt, or the failover/session reason when a -hedge upgraded a retry), and the alternate leg tagged `Hedging`. A leg that was -merged back afterwards is absorbed by the fan-out entry it repeats, so the -winner is listed once; genuine repeat dispatches are never collapsed. A dropped -leg has **no** `responded_regions()` entry (it never produced a service reply). +**Hedge journal.** A hedge leg records into its own private sub-builder, and the +race structurally drops the loser's future — so the loser's records must be +rescued out-of-band. Every attempt a leg *completes* is mirrored, at the moment +it reaches a terminal state, into an operation-scoped hedge journal shared by the +parent and all legs; the winner's copies are discarded when its sub-builder is +merged, and `complete()` folds the remaining copies back in and stable-sorts the +union by dispatch instant. The result is exactly one record per attempt in true +global dispatch order, no matter which leg won. Consequently a dropped leg that +had already received a reply (a `429` it was backing off from, say) still +contributes its region, its status and its RU charge to `requested_regions()`, +`responded_regions()` and the charge totals. + +An attempt that was still **in flight** when its leg was cancelled observed no +reply, so it is deliberately *not* recovered — reporting it would invent a +response that never arrived. + +**Hedge fan-out.** Each fan-out is additionally recorded on the *parent* builder +at dispatch time, before the race runs. This is a fallback for the one case the +journal cannot cover: `select` polls the primary first, so a primary that is +already `Ready` causes the alternate to be dropped **without ever being polled**, +meaning it never reaches `start_request` and has no attempt of its own to +mirror. Such a leg is spliced in as a synthetic entry, positioned by its dispatch +instant, so both legs always appear: the primary leg tagged with the reason it +was **actually** dispatched under (`Initial` for a first attempt, or the +failover/session reason when a hedge upgraded a retry), and the alternate leg +tagged `Hedging`. A leg that did dispatch describes itself through its own +(surviving) attempts and is never double-counted. A leg that never produced a +service reply has no `responded_regions()` entry. For an aggregated operation (e.g. `PATCH`) stitched from multiple sub-operations, every sub-operation's fan-out is preserved: the aggregated list @@ -215,6 +235,16 @@ middle, mirroring the "first and last of each run" policy the contract already applies to attempt compaction. The head preserves the initial dispatch and any early hedge fan-out; the tail preserves where the operation finally landed. +Because the cap is applied both per sub-operation and again to the concatenated +aggregate, a long enough PATCH conflict loop is bounded **twice** — the aggregate +keeps the head and tail of a list whose own entries are already head/tail +extracts. This compounding is deliberate: it costs some middle detail that was +already elided once, and in exchange it preserves the two properties consumers +actually assert on — the operation's first dispatch and where it finally landed — +under a hard bound that holds no matter how many sub-operations run. Raising the +cap would not remove the compounding, only move the threshold at which it starts, +while making the worst-case artifact proportionally larger. + Truncation is never silent. `total_requested_regions()` / `total_responded_regions()` report the exact pre-truncation counts, so `requested_regions().len() < total_requested_regions()` detects an elision. On the span, the matching `*_total` attribute is emitted only diff --git a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md index 4cc02ff164a..ee66f7c79a5 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md @@ -7,7 +7,7 @@ - Added an SDK-generated `x-ms-client-id` header that remains stable for each `CosmosDriver`, including metadata, retry, hedge, probe, and Gateway 2.0 outer HTTP requests. ([#4844](https://github.com/Azure/azure-sdk-for-rust/pull/4844)) - Added a schema-agnostic Cosmos binary JSON codec (`binary_json`) and driver-side binary encoding via `OperationOptions.binary_encoding` (`BinaryEncodingOptions`). When enabled, the driver transcodes item request/response bodies between text and Cosmos binary JSON and negotiates the wire format; it is honored only for point `Document` item operations. Off by default and inert on the wire when unset. ([#4671](https://github.com/Azure/azure-sdk-for-rust/pull/4671)) - Added `PlanOptions` (with `DEFAULT_MAX_FAN_OUT`) to `CosmosDriver::plan_operation`, enforcing a maximum fan-out on fresh cross-partition plans. A fresh plan spanning more leaf request nodes than `PlanOptions::max_fan_out` (default 100) is rejected with the new `CosmosStatus::CLIENT_CROSS_PARTITION_FAN_OUT_EXCEEDED` (HTTP 400). The limit is enforced only at initial plan time: resuming from a continuation token skips the check, and a partition split that raises the fan-out mid-execution does not abort the operation. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855)) -- Added the Hedging Detection API on `DiagnosticsContext`: `hedging_started()` (whether a hedge arm actually fanned out), `requested_regions()` (regions dispatched to, in dispatch order with duplicates, each tagged with a reason), and `responded_regions()` (regions that produced an actual service reply, in completion order). All three are materialized at finalization from the full pre-compaction attempt list plus a dispatch-time hedge fan-out log, so a structurally-dropped hedge leg, a retry storm that compacts `requests()`, and multi-round-trip aggregation all report a complete history. Both region histories are bounded by `max_request_diagnostics` (keeping head and tail, dropping the repetitive middle) so a retry storm cannot produce an unbounded diagnostics artifact; `total_requested_regions()` / `total_responded_regions()` report the exact pre-truncation counts, so truncation is detectable rather than silent. Adds the public `RequestedRegion` struct and `RequestedRegionReason` enum (both `#[non_exhaustive]`, with a total `From` mapping) and the new `ExecutionContext::OperationRetry` variant. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Added the Hedging Detection API on `DiagnosticsContext`: `hedging_started()` (whether a hedge arm actually fanned out), `requested_regions()` (regions dispatched to, in dispatch order with duplicates, each tagged with a reason), and `responded_regions()` (regions that produced an actual service reply, in completion order). All three are materialized at finalization from the full pre-compaction attempt list — including attempts rescued from a structurally-dropped hedge leg — plus a dispatch-time hedge fan-out log, so a cancelled hedge leg, a retry storm that compacts `requests()`, and multi-round-trip aggregation all report a complete history. Both region histories are bounded by `max_request_diagnostics` (keeping head and tail, dropping the repetitive middle) so a retry storm cannot produce an unbounded diagnostics artifact; `total_requested_regions()` / `total_responded_regions()` report the exact pre-truncation counts, so truncation is detectable rather than silent. Adds the public `RequestedRegion` struct and `RequestedRegionReason` enum (both `#[non_exhaustive]`, with a total `From` mapping) and the new `ExecutionContext::OperationRetry` variant. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Added `HedgeTerminalState::as_str()` (and a matching `Display`) returning a stable, low-cardinality `snake_case` identifier for the hedging race outcome, for use as an observability attribute / log-field value. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) ### Breaking Changes @@ -19,6 +19,8 @@ - Unified `403/3` and `403/1008` topology retries under a 5-second cumulative delay budget so a persistent topology error surfaces promptly instead of hanging. Multi-write `403/3` and all `403/1008` come down from ~120 seconds of fixed 1-second retries; single-write `403/3` moves up from three immediate generic retries onto the same topology policy. The first retry is always immediate; later retries use exponential backoff with jitter. ([#4740](https://github.com/Azure/azure-sdk-for-rust/pull/4740)) - Fixed the primary leg of a hedged request being recorded with `ExecutionContext::Initial` even when the hedge was dispatched from a retry. The primary leg now carries the execution context computed from the live retry state, so a hedge that upgraded a session retry or a region failover is no longer misreported as a first attempt in diagnostics. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Fixed a hedged operation losing the diagnostics of the leg that lost the race. Each leg records into its own builder, and the race cancels the loser by dropping its future, which also dropped every attempt that leg had already completed — so a leg that received a `429` and was retrying when it lost contributed nothing to `request_count()`, `regions_contacted()`, or `total_request_charge()`, under-reporting the RU the account was actually billed. Completed attempts are now mirrored into an operation-scoped hedge journal and folded back in at finalization. An attempt still in flight when its leg was cancelled observed no reply and remains unreported. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Fixed the finalized attempt list interleaving hedge legs by leg rather than by dispatch time. Attempts from both legs plus the hedge fan-out are now ordered by their actual dispatch instant, so a primary-leg retry that was dispatched before the alternate fanned out is reported before it instead of after. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) ### Other Changes diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/TRANSPORT_PIPELINE_SPEC.md b/sdk/cosmos/azure_data_cosmos_driver/docs/TRANSPORT_PIPELINE_SPEC.md index cbbbe1e63e5..adf671d6fba 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/TRANSPORT_PIPELINE_SPEC.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/TRANSPORT_PIPELINE_SPEC.md @@ -728,12 +728,25 @@ async fn execute_hedged( (preferring recent measurements for fast reaction). Hard-coded safety gates clamp the effective threshold to the range **50 ms – 4000 ms**; configurable min/max bounds can tighten this range further. -- The primary (first) attempt always uses `ExecutionContext::Initial` so that - diagnostics can clearly distinguish whether hedging occurred. A hedged attempt - uses `ExecutionContext::Hedging`. This makes it easy to identify concurrent - execution caused by the original attempt not producing a terminal result - within the configured threshold. -- Both attempts are tracked in `DiagnosticsContext` +- The primary (first) leg of the race carries the `ExecutionContext` the + *non-hedged* path would have used for that same attempt, so the reason a + hedged operation was already retrying is never lost. Concretely: + - **STAGE 2b** (a fresh operation that crosses the latency threshold) — the + retry counters are still zero, so the primary leg is `Initial`. + - **STAGE 7** (an already-retrying operation whose next attempt is upgraded + into a hedge) — the primary leg keeps the retry's own context, e.g. + `OperationRetry` or `RegionFailover`. + + The *hedged* leg is always `ExecutionContext::Hedging`. Hedging is therefore + identified by the presence of the `Hedging` leg, not by the primary being + `Initial`, which keeps concurrent execution unambiguous without erasing why + the operation was retrying in the first place. +- Both legs of the race are tracked in `DiagnosticsContext`, including the leg + that loses and is cancelled: each leg records into a private builder, but + every attempt it *completes* is mirrored into an operation-scoped hedge + journal, so a reply already received by the loser is still reported (region + history, RU charge, status). An attempt still in flight when its leg is + cancelled observed nothing and is deliberately not reported. - The hedged attempt's RU charge is always reported regardless of which wins - **Deadline enforcement**: The hedged attempt shares the original e2e deadline. By the time the hedged attempt starts, at least `hedging_threshold` has diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs index 82c4e776861..9e5ed0b8f50 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs @@ -17,7 +17,7 @@ use azure_core::http::StatusCode; use serde::Serialize; use std::{ collections::HashMap, - sync::{Arc, OnceLock}, + sync::{Arc, Mutex, OnceLock}, time::{Duration, Instant}, }; @@ -1433,28 +1433,77 @@ impl SystemUsageSnapshot { } } +/// Operation-scoped state shared by the parent [`DiagnosticsContextBuilder`] and +/// every hedge leg cloned from it. +/// +/// A hedge race resolves by *structurally dropping* the losing future. Without a +/// shared sink the loser's private builder — including service replies it had +/// already observed, their RU charge, and their timings — disappears with it, so +/// `responded_regions()` would omit real replies and the "exact totals" +/// guarantee would silently not hold. Every leg therefore mirrors each attempt +/// here the moment that attempt reaches a terminal state, tagged with the leg's +/// id; [`DiagnosticsContextBuilder::merge_hedge_attempt`] removes the winning +/// leg's copies (the winner's originals are merged directly), so exactly one +/// copy of every attempt survives into [`DiagnosticsContextBuilder::complete`]. +#[derive(Debug, Default)] +struct HedgeJournal { + /// Next leg id to hand out. Leg ids start at 1; `0` is reserved for the + /// parent builder, which is never a race participant. + next_leg_id: u64, + /// Legs that reached [`DiagnosticsContextBuilder::start_request`] at least + /// once, so materialization can tell "this leg described itself" from "this + /// leg was dropped before it dispatched anything". + dispatched_legs: Vec, + /// Terminal attempts mirrored by each leg, tagged with the leg id. What + /// remains at finalization is exactly the set of attempts that belonged to + /// legs the race dropped. + orphaned_attempts: Vec<(u64, RequestDiagnostics)>, +} + +/// One leg of a cross-region hedge race, as described to +/// [`DiagnosticsContextBuilder::record_hedge_fanout`]. +#[derive(Clone, Debug)] +pub(crate) struct HedgeLegDispatch { + /// The region the leg was routed to. `None` when the routed endpoint + /// carries no named region (global-endpoint accounts). + pub(crate) region: Option, + /// Why the orchestrator dispatched this leg to that region. + pub(crate) reason: RequestedRegionReason, + /// The leg builder's journal id, used to detect whether the leg ever + /// dispatched a request of its own. + leg_id: u64, + /// When the leg was launched, on the same clock as + /// [`RequestDiagnostics::started_at`]. + dispatched_at: Instant, +} + +impl HedgeLegDispatch { + /// Materializes this leg as a requested-region entry, if it was routed to a + /// named region. Used only to reconstruct a leg that never dispatched. + fn requested_region(&self) -> Option { + self.region.clone().map(|region| RequestedRegion { + region, + reason: self.reason, + }) + } +} + /// One cross-region hedge fan-out, recorded on the parent builder at the moment /// the race is dispatched. /// -/// A hedge race launches two legs concurrently and then structurally drops the -/// loser's future — and with it the loser's per-attempt [`RequestDiagnostics`] — -/// so the finalized attempt list alone can never describe the fan-out. Capturing -/// both legs here, before the race runs, makes the dispatch history authoritative -/// and independent of which leg won, of retry-storm compaction, and of -/// sub-operation aggregation. -#[derive(Clone, Debug, PartialEq, Eq)] +/// Legs describe themselves through their own [`RequestDiagnostics`], which the +/// [`HedgeJournal`] keeps alive even for the leg the race drops. This record is +/// the fallback for the one case that leaves no attempt behind: a leg cancelled +/// before it dispatched anything at all — most commonly the alternate, which +/// `select` never polls when the primary is already resolved. It also marks the +/// operation as having fanned out at all, independently of which leg won, of +/// retry-storm compaction, and of sub-operation aggregation. +#[derive(Clone, Debug)] pub(crate) struct HedgeFanout { - /// Number of attempts already recorded on the parent when the race was - /// dispatched. The winning leg's merged attempts start at this index, so - /// recovery can splice a dropped leg back in dispatch order. - at: usize, - /// The primary leg's dispatched region and reason. `None` when the routed - /// endpoint carries no named region (global-endpoint accounts). - primary: Option, - /// The speculative alternate leg's dispatched region, always tagged - /// [`RequestedRegionReason::Hedging`]. `None` when the routed endpoint - /// carries no named region. - alternate: Option, + /// The primary leg of the race. + primary: HedgeLegDispatch, + /// The speculative alternate leg of the race. + alternate: HedgeLegDispatch, } /// Internal mutable builder for constructing a [`DiagnosticsContext`]. @@ -1501,11 +1550,24 @@ pub(crate) struct DiagnosticsContextBuilder { /// Every hedge fan-out dispatched by this operation, in dispatch order. /// /// Recorded by the hedging orchestrator on the *parent* builder before the - /// race starts, so a structurally-dropped losing leg is still described. - /// Empty for the overwhelming majority of operations (no hedging), which - /// keeps the common path allocation-free. + /// race starts, so a leg dropped before it dispatched anything is still + /// described. Empty for the overwhelming majority of operations (no + /// hedging), which keeps the common path allocation-free. hedge_fanouts: Vec, + /// Operation-scoped state shared with every hedge leg cloned from this + /// builder, or `None` when the operation never hedged. + /// + /// Allocated lazily by the first + /// [`clone_for_hedge_attempt`](Self::clone_for_hedge_attempt), so the + /// non-hedged path — the overwhelming majority of operations — never pays + /// for the `Arc`/`Mutex`. + hedge_journal: Option>>, + + /// This builder's id within [`Self::hedge_journal`]. `0` on the parent + /// builder, which never races. + hedge_leg_id: u64, + /// Test-only override for system usage snapshot, bypassing the CPU monitor. #[cfg(test)] test_system_usage: Option, @@ -1526,6 +1588,8 @@ impl DiagnosticsContextBuilder { fault_injection_enabled: false, hedge_diagnostics: None, hedge_fanouts: Vec::new(), + hedge_journal: None, + hedge_leg_id: 0, #[cfg(test)] test_system_usage: None, } @@ -1548,30 +1612,41 @@ impl DiagnosticsContextBuilder { /// Records a cross-region hedge fan-out at the moment the race is dispatched. /// - /// Called by the hedging orchestrator on the *parent* builder before the two - /// legs are launched, so the dispatch history survives the race dropping the - /// loser's builder. Each leg is `None` when its routed endpoint carries no - /// named region (global-endpoint accounts), in which case that leg simply - /// contributes nothing to + /// Called by the hedging orchestrator on the *parent* builder once both leg + /// builders exist. Each leg normally describes itself through its own + /// attempts — the [`HedgeJournal`] keeps those alive even for the leg the + /// race drops — so this record only materializes a + /// [`RequestedRegion`](crate::diagnostics::RequestedRegion) for a leg that + /// was cancelled before it dispatched anything. Each leg's `region` is + /// `None` when its routed endpoint carries no named region (global-endpoint + /// accounts), in which case that leg contributes nothing to /// [`requested_regions`](DiagnosticsContext::requested_regions) — matching /// how a region-less attempt is skipped. pub(crate) fn record_hedge_fanout( &mut self, - primary_region: Option, - primary_reason: RequestedRegionReason, - alternate_region: Option, + primary: HedgeLegDispatch, + alternate: HedgeLegDispatch, ) { - self.hedge_fanouts.push(HedgeFanout { - at: self.requests.len(), - primary: primary_region.map(|region| RequestedRegion { - region, - reason: primary_reason, - }), - alternate: alternate_region.map(|region| RequestedRegion { - region, - reason: RequestedRegionReason::Hedging, - }), - }); + self.hedge_fanouts.push(HedgeFanout { primary, alternate }); + } + + /// Describes this (leg) builder as one side of a hedge race, for + /// [`record_hedge_fanout`](Self::record_hedge_fanout). + /// + /// The leg's `started_at` is its launch instant — `clone_for_hedge_attempt` + /// stamps it fresh — so it sits on the same clock as every + /// [`RequestDiagnostics::started_at`] and orders correctly against them. + pub(crate) fn leg_dispatch( + &self, + region: Option, + reason: RequestedRegionReason, + ) -> HedgeLegDispatch { + HedgeLegDispatch { + region, + reason, + leg_id: self.hedge_leg_id, + dispatched_at: self.started_at, + } } /// Creates a fresh builder for a single hedge attempt. @@ -1581,8 +1656,18 @@ impl DiagnosticsContextBuilder { /// but starts with an empty request list and a fresh `started_at`, so /// per-attempt durations measure from launch rather than from /// operation start. The winning attempt's requests are merged back - /// via [`merge_hedge_attempt`](Self::merge_hedge_attempt). - pub(crate) fn clone_for_hedge_attempt(&self) -> Self { + /// via [`merge_hedge_attempt`](Self::merge_hedge_attempt); the losing + /// attempt's are recovered from the shared [`HedgeJournal`], which is + /// allocated here on the first call. + pub(crate) fn clone_for_hedge_attempt(&mut self) -> Self { + let journal = self + .hedge_journal + .get_or_insert_with(|| Arc::new(Mutex::new(HedgeJournal::default()))); + let leg_id = { + let mut guard = journal.lock().unwrap_or_else(|e| e.into_inner()); + guard.next_leg_id += 1; + guard.next_leg_id + }; Self { activity_id: self.activity_id.clone(), started_at: Instant::now(), @@ -1598,6 +1683,8 @@ impl DiagnosticsContextBuilder { // builder is either merged back (winner) or dropped (loser), so // recording here would be lost exactly when it matters. hedge_fanouts: Vec::new(), + hedge_journal: Some(Arc::clone(journal)), + hedge_leg_id: leg_id, #[cfg(test)] test_system_usage: self.test_system_usage.clone(), } @@ -1609,7 +1696,17 @@ impl DiagnosticsContextBuilder { /// Only the request list is moved; the attempt's `status` and /// `hedge_diagnostics` are discarded because those operation-level /// fields are written directly on the parent. + /// + /// The merged leg's mirrored copies are dropped from the shared + /// [`HedgeJournal`] at the same time: the originals are now owned by the + /// parent, so leaving the mirror in place would double-count the winner. pub(crate) fn merge_hedge_attempt(&mut self, attempt: Self) { + if let Some(journal) = self.hedge_journal.as_ref() { + let mut guard = journal.lock().unwrap_or_else(|e| e.into_inner()); + guard + .orphaned_attempts + .retain(|(leg_id, _)| *leg_id != attempt.hedge_leg_id); + } if self.requests.is_empty() { self.requests = attempt.requests; } else { @@ -1681,9 +1778,55 @@ impl DiagnosticsContextBuilder { request.with_activity_id(self.activity_id.clone()); let handle = RequestHandle(self.requests.len()); self.requests.push(request); + self.journal_dispatch(); handle } + /// Records that this hedge leg has dispatched at least one request. + /// + /// Lets materialization tell a leg that described itself through its own + /// attempts from one the race cancelled before it dispatched anything — + /// only the latter needs a synthetic entry from the fan-out record. + fn journal_dispatch(&mut self) { + if self.hedge_leg_id == 0 { + return; + } + let Some(journal) = self.hedge_journal.as_ref() else { + return; + }; + let mut guard = journal.lock().unwrap_or_else(|e| e.into_inner()); + if !guard.dispatched_legs.contains(&self.hedge_leg_id) { + guard.dispatched_legs.push(self.hedge_leg_id); + } + } + + /// Mirrors a now-terminal attempt into the shared [`HedgeJournal`]. + /// + /// Attempts are immutable once terminal (`update_request` rejects a + /// completed handle), so the mirrored copy is final. If this leg goes on to + /// win the race its copies are dropped again by + /// [`merge_hedge_attempt`](Self::merge_hedge_attempt); if the race drops + /// this leg instead, the copy is the only surviving record of a reply the + /// service really sent. + /// + /// An attempt still in flight when the leg is cancelled is genuinely + /// unobserved — no reply arrived — and is intentionally not recovered. + fn journal_terminal_attempt(&mut self, handle: RequestHandle) { + if self.hedge_leg_id == 0 { + return; + } + let Some(journal) = self.hedge_journal.as_ref() else { + return; + }; + let Some(request) = self.requests.get(handle.0) else { + return; + }; + let mut guard = journal.lock().unwrap_or_else(|e| e.into_inner()); + guard + .orphaned_attempts + .push((self.hedge_leg_id, request.clone())); + } + /// Records the response headers and completion of a request in one shot. /// /// Convenience wrapper around [`update_request`](Self::update_request) + @@ -1738,6 +1881,7 @@ impl DiagnosticsContextBuilder { if let Some(request) = self.requests.get_mut(handle.0) { request.complete(status_code, sub_status); } + self.journal_terminal_attempt(handle); } /// Records end-to-end timeout of a request. @@ -1753,6 +1897,7 @@ impl DiagnosticsContextBuilder { if let Some(request) = self.requests.get_mut(handle.0) { request.timeout(); } + self.journal_terminal_attempt(handle); } /// Records a transport-level failure for a request that received no Cosmos response. @@ -1766,6 +1911,7 @@ impl DiagnosticsContextBuilder { if let Some(request) = self.requests.get_mut(handle.0) { request.fail_transport(error, request_sent, status); } + self.journal_terminal_attempt(handle); } /// Updates a request's diagnostics with additional data. @@ -1842,9 +1988,38 @@ impl DiagnosticsContextBuilder { /// This consumes the builder and creates a finalized diagnostics context /// with all data frozen. The `DiagnosticsContext` can then be safely /// shared via `Arc` without any locking overhead. - pub(crate) fn complete(self) -> DiagnosticsContext { + pub(crate) fn complete(mut self) -> DiagnosticsContext { let duration = self.started_at.elapsed(); + // Recover the attempts of any hedge leg the race dropped before it + // could be merged. Those legs may have observed real service replies + // (a 429 answered just before the partner won, say); losing them would + // under-report charge, drop a genuine entry from `responded_regions`, + // and break the exactness of the `total_*_regions` counters. + // + // `started_at` is a single operation-wide clock, and every builder + // appends in start order, so re-sorting the union restores true global + // dispatch order across the parent and both legs — something no + // parent-relative index could express while the legs ran concurrently. + // Handles are never used after `complete` consumes the builder, so + // reordering here cannot invalidate one. + let (orphaned, dispatched_legs) = self + .hedge_journal + .take() + .map(|journal| { + let mut guard = journal.lock().unwrap_or_else(|e| e.into_inner()); + ( + std::mem::take(&mut guard.orphaned_attempts), + std::mem::take(&mut guard.dispatched_legs), + ) + }) + .unwrap_or_default(); + if !orphaned.is_empty() { + self.requests + .extend(orphaned.into_iter().map(|(_, request)| request)); + self.requests.sort_by_key(RequestDiagnostics::started_at); + } + // Exact operation-level total charge, summed from the FULL attempt list // before any compaction so it stays exact even when the retained list is // bounded under a retry storm. @@ -1866,7 +2041,8 @@ impl DiagnosticsContextBuilder { // limit (DIAGNOSTICS-CONTRACT.md §8). The pre-truncation lengths are // retained so the truncation is explicit rather than silent. let cap = self.options.max_request_diagnostics(); - let requested_regions = requested_regions_from(&self.requests, &self.hedge_fanouts); + let requested_regions = + requested_regions_from(&self.requests, &self.hedge_fanouts, &dispatched_legs); let responded_regions = responded_regions_from(&self.requests); let total_requested_regions = requested_regions.len(); let total_responded_regions = responded_regions.len(); @@ -2182,7 +2358,7 @@ impl DiagnosticsContext { .sum::(), ); let regions_contacted = ordered_unique_regions(&requests); - let requested_regions = requested_regions_from(&requests, &[]); + let requested_regions = requested_regions_from(&requests, &[], &[]); let responded_regions = responded_regions_from(&requests); // The helper builds a fixed, hand-supplied attempt list, so the history // is never over the cap and the totals are simply its length. @@ -2245,24 +2421,58 @@ impl DiagnosticsContext { let not_sentinel = |region: &Region| { (region.as_str() != HedgeDiagnostics::UNKNOWN_REGION_SENTINEL).then(|| region.clone()) }; + // Both legs are stamped at (or before) the first supplied attempt, so a + // leg that has to be reconstructed lands at the head of the history — + // the same position the pipeline would produce for a race dispatched + // before any attempt was recorded. + let dispatched_at = requests + .iter() + .map(RequestDiagnostics::started_at) + .min() + .unwrap_or_else(Instant::now); + const PRIMARY_LEG: u64 = 1; + const ALTERNATE_LEG: u64 = 2; let fanouts: Vec = hedge_diagnostics .as_ref() .and_then(|hedge| { let alternate = hedge.alternate_region()?; Some(HedgeFanout { - at: 0, - primary: not_sentinel(hedge.primary_region()).map(|region| RequestedRegion { - region, + primary: HedgeLegDispatch { + region: not_sentinel(hedge.primary_region()), reason: RequestedRegionReason::Initial, - }), - alternate: not_sentinel(alternate).map(|region| RequestedRegion { - region, + leg_id: PRIMARY_LEG, + dispatched_at, + }, + alternate: HedgeLegDispatch { + region: not_sentinel(alternate), reason: RequestedRegionReason::Hedging, - }), + leg_id: ALTERNATE_LEG, + dispatched_at, + }, }) }) .into_iter() .collect(); + // A leg "dispatched" when the supplied attempt list already describes + // it; only a leg with no attempt of its own needs reconstructing. + let dispatched_legs: Vec = fanouts + .first() + .map(|fanout| { + [&fanout.primary, &fanout.alternate] + .into_iter() + .filter(|leg| { + leg.region.as_ref().is_some_and(|region| { + requests.iter().any(|request| { + request.region() == Some(region) + && RequestedRegionReason::from(request.execution_context()) + == leg.reason + }) + }) + }) + .map(|leg| leg.leg_id) + .collect() + }) + .unwrap_or_default(); let mut context = Self::for_testing_with_requests( activity_id, @@ -2272,7 +2482,9 @@ impl DiagnosticsContext { requests, ); if !fanouts.is_empty() { - context.requested_regions = requested_regions_from(&context.requests, &fanouts); + context.requested_regions = + requested_regions_from(&context.requests, &fanouts, &dispatched_legs); + context.total_requested_regions = context.requested_regions.len(); context.hedging_started = true; } context.hedge_diagnostics = hedge_diagnostics; @@ -2395,6 +2607,27 @@ impl DiagnosticsContext { // when each source is individually bounded. Re-bound the result under // the same cap and carry the summed pre-truncation totals, so the // aggregate honours the same guarantee as a single operation. + // + // Bounding the concatenation of already-bounded sources compresses the + // middle twice: a sub-op in the middle can have had its own middle + // elided, and then be dropped wholesale here. That is intentional, not + // an oversight, and a proportionally larger cap is deliberately *not* + // used: + // + // - The contract is a flat "the finalized artifact holds at most `cap` + // region entries", the same promise a single operation makes. Scaling + // the cap with sub-op count would make an aggregate's size grow with + // the length of a retry loop — exactly the unbounded growth the bound + // exists to prevent. + // - Nothing exact is lost to it: `total_requested_regions` and + // `total_responded_regions` are summed from the sources' + // pre-truncation totals, `total_request_charge` and `request_count()` + // stay exact via the compaction marker, and `hedging_started` is a + // disjunction over the sources, so a fan-out elided from the ordered + // view is still reported. + // - The interesting entries in a conflict loop are the head (how the + // operation started) and the tail (where it finally landed), which + // head/tail bounding keeps verbatim by construction. let total_requested_regions: usize = sources.iter().map(|c| c.total_requested_regions).sum(); let total_responded_regions: usize = @@ -3101,6 +3334,32 @@ fn ordered_unique_regions(requests: &[RequestDiagnostics]) -> Vec { /// operation finally landed) are kept verbatim, mirroring the "first + last of /// each run" policy `compact_requests` already uses. /// +/// # Deliberately atomicity-oblivious +/// +/// This helper takes no stride or grouping parameter, and none is planned. A +/// hedge fan-out contributes two adjacent entries that straddle the head/tail +/// boundary at exactly `cap`; one leg is kept and the other drained, so the +/// bounded view can show a "half" race. That is accepted, because: +/// +/// - It only occurs on an operation whose history already exceeds `cap`, i.e. +/// one that has been failing over or storming long enough that a single race +/// in the middle is not the interesting signal. +/// - Nothing exact is lost. +/// [`total_requested_regions`](DiagnosticsContext::total_requested_regions) +/// and [`total_responded_regions`](DiagnosticsContext::total_responded_regions) +/// are computed pre-truncation and stay exact, and +/// [`hedging_started`](DiagnosticsContext::hedging_started) is derived from +/// the fan-out records, not from this list — so "did this operation hedge?" +/// is still answered correctly even if both legs were elided. +/// - Keeping pairs intact would make the output length depend on where the +/// boundary falls, breaking the flat `<= cap` bound this exists to provide. +/// +/// The two histories are also *not* index-paired with each other: +/// `requested_regions` is dispatch-ordered and `responded_regions` is +/// arrival-ordered over the subset that actually replied, so they routinely +/// differ in both length and order. Bounding them independently therefore +/// cannot break a correspondence that never existed. +/// /// Truncation is never silent: the caller records the pre-truncation length, /// which [`DiagnosticsContext::total_requested_regions`] and /// [`DiagnosticsContext::total_responded_regions`] expose. @@ -3115,68 +3374,62 @@ fn bound_region_history(mut history: Vec, cap: usize) -> Vec { } /// Builds the dispatch-ordered requested-region history from the **full** -/// (pre-compaction) attempt list, splicing in each hedge fan-out at the point it -/// was dispatched. +/// (pre-compaction) attempt list. /// -/// A fan-out's two legs are recorded on the parent before the race runs, while -/// each surviving leg's own attempts are merged back afterwards. Splicing at -/// `HedgeFanout::at` therefore restores true dispatch order. +/// `requests` is already in global dispatch order — every builder appends in +/// start order and [`DiagnosticsContextBuilder::complete`] re-sorts the union of +/// the parent's attempts and any recovered hedge-leg attempts by `started_at` — +/// so attempts are emitted verbatim in the order given. /// -/// A merged-back leg repeats an attempt the fan-out already described, so each -/// fan-out entry is held in a small "pending" matcher that absorbs at most one -/// merged attempt apiece. Genuine repeat dispatches — a retry storm hammering -/// one region, say — are never collapsed, matching the documented contract that -/// duplicates are meaningful. +/// A hedge leg normally describes itself through its own attempts, including +/// when the race drops it (the [`HedgeJournal`] preserves them). The one leg +/// that leaves nothing behind is one cancelled before it dispatched anything — +/// most commonly the alternate, which `select` never polls when the primary is +/// already resolved. Only such a leg gets a synthetic entry here, positioned by +/// its dispatch instant, which is why `dispatched_legs` is required: it is the +/// set of legs that reached `start_request` at least once. fn requested_regions_from( requests: &[RequestDiagnostics], fanouts: &[HedgeFanout], + dispatched_legs: &[u64], ) -> Vec { - let mut regions: Vec = Vec::new(); - let mut next_fanout = 0usize; - // Fan-out legs awaiting the merged attempt that repeats them. Holds at most - // two entries (one race's legs) and is reset when the next race is spliced. - let mut pending: Vec = Vec::new(); - - for (index, request) in requests.iter().enumerate() { - // Emit every fan-out dispatched at or before this attempt first, so the - // race's two legs precede the winner's merged attempts. - while let Some(fanout) = fanouts.get(next_fanout) { - if fanout.at > index { + // Legs that never dispatched anything, in dispatch order. Nothing to + // reconstruct on the overwhelmingly common non-hedged path. + let mut silent_legs: Vec<&HedgeLegDispatch> = if fanouts.is_empty() { + Vec::new() + } else { + fanouts + .iter() + .flat_map(|fanout| [&fanout.primary, &fanout.alternate]) + .filter(|leg| leg.region.is_some() && !dispatched_legs.contains(&leg.leg_id)) + .collect() + }; + silent_legs.sort_by_key(|leg| leg.dispatched_at); + + let mut regions: Vec = Vec::with_capacity(requests.len()); + let mut next_silent_leg = 0usize; + + for request in requests { + // Emit any never-dispatched leg launched at or before this attempt, so + // the reconstructed entry lands in true dispatch order. + while let Some(leg) = silent_legs.get(next_silent_leg) { + if leg.dispatched_at > request.started_at() { break; } - let legs = fanout.primary.iter().chain(fanout.alternate.iter()); - pending.clear(); - pending.extend(legs.clone().cloned()); - regions.extend(legs.cloned()); - next_fanout += 1; + regions.extend(leg.requested_region()); + next_silent_leg += 1; } if let Some(region) = request.region() { - let candidate = RequestedRegion { + regions.push(RequestedRegion { region: region.clone(), reason: RequestedRegionReason::from(request.execution_context()), - }; - // A merged hedge leg repeats the entry its fan-out already recorded; - // consume the pending match instead of listing the leg twice. - match pending.iter().position(|pending| *pending == candidate) { - Some(matched) => { - pending.swap_remove(matched); - } - None => regions.push(candidate), - } + }); } } - // Fan-outs dispatched after the last recorded attempt (e.g. both legs failed - // before recording anything) still belong in the history. - for fanout in &fanouts[next_fanout.min(fanouts.len())..] { - regions.extend( - fanout - .primary - .iter() - .chain(fanout.alternate.iter()) - .cloned(), - ); + for leg in &silent_legs[next_silent_leg.min(silent_legs.len())..] { + regions.extend(leg.requested_region()); } regions @@ -4609,24 +4862,61 @@ mod tests { ) } + /// Mirrors the orchestrator's STAGE 1: spawn the primary leg's builder and + /// capture the dispatch record the parent keeps for it. + /// + /// Returned separately from [`spawn_alternate_leg`] so a test can dispatch + /// on the primary *between* the two, exactly as the pipeline does — the + /// alternate is only built once the hedge threshold has elapsed. + fn spawn_primary_leg( + parent: &mut DiagnosticsContextBuilder, + region: Option, + reason: RequestedRegionReason, + ) -> (DiagnosticsContextBuilder, HedgeLegDispatch) { + let leg = parent.clone_for_hedge_attempt(); + let dispatch = leg.leg_dispatch(region, reason); + (leg, dispatch) + } + + /// Mirrors the orchestrator's STAGE 3: spawn the alternate leg's builder + /// once the threshold has elapsed and record the fan-out on the parent. + fn spawn_alternate_leg( + parent: &mut DiagnosticsContextBuilder, + primary: HedgeLegDispatch, + region: Option, + ) -> DiagnosticsContextBuilder { + let leg = parent.clone_for_hedge_attempt(); + let dispatch = leg.leg_dispatch(region, RequestedRegionReason::Hedging); + parent.record_hedge_fanout(primary, dispatch); + leg + } + #[test] fn requested_regions_keeps_dropped_hedge_leg_on_primary_win() { - // PrimaryWonAfterHedge: the alternate leg is structurally dropped, so - // `requests` holds only the primary (Initial) record. Both legs must + // PrimaryWonAfterHedge: the alternate leg is cancelled before it + // dispatches anything, so it leaves no attempt behind. Both legs must // still appear, in dispatch order, from the fan-out the orchestrator - // recorded on the parent before the race. + // recorded on the parent. let ctx = make_context_with(ActivityId::new_uuid(), |builder| { - builder.record_hedge_fanout( + let (mut primary, primary_dispatch) = spawn_primary_leg( + builder, Some(Region::EAST_US_2), RequestedRegionReason::Initial, - Some(Region::WEST_US_2), ); - let h = builder.start_test_request( + let h = primary.start_test_request( ExecutionContext::Initial, Some(Region::EAST_US_2), "https://test.eastus2.documents.azure.com", ); - builder.complete_request(h, StatusCode::Ok, None); + primary.complete_request(h, StatusCode::Ok, None); + // Threshold elapsed, so the alternate launched — and was then + // dropped, unpolled, when the primary won the race. + drop(spawn_alternate_leg( + builder, + primary_dispatch, + Some(Region::WEST_US_2), + )); + builder.merge_hedge_attempt(primary); builder.set_hedge_diagnostics(HedgeDiagnostics::primary_won_after_hedge( hedge_config(), Region::EAST_US_2, @@ -4655,21 +4945,25 @@ mod tests { #[test] fn requested_regions_keeps_dropped_primary_leg_on_alternate_win() { - // AlternateWon: the primary leg is structurally dropped, so `requests` - // holds only the alternate (Hedging) record. The Initial primary must - // still be listed first (dispatch order). + // AlternateWon while the primary was still connecting: the primary leg + // is dropped without having dispatched, so the Initial primary must + // still be listed first (dispatch order) from the fan-out record. let ctx = make_context_with(ActivityId::new_uuid(), |builder| { - builder.record_hedge_fanout( + let (primary, primary_dispatch) = spawn_primary_leg( + builder, Some(Region::EAST_US_2), RequestedRegionReason::Initial, - Some(Region::WEST_US_2), ); - let h = builder.start_test_request( + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let h = alternate.start_test_request( ExecutionContext::Hedging, Some(Region::WEST_US_2), "https://test.westus2.documents.azure.com", ); - builder.complete_request(h, StatusCode::Ok, None); + alternate.complete_request(h, StatusCode::Ok, None); + drop(primary); + builder.merge_hedge_attempt(alternate); builder.set_hedge_diagnostics(HedgeDiagnostics::hedge_won( hedge_config(), Region::EAST_US_2, @@ -4707,17 +5001,23 @@ mod tests { "https://test.eastus2.documents.azure.com", ); builder.complete_request(h, StatusCode::ServiceUnavailable, None); - builder.record_hedge_fanout( + let (mut primary, primary_dispatch) = spawn_primary_leg( + builder, Some(Region::WEST_US_2), RequestedRegionReason::RegionFailover, - Some(Region::CENTRAL_US), ); - let h = builder.start_test_request( + let h = primary.start_test_request( ExecutionContext::RegionFailover, Some(Region::WEST_US_2), "https://test.westus2.documents.azure.com", ); - builder.complete_request(h, StatusCode::Ok, None); + primary.complete_request(h, StatusCode::Ok, None); + drop(spawn_alternate_leg( + builder, + primary_dispatch, + Some(Region::CENTRAL_US), + )); + builder.merge_hedge_attempt(primary); }); assert!(ctx.hedging_started()); @@ -4763,17 +5063,21 @@ mod tests { ); builder.complete_request(h, StatusCode::TooManyRequests, None); } - builder.record_hedge_fanout( + let (primary, primary_dispatch) = spawn_primary_leg( + &mut builder, Some(Region::EAST_US_2), RequestedRegionReason::OperationRetry, - Some(Region::WEST_US_2), ); - let h = builder.start_test_request( + let mut alternate = + spawn_alternate_leg(&mut builder, primary_dispatch, Some(Region::WEST_US_2)); + let h = alternate.start_test_request( ExecutionContext::Hedging, Some(Region::WEST_US_2), "https://test.westus2.documents.azure.com", ); - builder.complete_request(h, StatusCode::Ok, None); + alternate.complete_request(h, StatusCode::Ok, None); + drop(primary); + builder.merge_hedge_attempt(alternate); let ctx = builder.complete(); // The retained attempt list really was bounded... @@ -4781,9 +5085,8 @@ mod tests { assert!(ctx.compaction().is_some()); // ...and so is the dispatch history, independently of attempt count // (DIAGNOSTICS-CONTRACT.md §8) — but the elision is explicit, not - // silent: the exact count is still reported. 40 retries + both fan-out - // legs (the winning alternate's merged attempt is absorbed by the - // fan-out entry it repeats). + // silent: the exact count is still reported. 40 retries + the hedge's + // reconstructed primary leg + the alternate's own attempt. assert_eq!(ctx.total_requested_regions(), 42); assert_eq!(ctx.requested_regions().len(), cap); assert_eq!(ctx.total_responded_regions(), 41); @@ -4917,9 +5220,9 @@ mod tests { #[test] fn requested_regions_preserves_repeat_dispatches_around_a_fanout() { - // Duplicates are meaningful: the fan-out matcher absorbs only the merged - // leg that repeats it, never a genuine repeat dispatch to the same - // region under the same reason. + // Duplicates are meaningful: each leg's own attempts are listed as they + // were dispatched, and a genuine repeat dispatch to the same region + // under the same reason is never collapsed into a neighbor. let ctx = make_context_with(ActivityId::new_uuid(), |builder| { for _ in 0..2 { let h = builder.start_test_request( @@ -4929,25 +5232,29 @@ mod tests { ); builder.complete_request(h, StatusCode::TooManyRequests, None); } - builder.record_hedge_fanout( + // Both legs dispatch and are harvested back into the parent, so + // both describe themselves and neither is reconstructed... + let (mut primary, primary_dispatch) = spawn_primary_leg( + builder, Some(Region::EAST_US_2), RequestedRegionReason::OperationRetry, - Some(Region::WEST_US_2), ); - // Both legs get harvested back into the parent, so both fan-out - // entries are absorbed exactly once... - let primary = builder.start_test_request( + let h = primary.start_test_request( ExecutionContext::OperationRetry, Some(Region::EAST_US_2), "https://test.eastus2.documents.azure.com", ); - builder.complete_request(primary, StatusCode::Ok, None); - let alternate = builder.start_test_request( + primary.complete_request(h, StatusCode::Ok, None); + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let h = alternate.start_test_request( ExecutionContext::Hedging, Some(Region::WEST_US_2), "https://test.westus2.documents.azure.com", ); - builder.complete_request(alternate, StatusCode::Ok, None); + alternate.complete_request(h, StatusCode::Ok, None); + builder.merge_hedge_attempt(primary); + builder.merge_hedge_attempt(alternate); // ...and a later retry to the same region is still its own entry. let late = builder.start_test_request( ExecutionContext::OperationRetry, @@ -4990,7 +5297,10 @@ mod tests { // fan-out there contributes no requested-region entries, but still // counts as a fan-out. let ctx = make_context_with(ActivityId::new_uuid(), |builder| { - builder.record_hedge_fanout(None, RequestedRegionReason::Initial, None); + let (primary, primary_dispatch) = + spawn_primary_leg(builder, None, RequestedRegionReason::Initial); + drop(spawn_alternate_leg(builder, primary_dispatch, None)); + drop(primary); }); assert!(ctx.hedging_started()); @@ -5025,6 +5335,339 @@ mod tests { ); } + #[test] + fn hedge_loser_keeps_the_service_reply_it_already_observed() { + // The losing leg is not always empty. A leg that gets a 429, enters the + // transport pipeline's throttle-retry backoff, and only then loses the + // race has *already observed a real service reply*. Dropping its + // builder with the race would erase that reply, under-reporting charge + // and omitting a genuine entry from responded_regions. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (mut primary, primary_dispatch) = spawn_primary_leg( + builder, + Some(Region::EAST_US_2), + RequestedRegionReason::Initial, + ); + // Primary is throttled and sleeps before its next attempt. + let throttled = primary.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.update_request(throttled, |req| { + req.request_charge = RequestCharge::new(1.5) + }); + primary.complete_request(throttled, StatusCode::TooManyRequests, None); + + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let won = alternate.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.update_request(won, |req| req.request_charge = RequestCharge::new(2.5)); + alternate.complete_request(won, StatusCode::Ok, None); + + // The alternate wins; `select` drops the primary mid-backoff. + drop(primary); + builder.merge_hedge_attempt(alternate); + builder.set_hedge_diagnostics(HedgeDiagnostics::hedge_won( + hedge_config(), + Region::EAST_US_2, + Region::WEST_US_2, + )); + }); + + // Both attempts survive, in true dispatch order. + assert_eq!(ctx.request_count(), 2); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: RequestedRegionReason::Initial, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: RequestedRegionReason::Hedging, + }, + ] + ); + // The dropped leg's 429 is a real reply from East US 2, so it belongs + // in the arrival-ordered responded history — and both totals are exact. + assert_eq!( + ctx.responded_regions(), + vec![&Region::EAST_US_2, &Region::WEST_US_2] + ); + assert_eq!(ctx.total_requested_regions(), 2); + assert_eq!(ctx.total_responded_regions(), 2); + assert_eq!( + ctx.regions_contacted(), + vec![Region::EAST_US_2, Region::WEST_US_2] + ); + // The throttled attempt's RU is billed even though its leg lost. + assert_eq!(ctx.total_request_charge(), RequestCharge::new(4.0)); + } + + #[test] + fn requested_regions_orders_primary_retry_before_the_alternate() { + // A hedge leg can retry *before* the threshold elapses, so the alternate + // is dispatched after the primary's second attempt. The history must + // report that true order, not group both legs' fan-out ahead of the + // primary's own attempts. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (mut primary, primary_dispatch) = spawn_primary_leg( + builder, + Some(Region::EAST_US_2), + RequestedRegionReason::Initial, + ); + let first = primary.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(first, StatusCode::TooManyRequests, None); + let retry = primary.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(retry, StatusCode::Ok, None); + + // Threshold elapses only now, so the alternate is genuinely last. + drop(spawn_alternate_leg( + builder, + primary_dispatch, + Some(Region::WEST_US_2), + )); + builder.merge_hedge_attempt(primary); + }); + + assert!(ctx.hedging_started()); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: RequestedRegionReason::Initial, + }, + RequestedRegion { + region: Region::EAST_US_2, + reason: RequestedRegionReason::OperationRetry, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: RequestedRegionReason::Hedging, + }, + ] + ); + } + + #[test] + fn interleaved_leg_retries_report_true_global_dispatch_order() { + // Both legs retry concurrently and the loser is dropped. The finalized + // list must interleave the two legs by real dispatch time rather than + // concatenating one leg after the other. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (mut primary, primary_dispatch) = spawn_primary_leg( + builder, + Some(Region::EAST_US_2), + RequestedRegionReason::Initial, + ); + let p1 = primary.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(p1, StatusCode::TooManyRequests, None); + + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let a1 = alternate.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.complete_request(a1, StatusCode::TooManyRequests, None); + + let p2 = primary.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(p2, StatusCode::TooManyRequests, None); + + let a2 = alternate.start_test_request( + ExecutionContext::OperationRetry, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.complete_request(a2, StatusCode::Ok, None); + + drop(primary); + builder.merge_hedge_attempt(alternate); + }); + + assert_eq!(ctx.request_count(), 4); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: RequestedRegionReason::Initial, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: RequestedRegionReason::Hedging, + }, + RequestedRegion { + region: Region::EAST_US_2, + reason: RequestedRegionReason::OperationRetry, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: RequestedRegionReason::OperationRetry, + }, + ] + ); + assert_eq!(ctx.total_responded_regions(), 4); + } + + #[test] + fn both_transient_legs_keep_every_observed_reply() { + // BothTransient: neither leg wins, so neither is merged and the parent + // continues into the failover loop. Everything both legs observed must + // still reach the finalized context. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (mut primary, primary_dispatch) = spawn_primary_leg( + builder, + Some(Region::EAST_US_2), + RequestedRegionReason::Initial, + ); + let p = primary.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + primary.complete_request(p, StatusCode::ServiceUnavailable, None); + + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let a = alternate.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.complete_request(a, StatusCode::ServiceUnavailable, None); + + // Both transient: the race returns to the failover loop and neither + // leg builder is merged. + drop(primary); + drop(alternate); + + let failover = builder.start_test_request( + ExecutionContext::RegionFailover, + Some(Region::CENTRAL_US), + "https://test.centralus.documents.azure.com", + ); + builder.complete_request(failover, StatusCode::Ok, None); + }); + + assert_eq!(ctx.request_count(), 3); + assert_eq!( + ctx.requested_regions(), + vec![ + RequestedRegion { + region: Region::EAST_US_2, + reason: RequestedRegionReason::Initial, + }, + RequestedRegion { + region: Region::WEST_US_2, + reason: RequestedRegionReason::Hedging, + }, + RequestedRegion { + region: Region::CENTRAL_US, + reason: RequestedRegionReason::RegionFailover, + }, + ] + ); + assert_eq!( + ctx.responded_regions(), + vec![&Region::EAST_US_2, &Region::WEST_US_2, &Region::CENTRAL_US] + ); + assert_eq!(ctx.total_responded_regions(), 3); + } + + #[test] + fn hedge_leg_dropped_before_completing_is_not_recovered() { + // Only *observed* completions are recovered. An attempt still in flight + // when its leg is cancelled saw no reply, so it is intentionally absent + // from both histories — reporting it would invent a response. + let ctx = make_context_with(ActivityId::new_uuid(), |builder| { + let (mut primary, primary_dispatch) = spawn_primary_leg( + builder, + Some(Region::EAST_US_2), + RequestedRegionReason::Initial, + ); + // Dispatched, never completed. + let _in_flight = primary.start_test_request( + ExecutionContext::Initial, + Some(Region::EAST_US_2), + "https://test.eastus2.documents.azure.com", + ); + + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let won = alternate.start_test_request( + ExecutionContext::Hedging, + Some(Region::WEST_US_2), + "https://test.westus2.documents.azure.com", + ); + alternate.complete_request(won, StatusCode::Ok, None); + drop(primary); + builder.merge_hedge_attempt(alternate); + }); + + assert_eq!(ctx.request_count(), 1); + // The primary leg dispatched, so it is not reconstructed from the + // fan-out record either — that fallback is only for a leg that never + // reached the wire at all. + assert_eq!( + ctx.requested_regions(), + vec![RequestedRegion { + region: Region::WEST_US_2, + reason: RequestedRegionReason::Hedging, + }] + ); + assert_eq!(ctx.responded_regions(), vec![&Region::WEST_US_2]); + } + + #[test] + fn execution_context_retry_variant_still_serializes_as_retry() { + // `Retry` is deprecated but deliberately *not* a serde alias for + // `OperationRetry`: any value still constructed as `Retry` must keep + // emitting `"retry"` on the wire for the deprecation window. This + // guards the documented contract against an accidental + // `#[serde(rename)]`/alias edit while the variant lives on. + #[allow(deprecated)] + let retry = ExecutionContext::Retry; + #[allow(deprecated)] + { + assert_eq!( + serde_json::to_string(&retry).expect("ExecutionContext is Serialize"), + "\"retry\"" + ); + assert_eq!(retry.as_str(), "retry"); + } + // ...and the replacement variant is genuinely distinct on the wire. + assert_eq!( + serde_json::to_string(&ExecutionContext::OperationRetry) + .expect("ExecutionContext is Serialize"), + "\"operation_retry\"" + ); + } + #[test] fn aggregate_sub_operations_propagates_hedge_diagnostics() { // An aggregated operation (e.g. PATCH) whose sub-op hedged must still @@ -5072,17 +5715,21 @@ mod tests { // the Hedging Detection API must not depend on it: each sub-op's own // materialized dispatch history is concatenated in sub-op order. let read = make_context_with(ActivityId::new_uuid(), |builder| { - builder.record_hedge_fanout( + let (primary, primary_dispatch) = spawn_primary_leg( + builder, Some(Region::EAST_US_2), RequestedRegionReason::Initial, - Some(Region::WEST_US_2), ); - let h = builder.start_test_request( + let mut alternate = + spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); + let h = alternate.start_test_request( ExecutionContext::Hedging, Some(Region::WEST_US_2), "https://test.westus2.documents.azure.com", ); - builder.complete_request(h, StatusCode::Ok, None); + alternate.complete_request(h, StatusCode::Ok, None); + drop(primary); + builder.merge_hedge_attempt(alternate); builder.set_hedge_diagnostics(HedgeDiagnostics::hedge_won( hedge_config(), Region::EAST_US_2, @@ -5090,17 +5737,23 @@ mod tests { )); }); let replace = make_context_with(ActivityId::new_uuid(), |builder| { - builder.record_hedge_fanout( + let (mut primary, primary_dispatch) = spawn_primary_leg( + builder, Some(Region::EAST_US_2), RequestedRegionReason::Initial, - Some(Region::CENTRAL_US), ); - let h = builder.start_test_request( + let h = primary.start_test_request( ExecutionContext::Initial, Some(Region::EAST_US_2), "https://test.eastus2.documents.azure.com", ); - builder.complete_request(h, StatusCode::Ok, None); + primary.complete_request(h, StatusCode::Ok, None); + drop(spawn_alternate_leg( + builder, + primary_dispatch, + Some(Region::CENTRAL_US), + )); + builder.merge_hedge_attempt(primary); builder.set_hedge_diagnostics(HedgeDiagnostics::primary_won_after_hedge( hedge_config(), Region::EAST_US_2, diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs index 366b07477cd..5d30ac6756f 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs @@ -3109,6 +3109,11 @@ async fn execute_hedged( // `Initial` exactly as before. let primary_execution_context = compute_execution_context(retry_state_snapshot); let primary_diag = parent_diagnostics.clone_for_hedge_attempt(); + // Describe the primary as a race participant before its builder is moved + // into the future. `leg_dispatch` reads the leg's launch instant, so the + // fan-out record orders correctly against every attempt in the operation. + let primary_dispatch = + primary_diag.leg_dispatch(primary_region.clone(), primary_execution_context.into()); let primary_attempt = Box::pin(async move { let mut diag = primary_diag; // Primary is launched before Stage 2 elapses, so no shared @@ -3306,18 +3311,21 @@ async fn execute_hedged( ) .then(|| Arc::new(AtomicBool::new(ctx.hub_region_processing_only_initial))); let secondary_shared_latch = shared_hub_region_latch.clone(); - // Record the fan-out on the *parent* before the race runs. Whichever leg - // loses is structurally dropped along with its per-attempt diagnostics, so - // this is the only place the full dispatch history can be captured. The - // parent builder reaches every exit path (`finalize_hedge_attempt` on - // Terminal, `diagnostics: parent_diagnostics` on BothTransient), so the - // record always survives to `complete()`. - parent_diagnostics.record_hedge_fanout( - primary_region.clone(), - primary_execution_context.into(), + // Record the fan-out on the *parent* before the race runs, so the operation + // is known to have hedged regardless of which leg won. Each leg's own + // attempts survive the race via the diagnostics hedge journal, so this + // record only has to reconstruct a leg cancelled before it dispatched + // anything — most commonly the alternate, which `select` never polls when + // the primary is already resolved. The parent builder reaches every exit + // path (`finalize_hedge_attempt` on Terminal, `diagnostics: + // parent_diagnostics` on BothTransient), so the record always survives to + // `complete()`. + let secondary_diag = parent_diagnostics.clone_for_hedge_attempt(); + let secondary_dispatch = secondary_diag.leg_dispatch( secondary_region.clone(), + crate::diagnostics::RequestedRegionReason::Hedging, ); - let secondary_diag = parent_diagnostics.clone_for_hedge_attempt(); + parent_diagnostics.record_hedge_fanout(primary_dispatch, secondary_dispatch); let secondary_attempt = Box::pin(async move { let mut diag = secondary_diag; let result = perform_single_attempt( @@ -8608,7 +8616,7 @@ mod tests { #[test] fn diagnostics_clone_for_hedge_attempt_starts_empty() { - let parent = test_diagnostics(); + let mut parent = test_diagnostics(); let child = parent.clone_for_hedge_attempt(); // A fresh sub-builder must not carry the parent's request list, // status, or accumulated hedge diagnostics. From 04f474035a28a718c587367e62cefa45c85a60d0 Mon Sep 17 00:00:00 2001 From: Copilot App <223556219+Copilot@users.noreply.github.com> Date: Fri, 31 Jul 2026 15:20:17 -0700 Subject: [PATCH 13/21] Stop using deprecated ExecutionContext::Retry in tests The ExecutionContext::Retry -> OperationRetry deprecation broke CI: three pre-existing tests in diagnostics_context.rs still constructed the deprecated variant incidentally, and -D warnings turns the deprecation lint into a hard error, so zure_data_cosmos_driver failed to compile as a lib test across every clippy and test job. Those three sites (the PATCH sub-operation compaction test and the single-operation request-name test) don't assert anything about the deprecated variant, so they now use OperationRetry, which maps to the same RequestedRegionReason. The tests that deliberately exercise the deprecated variant's serialization contract keep their local #[allow(deprecated)] and are untouched. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 22b24639-d6f1-4849-98df-36b2f3c8630b --- .../src/diagnostics/diagnostics_context.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs index 031a212c1b4..1cd216d776c 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs @@ -6374,7 +6374,7 @@ mod tests { ); record_run( &mut read_b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", "https://east/", CosmosStatus::new(StatusCode::Ok), @@ -6395,7 +6395,7 @@ mod tests { ); record_run( &mut replace_b, - ExecutionContext::Retry, + ExecutionContext::OperationRetry, "East US", "https://east/", CosmosStatus::new(StatusCode::Ok), @@ -6918,7 +6918,7 @@ mod tests { "https://test.westus2.documents.azure.com", ); builder.start_test_request( - ExecutionContext::Retry, + ExecutionContext::OperationRetry, Some(Region::WEST_US_2), "https://test.westus2.documents.azure.com", ); From 85046527048b1445cf3adee326091baa8682c95d Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Tue, 4 Aug 2026 10:54:49 -0700 Subject: [PATCH 14/21] fix(cosmos): define SDK-owned RequestedRegion types Replace direct re-exports of RequestedRegion and RequestedRegionReason from zure_data_cosmos_driver with SDK-owned wrapper types in a new diagnostics::region module. - Add sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs with RequestedRegion, RequestedRegionReason, and From impls - Update diagnostics/mod.rs to export the SDK-owned types instead of the driver types directly Addresses review feedback that the driver types should not be part of the zure_data_cosmos public API surface, per the Cosmos versioning rule. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../azure_data_cosmos/src/diagnostics/mod.rs | 4 +- .../src/diagnostics/region.rs | 144 ++++++++++++++++++ 2 files changed, 147 insertions(+), 1 deletion(-) create mode 100644 sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs index 56813c731fc..9f46bc80472 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs @@ -42,7 +42,7 @@ #[doc(inline)] pub use azure_data_cosmos_driver::diagnostics::{ - DiagnosticsContext, RequestedRegion, RequestedRegionReason, ThresholdBreach, TransportKind, + DiagnosticsContext, ThresholdBreach, TransportKind, }; #[doc(inline)] pub use azure_data_cosmos_driver::DiagnosticsThresholds; @@ -52,6 +52,7 @@ pub use handler::{ pub use logging::{SamplingLogHandler, TracingLogHandler}; pub use operation_context::CosmosOperationContext; pub use rate_limiter::RateLimiterConfig; +pub use region::{RequestedRegion, RequestedRegionReason}; #[cfg(feature = "distributed_tracing")] pub use tracing::CosmosTracingHandler; @@ -72,6 +73,7 @@ mod handler; mod logging; mod operation_context; mod reason; +mod region; // Count-per-interval rate limiter shared by the sampling handlers (logging and, // when enabled, tracing) so they can bound emission under an error storm. pub(crate) mod rate_limiter; diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs new file mode 100644 index 00000000000..66ac4d3a25b --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs @@ -0,0 +1,144 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +//! SDK-owned region-request types for the Hedging Detection API. +//! +//! These types mirror the cross-SDK Hedging Detection API's `RequestedRegion` +//! and `RequestedRegionReason` while remaining wholly owned by +//! `azure_data_cosmos`. They are projected from the driver equivalents at +//! diagnostics-context finalisation, which lets the driver evolve its internal +//! model without forcing an SDK major-version bump. + +use crate::options::Region; + +/// The reason the SDK dispatched a request to a particular region. +/// +/// Carried by each [`RequestedRegion`] entry returned from +/// [`DiagnosticsContext::requested_regions`](crate::diagnostics::DiagnosticsContext::requested_regions). +/// +/// The enum is `#[non_exhaustive]`; always include a wildcard arm in `match` +/// expressions. +/// +/// # Example +/// +/// ```rust,no_run +/// # use azure_data_cosmos::diagnostics::{RequestedRegion, RequestedRegionReason}; +/// fn describe(r: &RequestedRegion) -> &'static str { +/// match r.reason { +/// RequestedRegionReason::Initial => "initial dispatch", +/// RequestedRegionReason::OperationRetry => "SDK-level retry", +/// RequestedRegionReason::TransportRetry => "transport-level retry", +/// RequestedRegionReason::Hedging => "speculative hedge", +/// RequestedRegionReason::RegionFailover => "region-failover retry", +/// RequestedRegionReason::CircuitBreakerProbe => "circuit-breaker probe", +/// _ => "unknown", +/// } +/// } +/// ``` +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum RequestedRegionReason { + /// The first dispatch of the operation. + Initial, + /// An operation-level retry decided by the SDK's client-retry policy. + OperationRetry, + /// A transport-level retry inside the per-region transport stack. + TransportRetry, + /// A speculative cross-region hedge fan-out dispatch. + Hedging, + /// An endpoint-failure-driven retry to a different region. + RegionFailover, + /// A probe dispatch to a previously circuit-broken region. + CircuitBreakerProbe, +} + +impl From for RequestedRegionReason { + fn from( + driver: azure_data_cosmos_driver::diagnostics::RequestedRegionReason, + ) -> RequestedRegionReason { + match driver { + azure_data_cosmos_driver::diagnostics::RequestedRegionReason::Initial => { + RequestedRegionReason::Initial + } + azure_data_cosmos_driver::diagnostics::RequestedRegionReason::OperationRetry => { + RequestedRegionReason::OperationRetry + } + azure_data_cosmos_driver::diagnostics::RequestedRegionReason::TransportRetry => { + RequestedRegionReason::TransportRetry + } + azure_data_cosmos_driver::diagnostics::RequestedRegionReason::Hedging => { + RequestedRegionReason::Hedging + } + azure_data_cosmos_driver::diagnostics::RequestedRegionReason::RegionFailover => { + RequestedRegionReason::RegionFailover + } + azure_data_cosmos_driver::diagnostics::RequestedRegionReason::CircuitBreakerProbe => { + RequestedRegionReason::CircuitBreakerProbe + } + // The driver enum is #[non_exhaustive]; map any future variants to + // the closest known reason rather than panicking. + _ => RequestedRegionReason::Initial, + } + } +} + +/// A single region the SDK dispatched a request to, tagged with the reason the +/// orchestrator chose to send it. +/// +/// Realizes the cross-SDK Hedging Detection API's `RequestedRegion` value type. +/// Returned by +/// [`DiagnosticsContext::requested_regions`](crate::diagnostics::DiagnosticsContext::requested_regions). +/// +/// The struct is `#[non_exhaustive]`; construct via the public fields only in +/// owned contexts, and pattern-match with `..` to remain forward-compatible. +#[derive(Clone, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct RequestedRegion { + /// The region the SDK dispatched to. + pub region: Region, + /// The reason the SDK chose this region for this dispatch attempt. + pub reason: RequestedRegionReason, +} + +impl From for RequestedRegion { + fn from(driver: azure_data_cosmos_driver::diagnostics::RequestedRegion) -> RequestedRegion { + RequestedRegion { + region: driver.region, + reason: driver.reason.into(), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use azure_data_cosmos_driver::diagnostics::RequestedRegionReason as DriverReason; + + #[test] + fn reason_from_driver_all_variants() { + assert_eq!( + RequestedRegionReason::from(DriverReason::Initial), + RequestedRegionReason::Initial + ); + assert_eq!( + RequestedRegionReason::from(DriverReason::OperationRetry), + RequestedRegionReason::OperationRetry + ); + assert_eq!( + RequestedRegionReason::from(DriverReason::TransportRetry), + RequestedRegionReason::TransportRetry + ); + assert_eq!( + RequestedRegionReason::from(DriverReason::Hedging), + RequestedRegionReason::Hedging + ); + assert_eq!( + RequestedRegionReason::from(DriverReason::RegionFailover), + RequestedRegionReason::RegionFailover + ); + assert_eq!( + RequestedRegionReason::from(DriverReason::CircuitBreakerProbe), + RequestedRegionReason::CircuitBreakerProbe + ); + } +} From 8cd38116b79e936752df6096760187943cbadce7 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Tue, 4 Aug 2026 11:16:49 -0700 Subject: [PATCH 15/21] Remove ExecutionContext::Retry and collapse RequestedRegionReason into ExecutionContext Per reviewer feedback (r3714936305, r3714941471): - Remove ExecutionContext::Retry variant entirely (pre-GA breaking change is fine) - Remove RequestedRegionReason from driver; RequestedRegion.reason is now typed as ExecutionContext directly - Update HedgeLegDispatch.reason, leg_dispatch(), and operation_pipeline.rs to use ExecutionContext directly - Update SDK-owned RequestedRegionReason to convert From (instead of the now-removed From) - Delete obsolete tests; bulk-update remaining test uses of RequestedRegionReason::* to ExecutionContext::* - Fix useless .into() in operation_pipeline.rs (clippy) - Update both CHANGELOGs to reflect full removal Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d51868ed-95ef-4dff-b237-4f8226e5ec9a --- sdk/cosmos/azure_data_cosmos/CHANGELOG.md | 2 +- .../src/diagnostics/region.rs | 30 +- .../azure_data_cosmos_driver/CHANGELOG.md | 4 +- .../src/diagnostics/diagnostics_context.rs | 279 +++++------------- .../src/diagnostics/mod.rs | 4 +- .../src/driver/pipeline/operation_pipeline.rs | 4 +- 6 files changed, 90 insertions(+), 233 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md index 39f7ae8c817..a36b326e551 100644 --- a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md @@ -18,7 +18,7 @@ ### Breaking Changes -- Serialized `DiagnosticsContext` output (the diagnostics JSON surfaced to consumers, and the sampled diagnostics log line) now serializes driver-generated operation retries with `execution_context` = `"operation_retry"` instead of `"retry"`, following the driver's `ExecutionContext::Retry` → `OperationRetry` rename. This is additive to the enum (the deprecated `Retry` variant still serializes as `"retry"`), but the wire value emitted for operation retries changes; telemetry/log parsers that match the literal `"retry"` execution context must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Serialized `DiagnosticsContext` output (the diagnostics JSON surfaced to consumers, and the sampled diagnostics log line) now serializes driver-generated operation retries with `execution_context` = `"operation_retry"` instead of `"retry"`. The driver's `ExecutionContext::Retry` variant has been removed entirely; use `ExecutionContext::OperationRetry` instead. Telemetry/log parsers that match the literal `"retry"` execution context must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Control-plane APIs are now gated behind the new `control_plane` feature, which is **not** enabled by default. Code using database or container management (`CosmosClient::create_database`/`query_databases`, `DatabaseClient::read`/`create_container`/`query_containers`/`delete`, `ContainerClient::replace`/`delete`), throughput management (`read_throughput`/`begin_replace_throughput`, `ThroughputPoller`), or the associated model and options types (`DatabaseProperties`, `ThroughputProperties`, and the container create/replace/delete/query, database, and throughput option types) must now enable the `control_plane` feature. Reading container properties via `ContainerClient::read()` — along with `ContainerProperties`, `IndexingPolicy`, `ResourceResponse`, and `ReadContainerOptions` — remains available without the feature, since it works with Entra ID authentication and mirrors the metadata read the SDK already performs internally. ([#4854](https://github.com/Azure/azure-sdk-for-rust/pull/4854)) - Fresh cross-partition queries and change feed reads now fail with a `BadRequest` error if they would fan out to more than 100 physical partitions. Raise `FeedOptions::max_fan_out` to run a broader operation. The limit is checked only at initial query setup — resuming from a continuation token is unaffected, and a partition split that raises the fan-out mid-execution does not abort the operation. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855)) diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs index 66ac4d3a25b..24b9c638205 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs @@ -52,27 +52,27 @@ pub enum RequestedRegionReason { CircuitBreakerProbe, } -impl From for RequestedRegionReason { +impl From for RequestedRegionReason { fn from( - driver: azure_data_cosmos_driver::diagnostics::RequestedRegionReason, + driver: azure_data_cosmos_driver::diagnostics::ExecutionContext, ) -> RequestedRegionReason { match driver { - azure_data_cosmos_driver::diagnostics::RequestedRegionReason::Initial => { + azure_data_cosmos_driver::diagnostics::ExecutionContext::Initial => { RequestedRegionReason::Initial } - azure_data_cosmos_driver::diagnostics::RequestedRegionReason::OperationRetry => { + azure_data_cosmos_driver::diagnostics::ExecutionContext::OperationRetry => { RequestedRegionReason::OperationRetry } - azure_data_cosmos_driver::diagnostics::RequestedRegionReason::TransportRetry => { + azure_data_cosmos_driver::diagnostics::ExecutionContext::TransportRetry => { RequestedRegionReason::TransportRetry } - azure_data_cosmos_driver::diagnostics::RequestedRegionReason::Hedging => { + azure_data_cosmos_driver::diagnostics::ExecutionContext::Hedging => { RequestedRegionReason::Hedging } - azure_data_cosmos_driver::diagnostics::RequestedRegionReason::RegionFailover => { + azure_data_cosmos_driver::diagnostics::ExecutionContext::RegionFailover => { RequestedRegionReason::RegionFailover } - azure_data_cosmos_driver::diagnostics::RequestedRegionReason::CircuitBreakerProbe => { + azure_data_cosmos_driver::diagnostics::ExecutionContext::CircuitBreakerProbe => { RequestedRegionReason::CircuitBreakerProbe } // The driver enum is #[non_exhaustive]; map any future variants to @@ -112,32 +112,32 @@ impl From for RequestedR #[cfg(test)] mod tests { use super::*; - use azure_data_cosmos_driver::diagnostics::RequestedRegionReason as DriverReason; + use azure_data_cosmos_driver::diagnostics::ExecutionContext as DriverCtx; #[test] fn reason_from_driver_all_variants() { assert_eq!( - RequestedRegionReason::from(DriverReason::Initial), + RequestedRegionReason::from(DriverCtx::Initial), RequestedRegionReason::Initial ); assert_eq!( - RequestedRegionReason::from(DriverReason::OperationRetry), + RequestedRegionReason::from(DriverCtx::OperationRetry), RequestedRegionReason::OperationRetry ); assert_eq!( - RequestedRegionReason::from(DriverReason::TransportRetry), + RequestedRegionReason::from(DriverCtx::TransportRetry), RequestedRegionReason::TransportRetry ); assert_eq!( - RequestedRegionReason::from(DriverReason::Hedging), + RequestedRegionReason::from(DriverCtx::Hedging), RequestedRegionReason::Hedging ); assert_eq!( - RequestedRegionReason::from(DriverReason::RegionFailover), + RequestedRegionReason::from(DriverCtx::RegionFailover), RequestedRegionReason::RegionFailover ); assert_eq!( - RequestedRegionReason::from(DriverReason::CircuitBreakerProbe), + RequestedRegionReason::from(DriverCtx::CircuitBreakerProbe), RequestedRegionReason::CircuitBreakerProbe ); } diff --git a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md index 0c8eb64f08e..79a172470fc 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md @@ -9,12 +9,12 @@ - Added `PlanOptions` (with `DEFAULT_MAX_FAN_OUT`) to `CosmosDriver::plan_operation`, enforcing a maximum fan-out on fresh cross-partition plans. A fresh plan spanning more leaf request nodes than `PlanOptions::max_fan_out` (default 100) is rejected with the new `CosmosStatus::CLIENT_CROSS_PARTITION_FAN_OUT_EXCEEDED` (HTTP 400). The limit is enforced only at initial plan time: resuming from a continuation token skips the check, and a partition split that raises the fan-out mid-execution does not abort the operation. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855)) - Added `CosmosOperation::db_operation_name`, returning the canonical OpenTelemetry `db.operation.name` (e.g. `read_item`, `query_items`, `execute_batch`, and `read_all_items_of_logical_partition` for a read feed scoped to one logical partition) for an operation. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) - Added `RequestDiagnostics::operation_name`, naming the operation that issued an individual attempt. It is set only where one `DiagnosticsContext` aggregates attempts from more than one operation — today a PATCH, whose attempts report `patch_read_item` / `patch_replace_item` while the context reports `patch_item` — and is `None` otherwise, meaning the attempt shares the context's operation name. `CosmosOperation::is_patch_sub_operation` reports the same distinction on the operation itself. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) -- Added the Hedging Detection API on `DiagnosticsContext`: `hedging_started()` (whether a hedge arm actually fanned out), `requested_regions()` (regions dispatched to, in dispatch order with duplicates, each tagged with a reason), and `responded_regions()` (regions that produced an actual service reply, in completion order). All three are materialized at finalization from the full pre-compaction attempt list — including attempts rescued from a structurally-dropped hedge leg — plus a dispatch-time hedge fan-out log, so a cancelled hedge leg, a retry storm that compacts `requests()`, and multi-round-trip aggregation all report a complete history. Both region histories are bounded by `max_request_diagnostics` (keeping head and tail, dropping the repetitive middle) so a retry storm cannot produce an unbounded diagnostics artifact; `total_requested_regions()` / `total_responded_regions()` report the exact pre-truncation counts, so truncation is detectable rather than silent. Adds the public `RequestedRegion` struct and `RequestedRegionReason` enum (both `#[non_exhaustive]`, with a total `From` mapping) and the new `ExecutionContext::OperationRetry` variant. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Added the Hedging Detection API on `DiagnosticsContext`: `hedging_started()` (whether a hedge arm actually fanned out), `requested_regions()` (regions dispatched to, in dispatch order with duplicates, each tagged with a reason), and `responded_regions()` (regions that produced an actual service reply, in completion order). All three are materialized at finalization from the full pre-compaction attempt list — including attempts rescued from a structurally-dropped hedge leg — plus a dispatch-time hedge fan-out log, so a cancelled hedge leg, a retry storm that compacts `requests()`, and multi-round-trip aggregation all report a complete history. Both region histories are bounded by `max_request_diagnostics` (keeping head and tail, dropping the repetitive middle) so a retry storm cannot produce an unbounded diagnostics artifact; `total_requested_regions()` / `total_responded_regions()` report the exact pre-truncation counts, so truncation is detectable rather than silent. Adds the public `RequestedRegion` struct and the new `ExecutionContext::OperationRetry` variant. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Added `HedgeTerminalState::as_str()` (and a matching `Display`) returning a stable, low-cardinality `snake_case` identifier for the hedging race outcome, for use as an observability attribute / log-field value. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) ### Breaking Changes -- Renamed `ExecutionContext::Retry` to `ExecutionContext::OperationRetry` to distinguish operation-level retries from transport-level `TransportRetry`. The old `Retry` remains for one release as a distinct `#[deprecated]` variant (**not** a serde alias); a `Retry` value still serializes as `"retry"`. The customer-visible wire-format change is that driver-generated operation retries now serialize as `"operation_retry"` instead of `"retry"`, because the dispatch sites emit `OperationRetry`; telemetry parsers that match the literal `"retry"` execution context must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Removed `ExecutionContext::Retry`. Use `ExecutionContext::OperationRetry` to represent an operation-level retry decided by the SDK's client-retry policy; `TransportRetry` continues to represent a transport-level retry inside the per-region transport stack. The `RequestedRegion::reason` field is now typed as `ExecutionContext` directly (the `RequestedRegionReason` type has also been removed from the driver). The customer-visible wire-format change is that driver-generated operation retries now serialize as `"operation_retry"` instead of `"retry"`; telemetry parsers that match the literal `"retry"` execution context must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - `CosmosDriver::plan_operation` now takes an additional `plan_options: &PlanOptions` argument (after `continuation`). The continuation token remains its own argument. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855)) ### Bugs Fixed diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs index 1cd216d776c..fecaf8f4554 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs @@ -58,18 +58,6 @@ pub enum ThresholdBreach { pub enum ExecutionContext { /// Initial request attempt (first try). Initial, - /// Retry due to transient error (e.g., 429, 503). - /// - /// **Deprecated:** superseded by [`ExecutionContext::OperationRetry`], which - /// aligns with the cross-SDK reason taxonomy and is distinct from the - /// transport-level [`ExecutionContext::TransportRetry`]. This is a distinct, - /// still-constructible variant — **not** a serde alias: a `Retry` value - /// continues to serialize as `"retry"`. The wire-format change to - /// `"operation_retry"` comes from the dispatch sites now emitting - /// `OperationRetry` instead of `Retry`, not from any change to this variant's - /// own serialization. Retained for one release for source compatibility. - #[deprecated(since = "0.7.0", note = "use `ExecutionContext::OperationRetry`")] - Retry, /// An operation-level retry decided by the SDK's client-retry policy. /// /// Distinguishes user-visible operation retries from transport-layer @@ -92,10 +80,8 @@ pub enum ExecutionContext { impl ExecutionContext { /// Returns the string representation of this execution context. pub fn as_str(&self) -> &'static str { - #[allow(deprecated)] match self { ExecutionContext::Initial => "initial", - ExecutionContext::Retry => "retry", ExecutionContext::OperationRetry => "operation_retry", ExecutionContext::TransportRetry => "transport_retry", ExecutionContext::Hedging => "hedging", @@ -105,48 +91,6 @@ impl ExecutionContext { } } -/// Reason the SDK chose to dispatch a request to a particular region. -/// -/// Realizes the cross-SDK Hedging Detection API's `RequestedRegionReason`. Each -/// entry returned by [`DiagnosticsContext::requested_regions`] carries one of -/// these, projected from the driver-internal [`ExecutionContext`] via the -/// [`From`] mapping below. -/// -/// The enum is `#[non_exhaustive]`; callers that `match` on it MUST include a -/// wildcard arm. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] -#[non_exhaustive] -pub enum RequestedRegionReason { - /// The first dispatch of the operation. - Initial, - /// An operation-level retry decided by the SDK's client-retry policy. - OperationRetry, - /// A transport-level retry inside the per-region transport stack. - TransportRetry, - /// A speculative cross-region hedge fan-out dispatch. - Hedging, - /// An endpoint-failure-driven retry to a different region. - RegionFailover, - /// A probe dispatch to a previously circuit-broken region. - CircuitBreakerProbe, -} - -impl From for RequestedRegionReason { - fn from(ctx: ExecutionContext) -> Self { - #[allow(deprecated)] - match ctx { - ExecutionContext::Initial => RequestedRegionReason::Initial, - ExecutionContext::Retry | ExecutionContext::OperationRetry => { - RequestedRegionReason::OperationRetry - } - ExecutionContext::TransportRetry => RequestedRegionReason::TransportRetry, - ExecutionContext::Hedging => RequestedRegionReason::Hedging, - ExecutionContext::RegionFailover => RequestedRegionReason::RegionFailover, - ExecutionContext::CircuitBreakerProbe => RequestedRegionReason::CircuitBreakerProbe, - } - } -} - /// A single region the SDK dispatched a request to, tagged with the reason the /// orchestrator chose to send it. /// @@ -159,7 +103,7 @@ pub struct RequestedRegion { /// The region the SDK dispatched to. pub region: Region, /// The reason the SDK chose this region for this dispatch attempt. - pub reason: RequestedRegionReason, + pub reason: ExecutionContext, } impl AsRef for ExecutionContext { @@ -1529,7 +1473,7 @@ pub(crate) struct HedgeLegDispatch { /// carries no named region (global-endpoint accounts). pub(crate) region: Option, /// Why the orchestrator dispatched this leg to that region. - pub(crate) reason: RequestedRegionReason, + pub(crate) reason: ExecutionContext, /// The leg builder's journal id, used to detect whether the leg ever /// dispatched a request of its own. leg_id: u64, @@ -1714,7 +1658,7 @@ impl DiagnosticsContextBuilder { pub(crate) fn leg_dispatch( &self, region: Option, - reason: RequestedRegionReason, + reason: ExecutionContext, ) -> HedgeLegDispatch { HedgeLegDispatch { region, @@ -2515,13 +2459,13 @@ impl DiagnosticsContext { Some(HedgeFanout { primary: HedgeLegDispatch { region: not_sentinel(hedge.primary_region()), - reason: RequestedRegionReason::Initial, + reason: ExecutionContext::Initial, leg_id: PRIMARY_LEG, dispatched_at, }, alternate: HedgeLegDispatch { region: not_sentinel(alternate), - reason: RequestedRegionReason::Hedging, + reason: ExecutionContext::Hedging, leg_id: ALTERNATE_LEG, dispatched_at, }, @@ -2540,8 +2484,7 @@ impl DiagnosticsContext { leg.region.as_ref().is_some_and(|region| { requests.iter().any(|request| { request.region() == Some(region) - && RequestedRegionReason::from(request.execution_context()) - == leg.reason + && request.execution_context() == leg.reason }) }) }) @@ -2891,7 +2834,7 @@ impl DiagnosticsContext { /// Duplicates are allowed: the same region may appear more than once if it /// was dispatched multiple times (e.g., a retry to the same region, or a /// hedge request to a region that was also the primary). The initial attempt - /// is included and tagged [`RequestedRegionReason::Initial`]. + /// is included and tagged [`ExecutionContext::Initial`]. /// /// Entries with no resolved region (pre-region-selection failures, and /// global-endpoint accounts that carry no named region) are skipped, so this @@ -2906,7 +2849,7 @@ impl DiagnosticsContext { /// parent at dispatch time, so **both** legs always appear here: the primary /// leg tagged with the reason it was actually dispatched under (`Initial` for /// a first attempt, or the failover/session reason when the hedge upgraded a - /// retry), and the alternate leg tagged [`RequestedRegionReason::Hedging`]. + /// retry), and the alternate leg tagged [`ExecutionContext::Hedging`]. /// A dropped leg has no corresponding [`responded_regions`](Self::responded_regions) /// entry, since it never produced a service reply. /// @@ -3619,7 +3562,7 @@ fn requested_regions_from( if let Some(region) = request.region() { regions.push(RequestedRegion { region: region.clone(), - reason: RequestedRegionReason::from(request.execution_context()), + reason: request.execution_context(), }); } } @@ -4920,10 +4863,6 @@ mod tests { #[test] fn execution_context_display() { assert_eq!(ExecutionContext::Initial.to_string(), "initial"); - #[allow(deprecated)] - { - assert_eq!(ExecutionContext::Retry.to_string(), "retry"); - } assert_eq!( ExecutionContext::OperationRetry.to_string(), "operation_retry" @@ -4967,11 +4906,11 @@ mod tests { assert_eq!(requested.len(), 3); // Dispatch order preserved, duplicates kept. assert_eq!(requested[0].region, Region::WEST_US_2); - assert_eq!(requested[0].reason, RequestedRegionReason::Initial); + assert_eq!(requested[0].reason, ExecutionContext::Initial); assert_eq!(requested[1].region, Region::WEST_US_2); - assert_eq!(requested[1].reason, RequestedRegionReason::OperationRetry); + assert_eq!(requested[1].reason, ExecutionContext::OperationRetry); assert_eq!(requested[2].region, Region::EAST_US_2); - assert_eq!(requested[2].reason, RequestedRegionReason::RegionFailover); + assert_eq!(requested[2].reason, ExecutionContext::RegionFailover); } #[test] @@ -5067,7 +5006,7 @@ mod tests { fn spawn_primary_leg( parent: &mut DiagnosticsContextBuilder, region: Option, - reason: RequestedRegionReason, + reason: ExecutionContext, ) -> (DiagnosticsContextBuilder, HedgeLegDispatch) { let leg = parent.clone_for_hedge_attempt(); let dispatch = leg.leg_dispatch(region, reason); @@ -5082,7 +5021,7 @@ mod tests { region: Option, ) -> DiagnosticsContextBuilder { let leg = parent.clone_for_hedge_attempt(); - let dispatch = leg.leg_dispatch(region, RequestedRegionReason::Hedging); + let dispatch = leg.leg_dispatch(region, ExecutionContext::Hedging); parent.record_hedge_fanout(primary, dispatch); leg } @@ -5094,11 +5033,8 @@ mod tests { // still appear, in dispatch order, from the fan-out the orchestrator // recorded on the parent. let ctx = make_context_with(ActivityId::new_uuid(), |builder| { - let (mut primary, primary_dispatch) = spawn_primary_leg( - builder, - Some(Region::EAST_US_2), - RequestedRegionReason::Initial, - ); + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); let h = primary.start_test_request( ExecutionContext::Initial, Some(Region::EAST_US_2), @@ -5126,11 +5062,11 @@ mod tests { vec![ RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::Initial, + reason: ExecutionContext::Initial, }, RequestedRegion { region: Region::WEST_US_2, - reason: RequestedRegionReason::Hedging, + reason: ExecutionContext::Hedging, }, ] ); @@ -5145,11 +5081,8 @@ mod tests { // is dropped without having dispatched, so the Initial primary must // still be listed first (dispatch order) from the fan-out record. let ctx = make_context_with(ActivityId::new_uuid(), |builder| { - let (primary, primary_dispatch) = spawn_primary_leg( - builder, - Some(Region::EAST_US_2), - RequestedRegionReason::Initial, - ); + let (primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); let mut alternate = spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); let h = alternate.start_test_request( @@ -5173,11 +5106,11 @@ mod tests { vec![ RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::Initial, + reason: ExecutionContext::Initial, }, RequestedRegion { region: Region::WEST_US_2, - reason: RequestedRegionReason::Hedging, + reason: ExecutionContext::Hedging, }, ] ); @@ -5200,7 +5133,7 @@ mod tests { let (mut primary, primary_dispatch) = spawn_primary_leg( builder, Some(Region::WEST_US_2), - RequestedRegionReason::RegionFailover, + ExecutionContext::RegionFailover, ); let h = primary.start_test_request( ExecutionContext::RegionFailover, @@ -5222,15 +5155,15 @@ mod tests { vec![ RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::Initial, + reason: ExecutionContext::Initial, }, RequestedRegion { region: Region::WEST_US_2, - reason: RequestedRegionReason::RegionFailover, + reason: ExecutionContext::RegionFailover, }, RequestedRegion { region: Region::CENTRAL_US, - reason: RequestedRegionReason::Hedging, + reason: ExecutionContext::Hedging, }, ] ); @@ -5262,7 +5195,7 @@ mod tests { let (primary, primary_dispatch) = spawn_primary_leg( &mut builder, Some(Region::EAST_US_2), - RequestedRegionReason::OperationRetry, + ExecutionContext::OperationRetry, ); let mut alternate = spawn_alternate_leg(&mut builder, primary_dispatch, Some(Region::WEST_US_2)); @@ -5293,10 +5226,10 @@ mod tests { // fan-out that ended it are both still visible. let requested = ctx.requested_regions(); assert_eq!(requested[0].region, Region::EAST_US_2); - assert_eq!(requested[0].reason, RequestedRegionReason::OperationRetry); + assert_eq!(requested[0].reason, ExecutionContext::OperationRetry); assert_eq!( requested.last().expect("non-empty").reason, - RequestedRegionReason::Hedging + ExecutionContext::Hedging ); assert_eq!( requested.last().expect("non-empty").region, @@ -5433,7 +5366,7 @@ mod tests { let (mut primary, primary_dispatch) = spawn_primary_leg( builder, Some(Region::EAST_US_2), - RequestedRegionReason::OperationRetry, + ExecutionContext::OperationRetry, ); let h = primary.start_test_request( ExecutionContext::OperationRetry, @@ -5465,23 +5398,23 @@ mod tests { vec![ RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::OperationRetry, + reason: ExecutionContext::OperationRetry, }, RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::OperationRetry, + reason: ExecutionContext::OperationRetry, }, RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::OperationRetry, + reason: ExecutionContext::OperationRetry, }, RequestedRegion { region: Region::WEST_US_2, - reason: RequestedRegionReason::Hedging, + reason: ExecutionContext::Hedging, }, RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::OperationRetry, + reason: ExecutionContext::OperationRetry, }, ] ); @@ -5494,7 +5427,7 @@ mod tests { // counts as a fan-out. let ctx = make_context_with(ActivityId::new_uuid(), |builder| { let (primary, primary_dispatch) = - spawn_primary_leg(builder, None, RequestedRegionReason::Initial); + spawn_primary_leg(builder, None, ExecutionContext::Initial); drop(spawn_alternate_leg(builder, primary_dispatch, None)); drop(primary); }); @@ -5526,7 +5459,7 @@ mod tests { ctx.requested_regions(), vec![RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::Initial, + reason: ExecutionContext::Initial, }] ); } @@ -5539,11 +5472,8 @@ mod tests { // builder with the race would erase that reply, under-reporting charge // and omitting a genuine entry from responded_regions. let ctx = make_context_with(ActivityId::new_uuid(), |builder| { - let (mut primary, primary_dispatch) = spawn_primary_leg( - builder, - Some(Region::EAST_US_2), - RequestedRegionReason::Initial, - ); + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); // Primary is throttled and sleeps before its next attempt. let throttled = primary.start_test_request( ExecutionContext::Initial, @@ -5582,11 +5512,11 @@ mod tests { vec![ RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::Initial, + reason: ExecutionContext::Initial, }, RequestedRegion { region: Region::WEST_US_2, - reason: RequestedRegionReason::Hedging, + reason: ExecutionContext::Hedging, }, ] ); @@ -5613,11 +5543,8 @@ mod tests { // report that true order, not group both legs' fan-out ahead of the // primary's own attempts. let ctx = make_context_with(ActivityId::new_uuid(), |builder| { - let (mut primary, primary_dispatch) = spawn_primary_leg( - builder, - Some(Region::EAST_US_2), - RequestedRegionReason::Initial, - ); + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); let first = primary.start_test_request( ExecutionContext::Initial, Some(Region::EAST_US_2), @@ -5646,15 +5573,15 @@ mod tests { vec![ RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::Initial, + reason: ExecutionContext::Initial, }, RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::OperationRetry, + reason: ExecutionContext::OperationRetry, }, RequestedRegion { region: Region::WEST_US_2, - reason: RequestedRegionReason::Hedging, + reason: ExecutionContext::Hedging, }, ] ); @@ -5666,11 +5593,8 @@ mod tests { // list must interleave the two legs by real dispatch time rather than // concatenating one leg after the other. let ctx = make_context_with(ActivityId::new_uuid(), |builder| { - let (mut primary, primary_dispatch) = spawn_primary_leg( - builder, - Some(Region::EAST_US_2), - RequestedRegionReason::Initial, - ); + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); let p1 = primary.start_test_request( ExecutionContext::Initial, Some(Region::EAST_US_2), @@ -5711,19 +5635,19 @@ mod tests { vec![ RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::Initial, + reason: ExecutionContext::Initial, }, RequestedRegion { region: Region::WEST_US_2, - reason: RequestedRegionReason::Hedging, + reason: ExecutionContext::Hedging, }, RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::OperationRetry, + reason: ExecutionContext::OperationRetry, }, RequestedRegion { region: Region::WEST_US_2, - reason: RequestedRegionReason::OperationRetry, + reason: ExecutionContext::OperationRetry, }, ] ); @@ -5736,11 +5660,8 @@ mod tests { // continues into the failover loop. Everything both legs observed must // still reach the finalized context. let ctx = make_context_with(ActivityId::new_uuid(), |builder| { - let (mut primary, primary_dispatch) = spawn_primary_leg( - builder, - Some(Region::EAST_US_2), - RequestedRegionReason::Initial, - ); + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); let p = primary.start_test_request( ExecutionContext::Initial, Some(Region::EAST_US_2), @@ -5776,15 +5697,15 @@ mod tests { vec![ RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::Initial, + reason: ExecutionContext::Initial, }, RequestedRegion { region: Region::WEST_US_2, - reason: RequestedRegionReason::Hedging, + reason: ExecutionContext::Hedging, }, RequestedRegion { region: Region::CENTRAL_US, - reason: RequestedRegionReason::RegionFailover, + reason: ExecutionContext::RegionFailover, }, ] ); @@ -5801,11 +5722,8 @@ mod tests { // when its leg is cancelled saw no reply, so it is intentionally absent // from both histories — reporting it would invent a response. let ctx = make_context_with(ActivityId::new_uuid(), |builder| { - let (mut primary, primary_dispatch) = spawn_primary_leg( - builder, - Some(Region::EAST_US_2), - RequestedRegionReason::Initial, - ); + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); // Dispatched, never completed. let _in_flight = primary.start_test_request( ExecutionContext::Initial, @@ -5833,37 +5751,12 @@ mod tests { ctx.requested_regions(), vec![RequestedRegion { region: Region::WEST_US_2, - reason: RequestedRegionReason::Hedging, + reason: ExecutionContext::Hedging, }] ); assert_eq!(ctx.responded_regions(), vec![&Region::WEST_US_2]); } - #[test] - fn execution_context_retry_variant_still_serializes_as_retry() { - // `Retry` is deprecated but deliberately *not* a serde alias for - // `OperationRetry`: any value still constructed as `Retry` must keep - // emitting `"retry"` on the wire for the deprecation window. This - // guards the documented contract against an accidental - // `#[serde(rename)]`/alias edit while the variant lives on. - #[allow(deprecated)] - let retry = ExecutionContext::Retry; - #[allow(deprecated)] - { - assert_eq!( - serde_json::to_string(&retry).expect("ExecutionContext is Serialize"), - "\"retry\"" - ); - assert_eq!(retry.as_str(), "retry"); - } - // ...and the replacement variant is genuinely distinct on the wire. - assert_eq!( - serde_json::to_string(&ExecutionContext::OperationRetry) - .expect("ExecutionContext is Serialize"), - "\"operation_retry\"" - ); - } - #[test] fn aggregate_sub_operations_propagates_hedge_diagnostics() { // An aggregated operation (e.g. PATCH) whose sub-op hedged must still @@ -5911,11 +5804,8 @@ mod tests { // the Hedging Detection API must not depend on it: each sub-op's own // materialized dispatch history is concatenated in sub-op order. let read = make_context_with(ActivityId::new_uuid(), |builder| { - let (primary, primary_dispatch) = spawn_primary_leg( - builder, - Some(Region::EAST_US_2), - RequestedRegionReason::Initial, - ); + let (primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); let mut alternate = spawn_alternate_leg(builder, primary_dispatch, Some(Region::WEST_US_2)); let h = alternate.start_test_request( @@ -5933,11 +5823,8 @@ mod tests { )); }); let replace = make_context_with(ActivityId::new_uuid(), |builder| { - let (mut primary, primary_dispatch) = spawn_primary_leg( - builder, - Some(Region::EAST_US_2), - RequestedRegionReason::Initial, - ); + let (mut primary, primary_dispatch) = + spawn_primary_leg(builder, Some(Region::EAST_US_2), ExecutionContext::Initial); let h = primary.start_test_request( ExecutionContext::Initial, Some(Region::EAST_US_2), @@ -5968,19 +5855,19 @@ mod tests { vec![ RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::Initial, + reason: ExecutionContext::Initial, }, RequestedRegion { region: Region::WEST_US_2, - reason: RequestedRegionReason::Hedging, + reason: ExecutionContext::Hedging, }, RequestedRegion { region: Region::EAST_US_2, - reason: RequestedRegionReason::Initial, + reason: ExecutionContext::Initial, }, RequestedRegion { region: Region::CENTRAL_US, - reason: RequestedRegionReason::Hedging, + reason: ExecutionContext::Hedging, }, ] ); @@ -5990,36 +5877,6 @@ mod tests { ); } - #[test] - fn requested_region_reason_mapping_is_total() { - // A wildcard-free match forces this to stay total as variants are added. - #[allow(deprecated)] - let all = [ - ExecutionContext::Initial, - ExecutionContext::Retry, - ExecutionContext::OperationRetry, - ExecutionContext::TransportRetry, - ExecutionContext::Hedging, - ExecutionContext::RegionFailover, - ExecutionContext::CircuitBreakerProbe, - ]; - for ctx in all { - let reason = RequestedRegionReason::from(ctx); - #[allow(deprecated)] - let expected = match ctx { - ExecutionContext::Initial => RequestedRegionReason::Initial, - ExecutionContext::Retry | ExecutionContext::OperationRetry => { - RequestedRegionReason::OperationRetry - } - ExecutionContext::TransportRetry => RequestedRegionReason::TransportRetry, - ExecutionContext::Hedging => RequestedRegionReason::Hedging, - ExecutionContext::RegionFailover => RequestedRegionReason::RegionFailover, - ExecutionContext::CircuitBreakerProbe => RequestedRegionReason::CircuitBreakerProbe, - }; - assert_eq!(reason, expected); - } - } - // ========================================================================= // Pipeline/Transport/RequestSentStatus tests (merged from request_diagnostics.rs) // ========================================================================= diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs index e64cb1b223c..f671ce376a4 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs @@ -27,8 +27,8 @@ pub(crate) use diagnostics_context::DiagnosticsContextBuilder; pub use diagnostics_context::{ DiagnosticsContext, ExecutionContext, FailedTransportShardDiagnostics, PipelineType, RequestDiagnostics, RequestEvent, RequestEventType, RequestHandle, RequestSentStatus, - RequestedRegion, RequestedRegionReason, ThresholdBreach, TransportHttpVersion, TransportKind, - TransportSecurity, TransportShardDiagnostics, + RequestedRegion, ThresholdBreach, TransportHttpVersion, TransportKind, TransportSecurity, + TransportShardDiagnostics, }; pub use proxy_configuration::ProxyConfiguration; diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs index 5d30ac6756f..a7a87424595 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs @@ -3113,7 +3113,7 @@ async fn execute_hedged( // into the future. `leg_dispatch` reads the leg's launch instant, so the // fan-out record orders correctly against every attempt in the operation. let primary_dispatch = - primary_diag.leg_dispatch(primary_region.clone(), primary_execution_context.into()); + primary_diag.leg_dispatch(primary_region.clone(), primary_execution_context); let primary_attempt = Box::pin(async move { let mut diag = primary_diag; // Primary is launched before Stage 2 elapses, so no shared @@ -3323,7 +3323,7 @@ async fn execute_hedged( let secondary_diag = parent_diagnostics.clone_for_hedge_attempt(); let secondary_dispatch = secondary_diag.leg_dispatch( secondary_region.clone(), - crate::diagnostics::RequestedRegionReason::Hedging, + crate::diagnostics::ExecutionContext::Hedging, ); parent_diagnostics.record_hedge_fanout(primary_dispatch, secondary_dispatch); let secondary_attempt = Box::pin(async move { From 98a6cde0b355b1b8fa7db49bb331b328009e9720 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Tue, 4 Aug 2026 11:25:38 -0700 Subject: [PATCH 16/21] Remove issue links from spec and CHANGELOG - Remove tracking issue link from HEDGING_DETECTION_API_SPEC.md header - Remove PR links from spec (lines formerly referencing previous PRs) - Simplify Section 3 in spec: remove deprecated Retry compat text - Remove Retry (deprecated) row from reason-mapping table in spec - Remove #4410 issue links from azure_data_cosmos CHANGELOG entries Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d51868ed-95ef-4dff-b237-4f8226e5ec9a --- sdk/cosmos/azure_data_cosmos/CHANGELOG.md | 6 ++-- .../docs/HEDGING_DETECTION_API_SPEC.md | 30 ++++++------------- 2 files changed, 12 insertions(+), 24 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md index a36b326e551..e16f5ff3755 100644 --- a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md @@ -7,18 +7,18 @@ - Added an SDK-generated `x-ms-client-id` header that remains stable for each `CosmosClient`. ([#4844](https://github.com/Azure/azure-sdk-for-rust/pull/4844)) - Added opt-in Cosmos binary JSON encoding for item operations (`create`/`read`/`replace`/`upsert`). Enable it via `CosmosClientBuilder::with_binary_encoding_options` (or the `AZURE_COSMOS_BINARY_ENCODING_ENABLED` environment-variable fallback). Off by default; when disabled, requests and responses are byte-for-byte unchanged. ([#4671](https://github.com/Azure/azure-sdk-for-rust/pull/4671)) - Added `FeedOptions::max_fan_out` (and `FeedOptions::with_max_fan_out`) to cap how many physical partitions a cross-partition query or change feed may fan out to. Applies to `ContainerClient::query_items` and `ContainerClient::query_change_feed`. The cap is enforced only at initial query setup; a partition that splits mid-execution and pushes the fan-out higher does not abort the operation. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855)) -- Surfaced the Hedging Detection API through `DiagnosticsContext`: `hedging_started()`, `requested_regions()`, `responded_regions()`, and the exact-count `total_requested_regions()` / `total_responded_regions()`, and re-exported the `RequestedRegion` / `RequestedRegionReason` types (mirroring the existing `DiagnosticsContext` re-export). ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Surfaced the Hedging Detection API through `DiagnosticsContext`: `hedging_started()`, `requested_regions()`, `responded_regions()`, and the exact-count `total_requested_regions()` / `total_responded_regions()`, and re-exported the `RequestedRegion` / `RequestedRegionReason` types (mirroring the existing `DiagnosticsContext` re-export). ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Added a pluggable client-side diagnostics emission layer — the `DiagnosticsHandler` trait and ordered `DiagnosticsHandlerChain` (registered via `CosmosClientBuilder::with_diagnostics_handler`) — invoked once per operation (singleton and paginated, on success and failure) with the completed `DiagnosticsContext` plus an SDK-supplied `CosmosOperationContext`; the empty default chain is a zero-overhead no-op. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added the `metrics`-gated `CosmosMetricsHandler` (with `MetricsOptions`), emitting the stable `db.client.operation.duration` histogram plus per-signal opt-in metrics (`with_request_charge_metric`, `with_returned_rows_metric`) and an opt-in extended attribute set (`with_extended_attributes`); a no-op when no meter provider is registered. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added `MetricsOptions::with_active_instance_metric`, an opt-in `azure.cosmosdb.client.active_instance.count` up-down counter reporting the number of live `CosmosClient` instances per account endpoint, keyed on `server.address` (plus `server.port` for a non-default port). ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) - Added `DiagnosticsHandler::on_client_created`, a defaulted hook that lets a handler observe client construction (`CosmosClientInfo`) and return a `ClientLifetimeToken` dropped with the client, for handlers that need to track client lifetime. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) - Added composable tail-sampled emission handlers — a `TracingLogHandler` leaf that writes a compact `tracing` line and a `SamplingLogHandler` wrapper (holding an `Arc`) that applies the sampling gate plus a shared per-window rate limit, defaulting to wrap a `TracingLogHandler` — and the `distributed_tracing`-gated `CosmosTracingHandler` (backdated span tree), also rate-limited so an error storm can't overwhelm exporters. All emit only for operations which fail or breach a configurable `DiagnosticsThresholds`, and stamp *why* they were sampled (a failure, or which threshold) on the emitted line and span. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) -- Surfaced hedging through the observability handlers, reusing the Hedging Detection API. When a cross-region hedge fans out, `CosmosTracingHandler` adds `azure.cosmosdb.operation.{hedging_started,hedge_region,hedge_terminal_state}` plus `requested_regions`/`responded_regions` (`string[]`, bounded by the driver's `max_request_diagnostics`, with a `*_total` companion attribute emitted only when a retry storm truncated the history) to the sampled operation span and tags the hedge-leg child span (`azure.cosmosdb.request.hedge`); `SamplingLogHandler` adds `hedging_started` / `hedge_region` / `hedge_terminal_state` fields to the sampled log line; and `CosmosMetricsHandler` gains an opt-in `azure.cosmosdb.client.operation.hedged` counter (`MetricsOptions::with_hedged_metric`) carrying the low-cardinality `hedge_terminal_state`, with the higher-cardinality `hedge_region` dimension added only under `with_extended_attributes`. All three surfaces decide fan-out from `hedging_started()`, so a hedge whose race ended both-transient and was then resolved by a failover attempt is still reported; the per-outcome `hedge_region` / `hedge_terminal_state` fields are omitted there (the counter's dimension carries the `unresolved` sentinel so its attribute schema stays uniform). ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Surfaced hedging through the observability handlers, reusing the Hedging Detection API. When a cross-region hedge fans out, `CosmosTracingHandler` adds `azure.cosmosdb.operation.{hedging_started,hedge_region,hedge_terminal_state}` plus `requested_regions`/`responded_regions` (`string[]`, bounded by the driver's `max_request_diagnostics`, with a `*_total` companion attribute emitted only when a retry storm truncated the history) to the sampled operation span and tags the hedge-leg child span (`azure.cosmosdb.request.hedge`); `SamplingLogHandler` adds `hedging_started` / `hedge_region` / `hedge_terminal_state` fields to the sampled log line; and `CosmosMetricsHandler` gains an opt-in `azure.cosmosdb.client.operation.hedged` counter (`MetricsOptions::with_hedged_metric`) carrying the low-cardinality `hedge_terminal_state`, with the higher-cardinality `hedge_region` dimension added only under `with_extended_attributes`. All three surfaces decide fan-out from `hedging_started()`, so a hedge whose race ended both-transient and was then resolved by a failover attempt is still reported; the per-outcome `hedge_region` / `hedge_terminal_state` fields are omitted there (the counter's dimension carries the `unresolved` sentinel so its attribute schema stays uniform). ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Added the non-default `control_plane` feature that gates the control-plane APIs (database and container CRUD, and throughput/offer management). It is intentionally independent of `key_auth` so these APIs are not tied to key-based authentication. ([#4854](https://github.com/Azure/azure-sdk-for-rust/pull/4854)) ### Breaking Changes -- Serialized `DiagnosticsContext` output (the diagnostics JSON surfaced to consumers, and the sampled diagnostics log line) now serializes driver-generated operation retries with `execution_context` = `"operation_retry"` instead of `"retry"`. The driver's `ExecutionContext::Retry` variant has been removed entirely; use `ExecutionContext::OperationRetry` instead. Telemetry/log parsers that match the literal `"retry"` execution context must update. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Serialized `DiagnosticsContext` output (the diagnostics JSON surfaced to consumers, and the sampled diagnostics log line) now serializes driver-generated operation retries with `execution_context` = `"operation_retry"` instead of `"retry"`. The driver's `ExecutionContext::Retry` variant has been removed entirely; use `ExecutionContext::OperationRetry` instead. Telemetry/log parsers that match the literal `"retry"` execution context must update. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Control-plane APIs are now gated behind the new `control_plane` feature, which is **not** enabled by default. Code using database or container management (`CosmosClient::create_database`/`query_databases`, `DatabaseClient::read`/`create_container`/`query_containers`/`delete`, `ContainerClient::replace`/`delete`), throughput management (`read_throughput`/`begin_replace_throughput`, `ThroughputPoller`), or the associated model and options types (`DatabaseProperties`, `ThroughputProperties`, and the container create/replace/delete/query, database, and throughput option types) must now enable the `control_plane` feature. Reading container properties via `ContainerClient::read()` — along with `ContainerProperties`, `IndexingPolicy`, `ResourceResponse`, and `ReadContainerOptions` — remains available without the feature, since it works with Entra ID authentication and mirrors the metadata read the SDK already performs internally. ([#4854](https://github.com/Azure/azure-sdk-for-rust/pull/4854)) - Fresh cross-partition queries and change feed reads now fail with a `BadRequest` error if they would fan out to more than 100 physical partitions. Raise `FeedOptions::max_fan_out` to run a broader operation. The limit is checked only at initial query setup — resuming from a continuation token is unaffected, and a partition split that raises the fan-out mid-execution does not abort the operation. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855)) diff --git a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md index e1c9b501e92..2efdd87712c 100644 --- a/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md +++ b/sdk/cosmos/azure_data_cosmos/docs/HEDGING_DETECTION_API_SPEC.md @@ -1,7 +1,6 @@ # Hedging Detection API — Spec **Status:** Implemented on `main`. -**Tracking issue:** [Azure/azure-sdk-for-rust#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410) **Cross-SDK contract:** The Azure Cosmos DB SDKs are converging on a "Hedging Detection" capability exposed on each SDK's per-operation diagnostics surface. This document specifies how the Rust SDK satisfies that contract. @@ -53,11 +52,9 @@ SDK depends on the driver (never the reverse), the diagnostics model is driver-owned and re-exported by `azure_data_cosmos`, exactly like `DiagnosticsContext` itself. -The hedging orchestrator/dispatch is **landed** on `main` -([#4432](https://github.com/Azure/azure-sdk-for-rust/pull/4432)): it emits +The hedging orchestrator/dispatch is **landed** on `main`: it emits `ExecutionContext::Hedging` for alternate legs and populates `HedgeDiagnostics` -(design: [`HEDGING_SPEC.md`](https://github.com/Azure/azure-sdk-for-rust/blob/main/sdk/cosmos/azure_data_cosmos_driver/docs/HEDGING_SPEC.md), -[PR #4330](https://github.com/Azure/azure-sdk-for-rust/pull/4330)). +(design: [`HEDGING_SPEC.md`](https://github.com/Azure/azure-sdk-for-rust/blob/main/sdk/cosmos/azure_data_cosmos_driver/docs/HEDGING_SPEC.md)). --- @@ -72,8 +69,6 @@ The hedging orchestrator/dispatch is **landed** on `main` #[non_exhaustive] pub enum ExecutionContext { Initial, - #[deprecated(since = "0.7.0", note = "use `ExecutionContext::OperationRetry`")] - Retry, OperationRetry, // was: Retry TransportRetry, Hedging, @@ -82,19 +77,12 @@ pub enum ExecutionContext { } ``` -`Retry` is renamed to `OperationRetry` so the operation-level retry reason is -clearly distinct from the transport-level `TransportRetry`. The hand-written -`ExecutionContext::as_str()` and every dispatch site (`operation_pipeline.rs`, -`transport_pipeline.rs`, `cosmos_driver.rs`) are updated accordingly. - -**Compatibility.** The old `Retry` variant is retained for one release as a -distinct `#[deprecated]` variant (**not** a serde alias) so existing source keeps -compiling; a `Retry` value still serializes as `"retry"`. The customer-visible -wire-format change is that the dispatch sites now emit `OperationRetry` for -driver-generated operation retries, so those attempts serialize as -`"operation_retry"` instead of `"retry"`; telemetry parsers that match the -literal `"retry"` execution context must update. (`ExecutionContext` derives -`Serialize` only, not `Deserialize`, so no `#[serde(alias)]` is needed.) +`Retry` has been removed and replaced by `OperationRetry` so the operation-level +retry reason is clearly distinct from the transport-level `TransportRetry`. The +hand-written `ExecutionContext::as_str()` and every dispatch site +(`operation_pipeline.rs`, `transport_pipeline.rs`, `cosmos_driver.rs`) are updated +accordingly. Telemetry parsers that matched the literal `"retry"` execution context +must update; serialized output now emits `"operation_retry"` instead. --- @@ -109,7 +97,7 @@ projected from the driver-internal `ExecutionContext` via a **total** | `ExecutionContext` | `RequestedRegionReason` | | --- | --- | | `Initial` | `Initial` | -| `Retry` (deprecated) / `OperationRetry` | `OperationRetry` | +| `OperationRetry` | `OperationRetry` | | `TransportRetry` | `TransportRetry` | | `Hedging` | `Hedging` | | `RegionFailover` | `RegionFailover` | From 5ab7a3d34b07c136211116944266275f9a41006d Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Tue, 4 Aug 2026 12:02:20 -0700 Subject: [PATCH 17/21] Resolve PR #4871 merge conflicts Resolve upstream/main merge conflicts in Cosmos hedging files and keep compatible changelog entries for both streams. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d51868ed-95ef-4dff-b237-4f8226e5ec9a --- sdk/cosmos/azure_data_cosmos/CHANGELOG.md | 2 +- .../src/diagnostics/metrics/attributes.rs | 53 ++ .../src/diagnostics/metrics/handler.rs | 78 +++ .../src/diagnostics/metrics/instruments.rs | 3 + .../azure_data_cosmos_driver/CHANGELOG.md | 4 + .../src/driver/cosmos_driver.rs | 215 ++++++-- .../src/driver/pipeline/operation_pipeline.rs | 475 ++++++++++++++++-- 7 files changed, 754 insertions(+), 76 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md index e16f5ff3755..a0633995d72 100644 --- a/sdk/cosmos/azure_data_cosmos/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos/CHANGELOG.md @@ -9,7 +9,7 @@ - Added `FeedOptions::max_fan_out` (and `FeedOptions::with_max_fan_out`) to cap how many physical partitions a cross-partition query or change feed may fan out to. Applies to `ContainerClient::query_items` and `ContainerClient::query_change_feed`. The cap is enforced only at initial query setup; a partition that splits mid-execution and pushes the fan-out higher does not abort the operation. ([#4855](https://github.com/Azure/azure-sdk-for-rust/pull/4855)) - Surfaced the Hedging Detection API through `DiagnosticsContext`: `hedging_started()`, `requested_regions()`, `responded_regions()`, and the exact-count `total_requested_regions()` / `total_responded_regions()`, and re-exported the `RequestedRegion` / `RequestedRegionReason` types (mirroring the existing `DiagnosticsContext` re-export). ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Added a pluggable client-side diagnostics emission layer — the `DiagnosticsHandler` trait and ordered `DiagnosticsHandlerChain` (registered via `CosmosClientBuilder::with_diagnostics_handler`) — invoked once per operation (singleton and paginated, on success and failure) with the completed `DiagnosticsContext` plus an SDK-supplied `CosmosOperationContext`; the empty default chain is a zero-overhead no-op. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) -- Added the `metrics`-gated `CosmosMetricsHandler` (with `MetricsOptions`), emitting the stable `db.client.operation.duration` histogram plus per-signal opt-in metrics (`with_request_charge_metric`, `with_returned_rows_metric`) and an opt-in extended attribute set (`with_extended_attributes`); a no-op when no meter provider is registered. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) +- Added the `metrics`-gated `CosmosMetricsHandler` (with `MetricsOptions`), emitting the stable `db.client.operation.duration` histogram plus per-signal opt-in metrics (`with_request_charge_metric`, `with_returned_rows_metric`) and an opt-in extended attribute set (`with_extended_attributes`); a no-op when no meter provider is registered. Each histogram declares explicit bucket boundaries (`attributes::BUCKETS_OPERATION_DURATION_SECONDS`, `BUCKETS_REQUEST_CHARGE_RU`, `BUCKETS_RETURNED_ROWS`) rather than inheriting OpenTelemetry's millisecond-scaled defaults, which would collapse every real observation of the seconds-valued duration metric into a single bucket and make latency percentiles constant. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) - Added `MetricsOptions::with_active_instance_metric`, an opt-in `azure.cosmosdb.client.active_instance.count` up-down counter reporting the number of live `CosmosClient` instances per account endpoint, keyed on `server.address` (plus `server.port` for a non-default port). ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) - Added `DiagnosticsHandler::on_client_created`, a defaulted hook that lets a handler observe client construction (`CosmosClientInfo`) and return a `ClientLifetimeToken` dropped with the client, for handlers that need to track client lifetime. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) - Added composable tail-sampled emission handlers — a `TracingLogHandler` leaf that writes a compact `tracing` line and a `SamplingLogHandler` wrapper (holding an `Arc`) that applies the sampling gate plus a shared per-window rate limit, defaulting to wrap a `TracingLogHandler` — and the `distributed_tracing`-gated `CosmosTracingHandler` (backdated span tree), also rate-limited so an error storm can't overwhelm exporters. All emit only for operations which fail or breach a configurable `DiagnosticsThresholds`, and stamp *why* they were sampled (a failure, or which threshold) on the emitted line and span. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789)) diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs index 68674c8cdd4..665930aa8f2 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs @@ -64,6 +64,59 @@ pub const UNIT_OPERATION: &str = "{operation}"; /// Unit for [`METRIC_ACTIVE_INSTANCE_COUNT`] — client instances. pub const UNIT_INSTANCE: &str = "{instance}"; +// ========================================================================= +// Explicit histogram bucket boundaries +// ========================================================================= +// +// These MUST be set on every histogram we create. OpenTelemetry's default +// boundaries are `[0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, +// 7500, 10000]`, which are scaled for *milliseconds*. `db.client.operation.duration` +// is recorded in *seconds*, so a typical few-millisecond Cosmos operation +// (0.003 s) lands in the very first bucket and every observation piles up there. +// A histogram whose observations all share one bucket carries no information: +// `histogram_quantile` degenerates to linear interpolation within that bucket +// and returns a constant that depends only on the requested quantile, not on +// the data. Latency percentiles then look plausible while being unable to +// register any change at all — so latency dashboards and alerts silently stop +// working rather than visibly breaking. +// +// The boundaries below come from the OpenTelemetry semantic conventions' +// per-instrument bucket *advice*, which exists for exactly this reason. Using +// the advised values (rather than hand-picked ones) also keeps these histograms +// directly comparable with the other Azure Cosmos DB SDKs. + +/// Bucket boundaries for [`METRIC_OPERATION_DURATION`], in seconds. +/// +/// Semconv advice for `db.client.operation.duration`. Spans sub-millisecond +/// (cache/emulator) through 10 s (a badly degraded or retried request). +pub const BUCKETS_OPERATION_DURATION_SECONDS: &[f64] = + &[0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0]; + +/// Bucket boundaries for [`METRIC_OPERATION_REQUEST_CHARGE`], in request units. +/// +/// Cosmos-specific, so there is no semconv advice to follow. Chosen to give +/// resolution where Cosmos operations actually sit: a point read is ~1 RU and a +/// small write ~5-10 RU, so the low end is finely divided, while cross-partition +/// queries reaching into the thousands still land in a meaningful bucket rather +/// than overflowing. +pub const BUCKETS_REQUEST_CHARGE_RU: &[f64] = &[ + 1.0, 2.5, 5.0, 10.0, 25.0, 50.0, 100.0, 250.0, 500.0, 1000.0, 2500.0, 5000.0, +]; + +/// Bucket boundaries for [`METRIC_RESPONSE_RETURNED_ROWS`], in rows. +/// +/// Semconv advice for `db.client.response.returned_rows`. +pub const BUCKETS_RETURNED_ROWS: &[f64] = &[ + 1.0, + 10.0, + 100.0, + 1000.0, + 10000.0, + 100_000.0, + 1_000_000.0, + 10_000_000.0, +]; + // ========================================================================= // Stable attributes (always emitted; operation scope, low cardinality) // diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs index 426765f3f03..89048d85ef3 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs @@ -576,6 +576,84 @@ mod tests { assert!(!attrs.contains_key(attributes::ATTR_SUB_STATUS_CODE)); } + /// Returns `(bounds, bucket_counts)` for the duration histogram. + fn duration_buckets(metrics: &[ResourceMetrics]) -> Option<(Vec, Vec)> { + for rm in metrics { + for sm in rm.scope_metrics() { + for m in sm.metrics() { + if m.name() != attributes::METRIC_OPERATION_DURATION { + continue; + } + if let AggregatedMetrics::F64(MetricData::Histogram(histogram)) = m.data() { + if let Some(point) = histogram.data_points().next() { + return Some(( + point.bounds().collect(), + point.bucket_counts().collect(), + )); + } + } + } + } + } + None + } + + /// The duration histogram must be able to *distinguish* realistic Cosmos + /// latencies, which is the whole point of recording it as a histogram. + /// + /// This guards a failure mode that is invisible from the outside: the metric + /// is recorded in seconds, but OpenTelemetry's default bucket boundaries are + /// scaled for milliseconds. With those defaults every real operation lands in + /// the first bucket, `histogram_quantile` degenerates to interpolation inside + /// that one bucket, and p50/p95/p99 become constants that depend only on the + /// quantile requested. Dashboards keep drawing plausible-looking lines that + /// can never move, so a latency regression cannot be detected — and nothing + /// errors to say so. + /// + /// Asserting on *separation* rather than on the literal boundary list keeps + /// this a test of the property we care about: the boundaries stay free to be + /// re-tuned, as long as they still resolve these three latencies apart. + #[test] + fn duration_histogram_separates_realistic_latencies() { + let harness = test_meter(); + let handler = CosmosMetricsHandler::with_meter(harness.meter.clone()); + let cx = Context::new().with_value(operation_context()); + + // A fast point read, a slow-but-normal query, and a degraded request. + for millis in [2_u64, 30, 300] { + handler.handle( + &DiagnosticsContext::for_testing_completed( + ActivityId::new_uuid(), + Duration::from_millis(millis), + Some(CosmosStatus::new(StatusCode::from(200))), + ), + &cx, + ); + } + + let metrics = harness.collect(); + let (bounds, counts) = + duration_buckets(&metrics).expect("duration histogram should be emitted"); + + assert_eq!(counts.iter().sum::(), 3, "all observations recorded"); + + let occupied = counts.iter().filter(|c| **c > 0).count(); + assert_eq!( + occupied, 3, + "2ms/30ms/300ms must land in 3 distinct buckets, but they occupy {occupied}; \ + boundaries are {bounds:?}. Boundaries scaled for milliseconds collapse every \ + real latency into one bucket and make percentiles constant." + ); + + // Guard the specific regression: seconds-valued data against + // millisecond-scaled boundaries. 10s is a degraded request, not a + // routine one, so nothing sane needs a boundary above it. + assert!( + bounds.iter().all(|b| *b <= 10.0), + "boundaries look millisecond-scaled for a seconds-valued metric: {bounds:?}" + ); + } + #[test] fn failure_sets_error_type_to_status_code() { let harness = test_meter(); diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs index c70a3cced85..2b5c950433d 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs @@ -53,18 +53,21 @@ impl Instruments { .f64_histogram(attributes::METRIC_OPERATION_DURATION) .with_unit(attributes::UNIT_SECONDS) .with_description("Total client-observed duration of a Cosmos DB operation.") + .with_boundaries(attributes::BUCKETS_OPERATION_DURATION_SECONDS.to_vec()) .build(); let request_charge = meter .f64_histogram(attributes::METRIC_OPERATION_REQUEST_CHARGE) .with_unit(attributes::UNIT_REQUEST_UNIT) .with_description("Request charge (RU) consumed by a Cosmos DB operation.") + .with_boundaries(attributes::BUCKETS_REQUEST_CHARGE_RU.to_vec()) .build(); let returned_rows = meter .u64_histogram(attributes::METRIC_RESPONSE_RETURNED_ROWS) .with_unit(attributes::UNIT_ROW) .with_description("Number of rows/items returned by a Cosmos DB operation.") + .with_boundaries(attributes::BUCKETS_RETURNED_ROWS.to_vec()) .build(); let hedged = meter diff --git a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md index 79a172470fc..6e77d05778a 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md +++ b/sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md @@ -11,6 +11,9 @@ - Added `RequestDiagnostics::operation_name`, naming the operation that issued an individual attempt. It is set only where one `DiagnosticsContext` aggregates attempts from more than one operation — today a PATCH, whose attempts report `patch_read_item` / `patch_replace_item` while the context reports `patch_item` — and is `None` otherwise, meaning the attempt shares the context's operation name. `CosmosOperation::is_patch_sub_operation` reports the same distinction on the operation itself. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874)) - Added the Hedging Detection API on `DiagnosticsContext`: `hedging_started()` (whether a hedge arm actually fanned out), `requested_regions()` (regions dispatched to, in dispatch order with duplicates, each tagged with a reason), and `responded_regions()` (regions that produced an actual service reply, in completion order). All three are materialized at finalization from the full pre-compaction attempt list — including attempts rescued from a structurally-dropped hedge leg — plus a dispatch-time hedge fan-out log, so a cancelled hedge leg, a retry storm that compacts `requests()`, and multi-round-trip aggregation all report a complete history. Both region histories are bounded by `max_request_diagnostics` (keeping head and tail, dropping the repetitive middle) so a retry storm cannot produce an unbounded diagnostics artifact; `total_requested_regions()` / `total_responded_regions()` report the exact pre-truncation counts, so truncation is detectable rather than silent. Adds the public `RequestedRegion` struct and the new `ExecutionContext::OperationRetry` variant. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Added `HedgeTerminalState::as_str()` (and a matching `Display`) returning a stable, low-cardinality `snake_case` identifier for the hedging race outcome, for use as an observability attribute / log-field value. ([#4410](https://github.com/Azure/azure-sdk-for-rust/issues/4410), [#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Extended cross-region hedging to the container and partition-key-range metadata reads, so a slow (but not failed) region no longer stalls a client's first operation against a container. Metadata hedges use a fixed 1.5s threshold and never let a hedged region override a definitive primary result. ([#4896](https://github.com/Azure/azure-sdk-for-rust/pull/4896)) +- Added `HedgingOptions` (via `DriverOptionsBuilder::with_hedging_options`), bounding how many metadata operations may make simultaneous cross-region attempts. Defaults to 32; `0` disables metadata hedging. An operation refused a slot follows the ordinary sequential failover path instead of queueing. Data-plane hedging is not budgeted — tracked by [#4916](https://github.com/Azure/azure-sdk-for-rust/issues/4916). ([#4896](https://github.com/Azure/azure-sdk-for-rust/pull/4896)) +- Added `CosmosResponse::serving_region`, returning the region that produced a response — the hedge winner when the operation raced, otherwise the region of the final attempt. ([#4896](https://github.com/Azure/azure-sdk-for-rust/pull/4896)) ### Breaking Changes @@ -25,6 +28,7 @@ - Fixed the primary leg of a hedged request being recorded with `ExecutionContext::Initial` even when the hedge was dispatched from a retry. The primary leg now carries the execution context computed from the live retry state, so a hedge that upgraded a session retry or a region failover is no longer misreported as a first attempt in diagnostics. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Fixed a hedged operation losing the diagnostics of the leg that lost the race. Each leg records into its own builder, and the race cancels the loser by dropping its future, which also dropped every attempt that leg had already completed — so a leg that received a `429` and was retrying when it lost contributed nothing to `request_count()`, `regions_contacted()`, or `total_request_charge()`, under-reporting the RU the account was actually billed. Completed attempts are now mirrored into an operation-scoped hedge journal and folded back in at finalization. An attempt still in flight when its leg was cancelled observed no reply and remains unreported. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) - Fixed the finalized attempt list interleaving hedge legs by leg rather than by dispatch time. Attempts from both legs plus the hedge fan-out are now ordered by their actual dispatch instant, so a primary-leg retry that was dispatched before the alternate fanned out is reported before it instead of after. ([#4871](https://github.com/Azure/azure-sdk-for-rust/pull/4871)) +- Fixed a partition-key-range refresh that could remain permanently pinned to an unreachable region. A refresh resuming a region-affine change-feed continuation is routed back to the region that served it, with hedging suppressed; if that region then became unavailable, every subsequent forced refresh repeated the same failing request forever. Such a refresh now retries once from cold, which clears both the continuation and the region pin together. The same clearing now also applies when an incremental routing-map merge falls back to a full refresh that comes back empty, which previously left the continuation in place after the pin protecting it had already been released. ([#4896](https://github.com/Azure/azure-sdk-for-rust/pull/4896)) ### Other Changes diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/cosmos_driver.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/cosmos_driver.rs index c2bde9f3932..4fbabde75bb 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/cosmos_driver.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/cosmos_driver.rs @@ -20,7 +20,8 @@ use crate::{ ThrottleRetryState, METADATA_MAX_PER_RETRY_DELAY, METADATA_MAX_THROTTLE_ATTEMPTS, METADATA_MAX_THROTTLE_WAIT, }, - pipeline::operation_pipeline::OperationOverrides, + pipeline::hedge_budget::HedgeBudget, + pipeline::operation_pipeline::{OperationOverrides, RegionPin}, routing::{ partition_key_range_id::PartitionKeyRangeId, session_manager::SessionManager, CosmosEndpoint, LocationStateStore, @@ -41,8 +42,9 @@ use crate::{ }; use arc_swap::ArcSwap; use futures::future::BoxFuture; +use std::collections::HashMap; use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::Arc; +use std::sync::{Arc, Mutex}; use std::time::Duration; #[cfg(feature = "preview_dtx")] use std::time::Instant; @@ -136,6 +138,20 @@ struct DriverRequestExecutor<'a> { options: &'a OperationOptions, } +/// Region pins for the PartitionKeyRange change feed, keyed by container. +/// +/// The `/pkranges` change-feed continuation is an ETag that only the region +/// which issued it can interpret, and [`PartitionKeyRangeCache`] persists that +/// continuation in `ContainerRoutingMap::change_feed_next_if_none_match` across +/// fetch operations — well beyond the lifetime of the closure that produced it. +/// Pinning per fetcher closure would therefore leak the token into normal +/// routing on the next refresh, so the pin lives at driver scope alongside the +/// cache whose entries it protects. +/// +/// A cold read (no carried continuation) starts a brand new ETag chain, so it +/// clears any existing pin and installs the region that served it. +type PkRangeRegionPins = Mutex>; + fn request_target_overrides( operation_partition_key: Option<&PartitionKey>, target: RequestTarget, @@ -180,6 +196,7 @@ fn request_target_overrides( // backend returns every document in the physical partition. partition_key: operation_partition_key.cloned(), continuation, + ..Default::default() }, RequestTarget::NonPartitioned => OperationOverrides { continuation, @@ -265,6 +282,13 @@ pub struct CosmosDriver { /// Used to pre-resolve partition key range IDs for PPAF/PPCB /// before the first request attempt. pk_range_cache: PartitionKeyRangeCache, + /// Region pins protecting the change-feed continuations held by + /// `pk_range_cache`. See [`PkRangeRegionPins`]. + pk_range_region_pins: PkRangeRegionPins, + /// Per-client ceiling on metadata operations making simultaneous cross-region attempts. + /// Bounds the request amplification a hedging client can inflict on an + /// alternate region during a brownout. See [`HedgeBudget`]. + hedge_budget: HedgeBudget, /// Session token cache for session consistency. session_manager: SessionManager, /// Set to `true` after [`initialize()`](Self::initialize) completes successfully. @@ -1625,6 +1649,10 @@ impl CosmosDriver { // Clone the per-driver registry as-is for the request hot path. let throughput_control_groups = options.throughput_control_groups().clone(); + // Read the hedge ceiling once, here: it is fixed for the driver's + // lifetime, and `options` is moved into `Self` below. + let hedge_budget = HedgeBudget::new(options.hedging_options()); + Ok(Self { runtime, options, @@ -1636,6 +1664,8 @@ impl CosmosDriver { ))] endpoint_probe_fn: TestEndpointProbeFn(endpoint_probe_fn_for_tests), pk_range_cache: PartitionKeyRangeCache::new(), + pk_range_region_pins: Mutex::new(HashMap::new()), + hedge_budget, session_manager: SessionManager::new(), initialized: AtomicBool::new(false), user_agent, @@ -2059,7 +2089,8 @@ impl CosmosDriver { &self, container: ContainerReference, continuation: Option, - ) -> Option { + region_pin: Option, + ) -> (Option, Option) { // Build the operation through the standard pipeline to get correct // URL construction, signing, and cross-region retry behavior. let mut operation = CosmosOperation::read_all_partition_key_ranges(container.clone()); @@ -2076,23 +2107,44 @@ impl CosmosDriver { request_headers.max_item_count = Some(crate::models::MaxItemCountHint::ServerDecides); operation = operation.with_request_headers(request_headers); + // Hedging is decided entirely by `region_pin`: absent (a cold read) the + // request is hedge-eligible; present it is not. The pin carries that + // suppression itself rather than going through + // `AvailabilityStrategy::Disabled`, which the + // `AZURE_COSMOS_HEDGING_ENABLED` env switch is allowed to override — + // a region-affine continuation must never be raced regardless of + // configuration. The pin's endpoint additionally routes the request back + // to the region that served the cold page, so a failover retry cannot + // move the chain either. let options = OperationOptions::default(); + let overrides = OperationOverrides { + region_pin: region_pin.map(Box::new), + ..Default::default() + }; match self - .execute_operation_direct(&operation, OperationOverrides::default(), &options) + .execute_operation_direct(&operation, overrides, &options) .await { Ok(response) => { + // Capture the region that served this page before the response + // body is consumed, so the caller can pin subsequent + // change-feed pages to it. + let serving_endpoint = self.response_endpoint(&response); + let etag = response.headers().etag.as_ref().map(|e| e.to_string()); // 304 Not Modified is a success outcome for conditional // changefeed reads: the cached routing map is still current. if response.status().status_code() == azure_core::http::StatusCode::NotModified { - return Some(PkRangeFetchResult { - ranges: vec![], - continuation, - not_modified: true, - }); + return ( + Some(PkRangeFetchResult { + ranges: vec![], + continuation, + not_modified: true, + }), + serving_endpoint, + ); } let body_bytes = match response.into_body().single() { @@ -2102,21 +2154,24 @@ impl CosmosDriver { container = %container.name(), "Partition key ranges response was a feed body, expected single payload" ); - return None; + return (None, serving_endpoint); } }; match parse_pk_ranges_response(&body_bytes) { - Some(ranges) => Some(PkRangeFetchResult { - ranges, - continuation: etag, - not_modified: false, - }), + Some(ranges) => ( + Some(PkRangeFetchResult { + ranges, + continuation: etag, + not_modified: false, + }), + serving_endpoint, + ), None => { tracing::error!( container = %container.name(), "Failed to parse partition key ranges response body" ); - None + (None, serving_endpoint) } } } @@ -2146,7 +2201,7 @@ impl CosmosDriver { error = %e, "Permanent error fetching partition key ranges — check account credentials and container existence" ); - return None; + return (None, None); } } @@ -2155,11 +2210,103 @@ impl CosmosDriver { error = %e, "Transient error fetching partition key ranges from service after exhausting pipeline cross-region retries" ); - None + (None, None) } } } + /// Maps the region that produced `response` onto the account endpoint that + /// serves it. + /// + /// Used to pin the pages that follow a change-feed cold read. The ETag a + /// page returns is only meaningful to the region that issued it, so **every** + /// successful cold page must be recorded — not just the ones an alternate + /// region won via hedging. Recording only hedge wins would leave the common + /// primary-answered case unpinned, and a `FailoverRetry`/`SessionRetry` on a + /// later page would then be free to carry that region-affine ETag into a + /// different region. + /// + /// `None` when the response names no serving region, or when that region is + /// absent from the account's preferred read endpoints; the caller then falls + /// back to a pin that carries no endpoint but still forbids hedging. + fn response_endpoint( + &self, + response: &crate::models::CosmosResponse, + ) -> Option { + let region = response.serving_region()?; + let snapshot = self.location_state_store.snapshot(); + snapshot + .account + .preferred_read_endpoints + .iter() + .find(|ep| ep.region() == Some(®ion)) + .cloned() + } + + /// Builds the per-fetch-operation closure that the PartitionKeyRange cache + /// drives page-by-page. + /// + /// Encapsulates the change-feed hedging policy. A **cold** call (no carried + /// continuation) starts a fresh ETag chain, so it drops any existing pin for + /// the container and is hedge-eligible; whichever region serves it — the + /// primary or a hedge-winning alternate — is recorded as the container's + /// pin. Every call that carries a continuation is region-pinned instead: + /// hedging is suppressed, and the call is routed back to the recorded region + /// so neither a hedge nor a mid-page failover can carry the region-affine + /// ETag somewhere it means nothing. + /// + /// The pin is held on the driver rather than in the closure because the + /// cache persists the continuation past the closure's lifetime — see + /// [`PkRangeRegionPins`]. + /// + /// A pinned chain that fails is not left pinned: the cache discards the + /// failed continuation and retries the fetch cold, which routes back through + /// the `continuation.is_none()` branch below and clears the pin. Both halves + /// of the region-affine state therefore die together, so an unreachable + /// pinned region cannot wedge later force-refreshes. + fn pk_range_page_fetcher<'a>( + &'a self, + ) -> impl Fn(ContainerReference, Option) -> BoxFuture<'a, Option> + + Send + + 'a { + move |container, continuation| { + Box::pin(async move { + let region_pin = { + let mut pins = self + .pk_range_region_pins + .lock() + .expect("pk-range region pin mutex poisoned"); + if continuation.is_none() { + // A cold read starts a new ETag chain, so the previous + // chain's pin no longer applies and must not outlive it. + pins.remove(&container); + None + } else { + Some(RegionPin { + endpoint: pins.get(&container).cloned(), + }) + } + }; + let is_cold = region_pin.is_none(); + + let (result, serving_endpoint) = self + .fetch_pk_ranges_from_service(container.clone(), continuation, region_pin) + .await; + // Record the serving region for every successful cold page, so + // the continuation pages that follow are pinned to it. Pages + // that already carry a pin leave it untouched: the chain must + // stay on the region that opened it. + if let Some(endpoint) = serving_endpoint.filter(|_| is_cold) { + self.pk_range_region_pins + .lock() + .expect("pk-range region pin mutex poisoned") + .insert(container, endpoint); + } + result + }) + } + } + /// Pre-resolves the partition key range ID for a data plane operation. /// /// When PPAF/PPCB is enabled, seeds the partition key range ID before the @@ -2252,9 +2399,12 @@ impl CosmosDriver { if let Some(partition_key) = partition_key { return self .pk_range_cache - .resolve_partition_key_range_id(container, partition_key, false, |c, cont| { - Box::pin(self.fetch_pk_ranges_from_service(c, cont)) - }) + .resolve_partition_key_range_id( + container, + partition_key, + false, + self.pk_range_page_fetcher(), + ) .await .map(PartitionKeyRangeId::from); } @@ -2279,7 +2429,7 @@ impl CosmosDriver { container, target.min_inclusive()..target.max_exclusive(), false, - |c, cont| Box::pin(self.fetch_pk_ranges_from_service(c, cont)), + self.pk_range_page_fetcher(), ) .await .map(PartitionKeyRangeId::from) @@ -2699,9 +2849,7 @@ impl CosmosDriver { }; let mut topology = container.map(|c| { - CachedTopologyProvider::new(&self.pk_range_cache, c, |container, continuation| { - self.fetch_pk_ranges_from_service(container, continuation) - }) + CachedTopologyProvider::new(&self.pk_range_cache, c, self.pk_range_page_fetcher()) }); let mut context = PipelineContext::new( @@ -2844,6 +2992,7 @@ impl CosmosDriver { .default_consistency_level, effective_throughput_control, pre_resolved_pk_range_id, + &self.hedge_budget, ) .await } @@ -3055,9 +3204,7 @@ impl CosmosDriver { let mut topology = CachedTopologyProvider::new( &self.pk_range_cache, container_ref, - |container, continuation| { - self.fetch_pk_ranges_from_service(container, continuation) - }, + self.pk_range_page_fetcher(), ); let pipeline = planner::build_unordered_merge( &feed_range, @@ -3090,7 +3237,7 @@ impl CosmosDriver { let mut topology = CachedTopologyProvider::new( &self.pk_range_cache, container_ref, - |container, continuation| self.fetch_pk_ranges_from_service(container, continuation), + self.pk_range_page_fetcher(), ); let pipeline = @@ -3239,9 +3386,7 @@ impl CosmosDriver { ) -> Option> { let routing_map = self .pk_range_cache - .try_lookup(container, force_refresh, |c, cont| { - Box::pin(self.fetch_pk_ranges_from_service(c, cont)) - }) + .try_lookup(container, force_refresh, self.pk_range_page_fetcher()) .await?; let ranges = routing_map.ranges(); @@ -3284,9 +3429,7 @@ impl CosmosDriver { // Full key — point lookup let routing_map = self .pk_range_cache - .try_lookup(container, force_refresh, |c, cont| { - Box::pin(self.fetch_pk_ranges_from_service(c, cont)) - }) + .try_lookup(container, force_refresh, self.pk_range_page_fetcher()) .await?; if routing_map.ranges().is_empty() { return None; @@ -3304,7 +3447,7 @@ impl CosmosDriver { container, &epk_range.start..&epk_range.end, force_refresh, - |c, cont| Box::pin(self.fetch_pk_ranges_from_service(c, cont)), + self.pk_range_page_fetcher(), ) .await } diff --git a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs index a7a87424595..93f26e5de32 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs +++ b/sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs @@ -44,6 +44,7 @@ use super::{ DATA_PLANE_MAX_THROTTLE_ATTEMPTS, DATA_PLANE_MAX_THROTTLE_WAIT, METADATA_MAX_PER_RETRY_DELAY, METADATA_MAX_THROTTLE_ATTEMPTS, METADATA_MAX_THROTTLE_WAIT, }, + hedge_budget::{HedgeBudget, HedgePermit}, hedging_diagnostics::{HedgeDiagnostics, HedgingStrategyConfig}, hedging_eligibility::evaluate_hedge_eligibility, retry_evaluation::{ @@ -79,12 +80,45 @@ fn default_throttle_budget(pipeline_type: PipelineType) -> (u32, Duration, Durat } } +/// Internal, non-customer-overridable hedging and routing pin carried on +/// [`OperationOverrides::region_pin`]. +/// +/// Attached to a read whose continuation token is region-affine — today the +/// PartitionKeyRange change feed, whose ETag is only meaningful to the region +/// that issued it. Its presence alone suppresses hedging; the optional +/// `endpoint` additionally forces the attempt onto a specific region. +/// +/// This deliberately does **not** reuse [`AvailabilityStrategy::Disabled`]: +/// that is a customer-facing option, and `resolve_availability_strategy` lets +/// the `AZURE_COSMOS_HEDGING_ENABLED=true` environment switch override it. A +/// correctness constraint must not be something configuration can turn off, so +/// it lives here instead. +/// +/// [`AvailabilityStrategy::Disabled`]: crate::options::AvailabilityStrategy::Disabled +#[derive(Debug, Clone, Default)] +pub(crate) struct RegionPin { + /// Region this attempt must be routed to, bypassing the normal region + /// selection in [`resolve_endpoint`]. + /// + /// Normally always populated: the region that served the cold page is + /// recorded and every later page in the chain is pinned to it, so a + /// `FailoverRetry`/`SessionRetry` cannot carry the region-affine + /// continuation into a region that never issued it. + /// + /// `None` is the degraded fallback for the rare case where the serving + /// region could not be identified. Normal region selection then applies, + /// but hedging stays suppressed — racing a second region would send the + /// token somewhere it means nothing. + pub endpoint: Option, +} + /// Per-request overrides that take precedence over values from [`CosmosOperation`]. /// /// Used by the dataflow pipeline to inject routing and pagination state that /// varies per physical partition or per page, without mutating the shared -/// `CosmosOperation`. Each field, when `Some`, emits the corresponding request -/// header in [`OperationOverrides::apply_headers`]. +/// `CosmosOperation`. Most fields, when `Some`, emit the corresponding request +/// header in [`OperationOverrides::apply_headers`]; `region_pin` instead +/// constrains hedging and the STAGE 2 routing decision. #[derive(Debug, Clone, Default)] pub(crate) struct OperationOverrides { /// Feed range to constrain the request to (emits `x-ms-start-epk` / `x-ms-end-epk`). @@ -105,9 +139,41 @@ pub(crate) struct OperationOverrides { /// Continuation token for pagination (emits `x-ms-continuation`). pub continuation: Option, + + /// Constrains this attempt to a single region and forbids hedging. + /// + /// Unlike the other fields this is NOT a request header — it is consumed by + /// STAGE 2 (routing), STAGE 2b (pre-attempt hedge dispatch), and STAGE 5b + /// (post-attempt hedge upgrade) of the operation pipeline. It keeps a paged + /// read on one region: the PartitionKeyRange change feed carries its + /// continuation as an ETag that is only meaningful to the region that + /// issued it, so every page after the first must neither race another + /// region nor be failed over into one. + /// + /// See [`RegionPin`] for why this is an internal signal rather than + /// `AvailabilityStrategy::Disabled`. Boxed so the field stays pointer-sized + /// — this struct is captured by the operation future, which is already + /// close to the `clippy::large_futures` budget. + pub region_pin: Option>, } impl OperationOverrides { + /// The endpoint this attempt is pinned to, if any. + /// + /// `None` either because there is no pin at all, or because the pin only + /// suppresses hedging and leaves region selection alone. + pub fn pinned_endpoint(&self) -> Option<&crate::driver::routing::CosmosEndpoint> { + self.region_pin.as_ref().and_then(|p| p.endpoint.as_ref()) + } + + /// Whether hedging must not fire for this attempt. + /// + /// Distinct from the customer-facing `AvailabilityStrategy::Disabled`, + /// which `AZURE_COSMOS_HEDGING_ENABLED=true` deliberately overrides. + pub fn hedging_suppressed(&self) -> bool { + self.region_pin.is_some() + } + /// Applies the override headers to the given header map. /// /// Headers set here take precedence over any previously-set values for @@ -234,6 +300,7 @@ pub(crate) async fn execute_operation_pipeline( account_default_consistency: DefaultConsistencyLevel, throughput_control: Option, pre_resolved_pk_range_id: Option, + hedge_budget: &HedgeBudget, ) -> crate::error::Result { let mut diagnostics = diagnostics; let location_snapshot = location_state_store.snapshot(); @@ -382,14 +449,22 @@ pub(crate) async fn execute_operation_pipeline( // parity), falling back to parsing the customer-provided global // endpoint hostname when metadata has not synced yet. let account_name = location_state_store.global_database_account_name(); - let routing = resolve_endpoint( - operation, - &retry_state, - &location, - pipeline_type.is_data_plane(), - account_name.is_some(), - location_state_store.endpoint_unavailability_ttl(), - ); + // A pinned attempt (e.g. PartitionKeyRange pages 2..N after a first-page + // hedge win) bypasses normal region selection and routes straight to the + // pinned region so the change-feed continuation stays region-consistent. + let routing = match overrides.pinned_endpoint() { + Some(pinned) => { + routing_decision_for_pinned_endpoint(pinned, pipeline_type.is_data_plane()) + } + None => resolve_endpoint( + operation, + &retry_state, + &location, + pipeline_type.is_data_plane(), + account_name.is_some(), + location_state_store.endpoint_unavailability_ttl(), + ), + }; // Emit one structured debug record per attempt with the chosen // routing decision. Tests and SREs filter on this to verify which @@ -421,14 +496,43 @@ pub(crate) async fn execute_operation_pipeline( // applicable read endpoints, env-disabled hedging, or per-op // `AvailabilityStrategy::Disabled`. All gated by // [`evaluate_hedge_eligibility`]. - if retry_state.failover_retry_count == 0 && retry_state.session_token_retry_count == 0 { - if let Some(upgrade) = evaluate_hedge_eligibility( + // * **Region-pinned attempts** — when `overrides.region_pin` is set the + // attempt carries a region-affine continuation token, so racing a + // second region would send that token somewhere it means nothing. + // This is checked directly rather than through + // `AvailabilityStrategy::Disabled` because the latter is a customer + // option that `AZURE_COSMOS_HEDGING_ENABLED=true` can override; a + // correctness constraint must not be something configuration can + // turn off. + // * **An exhausted hedge concurrency budget** — see [`HedgeBudget`]. + // The permit is held for the lifetime of the race and released when it + // ends, so a refusal here means the client already has as many hedge + // races open as it is allowed. + if retry_state.failover_retry_count == 0 + && retry_state.session_token_retry_count == 0 + && !overrides.hedging_suppressed() + { + let admitted = evaluate_hedge_eligibility( operation, options, &location.account, &routing, configured_request_timeout, - ) { + ) + .and_then(|upgrade| match hedge_budget.try_admit(pipeline_type) { + Some(permit) => Some((upgrade, permit)), + None => { + // Refuse rather than queue: an operation that waits its turn + // to hedge has already lost the latency argument. It falls + // through to the ordinary sequential path instead. + tracing::debug!( + activity_id = %activity_id, + "cosmos.hedge.concurrency_budget_exhausted", + ); + None + } + }); + if let Some((upgrade, _hedge_permit)) = admitted { let attempt_ctx = AttemptContext { operation, overrides: &overrides, @@ -692,18 +796,27 @@ pub(crate) async fn execute_operation_pipeline( // back-to-back upgrades would compound RU consumption without // letting the surrounding failover loop make sequential // progress against the remaining regions. - let action = if retry_state.hedge_already_fired { - action - } else { - maybe_upgrade_to_hedge( - action, - operation, - options, - &location.account, - &routing, - configured_request_timeout, - ) - }; + // + // A region-pinned attempt (`overrides.region_pin`) is likewise never + // upgraded: it carries a region-affine continuation token, so racing + // (or failing over) to another region would send that token somewhere + // it is not meaningful. This mirrors the STAGE 2b suppression above. + let (action, _hedge_permit) = + if retry_state.hedge_already_fired || overrides.hedging_suppressed() { + (action, None) + } else { + maybe_upgrade_to_hedge( + action, + operation, + options, + &location.account, + &routing, + configured_request_timeout, + hedge_budget, + pipeline_type, + activity_id, + ) + }; // ── STAGE 6: Apply location effects ──────────────────────────── // Single-master write effects are deferred into @@ -1144,6 +1257,38 @@ fn is_effect_already_applied(effect: &LocationEffect, snapshot: &LocationSnapsho /// /// Uses `LocationSnapshot` and `AccountEndpointState` to select the best /// available endpoint, respecting excluded regions and unavailability TTL. +/// Builds a [`RoutingDecision`] that routes an attempt to a specific `pinned` +/// endpoint (see [`OperationOverrides::region_pin`]), bypassing the normal +/// region selection in [`resolve_endpoint`]. +/// +/// Mirrors the routing construction used for the hedge secondary in +/// `evaluate_hedge_eligibility`, so a pinned attempt shares the same +/// gateway-version preference and connection-pool keying. +fn routing_decision_for_pinned_endpoint( + pinned: &crate::driver::routing::CosmosEndpoint, + prefer_gateway_v2: bool, +) -> RoutingDecision { + let use_gateway_v2 = pinned.uses_gateway_v2(prefer_gateway_v2); + let transport_mode = if use_gateway_v2 { + TransportMode::GatewayV2 + } else { + TransportMode::Gateway + }; + let selected_url = pinned.selected_url(use_gateway_v2).clone(); + let endpoint_key = if use_gateway_v2 { + crate::driver::transport::EndpointKey::try_from(&selected_url) + .expect("selected URL must have a valid host and port") + } else { + pinned.endpoint_key() + }; + RoutingDecision { + selected_url, + transport_mode, + endpoint_key, + endpoint: pinned.clone(), + } +} + fn resolve_endpoint( operation: &CosmosOperation, retry_state: &OperationRetryState, @@ -2365,6 +2510,34 @@ fn classify_hedge_result(result: crate::error::Result) -> Hedge } } +/// Classifies the SECONDARY (hedge) leg, applying the metadata +/// primary-authoritative rule. +/// +/// For the two metadata cache reads the primary region is authoritative: a +/// hedge may improve latency by winning with a definitive **success**, but it +/// must never override the primary with a definitive **error** (e.g. a +/// not-yet-replicated secondary returning `404` for a freshly-created container, +/// or a `409`/`412`/`429`-final). When `primary_authoritative` is set, a +/// secondary that produced a `Final` but non-`Success` outcome is therefore +/// downgraded to [`HedgeClass::Transient`] so the race discards it and awaits +/// the primary's authoritative outcome. A secondary definitive success still +/// wins; a secondary transient stays transient. Data-plane hedging passes +/// `false` and keeps the first-`Final`-wins semantics of +/// [`classify_hedge_result`]. +fn classify_secondary_hedge_result( + result: crate::error::Result, + primary_authoritative: bool, +) -> HedgeClass { + match classify_hedge_result(result) { + HedgeClass::Final(tr) + if primary_authoritative && !matches!(tr.outcome, TransportOutcome::Success { .. }) => + { + HedgeClass::Transient + } + other => other, + } +} + /// Non-consuming version of [`classify_hedge_result`] for the /// pre-threshold primary-completion branch, where the caller still /// needs the original `TransportResult` to surface the response via @@ -2520,14 +2693,22 @@ fn finalize_hedge_attempt( /// `Hedge` variant so STAGE 7 can apply it to the live `retry_state` /// before building `AttemptContext` — see /// `OperationAction::Hedge::new_state`. -fn maybe_upgrade_to_hedge( +/// +/// Returns the (possibly rewritten) action alongside the [`HedgePermit`] that +/// admitted the race. The caller must hold the permit for as long as the race is +/// open; dropping it returns the slot to the [`HedgeBudget`]. +#[allow(clippy::too_many_arguments)] +fn maybe_upgrade_to_hedge<'a>( action: OperationAction, operation: &CosmosOperation, options: &OperationOptionsView<'_>, account_state: &AccountEndpointState, primary: &RoutingDecision, request_timeout: Option, -) -> OperationAction { + hedge_budget: &'a HedgeBudget, + pipeline_type: PipelineType, + activity_id: &ActivityId, +) -> (OperationAction, Option>) { // Extract `new_state` from the retry-upgrade-eligible variants; // return everything else unchanged. let new_state = match &action { @@ -2535,10 +2716,12 @@ fn maybe_upgrade_to_hedge( // handlers, so this preserves the backend-failover backoff instead of // replacing it with an immediate hedge. A zero delay carries no backoff // to preserve, so it stays hedge-eligible like `None`. - OperationAction::FailoverRetry { delay: Some(d), .. } if !d.is_zero() => return action, + OperationAction::FailoverRetry { delay: Some(d), .. } if !d.is_zero() => { + return (action, None) + } OperationAction::FailoverRetry { new_state, .. } => new_state.clone(), OperationAction::SessionRetry { new_state } => new_state.clone(), - _ => return action, + _ => return (action, None), }; match evaluate_hedge_eligibility(operation, options, account_state, primary, request_timeout) { @@ -2554,8 +2737,18 @@ fn maybe_upgrade_to_hedge( max_failover_retries = new_state.max_failover_retries, "cosmos.hedge.budget_exhausted_skipping_upgrade", ); - return action; + return (action, None); } + // Concurrency admission control. Refusing here leaves the operation + // on its original sequential retry action rather than queueing it + // for a slot — see [`HedgeBudget`]. + let Some(permit) = hedge_budget.try_admit(pipeline_type) else { + tracing::debug!( + activity_id = %activity_id, + "cosmos.hedge.concurrency_budget_exhausted", + ); + return (action, None); + }; // Emit a structured event when an operation is upgraded // into the hedge race. Fields mirror the inputs that drove // the eligibility decision so operators can correlate @@ -2567,14 +2760,17 @@ fn maybe_upgrade_to_hedge( hub_region_processing_only = new_state.hub_region_processing_only, "cosmos.hedge.enabled_for_operation", ); - OperationAction::Hedge { - secondary_routing: upgrade.secondary_routing, - threshold: upgrade.threshold, - strategy_config: upgrade.strategy_config, - new_state, - } + ( + OperationAction::Hedge { + secondary_routing: upgrade.secondary_routing, + threshold: upgrade.threshold, + strategy_config: upgrade.strategy_config, + new_state, + }, + Some(permit), + ) } - None => action, + None => (action, None), } } @@ -3089,6 +3285,13 @@ async fn execute_hedged( .clone() .unwrap_or_else(|| Region::new(HedgeDiagnostics::UNKNOWN_REGION_SENTINEL)); + // Metadata cache reads keep the PRIMARY authoritative: a secondary may win + // only with a definitive success, never with a definitive error (guards the + // replication-lag race where a not-yet-consistent secondary returns 404/409 + // before a slow-but-good primary). Data-plane hedging keeps first-Final-wins. + // The metadata pipeline is the only place the two metadata read pairs run. + let metadata_primary_authoritative = ctx.pipeline_type.is_metadata(); + tracing::debug!( activity_id = %ctx.activity_id, threshold_ms = ?threshold.get().as_millis(), @@ -3468,7 +3671,10 @@ async fn execute_hedged( &mut race_observed_session_unavailable, ) .await; - match classify_hedge_result(secondary_result) { + match classify_secondary_hedge_result( + secondary_result, + metadata_primary_authoritative, + ) { HedgeClass::Final(tr) => { parent_diagnostics.set_hedge_diagnostics(HedgeDiagnostics::hedge_won( strategy_config, @@ -3537,7 +3743,8 @@ async fn execute_hedged( &mut race_observed_session_unavailable, ) .await; - match classify_hedge_result(secondary_result) { + match classify_secondary_hedge_result(secondary_result, metadata_primary_authoritative) + { HedgeClass::Final(tr) => { parent_diagnostics.set_hedge_diagnostics(HedgeDiagnostics::hedge_won( strategy_config, @@ -8458,6 +8665,196 @@ mod tests { )); } + // ── classify_secondary_hedge_result (metadata primary-authoritative) ── + + #[test] + fn secondary_definitive_error_defers_to_primary_when_metadata() { + // Metadata primary-authoritative: a secondary 404 (Final, non-success) + // must NOT win — downgraded to Transient so the primary is awaited. + let tr = http_result(404, None); + assert!(matches!( + super::classify_secondary_hedge_result(Ok(tr), true), + super::HedgeClass::Transient + )); + } + + #[test] + fn secondary_definitive_error_wins_when_not_metadata() { + // Data-plane (primary_authoritative = false) keeps first-Final-wins. + let tr = http_result(404, None); + assert!(matches!( + super::classify_secondary_hedge_result(Ok(tr), false), + super::HedgeClass::Final(_) + )); + } + + #[test] + fn secondary_definitive_success_wins_when_metadata() { + // A secondary definitive success still wins under primary-authoritative + // (the latency benefit). + let tr = http_result(200, None); + assert!(matches!( + super::classify_secondary_hedge_result(Ok(tr), true), + super::HedgeClass::Final(_) + )); + } + + #[test] + fn secondary_transient_stays_transient_when_metadata() { + let tr = http_result(503, None); + assert!(matches!( + super::classify_secondary_hedge_result(Ok(tr), true), + super::HedgeClass::Transient + )); + } + + // ── OperationOverrides::region_pin ──────────────────────────────── + + #[test] + fn routing_decision_for_pinned_endpoint_routes_to_the_pinned_endpoint() { + use crate::driver::routing::CosmosEndpoint; + use crate::options::Region; + let url = url::Url::parse("https://acct-westus2.documents.azure.com/").unwrap(); + let pinned = CosmosEndpoint::regional(Region::WEST_US_2, url.clone()); + + let routing = super::routing_decision_for_pinned_endpoint(&pinned, false); + + assert_eq!(routing.endpoint.region(), Some(&Region::WEST_US_2)); + assert_eq!(routing.selected_url, url); + assert!(matches!( + routing.transport_mode, + super::TransportMode::Gateway + )); + } + + #[test] + fn no_region_pin_allows_hedging_and_normal_routing() { + let overrides = super::OperationOverrides::default(); + + assert!(!overrides.hedging_suppressed()); + assert!(overrides.pinned_endpoint().is_none()); + } + + #[test] + fn region_pin_without_endpoint_suppresses_hedging_only() { + // The degraded fallback: the serving region could not be identified, so + // there is no region to force, but the page still carries a + // region-affine ETag and must not be raced. + let overrides = super::OperationOverrides { + region_pin: Some(Box::new(super::RegionPin::default())), + ..Default::default() + }; + + assert!(overrides.hedging_suppressed()); + assert!(overrides.pinned_endpoint().is_none()); + } + + #[test] + fn region_pin_with_endpoint_suppresses_hedging_and_pins_routing() { + use crate::driver::routing::CosmosEndpoint; + use crate::options::Region; + let url = url::Url::parse("https://acct-westus2.documents.azure.com/").unwrap(); + let overrides = super::OperationOverrides { + region_pin: Some(Box::new(super::RegionPin { + endpoint: Some(CosmosEndpoint::regional(Region::WEST_US_2, url)), + })), + ..Default::default() + }; + + assert!(overrides.hedging_suppressed()); + assert_eq!( + overrides.pinned_endpoint().and_then(|e| e.region()), + Some(&Region::WEST_US_2) + ); + } + + #[test] + fn region_pin_holds_page_two_on_its_region_across_a_failover_retry() { + // The failure this guards: a `/pkranges` page 2 whose primary region has + // just been marked unavailable by a failover retry. Normal routing would + // move that attempt to the next preferred region and send the + // region-affine change-feed ETag somewhere that never issued it. The + // STAGE 2 pin must win over `resolve_endpoint` in that state. + let operation = CosmosOperation::read_all_partition_key_ranges(test_container()); + + let east = CosmosEndpoint::regional( + "eastus".into(), + Url::parse("https://test-eastus.documents.azure.com:443/").unwrap(), + ); + let west = CosmosEndpoint::regional( + "westus2".into(), + Url::parse("https://test-westus2.documents.azure.com:443/").unwrap(), + ); + + // East US — the region that served the cold page and issued the ETag — + // has just been marked unavailable, exactly as an in-flight failover + // retry would leave it. + let mut unavailable = std::collections::HashMap::new(); + unavailable.insert( + east.url().clone(), + ( + std::time::Instant::now(), + crate::driver::routing::UnavailableReason::ServiceUnavailable, + ), + ); + let location = LocationSnapshot::for_tests(Arc::new(AccountEndpointState { + generation: 0, + preferred_read_endpoints: vec![east.clone(), west.clone()].into(), + preferred_write_endpoints: vec![east.clone()].into(), + account_write_endpoints: vec![east.clone()].into(), + unavailable_endpoints: unavailable, + multiple_write_locations_enabled: false, + default_endpoint: east.clone(), + })); + + let mut retry_state = crate::driver::pipeline::components::OperationRetryState::initial( + 0, + false, + Vec::new(), + 3, + 2, + ); + retry_state.failover_retry_count = 1; + + // Baseline: without a pin this attempt leaves East US. + let unpinned = super::resolve_endpoint( + &operation, + &retry_state, + &location, + false, + false, + Duration::from_secs(60), + ); + assert_eq!( + unpinned.endpoint, west, + "sanity check: normal routing must fail over off the unavailable region, \ + otherwise this test proves nothing", + ); + + // With the pin, STAGE 2 bypasses `resolve_endpoint` entirely and the + // page stays on the region that issued its continuation. + let overrides = super::OperationOverrides { + region_pin: Some(Box::new(super::RegionPin { + endpoint: Some(east.clone()), + })), + ..Default::default() + }; + let pinned = overrides + .pinned_endpoint() + .expect("a recorded pin carries its endpoint"); + let routing = super::routing_decision_for_pinned_endpoint(pinned, false); + + assert_eq!( + routing.endpoint, east, + "a pinned continuation page must stay on its issuing region even when \ + that region is unavailable and a failover retry is in flight", + ); + assert!( + overrides.hedging_suppressed(), + "a pinned continuation page must also never be raced", + ); + } + #[test] fn classify_hedge_result_404_1002_is_transient() { // 404/1002 ReadSessionNotAvailable is retriable. From d9dfadc5221b2ee4b910a052cf0c539c124f1b70 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Tue, 4 Aug 2026 12:14:13 -0700 Subject: [PATCH 18/21] Drop workflow-file deltas from merge Keep PR #4871 conflict resolutions while resetting workflow and action-lock files to remote head so push is not blocked by workflow-scope enforcement. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d51868ed-95ef-4dff-b237-4f8226e5ec9a --- .github/aw/actions-lock.json | 6 +- .github/workflows/issue-triage.lock.yml | 144 ++++++++++++----------- .github/workflows/issue-triage.md | 3 +- .github/workflows/review-sdk.lock.yml | 148 +++++++++++++----------- .github/workflows/verify-links.yml | 4 +- 5 files changed, 162 insertions(+), 143 deletions(-) diff --git a/.github/aw/actions-lock.json b/.github/aw/actions-lock.json index c6e1d47550e..4581b1f584d 100644 --- a/.github/aw/actions-lock.json +++ b/.github/aw/actions-lock.json @@ -30,10 +30,10 @@ "version": "v7.0.1", "sha": "043fb46d1a93c77aae656e7c1c64a875d1fc6a0a" }, - "github/gh-aw-actions/setup@v0.84.3": { + "github/gh-aw-actions/setup@v0.83.4": { "repo": "github/gh-aw-actions/setup", - "version": "v0.84.3", - "sha": "c863074b673419603d146aab585e2986ef08deec" + "version": "v0.83.4", + "sha": "e89c65e17eb281bbd5ff2ff9e9199a03e96654c7" } } } diff --git a/.github/workflows/issue-triage.lock.yml b/.github/workflows/issue-triage.lock.yml index 3e5acd12715..1dcbbcd356b 100644 --- a/.github/workflows/issue-triage.lock.yml +++ b/.github/workflows/issue-triage.lock.yml @@ -1,6 +1,6 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"311a18aa564b594a480c8f759d9514808444d70dd631916b8dcad2a882d2400d","body_hash":"6a8f1950d11870316a376a455938cc3bd282bd0ce9c16b235f99948710110519","compiler_version":"v0.84.3","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.77"}} -# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"373c709c69115d41ff229c7e5df9f8788daa9553","version":"v9"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"c863074b673419603d146aab585e2986ef08deec","version":"v0.84.3"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.43","digest":"sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43","digest":"sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.43","digest":"sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.7","digest":"sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.7@sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196"},{"image":"ghcr.io/github/github-mcp-server:v1.8.0","digest":"sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520","pinned_image":"ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520"}]} -# This file was automatically generated by gh-aw (v0.84.3). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"21695fba876d02c38f79a1b4e7d8fbff7641224a0b4007e499af975192cab924","body_hash":"6a8f1950d11870316a376a455938cc3bd282bd0ce9c16b235f99948710110519","compiler_version":"v0.83.4","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.75"}} +# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"373c709c69115d41ff229c7e5df9f8788daa9553","version":"v9"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"e89c65e17eb281bbd5ff2ff9e9199a03e96654c7","version":"v0.83.4"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.42","digest":"sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42","digest":"sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.42","digest":"sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.6","digest":"sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.6@sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748","pinned_image":"ghcr.io/github/gh-aw-node@sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748"},{"image":"ghcr.io/github/github-mcp-server:v1.7.0","digest":"sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308","pinned_image":"ghcr.io/github/github-mcp-server:v1.7.0@sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308"}]} +# This file was automatically generated by gh-aw (v0.83.4). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # # ___ _ _ # / _ \ | | (_) @@ -46,15 +46,15 @@ # - actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 # - actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 # - actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 -# - github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 +# - github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 # # Container images used: -# - ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6 -# - ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1 -# - ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d -# - ghcr.io/github/gh-aw-mcpg:v0.4.7@sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00 -# - ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196 -# - ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520 +# - ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b +# - ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607 +# - ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0 +# - ghcr.io/github/gh-aw-mcpg:v0.4.6@sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c +# - ghcr.io/github/gh-aw-node@sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748 +# - ghcr.io/github/github-mcp-server:v1.7.0@sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308 name: "Agentic Triage" on: @@ -100,7 +100,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 + uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -108,8 +108,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Agentic Triage" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/issue-triage.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.77" - GH_AW_INFO_AWF_VERSION: "v0.27.43" + GH_AW_INFO_VERSION: "1.0.75" + GH_AW_INFO_AWF_VERSION: "v0.27.42" GH_AW_INFO_BODY_MODIFIED: "false" GH_AW_INFO_ENGINE_ID: "copilot" - name: Generate agentic run info @@ -117,17 +117,17 @@ jobs: env: GH_AW_INFO_ENGINE_ID: "copilot" GH_AW_INFO_ENGINE_NAME: "GitHub Copilot CLI" - GH_AW_INFO_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'auto' }} - GH_AW_INFO_VERSION: "1.0.77" - GH_AW_INFO_AGENT_VERSION: "1.0.77" - GH_AW_INFO_CLI_VERSION: "v0.84.3" + GH_AW_INFO_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }} + GH_AW_INFO_VERSION: "1.0.75" + GH_AW_INFO_AGENT_VERSION: "1.0.75" + GH_AW_INFO_CLI_VERSION: "v0.83.4" GH_AW_INFO_WORKFLOW_NAME: "Agentic Triage" GH_AW_INFO_EXPERIMENTAL: "false" GH_AW_INFO_SUPPORTS_TOOLS_ALLOWLIST: "true" GH_AW_INFO_STAGED: "false" GH_AW_INFO_ALLOWED_DOMAINS: '["github","threat-detection"]' GH_AW_INFO_FIREWALL_ENABLED: "true" - GH_AW_INFO_AWF_VERSION: "v0.27.43" + GH_AW_INFO_AWF_VERSION: "v0.27.42" GH_AW_INFO_AWMG_VERSION: "" GH_AW_INFO_FIREWALL_TYPE: "squid" GH_AW_INFO_FRONTMATTER_SOURCE: "githubnext/agentics/workflows/issue-triage.md@8e6d7c86bba37371d2d0eee1a23563db3e561eb5" @@ -240,7 +240,7 @@ jobs: - name: Check compile-agentic version uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 env: - GH_AW_COMPILED_VERSION: "v0.84.3" + GH_AW_COMPILED_VERSION: "v0.83.4" with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); @@ -433,10 +433,7 @@ jobs: http_400_response_error: ${{ steps.detect-agent-errors.outputs.http_400_response_error || 'false' }} inference_access_error: ${{ steps.detect-agent-errors.outputs.inference_access_error || 'false' }} invocation_cap_exceeded: ${{ steps.detect-agent-errors.outputs.invocation_cap_exceeded || 'false' }} - max_cache_misses_exceeded: ${{ steps.detect-agent-errors.outputs.max_cache_misses_exceeded || 'false' }} mcp_policy_error: ${{ steps.detect-agent-errors.outputs.mcp_policy_error || 'false' }} - missing_model_pricing_error: ${{ steps.detect-agent-errors.outputs.missing_model_pricing_error || 'false' }} - missing_model_pricing_model_name: ${{ steps.detect-agent-errors.outputs.missing_model_pricing_model_name || '' }} model: ${{ needs.activation.outputs.model }} model_not_supported_error: ${{ steps.detect-agent-errors.outputs.model_not_supported_error || 'false' }} output: ${{ steps.collect_output.outputs.output }} @@ -448,7 +445,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 + uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -457,8 +454,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Agentic Triage" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/issue-triage.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.77" - GH_AW_INFO_AWF_VERSION: "v0.27.43" + GH_AW_INFO_VERSION: "1.0.75" + GH_AW_INFO_AWF_VERSION: "v0.27.42" GH_AW_INFO_BODY_MODIFIED: "false" GH_AW_INFO_ENGINE_ID: "copilot" - name: Set runtime paths @@ -505,12 +502,11 @@ jobs: const { main } = require('${{ runner.temp }}/gh-aw/actions/checkout_pr_branch.cjs'); await main(); - name: Install GitHub Copilot CLI - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" 1.0.75 env: GH_HOST: github.com - GH_AW_COMPILED_VERSION: v0.84.3 - name: Install AWF binary - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.43 --rootless + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.42 --rootless - name: Determine automatic lockdown mode for GitHub MCP Server id: determine-automatic-lockdown uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 @@ -546,7 +542,7 @@ jobs: GH_AW_SKILL_DIR: ".github/skills" run: bash "${RUNNER_TEMP}/gh-aw/actions/restore_inline_skills.sh" - name: Download container images - run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6 ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1 ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d ghcr.io/github/gh-aw-mcpg:v0.4.7@sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00 ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196 ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520 + run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607 ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0 ghcr.io/github/gh-aw-mcpg:v0.4.6@sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c ghcr.io/github/gh-aw-node@sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748 ghcr.io/github/github-mcp-server:v1.7.0@sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308 - name: Generate Safe Outputs Config run: | mkdir -p "${RUNNER_TEMP}/gh-aw/safeoutputs" @@ -751,7 +747,6 @@ jobs: GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} GH_AW_SAFE_OUTPUTS_CONFIG_PATH: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS_CONFIG_PATH }} GH_AW_SAFE_OUTPUTS_TOOLS_PATH: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS_TOOLS_PATH }} - GH_AW_SINK_VISIBILITY: ${{ steps.determine-automatic-lockdown.outputs.visibility }} GITHUB_MCP_SERVER_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | @@ -774,22 +769,22 @@ jobs: MCP_GATEWAY_UID=$(id -u 2>/dev/null || echo '0') MCP_GATEWAY_GID=$(id -g 2>/dev/null || echo '0') source "${RUNNER_TEMP}/gh-aw/actions/resolve_docker_socket_gid.sh" - export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network bridge -p 127.0.0.1:'"${MCP_GATEWAY_PORT}"':'"${MCP_GATEWAY_PORT}"' --name awmg-mcpg --add-host host.docker.internal:host-gateway --user '"${MCP_GATEWAY_UID}"':'"${MCP_GATEWAY_GID}"' --group-add '"${DOCKER_SOCK_GID}"' -v '"${DOCKER_SOCK_PATH}"':/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e MCP_GATEWAY_PAYLOAD_DIR -e MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD -e DOCKER_HOST=unix:///var/run/docker.sock -e DEBUG -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_POLICY_ALLOW_CREATE_PULL_REQUEST -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_GUARD_MIN_INTEGRITY -e GITHUB_MCP_GUARD_REPOS -e GH_AW_SINK_VISIBILITY -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e GITHUB_JOB -e GITHUB_ACTION -e GITHUB_EVENT_NAME -e GITHUB_EVENT_PATH -e GITHUB_ACTOR -e GITHUB_ACTOR_ID -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_WORKFLOW_REF -e GITHUB_WORKFLOW_SHA -e GITHUB_REF -e GITHUB_REF_NAME -e GITHUB_REF_TYPE -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e RUNNER_TEMP -v /tmp/gh-aw/mcp-payloads:/tmp/gh-aw/mcp-payloads:rw -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw -v '"${RUNNER_TEMP}"'/gh-aw/safeoutputs:'"${RUNNER_TEMP}"'/gh-aw/safeoutputs:rw ghcr.io/github/gh-aw-mcpg:v0.4.7' + export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network bridge -p 127.0.0.1:'"${MCP_GATEWAY_PORT}"':'"${MCP_GATEWAY_PORT}"' --name awmg-mcpg --add-host host.docker.internal:host-gateway --user '"${MCP_GATEWAY_UID}"':'"${MCP_GATEWAY_GID}"' --group-add '"${DOCKER_SOCK_GID}"' -v '"${DOCKER_SOCK_PATH}"':/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e MCP_GATEWAY_PAYLOAD_DIR -e MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD -e DOCKER_HOST=unix:///var/run/docker.sock -e DEBUG -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_POLICY_ALLOW_CREATE_PULL_REQUEST -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_GUARD_MIN_INTEGRITY -e GITHUB_MCP_GUARD_REPOS -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e GITHUB_JOB -e GITHUB_ACTION -e GITHUB_EVENT_NAME -e GITHUB_EVENT_PATH -e GITHUB_ACTOR -e GITHUB_ACTOR_ID -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_WORKFLOW_REF -e GITHUB_WORKFLOW_SHA -e GITHUB_REF -e GITHUB_REF_NAME -e GITHUB_REF_TYPE -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e RUNNER_TEMP -v /tmp/gh-aw/mcp-payloads:/tmp/gh-aw/mcp-payloads:rw -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw -v '"${RUNNER_TEMP}"'/gh-aw/safeoutputs:'"${RUNNER_TEMP}"'/gh-aw/safeoutputs:rw ghcr.io/github/gh-aw-mcpg:v0.4.6' mkdir -p "$HOME/.copilot" GH_AW_NODE=$(which node 2>/dev/null || command -v node 2>/dev/null || echo node) - cat << GH_AW_MCP_CONFIG_aaacf455c8d7e251_EOF | "$GH_AW_NODE" "${RUNNER_TEMP}/gh-aw/actions/start_mcp_gateway.cjs" + cat << GH_AW_MCP_CONFIG_d808373827ca217a_EOF | "$GH_AW_NODE" "${RUNNER_TEMP}/gh-aw/actions/start_mcp_gateway.cjs" { "mcpServers": { "github": { "type": "stdio", - "container": "ghcr.io/github/github-mcp-server:v1.8.0", + "container": "ghcr.io/github/github-mcp-server:v1.7.0", "env": { "GITHUB_FEATURES": "fields_param", "GITHUB_HOST": "${GITHUB_SERVER_URL}", "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_MCP_SERVER_TOKEN}", "GITHUB_READ_ONLY": "1", - "GITHUB_TOOLSETS": "issues,pull_requests,repos" + "GITHUB_TOOLSETS": "issues,pull_requests" }, "guard-policies": { "allow-only": { @@ -832,7 +827,7 @@ jobs: "accept": [ "private:azure/azure-sdk-for-rust" ], - "sink-visibility": "${GH_AW_SINK_VISIBILITY}" + "sink-visibility": ${{ toJSON(steps.determine-automatic-lockdown.outputs.visibility) }} } } } @@ -845,7 +840,7 @@ jobs: "startupTimeout": 120 } } - GH_AW_MCP_CONFIG_aaacf455c8d7e251_EOF + GH_AW_MCP_CONFIG_d808373827ca217a_EOF - name: Mount MCP servers as CLIs id: mount-mcp-clis continue-on-error: true @@ -888,7 +883,7 @@ jobs: export COPILOT_API_KEY="$COPILOT_DUMMY_BYOK" (umask 177 && touch /tmp/gh-aw/agent-stdio.log) GH_AW_MAX_AI_CREDITS="${GH_AW_MAX_AI_CREDITS:-1000}" - printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.43/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"*.githubusercontent.com\",\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"codeload.github.com\",\"docs.github.com\",\"github-cloud.githubusercontent.com\",\"github-cloud.s3.amazonaws.com\",\"github.blog\",\"github.com\",\"github.githubassets.com\",\"host.docker.internal\",\"lfs.github.com\",\"objects.githubusercontent.com\",\"patch-diff.githubusercontent.com\",\"patchdiff.githubusercontent.com\",\"raw.githubusercontent.com\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"],\"blockDomains\":[\"registry.npmjs.org\"],\"isolation\":true,\"topologyAttach\":[\"awmg-mcpg\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"auto\":[\"copilot/auto\",\"large\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"detection\":[\"small\"],\"evals\":[\"small\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"grok\":[\"copilot/*grok*\",\"openai/*grok*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.43,squid=sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d,agent=sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6,api-proxy=sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1,cli-proxy=sha256:65c45ea2967984d0024f3df61bc71335658a77ede96c8d9665da7a5f33a795ab\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" + printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.42/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"*.githubusercontent.com\",\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"codeload.github.com\",\"docs.github.com\",\"github-cloud.githubusercontent.com\",\"github-cloud.s3.amazonaws.com\",\"github.blog\",\"github.com\",\"github.githubassets.com\",\"host.docker.internal\",\"lfs.github.com\",\"objects.githubusercontent.com\",\"patch-diff.githubusercontent.com\",\"patchdiff.githubusercontent.com\",\"raw.githubusercontent.com\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"],\"blockDomains\":[\"registry.npmjs.org\"],\"isolation\":true,\"topologyAttach\":[\"awmg-mcpg\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"fable\",\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.42,squid=sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0,agent=sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b,agent-act=sha256:a14ad974484aa518aab83d40f3f141175dfd171d3745e01c092375b970f73a20,api-proxy=sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607,cli-proxy=sha256:da006bf96d2d246dd269d57b233c1798d2ad63d6cd64ca02f7bf71045028781f\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" cp "${RUNNER_TEMP}/gh-aw/awf-config.json" /tmp/gh-aw/awf-config.json export GH_AW_MODELS_JSON_PATH="/tmp/gh-aw/models.json" GH_AW_DOCKER_HOST="" @@ -913,7 +908,7 @@ jobs: COPILOT_AGENT_RUNNER_TYPE: STANDALONE COPILOT_DUMMY_BYOK: dummy-byok-key-for-offline-mode COPILOT_GITHUB_TOKEN: ${{ github.token }} - COPILOT_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'auto' }} + COPILOT_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }} GH_AW_LLM_PROVIDER: github GH_AW_MAX_AI_CREDITS: ${{ vars.GH_AW_DEFAULT_MAX_AI_CREDITS || '1000' }} GH_AW_MAX_TURNS: ${{ vars.GH_AW_DEFAULT_MAX_TURNS || '' }} @@ -921,7 +916,7 @@ jobs: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} GH_AW_TIMEOUT_MINUTES: 10 - GH_AW_VERSION: v0.84.3 + GH_AW_VERSION: v0.83.4 GITHUB_API_URL: ${{ github.api_url }} GITHUB_AW: true GITHUB_COPILOT_INTEGRATION_ID: agentic-workflows @@ -1095,6 +1090,7 @@ jobs: needs.activation.outputs.daily_ai_credits_exceeded == 'true') runs-on: ubuntu-slim permissions: + contents: read issues: write pull-requests: write concurrency: @@ -1111,7 +1107,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 + uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1120,8 +1116,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Agentic Triage" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/issue-triage.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.77" - GH_AW_INFO_AWF_VERSION: "v0.27.43" + GH_AW_INFO_VERSION: "1.0.75" + GH_AW_INFO_AWF_VERSION: "v0.27.42" GH_AW_INFO_BODY_MODIFIED: "false" GH_AW_INFO_ENGINE_ID: "copilot" - name: Download agent output artifact @@ -1149,7 +1145,30 @@ jobs: - name: Collect usage artifact files if: always() continue-on-error: true - run: bash "${RUNNER_TEMP}/gh-aw/actions/collect_usage_artifact_files.sh" + run: | + mkdir -p /tmp/gh-aw/usage/agent /tmp/gh-aw/usage/detection + echo "Usage artifact source file status:" + for file in /tmp/gh-aw/aw_info.json /tmp/gh-aw/aw-info.jsonl /tmp/gh-aw/agent_usage.json /tmp/gh-aw/agent_usage.jsonl /tmp/gh-aw/detection_usage.jsonl /tmp/gh-aw/evals/evals.jsonl /tmp/gh-aw/github_rate_limits.jsonl /tmp/gh-aw/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/threat-detection/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/threat-detection/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl; do + [ -f "$file" ] && echo "FOUND: $file" || echo "MISSING: $file" + done + [ -f /tmp/gh-aw/aw_info.json ] && cp /tmp/gh-aw/aw_info.json /tmp/gh-aw/usage/aw_info.json || true + [ -f /tmp/gh-aw/aw-info.jsonl ] && cp /tmp/gh-aw/aw-info.jsonl /tmp/gh-aw/usage/aw-info.jsonl || true + [ -f /tmp/gh-aw/agent_usage.json ] && cp /tmp/gh-aw/agent_usage.json /tmp/gh-aw/usage/agent_usage.json || true + [ -f /tmp/gh-aw/agent_usage.jsonl ] && cp /tmp/gh-aw/agent_usage.jsonl /tmp/gh-aw/usage/agent_usage.jsonl || true + [ -f /tmp/gh-aw/detection_usage.jsonl ] && cp /tmp/gh-aw/detection_usage.jsonl /tmp/gh-aw/usage/detection_usage.jsonl || true + [ -f /tmp/gh-aw/evals/evals.jsonl ] && cp /tmp/gh-aw/evals/evals.jsonl /tmp/gh-aw/usage/evals.jsonl || true + [ -f /tmp/gh-aw/github_rate_limits.jsonl ] && cp /tmp/gh-aw/github_rate_limits.jsonl /tmp/gh-aw/usage/github_rate_limits.jsonl || true + [ -s /tmp/gh-aw/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/agent/token_usage.jsonl || true + [ -s /tmp/gh-aw/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/agent/token_usage.jsonl || true + [ -s /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/agent/token_usage.jsonl || true + [ -s /tmp/gh-aw/threat-detection/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/threat-detection/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/detection/token_usage.jsonl || true + [ -s /tmp/gh-aw/threat-detection/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/threat-detection/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/detection/token_usage.jsonl || true + [ -s /tmp/gh-aw/threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/detection/token_usage.jsonl || true + [ -f /tmp/gh-aw/usage/agent/token_usage.jsonl ] || : > /tmp/gh-aw/usage/agent/token_usage.jsonl + [ -f /tmp/gh-aw/usage/detection/token_usage.jsonl ] || : > /tmp/gh-aw/usage/detection/token_usage.jsonl + mkdir -p /tmp/gh-aw/usage/activity + node "${RUNNER_TEMP}/gh-aw/actions/generate_usage_activity_summary.cjs" + find /tmp/gh-aw/usage -type f -print | sort - name: Upload usage artifact if: always() continue-on-error: true @@ -1306,9 +1325,6 @@ jobs: GH_AW_AGENTIC_ENGINE_TIMEOUT: ${{ needs.agent.outputs.agentic_engine_timeout }} GH_AW_MODEL_NOT_SUPPORTED_ERROR: ${{ needs.agent.outputs.model_not_supported_error }} GH_AW_HTTP_400_RESPONSE_ERROR: ${{ needs.agent.outputs.http_400_response_error }} - GH_AW_MAX_CACHE_MISSES_EXCEEDED: ${{ needs.agent.outputs.max_cache_misses_exceeded }} - GH_AW_MISSING_MODEL_PRICING_ERROR: ${{ needs.agent.outputs.missing_model_pricing_error }} - GH_AW_MISSING_MODEL_PRICING_MODEL_NAME: ${{ needs.agent.outputs.missing_model_pricing_model_name }} GH_AW_ENGINE_API_HOSTS: "api.enterprise.githubcopilot.com,api.githubcopilot.com,api.business.githubcopilot.com,api.individual.githubcopilot.com" GH_AW_LOCKDOWN_CHECK_FAILED: ${{ needs.activation.outputs.lockdown_check_failed }} GH_AW_OAUTH_TOKEN_CHECK_FAILED: ${{ needs.activation.outputs.oauth_token_check_failed }} @@ -1348,7 +1364,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 + uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1357,8 +1373,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Agentic Triage" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/issue-triage.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.77" - GH_AW_INFO_AWF_VERSION: "v0.27.43" + GH_AW_INFO_VERSION: "1.0.75" + GH_AW_INFO_AWF_VERSION: "v0.27.42" GH_AW_INFO_BODY_MODIFIED: "false" GH_AW_INFO_ENGINE_ID: "copilot" - name: Download agent output artifact @@ -1386,7 +1402,7 @@ jobs: rm -rf /tmp/gh-aw/sandbox/firewall/logs rm -rf /tmp/gh-aw/sandbox/firewall/audit - name: Download container images - run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6 ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1 ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d + run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607 ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0 - name: Check if detection needed id: detection_guard if: always() @@ -1418,14 +1434,10 @@ jobs: fi cp /tmp/gh-aw/agent_output.json /tmp/gh-aw/threat-detection/agent_output.json 2>/dev/null || true for f in /tmp/gh-aw/aw-*.patch; do - if [ -f "$f" ]; then - cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true - fi + [ -f "$f" ] && cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true done for f in /tmp/gh-aw/aw-*.bundle; do - if [ -f "$f" ]; then - cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true - fi + [ -f "$f" ] && cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true done echo "Prepared threat detection files:" ls -la /tmp/gh-aw/threat-detection/ 2>/dev/null || true @@ -1436,7 +1448,6 @@ jobs: WORKFLOW_NAME: "Agentic Triage" WORKFLOW_DESCRIPTION: "Intelligent issue triage assistant that processes new issues.\nAnalyzes issue content, evaluates whether the author is an external customer,\npredicts category and service labels, looks up owners from CODEOWNERS, and\nposts an analysis comment. Implements the initial issue triage rules for the\nAzure SDK for Rust repository." HAS_PATCH: ${{ needs.agent.outputs.has_patch }} - GH_AW_DETECTION_CONTINUE_ON_ERROR: "true" with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); @@ -1454,12 +1465,11 @@ jobs: node-version: '24' package-manager-cache: false - name: Install GitHub Copilot CLI - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" 1.0.75 env: GH_HOST: github.com - GH_AW_COMPILED_VERSION: v0.84.3 - name: Install AWF binary - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.43 + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.42 - name: Execute GitHub Copilot CLI if: always() && steps.detection_guard.outputs.run_detection == 'true' continue-on-error: true @@ -1479,7 +1489,7 @@ jobs: export COPILOT_API_KEY="$COPILOT_DUMMY_BYOK" (umask 177 && touch /tmp/gh-aw/threat-detection/detection.log) GH_AW_MAX_AI_CREDITS="${GH_AW_MAX_AI_CREDITS:-400}" - printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.43/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"github.com\",\"host.docker.internal\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"auto\":[\"copilot/auto\",\"large\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"detection\":[\"small\"],\"evals\":[\"small\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"grok\":[\"copilot/*grok*\",\"openai/*grok*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.43,squid=sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d,agent=sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6,api-proxy=sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1,cli-proxy=sha256:65c45ea2967984d0024f3df61bc71335658a77ede96c8d9665da7a5f33a795ab\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" + printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.42/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"github.com\",\"host.docker.internal\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"fable\",\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.42,squid=sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0,agent=sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b,agent-act=sha256:a14ad974484aa518aab83d40f3f141175dfd171d3745e01c092375b970f73a20,api-proxy=sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607,cli-proxy=sha256:da006bf96d2d246dd269d57b233c1798d2ad63d6cd64ca02f7bf71045028781f\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" cp "${RUNNER_TEMP}/gh-aw/awf-config.json" /tmp/gh-aw/awf-config.json export GH_AW_MODELS_JSON_PATH="/tmp/gh-aw/models.json" GH_AW_DOCKER_HOST="" @@ -1506,14 +1516,14 @@ jobs: COPILOT_AGENT_RUNNER_TYPE: STANDALONE COPILOT_DUMMY_BYOK: dummy-byok-key-for-offline-mode COPILOT_GITHUB_TOKEN: ${{ github.token }} - COPILOT_MODEL: detection + COPILOT_MODEL: ${{ vars.GH_AW_MODEL_DETECTION_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }} GH_AW_LLM_PROVIDER: github GH_AW_MAX_AI_CREDITS: ${{ vars.GH_AW_DEFAULT_DETECTION_MAX_AI_CREDITS || '400' }} GH_AW_MAX_TURNS: ${{ vars.GH_AW_DEFAULT_MAX_TURNS || '' }} GH_AW_PHASE: detection GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_TIMEOUT_MINUTES: 20 - GH_AW_VERSION: v0.84.3 + GH_AW_VERSION: v0.83.4 GITHUB_API_URL: ${{ github.api_url }} GITHUB_AW: true GITHUB_COPILOT_INTEGRATION_ID: agentic-workflows @@ -1590,6 +1600,7 @@ jobs: if: (!cancelled()) && needs.agent.result != 'skipped' && needs.detection.result == 'success' runs-on: ubuntu-slim permissions: + contents: read issues: write pull-requests: write timeout-minutes: 45 @@ -1603,6 +1614,7 @@ jobs: GH_AW_EFFECTIVE_TOKENS: ${{ needs.agent.outputs.effective_tokens }} GH_AW_ENGINE_ID: "copilot" GH_AW_ENGINE_MODEL: ${{ needs.agent.outputs.model }} + GH_AW_ENGINE_VERSION: "1.0.75" GH_AW_RUNTIME_FEATURES: ${{ vars.GH_AW_RUNTIME_FEATURES }} GH_AW_THREAT_DETECTION_AIC: ${{ needs.detection.outputs.aic }} GH_AW_WORKFLOW_ID: "issue-triage" @@ -1622,7 +1634,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 + uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1631,8 +1643,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Agentic Triage" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/issue-triage.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.77" - GH_AW_INFO_AWF_VERSION: "v0.27.43" + GH_AW_INFO_VERSION: "1.0.75" + GH_AW_INFO_AWF_VERSION: "v0.27.42" GH_AW_INFO_BODY_MODIFIED: "false" GH_AW_INFO_ENGINE_ID: "copilot" - name: Download agent output artifact diff --git a/.github/workflows/issue-triage.md b/.github/workflows/issue-triage.md index bc067c1542d..d8e45295ad8 100644 --- a/.github/workflows/issue-triage.md +++ b/.github/workflows/issue-triage.md @@ -45,8 +45,7 @@ safe-outputs: tools: bash: false github: - # The gateway uses search_repositories to determine repository visibility. - toolsets: [issues, pull_requests, repos] + toolsets: [issues, pull_requests] # If in a public repo, setting `lockdown: false` allows # reading issues, pull requests and comments from 3rd-parties. # If in a private repo this has no particular effect. diff --git a/.github/workflows/review-sdk.lock.yml b/.github/workflows/review-sdk.lock.yml index 3f0063d5d0c..b6b593b6f10 100644 --- a/.github/workflows/review-sdk.lock.yml +++ b/.github/workflows/review-sdk.lock.yml @@ -1,6 +1,6 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"bfad664f863791b31456b318276c13868bd56a2afdb0dd7b7e72563f525b8608","body_hash":"70079c5f1c5bf0de92b2032b7b41dff0b140492a1386fc98d63b6b0ea8559c5c","compiler_version":"v0.84.3","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.77"}} -# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"373c709c69115d41ff229c7e5df9f8788daa9553","version":"v9"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"c863074b673419603d146aab585e2986ef08deec","version":"v0.84.3"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.43","digest":"sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43","digest":"sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.43","digest":"sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.7","digest":"sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.7@sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196"},{"image":"ghcr.io/github/github-mcp-server:v1.8.0","digest":"sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520","pinned_image":"ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520"}],"has_pull_request":true} -# This file was automatically generated by gh-aw (v0.84.3). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"bfad664f863791b31456b318276c13868bd56a2afdb0dd7b7e72563f525b8608","body_hash":"70079c5f1c5bf0de92b2032b7b41dff0b140492a1386fc98d63b6b0ea8559c5c","compiler_version":"v0.83.4","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.75"}} +# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"373c709c69115d41ff229c7e5df9f8788daa9553","version":"v9"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"e89c65e17eb281bbd5ff2ff9e9199a03e96654c7","version":"v0.83.4"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.42","digest":"sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42","digest":"sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.42","digest":"sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.6","digest":"sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.6@sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748","pinned_image":"ghcr.io/github/gh-aw-node@sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748"},{"image":"ghcr.io/github/github-mcp-server:v1.7.0","digest":"sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308","pinned_image":"ghcr.io/github/github-mcp-server:v1.7.0@sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308"}],"has_pull_request":true} +# This file was automatically generated by gh-aw (v0.83.4). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # # ___ _ _ # / _ \ | | (_) @@ -42,15 +42,15 @@ # - actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 # - actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 # - actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 -# - github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 +# - github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 # # Container images used: -# - ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6 -# - ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1 -# - ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d -# - ghcr.io/github/gh-aw-mcpg:v0.4.7@sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00 -# - ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196 -# - ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520 +# - ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b +# - ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607 +# - ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0 +# - ghcr.io/github/gh-aw-mcpg:v0.4.6@sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c +# - ghcr.io/github/gh-aw-node@sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748 +# - ghcr.io/github/github-mcp-server:v1.7.0@sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308 name: "Review SDK PR" on: @@ -69,10 +69,7 @@ run-name: "Review SDK PR" jobs: activation: - if: > - (github.event_name != 'pull_request' || github.event.pull_request.head.repo.id == github.repository_id) && - ((github.event_name != 'pull_request' && github.event_name != 'pull_request_review') || github.event.pull_request.stack == null || - github.event.pull_request.stack.position == github.event.pull_request.stack.size) + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.id == github.repository_id runs-on: ubuntu-slim permissions: actions: read @@ -102,7 +99,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 + uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -110,25 +107,25 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Review SDK PR" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/review-sdk.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.77" - GH_AW_INFO_AWF_VERSION: "v0.27.43" + GH_AW_INFO_VERSION: "1.0.75" + GH_AW_INFO_AWF_VERSION: "v0.27.42" GH_AW_INFO_ENGINE_ID: "copilot" - name: Generate agentic run info id: generate_aw_info env: GH_AW_INFO_ENGINE_ID: "copilot" GH_AW_INFO_ENGINE_NAME: "GitHub Copilot CLI" - GH_AW_INFO_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'auto' }} - GH_AW_INFO_VERSION: "1.0.77" - GH_AW_INFO_AGENT_VERSION: "1.0.77" - GH_AW_INFO_CLI_VERSION: "v0.84.3" + GH_AW_INFO_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }} + GH_AW_INFO_VERSION: "1.0.75" + GH_AW_INFO_AGENT_VERSION: "1.0.75" + GH_AW_INFO_CLI_VERSION: "v0.83.4" GH_AW_INFO_WORKFLOW_NAME: "Review SDK PR" GH_AW_INFO_EXPERIMENTAL: "false" GH_AW_INFO_SUPPORTS_TOOLS_ALLOWLIST: "true" GH_AW_INFO_STAGED: "false" GH_AW_INFO_ALLOWED_DOMAINS: '["github","threat-detection","azure.github.io","docs.microsoft.com","learn.microsoft.com"]' GH_AW_INFO_FIREWALL_ENABLED: "true" - GH_AW_INFO_AWF_VERSION: "v0.27.43" + GH_AW_INFO_AWF_VERSION: "v0.27.42" GH_AW_INFO_AWMG_VERSION: "" GH_AW_INFO_FIREWALL_TYPE: "squid" GH_AW_COMPILED_STRICT: "true" @@ -239,7 +236,7 @@ jobs: - name: Check compile-agentic version uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 env: - GH_AW_COMPILED_VERSION: "v0.84.3" + GH_AW_COMPILED_VERSION: "v0.83.4" with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); @@ -432,10 +429,7 @@ jobs: http_400_response_error: ${{ steps.detect-agent-errors.outputs.http_400_response_error || 'false' }} inference_access_error: ${{ steps.detect-agent-errors.outputs.inference_access_error || 'false' }} invocation_cap_exceeded: ${{ steps.detect-agent-errors.outputs.invocation_cap_exceeded || 'false' }} - max_cache_misses_exceeded: ${{ steps.detect-agent-errors.outputs.max_cache_misses_exceeded || 'false' }} mcp_policy_error: ${{ steps.detect-agent-errors.outputs.mcp_policy_error || 'false' }} - missing_model_pricing_error: ${{ steps.detect-agent-errors.outputs.missing_model_pricing_error || 'false' }} - missing_model_pricing_model_name: ${{ steps.detect-agent-errors.outputs.missing_model_pricing_model_name || '' }} model: ${{ needs.activation.outputs.model }} model_not_supported_error: ${{ steps.detect-agent-errors.outputs.model_not_supported_error || 'false' }} output: ${{ steps.collect_output.outputs.output }} @@ -447,7 +441,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 + uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -456,8 +450,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Review SDK PR" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/review-sdk.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.77" - GH_AW_INFO_AWF_VERSION: "v0.27.43" + GH_AW_INFO_VERSION: "1.0.75" + GH_AW_INFO_AWF_VERSION: "v0.27.42" GH_AW_INFO_ENGINE_ID: "copilot" - name: Set runtime paths id: set-runtime-paths @@ -503,12 +497,11 @@ jobs: const { main } = require('${{ runner.temp }}/gh-aw/actions/checkout_pr_branch.cjs'); await main(); - name: Install GitHub Copilot CLI - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" 1.0.75 env: GH_HOST: github.com - GH_AW_COMPILED_VERSION: v0.84.3 - name: Install AWF binary - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.43 --rootless + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.42 --rootless - name: Determine automatic lockdown mode for GitHub MCP Server id: determine-automatic-lockdown uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 @@ -544,7 +537,7 @@ jobs: GH_AW_SKILL_DIR: ".github/skills" run: bash "${RUNNER_TEMP}/gh-aw/actions/restore_inline_skills.sh" - name: Download container images - run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6 ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1 ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d ghcr.io/github/gh-aw-mcpg:v0.4.7@sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00 ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196 ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520 + run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607 ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0 ghcr.io/github/gh-aw-mcpg:v0.4.6@sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c ghcr.io/github/gh-aw-node@sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748 ghcr.io/github/github-mcp-server:v1.7.0@sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308 - name: Generate Safe Outputs Config run: | mkdir -p "${RUNNER_TEMP}/gh-aw/safeoutputs" @@ -675,7 +668,6 @@ jobs: GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} GH_AW_SAFE_OUTPUTS_CONFIG_PATH: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS_CONFIG_PATH }} GH_AW_SAFE_OUTPUTS_TOOLS_PATH: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS_TOOLS_PATH }} - GH_AW_SINK_VISIBILITY: ${{ steps.determine-automatic-lockdown.outputs.visibility }} GITHUB_MCP_SERVER_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | @@ -698,16 +690,16 @@ jobs: MCP_GATEWAY_UID=$(id -u 2>/dev/null || echo '0') MCP_GATEWAY_GID=$(id -g 2>/dev/null || echo '0') source "${RUNNER_TEMP}/gh-aw/actions/resolve_docker_socket_gid.sh" - export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network bridge -p 127.0.0.1:'"${MCP_GATEWAY_PORT}"':'"${MCP_GATEWAY_PORT}"' --name awmg-mcpg --add-host host.docker.internal:host-gateway --user '"${MCP_GATEWAY_UID}"':'"${MCP_GATEWAY_GID}"' --group-add '"${DOCKER_SOCK_GID}"' -v '"${DOCKER_SOCK_PATH}"':/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e MCP_GATEWAY_PAYLOAD_DIR -e MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD -e DOCKER_HOST=unix:///var/run/docker.sock -e DEBUG -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_POLICY_ALLOW_CREATE_PULL_REQUEST -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_GUARD_MIN_INTEGRITY -e GITHUB_MCP_GUARD_REPOS -e GH_AW_SINK_VISIBILITY -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e GITHUB_JOB -e GITHUB_ACTION -e GITHUB_EVENT_NAME -e GITHUB_EVENT_PATH -e GITHUB_ACTOR -e GITHUB_ACTOR_ID -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_WORKFLOW_REF -e GITHUB_WORKFLOW_SHA -e GITHUB_REF -e GITHUB_REF_NAME -e GITHUB_REF_TYPE -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e RUNNER_TEMP -v /tmp/gh-aw/mcp-payloads:/tmp/gh-aw/mcp-payloads:rw -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw -v '"${RUNNER_TEMP}"'/gh-aw/safeoutputs:'"${RUNNER_TEMP}"'/gh-aw/safeoutputs:rw ghcr.io/github/gh-aw-mcpg:v0.4.7' + export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network bridge -p 127.0.0.1:'"${MCP_GATEWAY_PORT}"':'"${MCP_GATEWAY_PORT}"' --name awmg-mcpg --add-host host.docker.internal:host-gateway --user '"${MCP_GATEWAY_UID}"':'"${MCP_GATEWAY_GID}"' --group-add '"${DOCKER_SOCK_GID}"' -v '"${DOCKER_SOCK_PATH}"':/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e MCP_GATEWAY_PAYLOAD_DIR -e MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD -e DOCKER_HOST=unix:///var/run/docker.sock -e DEBUG -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_POLICY_ALLOW_CREATE_PULL_REQUEST -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_GUARD_MIN_INTEGRITY -e GITHUB_MCP_GUARD_REPOS -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e GITHUB_JOB -e GITHUB_ACTION -e GITHUB_EVENT_NAME -e GITHUB_EVENT_PATH -e GITHUB_ACTOR -e GITHUB_ACTOR_ID -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_WORKFLOW_REF -e GITHUB_WORKFLOW_SHA -e GITHUB_REF -e GITHUB_REF_NAME -e GITHUB_REF_TYPE -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e RUNNER_TEMP -v /tmp/gh-aw/mcp-payloads:/tmp/gh-aw/mcp-payloads:rw -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw -v '"${RUNNER_TEMP}"'/gh-aw/safeoutputs:'"${RUNNER_TEMP}"'/gh-aw/safeoutputs:rw ghcr.io/github/gh-aw-mcpg:v0.4.6' mkdir -p "$HOME/.copilot" GH_AW_NODE=$(which node 2>/dev/null || command -v node 2>/dev/null || echo node) - cat << GH_AW_MCP_CONFIG_2455694971a4678a_EOF | "$GH_AW_NODE" "${RUNNER_TEMP}/gh-aw/actions/start_mcp_gateway.cjs" + cat << GH_AW_MCP_CONFIG_cd0fedea8e038870_EOF | "$GH_AW_NODE" "${RUNNER_TEMP}/gh-aw/actions/start_mcp_gateway.cjs" { "mcpServers": { "github": { "type": "stdio", - "container": "ghcr.io/github/github-mcp-server:v1.8.0", + "container": "ghcr.io/github/github-mcp-server:v1.7.0", "env": { "GITHUB_FEATURES": "fields_param", "GITHUB_HOST": "${GITHUB_SERVER_URL}", @@ -758,7 +750,7 @@ jobs: "private:azure/azure-sdk-for-rust", "private:azure/azure-rest-api-specs" ], - "sink-visibility": "${GH_AW_SINK_VISIBILITY}" + "sink-visibility": ${{ toJSON(steps.determine-automatic-lockdown.outputs.visibility) }} } } } @@ -771,7 +763,7 @@ jobs: "startupTimeout": 120 } } - GH_AW_MCP_CONFIG_2455694971a4678a_EOF + GH_AW_MCP_CONFIG_cd0fedea8e038870_EOF - name: Mount MCP servers as CLIs id: mount-mcp-clis continue-on-error: true @@ -815,7 +807,7 @@ jobs: export COPILOT_API_KEY="$COPILOT_DUMMY_BYOK" (umask 177 && touch /tmp/gh-aw/agent-stdio.log) GH_AW_MAX_AI_CREDITS="${GH_AW_MAX_AI_CREDITS:-1000}" - printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.43/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"*.githubusercontent.com\",\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"azure.github.io\",\"codeload.github.com\",\"docs.github.com\",\"docs.microsoft.com\",\"github-cloud.githubusercontent.com\",\"github-cloud.s3.amazonaws.com\",\"github.blog\",\"github.com\",\"github.githubassets.com\",\"host.docker.internal\",\"learn.microsoft.com\",\"lfs.github.com\",\"objects.githubusercontent.com\",\"patch-diff.githubusercontent.com\",\"patchdiff.githubusercontent.com\",\"raw.githubusercontent.com\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"],\"isolation\":true,\"topologyAttach\":[\"awmg-mcpg\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"auto\":[\"copilot/auto\",\"large\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"detection\":[\"small\"],\"evals\":[\"small\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"grok\":[\"copilot/*grok*\",\"openai/*grok*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.43,squid=sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d,agent=sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6,api-proxy=sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1,cli-proxy=sha256:65c45ea2967984d0024f3df61bc71335658a77ede96c8d9665da7a5f33a795ab\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" + printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.42/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"*.githubusercontent.com\",\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"azure.github.io\",\"codeload.github.com\",\"docs.github.com\",\"docs.microsoft.com\",\"github-cloud.githubusercontent.com\",\"github-cloud.s3.amazonaws.com\",\"github.blog\",\"github.com\",\"github.githubassets.com\",\"host.docker.internal\",\"learn.microsoft.com\",\"lfs.github.com\",\"objects.githubusercontent.com\",\"patch-diff.githubusercontent.com\",\"patchdiff.githubusercontent.com\",\"raw.githubusercontent.com\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"],\"isolation\":true,\"topologyAttach\":[\"awmg-mcpg\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"fable\",\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.42,squid=sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0,agent=sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b,agent-act=sha256:a14ad974484aa518aab83d40f3f141175dfd171d3745e01c092375b970f73a20,api-proxy=sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607,cli-proxy=sha256:da006bf96d2d246dd269d57b233c1798d2ad63d6cd64ca02f7bf71045028781f\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" cp "${RUNNER_TEMP}/gh-aw/awf-config.json" /tmp/gh-aw/awf-config.json export GH_AW_MODELS_JSON_PATH="/tmp/gh-aw/models.json" GH_AW_DOCKER_HOST="" @@ -840,7 +832,7 @@ jobs: COPILOT_AGENT_RUNNER_TYPE: STANDALONE COPILOT_DUMMY_BYOK: dummy-byok-key-for-offline-mode COPILOT_GITHUB_TOKEN: ${{ github.token }} - COPILOT_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'auto' }} + COPILOT_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }} GH_AW_LLM_PROVIDER: github GH_AW_MAX_AI_CREDITS: ${{ vars.GH_AW_DEFAULT_MAX_AI_CREDITS || '1000' }} GH_AW_MAX_TURNS: ${{ vars.GH_AW_DEFAULT_MAX_TURNS || '' }} @@ -848,7 +840,7 @@ jobs: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} GH_AW_TIMEOUT_MINUTES: 12 - GH_AW_VERSION: v0.84.3 + GH_AW_VERSION: v0.83.4 GITHUB_API_URL: ${{ github.api_url }} GITHUB_AW: true GITHUB_COPILOT_INTEGRATION_ID: agentic-workflows @@ -1022,6 +1014,7 @@ jobs: needs.activation.outputs.daily_ai_credits_exceeded == 'true') runs-on: ubuntu-slim permissions: + contents: read issues: write pull-requests: write concurrency: @@ -1038,7 +1031,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 + uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1047,8 +1040,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Review SDK PR" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/review-sdk.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.77" - GH_AW_INFO_AWF_VERSION: "v0.27.43" + GH_AW_INFO_VERSION: "1.0.75" + GH_AW_INFO_AWF_VERSION: "v0.27.42" GH_AW_INFO_ENGINE_ID: "copilot" - name: Download agent output artifact id: download-agent-output @@ -1075,7 +1068,30 @@ jobs: - name: Collect usage artifact files if: always() continue-on-error: true - run: bash "${RUNNER_TEMP}/gh-aw/actions/collect_usage_artifact_files.sh" + run: | + mkdir -p /tmp/gh-aw/usage/agent /tmp/gh-aw/usage/detection + echo "Usage artifact source file status:" + for file in /tmp/gh-aw/aw_info.json /tmp/gh-aw/aw-info.jsonl /tmp/gh-aw/agent_usage.json /tmp/gh-aw/agent_usage.jsonl /tmp/gh-aw/detection_usage.jsonl /tmp/gh-aw/evals/evals.jsonl /tmp/gh-aw/github_rate_limits.jsonl /tmp/gh-aw/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/threat-detection/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/threat-detection/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl; do + [ -f "$file" ] && echo "FOUND: $file" || echo "MISSING: $file" + done + [ -f /tmp/gh-aw/aw_info.json ] && cp /tmp/gh-aw/aw_info.json /tmp/gh-aw/usage/aw_info.json || true + [ -f /tmp/gh-aw/aw-info.jsonl ] && cp /tmp/gh-aw/aw-info.jsonl /tmp/gh-aw/usage/aw-info.jsonl || true + [ -f /tmp/gh-aw/agent_usage.json ] && cp /tmp/gh-aw/agent_usage.json /tmp/gh-aw/usage/agent_usage.json || true + [ -f /tmp/gh-aw/agent_usage.jsonl ] && cp /tmp/gh-aw/agent_usage.jsonl /tmp/gh-aw/usage/agent_usage.jsonl || true + [ -f /tmp/gh-aw/detection_usage.jsonl ] && cp /tmp/gh-aw/detection_usage.jsonl /tmp/gh-aw/usage/detection_usage.jsonl || true + [ -f /tmp/gh-aw/evals/evals.jsonl ] && cp /tmp/gh-aw/evals/evals.jsonl /tmp/gh-aw/usage/evals.jsonl || true + [ -f /tmp/gh-aw/github_rate_limits.jsonl ] && cp /tmp/gh-aw/github_rate_limits.jsonl /tmp/gh-aw/usage/github_rate_limits.jsonl || true + [ -s /tmp/gh-aw/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/agent/token_usage.jsonl || true + [ -s /tmp/gh-aw/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/agent/token_usage.jsonl || true + [ -s /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/agent/token_usage.jsonl || true + [ -s /tmp/gh-aw/threat-detection/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/threat-detection/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/detection/token_usage.jsonl || true + [ -s /tmp/gh-aw/threat-detection/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/threat-detection/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/detection/token_usage.jsonl || true + [ -s /tmp/gh-aw/threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/detection/token_usage.jsonl || true + [ -f /tmp/gh-aw/usage/agent/token_usage.jsonl ] || : > /tmp/gh-aw/usage/agent/token_usage.jsonl + [ -f /tmp/gh-aw/usage/detection/token_usage.jsonl ] || : > /tmp/gh-aw/usage/detection/token_usage.jsonl + mkdir -p /tmp/gh-aw/usage/activity + node "${RUNNER_TEMP}/gh-aw/actions/generate_usage_activity_summary.cjs" + find /tmp/gh-aw/usage -type f -print | sort - name: Upload usage artifact if: always() continue-on-error: true @@ -1227,9 +1243,6 @@ jobs: GH_AW_AGENTIC_ENGINE_TIMEOUT: ${{ needs.agent.outputs.agentic_engine_timeout }} GH_AW_MODEL_NOT_SUPPORTED_ERROR: ${{ needs.agent.outputs.model_not_supported_error }} GH_AW_HTTP_400_RESPONSE_ERROR: ${{ needs.agent.outputs.http_400_response_error }} - GH_AW_MAX_CACHE_MISSES_EXCEEDED: ${{ needs.agent.outputs.max_cache_misses_exceeded }} - GH_AW_MISSING_MODEL_PRICING_ERROR: ${{ needs.agent.outputs.missing_model_pricing_error }} - GH_AW_MISSING_MODEL_PRICING_MODEL_NAME: ${{ needs.agent.outputs.missing_model_pricing_model_name }} GH_AW_ENGINE_API_HOSTS: "api.enterprise.githubcopilot.com,api.githubcopilot.com,api.business.githubcopilot.com,api.individual.githubcopilot.com" GH_AW_LOCKDOWN_CHECK_FAILED: ${{ needs.activation.outputs.lockdown_check_failed }} GH_AW_OAUTH_TOKEN_CHECK_FAILED: ${{ needs.activation.outputs.oauth_token_check_failed }} @@ -1269,7 +1282,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 + uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1278,8 +1291,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Review SDK PR" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/review-sdk.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.77" - GH_AW_INFO_AWF_VERSION: "v0.27.43" + GH_AW_INFO_VERSION: "1.0.75" + GH_AW_INFO_AWF_VERSION: "v0.27.42" GH_AW_INFO_ENGINE_ID: "copilot" - name: Download agent output artifact id: download-agent-output @@ -1306,7 +1319,7 @@ jobs: rm -rf /tmp/gh-aw/sandbox/firewall/logs rm -rf /tmp/gh-aw/sandbox/firewall/audit - name: Download container images - run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6 ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1 ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d + run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607 ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0 - name: Check if detection needed id: detection_guard if: always() @@ -1338,14 +1351,10 @@ jobs: fi cp /tmp/gh-aw/agent_output.json /tmp/gh-aw/threat-detection/agent_output.json 2>/dev/null || true for f in /tmp/gh-aw/aw-*.patch; do - if [ -f "$f" ]; then - cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true - fi + [ -f "$f" ] && cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true done for f in /tmp/gh-aw/aw-*.bundle; do - if [ -f "$f" ]; then - cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true - fi + [ -f "$f" ] && cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true done echo "Prepared threat detection files:" ls -la /tmp/gh-aw/threat-detection/ 2>/dev/null || true @@ -1356,7 +1365,6 @@ jobs: WORKFLOW_NAME: "Review SDK PR" WORKFLOW_DESCRIPTION: "Agentic PR reviewer for Azure SDK for Rust changes.\nReviews API surface, crate/package conventions, workspace/Cargo wiring,\nrequired SDK metadata files, and security risks. Posts one PR comment." HAS_PATCH: ${{ needs.agent.outputs.has_patch }} - GH_AW_DETECTION_CONTINUE_ON_ERROR: "true" with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); @@ -1374,12 +1382,11 @@ jobs: node-version: '24' package-manager-cache: false - name: Install GitHub Copilot CLI - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" 1.0.75 env: GH_HOST: github.com - GH_AW_COMPILED_VERSION: v0.84.3 - name: Install AWF binary - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.43 + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.42 - name: Execute GitHub Copilot CLI if: always() && steps.detection_guard.outputs.run_detection == 'true' continue-on-error: true @@ -1399,7 +1406,7 @@ jobs: export COPILOT_API_KEY="$COPILOT_DUMMY_BYOK" (umask 177 && touch /tmp/gh-aw/threat-detection/detection.log) GH_AW_MAX_AI_CREDITS="${GH_AW_MAX_AI_CREDITS:-400}" - printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.43/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"github.com\",\"host.docker.internal\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"auto\":[\"copilot/auto\",\"large\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"detection\":[\"small\"],\"evals\":[\"small\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"grok\":[\"copilot/*grok*\",\"openai/*grok*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.43,squid=sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d,agent=sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6,api-proxy=sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1,cli-proxy=sha256:65c45ea2967984d0024f3df61bc71335658a77ede96c8d9665da7a5f33a795ab\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" + printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.42/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"github.com\",\"host.docker.internal\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"fable\",\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.42,squid=sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0,agent=sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b,agent-act=sha256:a14ad974484aa518aab83d40f3f141175dfd171d3745e01c092375b970f73a20,api-proxy=sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607,cli-proxy=sha256:da006bf96d2d246dd269d57b233c1798d2ad63d6cd64ca02f7bf71045028781f\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" cp "${RUNNER_TEMP}/gh-aw/awf-config.json" /tmp/gh-aw/awf-config.json export GH_AW_MODELS_JSON_PATH="/tmp/gh-aw/models.json" GH_AW_DOCKER_HOST="" @@ -1426,14 +1433,14 @@ jobs: COPILOT_AGENT_RUNNER_TYPE: STANDALONE COPILOT_DUMMY_BYOK: dummy-byok-key-for-offline-mode COPILOT_GITHUB_TOKEN: ${{ github.token }} - COPILOT_MODEL: detection + COPILOT_MODEL: ${{ vars.GH_AW_MODEL_DETECTION_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }} GH_AW_LLM_PROVIDER: github GH_AW_MAX_AI_CREDITS: ${{ vars.GH_AW_DEFAULT_DETECTION_MAX_AI_CREDITS || '400' }} GH_AW_MAX_TURNS: ${{ vars.GH_AW_DEFAULT_MAX_TURNS || '' }} GH_AW_PHASE: detection GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_TIMEOUT_MINUTES: 20 - GH_AW_VERSION: v0.84.3 + GH_AW_VERSION: v0.83.4 GITHUB_API_URL: ${{ github.api_url }} GITHUB_AW: true GITHUB_COPILOT_INTEGRATION_ID: agentic-workflows @@ -1510,6 +1517,7 @@ jobs: if: (!cancelled()) && needs.agent.result != 'skipped' && needs.detection.result == 'success' runs-on: ubuntu-slim permissions: + contents: read issues: write pull-requests: write timeout-minutes: 45 @@ -1523,7 +1531,7 @@ jobs: GH_AW_EFFECTIVE_TOKENS: ${{ needs.agent.outputs.effective_tokens }} GH_AW_ENGINE_ID: "copilot" GH_AW_ENGINE_MODEL: ${{ needs.agent.outputs.model }} - GH_AW_HEAD_SHA: ${{ github.event.pull_request.head.sha }} + GH_AW_ENGINE_VERSION: "1.0.75" GH_AW_RUNTIME_FEATURES: ${{ vars.GH_AW_RUNTIME_FEATURES }} GH_AW_THREAT_DETECTION_AIC: ${{ needs.detection.outputs.aic }} GH_AW_WORKFLOW_ID: "review-sdk" @@ -1541,7 +1549,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 + uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1550,8 +1558,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Review SDK PR" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/review-sdk.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.77" - GH_AW_INFO_AWF_VERSION: "v0.27.43" + GH_AW_INFO_VERSION: "1.0.75" + GH_AW_INFO_AWF_VERSION: "v0.27.42" GH_AW_INFO_ENGINE_ID: "copilot" - name: Download agent output artifact id: download-agent-output diff --git a/.github/workflows/verify-links.yml b/.github/workflows/verify-links.yml index f41d380f17a..540b7cb1cdc 100644 --- a/.github/workflows/verify-links.yml +++ b/.github/workflows/verify-links.yml @@ -27,10 +27,10 @@ jobs: (github.repository == 'Azure/azure-sdk-for-python' && contains(github.event.check_run.name, 'Analyze')) || (github.repository == 'Azure/azure-sdk-for-java' && contains(github.event.check_run.name, 'Analyze')) || (github.repository == 'Azure/azure-sdk-for-js' && contains(github.event.check_run.name, 'Analyze')) || + (github.repository == 'Azure/azure-sdk-for-c' && contains(github.event.check_run.name, 'GenerateReleaseArtifacts')) || (github.repository == 'Azure/azure-sdk-for-cpp' && contains(github.event.check_run.name, 'GenerateReleaseArtifacts')) || (github.repository == 'Azure/azure-sdk-for-go' && contains(github.event.check_run.name, 'Analyze')) || - (github.repository == 'Azure/azure-sdk-for-ios' && contains(github.event.check_run.name, 'Analyze')) || - (github.repository == 'Azure/azure-sdk-for-rust' && contains(github.event.check_run.name, 'Analyze')) + (github.repository == 'Azure/azure-sdk-for-ios' && contains(github.event.check_run.name, 'Analyze')) ) ) runs-on: ubuntu-latest From 0c2947545d98ea1629241bfe4a8a6044dcd97c12 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Tue, 4 Aug 2026 12:58:35 -0700 Subject: [PATCH 19/21] Build hedged counter in Instruments::new The hedged field was referenced in the Self { .. } literal but its let hedged = .. binding was lost in an earlier merge, so every build of azure_data_cosmos with the metrics feature failed with E0425. This broke all six ADO pullrequest jobs. Construct the counter from METRIC_OPERATION_HEDGED with the {operation} unit, matching how the sibling instruments are built. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../src/diagnostics/metrics/instruments.rs | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs index 07bbb8c20a9..86923f7f0c1 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs @@ -70,6 +70,14 @@ impl Instruments { .with_boundaries(attributes::BUCKETS_RETURNED_ROWS.to_vec()) .build(); + let hedged = meter + .u64_counter(attributes::METRIC_OPERATION_HEDGED) + .with_unit(attributes::UNIT_OPERATION) + .with_description( + "Number of Cosmos DB operations that dispatched a cross-region hedge.", + ) + .build(); + let active_instance = meter .i64_up_down_counter(attributes::METRIC_ACTIVE_INSTANCE_COUNT) .with_unit(attributes::UNIT_INSTANCE) From a2d4581bbd6e7c7286f211ea8d5f992dab24e6a7 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Tue, 4 Aug 2026 13:26:30 -0700 Subject: [PATCH 20/21] Use US spelling of finalization in region docs The Check spelling (cspell) task in the Build Analyze job rejected the British `finalisation` in the module docs for the SDK-owned region types. The repo dictionary standardises on US spellings, so switch to `finalization` rather than widening the dictionary for a doc comment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs b/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs index 24b9c638205..2af2204d744 100644 --- a/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs +++ b/sdk/cosmos/azure_data_cosmos/src/diagnostics/region.rs @@ -6,7 +6,7 @@ //! These types mirror the cross-SDK Hedging Detection API's `RequestedRegion` //! and `RequestedRegionReason` while remaining wholly owned by //! `azure_data_cosmos`. They are projected from the driver equivalents at -//! diagnostics-context finalisation, which lets the driver evolve its internal +//! diagnostics-context finalization, which lets the driver evolve its internal //! model without forcing an SDK major-version bump. use crate::options::Region; From d1af5e3f1358ea1f6a5cb3bcdfdf7ba3678b5b36 Mon Sep 17 00:00:00 2001 From: Nalu Tripician Date: Tue, 4 Aug 2026 15:25:32 -0700 Subject: [PATCH 21/21] Remove unrelated workflow changes Restore agentic workflow and link-verification files to upstream main so PR #4871 remains scoped to Cosmos hedging diagnostics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 821eaa5c-6f91-4dcd-b744-c098fa1550c6 --- .github/aw/actions-lock.json | 6 +- .github/workflows/issue-triage.lock.yml | 144 +++++++++++------------ .github/workflows/issue-triage.md | 3 +- .github/workflows/review-sdk.lock.yml | 148 +++++++++++------------- .github/workflows/verify-links.yml | 4 +- 5 files changed, 143 insertions(+), 162 deletions(-) diff --git a/.github/aw/actions-lock.json b/.github/aw/actions-lock.json index 4581b1f584d..c6e1d47550e 100644 --- a/.github/aw/actions-lock.json +++ b/.github/aw/actions-lock.json @@ -30,10 +30,10 @@ "version": "v7.0.1", "sha": "043fb46d1a93c77aae656e7c1c64a875d1fc6a0a" }, - "github/gh-aw-actions/setup@v0.83.4": { + "github/gh-aw-actions/setup@v0.84.3": { "repo": "github/gh-aw-actions/setup", - "version": "v0.83.4", - "sha": "e89c65e17eb281bbd5ff2ff9e9199a03e96654c7" + "version": "v0.84.3", + "sha": "c863074b673419603d146aab585e2986ef08deec" } } } diff --git a/.github/workflows/issue-triage.lock.yml b/.github/workflows/issue-triage.lock.yml index 1dcbbcd356b..3e5acd12715 100644 --- a/.github/workflows/issue-triage.lock.yml +++ b/.github/workflows/issue-triage.lock.yml @@ -1,6 +1,6 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"21695fba876d02c38f79a1b4e7d8fbff7641224a0b4007e499af975192cab924","body_hash":"6a8f1950d11870316a376a455938cc3bd282bd0ce9c16b235f99948710110519","compiler_version":"v0.83.4","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.75"}} -# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"373c709c69115d41ff229c7e5df9f8788daa9553","version":"v9"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"e89c65e17eb281bbd5ff2ff9e9199a03e96654c7","version":"v0.83.4"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.42","digest":"sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42","digest":"sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.42","digest":"sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.6","digest":"sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.6@sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748","pinned_image":"ghcr.io/github/gh-aw-node@sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748"},{"image":"ghcr.io/github/github-mcp-server:v1.7.0","digest":"sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308","pinned_image":"ghcr.io/github/github-mcp-server:v1.7.0@sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308"}]} -# This file was automatically generated by gh-aw (v0.83.4). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"311a18aa564b594a480c8f759d9514808444d70dd631916b8dcad2a882d2400d","body_hash":"6a8f1950d11870316a376a455938cc3bd282bd0ce9c16b235f99948710110519","compiler_version":"v0.84.3","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.77"}} +# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"373c709c69115d41ff229c7e5df9f8788daa9553","version":"v9"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"c863074b673419603d146aab585e2986ef08deec","version":"v0.84.3"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.43","digest":"sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43","digest":"sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.43","digest":"sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.7","digest":"sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.7@sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196"},{"image":"ghcr.io/github/github-mcp-server:v1.8.0","digest":"sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520","pinned_image":"ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520"}]} +# This file was automatically generated by gh-aw (v0.84.3). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # # ___ _ _ # / _ \ | | (_) @@ -46,15 +46,15 @@ # - actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 # - actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 # - actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 -# - github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 +# - github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 # # Container images used: -# - ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b -# - ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607 -# - ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0 -# - ghcr.io/github/gh-aw-mcpg:v0.4.6@sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c -# - ghcr.io/github/gh-aw-node@sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748 -# - ghcr.io/github/github-mcp-server:v1.7.0@sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308 +# - ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6 +# - ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1 +# - ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d +# - ghcr.io/github/gh-aw-mcpg:v0.4.7@sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00 +# - ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196 +# - ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520 name: "Agentic Triage" on: @@ -100,7 +100,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 + uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -108,8 +108,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Agentic Triage" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/issue-triage.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.75" - GH_AW_INFO_AWF_VERSION: "v0.27.42" + GH_AW_INFO_VERSION: "1.0.77" + GH_AW_INFO_AWF_VERSION: "v0.27.43" GH_AW_INFO_BODY_MODIFIED: "false" GH_AW_INFO_ENGINE_ID: "copilot" - name: Generate agentic run info @@ -117,17 +117,17 @@ jobs: env: GH_AW_INFO_ENGINE_ID: "copilot" GH_AW_INFO_ENGINE_NAME: "GitHub Copilot CLI" - GH_AW_INFO_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }} - GH_AW_INFO_VERSION: "1.0.75" - GH_AW_INFO_AGENT_VERSION: "1.0.75" - GH_AW_INFO_CLI_VERSION: "v0.83.4" + GH_AW_INFO_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'auto' }} + GH_AW_INFO_VERSION: "1.0.77" + GH_AW_INFO_AGENT_VERSION: "1.0.77" + GH_AW_INFO_CLI_VERSION: "v0.84.3" GH_AW_INFO_WORKFLOW_NAME: "Agentic Triage" GH_AW_INFO_EXPERIMENTAL: "false" GH_AW_INFO_SUPPORTS_TOOLS_ALLOWLIST: "true" GH_AW_INFO_STAGED: "false" GH_AW_INFO_ALLOWED_DOMAINS: '["github","threat-detection"]' GH_AW_INFO_FIREWALL_ENABLED: "true" - GH_AW_INFO_AWF_VERSION: "v0.27.42" + GH_AW_INFO_AWF_VERSION: "v0.27.43" GH_AW_INFO_AWMG_VERSION: "" GH_AW_INFO_FIREWALL_TYPE: "squid" GH_AW_INFO_FRONTMATTER_SOURCE: "githubnext/agentics/workflows/issue-triage.md@8e6d7c86bba37371d2d0eee1a23563db3e561eb5" @@ -240,7 +240,7 @@ jobs: - name: Check compile-agentic version uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 env: - GH_AW_COMPILED_VERSION: "v0.83.4" + GH_AW_COMPILED_VERSION: "v0.84.3" with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); @@ -433,7 +433,10 @@ jobs: http_400_response_error: ${{ steps.detect-agent-errors.outputs.http_400_response_error || 'false' }} inference_access_error: ${{ steps.detect-agent-errors.outputs.inference_access_error || 'false' }} invocation_cap_exceeded: ${{ steps.detect-agent-errors.outputs.invocation_cap_exceeded || 'false' }} + max_cache_misses_exceeded: ${{ steps.detect-agent-errors.outputs.max_cache_misses_exceeded || 'false' }} mcp_policy_error: ${{ steps.detect-agent-errors.outputs.mcp_policy_error || 'false' }} + missing_model_pricing_error: ${{ steps.detect-agent-errors.outputs.missing_model_pricing_error || 'false' }} + missing_model_pricing_model_name: ${{ steps.detect-agent-errors.outputs.missing_model_pricing_model_name || '' }} model: ${{ needs.activation.outputs.model }} model_not_supported_error: ${{ steps.detect-agent-errors.outputs.model_not_supported_error || 'false' }} output: ${{ steps.collect_output.outputs.output }} @@ -445,7 +448,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 + uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -454,8 +457,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Agentic Triage" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/issue-triage.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.75" - GH_AW_INFO_AWF_VERSION: "v0.27.42" + GH_AW_INFO_VERSION: "1.0.77" + GH_AW_INFO_AWF_VERSION: "v0.27.43" GH_AW_INFO_BODY_MODIFIED: "false" GH_AW_INFO_ENGINE_ID: "copilot" - name: Set runtime paths @@ -502,11 +505,12 @@ jobs: const { main } = require('${{ runner.temp }}/gh-aw/actions/checkout_pr_branch.cjs'); await main(); - name: Install GitHub Copilot CLI - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" 1.0.75 + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" env: GH_HOST: github.com + GH_AW_COMPILED_VERSION: v0.84.3 - name: Install AWF binary - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.42 --rootless + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.43 --rootless - name: Determine automatic lockdown mode for GitHub MCP Server id: determine-automatic-lockdown uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 @@ -542,7 +546,7 @@ jobs: GH_AW_SKILL_DIR: ".github/skills" run: bash "${RUNNER_TEMP}/gh-aw/actions/restore_inline_skills.sh" - name: Download container images - run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607 ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0 ghcr.io/github/gh-aw-mcpg:v0.4.6@sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c ghcr.io/github/gh-aw-node@sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748 ghcr.io/github/github-mcp-server:v1.7.0@sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308 + run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6 ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1 ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d ghcr.io/github/gh-aw-mcpg:v0.4.7@sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00 ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196 ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520 - name: Generate Safe Outputs Config run: | mkdir -p "${RUNNER_TEMP}/gh-aw/safeoutputs" @@ -747,6 +751,7 @@ jobs: GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} GH_AW_SAFE_OUTPUTS_CONFIG_PATH: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS_CONFIG_PATH }} GH_AW_SAFE_OUTPUTS_TOOLS_PATH: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS_TOOLS_PATH }} + GH_AW_SINK_VISIBILITY: ${{ steps.determine-automatic-lockdown.outputs.visibility }} GITHUB_MCP_SERVER_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | @@ -769,22 +774,22 @@ jobs: MCP_GATEWAY_UID=$(id -u 2>/dev/null || echo '0') MCP_GATEWAY_GID=$(id -g 2>/dev/null || echo '0') source "${RUNNER_TEMP}/gh-aw/actions/resolve_docker_socket_gid.sh" - export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network bridge -p 127.0.0.1:'"${MCP_GATEWAY_PORT}"':'"${MCP_GATEWAY_PORT}"' --name awmg-mcpg --add-host host.docker.internal:host-gateway --user '"${MCP_GATEWAY_UID}"':'"${MCP_GATEWAY_GID}"' --group-add '"${DOCKER_SOCK_GID}"' -v '"${DOCKER_SOCK_PATH}"':/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e MCP_GATEWAY_PAYLOAD_DIR -e MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD -e DOCKER_HOST=unix:///var/run/docker.sock -e DEBUG -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_POLICY_ALLOW_CREATE_PULL_REQUEST -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_GUARD_MIN_INTEGRITY -e GITHUB_MCP_GUARD_REPOS -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e GITHUB_JOB -e GITHUB_ACTION -e GITHUB_EVENT_NAME -e GITHUB_EVENT_PATH -e GITHUB_ACTOR -e GITHUB_ACTOR_ID -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_WORKFLOW_REF -e GITHUB_WORKFLOW_SHA -e GITHUB_REF -e GITHUB_REF_NAME -e GITHUB_REF_TYPE -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e RUNNER_TEMP -v /tmp/gh-aw/mcp-payloads:/tmp/gh-aw/mcp-payloads:rw -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw -v '"${RUNNER_TEMP}"'/gh-aw/safeoutputs:'"${RUNNER_TEMP}"'/gh-aw/safeoutputs:rw ghcr.io/github/gh-aw-mcpg:v0.4.6' + export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network bridge -p 127.0.0.1:'"${MCP_GATEWAY_PORT}"':'"${MCP_GATEWAY_PORT}"' --name awmg-mcpg --add-host host.docker.internal:host-gateway --user '"${MCP_GATEWAY_UID}"':'"${MCP_GATEWAY_GID}"' --group-add '"${DOCKER_SOCK_GID}"' -v '"${DOCKER_SOCK_PATH}"':/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e MCP_GATEWAY_PAYLOAD_DIR -e MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD -e DOCKER_HOST=unix:///var/run/docker.sock -e DEBUG -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_POLICY_ALLOW_CREATE_PULL_REQUEST -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_GUARD_MIN_INTEGRITY -e GITHUB_MCP_GUARD_REPOS -e GH_AW_SINK_VISIBILITY -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e GITHUB_JOB -e GITHUB_ACTION -e GITHUB_EVENT_NAME -e GITHUB_EVENT_PATH -e GITHUB_ACTOR -e GITHUB_ACTOR_ID -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_WORKFLOW_REF -e GITHUB_WORKFLOW_SHA -e GITHUB_REF -e GITHUB_REF_NAME -e GITHUB_REF_TYPE -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e RUNNER_TEMP -v /tmp/gh-aw/mcp-payloads:/tmp/gh-aw/mcp-payloads:rw -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw -v '"${RUNNER_TEMP}"'/gh-aw/safeoutputs:'"${RUNNER_TEMP}"'/gh-aw/safeoutputs:rw ghcr.io/github/gh-aw-mcpg:v0.4.7' mkdir -p "$HOME/.copilot" GH_AW_NODE=$(which node 2>/dev/null || command -v node 2>/dev/null || echo node) - cat << GH_AW_MCP_CONFIG_d808373827ca217a_EOF | "$GH_AW_NODE" "${RUNNER_TEMP}/gh-aw/actions/start_mcp_gateway.cjs" + cat << GH_AW_MCP_CONFIG_aaacf455c8d7e251_EOF | "$GH_AW_NODE" "${RUNNER_TEMP}/gh-aw/actions/start_mcp_gateway.cjs" { "mcpServers": { "github": { "type": "stdio", - "container": "ghcr.io/github/github-mcp-server:v1.7.0", + "container": "ghcr.io/github/github-mcp-server:v1.8.0", "env": { "GITHUB_FEATURES": "fields_param", "GITHUB_HOST": "${GITHUB_SERVER_URL}", "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_MCP_SERVER_TOKEN}", "GITHUB_READ_ONLY": "1", - "GITHUB_TOOLSETS": "issues,pull_requests" + "GITHUB_TOOLSETS": "issues,pull_requests,repos" }, "guard-policies": { "allow-only": { @@ -827,7 +832,7 @@ jobs: "accept": [ "private:azure/azure-sdk-for-rust" ], - "sink-visibility": ${{ toJSON(steps.determine-automatic-lockdown.outputs.visibility) }} + "sink-visibility": "${GH_AW_SINK_VISIBILITY}" } } } @@ -840,7 +845,7 @@ jobs: "startupTimeout": 120 } } - GH_AW_MCP_CONFIG_d808373827ca217a_EOF + GH_AW_MCP_CONFIG_aaacf455c8d7e251_EOF - name: Mount MCP servers as CLIs id: mount-mcp-clis continue-on-error: true @@ -883,7 +888,7 @@ jobs: export COPILOT_API_KEY="$COPILOT_DUMMY_BYOK" (umask 177 && touch /tmp/gh-aw/agent-stdio.log) GH_AW_MAX_AI_CREDITS="${GH_AW_MAX_AI_CREDITS:-1000}" - printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.42/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"*.githubusercontent.com\",\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"codeload.github.com\",\"docs.github.com\",\"github-cloud.githubusercontent.com\",\"github-cloud.s3.amazonaws.com\",\"github.blog\",\"github.com\",\"github.githubassets.com\",\"host.docker.internal\",\"lfs.github.com\",\"objects.githubusercontent.com\",\"patch-diff.githubusercontent.com\",\"patchdiff.githubusercontent.com\",\"raw.githubusercontent.com\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"],\"blockDomains\":[\"registry.npmjs.org\"],\"isolation\":true,\"topologyAttach\":[\"awmg-mcpg\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"fable\",\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.42,squid=sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0,agent=sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b,agent-act=sha256:a14ad974484aa518aab83d40f3f141175dfd171d3745e01c092375b970f73a20,api-proxy=sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607,cli-proxy=sha256:da006bf96d2d246dd269d57b233c1798d2ad63d6cd64ca02f7bf71045028781f\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" + printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.43/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"*.githubusercontent.com\",\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"codeload.github.com\",\"docs.github.com\",\"github-cloud.githubusercontent.com\",\"github-cloud.s3.amazonaws.com\",\"github.blog\",\"github.com\",\"github.githubassets.com\",\"host.docker.internal\",\"lfs.github.com\",\"objects.githubusercontent.com\",\"patch-diff.githubusercontent.com\",\"patchdiff.githubusercontent.com\",\"raw.githubusercontent.com\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"],\"blockDomains\":[\"registry.npmjs.org\"],\"isolation\":true,\"topologyAttach\":[\"awmg-mcpg\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"auto\":[\"copilot/auto\",\"large\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"detection\":[\"small\"],\"evals\":[\"small\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"grok\":[\"copilot/*grok*\",\"openai/*grok*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.43,squid=sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d,agent=sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6,api-proxy=sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1,cli-proxy=sha256:65c45ea2967984d0024f3df61bc71335658a77ede96c8d9665da7a5f33a795ab\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" cp "${RUNNER_TEMP}/gh-aw/awf-config.json" /tmp/gh-aw/awf-config.json export GH_AW_MODELS_JSON_PATH="/tmp/gh-aw/models.json" GH_AW_DOCKER_HOST="" @@ -908,7 +913,7 @@ jobs: COPILOT_AGENT_RUNNER_TYPE: STANDALONE COPILOT_DUMMY_BYOK: dummy-byok-key-for-offline-mode COPILOT_GITHUB_TOKEN: ${{ github.token }} - COPILOT_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }} + COPILOT_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'auto' }} GH_AW_LLM_PROVIDER: github GH_AW_MAX_AI_CREDITS: ${{ vars.GH_AW_DEFAULT_MAX_AI_CREDITS || '1000' }} GH_AW_MAX_TURNS: ${{ vars.GH_AW_DEFAULT_MAX_TURNS || '' }} @@ -916,7 +921,7 @@ jobs: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} GH_AW_TIMEOUT_MINUTES: 10 - GH_AW_VERSION: v0.83.4 + GH_AW_VERSION: v0.84.3 GITHUB_API_URL: ${{ github.api_url }} GITHUB_AW: true GITHUB_COPILOT_INTEGRATION_ID: agentic-workflows @@ -1090,7 +1095,6 @@ jobs: needs.activation.outputs.daily_ai_credits_exceeded == 'true') runs-on: ubuntu-slim permissions: - contents: read issues: write pull-requests: write concurrency: @@ -1107,7 +1111,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 + uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1116,8 +1120,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Agentic Triage" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/issue-triage.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.75" - GH_AW_INFO_AWF_VERSION: "v0.27.42" + GH_AW_INFO_VERSION: "1.0.77" + GH_AW_INFO_AWF_VERSION: "v0.27.43" GH_AW_INFO_BODY_MODIFIED: "false" GH_AW_INFO_ENGINE_ID: "copilot" - name: Download agent output artifact @@ -1145,30 +1149,7 @@ jobs: - name: Collect usage artifact files if: always() continue-on-error: true - run: | - mkdir -p /tmp/gh-aw/usage/agent /tmp/gh-aw/usage/detection - echo "Usage artifact source file status:" - for file in /tmp/gh-aw/aw_info.json /tmp/gh-aw/aw-info.jsonl /tmp/gh-aw/agent_usage.json /tmp/gh-aw/agent_usage.jsonl /tmp/gh-aw/detection_usage.jsonl /tmp/gh-aw/evals/evals.jsonl /tmp/gh-aw/github_rate_limits.jsonl /tmp/gh-aw/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/threat-detection/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/threat-detection/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl; do - [ -f "$file" ] && echo "FOUND: $file" || echo "MISSING: $file" - done - [ -f /tmp/gh-aw/aw_info.json ] && cp /tmp/gh-aw/aw_info.json /tmp/gh-aw/usage/aw_info.json || true - [ -f /tmp/gh-aw/aw-info.jsonl ] && cp /tmp/gh-aw/aw-info.jsonl /tmp/gh-aw/usage/aw-info.jsonl || true - [ -f /tmp/gh-aw/agent_usage.json ] && cp /tmp/gh-aw/agent_usage.json /tmp/gh-aw/usage/agent_usage.json || true - [ -f /tmp/gh-aw/agent_usage.jsonl ] && cp /tmp/gh-aw/agent_usage.jsonl /tmp/gh-aw/usage/agent_usage.jsonl || true - [ -f /tmp/gh-aw/detection_usage.jsonl ] && cp /tmp/gh-aw/detection_usage.jsonl /tmp/gh-aw/usage/detection_usage.jsonl || true - [ -f /tmp/gh-aw/evals/evals.jsonl ] && cp /tmp/gh-aw/evals/evals.jsonl /tmp/gh-aw/usage/evals.jsonl || true - [ -f /tmp/gh-aw/github_rate_limits.jsonl ] && cp /tmp/gh-aw/github_rate_limits.jsonl /tmp/gh-aw/usage/github_rate_limits.jsonl || true - [ -s /tmp/gh-aw/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/agent/token_usage.jsonl || true - [ -s /tmp/gh-aw/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/agent/token_usage.jsonl || true - [ -s /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/agent/token_usage.jsonl || true - [ -s /tmp/gh-aw/threat-detection/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/threat-detection/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/detection/token_usage.jsonl || true - [ -s /tmp/gh-aw/threat-detection/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/threat-detection/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/detection/token_usage.jsonl || true - [ -s /tmp/gh-aw/threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/detection/token_usage.jsonl || true - [ -f /tmp/gh-aw/usage/agent/token_usage.jsonl ] || : > /tmp/gh-aw/usage/agent/token_usage.jsonl - [ -f /tmp/gh-aw/usage/detection/token_usage.jsonl ] || : > /tmp/gh-aw/usage/detection/token_usage.jsonl - mkdir -p /tmp/gh-aw/usage/activity - node "${RUNNER_TEMP}/gh-aw/actions/generate_usage_activity_summary.cjs" - find /tmp/gh-aw/usage -type f -print | sort + run: bash "${RUNNER_TEMP}/gh-aw/actions/collect_usage_artifact_files.sh" - name: Upload usage artifact if: always() continue-on-error: true @@ -1325,6 +1306,9 @@ jobs: GH_AW_AGENTIC_ENGINE_TIMEOUT: ${{ needs.agent.outputs.agentic_engine_timeout }} GH_AW_MODEL_NOT_SUPPORTED_ERROR: ${{ needs.agent.outputs.model_not_supported_error }} GH_AW_HTTP_400_RESPONSE_ERROR: ${{ needs.agent.outputs.http_400_response_error }} + GH_AW_MAX_CACHE_MISSES_EXCEEDED: ${{ needs.agent.outputs.max_cache_misses_exceeded }} + GH_AW_MISSING_MODEL_PRICING_ERROR: ${{ needs.agent.outputs.missing_model_pricing_error }} + GH_AW_MISSING_MODEL_PRICING_MODEL_NAME: ${{ needs.agent.outputs.missing_model_pricing_model_name }} GH_AW_ENGINE_API_HOSTS: "api.enterprise.githubcopilot.com,api.githubcopilot.com,api.business.githubcopilot.com,api.individual.githubcopilot.com" GH_AW_LOCKDOWN_CHECK_FAILED: ${{ needs.activation.outputs.lockdown_check_failed }} GH_AW_OAUTH_TOKEN_CHECK_FAILED: ${{ needs.activation.outputs.oauth_token_check_failed }} @@ -1364,7 +1348,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 + uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1373,8 +1357,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Agentic Triage" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/issue-triage.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.75" - GH_AW_INFO_AWF_VERSION: "v0.27.42" + GH_AW_INFO_VERSION: "1.0.77" + GH_AW_INFO_AWF_VERSION: "v0.27.43" GH_AW_INFO_BODY_MODIFIED: "false" GH_AW_INFO_ENGINE_ID: "copilot" - name: Download agent output artifact @@ -1402,7 +1386,7 @@ jobs: rm -rf /tmp/gh-aw/sandbox/firewall/logs rm -rf /tmp/gh-aw/sandbox/firewall/audit - name: Download container images - run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607 ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0 + run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6 ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1 ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d - name: Check if detection needed id: detection_guard if: always() @@ -1434,10 +1418,14 @@ jobs: fi cp /tmp/gh-aw/agent_output.json /tmp/gh-aw/threat-detection/agent_output.json 2>/dev/null || true for f in /tmp/gh-aw/aw-*.patch; do - [ -f "$f" ] && cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true + if [ -f "$f" ]; then + cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true + fi done for f in /tmp/gh-aw/aw-*.bundle; do - [ -f "$f" ] && cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true + if [ -f "$f" ]; then + cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true + fi done echo "Prepared threat detection files:" ls -la /tmp/gh-aw/threat-detection/ 2>/dev/null || true @@ -1448,6 +1436,7 @@ jobs: WORKFLOW_NAME: "Agentic Triage" WORKFLOW_DESCRIPTION: "Intelligent issue triage assistant that processes new issues.\nAnalyzes issue content, evaluates whether the author is an external customer,\npredicts category and service labels, looks up owners from CODEOWNERS, and\nposts an analysis comment. Implements the initial issue triage rules for the\nAzure SDK for Rust repository." HAS_PATCH: ${{ needs.agent.outputs.has_patch }} + GH_AW_DETECTION_CONTINUE_ON_ERROR: "true" with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); @@ -1465,11 +1454,12 @@ jobs: node-version: '24' package-manager-cache: false - name: Install GitHub Copilot CLI - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" 1.0.75 + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" env: GH_HOST: github.com + GH_AW_COMPILED_VERSION: v0.84.3 - name: Install AWF binary - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.42 + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.43 - name: Execute GitHub Copilot CLI if: always() && steps.detection_guard.outputs.run_detection == 'true' continue-on-error: true @@ -1489,7 +1479,7 @@ jobs: export COPILOT_API_KEY="$COPILOT_DUMMY_BYOK" (umask 177 && touch /tmp/gh-aw/threat-detection/detection.log) GH_AW_MAX_AI_CREDITS="${GH_AW_MAX_AI_CREDITS:-400}" - printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.42/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"github.com\",\"host.docker.internal\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"fable\",\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.42,squid=sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0,agent=sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b,agent-act=sha256:a14ad974484aa518aab83d40f3f141175dfd171d3745e01c092375b970f73a20,api-proxy=sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607,cli-proxy=sha256:da006bf96d2d246dd269d57b233c1798d2ad63d6cd64ca02f7bf71045028781f\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" + printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.43/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"github.com\",\"host.docker.internal\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"auto\":[\"copilot/auto\",\"large\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"detection\":[\"small\"],\"evals\":[\"small\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"grok\":[\"copilot/*grok*\",\"openai/*grok*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.43,squid=sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d,agent=sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6,api-proxy=sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1,cli-proxy=sha256:65c45ea2967984d0024f3df61bc71335658a77ede96c8d9665da7a5f33a795ab\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" cp "${RUNNER_TEMP}/gh-aw/awf-config.json" /tmp/gh-aw/awf-config.json export GH_AW_MODELS_JSON_PATH="/tmp/gh-aw/models.json" GH_AW_DOCKER_HOST="" @@ -1516,14 +1506,14 @@ jobs: COPILOT_AGENT_RUNNER_TYPE: STANDALONE COPILOT_DUMMY_BYOK: dummy-byok-key-for-offline-mode COPILOT_GITHUB_TOKEN: ${{ github.token }} - COPILOT_MODEL: ${{ vars.GH_AW_MODEL_DETECTION_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }} + COPILOT_MODEL: detection GH_AW_LLM_PROVIDER: github GH_AW_MAX_AI_CREDITS: ${{ vars.GH_AW_DEFAULT_DETECTION_MAX_AI_CREDITS || '400' }} GH_AW_MAX_TURNS: ${{ vars.GH_AW_DEFAULT_MAX_TURNS || '' }} GH_AW_PHASE: detection GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_TIMEOUT_MINUTES: 20 - GH_AW_VERSION: v0.83.4 + GH_AW_VERSION: v0.84.3 GITHUB_API_URL: ${{ github.api_url }} GITHUB_AW: true GITHUB_COPILOT_INTEGRATION_ID: agentic-workflows @@ -1600,7 +1590,6 @@ jobs: if: (!cancelled()) && needs.agent.result != 'skipped' && needs.detection.result == 'success' runs-on: ubuntu-slim permissions: - contents: read issues: write pull-requests: write timeout-minutes: 45 @@ -1614,7 +1603,6 @@ jobs: GH_AW_EFFECTIVE_TOKENS: ${{ needs.agent.outputs.effective_tokens }} GH_AW_ENGINE_ID: "copilot" GH_AW_ENGINE_MODEL: ${{ needs.agent.outputs.model }} - GH_AW_ENGINE_VERSION: "1.0.75" GH_AW_RUNTIME_FEATURES: ${{ vars.GH_AW_RUNTIME_FEATURES }} GH_AW_THREAT_DETECTION_AIC: ${{ needs.detection.outputs.aic }} GH_AW_WORKFLOW_ID: "issue-triage" @@ -1634,7 +1622,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 + uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1643,8 +1631,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Agentic Triage" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/issue-triage.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.75" - GH_AW_INFO_AWF_VERSION: "v0.27.42" + GH_AW_INFO_VERSION: "1.0.77" + GH_AW_INFO_AWF_VERSION: "v0.27.43" GH_AW_INFO_BODY_MODIFIED: "false" GH_AW_INFO_ENGINE_ID: "copilot" - name: Download agent output artifact diff --git a/.github/workflows/issue-triage.md b/.github/workflows/issue-triage.md index d8e45295ad8..bc067c1542d 100644 --- a/.github/workflows/issue-triage.md +++ b/.github/workflows/issue-triage.md @@ -45,7 +45,8 @@ safe-outputs: tools: bash: false github: - toolsets: [issues, pull_requests] + # The gateway uses search_repositories to determine repository visibility. + toolsets: [issues, pull_requests, repos] # If in a public repo, setting `lockdown: false` allows # reading issues, pull requests and comments from 3rd-parties. # If in a private repo this has no particular effect. diff --git a/.github/workflows/review-sdk.lock.yml b/.github/workflows/review-sdk.lock.yml index b6b593b6f10..3f0063d5d0c 100644 --- a/.github/workflows/review-sdk.lock.yml +++ b/.github/workflows/review-sdk.lock.yml @@ -1,6 +1,6 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"bfad664f863791b31456b318276c13868bd56a2afdb0dd7b7e72563f525b8608","body_hash":"70079c5f1c5bf0de92b2032b7b41dff0b140492a1386fc98d63b6b0ea8559c5c","compiler_version":"v0.83.4","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.75"}} -# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"373c709c69115d41ff229c7e5df9f8788daa9553","version":"v9"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"e89c65e17eb281bbd5ff2ff9e9199a03e96654c7","version":"v0.83.4"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.42","digest":"sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42","digest":"sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.42","digest":"sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.6","digest":"sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.6@sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748","pinned_image":"ghcr.io/github/gh-aw-node@sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748"},{"image":"ghcr.io/github/github-mcp-server:v1.7.0","digest":"sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308","pinned_image":"ghcr.io/github/github-mcp-server:v1.7.0@sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308"}],"has_pull_request":true} -# This file was automatically generated by gh-aw (v0.83.4). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"bfad664f863791b31456b318276c13868bd56a2afdb0dd7b7e72563f525b8608","body_hash":"70079c5f1c5bf0de92b2032b7b41dff0b140492a1386fc98d63b6b0ea8559c5c","compiler_version":"v0.84.3","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.77"}} +# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"373c709c69115d41ff229c7e5df9f8788daa9553","version":"v9"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"c863074b673419603d146aab585e2986ef08deec","version":"v0.84.3"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.43","digest":"sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43","digest":"sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.43","digest":"sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.7","digest":"sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.7@sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196"},{"image":"ghcr.io/github/github-mcp-server:v1.8.0","digest":"sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520","pinned_image":"ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520"}],"has_pull_request":true} +# This file was automatically generated by gh-aw (v0.84.3). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # # ___ _ _ # / _ \ | | (_) @@ -42,15 +42,15 @@ # - actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 # - actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 # - actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 -# - github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 +# - github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 # # Container images used: -# - ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b -# - ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607 -# - ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0 -# - ghcr.io/github/gh-aw-mcpg:v0.4.6@sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c -# - ghcr.io/github/gh-aw-node@sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748 -# - ghcr.io/github/github-mcp-server:v1.7.0@sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308 +# - ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6 +# - ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1 +# - ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d +# - ghcr.io/github/gh-aw-mcpg:v0.4.7@sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00 +# - ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196 +# - ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520 name: "Review SDK PR" on: @@ -69,7 +69,10 @@ run-name: "Review SDK PR" jobs: activation: - if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.id == github.repository_id + if: > + (github.event_name != 'pull_request' || github.event.pull_request.head.repo.id == github.repository_id) && + ((github.event_name != 'pull_request' && github.event_name != 'pull_request_review') || github.event.pull_request.stack == null || + github.event.pull_request.stack.position == github.event.pull_request.stack.size) runs-on: ubuntu-slim permissions: actions: read @@ -99,7 +102,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 + uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -107,25 +110,25 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Review SDK PR" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/review-sdk.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.75" - GH_AW_INFO_AWF_VERSION: "v0.27.42" + GH_AW_INFO_VERSION: "1.0.77" + GH_AW_INFO_AWF_VERSION: "v0.27.43" GH_AW_INFO_ENGINE_ID: "copilot" - name: Generate agentic run info id: generate_aw_info env: GH_AW_INFO_ENGINE_ID: "copilot" GH_AW_INFO_ENGINE_NAME: "GitHub Copilot CLI" - GH_AW_INFO_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }} - GH_AW_INFO_VERSION: "1.0.75" - GH_AW_INFO_AGENT_VERSION: "1.0.75" - GH_AW_INFO_CLI_VERSION: "v0.83.4" + GH_AW_INFO_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'auto' }} + GH_AW_INFO_VERSION: "1.0.77" + GH_AW_INFO_AGENT_VERSION: "1.0.77" + GH_AW_INFO_CLI_VERSION: "v0.84.3" GH_AW_INFO_WORKFLOW_NAME: "Review SDK PR" GH_AW_INFO_EXPERIMENTAL: "false" GH_AW_INFO_SUPPORTS_TOOLS_ALLOWLIST: "true" GH_AW_INFO_STAGED: "false" GH_AW_INFO_ALLOWED_DOMAINS: '["github","threat-detection","azure.github.io","docs.microsoft.com","learn.microsoft.com"]' GH_AW_INFO_FIREWALL_ENABLED: "true" - GH_AW_INFO_AWF_VERSION: "v0.27.42" + GH_AW_INFO_AWF_VERSION: "v0.27.43" GH_AW_INFO_AWMG_VERSION: "" GH_AW_INFO_FIREWALL_TYPE: "squid" GH_AW_COMPILED_STRICT: "true" @@ -236,7 +239,7 @@ jobs: - name: Check compile-agentic version uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 env: - GH_AW_COMPILED_VERSION: "v0.83.4" + GH_AW_COMPILED_VERSION: "v0.84.3" with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); @@ -429,7 +432,10 @@ jobs: http_400_response_error: ${{ steps.detect-agent-errors.outputs.http_400_response_error || 'false' }} inference_access_error: ${{ steps.detect-agent-errors.outputs.inference_access_error || 'false' }} invocation_cap_exceeded: ${{ steps.detect-agent-errors.outputs.invocation_cap_exceeded || 'false' }} + max_cache_misses_exceeded: ${{ steps.detect-agent-errors.outputs.max_cache_misses_exceeded || 'false' }} mcp_policy_error: ${{ steps.detect-agent-errors.outputs.mcp_policy_error || 'false' }} + missing_model_pricing_error: ${{ steps.detect-agent-errors.outputs.missing_model_pricing_error || 'false' }} + missing_model_pricing_model_name: ${{ steps.detect-agent-errors.outputs.missing_model_pricing_model_name || '' }} model: ${{ needs.activation.outputs.model }} model_not_supported_error: ${{ steps.detect-agent-errors.outputs.model_not_supported_error || 'false' }} output: ${{ steps.collect_output.outputs.output }} @@ -441,7 +447,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 + uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -450,8 +456,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Review SDK PR" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/review-sdk.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.75" - GH_AW_INFO_AWF_VERSION: "v0.27.42" + GH_AW_INFO_VERSION: "1.0.77" + GH_AW_INFO_AWF_VERSION: "v0.27.43" GH_AW_INFO_ENGINE_ID: "copilot" - name: Set runtime paths id: set-runtime-paths @@ -497,11 +503,12 @@ jobs: const { main } = require('${{ runner.temp }}/gh-aw/actions/checkout_pr_branch.cjs'); await main(); - name: Install GitHub Copilot CLI - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" 1.0.75 + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" env: GH_HOST: github.com + GH_AW_COMPILED_VERSION: v0.84.3 - name: Install AWF binary - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.42 --rootless + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.43 --rootless - name: Determine automatic lockdown mode for GitHub MCP Server id: determine-automatic-lockdown uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 @@ -537,7 +544,7 @@ jobs: GH_AW_SKILL_DIR: ".github/skills" run: bash "${RUNNER_TEMP}/gh-aw/actions/restore_inline_skills.sh" - name: Download container images - run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607 ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0 ghcr.io/github/gh-aw-mcpg:v0.4.6@sha256:fecabec51bbc41f2ad61076d6bcd9a36ef23b142e672a444e054d37fc29de93c ghcr.io/github/gh-aw-node@sha256:a8082161d7dceda14b68f32eb39d0eaa96b825d07f5895b096afab9d9e0c7748 ghcr.io/github/github-mcp-server:v1.7.0@sha256:c491ffdf6f4c85cb5397021bc655edb8ab825c6f5f568e7597d77a1bd7c4d308 + run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6 ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1 ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d ghcr.io/github/gh-aw-mcpg:v0.4.7@sha256:7545220a9aca134b71e51193ee0eaf4c50756ebf8fbd25a63ae7556e62815c00 ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196 ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520 - name: Generate Safe Outputs Config run: | mkdir -p "${RUNNER_TEMP}/gh-aw/safeoutputs" @@ -668,6 +675,7 @@ jobs: GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} GH_AW_SAFE_OUTPUTS_CONFIG_PATH: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS_CONFIG_PATH }} GH_AW_SAFE_OUTPUTS_TOOLS_PATH: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS_TOOLS_PATH }} + GH_AW_SINK_VISIBILITY: ${{ steps.determine-automatic-lockdown.outputs.visibility }} GITHUB_MCP_SERVER_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | @@ -690,16 +698,16 @@ jobs: MCP_GATEWAY_UID=$(id -u 2>/dev/null || echo '0') MCP_GATEWAY_GID=$(id -g 2>/dev/null || echo '0') source "${RUNNER_TEMP}/gh-aw/actions/resolve_docker_socket_gid.sh" - export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network bridge -p 127.0.0.1:'"${MCP_GATEWAY_PORT}"':'"${MCP_GATEWAY_PORT}"' --name awmg-mcpg --add-host host.docker.internal:host-gateway --user '"${MCP_GATEWAY_UID}"':'"${MCP_GATEWAY_GID}"' --group-add '"${DOCKER_SOCK_GID}"' -v '"${DOCKER_SOCK_PATH}"':/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e MCP_GATEWAY_PAYLOAD_DIR -e MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD -e DOCKER_HOST=unix:///var/run/docker.sock -e DEBUG -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_POLICY_ALLOW_CREATE_PULL_REQUEST -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_GUARD_MIN_INTEGRITY -e GITHUB_MCP_GUARD_REPOS -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e GITHUB_JOB -e GITHUB_ACTION -e GITHUB_EVENT_NAME -e GITHUB_EVENT_PATH -e GITHUB_ACTOR -e GITHUB_ACTOR_ID -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_WORKFLOW_REF -e GITHUB_WORKFLOW_SHA -e GITHUB_REF -e GITHUB_REF_NAME -e GITHUB_REF_TYPE -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e RUNNER_TEMP -v /tmp/gh-aw/mcp-payloads:/tmp/gh-aw/mcp-payloads:rw -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw -v '"${RUNNER_TEMP}"'/gh-aw/safeoutputs:'"${RUNNER_TEMP}"'/gh-aw/safeoutputs:rw ghcr.io/github/gh-aw-mcpg:v0.4.6' + export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network bridge -p 127.0.0.1:'"${MCP_GATEWAY_PORT}"':'"${MCP_GATEWAY_PORT}"' --name awmg-mcpg --add-host host.docker.internal:host-gateway --user '"${MCP_GATEWAY_UID}"':'"${MCP_GATEWAY_GID}"' --group-add '"${DOCKER_SOCK_GID}"' -v '"${DOCKER_SOCK_PATH}"':/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e MCP_GATEWAY_PAYLOAD_DIR -e MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD -e DOCKER_HOST=unix:///var/run/docker.sock -e DEBUG -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_POLICY_ALLOW_CREATE_PULL_REQUEST -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_GUARD_MIN_INTEGRITY -e GITHUB_MCP_GUARD_REPOS -e GH_AW_SINK_VISIBILITY -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e GITHUB_JOB -e GITHUB_ACTION -e GITHUB_EVENT_NAME -e GITHUB_EVENT_PATH -e GITHUB_ACTOR -e GITHUB_ACTOR_ID -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_WORKFLOW_REF -e GITHUB_WORKFLOW_SHA -e GITHUB_REF -e GITHUB_REF_NAME -e GITHUB_REF_TYPE -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e RUNNER_TEMP -v /tmp/gh-aw/mcp-payloads:/tmp/gh-aw/mcp-payloads:rw -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw -v '"${RUNNER_TEMP}"'/gh-aw/safeoutputs:'"${RUNNER_TEMP}"'/gh-aw/safeoutputs:rw ghcr.io/github/gh-aw-mcpg:v0.4.7' mkdir -p "$HOME/.copilot" GH_AW_NODE=$(which node 2>/dev/null || command -v node 2>/dev/null || echo node) - cat << GH_AW_MCP_CONFIG_cd0fedea8e038870_EOF | "$GH_AW_NODE" "${RUNNER_TEMP}/gh-aw/actions/start_mcp_gateway.cjs" + cat << GH_AW_MCP_CONFIG_2455694971a4678a_EOF | "$GH_AW_NODE" "${RUNNER_TEMP}/gh-aw/actions/start_mcp_gateway.cjs" { "mcpServers": { "github": { "type": "stdio", - "container": "ghcr.io/github/github-mcp-server:v1.7.0", + "container": "ghcr.io/github/github-mcp-server:v1.8.0", "env": { "GITHUB_FEATURES": "fields_param", "GITHUB_HOST": "${GITHUB_SERVER_URL}", @@ -750,7 +758,7 @@ jobs: "private:azure/azure-sdk-for-rust", "private:azure/azure-rest-api-specs" ], - "sink-visibility": ${{ toJSON(steps.determine-automatic-lockdown.outputs.visibility) }} + "sink-visibility": "${GH_AW_SINK_VISIBILITY}" } } } @@ -763,7 +771,7 @@ jobs: "startupTimeout": 120 } } - GH_AW_MCP_CONFIG_cd0fedea8e038870_EOF + GH_AW_MCP_CONFIG_2455694971a4678a_EOF - name: Mount MCP servers as CLIs id: mount-mcp-clis continue-on-error: true @@ -807,7 +815,7 @@ jobs: export COPILOT_API_KEY="$COPILOT_DUMMY_BYOK" (umask 177 && touch /tmp/gh-aw/agent-stdio.log) GH_AW_MAX_AI_CREDITS="${GH_AW_MAX_AI_CREDITS:-1000}" - printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.42/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"*.githubusercontent.com\",\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"azure.github.io\",\"codeload.github.com\",\"docs.github.com\",\"docs.microsoft.com\",\"github-cloud.githubusercontent.com\",\"github-cloud.s3.amazonaws.com\",\"github.blog\",\"github.com\",\"github.githubassets.com\",\"host.docker.internal\",\"learn.microsoft.com\",\"lfs.github.com\",\"objects.githubusercontent.com\",\"patch-diff.githubusercontent.com\",\"patchdiff.githubusercontent.com\",\"raw.githubusercontent.com\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"],\"isolation\":true,\"topologyAttach\":[\"awmg-mcpg\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"fable\",\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.42,squid=sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0,agent=sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b,agent-act=sha256:a14ad974484aa518aab83d40f3f141175dfd171d3745e01c092375b970f73a20,api-proxy=sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607,cli-proxy=sha256:da006bf96d2d246dd269d57b233c1798d2ad63d6cd64ca02f7bf71045028781f\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" + printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.43/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"*.githubusercontent.com\",\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"azure.github.io\",\"codeload.github.com\",\"docs.github.com\",\"docs.microsoft.com\",\"github-cloud.githubusercontent.com\",\"github-cloud.s3.amazonaws.com\",\"github.blog\",\"github.com\",\"github.githubassets.com\",\"host.docker.internal\",\"learn.microsoft.com\",\"lfs.github.com\",\"objects.githubusercontent.com\",\"patch-diff.githubusercontent.com\",\"patchdiff.githubusercontent.com\",\"raw.githubusercontent.com\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"],\"isolation\":true,\"topologyAttach\":[\"awmg-mcpg\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"auto\":[\"copilot/auto\",\"large\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"detection\":[\"small\"],\"evals\":[\"small\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"grok\":[\"copilot/*grok*\",\"openai/*grok*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.43,squid=sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d,agent=sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6,api-proxy=sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1,cli-proxy=sha256:65c45ea2967984d0024f3df61bc71335658a77ede96c8d9665da7a5f33a795ab\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" cp "${RUNNER_TEMP}/gh-aw/awf-config.json" /tmp/gh-aw/awf-config.json export GH_AW_MODELS_JSON_PATH="/tmp/gh-aw/models.json" GH_AW_DOCKER_HOST="" @@ -832,7 +840,7 @@ jobs: COPILOT_AGENT_RUNNER_TYPE: STANDALONE COPILOT_DUMMY_BYOK: dummy-byok-key-for-offline-mode COPILOT_GITHUB_TOKEN: ${{ github.token }} - COPILOT_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }} + COPILOT_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'auto' }} GH_AW_LLM_PROVIDER: github GH_AW_MAX_AI_CREDITS: ${{ vars.GH_AW_DEFAULT_MAX_AI_CREDITS || '1000' }} GH_AW_MAX_TURNS: ${{ vars.GH_AW_DEFAULT_MAX_TURNS || '' }} @@ -840,7 +848,7 @@ jobs: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} GH_AW_TIMEOUT_MINUTES: 12 - GH_AW_VERSION: v0.83.4 + GH_AW_VERSION: v0.84.3 GITHUB_API_URL: ${{ github.api_url }} GITHUB_AW: true GITHUB_COPILOT_INTEGRATION_ID: agentic-workflows @@ -1014,7 +1022,6 @@ jobs: needs.activation.outputs.daily_ai_credits_exceeded == 'true') runs-on: ubuntu-slim permissions: - contents: read issues: write pull-requests: write concurrency: @@ -1031,7 +1038,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 + uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1040,8 +1047,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Review SDK PR" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/review-sdk.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.75" - GH_AW_INFO_AWF_VERSION: "v0.27.42" + GH_AW_INFO_VERSION: "1.0.77" + GH_AW_INFO_AWF_VERSION: "v0.27.43" GH_AW_INFO_ENGINE_ID: "copilot" - name: Download agent output artifact id: download-agent-output @@ -1068,30 +1075,7 @@ jobs: - name: Collect usage artifact files if: always() continue-on-error: true - run: | - mkdir -p /tmp/gh-aw/usage/agent /tmp/gh-aw/usage/detection - echo "Usage artifact source file status:" - for file in /tmp/gh-aw/aw_info.json /tmp/gh-aw/aw-info.jsonl /tmp/gh-aw/agent_usage.json /tmp/gh-aw/agent_usage.jsonl /tmp/gh-aw/detection_usage.jsonl /tmp/gh-aw/evals/evals.jsonl /tmp/gh-aw/github_rate_limits.jsonl /tmp/gh-aw/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/threat-detection/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/threat-detection/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl; do - [ -f "$file" ] && echo "FOUND: $file" || echo "MISSING: $file" - done - [ -f /tmp/gh-aw/aw_info.json ] && cp /tmp/gh-aw/aw_info.json /tmp/gh-aw/usage/aw_info.json || true - [ -f /tmp/gh-aw/aw-info.jsonl ] && cp /tmp/gh-aw/aw-info.jsonl /tmp/gh-aw/usage/aw-info.jsonl || true - [ -f /tmp/gh-aw/agent_usage.json ] && cp /tmp/gh-aw/agent_usage.json /tmp/gh-aw/usage/agent_usage.json || true - [ -f /tmp/gh-aw/agent_usage.jsonl ] && cp /tmp/gh-aw/agent_usage.jsonl /tmp/gh-aw/usage/agent_usage.jsonl || true - [ -f /tmp/gh-aw/detection_usage.jsonl ] && cp /tmp/gh-aw/detection_usage.jsonl /tmp/gh-aw/usage/detection_usage.jsonl || true - [ -f /tmp/gh-aw/evals/evals.jsonl ] && cp /tmp/gh-aw/evals/evals.jsonl /tmp/gh-aw/usage/evals.jsonl || true - [ -f /tmp/gh-aw/github_rate_limits.jsonl ] && cp /tmp/gh-aw/github_rate_limits.jsonl /tmp/gh-aw/usage/github_rate_limits.jsonl || true - [ -s /tmp/gh-aw/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/agent/token_usage.jsonl || true - [ -s /tmp/gh-aw/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/agent/token_usage.jsonl || true - [ -s /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/agent/token_usage.jsonl || true - [ -s /tmp/gh-aw/threat-detection/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/threat-detection/sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/detection/token_usage.jsonl || true - [ -s /tmp/gh-aw/threat-detection/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/threat-detection/sandbox/firewall/audit/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/detection/token_usage.jsonl || true - [ -s /tmp/gh-aw/threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl ] && cp /tmp/gh-aw/threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl /tmp/gh-aw/usage/detection/token_usage.jsonl || true - [ -f /tmp/gh-aw/usage/agent/token_usage.jsonl ] || : > /tmp/gh-aw/usage/agent/token_usage.jsonl - [ -f /tmp/gh-aw/usage/detection/token_usage.jsonl ] || : > /tmp/gh-aw/usage/detection/token_usage.jsonl - mkdir -p /tmp/gh-aw/usage/activity - node "${RUNNER_TEMP}/gh-aw/actions/generate_usage_activity_summary.cjs" - find /tmp/gh-aw/usage -type f -print | sort + run: bash "${RUNNER_TEMP}/gh-aw/actions/collect_usage_artifact_files.sh" - name: Upload usage artifact if: always() continue-on-error: true @@ -1243,6 +1227,9 @@ jobs: GH_AW_AGENTIC_ENGINE_TIMEOUT: ${{ needs.agent.outputs.agentic_engine_timeout }} GH_AW_MODEL_NOT_SUPPORTED_ERROR: ${{ needs.agent.outputs.model_not_supported_error }} GH_AW_HTTP_400_RESPONSE_ERROR: ${{ needs.agent.outputs.http_400_response_error }} + GH_AW_MAX_CACHE_MISSES_EXCEEDED: ${{ needs.agent.outputs.max_cache_misses_exceeded }} + GH_AW_MISSING_MODEL_PRICING_ERROR: ${{ needs.agent.outputs.missing_model_pricing_error }} + GH_AW_MISSING_MODEL_PRICING_MODEL_NAME: ${{ needs.agent.outputs.missing_model_pricing_model_name }} GH_AW_ENGINE_API_HOSTS: "api.enterprise.githubcopilot.com,api.githubcopilot.com,api.business.githubcopilot.com,api.individual.githubcopilot.com" GH_AW_LOCKDOWN_CHECK_FAILED: ${{ needs.activation.outputs.lockdown_check_failed }} GH_AW_OAUTH_TOKEN_CHECK_FAILED: ${{ needs.activation.outputs.oauth_token_check_failed }} @@ -1282,7 +1269,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 + uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1291,8 +1278,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Review SDK PR" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/review-sdk.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.75" - GH_AW_INFO_AWF_VERSION: "v0.27.42" + GH_AW_INFO_VERSION: "1.0.77" + GH_AW_INFO_AWF_VERSION: "v0.27.43" GH_AW_INFO_ENGINE_ID: "copilot" - name: Download agent output artifact id: download-agent-output @@ -1319,7 +1306,7 @@ jobs: rm -rf /tmp/gh-aw/sandbox/firewall/logs rm -rf /tmp/gh-aw/sandbox/firewall/audit - name: Download container images - run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.42@sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b ghcr.io/github/gh-aw-firewall/api-proxy:0.27.42@sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607 ghcr.io/github/gh-aw-firewall/squid:0.27.42@sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0 + run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.27.43@sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6 ghcr.io/github/gh-aw-firewall/api-proxy:0.27.43@sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1 ghcr.io/github/gh-aw-firewall/squid:0.27.43@sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d - name: Check if detection needed id: detection_guard if: always() @@ -1351,10 +1338,14 @@ jobs: fi cp /tmp/gh-aw/agent_output.json /tmp/gh-aw/threat-detection/agent_output.json 2>/dev/null || true for f in /tmp/gh-aw/aw-*.patch; do - [ -f "$f" ] && cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true + if [ -f "$f" ]; then + cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true + fi done for f in /tmp/gh-aw/aw-*.bundle; do - [ -f "$f" ] && cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true + if [ -f "$f" ]; then + cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true + fi done echo "Prepared threat detection files:" ls -la /tmp/gh-aw/threat-detection/ 2>/dev/null || true @@ -1365,6 +1356,7 @@ jobs: WORKFLOW_NAME: "Review SDK PR" WORKFLOW_DESCRIPTION: "Agentic PR reviewer for Azure SDK for Rust changes.\nReviews API surface, crate/package conventions, workspace/Cargo wiring,\nrequired SDK metadata files, and security risks. Posts one PR comment." HAS_PATCH: ${{ needs.agent.outputs.has_patch }} + GH_AW_DETECTION_CONTINUE_ON_ERROR: "true" with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); @@ -1382,11 +1374,12 @@ jobs: node-version: '24' package-manager-cache: false - name: Install GitHub Copilot CLI - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" 1.0.75 + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" env: GH_HOST: github.com + GH_AW_COMPILED_VERSION: v0.84.3 - name: Install AWF binary - run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.42 + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.43 - name: Execute GitHub Copilot CLI if: always() && steps.detection_guard.outputs.run_detection == 'true' continue-on-error: true @@ -1406,7 +1399,7 @@ jobs: export COPILOT_API_KEY="$COPILOT_DUMMY_BYOK" (umask 177 && touch /tmp/gh-aw/threat-detection/detection.log) GH_AW_MAX_AI_CREDITS="${GH_AW_MAX_AI_CREDITS:-400}" - printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.42/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"github.com\",\"host.docker.internal\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"fable\",\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.42,squid=sha256:42dfeb649c680a8558cd5423dbc530b653a69413e35ffbe5e71da5d48c94bdf0,agent=sha256:26a8af4e5566485b02f52af59ee03803ae798271a9619d4767e94d07806deb9b,agent-act=sha256:a14ad974484aa518aab83d40f3f141175dfd171d3745e01c092375b970f73a20,api-proxy=sha256:944f2686c9ab9bec338fd14b662461662f77cd12cd0ea8a3e7cb8c0987cd1607,cli-proxy=sha256:da006bf96d2d246dd269d57b233c1798d2ad63d6cd64ca02f7bf71045028781f\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" + printf '%s\n' "{\"\$schema\":\"https://github.com/github/gh-aw-firewall/releases/download/v0.27.43/awf-config.schema.json\",\"network\":{\"allowDomains\":[\"api.business.githubcopilot.com\",\"api.enterprise.githubcopilot.com\",\"api.github.com\",\"api.githubcopilot.com\",\"api.individual.githubcopilot.com\",\"github.com\",\"host.docker.internal\",\"registry.npmjs.org\",\"telemetry.enterprise.githubcopilot.com\"]},\"apiProxy\":{\"enabled\":true,\"enableTokenSteering\":true,\"maxRuns\":500,\"maxAiCredits\":${GH_AW_MAX_AI_CREDITS},\"maxCacheMisses\":5,\"models\":{\"agent\":[\"sonnet-6x\",\"gpt-5.4\",\"gpt-5.5\",\"gpt-5.6\",\"gpt-5.3\",\"gemini-pro\",\"any\"],\"antigravity\":[\"copilot/antigravity*\",\"google/antigravity*\",\"gemini/antigravity*\"],\"any\":[\"copilot/*\",\"anthropic/*\",\"openai/*\",\"google/*\",\"gemini/*\"],\"auto\":[\"copilot/auto\",\"large\"],\"claude\":[\"agent\"],\"codex\":[\"agent\"],\"coding\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\",\"gpt-5-codex\",\"kimi\"],\"computer-use\":[\"copilot/*computer-use*\",\"google/*computer-use*\",\"gemini/*computer-use*\",\"openai/*computer-use*\"],\"copilot\":[\"agent\"],\"deep-research\":[\"copilot/deep-research*\",\"copilot/o3-deep-research*\",\"copilot/o4-mini-deep-research*\",\"google/deep-research*\",\"gemini/deep-research*\",\"openai/o3-deep-research*\",\"openai/o4-mini-deep-research*\"],\"detection\":[\"small\"],\"evals\":[\"small\"],\"fable\":[\"copilot/*fable*\",\"anthropic/*fable*\"],\"gemini\":[\"agent\"],\"gemini-3-flash\":[\"copilot/gemini-3*flash*\",\"google/gemini-3*flash*\",\"gemini/gemini-3*flash*\"],\"gemini-3-pro\":[\"copilot/gemini-3*pro*\",\"google/gemini-3*pro*\",\"google/nano-banana*\",\"gemini/gemini-3*pro*\"],\"gemini-3.1-flash\":[\"copilot/gemini-3.1*flash*\",\"google/gemini-3.1*flash*\",\"gemini/gemini-3.1*flash*\"],\"gemini-3.1-pro\":[\"copilot/gemini-3.1*pro*\",\"google/gemini-3.1*pro*\",\"gemini/gemini-3.1*pro*\"],\"gemini-3.5-flash\":[\"copilot/gemini-3.5*flash*\",\"google/gemini-3.5*flash*\",\"gemini/gemini-3.5*flash*\"],\"gemini-3.6-flash\":[\"copilot/gemini-3.6*flash*\",\"google/gemini-3.6*flash*\",\"gemini/gemini-3.6*flash*\"],\"gemini-flash\":[\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"],\"gemini-flash-lite\":[\"copilot/gemini-*flash*lite*\",\"google/gemini-*flash*lite*\",\"gemini/gemini-*flash*lite*\"],\"gemini-omni\":[\"copilot/gemini-omni*\",\"google/gemini-omni*\",\"gemini/gemini-omni*\"],\"gemini-pro\":[\"copilot/gemini-*pro*\",\"google/gemini-*pro*\",\"gemini/gemini-*pro*\"],\"gemma\":[\"copilot/gemma*\",\"google/gemma*\",\"gemini/gemma*\"],\"gpt-5\":[\"copilot/gpt-5*\",\"openai/gpt-5*\"],\"gpt-5-codex\":[\"copilot/gpt-5*codex*\",\"openai/gpt-5*codex*\"],\"gpt-5-mini\":[\"copilot/gpt-5*mini*\",\"openai/gpt-5*mini*\"],\"gpt-5-nano\":[\"copilot/gpt-5*nano*\",\"openai/gpt-5*nano*\"],\"gpt-5-pro\":[\"copilot/gpt-5*pro*\",\"openai/gpt-5*pro*\"],\"gpt-5.1\":[\"copilot/gpt-5.1*\",\"openai/gpt-5.1*\"],\"gpt-5.2\":[\"copilot/gpt-5.2*\",\"openai/gpt-5.2*\"],\"gpt-5.3\":[\"copilot/gpt-5.3*\",\"openai/gpt-5.3*\"],\"gpt-5.4\":[\"copilot/gpt-5.4*\",\"openai/gpt-5.4*\"],\"gpt-5.5\":[\"copilot/gpt-5.5*\",\"openai/gpt-5.5*\"],\"gpt-5.6\":[\"copilot/gpt-5.6*\",\"openai/gpt-5.6*\"],\"grok\":[\"copilot/*grok*\",\"openai/*grok*\"],\"haiku\":[\"copilot/*haiku*\",\"anthropic/*haiku*\"],\"image-generation\":[\"copilot/gpt-image*\",\"openai/gpt-image*\",\"openai/chatgpt-image*\",\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"google/imagen*\"],\"kimi\":[\"copilot/kimi*\",\"openai/kimi*\"],\"kiwi\":[\"copilot/kiwi*\",\"openai/kiwi*\"],\"large\":[\"sonnet\",\"gpt-5-pro\",\"gpt-5\",\"gemini-pro\"],\"lyria\":[\"google/lyria*\",\"gemini/lyria*\",\"copilot/lyria*\"],\"mai-code\":[\"copilot/MAI-Code*\",\"copilot/mai-code*\",\"openai/MAI-Code*\"],\"mai-code-1-flash-picker\":[\"copilot/MAI-Code-1-Flash-picker*\",\"copilot/mai-code-1-flash-picker*\",\"openai/MAI-Code-1-Flash-picker*\"],\"mini\":[\"haiku\",\"gpt-5-mini\",\"gpt-5-nano\",\"gemini-flash-lite\"],\"nano-banana\":[\"copilot/nano-banana*\",\"google/nano-banana*\",\"gemini/nano-banana*\"],\"opus\":[\"copilot/*opus*\",\"anthropic/*opus*\"],\"opusplan\":[\"opus?effort=high\"],\"raptor-mini\":[\"copilot/raptor*\",\"openai/raptor*\"],\"reasoning\":[\"copilot/o1*\",\"copilot/o3*\",\"copilot/o4*\",\"openai/o1*\",\"openai/o3*\",\"openai/o4*\"],\"robotics\":[\"copilot/*robotics*\",\"google/*robotics*\",\"gemini/*robotics*\"],\"small\":[\"mini\"],\"small-agent\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash\"],\"sonnet\":[\"copilot/*sonnet*\",\"anthropic/*sonnet*\"],\"sonnet-6x\":[\"copilot/*sonnet-4.5*\",\"copilot/*sonnet-4.6*\",\"copilot/*sonnet-5*\",\"copilot/*sonnet-4-5-*\",\"anthropic/*sonnet-4-5-*\",\"copilot/*sonnet-4-6*\",\"anthropic/*sonnet-4-6*\",\"anthropic/*sonnet-5*\"],\"summarization\":[\"haiku\",\"gpt-5-mini\",\"gemini-flash-lite\",\"mini\"],\"veo\":[\"google/veo*\",\"gemini/veo*\"],\"vision\":[\"copilot/gemini-*image*\",\"google/gemini-*image*\",\"gemini/gemini-*image*\",\"copilot/gemini-*flash*\",\"google/gemini-*flash*\",\"gemini/gemini-*flash*\"]}},\"container\":{\"imageTag\":\"0.27.43,squid=sha256:26be5e0b8c8f4c41c8a59126b29bb5d80b07253597472ded2a16bdd75abcbf9d,agent=sha256:04e2d1987a565000a8f114b89d806ae7a3864dd4f944be65275b28c93d8690e6,api-proxy=sha256:d85f57975af5ea23af4996e41ed73fbc8f5b4a47402472bfe82e508f352cb0c1,cli-proxy=sha256:65c45ea2967984d0024f3df61bc71335658a77ede96c8d9665da7a5f33a795ab\"},\"logging\":{\"proxyLogsDir\":\"/tmp/gh-aw/sandbox/firewall/logs\",\"auditDir\":\"/tmp/gh-aw/sandbox/firewall/audit\"}}" > "${RUNNER_TEMP}/gh-aw/awf-config.json" cp "${RUNNER_TEMP}/gh-aw/awf-config.json" /tmp/gh-aw/awf-config.json export GH_AW_MODELS_JSON_PATH="/tmp/gh-aw/models.json" GH_AW_DOCKER_HOST="" @@ -1433,14 +1426,14 @@ jobs: COPILOT_AGENT_RUNNER_TYPE: STANDALONE COPILOT_DUMMY_BYOK: dummy-byok-key-for-offline-mode COPILOT_GITHUB_TOKEN: ${{ github.token }} - COPILOT_MODEL: ${{ vars.GH_AW_MODEL_DETECTION_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }} + COPILOT_MODEL: detection GH_AW_LLM_PROVIDER: github GH_AW_MAX_AI_CREDITS: ${{ vars.GH_AW_DEFAULT_DETECTION_MAX_AI_CREDITS || '400' }} GH_AW_MAX_TURNS: ${{ vars.GH_AW_DEFAULT_MAX_TURNS || '' }} GH_AW_PHASE: detection GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_TIMEOUT_MINUTES: 20 - GH_AW_VERSION: v0.83.4 + GH_AW_VERSION: v0.84.3 GITHUB_API_URL: ${{ github.api_url }} GITHUB_AW: true GITHUB_COPILOT_INTEGRATION_ID: agentic-workflows @@ -1517,7 +1510,6 @@ jobs: if: (!cancelled()) && needs.agent.result != 'skipped' && needs.detection.result == 'success' runs-on: ubuntu-slim permissions: - contents: read issues: write pull-requests: write timeout-minutes: 45 @@ -1531,7 +1523,7 @@ jobs: GH_AW_EFFECTIVE_TOKENS: ${{ needs.agent.outputs.effective_tokens }} GH_AW_ENGINE_ID: "copilot" GH_AW_ENGINE_MODEL: ${{ needs.agent.outputs.model }} - GH_AW_ENGINE_VERSION: "1.0.75" + GH_AW_HEAD_SHA: ${{ github.event.pull_request.head.sha }} GH_AW_RUNTIME_FEATURES: ${{ vars.GH_AW_RUNTIME_FEATURES }} GH_AW_THREAT_DETECTION_AIC: ${{ needs.detection.outputs.aic }} GH_AW_WORKFLOW_ID: "review-sdk" @@ -1549,7 +1541,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@e89c65e17eb281bbd5ff2ff9e9199a03e96654c7 # v0.83.4 + uses: github/gh-aw-actions/setup@c863074b673419603d146aab585e2986ef08deec # v0.84.3 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1558,8 +1550,8 @@ jobs: env: GH_AW_SETUP_WORKFLOW_NAME: "Review SDK PR" GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/review-sdk.lock.yml@${{ github.ref }} - GH_AW_INFO_VERSION: "1.0.75" - GH_AW_INFO_AWF_VERSION: "v0.27.42" + GH_AW_INFO_VERSION: "1.0.77" + GH_AW_INFO_AWF_VERSION: "v0.27.43" GH_AW_INFO_ENGINE_ID: "copilot" - name: Download agent output artifact id: download-agent-output diff --git a/.github/workflows/verify-links.yml b/.github/workflows/verify-links.yml index 540b7cb1cdc..f41d380f17a 100644 --- a/.github/workflows/verify-links.yml +++ b/.github/workflows/verify-links.yml @@ -27,10 +27,10 @@ jobs: (github.repository == 'Azure/azure-sdk-for-python' && contains(github.event.check_run.name, 'Analyze')) || (github.repository == 'Azure/azure-sdk-for-java' && contains(github.event.check_run.name, 'Analyze')) || (github.repository == 'Azure/azure-sdk-for-js' && contains(github.event.check_run.name, 'Analyze')) || - (github.repository == 'Azure/azure-sdk-for-c' && contains(github.event.check_run.name, 'GenerateReleaseArtifacts')) || (github.repository == 'Azure/azure-sdk-for-cpp' && contains(github.event.check_run.name, 'GenerateReleaseArtifacts')) || (github.repository == 'Azure/azure-sdk-for-go' && contains(github.event.check_run.name, 'Analyze')) || - (github.repository == 'Azure/azure-sdk-for-ios' && contains(github.event.check_run.name, 'Analyze')) + (github.repository == 'Azure/azure-sdk-for-ios' && contains(github.event.check_run.name, 'Analyze')) || + (github.repository == 'Azure/azure-sdk-for-rust' && contains(github.event.check_run.name, 'Analyze')) ) ) runs-on: ubuntu-latest