Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sdk/cosmos/azure_data_cosmos/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
### Features Added

- Added a pluggable client-side diagnostics emission layer — the `DiagnosticsHandler` trait and ordered `DiagnosticsHandlerChain` (registered via `CosmosClientBuilder::with_diagnostics_handler`) — invoked once per operation (singleton and paginated, on success and failure) with the completed `DiagnosticsContext` plus an SDK-supplied `CosmosOperationContext`; the empty default chain is a zero-overhead no-op. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789))
- Added the `metrics`-gated `CosmosMetricsHandler` (with `MetricsOptions`), emitting the stable `db.client.operation.duration` histogram plus per-signal opt-in metrics (`with_request_charge_metric`, `with_returned_rows_metric`) and an opt-in extended attribute set (`with_extended_attributes`); a no-op when no meter provider is registered. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789))
- Added the `metrics`-gated `CosmosMetricsHandler` (with `MetricsOptions`), emitting the stable `db.client.operation.duration` histogram plus per-signal opt-in metrics (`with_request_charge_metric`, `with_returned_rows_metric`, and `with_active_instance_metric` — an `azure.cosmosdb.client.active_instance.count` up-down counter that is incremented on handler construction and decremented on drop to track live client instances) and an opt-in extended attribute set (`with_extended_attributes`); a no-op when no meter provider is registered. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789), [#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874))
- Added composable tail-sampled emission handlers — a `TracingLogHandler` leaf that writes a compact `tracing` line and a `SamplingLogHandler` wrapper (holding an `Arc<dyn DiagnosticsHandler>`) that applies the sampling gate plus a shared per-window rate limit, defaulting to wrap a `TracingLogHandler` — and the `distributed_tracing`-gated `CosmosTracingHandler` (backdated span tree), also rate-limited so an error storm can't overwhelm exporters. All emit only for operations which fail or breach a configurable `DiagnosticsThresholds`, and stamp *why* they were sampled (a failure, or which threshold) on the emitted line and span. ([#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789))

### Breaking Changes
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ pub const METRIC_OPERATION_REQUEST_CHARGE: &str = "azure.cosmosdb.client.operati
/// Optional histogram (rows): number of rows/items returned by an operation.
pub const METRIC_RESPONSE_RETURNED_ROWS: &str = "db.client.response.returned_rows";

/// Optional up-down counter (instances): number of live
/// [`CosmosMetricsHandler`](super::CosmosMetricsHandler) instances (one per
/// instrumented client, under the intended one-handler-per-client registration).
pub const METRIC_ACTIVE_INSTANCE_COUNT: &str = "azure.cosmosdb.client.active_instance.count";

// =========================================================================
// Instrument units
// =========================================================================
Expand All @@ -49,6 +54,9 @@ pub const UNIT_REQUEST_UNIT: &str = "{request_unit}";
/// Unit for [`METRIC_RESPONSE_RETURNED_ROWS`] — rows.
pub const UNIT_ROW: &str = "{row}";

/// Unit for [`METRIC_ACTIVE_INSTANCE_COUNT`] — client instances.
pub const UNIT_INSTANCE: &str = "{instance}";

// =========================================================================
// Stable attributes (always emitted; operation scope, low cardinality)
//
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@ const METER_NAME: &str = "azure_data_cosmos";
/// histogram. The optional per-signal metrics and the extended attribute set are
/// opt-in via [`MetricsOptions`] (see [`with_options`](CosmosMetricsHandler::with_options)).
///
/// When the active-instance metric is enabled
/// ([`MetricsOptions::with_active_instance_metric`]), the handler increments the
/// `azure.cosmosdb.client.active_instance.count` up-down counter on construction
/// and decrements it on [`Drop`], so the reported value tracks the number of
/// live handler instances (one per instrumented client, under the intended
/// one-handler-per-client registration).
///
/// The handler captures a [`Meter`] from the globally-registered provider at
/// construction. Install your meter provider **before** constructing the handler:
/// a `Meter` obtained while the global provider is still the default no-op stays a
Expand Down Expand Up @@ -76,10 +83,34 @@ impl CosmosMetricsHandler {
}

fn from_meter(meter: &Meter, options: MetricsOptions) -> Self {
Self {
let handler = Self {
instruments: Instruments::new(meter),
options,
};
// Record the +1 half of the active-instance up-down counter at
// construction. One handler is created per instrumented client and
// held for that client's lifetime, so the handler's own lifecycle is
// a faithful proxy for a live client instance. The matching -1 is
// recorded in `Drop`.
Comment thread
NaluTripician marked this conversation as resolved.
Outdated
if handler.options.active_instance_metric_enabled() {
handler
.instruments
.active_instance
.add(1, &Self::active_instance_attributes());
}
handler
}

/// Low-cardinality attribute set for the active-instance up-down counter.
///
/// Deliberately keyed on `db.system.name` alone so every client instance
/// aggregates into a single series whose value is the live instance count,
/// rather than fanning out per-instance.
fn active_instance_attributes() -> [KeyValue; 1] {
[KeyValue::new(
attributes::ATTR_DB_SYSTEM_NAME,
attributes::DB_SYSTEM_NAME_VALUE,
)]
Comment thread
NaluTripician marked this conversation as resolved.
Outdated
}

/// Resolves `server.address`: the operation-context override if present,
Expand Down Expand Up @@ -215,6 +246,18 @@ impl Default for CosmosMetricsHandler {
}
}

impl Drop for CosmosMetricsHandler {
fn drop(&mut self) {
// Record the -1 half of the active-instance up-down counter: the client
// instrumentation instance this handler represents is going away.
if self.options.active_instance_metric_enabled() {
self.instruments
.active_instance
.add(-1, &Self::active_instance_attributes());
}
}
}

impl std::fmt::Debug for CosmosMetricsHandler {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
// Instruments and the seen-set are not meaningfully printable; surface
Expand Down Expand Up @@ -353,6 +396,29 @@ mod tests {
None
}

/// Returns the summed value of the `active_instance.count` up-down counter
/// from the most recent export, or `None` if the metric was never emitted.
///
/// The in-memory exporter accumulates one snapshot per `collect()` call, so
/// we scan every snapshot and keep the value from the last one — the current
/// cumulative count.
fn active_instance_value(metrics: &[ResourceMetrics]) -> Option<i64> {
let mut latest = None;
for rm in metrics {
for sm in rm.scope_metrics() {
for m in sm.metrics() {
if m.name() != attributes::METRIC_ACTIVE_INSTANCE_COUNT {
continue;
}
if let AggregatedMetrics::I64(MetricData::Sum(sum)) = m.data() {
latest = Some(sum.data_points().map(|point| point.value()).sum());
}
}
}
}
latest
}

#[test]
fn stable_duration_metric_carries_expected_attributes() {
let harness = test_meter();
Expand Down Expand Up @@ -530,6 +596,32 @@ mod tests {
handler.handle(&completed(200), &Context::new());
}

#[test]
fn active_instance_metric_off_by_default() {
// With default options the active-instance counter is never touched, so
// no such series is exported even across the handler's whole lifecycle.
let harness = test_meter();
let handler = CosmosMetricsHandler::with_meter(harness.meter.clone());
assert_eq!(active_instance_value(&harness.collect()), None);
drop(handler);
assert_eq!(active_instance_value(&harness.collect()), None);
}

#[test]
fn active_instance_metric_tracks_handler_lifecycle() {
let harness = test_meter();
let options = MetricsOptions::default().with_active_instance_metric(true);

let handler = CosmosMetricsHandler::with_meter_and_options(harness.meter.clone(), options);
// +1 recorded at construction.
assert_eq!(active_instance_value(&harness.collect()), Some(1));

// Dropping the handler records the matching -1, so the up-down counter
// returns to zero — it reflects live instances, not a monotonic total.
drop(handler);
assert_eq!(active_instance_value(&harness.collect()), Some(0));
}

#[test]
fn host_of_extracts_host() {
assert_eq!(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
//! reference-counted). Building them eagerly keeps the per-operation hot path to
//! just `record`/`add` calls with no allocation of instrument state.

use opentelemetry::metrics::{Histogram, Meter};
use opentelemetry::metrics::{Histogram, Meter, UpDownCounter};

use crate::diagnostics::metrics::attributes;

Expand All @@ -17,7 +17,8 @@ use crate::diagnostics::metrics::attributes;
/// The stable operation-duration histogram is always recorded; the remaining
/// per-signal instruments are recorded only when the matching
/// [`MetricsOptions`](super::MetricsOptions) toggle
/// (`request_charge_metric_enabled` / `returned_rows_metric_enabled`) is set.
/// (`request_charge_metric_enabled` / `returned_rows_metric_enabled` /
/// `active_instance_metric_enabled`) is set.
/// They are still created unconditionally because instrument creation is cheap
/// and idempotent, and doing so keeps the handler's record path branch-free per
/// instrument.
Expand All @@ -31,6 +32,15 @@ pub(crate) struct Instruments {

/// Development: `db.client.response.returned_rows` (rows).
pub(crate) returned_rows: Histogram<u64>,

/// Development: `azure.cosmosdb.client.active_instance.count` (instances).
///
/// An up-down counter incremented when the handler is constructed and
/// decremented when it is dropped, so the reported value tracks the number
/// of live [`CosmosMetricsHandler`](super::CosmosMetricsHandler) instances
/// (one per instrumented client, under the intended one-handler-per-client
/// registration).
pub(crate) active_instance: UpDownCounter<i64>,
}

impl Instruments {
Expand All @@ -54,10 +64,17 @@ impl Instruments {
.with_description("Number of rows/items returned by a Cosmos DB operation.")
.build();

let active_instance = meter
.i64_up_down_counter(attributes::METRIC_ACTIVE_INSTANCE_COUNT)
.with_unit(attributes::UNIT_INSTANCE)
.with_description("Number of active Cosmos DB client instances.")
.build();

Self {
operation_duration,
request_charge,
returned_rows,
active_instance,
}
}
}
21 changes: 21 additions & 0 deletions sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/options.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,15 +33,18 @@
/// let full = MetricsOptions::default()
/// .with_request_charge_metric(true)
/// .with_returned_rows_metric(true)
/// .with_active_instance_metric(true)
/// .with_extended_attributes(true);
/// assert!(full.request_charge_metric_enabled());
/// assert!(full.returned_rows_metric_enabled());
/// assert!(full.active_instance_metric_enabled());
/// assert!(full.extended_attributes_enabled());
/// ```
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct MetricsOptions {
request_charge_metric: bool,
returned_rows_metric: bool,
active_instance_metric: bool,
extended_attributes: bool,
}

Expand All @@ -68,6 +71,19 @@ impl MetricsOptions {
self
}

/// Enables (or disables) the
/// `azure.cosmosdb.client.active_instance.count` up-down counter, which
/// tracks the number of live [`CosmosMetricsHandler`](super::CosmosMetricsHandler)
/// instances: it is incremented by one when the handler is created and
/// decremented by one when it is dropped. With the intended one-handler-per-client
/// registration this equals the number of live instrumented clients; sharing
/// a single handler across several clients reports one. Off by default.
#[must_use]
pub fn with_active_instance_metric(mut self, enabled: bool) -> Self {
self.active_instance_metric = enabled;
self
}

/// Enables (or disables) the extended attribute set on every emitted metric:
/// consistency level, contacted regions, sub-status code, and connection
/// mode. These can be higher cardinality, so they are opt-in and off by
Expand All @@ -88,6 +104,11 @@ impl MetricsOptions {
self.returned_rows_metric
}

/// Whether the active-instance up-down counter is emitted.
pub fn active_instance_metric_enabled(&self) -> bool {
self.active_instance_metric
}

/// Whether the extended attribute set is attached to emitted metrics.
pub fn extended_attributes_enabled(&self) -> bool {
self.extended_attributes
Expand Down
44 changes: 41 additions & 3 deletions sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -208,9 +208,9 @@ mod tests {

let now_instant = Instant::now();
let now_system = SystemTime::now();
// A slow operation with NO driver-side operation name — the production
// case, since the driver never records one. The SDK-supplied
// CosmosOperationContext carries the operation identity instead.
// A slow operation with NO driver-side operation name — exercising the
// op-context fallback path. The SDK-supplied CosmosOperationContext
// carries the operation identity instead.
let ctx = context(
Duration::from_millis(1500),
Some(CosmosStatus::new(StatusCode::Ok)),
Expand Down Expand Up @@ -244,6 +244,44 @@ mod tests {
}));
}

#[test]
fn op_context_operation_name_wins_over_driver_context() {
// When both the driver context and the SDK operation context carry a
// name, the span must use the caller-facing op-context identity — so a
// PATCH whose surfaced sub-op is a Replace is still labeled `patch_item`,
// consistent with the `db.operation.name` metric and the tail-sampling
// classifier (which both read the op context).
let (provider, exporter) = exportable();
let tracer = provider.tracer("test");

let now_instant = Instant::now();
let now_system = SystemTime::now();
let ctx = context(
Duration::from_millis(1500),
Some(CosmosStatus::new(StatusCode::Ok)),
Some("replace_item"), // driver-side sub-op name
&[(1500, 1500, CosmosStatus::new(StatusCode::Ok))],
now_instant,
);
let op = CosmosOperationContext::new().with_operation_name("patch_item");

emit_backdated_span_tree(&tracer, &ctx, Some(&op), None, now_instant, now_system);
provider.force_flush().unwrap();

let spans = exporter.get_finished_spans().unwrap();
let root = spans
.iter()
.find(|s| s.name == "patch_item")
.expect("root span named from op context, not the driver sub-op");
assert!(root.attributes.iter().any(|kv| {
kv.key.as_str() == attributes::DB_OPERATION_NAME && kv.value.as_str() == "patch_item"
}));
assert!(
!spans.iter().any(|s| s.name == "replace_item"),
"the driver sub-op name must not label the operation span"
);
}

#[test]
fn incomplete_context_is_not_sampled_even_when_slow() {
// A finalized context with neither a status nor any attempts does not
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -101,11 +101,18 @@ pub(crate) fn emit_backdated_span_tree<T>(
let op_failed = diagnostics.is_failure();

// --- Operation (root) span ---
// Prefer the driver context's operation name; fall back to the SDK-supplied
// operation identity when the driver did not record one.
let op_name_ref = diagnostics
.operation_name()
.or_else(|| op.and_then(CosmosOperationContext::operation_name));
// Prefer the SDK-supplied operation identity (the caller-facing operation,
// e.g. `patch_item`) so the span label agrees with the metric
// `db.operation.name` and the tail-sampling classifier, which both read the
// same `CosmosOperationContext`. Fall back to the driver context's operation
// name for operations not surfaced through the SDK wrapper (which therefore
// carry no `CosmosOperationContext`). Preferring the driver value here would
// mislabel an aggregate whose surfaced sub-op differs from the operation —
// e.g. a PATCH that fails during its internal Read would report `read_item`
// on the span while the metric reports `patch_item`.
let op_name_ref = op
.and_then(CosmosOperationContext::operation_name)
.or_else(|| diagnostics.operation_name());
let op_name = op_name_ref
.unwrap_or(DEFAULT_OPERATION_SPAN_NAME)
.to_string();
Expand Down
2 changes: 2 additions & 0 deletions sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

### Features Added

- Added `CosmosOperation::db_operation_name`, returning the canonical OpenTelemetry `db.operation.name` (e.g. `read_item`, `query_items`, `execute_batch`) for an operation. The operation pipeline now populates `DiagnosticsContext::operation_name` from it in production (previously always `None`), so `DiagnosticsContext::operation_name()` is populated for callers inspecting diagnostics and supplies the operation name to the emission layer's tail-sampling classifier and tracing span when no SDK-supplied `CosmosOperationContext` is present. The tracing span's operation label prefers the caller-facing `CosmosOperationContext` identity (consistent with the `db.operation.name` metric), and PATCH aggregates report `patch_item` rather than the underlying Replace. ([#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874))
Comment thread
NaluTripician marked this conversation as resolved.
Outdated

### Breaking Changes

### Bugs Fixed
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -351,7 +351,7 @@ attributes. This powers client-side Grafana dashboards (R7) with per-combination
| `db.client.operation.duration` | stable | histogram | `s` | End-to-end operation duration — **the primary metric**. |
| `db.client.response.returned_rows` | development | histogram | `{row}` | Rows returned in the result set. |
| `azure.cosmosdb.client.operation.request_charge` | development | histogram | `{request_unit}` | Request units (RU) consumed by the operation. |
| `azure.cosmosdb.client.active_instance.count` | development | up-down counter | `{instance}` | Number of active Cosmos client instances. *(Deferred — not emitted yet; pending real client-lifecycle wiring, see [#4789](https://github.com/Azure/azure-sdk-for-rust/pull/4789).)* |
| `azure.cosmosdb.client.active_instance.count` | development | up-down counter | `{instance}` | Number of active Cosmos client instances. *(Opt-in via `MetricsOptions::with_active_instance_metric`; `CosmosMetricsHandler` records +1 on construction and −1 on `Drop`, so the value tracks live `CosmosMetricsHandler` instances — one per instrumented client under the intended one-handler-per-client registration — [#4874](https://github.com/Azure/azure-sdk-for-rust/pull/4874).)* |

**Always-on metric attributes (low cardinality, D7):** `db.operation.name`,
`db.response.status_code`, `db.collection.name`, `db.namespace`, `error.type`, `server.address`,
Expand Down
Loading
Loading