feat(llm): add shared model telemetry and instrumented decorator

[Pouyanpi]

Description

Adds shared, engine-neutral OpenTelemetry GenAI instrumentation for model calls.

Commit 1 (the deliverable): feat(llm): add shared model telemetry and instrumented decorator2b680f0

• nemoguardrails/llm/telemetry.py: engine-neutral helpers for GenAI CLIENT spans and request/response/usage attributes.
• InstrumentedLLMModel: a transparent decorator wrapping any LLMModel to emit GenAI spans and client metrics. No-op when tracing and metrics are both off.
• No engine is wired to it. Self-contained, unit-tested, changes no existing behavior.

Commit 2 (temporary): refactor(guardrails): route IORails model calls through the shared decorator

• Switches IORails to the shared decorator to show its tracing/metrics are preserved.
• Not for merge here; I will drop it before merging. Review as a preservation proof.

Related Issue(s)

• Fixes #<issue_number>
• Issue assignee: @Pouyanpi

Verification

• Live IORails (OpenAI gpt-4o-mini): unchanged spans plus both guardrails.* and gen_ai.client.* metrics. Examples were run against every IORails telemetry doc:
  • docs/observability/metrics/: enable-metrics.mdx, index.mdx, opentelemetry-integration.mdx, reference.mdx
  • docs/observability/tracing/: content-capture.mdx, span-reference.mdx, index.mdx

AI Assistance

• No AI tools were used.
• AI tools were used; a human reviewed and can explain every change (tool: Claude Code).

Checklist

• I've read the CONTRIBUTING guidelines.
• This PR links to a triaged issue assigned to me.
• My PR title follows the project commit convention.
• I've updated the documentation if applicable.
• I've added tests if applicable.
• I've noted any verification beyond CI and any checks I couldn't run.
• I did not update generated changelog files manually.
• I addressed all CodeRabbit, Greptile, and other review comments, or replied with why no change is needed.
• @mentions of the person or team responsible for reviewing proposed changes.

[codecov]

Codecov Report

❌ Patch coverage is 98.10606% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
nemoguardrails/llm/telemetry.py	97.14%	3 Missing ⚠️
nemoguardrails/guardrails/engine_registry.py	95.55%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[coderabbitai]

📝 Walkthrough

Walkthrough

The change centralizes LLM telemetry in a new reusable instrumented model wrapper, adds dedicated LLM telemetry helpers, integrates wrappers into EngineRegistry, re-exports the wrapper, and adds coverage for tracing, streaming, metrics, content capture, errors, and lifecycle behavior.

Changes

LLM instrumentation and telemetry

Layer / File(s)	Summary
LLM telemetry foundation nemoguardrails/llm/telemetry.py, nemoguardrails/guardrails/telemetry.py, nemoguardrails/tracing/constants.py	LLM span creation, request/response attributes, content capture, and error recording move into dedicated LLM telemetry helpers; Guardrails telemetry exposes api_call_span.
Instrumented model wrapper nemoguardrails/llm/models/instrumented.py, nemoguardrails/__init__.py	InstrumentedLLMModel instruments generation and streaming calls with spans, metrics, metadata, content capture, error handling, and cleanup, and is exported from the top-level package.
Engine registry integration nemoguardrails/guardrails/engine_registry.py	EngineRegistry adapts model engines and delegates generation and streaming calls to configured instrumented wrappers.
Instrumentation validation tests/llm/models/test_instrumented.py, tests/guardrails/test_telemetry_content_capture.py, tests/guardrails/test_telemetry_spans.py	Tests cover instrumentation, streaming lifecycle, errors, content capture, metrics, disabled behavior, idempotency, and updated telemetry imports.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant EngineRegistry
  participant InstrumentedLLMModel
  participant ModelEngine
  participant LLMTelemetry
  Caller->>EngineRegistry: model_call or stream_model_call
  EngineRegistry->>InstrumentedLLMModel: generate_async or stream_async
  InstrumentedLLMModel->>LLMTelemetry: start span and record request data
  InstrumentedLLMModel->>ModelEngine: execute model request
  ModelEngine-->>InstrumentedLLMModel: response or streamed chunks
  InstrumentedLLMModel->>LLMTelemetry: record response data, content, and metrics
  InstrumentedLLMModel-->>Caller: response or streamed chunks

Loading

🚥 Pre-merge checks | ✅ 5 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: shared LLM telemetry plus the new instrumented model decorator.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Test Results For Major Changes	✅ Passed	PR description includes a Verification section with unit-test results and live validation details for the major instrumentation changes.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch pouyanpi/shared-model-telemetry

---

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Greptile Summary

This PR extracts LLM-call OpenTelemetry instrumentation from guardrails/telemetry.py into a new, engine-neutral llm/telemetry.py module and pairs it with InstrumentedLLMModel — a transparent decorator that emits GenAI CLIENT spans and gen_ai.client.* metrics for any LLMModel implementation. Commit 2 (explicitly marked as temporary and not for merge) demonstrates backward-compatibility by routing IORails model calls through the shared decorator.

• nemoguardrails/llm/telemetry.py (new): engine-agnostic helpers for LLM span creation, request/response/content attributes. Adds suppress(Exception) guards throughout for telemetry-safe resilience.
• nemoguardrails/llm/models/instrumented.py (new): InstrumentedLLMModel decorator with idempotent wrapping, idempotency warnings, correct stream lifecycle (duration metric ends before cleanup, span attributes set only on natural exhaustion), and instrument_llm_model factory that short-circuits when telemetry is disabled.
• nemoguardrails/guardrails/telemetry.py (refactored): Removes the now-shared LLM-span helpers (moved to llm/telemetry.py) and imports record_span_error from there instead.

Confidence Score: 5/5

Commit 1 (the deliverable) is self-contained, adds no new dependencies on existing behavior, and is guarded by a thorough test suite. Safe to merge.

The new InstrumentedLLMModel correctly handles all stream lifecycle paths (natural exhaustion, consumer close, task cancellation), the finally/duration ordering is correct, idempotency warnings work as designed, and the telemetry helpers use suppress(Exception) consistently to avoid affecting callers. Tests cover every non-trivial path. Commit 2 is explicitly temporary and flagged as not for merge.

Files Needing Attention: No files require special attention. Commit 2 changes in engine_registry.py are explicitly temporary and should be dropped before merge as the author noted.

Important Files Changed

Filename	Overview
nemoguardrails/llm/models/instrumented.py	New InstrumentedLLMModel decorator with correct stream lifecycle, idempotency guard with warnings, and clean delegation to the wrapped model. No issues found.
nemoguardrails/llm/telemetry.py	New engine-neutral GenAI telemetry helpers migrated from guardrails/telemetry.py with added suppress(Exception) resilience guards.
nemoguardrails/guardrails/engine_registry.py	Commit 2 (temporary): replaces inline LLM telemetry in _run_model_call/_run_model_stream with InstrumentedLLMModel delegation via _ModelEngineAdapter. Not for merge.
nemoguardrails/guardrails/telemetry.py	Removes LLM-span helpers now living in llm/telemetry.py; imports record_span_error from there. Clean refactor.
tests/llm/models/test_instrumented.py	Comprehensive test suite covering delegation, error identity, stream lifecycle (natural exhaustion, consumer close, cancellation), content capture, metrics, and idempotency. Well-structured.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant ILM as InstrumentedLLMModel
    participant Span as llm_call_span
    participant Dur as llm_operation_duration
    participant Model as Wrapped LLMModel
    participant Metrics as OTEL Metrics

    Caller->>ILM: "generate_async(prompt, **kwargs)"
    ILM->>Span: open CLIENT span
    ILM->>Span: set_llm_request_attributes
    ILM->>Dur: start duration timer
    ILM->>Model: "generate_async(prompt, stop, **kwargs)"
    Model-->>ILM: LLMResponse
    ILM->>Dur: record duration (stop timer)
    ILM->>Span: set_llm_response_attributes
    ILM->>Span: set_llm_call_content (if capture enabled)
    ILM->>Span: close span
    ILM->>Metrics: record_token_usage
    ILM-->>Caller: LLMResponse

    Note over ILM,Span: On error: record_span_error re-raise

    Caller->>ILM: "stream_async(prompt, **kwargs)"
    ILM->>Span: open CLIENT span
    ILM->>Span: "set_llm_request_attributes (stream=True)"
    ILM->>Dur: start duration timer
    loop each chunk
        ILM->>Model: yield chunk
        Model-->>ILM: LLMResponseChunk
        ILM->>Metrics: record_time_to_first_chunk / per_chunk
        ILM-->>Caller: yield chunk
    end
    ILM->>Dur: record duration (stop timer)
    ILM->>Model: aclose() [finally]
    ILM->>Span: set_llm_response_attributes
    ILM->>Span: set_llm_call_content (if capture enabled)
    ILM->>Span: close span
    ILM->>Metrics: record_token_usage

Loading

_{Reviews (4): Last reviewed commit: "docs(llm): document model telemetry surf..." | Re-trigger Greptile}

[coderabbitai]

🧹 Nitpick comments (1)

nemoguardrails/llm/models/instrumented.py (1)

177-192: 🩺 Stability & Availability | 🔵 Trivial

Confirm marking consumer cancellation / early close as span ERROR is intended.

When the consumer closes the stream early (GeneratorExit) or the task is cancelled (CancelledError), the exception propagates through llm_call_span, which sets error.type and StatusCode.ERROR on the CLIENT span (as the tests assert). This conflates normal client-initiated cancellation with genuine provider failures and can inflate error rates on dashboards/alerts derived from span status. If that's not desired, consider treating GeneratorExit/CancelledError as non-error terminations.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nemoguardrails/llm/models/instrumented.py` around lines 177 - 192, Update the
streaming cleanup/error handling around llm_call_span so consumer-initiated
GeneratorExit and CancelledError terminations are treated as normal
cancellations rather than marking the CLIENT span with error.type or
StatusCode.ERROR. Preserve error recording for genuine provider failures and
retain the existing stream close and token-usage behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@nemoguardrails/llm/models/instrumented.py`:
- Around line 177-192: Update the streaming cleanup/error handling around
llm_call_span so consumer-initiated GeneratorExit and CancelledError
terminations are treated as normal cancellations rather than marking the CLIENT
span with error.type or StatusCode.ERROR. Preserve error recording for genuine
provider failures and retain the existing stream close and token-usage behavior.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b3e7b59e-0716-48f2-a40b-d464477f10c6

📥 Commits

Reviewing files that changed from the base of the PR and between 77168c4 and da1db40.

📒 Files selected for processing (9)

• nemoguardrails/__init__.py
• nemoguardrails/guardrails/engine_registry.py
• nemoguardrails/guardrails/telemetry.py
• nemoguardrails/llm/models/instrumented.py
• nemoguardrails/llm/telemetry.py
• nemoguardrails/tracing/constants.py
• tests/guardrails/test_telemetry_content_capture.py
• tests/guardrails/test_telemetry_spans.py
• tests/llm/models/test_instrumented.py