Skip to content

Fix: record the thrown error on the invocation, invoke_agent and call_llm spans - #817

Open
AmaadMartin wants to merge 3 commits into
mainfrom
fix/span-error-status-invocation-agent-llm
Open

Fix: record the thrown error on the invocation, invoke_agent and call_llm spans#817
AmaadMartin wants to merge 3 commits into
mainfrom
fix/span-error-status-invocation-agent-llm

Conversation

@AmaadMartin

@AmaadMartin AmaadMartin commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):

No existing issue.

  1. Or, if no issue exists, describe the change:

Problem: A crashed ADK run does not look crashed in a trace backend. OpenTelemetry JS records nothing when a span body throws, so invocation, invoke_agent and call_llm all finish with UNSET status, no exception event and no error.type. The call_llm span is worse: it ended with a statement placed after the delegation, so a throw skipped span.end() and the span never reached an exporter at all. adk-python does not have this gap, because its SDK records the exception for a start_as_current_span context manager.

Solution: Add runAsyncGeneratorInSpan(spanName, thisArg, fn) to core/src/telemetry/tracing.ts, which starts the span, binds it as the active OTEL context, records any error that ends the generator, rethrows it unchanged, and always ends the span. All four sites now delegate to it, so the span lifecycle lives in one place. resolveErrorType(error) in core/src/utils/error_utils.ts derives the label: a duck-typed HTTP status first, otherwise the class name. The status message carries the error type, not the message, because the message can hold user or model content that ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS does not gate there.

Notes for the reviewer:

  • The error object reaches the caller unchanged (===). One test asserts identity.
  • base_agent.ts shows a large hunk, but git diff main -w reports 5 added and 19 removed lines. The rest is one level of de-indentation from dropping the wrapper.
  • runner.ts keeps its own finally for the toolset teardown. The helper's finally is inner, so span.end() still runs first.
  • resolveErrorType omits the self-classified errorType branch that adk-python has. Nothing in adk-js, @google/genai or the MCP SDK sets that field, so the branch would classify a shape the JS runtime never produces.
  • Out of scope, on purpose: execute_tool (PR Fix: record the tool exception on the execute_tool span #710) and runAndHandleError, which absorbs an Error-typed model failure into an error event instead of rethrowing. Widening that changes behaviour, not telemetry.
  • Overlap check against the 706 open PRs on this fork: no PR lands this change. PR Fix: record the tool exception on the execute_tool span #710 and PR Feat: Port TokenUsage and resolveErrorType from adk-python (Part 1/2) #399 each add their own resolveErrorType to the same file, so whichever lands second reconciles that one function.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

New file core/test/telemetry/span_error_recording_test.ts (11 tests) drives each span through its real call path with a NodeTracerProvider and an InMemorySpanExporter. core/test/utils/error_utils_test.ts gains a resolveErrorType block (7 tests); the existing formatError tests are untouched.

npx vitest run --project unit:core core/test/telemetry core/test/agents core/test/runner core/test/utils
  -> 54 files, 714 passed
npm run build && npm run lint && npm run format:check
  -> all clean

Every line this change adds is covered. error_utils.ts reports 100% line and branch coverage, and every line of runAsyncGeneratorInSpan and recordSpanError reports a non-zero hit count in the v8 report.

Proof the tests can fail. Each mutation was applied to the source, the tests were run, and the source was restored:

Mutation Result
Delete both catch blocks in base_agent.ts 2 failed: expected +0 to be 2 (UNSET, not ERROR)
Delete only the runLive catch 1 failed, the runAsync case still passed
Delete the catch in runner.ts 2 failed: expected +0 to be 2
Restore llm_agent.ts to span.end() after the delegation 3 failed: expected [] to have a length of 1 but got +0
Keep the catch but drop span.end() from the finally 4 failed, including the success path
error.constructor.name -> error.name 3 failed: expected 'Error' to be 'QuotaExceededError'
Remove the 100-599 status bound 1 failed: expected '0' to be 'QuotaExceededError'
Remove the errorType precedence 3 failed: expected '429' to be 'RATE_LIMIT'

Manual End-to-End (E2E) Tests:

Build the package, then run a script that registers a NodeTracerProvider with an InMemorySpanExporter, drives Runner.runAsync over an agent that throws, over an agent whose model throws the string MODEL_UNAVAILABLE, and over a model that answers, and prints each finished span.

Before this change:

--- throwing agent ---   caught: Boom: agent blew up
invoke_agent throwing_agent | status=0 | error.type=- | exception events=0
invocation                  | status=0 | error.type=- | exception events=0
--- throwing model ---   caught: "MODEL_UNAVAILABLE"
invoke_agent llm_agent      | status=0 | error.type=- | exception events=0
invocation                  | status=0 | error.type=- | exception events=0

The call_llm span is absent from the failing model run.

After this change:

--- throwing agent ---   caught: Boom: agent blew up
invoke_agent throwing_agent | status=2 | error.type=Boom | exception events=1
invocation                  | status=2 | error.type=Boom | exception events=1
--- throwing model ---   caught: "MODEL_UNAVAILABLE"
call_llm                    | status=2 | error.type=MODEL_UNAVAILABLE | exception events=1
invoke_agent llm_agent      | status=2 | error.type=MODEL_UNAVAILABLE | exception events=1
invocation                  | status=2 | error.type=MODEL_UNAVAILABLE | exception events=1
--- success path ---     caught: <none>
call_llm                    | status=0 | error.type=- | exception events=0
invoke_agent llm_agent      | status=0 | error.type=- | exception events=0
invocation                  | status=0 | error.type=- | exception events=0

The success-path output is identical to main.

CI note: the first macOS run failed on tests/integration/app_loader/app_loader_test.ts with Test timed out in 40000ms. That test is unrelated to this change, it passed on Ubuntu and Windows in the same run, and it passed on a re-run of the same commit.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.

Amaad Martin added 3 commits August 8, 2026 11:26
…and call_llm spans

OpenTelemetry JS does not record anything when a span body throws, so a
crashed ADK run exported spans with status UNSET and no exception event.
The call_llm span also ended with a statement after the delegation, so a
throw skipped span.end() and the span never reached an exporter.

Add recordSpanError() to the telemetry module and resolveErrorType() to
error_utils, and catch-record-rethrow at the four span sites. The
call_llm span now ends from a finally.
Drive each span through its production call path with the real OTel SDK
and an in-memory exporter, and pin both the failure and the success path.
The call_llm case also pins that the span is exported at all, which it
was not before.
…InSpan

The start-bind-record-end sequence was copy-pasted at all four span
sites. Move it into one helper next to runAsyncGeneratorWithOtelContext,
so each site is a single delegation and recordSpanError stays private to
the telemetry module. The runner keeps its own finally for the toolset
teardown, which still runs after span.end().

Also drop the errorType branch of resolveErrorType. Nothing in adk-js,
@google/genai or the MCP SDK sets that field, so the branch classified a
shape the runtime never produces.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant