Skip to content

Feat: Port SqliteSpanExporter from adk-python to core telemetry - #502

Open
AmaadMartin wants to merge 7 commits into
mainfrom
feat/sqlite-span-exporter
Open

Feat: Port SqliteSpanExporter from adk-python to core telemetry#502
AmaadMartin wants to merge 7 commits into
mainfrom
feat/sqlite-span-exporter

Conversation

@AmaadMartin

@AmaadMartin AmaadMartin commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):
    Closes: N/A
    Related: N/A
  2. Or, if no issue exists, describe the change:
    Problem: adk-python can persist OpenTelemetry spans to a local SQLite database (src/google/adk/telemetry/sqlite_span_exporter.py), so the local dev UI can reload trace history for older sessions after the process restarts. adk-js has no equivalent: core/src/telemetry/ contains only google_cloud.ts, setup.ts and tracing.ts, and the only span sink in the JS dev server is the process-local InMemoryExporter in dev/src/utils/telemetry_utils.ts, whose contents are lost on restart.

Solution: Port SqliteSpanExporter to core/src/telemetry/sqlite_span_exporter.ts, with the ReadableSpan <-> row mapping in core/src/telemetry/db/span_mapper.ts next to the StorageSpan entity it maps. The exporter is a plain @opentelemetry/sdk-trace-base SpanExporter, so it plugs into the existing OTelHooks.spanProcessors surface with no changes to setup.ts and no second OTel init path:

import {SimpleSpanProcessor} from '@opentelemetry/sdk-trace-base';
import {maybeSetOtelProviders, SqliteSpanExporter} from '@google/adk';

const exporter = new SqliteSpanExporter({dbPath: '/tmp/adk_traces.db'});
maybeSetOtelProviders([{spanProcessors: [new SimpleSpanProcessor(exporter)]}]);

getAllSpansForSession(sessionId) returns whole trace trees: it resolves the trace ids belonging to the session first, then returns every span of those traces, so parent and sibling spans that carry no session id of their own come back too.

Persistence reuses the MikroORM stack that already ships for DatabaseSessionService rather than adding a second SQLite client. @mikro-orm/sqlite is a dev/peer dependency only, so it is loaded through a dynamic import() exactly as core/src/sessions/db/operations.ts does — installing @google/adk without the peer keeps every other import path working. Schema creation reuses ensureDatabaseCreated() (ensureDatabase() + updateSchema({safe: true})), which never drops tables or columns, so the exporter can safely point at the same .db file as DatabaseSessionService. (Safe mode may still re-create a spans table another writer declared with INTEGER rather than bigint; row data survives and both declarations carry INTEGER affinity, so this is inert.)

Scope: this PR ships the write path plus its read API, and nothing consumes the read API yet. getAllSpansForSession has no in-repo caller. Its intended consumer is dev/src/server/adk_api_server.ts, which still serves /debug/trace/session/:sessionId from InMemoryExporter.getFinishedSpans (dev/src/utils/telemetry_utils.ts) — the same method shape, which is why the return type is ReadableSpan[] rather than a bespoke row type. Repointing that endpoint is a deliberate follow-up, queued separately, because it changes the dev/ CLI surface (a flag to choose the sink, a migration for the in-memory default) and would make this diff unreviewable. adk-python is in the same state: its SqliteSpanExporter appears only in its own module and its tests, wired into no entry point, so parity does not require it here either. core/src/telemetry/setup.ts, core/src/index_web.ts (the browser bundle must not pull in SQLite) and everything under dev/ are untouched.

Collision check. gh pr list --repo AmaadMartin/adk-js --state open --limit 1000 returned 401 open PRs; none touches core/src/telemetry/sqlite_span_exporter.ts or core/src/telemetry/db/, and no open PR title or branch mentions a SQLite span exporter. The adjacent telemetry PRs (#400, #399, #395, #394, #387, #357, #348, #308) touch metrics.ts, token_usage.ts, google_cloud.ts and tracing.ts only, and the adjacent MikroORM PRs (#500, #291, #386) touch driver peer-dependency declarations and the session service. No overlap, so this branches from main rather than stacking.

Parity wins (observable across the boundary, so adk-python decides):

  • Table name spans and all nine column names/nullability, verified against the generated DDL:
    CREATE TABLE `spans` (`span_id` text not null, `trace_id` text not null, `parent_span_id` text null, `name` text not null, `start_time_unix_nano` bigint null, `end_time_unix_nano` bigint null, `session_id` text null, `invocation_id` text null, `attributes_json` text null, primary key (`span_id`))
  • Both index names: spans_session_id_idx, spans_trace_id_idx.
  • The gcp.vertex.agent.session_id -> gen_ai.conversation.id session-id fallback, and gcp.vertex.agent.invocation_id. These are the keys adk-js already emits (core/src/telemetry/tracing.ts), so no key translation is needed. The fallback uses ||, not ??, so an empty session id falls through exactly as Python's or does.
  • The trace-wide, two-step session query.
  • busy_timeout = 30000 on the connection, matching the reference's sqlite3.connect(timeout=30.0). Without it a contended write fails immediately at the driver default of 1s — plausible precisely in the shared-file-with-DatabaseSessionService configuration this class documents.
  • A failure result plus a log line rather than a throw.

Idiom wins (never leaves the process, so local convention decides):

  • forceFlush() returns Promise<void>, not bool — mandated by the OTel JS SpanExporter interface.
  • export() reports through the resultCallback (ExportResultCode.SUCCESS / FAILED) instead of returning a value, again per the JS interface. It never throws synchronously and never leaves an unhandled rejection.
  • Attribute entries whose JSON value is not a legal OTel AttributeValue (nested objects, mixed arrays) are dropped on read. adk-python round-trips them because Python span attributes are untyped; OTel JS types Attributes as primitives and homogeneous primitive arrays, so such a value could never have been a real JS span attribute, and smuggling it through would need a cast.
  • No '<not serializable>' sentinel here. Python needs one because json.dumps meets arbitrary objects; in JS the only values JSON.stringify cannot represent are bigint and cycles, neither of which is an AttributeValue — and tracing.ts's safeJsonSerialize already substitutes that exact literal before an attribute is set. Reimplementing it in the exporter would have been the second copy of the same sentinel in the same directory, and for functions and symbols it was actively wrong: JSON.stringify drops those silently, so a replacer fabricated a '<not serializable>' string attribute that then read back as a real value. A try/catch returning '{}' is all that remains.
  • shutdown() awaits an in-flight open before closing, so a shutdown issued mid-export cannot orphan the connection that open is still creating (Python, being synchronous, has no equivalent race).
  • Python's threading.Lock is replaced by a memoized init promise: the event loop is single-threaded and MikroORM serialises its own connection, so N concurrent export() calls perform exactly one MikroORM.init. A failed open is not memoized, so a later export retries instead of being poisoned forever (Python reconnects the same way).

Three implementation notes where the obvious approach is wrong:

  1. Timestamps. Epoch nanoseconds (~1.7e18) exceed Number.MAX_SAFE_INTEGER (9.0e15). The bigint column stores them exactly, but the sqlite3 driver behind @mikro-orm/sqlite hands JavaScript a double for integer columns, which silently rounds 1750000000123456789 to ...800. StorageSpan therefore uses a BigIntType('string') subclass that overrides convertToJSValueSQL to cast(<col> as text). That cast also applies to ORDER BY, which would then sort lexicographically ('1000000000' < '1750000000' < '999'), so ordering is done in JS with a BigInt comparison instead. Both effects are pinned by tests.
  2. Reporting a failure. A driver error inlines the failing statement together with its bound values, so logging it verbatim dumped attributes_json — which by default carries the serialized LlmRequest/LlmResponse and tool arguments, since shouldAddRequestResponseToSpans() defaults to true — into the application log. The same string was handed back as ExportResult.error, and SimpleSpanProcessor routes that into OpenTelemetry's global error handler, whose flattenException stringifies every property it can reach, cause included, so it was logged twice. A failure is now named by its error class and SQLite code only (Failed to export spans to SQLite: NotNullConstraintViolationException (SQLITE_CONSTRAINT)), and the reported Error carries no cause and no driver text on any property. The allowlist is /^SQLITE_[A-Z_]+$/ — the underscore matters, or extended codes like SQLITE_CONSTRAINT_NOTNULL and SQLITE_IOERR_READ would degrade to a bare Error and lose the diagnostic the allowlist exists to keep.
  3. Opening the database. MikroORM.init() opens the SQLite file before it can fail, and it fails without returning a handle, so a bad file leaked an open connection with no way to close it. The first CI run on this PR caught exactly that: run-tests (windows-latest) failed with EBUSY: resource busy or locked, unlink '...corrupt.db' while Linux and macOS passed. open() now uses connect: false so the exporter owns the instance before any I/O happens, and closes it if connecting or schema setup throws. Verified locally too: a Node process that runs one failing export and shuts down now exits cleanly instead of hanging on the leaked handle.

New direct dependency: @opentelemetry/core@^2.8.0. ExportResult and ExportResultCode live there and are named in the export() signature (TypeScript does not contextually type method parameters from an implements clause), and @opentelemetry/sdk-trace-base re-exports neither. It was already present at exactly 2.8.0 as a transitive dependency of @opentelemetry/sdk-trace-base@2.8.0; the lockfile diff is the single added line. No other package.json or version field is touched.

No suppressions of any kind are added: the diff against the branch base returns nothing for @ts-expect-error, @ts-ignore, eslint-disable, v8 ignore, as any, as never or a bare : any, tests included.

Follow-up worth naming: hrTimeToUnixNanos (exact, string) now coexists with dev/src/utils/telemetry_utils.ts's hrTimeToNanoseconds (lossy number, and the one behind the live /debug/trace/session/:sessionId endpoint). Two conversions of the same quantity in two packages, one silently wrong above Number.MAX_SAFE_INTEGER. Consolidating them belongs with the dev-server wiring change, not here.

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.

40 new tests across two files that mirror the source layout — core/test/telemetry/sqlite_span_exporter_test.ts (33) and core/test/telemetry/db/span_mapper_test.ts (7):

npx vitest run --project unit:core core/test/telemetry core/test/sessions/database_session_service_test.ts
  Test Files  6 passed (6)
       Tests  85 passed (85)

Coverage of the three new source files is 100% of statements, branches, functions and lines (sqlite_span_exporter.ts, db/span_mapper.ts, db/schema.ts). No coverage suppression is used and no structure was weakened to reach the number.

All 16 contract tests from adk-python/tests/unittests/telemetry/test_sqlite_span_exporter.py have a counterpart, plus JS-specific cases the Python suite does not have: persistence across a restart, two concurrent first exports, a corrupt database file, a retry after the fault is cleared, nanosecond precision beyond Number.MAX_SAFE_INTEGER, a non-string session-id attribute, a row written by another writer (NULL timestamps + invalid attributes JSON), a failure that must not log its payload, two shutdown-races-open cases, and an end-to-end pass through a real BasicTracerProvider + SimpleSpanProcessor (a local provider, not maybeSetOtelProviders, so the global OTel provider is not mutated).

test_shutdown_closes_connection asserts on Python's private _conn. The JS counterpart asserts the same behaviour through the public API instead of widening a private member: after shutdown() the previously written spans are still readable (proving a clean reopen), and a second shutdown() does not throw.

Every test was proved able to fail. Each mutation below was applied to the source, the suite re-run, and the source reverted:

# Mutation Test that failed Failure
1 Drop the gen_ai.conversation.id session-id fallback indexes a span that only carries the conversation id attribute expected [] to have a length of 1 but got +0
2 Reverse compareByStartTime orders spans numerically by start time (+1) expected '0000000000000300,…' to be '0000000000000100,…'
3 Query by session_id only instead of trace-wide includes trace siblings that carry no session id (+1) expected [ 'call_llm' ] to deeply equal [ 'call_llm', 'invocation', …(1) ]
4 Compute epoch nanos with Number instead of BigInt converts a timestamp above Number.MAX_SAFE_INTEGER exactly (+1) expected '1750000000123456800' to be '1750000000123456789'
5 Remove the cast(<col> as text) read override preserves nanosecond precision beyond Number.MAX_SAFE_INTEGER expected [ 1750000000, 123456800 ] to deeply equal [ 1750000000, 123456789 ]
6 Report SUCCESS on a persistence failure reports failure without throwing when the database file is not a database (+1) expected +0 to be 1
7 shutdown() stops clearing the memoized init promise reopens the database after shutdown … (+2) Unable to acquire a connection
8 Coerce a non-string session-id attribute instead of ignoring it stores a span whose session id attribute is not a string without indexing it expected [ { name: 'test_span', …(16) } ] to deeply equal []
9 Accept mixed arrays and nested objects on read drops nested objects and mixed or non-primitive arrays expected { keep: 'value', …(3) } to deeply equal { keep: 'value', nulls: […] }
10 Log String(cause) instead of describeError(cause) reports a failure without logging the span payload that caused it logged line contained the raw insert into spans … 'SUPER_SECRET_PROMPT' …
11 Re-attach the driver error as Error.cause reports a failure without logging the span payload that caused it expected NotNullConstraintViolationException: inse… to be undefined
12 Memoize a failed open (no retry) reopens the database on the next export once the fault is cleared expected 1 to be +0
13 Narrow the code allowlist to /^SQLITE_[A-Z]+$/ keeps an extended SQLite driver code expected 'Error' to be 'Error (SQLITE_CONSTRAINT_NOTNULL)'

Mutation 14 — removing an intra-batch de-duplication map by span id — killed no test: MikroORM's upsertMany applies the rows in order and the last version wins on its own, so that code was deleted rather than kept, and the test was strengthened to interleave three copies of one span id with a distinct span.

Connection-leak regressions are the one class of defect these mutations cannot surface on Linux, because POSIX happily unlinks an open file. They are pinned two other ways: the failure and shutdown-race tests unlink the database file themselves right after shutdown() (await expect(rm(...)).resolves.toBeUndefined()), which is a real assertion on Windows, and the run-tests (windows-latest) job on this PR is the end-to-end proof. The same honesty applies to busy_timeout, a per-connection pragma with no observable effect through the public API: it is covered but not asserted, and was verified out of band — a fresh connection reports 1000, the exporter's reports 30000.

Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.

npm install
npx vitest run --project unit:core core/test/telemetry
npm run build
npm run lint
npm run format:check
npm run docs:check

To inspect the on-disk artefact by hand, export a couple of spans to a temp path, shut the exporter down, construct a second one on the same path and read the session back, then open the file with any SQLite client:

.schema spans
select span_id, trace_id, session_id, name, typeof(start_time_unix_nano),
       cast(start_time_unix_nano as text) from spans;

I ran exactly that. The schema is the nine snake*case columns plus both spans*\*\_idxindexes shown above,typeof(start_time_unix_nano)isinteger, and cast(start_time_unix_nano as text)is1750000000123456789— stored and read back with no rounding. The second exporter returned both spans in start-time order, including the parent span that carries no session id, with the child'sparentSpanContext.spanId pointing at the parent.

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.

Amaad Martin added 5 commits August 1, 2026 22:59
Ports adk-python's SqliteSpanExporter to adk-js. Spans exported through the
existing OTelHooks.spanProcessors surface are persisted to a SQLite file, and
whole trace trees can be read back for a session id after a process restart.

Reuses the MikroORM stack already shipped for DatabaseSessionService: the
@mikro-orm/sqlite driver is loaded through a dynamic import so consumers who
install @google/adk without the optional peer are unaffected.

@opentelemetry/core becomes a direct dependency because ExportResult and
ExportResultCode are named in the export() signature.
Ports all 16 contract tests from adk-python's test_sqlite_span_exporter.py and
adds JS-specific cases: persistence across a restart, concurrent first export,
a corrupt database file, nanosecond precision beyond Number.MAX_SAFE_INTEGER,
a non-string session id attribute, rows written by another writer, and an
end-to-end pass through a real BasicTracerProvider + SimpleSpanProcessor.
The upsert already applies rows in order, so the last version of a repeated
span id wins without a pre-pass. Removing the map with the test still in place
proved the behaviour is unchanged, and the test now interleaves three copies of
one span id with a distinct span to pin it.
Move the JSON replacer to a module-level factory so it is not defined inside
another function, convert epoch nanoseconds without a string round trip, and
drop a default for a NOT NULL column that can never be missing.
MikroORM.init opens the file before it can throw, and it throws without
returning a handle, so a failed open leaked the connection: the Windows CI job
then failed to unlink the database file with EBUSY. Initialize with
connect: false so the exporter owns the instance before any I/O, and close it
if connecting or schema setup fails.
Amaad Martin added 2 commits August 1, 2026 23:55
A driver error inlines the failing statement together with its bound values,
so logging it verbatim dumped attributes_json - which carries the serialized
LLM request and response by default - into the application log, and again
through OpenTelemetry's global error handler, which stringifies every property
it can reach on ExportResult.error including cause.

The failure is now named by its error class and SQLite code only, and the
reported Error carries no cause. Also: shutdown awaits an in-flight open so a
shutdown issued mid-export cannot orphan the connection, and the connection
sets busy_timeout to 30s, matching the reference exporter's sqlite3 connect
timeout, so a contended write waits instead of dropping the batch.
The exporter file was mostly free functions: attribute keys, time conversion,
attribute (de)serialization and the ReadableSpan <-> StorageSpan mappers left
the class itself as a minority of the file. That mapping concern is meaningless
outside this feature, so it belongs next to the entity it maps rather than in
the shared utils directory, and its tests move with it.

Also drops the JSON replacer. It defended against bigint, function and symbol
attribute values, none of which can reach it: OpenTelemetry types attributes as
primitives and primitive arrays, tracing.ts already substitutes
'<not serializable>' for an unstringifiable payload before setting one, and
JSON.stringify silently drops functions and symbols anyway - so for those two
the replacer fabricated a '<not serializable>' string attribute that then read
back as a real value. The plain try/catch fallback remains.

Deletes the unixNanosToHrTime wrapper, which had no production caller, shrinks
compareByStartTime to a signed subtraction, projects the trace-id lookup with
fields: ['traceId'] instead of re-reading whole rows, and widens the SQLite
code allowlist to /^SQLITE_[A-Z_]+$/ so extended codes such as
SQLITE_CONSTRAINT_NOTNULL keep their diagnostic instead of degrading to a bare
Error.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant