Feat: export GCP metrics via OTLP to telemetry.googleapis.com instead of the Cloud Monitoring exporter

[AmaadMartin]

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):
   Closes: #issue_number
   Related: #issue_number
2. Or, if no issue exists, describe the change:

Problem: @google/adk depends on @google-cloud/opentelemetry-cloud-monitoring-exporter@^0.21.0 for GCP metric export. That package itself is tiny (npm view @google-cloud/opentelemetry-cloud-monitoring-exporter@0.21.0 dist.unpackedSize → 53133, 53 KB) but it hard-depends on googleapis: "^137.0.0", which is 110.6 MB unpacked (npm view googleapis@137.1.0 dist.unpackedSize → 110624690). Its compiled build/src/monitoring.js reaches in with require("googleapis/build/src/apis/monitoring"). Every consumer of @google/adk downloaded that subtree whether or not they ever constructed a Cloud Monitoring exporter — the exporter is only reachable through getGcpExporters({enableMetrics: true}), i.e. the --otel_to_cloud CLI flag, which defaults to false (dev/src/cli/cli.ts:130-133).

In the lockfile, that exporter was the only dependent of googleapis, and googleapis the only dependent of googleapis-common; between them they carried three of the six nested google-auth-library@9.15.1 copies.

Solution: Export GCP metrics with a direct gRPC OTLP exporter pointed at the Google Cloud Telemetry API (https://telemetry.googleapis.com) instead, and drop the Cloud Monitoring exporter.

This is a cross-language parity change. adk-python already made OTLP-to-telemetry.googleapis.com its default GCP metric exporter: src/google/adk/telemetry/google_cloud.py defines _get_gcp_otlp_metric_exporter, whose docstring reads "Returns a raw OTLP push metric exporter to telemetry.googleapis.com. This is the default GCP metric exporter (over Cloud Monitoring)." It targets _DEFAULT_TELEMETRY_METRICS_ENDPOINT = "https://telemetry.googleapis.com/v1/metrics" and wraps it in a PeriodicExportingMetricReader at MIN_EXPORT_INTERVAL_MS = 5000.0 (_agent_engine_metric_exporter.py:162). Verified by reading those files at google/adk-python main, not inferred from the task description. adk-js was the outlier.

Concretely:

• core/src/telemetry/google_cloud.ts builds new OTLPMetricExporter({url, credentials}) from @opentelemetry/exporter-metrics-otlp-grpc inside the existing PeriodicExportingMetricReader, with two new module-private constants (TELEMETRY_ENDPOINT, METRIC_EXPORT_INTERVAL_MS; the latter replaces the inline 5000 and keeps parity with MIN_EXPORT_INTERVAL_MS).
• core/package.json drops @google-cloud/opentelemetry-cloud-monitoring-exporter and adds @opentelemetry/exporter-metrics-otlp-grpc@^0.205.0 (28.7 KB unpacked, matching the 0.205.0 line the repo already pins for the other OTLP exporters) plus @grpc/grpc-js@^1.14.4, which becomes a direct dependency because the new code imports credentials and Metadata from it rather than relying on it transitively.
• getGcpResource now takes the destination project and adds two resource detectors — see Resource identity below, which is the non-obvious half of this change.
• The tracing branch, core/src/telemetry/setup.ts and the exported getGcpExporters / OTelHooks contract are unchanged. @opentelemetry/exporter-metrics-otlp-http stays — setup.ts still uses it for the generic OTEL_EXPORTER_OTLP_* env-var path.

Resource identity: why this is not just an exporter swap

The Cloud Monitoring exporter was constructed as new MetricExporter({projectId}) — the destination project was an exporter argument. OTLP has no such argument: the Telemetry API takes the destination project as the gcp.project_id resource attribute on the exported payload. The collector config Google publishes at stackdriver/docs/otlp-metrics/deploy-collector says so outright: "When sending telemetry to the GCP OTLP endpoint, the gcp.project_id resource attribute is required to be set to your project ID." All three official SDK samples set it via OTEL_RESOURCE_ATTRIBUTES in their run instructions.

Swapping the exporter alone would therefore have shipped a silent regression. getGcpResource() ran only the GCP detector, which yields cloud.account.id on GCE and nothing at all off Google Cloud — so adk web --otel_to_cloud on a workstation, the primary dev workflow, would have retried a rejected export every five seconds with no error surfaced. I confirmed the empty payload against a real MeterProvider before fixing it.

Three changes close it, all in getGcpResource:

1. getGcpResource(projectId?: string) merges gcp.project_id. Additive, and the exact signature adk-python uses (get_gcp_resource(project_id: Optional[str] = None), src/google/adk/telemetry/google_cloud.py:228, setting "gcp.project_id": project_id at lines 241/265). adk-js previously took no project at all — an asymmetry that was harmless while metrics went to the TimeSeries API and is not harmless now.
2. envDetector, so OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES reach the payload. Before this they did not: maybeSetOtelProviders uses the passed resource verbatim, so a user could not supply location from the environment even as a workaround. adk-python gets this from OTELResourceDetector, and the detector ordering here reproduces its documented precedence — a gcp.project_id in OTEL_RESOURCE_ATTRIBUTES overrides the argument.
3. serviceInstanceIdDetector, supplying service.instance.id. Per stackdriver/docs/reference/telemetry/v1.metrics, OTLP metrics land on the prometheus_target monitored resource, where location and instance are both required and each documented "Reject the point if empty." instance is sourced from instance → service.instance.id → faas.instance → k8s.pod.name → host.id; off Google Cloud none of those exist, so without this the point is rejected.

Parity deviation, stated explicitly (per the "know which side wins" rule): service.instance.id is observable on the wire, where parity normally wins, and adk-python sets it only on its Agent Engine path. I add it unconditionally anyway, because it is a documented-required label with no other source off Google Cloud, and it stays overridable via OTEL_RESOURCE_ATTRIBUTES. Everything else here follows adk-python.

Known remaining gap, not silently reduced: off Google Cloud, location still has to come from OTEL_RESOURCE_ATTRIBUTES=location=.... No detector can infer a location off-platform, and inventing one (global) is not something I can ground in the docs — so this change makes it supplyable, which it previously was not, and documents it on getGcpResource. On GCE/GKE/Cloud Run the GCP detector fills cloud.region / cloud.availability_zone and nothing extra is needed.

getGcpProjectId() is newly exported so the resource can actually be built with a project. It is the same ADC lookup getGcpExporters already performed internally; without it the new projectId parameter would be dead config, since neither the dev CLI nor a library user has another way to obtain the value. dev/src/utils/telemetry_utils.ts resolves it concurrently with the exporters (Promise.all) and passes it to getGcpResource.

Measured effect (clean rm -rf node_modules && npm install before and after, same machine):

	before	after
du -sh node_modules	620M	515M
packages installed	1096	1086
npm ls googleapis	googleapis@137.1.0 via the Cloud Monitoring exporter	(empty)
nested google-auth-library@9 copies in the lockfile	6	3 (cloud-trace-exporter, storage, vertexai)

googleapis, googleapis-common, and their transitive @google-cloud/precise-date / url-template / nested uuid@9 rows are gone from package-lock.json. The lockfile was regenerated with npm install and is a fixed point (a second install leaves it untouched); no version fields were hand-edited. Every resolved URL in the six new entries points at https://registry.npmjs.org, and each tarball was fetched anonymously from there and checked against the lockfile integrity hash, so the lockfile installs in CI with no registry credentials.

Why gRPC and not the HTTP exporter already in the dependency list

@opentelemetry/exporter-metrics-otlp-http is already a dependency, so it looks like the cheaper option, but its public headers option is typed Record<string, string> — a static header map. The OAuth token would be frozen at construction and every export would fail once it expired. Google's own guidance says the same thing: "due to the lack of support for dynamic token refreshing in most SDK exporters, we recommend using only the gRPC OTLP exporter, not the HTTP exporters, when exporting directly from SDKs" (Google Cloud Observability, OTLP metric ingestion overview, "Protocol support"). adk-python can use the HTTP exporter because google.auth.transport.requests.AuthorizedSession refreshes for it; the JS HTTP exporter has no equivalent hook.

Parity conflict resolution: the observable wire behaviour (destination host, 5000 ms interval, degrade-don't-throw semantics, the byte-identical "Cannot determine GCP Project…" warning) follows adk-python. The transport (gRPC here, HTTP there) and all in-process conventions follow local JS/TS reality, because the JS HTTP exporter cannot refresh a token. The endpoint string also differs by necessity: gRPC targets a host (https://telemetry.googleapis.com, normalised by the exporter to telemetry.googleapis.com and dialled on the gRPC default port 443), where the Python HTTP path needs the /v1/metrics signal path.

The google-auth-library v10 Headers pitfall

Google's published Node.js sample uses credentials.createFromGoogleCredential(authClient). That call is broken against the google-auth-library major this repo pins, and silently so. google-auth-library@10.7.0 declares getRequestHeaders(url?: string | URL): Promise<Headers> — a WHATWG Headers instance (build/src/auth/authclient.d.ts:145) — while @grpc/grpc-js@1.14.4's createFromGoogleCredential builds its metadata with for (const key of Object.keys(headers)) (build/src/call-credentials.js:68) and types its client as returning Promise<{[index: string]: string}>. Object.keys(new Headers(...)) is [], so that path produces empty gRPC metadata and unauthenticated export requests — no crash, no log, just rejected exports.

This change therefore bridges with credentials.createFromMetadataGenerator + Headers.forEach, and there is a why comment at that call site naming the mismatch. it('mints gRPC metadata from ADC request headers') pins it (see the mutation table below).

Reviewed alternatives, and why they were not taken

• @google-cloud/monitoring — rejected. It is a GAPIC client for the Cloud Monitoring v3 API, not an OpenTelemetry PushMetricExporter; adopting it means re-implementing the OTel→TimeSeries translation layer (metric-descriptor creation, monitored-resource mapping, cumulative/delta handling, histogram conversion, label sanitisation, batching). It also pulls google-gax@5, which pins google-auth-library to an exact 10.5.0 and would add a new nested copy alongside the repo's 10.7.0.
• Keeping the exporter as an optional peer dependency — cuts install size but leaves adk-js exporting to a different backend than adk-python, entrenching the parity gap this change exists to close. See the collision note below.

Breaking change

This is a user-visible behaviour change for anyone running with --otel_to_cloud (or calling getGcpExporters({enableMetrics: true}) directly). No API is renamed or removed, so no deprecation shim is required.

1. Destination and metric naming. Metrics previously landed in Cloud Monitoring under the Cloud Monitoring exporter's workload.googleapis.com/ prefix. They now go through the Telemetry API and are stored in Managed Service for Prometheus format. Dashboards, alerting policies and SLOs built on the old names must be repointed. Querying a UTF-8 name in PromQL needs brace-and-quote syntax, e.g. {"gen_ai.client.token.usage"}.
2. Billing. Per Google's OTLP metric ingestion overview: "Billing for OTLP metrics is accounted for under the 'Prometheus Samples Ingested' SKU, the same one used for metrics from Google Cloud Managed Service for Prometheus."
3. API enablement. Callers must enable the Telemetry API: gcloud services enable telemetry.googleapis.com.
4. Launch stage. The Telemetry API OTLP metric path is Pre-GA: "This feature is subject to the 'Pre-GA Offerings Terms' … Pre-GA features are available 'as is' and might have limited support." Shipping it anyway is justified by parity — adk-python already makes it the default GCP metric exporter.
5. Blast radius is narrow. --otel_to_cloud defaults to false, and no ADK documentation names the Cloud Monitoring destination (README.md, CONTRIBUTING.md and docs/ do not mention it), so no docs are invalidated.

Error handling is unchanged in spirit and slightly more forgiving: if ADC cannot produce a client while enableMetrics is true, metrics are disabled with a logger.warn and tracing is left intact — mirroring adk-python's _get_gcp_metrics_exporter, which returns None rather than raising. A metrics setup failure must never take down tracing.

Why the eager getClient() probe is kept

GoogleAuth exposes the same getRequestHeaders(url?): Promise<Headers> signature as AuthClient (google-auth-library@10.7.0, build/src/auth/googleauth.d.ts:470), so auth could be handed straight to the credential builder and the getClient() probe dropped entirely. That would also delete GCP_CREDENTIALS_ERROR_MESSAGE and the metricReaders accumulator, collapsing back to a one-line ternary. It is deliberately not done, for three reasons:

1. It is the specified behaviour. A credential client that cannot be resolved must disable metrics with an actionable warning and leave the span processors installed; that contract is pinned by disables metrics but keeps tracing when the auth client is unavailable. Removing the probe deletes that test along with the behaviour.
2. It trades one warning for an unbounded error loop. Without the probe, a missing ADC surfaces as a gRPC export failure every 5 s for the life of the process instead of a single actionable line at startup. adk web --otel_to_cloud with GOOGLE_CLOUD_PROJECT set but no ADC is exactly that case.
3. Silent-until-you-read-the-logs failure is the defect class this PR already had to fix once (see Resource identity), so shortening the diff by removing early detection is the wrong direction here.

The probe is one line — await auth.getClient().catch(() => undefined). The .catch is scoped to getClient() rather than wrapping the block, so a throw from anywhere else in the metrics branch can never be misreported as a credentials failure.

Aggregation temporality

The replaced Cloud Monitoring exporter implemented its own selectAggregationTemporality; nothing in this diff sets one, so the generic exporter's default applies. Verified rather than assumed: chooseTemporalitySelectorFromEnvironment in @opentelemetry/exporter-metrics-otlp-http@0.205.0 (build/src/OTLPMetricExporterBase.js:54-56) defaults to cumulative when OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is unset. Cumulative is what Prometheus-format ingestion expects — Google's OTLP overview warns the other way round, that "Delta metrics might not query properly in certain circumstances" — and it matches adk-python, which likewise constructs OTLPMetricExporter with no temporality preference. No override is needed, and adding one would be a parity deviation.

Collision check

gh pr list --repo AmaadMartin/adk-js --state open --limit 100 plus gh pr diff --name-only on the plausibly adjacent PRs. Findings:

• Feat: make @google-cloud/storage and the OTel GCP exporters optional peer dependencies #387 "make @google-cloud/storage and the OTel GCP exporters optional peer dependencies" overlaps in core/package.json, core/src/telemetry/google_cloud.ts, core/test/telemetry/google_cloud_test.ts and package-lock.json. It does not land this change: it keeps MetricExporter from @google-cloud/opentelemetry-cloud-monitoring-exporter and lazy-loads it, so googleapis stays in the tree for anyone who opts in, and the adk-python parity gap remains. It is an alternative design for the same file rather than a prerequisite, so this PR branches from main rather than stacking on it — stacking would make an approach this change supersedes a dependency of it, and would make each PR unportable without the other. Whichever merges second will need a small rebase in google_cloud.ts and core/package.json.
• Fix: document why google-auth-library v9 stays duplicated, and pin the allowlist #336 (docs/dependency-duplication.md + a google_auth_library_duplication_test) and Fix: stop shipping a second google-auth-library copy under @google-cloud/vertexai #295 (google_auth_library_dedupe_test) touch dependency-duplication docs/tests only. Neither is in this diff; if Fix: document why google-auth-library v9 stays duplicated, and pin the allowlist #336 merges, its allowlist will need updating because this change takes the nested google-auth-library@9 count from six to three.

No competing implementation of the OTLP metric exporter is open.

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.

Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.

core/test/telemetry/google_cloud_test.ts — the four parametrised getGcpExporters cases and the two project-id-failure cases keep their bodies and assertions. No existing test was deleted, skipped, or weakened. Three existing lines did have to change, all disclosed here:

1. vi.mock('@google-cloud/opentelemetry-cloud-monitoring-exporter') became vi.mock('@opentelemetry/exporter-metrics-otlp-grpc') — the old specifier no longer resolves.
2. The shared auth double gained a getClient stub, because the metrics path now resolves an ADC client.
3. An assertion was rewritten, not added to. should detect GCP resources using gcpDetector asserted detectors: [expect.any(Object)], pinning "exactly one detector" — behaviour this change deliberately alters. It now asserts detectors: [serviceInstanceIdDetector, envDetector, gcpDetector] by identity, so it pins the order the getGcpResource doc comment promises rather than only the arity. An arity-only form would pass for any three detectors in any arrangement; the identity form fails on either transposition (both are in the mutation table below).

Eight new cases were appended in a getGcpExporters OTLP metric export block. The metadata cases deliberately drive the real gRPC CallCredentials captured from credentials.combineChannelCredentials and call its public generateMetadata, rather than asserting which factory function was used — so they fail on empty metadata, which is the actual defect.

Every new test was run against mutated source and observed to fail:

Mutation to core/src/telemetry/google_cloud.ts	Test that failed	Failure message
TELEMETRY_ENDPOINT → 'https://example.invalid'	exports metrics to the Cloud Telemetry OTLP endpoint	expected 'https://example.invalid' to be 'https://telemetry.googleapis.com'
METRIC_EXPORT_INTERVAL_MS → 10000	wraps the OTLP exporter in a 5s periodic reader	expected 10000 to be 5000
createFromMetadataGenerator(...) → createFromGoogleCredential(authClient)	mints gRPC metadata from ADC request headers	expected [] to deeply equal [ 'Bearer test-token' ]
drop the .catch(() => undefined) on auth.getClient()	disables metrics but keeps tracing when the auth client is unavailable	Error: no ADC (rejected instead of degrading)
if (enableMetrics) → if (true)	constructs no metric exporter when metrics are not enabled	expected [ …(1) ] to deeply equal []
e instanceof Error ? e : new Error(String(e)) → always re-wrap	forwards a credential Error to the export RPC	expected Error: Error: token refresh failed to be Error: token refresh failed
same ternary → never wrap (e as Error)	wraps a non-Error credential rejection before forwarding it	expected 'boom' to be an instance of Error
getRequestHeaders called eagerly in createGcpMetricReader	constructs the exporter without performing network I/O	expected "getRequestHeaders" to not be called at all, but actually been called 1 times
getGcpProjectId returns getProjectId() raw instead of degrading	resolves undefined when the project cannot be determined	promise rejected "Error: no ADC" instead of resolving
never attach gcp.project_id (the pre-fix behaviour)	carries the project id the Telemetry API routes on	expected undefined to be 'test-project'
drop envDetector (the pre-fix behaviour)	lets OTEL_RESOURCE_ATTRIBUTES supply the location label	expected undefined to be 'us-central1'
drop serviceInstanceIdDetector	supplies the instance label Prometheus ingestion requires	expected undefined to deeply equal Any<String>
getGcpResource(projectId) → getGcpResource() in the dev CLI	builds the resource with the project the Telemetry API routes on	expected "spy" to be called with arguments: [ 'adc-project' ]
dev CLI stops calling getGcpProjectId()	builds the resource with the project the Telemetry API routes on	expected "spy" to be called 1 times, but got 0 times
swap envDetector / gcpDetector (arity unchanged)	should detect GCP resources using gcpDetector	expected "detectResources" to be called with arguments: [ { detectors: [ … ] } ]
swap serviceInstanceIdDetector / envDetector (arity unchanged)	should detect GCP resources using gcpDetector	expected "detectResources" to be called with arguments: [ { detectors: [ … ] } ]
getGcpProjectId ignores its auth argument and builds a fresh GoogleAuth	prefers injected credentials over the ambient default	expected 'ambient-project' to be 'injected-project'

The last two mutations are the reason dev/test/utils/telemetry_utils_test.ts exists. setupTelemetry(true) is the only production caller of getGcpResource, and dropping the project id argument there compiled cleanly and passed every existing suite — the resource wiring had no coverage at all. That gap is what let the original version of this PR ship a payload with no project on it.

Coverage of core/src/telemetry/google_cloud.ts is 100% statements / branches / functions / lines, with no coverage suppressions:

npx vitest run --project unit:core --project integration \
  core/test/telemetry/google_cloud_test.ts \
  tests/integration/telemetry/gcp_otlp_metrics_test.ts \
  --coverage --coverage.include='core/src/telemetry/google_cloud.ts'
 google_cloud.ts | 100 | 100 | 100 | 100 |

Every line this change adds to dev/src/utils/telemetry_utils.ts (167-174, 177) is covered by the new dev suite; the uncovered statements reported for that file are all pre-existing code outside this diff.

Integration test — tests/integration/telemetry/gcp_otlp_metrics_test.ts (new). Unlike the unit tests it does not mock @opentelemetry/exporter-metrics-otlp-grpc or @opentelemetry/resources, so the real packages have to compose. Two groups:

• The exporter: exactly one reader, that it is a PeriodicExportingMetricReader, and that no credential is minted (hence no gRPC channel opened) before a MeterProvider drains it; the reader is shut down in a finally so its 5 s timer cannot leak.
• The payload: a real MeterProvider is built from getGcpResource(...) the way maybeSetOtelProviders builds one, a counter is recorded, and reader.collect() is asserted against — so the assertions are on the resource attributes an exporter would actually transmit, not on how the resource was constructed. These pin gcp.project_id, service.instance.id, the OTEL_RESOURCE_ATTRIBUTES path for location, and the env-overrides-argument precedence. METADATA_SERVER_DETECTION=none is stubbed so they run the off-Google-Cloud path deterministically and never touch the metadata server from CI.

Regression suites re-run:

npx vitest run --project unit:core core/test/telemetry/           29 passed
npx vitest run --project integration tests/integration/telemetry   6 passed
npx vitest run --project integration tests/integration/build_setup 20 passed | 4 skipped
npx vitest run --project unit:dev dev/test/                      225 passed | 1 failed (pre-existing)

build_setup is the manifest regression guard — it installs core via file: into six fixture projects and builds/runs each. The single dev/test failure is cli_create_test.ts:214 (initialValue: 'gcloud-project'); it reproduces identically on a clean main worktree with no changes, so it is ambient gcloud config leaking into that test and is unrelated to this diff.

Repository gates, all clean on the pushed commit:

npm run build          exit 0   (esm, cjs, web for all three workspaces)
npm run lint           exit 0
npm run format:check   exit 0
npm run docs:check     exit 0

npm run ts:check fails on main today with pre-existing errors in unrelated test files; none of the files in this diff appear in its output.

Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.

Install-size verification (run from a clean clone):

rm -rf node_modules && npm install
du -sh node_modules     # 620M before this change, 515M after
npm ls googleapis       # googleapis@137.1.0 before, "(empty)" after
npm run build

Live export verification (requires a GCP project; not executed in this sandbox — it has no GCP project or ADC, so a reviewer with a project should confirm it):

1. gcloud services enable telemetry.googleapis.com monitoring.googleapis.com
2. gcloud auth application-default login
3. Off Google Cloud only, also export OTEL_RESOURCE_ATTRIBUTES="location=us-central1" — Managed Service for Prometheus rejects points with an empty location, and no detector can infer one off-platform. On GCE/GKE/Cloud Run skip this; the GCP detector supplies it.
4. adk web --otel_to_cloud against a sample agent; send a few turns and wait >10 s (two export intervals).
5. Query in Cloud Monitoring with PromQL. Names are Prometheus-format now, so a UTF-8 name needs brace-and-quote syntax, e.g. {"gen_ai.client.token.usage"}.
6. Re-run step 4 with ADC unavailable (e.g. gcloud auth application-default revoke) and confirm the process logs Cannot obtain Application Default Credentials… and keeps serving, with tracing still installed, rather than crashing.

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.