Skip to content

Feat: make @google-cloud/storage and the OTel GCP exporters optional peer dependencies - #387

Open
AmaadMartin wants to merge 8 commits into
mainfrom
feat/optional-gcp-dependencies
Open

Feat: make @google-cloud/storage and the OTel GCP exporters optional peer dependencies#387
AmaadMartin wants to merge 8 commits into
mainfrom
feat/optional-gcp-dependencies

Conversation

@AmaadMartin

@AmaadMartin AmaadMartin commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):
    Closes: #issue_number
    Related: #issue_number
  2. Or, if no issue exists, describe the change:
    Problem: Every consumer of @google/adk downloads a large dependency closure for three packages that only two optional features use:
Package Only consumer in core/src
@google-cloud/storage artifacts/gcs_artifact_service.ts (GcsArtifactService)
@google-cloud/opentelemetry-cloud-trace-exporter telemetry/google_cloud.ts (getGcpExporters, only when enableTracing)
@google-cloud/opentelemetry-cloud-monitoring-exporter telemetry/google_cloud.ts (getGcpExporters, only when enableMetrics)

The monitoring exporter is small itself but declares googleapis ^137.0.0, which is by far the largest thing in the tree. Users who never touch GCS artifacts or Cloud Trace / Cloud Monitoring export pay for all of it.

Solution: Move the three packages out of core's runtime dependencies into optional peer dependencies, and load each with a lazy import() at the point its feature is actually used. This follows the lazy-loading convention already established in core/src/sessions/db/operations.ts and tests/integration/lazy_load_db_drivers/ rather than inventing a second pattern.

Two deliberate improvements over that precedent:

  1. A shared helper, core/src/utils/optional_dependency_utils.ts, turns Node's raw module-resolution failure into an actionable message: GcsArtifactService requires the optional peer dependency '@google-cloud/storage', which is not installed. Run `npm install @google-cloud/storage` to enable it. It is generically named and lives in the shared utils directory (an express caller is a plausible fourth). It is internal — deliberately not added to core/src/index.ts or core/src/common.ts.
  2. core/package.json gains a peerDependenciesMeta block marking only these three optional. Without it npm 7+ auto-installs peers and the change would achieve nothing. The five existing @mikro-orm/* peers are intentionally left alone — retro-fitting them would silently alter install behaviour for database users and muddy this diff.

Design notes:

  • GcsArtifactService's constructor stays synchronous and keeps its exact signature. getArtifactServiceFromUri stays synchronous. The Storage client is created on first use by a module-level createBucket() and memoized with ??=, so there is exactly one client per service instance and a rejected promise stays cached (a missing package keeps producing the same error rather than retrying resolution on every call).
  • loadArtifact and getArtifactVersion wrap their bodies in try { ... } catch { logger.warn(); return undefined; }. await this.getBucket() is hoisted above the try in both, so a missing package surfaces as a real error instead of being swallowed into undefined. Integration cases 3 and 4 pin this. See the "Two public methods can now reject" disclosure below — this is wider than just the missing-package case.
  • getGcpExporters was already async, so no signature changed anywhere, including dev/src/utils/telemetry_utils.ts.
  • dev/package.json takes the three on as direct dependencies, mirroring how it already carries all five @mikro-orm drivers so the CLI supports every --session_service_uri scheme out of the box. The adk CLI's behaviour is unchanged.
  • Only value imports were removed; Bucket, File, StorageOptions, SpanProcessor and MetricReader are now import type so esbuild erases them and no runtime import survives.
  • No suppressions were added: no any, @ts-expect-error, @ts-ignore, eslint-disable, as any or as unknown as anywhere in this diff (verified by grepping git diff main -U0 for added lines).

Collision check (required before starting): gh pr list --repo AmaadMartin/adk-js --state open --limit 100 returned 100 open PRs. I diffed the plausibly-adjacent ones — #336, #295, #347, #291, #323, #303, #292, #382 — with gh pr diff <n> --name-only. None implements lazy loading or optional peer dependencies for these packages. Four touch core/package.json or dev/package.json in unrelated ways (@types/express declaration, @mikro-orm/reflection removal, an import/no-extraneous-dependencies rule, a google-auth-library dedupe), so they overlap only as potential textual merge conflicts in different fields of the same file, not semantically. I branched from main rather than stacking, because stacking on an arbitrary one of four peripheral overlaps would be misleading and would also suppress CI (the workflow triggers only on pull_request: branches: [main]).

Two claims in the task spec turned out to be wrong; I verified against the repo and report the measured facts instead:

  1. The spec said the ESM build emits a native import() (ERR_MODULE_NOT_FOUND) while only the CJS build is lowered. In fact core/build.js builds both node outputs with platform: 'node'target: ['node10.4'], so esbuild lowers await import() to a require() wrapper in both. Evidence, since this is easy to assume the other way round:

    • core/dist/esm/artifacts/gcs_artifact_service.js:250 and core/dist/cjs/artifacts/gcs_artifact_service.js:265 both contain () => Promise.resolve().then(() => __toESM(require("@google-cloud/storage"), 1)).
    • core/build.js:72-78 prepends a createRequire banner to the esm output specifically (import {createRequire as topLevelCreateRequire} from 'module'; const require = topLevelCreateRequire(import.meta.url);), which exists precisely so that lowered require() works in an ES module. That banner would be dead code if the esm build kept a native import().

    Both error codes are still needed, and the helper's comment now says why each one is reachable: running from source (vitest, ts-node, or a bundler that preserves import()) yields the native import and ERR_MODULE_NOT_FOUND; the published builds yield MODULE_NOT_FOUND.

  2. The spec said @grpc/grpc-js "will not disappear" because @a2a-js/sdk depends on it. It does disappear: @a2a-js/sdk@0.3.14 declares @grpc/grpc-js under peerDependenciesMeta as {"optional": true}, so with the exporters gone nothing pulls it in and it shows as UNMET OPTIONAL DEPENDENCY in the measured tree.

One file the spec listed as "must not change" did have to change: core/test/artifacts/registry_test.ts. Its gs:// case asserted the bucket name by casting into the private field the refactor removes — (service as unknown as {bucket: {name: string}}).bucket.name. Re-adding an equivalent private reach for the new bucketName field would both perpetuate the anti-pattern and add an as unknown as to the diff, so the case now mocks @google-cloud/storage and asserts the same behaviour through the public path (expect(bucketMock).toHaveBeenCalledWith('my-bucket')). The toBeInstanceOf(GcsArtifactService) assertion is untouched, the test name and intent are unchanged, and mutation M8 below proves the new form still catches a wrong bucket name. This is the only existing test modified by this PR — every other test change in the diff is a pure addition. (The rewrite also removes an as unknown as from the repo rather than adding one.)

Two public methods can now reject where they previously always resolved. GcsArtifactService.loadArtifact and getArtifactVersion have always wrapped their whole body in try { ... } catch (e) { logger.warn(...); return undefined; }. Because await this.getBucket() is now hoisted above that try (gcs_artifact_service.ts:114 and :258), any storage-client construction failure escapes the catch — not only the missing-package error. In practice new Storage(options).bucket(name) does no I/O, so the realistic triggers are the missing package and an invalid StorageOptions, but the widening is real and deliberate: leaving the resolution inside the try is exactly the bug that returns a silent undefined to a user who forgot to install the package (mutations M3 and M6 demonstrate it). Failures of the GCS calls themselves still log and return undefined exactly as before. This is now stated in the README section too.

Likewise getGcpExporters can now reject rather than always resolving. The only in-repo caller, dev/src/utils/telemetry_utils.ts:166, ships all three packages so it cannot hit this, but library consumers calling it directly with enableTracing/enableMetrics can.

Behaviour change (please read): this is a behaviour change for a subset of users. Anyone who today uses GcsArtifactService, getArtifactServiceFromUri('gs://...'), or getGcpExporters with enableTracing/enableMetrics and relied on the transitive install will now get the actionable error until they run one npm install. Users of the adk CLI are unaffected. I used a non-breaking feat: prefix rather than feat!: because release-please-config.json uses release-type: node with linked versions across all four components, so feat! would force a simultaneous major bump on main, adk, devtools and integrationsthat call belongs to the maintainers, so please tell me if you would prefer this re-tagged as breaking. No migration script, codemod, or deprecation shim was added.

Accepted trade-off — maintainers may want to rule on this. dist/types/artifacts/gcs_artifact_service.d.ts keeps import type ... from '@google-cloud/storage' because StorageOptions appears in the public constructor signature. The net effect is that the dependency is optional at runtime but not at type-check time: consumers who do not install the package and compile with skipLibCheck: false (not the default) will see TS2307 on that declaration file. This is documented in the new README subsection.

I deliberately did not "fix" it by widening the parameter to unknown/any or hand-duplicating the SDK interface. The only two fixes I can see both change the public surface, so they are your call rather than mine: (a) accept a structural options type instead of StorageOptions, or (b) make the bucket field truly private (#bucketPromise) so Bucket/StorageOptions stay out of the emitted declaration. Happy to do either if you prefer it.

One intentional deviation from the task spec: the spec asked for the exportIntervalMillis: 5000 literal to be extracted into a named METRIC_EXPORT_INTERVAL_MS constant. A complexity review flagged that as single-use scope creep in a diff whose job is lazy loading, given that exportIntervalMillis: already names the value at the call site. I agreed and left the literal inline, so the telemetry diff is now purely the lazy-import change.

Scope: nothing was silently dropped. Explicitly out of scope and left alone (each is separately queued): making express optional, slimming @google/adk-devtools so the CLI stops pulling googleapis, and adding peerDependenciesMeta.optional for the five @mikro-orm/* drivers. docs/dependency-duplication.md was not created or referenced. This shipped as one PR rather than a stack because the manifest change and the lazy loading are not independently shippable — part 1 alone achieves nothing and part 2 alone breaks users; the human-authored diff is 486 insertions across 12 files, of which 238 are new tests.

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.

Targeted runs (per the repo guideline against running the whole suite):

npx vitest run --project unit:core core/test/utils/optional_dependency_utils_test.ts \
  core/test/artifacts/gcs_artifact_service_test.ts core/test/artifacts/registry_test.ts \
  core/test/telemetry/google_cloud_test.ts
npx vitest run --project integration tests/integration/lazy_load_optional_gcp_deps \
  tests/integration/lazy_load_db_drivers

Result: 67 passed (6 files). lazy_load_db_drivers is re-run to prove the shared helper did not disturb the existing convention. All 33 pre-existing gcs_artifact_service_test.ts cases and all 7 pre-existing google_cloud_test.ts cases pass unmodified; no test was skipped, disabled, .only'd, weakened, or deleted.

New tests:

  • core/test/utils/optional_dependency_utils_test.ts (new, 9 cases): happy path; ESM miss; CJS miss; transitive miss rethrown by identity; module throws while evaluating; wrong error code; thrown object with a code but no message; non-Error rejection; feature name in the message.
  • core/test/artifacts/gcs_artifact_service_test.ts: one added describe proving the constructor calls no Storage and two operations share one memoized client.
  • core/test/telemetry/google_cloud_test.ts: one added case for both flags disabled (there was no existing both-false case).
  • tests/integration/lazy_load_optional_gcp_deps/ (new, 7 cases): all three packages mocked as unresolvable, plus google-auth-library mocked so getGcpExporters does not short-circuit on a missing project id and pass vacuously on a credential-less runner.

Coverage. optional_dependency_utils.ts is at 100% statements / branches / functions / lines. google_cloud.ts is at 100% on all four. gcs_artifact_service.ts is at 97.81% / 92.75%; the only uncovered lines are 331-332 and 346-347 inside the pre-existing extractArtifactKeys / getFileNameFromPath helpers, which this change does not touch (the highest hunk in the diff ends at line 305). No structure was made less safe to raise a number.

Proof the tests can fail. Every new test was run against mutated source and confirmed to FAIL:

# Mutation Test that failed Failure message
M1 drop the message.includes(`'${packageName}'`) clause transitive-miss negative test expected Error: GcsArtifactService requires the op… to be Error: Cannot find package 'teeny-request… // Object.is equality
M2 remove 'MODULE_NOT_FOUND' from MODULE_NOT_FOUND_CODES CJS-miss case (+ feature-name case) expected [Function] to throw error including 'Run `npm install @google-cloud/storag…' but got 'Cannot find module \'@google-cloud/st…'
M3 move await this.getBucket() back inside the try in loadArtifact integration rejects loadArtifact promise resolved "undefined" instead of rejecting
M4 replace ??= with a plain call in getBucket() unit lazy Storage construction expected "spy" to be called 1 times, but got 2 times
M5 import the trace exporter unconditionally integration resolves with no exporters when both flags are disabled promise rejected "Error: [vitest] There was an error when m…" instead of resolving
M6 move await this.getBucket() inside the try in getArtifactVersion integration rejects getArtifactVersion promise resolved "undefined" instead of rejecting
M7 construct the bucket eagerly in the constructor unit lazy Storage construction failed on expect(StorageMock).not.toHaveBeenCalled()
M8 new GcsArtifactService(bucket.toUpperCase()) in registry.ts registry_test gs uri case expected "spy" to be called with arguments: [ 'my-bucket' ]
M9 return {} from getGcpExporters when both flags are off unit both-flags-disabled case expected {} to deeply equal { spanProcessors: [], …(2) }

Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.

Install-size measurement (real consumer install, honours peerDependenciesMeta). Run on main and again on this branch:

mkdir -p /tmp/pack /tmp/consumer
(cd core && npm pack --ignore-scripts --pack-destination /tmp/pack)
cd /tmp/consumer && npm init -y
npm install --omit=dev --ignore-scripts /tmp/pack/google-adk-*.tgz
du -sb node_modules
npm ls --all --omit=dev

Measured on this machine (npm 9.2.0, Node v22.22.2):

node_modules (bytes)
main 315,800,960
this branch 192,320,499
delta −123,480,461 (−117.8 MiB, −39.1%)

npm ls --all --omit=dev diff — what actually left the closure (measured, not estimated):

  • googleapis@137.1.0 (111,236,216 bytes on disk here) and googleapis-common: gone.
  • @google-cloud/storage (2,249,521 bytes), both Cloud exporters, @google-cloud/opentelemetry-resource-util, @google-cloud/precise-date, @google-cloud/paginator, @google-cloud/projectify, @google-cloud/promisify, teeny-request, retry-request, fast-xml-parser, form-data, yargs: gone.
  • google-auth-library@9.15.1: 6 copies → 1. The survivor is under @google-cloud/vertexai@1.12.0. The three google-auth-library@10.9.1 copies are unchanged.
  • @grpc/grpc-js and @grpc/proto-loader: gone (2 copies each → 0). See the corrected claim above — @a2a-js/sdk declares them optional.
  • The three packages now correctly appear as UNMET OPTIONAL DEPENDENCY in the consumer tree, which is the direct evidence that peerDependenciesMeta is doing its job.

CLI unchanged, verified end to end:

npm run build && node dev/dist/esm/cli_entrypoint.js --help

prints the full command list (web, api_server, create, run, deploy, integration) as before.

CI: green. run-tests passed on ubuntu-latest, macos-latest and windows-latest.

The latest commit passed all three OS legs on the first attempt. On the first pushed commit the Windows leg needed two re-runs, both for reasons unrelated to this diff, which I am recording rather than hiding (the clean run since is further evidence they were flakes):

  1. First attempt: core/test/code_executors/unsafe_local_code_executor_test.ts > should execute shell code and return stdout timed out at 5000ms (2692 of 2735 tests passed). That file spawns a shell and is not touched by this change — git diff main --name-only contains no code_executors path — and it is a known Windows timeout flake with dedicated fixes already in flight.
  2. Second attempt: the build step failed with Error: read ECONNRESET while fetching assets — a network flake.
  3. Third attempt: pass, 8m57s.

Full local validation on the same commit:

npm install        # package-lock.json is a fixed point; re-running produces no diff
npm run build      # all three workspaces, all three targets: OK
npm run lint       # clean
npm run format:check  # "All matched files use Prettier code style!"
npm run docs:check    # typedoc --treatWarningsAsErrors: clean

docs:check is the one that would catch an unresolvable public-signature type, and it passes because the three packages remain installed in the repo via the new core devDependencies.

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.

Amaad Martin added 8 commits July 31, 2026 05:57
Wraps a lazy import() and converts Node's module-resolution failure into an
error naming the package and the npm install command. Any other failure --
including a missing transitive dependency of the package -- is rethrown
unchanged so a genuine crash is never mislabelled as "not installed".
The constructor stays synchronous and keeps its signature; the Storage client
is created on first use and memoized, so getArtifactServiceFromUri('gs://...')
never resolves @google-cloud/storage. loadArtifact and getArtifactVersion
resolve the bucket above their try block so a missing package surfaces instead
of being swallowed into undefined.

registry_test asserted the bucket name by casting into the now-removed private
'bucket' field; it now asserts the same behaviour through the public path.
getGcpExporters is already async, so each exporter is now imported only when
its flag is set. Names the 5000 ms metric export interval as a constant.
…le/adk

@google-cloud/storage and the two OTel Cloud exporters move out of core's
runtime dependencies into optional peerDependencies (with peerDependenciesMeta,
without which npm 7+ installs peers anyway) plus devDependencies so the repo's
own tsc, vitest, eslint and typedoc still resolve them.

@google/adk-devtools takes them on as direct dependencies, matching how it
already carries all five @mikro-orm drivers, so the adk CLI keeps working
unchanged.
Takes loadOptionalDependency to 100% line and branch coverage.
Replaces the two-element MODULE_NOT_FOUND_CODES Set with a direct comparison
and inlines the single-use METRIC_EXPORT_INTERVAL_MS back at its call site,
where exportIntervalMillis already names the value.

Rewrites the error-code rationale to say when EACH code is reachable, and
documents in the README that loadArtifact and getArtifactVersion now reject on
a storage-client construction failure instead of returning undefined.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant