Skip to content

ci: run cross-language integration tests on ubuntu-latest - #648

Open
AmaadMartin wants to merge 1 commit into
mainfrom
feat/ci-cross-language-ubuntu-runner
Open

ci: run cross-language integration tests on ubuntu-latest#648
AmaadMartin wants to merge 1 commit into
mainfrom
feat/ci-cross-language-ubuntu-runner

Conversation

@AmaadMartin

@AmaadMartin AmaadMartin commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):
    Closes: #issue_number
    Related: #issue_number
  2. Or, if no issue exists, describe the change:

No existing issue; description follows.

Problem: .github/workflows/cross-language-integration.yml was the only workflow in the repository whose sole runner is macos-latest. macOS is the tightest GitHub-hosted runner pool an account has — the Actions limits cap concurrent macOS jobs at 5 (Free/Pro/Team) or 50 (Enterprise) account-wide, against a total concurrent-job budget of 20–500 — so every macOS job queues behind, and delays, every other macOS job in the account. It is also the most expensive runner class (billing docs: macOS $0.062/min vs Linux $0.006/min, ~10.3x), which matters if this repository is ever made private or moved to larger runners. The suite that job runs has no macOS-specific surface, so it was paying that premium for signal ubuntu-latest already produces.

Solution: Flip runs-on to ubuntu-latest and record why the runner is Linux in a three-line comment, so a future contributor does not "restore" macOS coverage by reflex. This is the whole change — 4 insertions, 1 deletion, in one file:

 jobs:
   run-tests:
-    runs-on: macos-latest
+    # This suite is loopback HTTP between a Node process and a Go process and
+    # has no macOS-specific surface; validation.yaml keeps the repository's
+    # macOS coverage via its ubuntu/windows/macos matrix.
+    runs-on: ubuntu-latest

Why this is safe — the suite has no macOS-specific surface. An exhaustive search of core/src, dev/src, integrations/src and tests for process.platform, os.platform() and 'darwin' returns four hits, and every one is a Windows branch:

  • core/src/code_executors/unsafe_local_code_executor.ts:21const IS_WINDOWS = os.platform() === 'win32';
  • dev/src/utils/agent_loader.ts:634process.platform === 'win32' ? 'junction' : 'dir'
  • tests/integration/tools/run_skill_script_tool_test.ts:21-22IS_WINDOWS, plus const IS_UNIX = os.platform() === 'linux' || os.platform() === 'darwin';

That last line is the decisive one: the repository's own test code already treats Linux and macOS as a single behavioural class. There is no code path anywhere that macOS reaches and Linux does not. The tests this workflow runs are loopback HTTP between a Node process and a Go process (go_server.ts binds 127.0.0.1, test_api_server.ts uses localhost), the two Go modules have pure-Go dependency sets with no cgo and no darwin-only packages, and tests/integration/test_case_utils.ts's BaseTestServer contains no platform branching at all.

For the record, the native-addon fixtures are not real native addons and are not run by this workflow (they belong to the integration vitest project, driven by validation.yaml): tests/integration/build_setup/ts_esm_native_addon/.../binding.node is a plain ASCII text file containing placeholder native addon binary, and nothing is ever dlopened on any OS. They already run on validation.yaml's ubuntu-latest leg today.

macOS coverage for the repository is retained. .github/workflows/validation.yaml:17 still runs its full matrix — os: [ubuntu-latest, windows-latest, macos-latest] — and is left byte-for-byte alone by this PR.

Invariants preserved. The workflow name (Cross-Language Tests), the job id (run-tests), the trigger block, and all 7 steps are unchanged, so the GitHub check-run name that branch-protection rules and rulesets match on is preserved and no admin coordination is required. Verified structurally, not by eye — see check 1 below. No source file, test file, package.json, vitest.config.ts or go.mod is touched.

Deliberately out of scope (each queued as its own task): gating validation.yaml's macos-latest leg on the event (that one does remove a check name from PR runs and so needs branch-protection coordination), and adding npm caching / switching npm install to npm ci.

Collision check. Ran gh pr list --repo AmaadMartin/adk-js --state open --limit 1000 (547 open PRs) and diffed every plausibly adjacent one. 16 open PRs touch cross-language-integration.yml (#505, #427, #428, #416, #406, #403, #393, #306, #415, #338, #574, #509, #593, #571, #572, #504) — they variously pin Node, add npm/Go caching, switch to npm ci, pin actions to commit SHAs, add timeout-minutes and add concurrency groups. Grepping each of those diffs for runs-on|macos|ubuntu inside the cross-language-integration.yml hunk returned zero hits: none of them changes the runner, so this change is orthogonal to all of them and does not stack on any one of them.

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.

Note on unit tests / coverage: the deliverable is a GitHub Actions workflow definition, so there is no new line of executable product code to cover. vitest.config.ts's coverage.include globs are core/src/**, dev/src/**, integrations/src/**.github/** is not and cannot be instrumented. No coverage threshold was touched and no unit test was invented for a YAML file. The verification burden is met by the three checks below instead, all run locally on Linux x86_64 (Node v22.22.2, Go 1.26.5).

Check 1 — the workflow still parses and still has the intended shape, and this check is discriminating (it reports the old value against the unchanged file, i.e. it fails before the fix). Run against the pre-change file (git show fork/main:...) and the post-change file:

$ python3 -c "import yaml; d=yaml.safe_load(open(...)); j=d['jobs']['run-tests']; print(j['runs-on']); print(len(j['steps']),'steps')"
BEFORE:  macos-latest   /  7 steps
AFTER:   ubuntu-latest  /  7 steps
         workflow name: Cross-Language Tests
         job ids: ['run-tests']
         triggers: {'push': {'branches': ['main']}, 'pull_request': {'branches': ['main']}}

The step count does not move, and the name/job-id/trigger invariants hold. A stronger structural assertion was also run — load both YAML documents, set the before document's runs-on to ubuntu-latest, and deep-compare:

structurally identical apart from runs-on: True

so the diff provably changes nothing else semantically.

Check 2 — the diff is exactly what it claims to be. (The PR base is the fork's main, i.e. fork/main; the local main ref in this clone is stale.)

$ git diff fork/main --stat
 .github/workflows/cross-language-integration.yml | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

$ git diff fork/main -- . ':!.github/workflows/cross-language-integration.yml'
(empty)

One file, and the only non-comment change is the runs-on value. The go.sum files that go mod tidy generates under tests/cross_language/a2a/*/go*/ are gitignored (.gitignore:11) and are not committed; git status is clean after a full install/build/test cycle.

Check 3 — the real suite passes on Linux, run with exactly the command sequence the workflow uses:

$ npm install
$ (cd tests/cross_language/a2a/go_ts/go_client && go mod tidy)     # exit 0
$ (cd tests/cross_language/a2a/ts_go/go_backend && go mod tidy)    # exit 0
$ npm run build                                                     # ⚡ Done
$ npm run test:cross-language

 ✓ |cross-language| tests/cross_language/a2a/ts_go/ts_a2a_go_test.ts (2 tests) 1673ms
 ✓ |cross-language| tests/cross_language/a2a/go_ts/go_a2a_ts_test.ts (2 tests) 6124ms

 Test Files  2 passed (2)
      Tests  4 passed (4)
   Duration  13.55s

The set of tests executed is identical before and after (the cross-language vitest project, tests/cross_language/**/*_test.ts). Both loopback directions were exercised — the Go server bound http://127.0.0.1:40973 and the TS ADK API server came up on localhost — so the one plausible Linux/macOS difference (whether localhost resolves to ::1 first) is empirically a non-issue here; no ECONNREFUSED ::1 occurred in either direction.

Unit Tests:
[x] I have added or updated unit tests for my change. — N/A and deliberately so: zero new lines of executable code, and .github/** is not instrumentable. Substituted by the mandatory structural checks 1 and 2 above.
[x] All unit tests pass locally.

Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.

This pull request is its own end-to-end test. pull_request workflows execute the workflow file from the PR's merge commit, so the PR verifies the change against real GitHub infrastructure. This has now been observed, not merely predicted — the five acceptance criteria and their measured results:

  1. The Cross-Language Tests / run-tests check ran on this PR and passed.

  2. The runner image changed from macOS to Ubuntu. Read from the Set up job step of each run:

    Workflow run Operating System Image Job wall-clock
    Before 30933159356 (last run prior to this change) macOS 26.5.2 macos-26-arm64 1m48s
    After 30937462257 (this PR) Ubuntu 24.04.4 ubuntu-24.04 1m31s

    That is the primary acceptance evidence. One macos-latest job is now removed from every push-to-main run and every pull_request run, freeing one slot out of the account-wide macOS concurrency cap of 5 (or 50) per run — and the job is not slower for it (1m31s vs 1m48s; sampling three prior macOS runs gives 1m48s / 1m54s / 1m48s).

  3. The suite is green on the Ubuntu runner, with the same test count as before:

     Test Files  2 passed (2)
          Tests  4 passed (4)
       Duration  43.57s
    
  4. The check-run name is unchangedrun-tests under Cross-Language Tests. No previously-reporting check disappeared from this PR's check list.

  5. No macos-latest job remains in this workflow's run, and validation.yaml is unperturbed — this PR's check list still shows all three of its legs: run-tests (ubuntu-latest), run-tests (windows-latest), run-tests (macos-latest).

To reproduce locally instead, run the Check 3 command sequence above on a Linux host with Go >= 1.25, Node, and network access to the npm registry and the Go module proxy.

If this suite ever fails on ubuntu-latest for a genuine platform reason, the right response is a Linux+macOS matrix and a bug report — not an OS-conditional skip, and not reverting to macOS.

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.

The Cross-Language Tests workflow was the only workflow in the repository
whose sole runner was macos-latest. The suite it runs is loopback HTTP
between a Node process and a Go process and has no macOS-specific surface:
the only platform branches in core/src, dev/src, integrations/src and tests
are Windows branches, and the repository's own test code treats Linux and
macOS as a single behavioural class (IS_UNIX in
tests/integration/tools/run_skill_script_tool_test.ts:22).

macOS is the tightest GitHub-hosted runner pool an account has, so the job
was queueing against a scarce shared resource for signal ubuntu-latest
already produces. validation.yaml keeps the repository's macOS coverage via
its ubuntu/windows/macos matrix.

The workflow name (Cross-Language Tests), the job id (run-tests), the
triggers and all 7 steps are unchanged, so the check-run name that
branch-protection rules match on is preserved.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant