Add scratch-utils CI image, ci-base VM builder, and go-build preload disk

[Brian-McM]

Summary

Adds the CI VM tooling the ArgoCI kind-rig needs, as three pieces that share one
source of truth for versions:

• scratch-utils — one static binary (createvm / deletevm / secret /
  runonvm) that drives the GCE VM lifecycle over the GCP APIs directly, with no
  gcloud and no bash, so it runs from a distroless image. Published as
  calico/scratch-utils.
• vm-images/ci-base/ — builds the ci-base GCE image: docker, go, kind, kubectl and
  gh baked in, plus the heavy CI container images pre-pulled, so a CI VM boots
  ready and does zero per-run installs.
• preload-disks/ci-cache/ — builds a GKE secondary-boot-disk image with
  calico/go-build preloaded, so build pods start with it already on the node
  instead of pulling several hundred MB per run.

It's CI VM tooling that belongs alongside go-build, so it's folded into this
module rather than a separate one.

Layout

Both builder trees are plural with the artifact as a subfolder, so a second one
lands as a sibling rather than a rewrite:

vm-images/ci-base/        # the GCE image family it produces
preload-disks/ci-cache/   # what it caches

Each subfolder is named for the axis a sibling would vary on. No shared driver is
factored out yet — with one type in each tree that would be guessing at what the
second needs, and the two trees share nothing anyway (one drives gcloud, the other
drives daisy through a fetched Go tool).

Everything derives from images/calico-go-build/versions.yaml

This is the part worth reviewing closely. A job that runs on one of these VMs
would otherwise have run inside calico/go-build, so it has to see the same Go —
which means the VM's toolchain must track that file, not this module's go.mod.

vm-images/ci-base/build-image.sh resolves the Go version, its checksum and the
calico/go-build tag from it (via the existing hack/generate-version-tag-name.sh,
the same helper the release tagging uses) and injects them into provision.sh,
which runs as the builder VM's startup-script where it cannot read the repo.
preload-disks/ci-cache resolves the same tag for the image it preloads. Both require
those values rather than defaulting them: silently baking a stale Go is the exact
drift this indirection exists to prevent.

Tools with no home in that file — kind, gh, the kind node image — are pinned in
vm-images/ci-base/versions.yaml. Nothing floats on latest or stable.txt, so two
builds of one commit produce the same image.

Image naming

Images are named off the go-build release tag, so a VM image and the toolchain it
was built against are matchable by eye:

go-build image	calico/go-build:1.27.1-llvm21.1.8-k8s1.37.0
ci-base VM image	ci-base-1-27-1-llvm21-1-8-k8s1-37-0
preload disk	cic-1-27-1-llvm21-1-8-k8s1-37-0

hack/generate-image-name.sh builds these. Two constraints shape it: GCE names
are RFC1035 so the dots become hyphens (the exact tag is kept as a go-build-tag
label), and GKE caps a secondary boot disk name at 39 characters — enforced
when a node pool attaches the image, long after it built cleanly — which is why
the preload prefix is terse and why the generator takes -m to fail at build time
instead.

On a tag build the version comes from $SEMAPHORE_GIT_TAG_NAME, matching what
calico-go-build-cd already does; that is the only place the re-release suffix
lives (-1, -2), which generate-version-tag-name.sh cannot know about.
Otherwise it uses the branch.

CI

• Build check for the calico/scratch-utils image on every PR.
• New go test block, run in the golang image at the version versions.yaml
  pins — so tests see the same Go the go-build image ships and the agent needs
  none of its own. This repo ran no Go tests before.
• promotions/calico-scratch-utils.yml — change-gated publish, mirroring the
  other image promotions.
• promotions/ci-base-vm-image.yml and promotions/preload-disk-image.yml —
  chained off the go-build publish, auto-promoted on a release tag, so both
  images rebuild once the go-build image they bake or cache actually exists.
  Gated on the tag rather than merely on that pipeline passing, since it also runs
  for master, where a rebuild would spend a builder VM per merge for a throwaway
  branch image. Both remain promotable by hand for a one-off rebuild.

New dependencies

google.golang.org/api (compute) and golang.org/x/crypto (ssh). Both are for
scratch-utils; nothing else in the module imports them.

Deliberately not added: gke-disk-image-builder. It has a clean library API,
but its go.mod still declares the old GoogleCloudPlatform/ai-on-gke path while
the code lives at ai-on-gke/tools, so go get at the real location fails on a
path mismatch and the declared path resolves only to an abandoned 2023 snapshot.
preload-disks/ci-cache fetches a pinned commit instead, which also keeps compute-daisy
and the cloud.google.com/go stack — about 17 modules — out of this module's
dependency graph.

Testing

• 21 unit tests across 6 files: gofmt, go vet and go test clean on Go 1.27.1.
  Coverage is the pure logic plus the security-relevant path — untar's guard
  against tar entries escaping the destination, which GetDir applies to whatever
  a remote sends. That test was mutation-checked: it fails when the guard is
  removed. shellQuote is verified by round-tripping hostile values through a real
  bash, including inside the export NAME=... line the env file is built from.
• Both images were actually built and used: ci-base-1-27-1-llvm21-1-8-k8s1-37-0
  booted real CI VMs, and the preload disk builder was run end-to-end with only its
  defaults. The 1.27.1 rebuilds follow this merge.
• scratch-utils:v0.3 built, pushed and pulled back to confirm the binary
  dispatches and errors cleanly without credentials.

Notes for review

• Subcommand, not command. The image ENTRYPOINT is the binary, so a pod passes
  args: [createvm] (or command: [scratch-utils, createvm]). A bare
  command: [createvm] overrides the entrypoint and fails to exec.
• max-run-duration is eventual, not guaranteed. It fires on time, but it is
  itself an operation: observed in us-central1-a, an insert stayed RUNNING for over
  90 minutes and the deadline's own DELETE then sat PENDING behind it. Hence the
  per-zone timeout in the zone loop rather than relying on the backstop.
• GOOGLE_VM_IMAGE pins an exact VM image; unset, the ci-base family gives
  the newest. The pin is what lets the image be rolled without rebuilding the
  binary.
• The kindest/node tag is still hand-maintained in vm-images/ci-base/versions.yaml — it
  comes from calico's lib/kind, which this repo has no view of.

---

Authored with AI assistance (Claude Code).

[Copilot]

🟡 Changes recommended

There are a few concrete correctness/documentation issues (SSH PutData stderr capture, tar file descriptor handling, misleading GCE package/docs, and some mismatched/incorrect docs/config defaults) that should be addressed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR introduces cctools (a single, distroless-friendly Go binary for CI GCE VM lifecycle + SSH-based remote execution) and adds vm-image scripts for building a reusable ci-base GCE image with common CI tooling preinstalled, along with CI wiring to build/publish the calico/cctools image.

Changes:

• Add cmd/cctools dispatcher plus cctools/ packages implementing createvm, deletevm, secret, and runonvm over GCP APIs and native SSH.
• Add vm-image/ scripts/docs to provision and snapshot a ci-base GCE image.
• Add images/calico-cctools image + Makefile/Semaphore plumbing for build and master-branch publish.

File summaries

File	Description
vm-image/README.md	Documents the ci-base image and how to build/refresh it.
vm-image/provision.sh	Boot-time provisioning script to bake CI tooling and pre-pulled images into the base VM image.
vm-image/build-image.sh	Script to create a builder VM, wait for provisioning, snapshot to an image family, and clean up.
images/Makefile	Adds build/publish targets for the new calico/cctools image.
images/calico-cctools/versions.yaml	Introduces version pinning for the calico/cctools image tag.
images/calico-cctools/Dockerfile	Defines a distroless image that only contains the cctools binary.
go.sum	Records checksums for newly introduced Go dependencies (GCP + SSH).
go.mod	Adds module dependencies required for cctools (and additional new requirements).
cmd/Makefile	Builds cctools alongside existing cmd binaries.
cmd/cctools/main.go	Subcommand dispatcher for cctools binary.
cctools/subcmd/secret/secret.go	Implements cctools secret for env-var-to-file materialization.
cctools/subcmd/runonvm/runonvm.go	Implements SSH-based upload/run/download workflow for executing scripts on a VM.
cctools/subcmd/deletevm/deletevm.go	Implements best-effort VM deletion by name/zone discovery over compute API.
cctools/subcmd/createvm/createvm.go	Implements VM creation from an image family and writes chosen zone to a file.
cctools/gce/vm.go	Compute API client and VM create/delete/zone-discovery logic.
cctools/gce/ssh.go	SSH key injection + native SSH transport and tar-based put/get helpers.
cctools/ccutil/secret.go	Shared helpers for secret materialization and setting up compute ADC.
.semaphore/semaphore.yml	Adds build block + auto-promotion trigger for the cctools image.
.semaphore/promotions/calico-cctools.yml	Adds a promotion pipeline to publish calico/cctools images on master merges.
.gitignore	Ignores the new images/calico-cctools/bin staging directory.

Review details

• Files reviewed: 18/20 changed files
• Comments generated: 10
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

It introduces confirmed command-injection risks in the SSH helper and an invalid GCE label value that would break gcloud compute images create.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

vm-images/ci-base/build-image.sh:123

• The go-build-tag label value produced here is not a valid GCE label value: it starts with a digit and uses underscores. gcloud compute images create --labels requires label values to match [a-z]([-a-z0-9]*[a-z0-9])? (lowercase, digits, hyphens; must start with a letter), so this will fail at image-create time. Consider prefixing with a letter and normalizing to hyphens.

• Files reviewed: 32/34 changed files
• Comments generated: 8
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The VM image build/provisioning path currently risks producing “successful” images with broken tool installs (non-failing curl downloads and non-enforcing post-provision checks), and SetupComputeADC writes secrets to a predictable temp path.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 32/34 changed files
• Comments generated: 5
• Review effort level: Lite

[hjiawei]

Review of the scratch-utils / ci-base / preload-disk changes. Findings are inline. The one at .semaphore/promotions/calico-go-build.yml:41 is blocking: as written, neither new pipeline can be reached.

[hjiawei]

llvm\d+\.\d\.\d only matches a single-digit LLVM minor and patch. On a bump to llvm 21.1.10 the regex consumes llvm21.1.1 and then needs -k8s1. where the tag has 0-k8s1., so it stops matching.

Nothing errors when that happens: auto_promote just never fires, and the identical run.when gate in ci-base-vm-image.yml:48 / preload-disk-image.yml:54 also goes false, so even a manual promotion gives a green pipeline with the block skipped. The release ships with no ci-base image and no preload disk.

llvm\d+(\.\d+){2} fixes it. Four copies of this pattern now exist — worth a single anchor they all reference.

[Brian-McM]

Not that this isn't true, but I copied this verbatim from our other pipelines so we should probably fix those there too.

[Brian-McM]

I just fixed this for all instances of the regex in this PR

[hjiawei]

Go randomizes map iteration order, so returning on the first non-empty scope picks an arbitrary zone when the same VM name exists in two.

That isn't hypothetical here: PerZoneTimeout (3m) can leave an insert still RUNNING in an earlier zone whose best-effort DELETE is also still PENDING — the 85-minute us-central1-a case the comment at :55 describes — while Create succeeds in the next zone. Instance names are zone-scoped, so GCE happily holds both. deletevm with ZONE unset then deletes one at a coin flip and leaks the other, and runonvm with ZONE unset can SSH into the abandoned VM and run the job on a machine createvm never reported.

Collect every matching zone, then either fail loudly on >1 or act on all of them.

[hjiawei]

This deadlocks. StdoutPipe's contract is that the reader must keep draining or the remote command blocks; on the untar error path the pipe is abandoned with the remote tar czf - . still writing. The SSH window fills, the remote tar blocks on write and never exits, and Wait never returns.

Reachable from the escapes dest guard, ENOSPC locally, or a permission error creating a target file. Because GetDir runs from runonvm's deferred --get loop, the hang happens after the user's script finished and reported its exit code — it looks like the job wedged at 100%.

io.Copy(io.Discard, stdout) or sess.Close() before Wait on the error path.

[hjiawei]

ssh.ClientConfig.Timeout bounds only the TCP connect — ssh.Dial passes it to net.DialTimeout and then runs NewClientConn with no deadline on the conn.

The state this loop exists to ride out is exactly the one that hangs: a freshly-booted VM whose sshd has bound the socket but isn't ready to send its version string accepts the TCP connection and then stalls. ssh.Dial never returns, so neither the deadline check nor the ctx.Done() select below is ever reached, and runonvm's 10-minute setupCtx doesn't fire either. The step hangs until CI kills it.

Dial with net.DialTimeout, conn.SetDeadline, then ssh.NewClientConn — or race each attempt against ctx in a goroutine.

[Brian-McM]

Fixed.

[hjiawei]

These two are the only places the remote paths go into a shell command unquoted — PutData runs the same value through util.ShellQuote one call earlier (ssh.go:187).

remoteScript is path.Join("/tmp", path.Base(script)) from caller-supplied --script or Argo's trailing arg. --script '/work/my step.sh' uploads to /tmp/my step.sh correctly (quoted) and then runs bash /tmp/my step.sh, which the remote shell splits in two: "No such file or directory" after every upload already succeeded, pointing at the wrong thing.

util.ShellQuote on both.

[Brian-McM]

Fixed.

[hjiawei]

Two things here:

= is recursive, so this find re-runs every time make expands the prerequisite list. := runs it once.

Bigger: build: now depends on bin/scratch-utils-$(ARCH), and build is a prerequisite of calico-go-build-image (images/Makefile:67) and calico-binfmt-image (:100). Every arch of the go-build image matrix now downloads and compiles google.golang.org/api, gRPC and the OTel stack — roughly 20 modules those images never contain — before it can start its own Dockerfile.

Give bin/scratch-utils-$(ARCH) its own target that only calico-scratch-utils-image depends on.

[Brian-McM]

Fixed.

[hjiawei]

The comment above says the label keeps the exact tag, and the PR description repeats it, but the sed rewrites every dot: the stored value is 1_27_1-llvm21_1_8-k8s1_37_0.

So gcloud compute images list --filter="labels.go-build-tag=1.27.1-llvm21.1.8-k8s1.37.0" returns nothing, which is the label's only stated purpose — the image name already carries the same munged form.

GCE label values allow dashes, so s|\.|-|g would at least match the name. Either way the comment should say what's actually stored.

[hjiawei]

This same eight-line envOr is also in deletevm.go:67 and runonvm.go:225, and all three files already import scratch-utils/util — whose doc comment says it's the home for exactly this. The empty-vs-unset semantics here are the distinction LocalSecret and --put-env care about, so a change to it now has to land in three places or the subcommands quietly diverge.

Separately, createvm_test.go:51 hand-rolls containsStr; every other test file in the PR just uses strings.Contains.

[Brian-McM]

simplified.

[hjiawei]

This comment (and a few others) explains what the code used to do and why the old bug is gone, rather than the present state — that belongs in the commit message.

Same shape at deletevm.go:32 ("pointing ADC at a nonexistent /secrets path made every delete fail auth..."), createvm.go:63 ("a fixed cap silently starved the later zones..."), build-image.sh:106 ("This used to end in || true..."), and util/secret.go:61 + util/secret_test.go:71 ("The key used to go to a fixed /tmp path").

The forward-looking half of each is the part worth keeping — here, that the timeout is per-zone so a degraded zone can be skipped, and that 3 minutes is chosen because the failure mode is slowness.

[Brian-McM]

I trimmed all the comments.