-
Notifications
You must be signed in to change notification settings - Fork 81
Add scratch-utils CI image, ci-base VM builder, and go-build preload disk #908
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
hjiawei
merged 34 commits into
projectcalico:master
from
Brian-McM:bm-add-cctools-and-ci-base
Sep 12, 2026
+3,079
−9
Merged
Changes from 26 commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
d90f9d0
Add cctools CI-tools image + ci-base VM builder
Brian-McM 62b5712
Rename cctools -> scratch-utils; address review feedback
Brian-McM 50457c0
vm-image: source Go + go-build versions from calico-go-build/versions…
Brian-McM e7601d0
Add VM-image promotions, tests, and the remaining review fixes
Brian-McM cf65a97
preload-disk: pin the gke-disk-image-builder commit
Brian-McM ace7e97
Name the VM and disk images off the go-build release tag
Brian-McM 02c2835
Name non-tag image builds after the branch, as go-build does
Brian-McM 7c32cf7
preload-disk: correct the branch-image replacement warning
Brian-McM 7a59bfb
Tighten comments across scratch-utils, vm-image and preload-disk
Brian-McM 6e9da91
createvm: pin an exact VM image; preload-disk: make the network confi…
Brian-McM befd125
Document that the subcommand is an argument, not the command
Brian-McM 05ec192
createvm: give each zone its own deadline and report every zone's error
Brian-McM bbf168f
scratch-utils: bump image to v0.3
Brian-McM a373172
gce: correct the max-run-duration backstop claim
Brian-McM 8f7b1ef
preload-disk: correct the cross-project claim
Brian-McM 4fb9055
preload-disk: fit the GKE 39-character secondary boot disk name cap
Brian-McM a1ff5f6
preload-disk: build in tigera-cc-dev, where the clusters are
Brian-McM 1412cd9
gce: retry transient control-plane errors, keeping the call in the me…
Brian-McM 95ab59b
Merge remote-tracking branch 'upstream/master' into bm-add-cctools-an…
Brian-McM 40999e1
preload-disk: drop the next-steps log block
Brian-McM fd4234d
Make the builder trees plural, with the artifact as a subfolder
Brian-McM 217acd5
Tighten comments again: fewer, and shorter
Brian-McM b1bc61e
ci-base: pre-pull a list of kind node images, not just one
Brian-McM 2981a0c
preload-disks: name it ci-cache, not go-build
Brian-McM 3d1b7ea
Drop the vm-images and preload-disks READMEs
Brian-McM 153649c
Quote remote paths for the shell; validate env names; reject bad disk…
Brian-McM 4387ea8
Fail the image build on a bad download; harden ADC and not-found dete…
Brian-McM 4ea9dd6
scratch-utils: build on scratch rather than distroless
Brian-McM 5ebfe3c
scratch-utils: release by tag, publish moving images from master
Brian-McM 8289024
Chain the VM and preload-disk builds off the go-build publish
Brian-McM cc112cd
Gate the VM and preload-disk blocks like calico-go-build's
Brian-McM e2f55b3
Address review feedback on scratch-utils, ci-base and ci-cache
Brian-McM d8d5d81
gce: never hand a dying VM to a caller that wants to run on one
Brian-McM e706f7c
gce: treat STOPPED as going away, like TERMINATED
Brian-McM File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,3 +2,4 @@ | |
| cmd/bin | ||
| images/calico-binfmt/bin | ||
| images/calico-go-build/bin | ||
| images/calico-scratch-utils/bin | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| version: v1.0 | ||
| name: Publish calico/scratch-utils images | ||
| agent: | ||
| machine: | ||
| type: f1-standard-2 | ||
| os_image: ubuntu2204 | ||
|
|
||
| execution_time_limit: | ||
| minutes: 30 | ||
|
|
||
| global_job_config: | ||
| env_vars: | ||
| - name: DEV_REGISTRIES | ||
| value: calico | ||
| secrets: | ||
| - name: docker | ||
| prologue: | ||
| commands: | ||
| - echo $DOCKER_TOKEN | docker login --username "$DOCKER_USER" --password-stdin | ||
| - checkout | ||
|
|
||
| blocks: | ||
| - name: Publish calico/scratch-utils amd64 images | ||
| dependencies: [] | ||
| run: | ||
| when: "branch = 'master'" | ||
| task: | ||
| jobs: | ||
| - name: Linux amd64 | ||
| commands: | ||
| - if [ -z "${SEMAPHORE_GIT_PR_NUMBER}" ]; then make -C images calico-scratch-utils-cd CONFIRM=true; fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| version: v1.0 | ||
| name: Build ci-base VM image | ||
| agent: | ||
| machine: | ||
| type: f1-standard-2 | ||
| os_image: ubuntu2204 | ||
|
|
||
| execution_time_limit: | ||
| minutes: 30 | ||
|
|
||
| # Manual-only (no auto_promote in semaphore.yml): trigger from the Semaphore UI to | ||
| # (re)build the ci-base GCE VM image. The work runs on a throwaway builder VM (see | ||
| # vm-images/ci-base/build-image.sh); this agent just drives gcloud. The image lives in a | ||
| # family, so createvm always gets the newest. | ||
| # | ||
| # The baked toolchain is pinned: Go and calico/go-build from | ||
| # images/calico-go-build/versions.yaml, kind/gh from vm-images/ci-base/versions.yaml. Re-run | ||
| # after bumping either. | ||
| global_job_config: | ||
| env_vars: | ||
| - name: PROJECT | ||
| value: unique-caldron-775 | ||
| - name: ZONE | ||
| value: us-central1-a | ||
| - name: FAMILY | ||
| value: ci-base | ||
| # Same secret the projectcalico/calico GCE jobs use. Needs compute instance + | ||
| # image create/delete in PROJECT. | ||
| secrets: | ||
| - name: google-service-account-for-gce | ||
| prologue: | ||
| commands: | ||
| - export GOOGLE_APPLICATION_CREDENTIALS=$HOME/secrets/secret.google-service-account-key.json | ||
| - gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS | ||
| - gcloud config set project $PROJECT | ||
| - checkout | ||
| # build-image.sh reads the versions.yaml files with yq. | ||
| - which yq || (sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 && sudo chmod +x /usr/local/bin/yq) | ||
|
|
||
| blocks: | ||
| - name: Build ci-base VM image | ||
| dependencies: [] | ||
| task: | ||
| jobs: | ||
| - name: build-image | ||
| commands: | ||
| - bash vm-images/ci-base/build-image.sh |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| version: v1.0 | ||
| name: Build go-build preload disk image | ||
| agent: | ||
| machine: | ||
| type: f1-standard-2 | ||
| os_image: ubuntu2204 | ||
|
|
||
| execution_time_limit: | ||
| minutes: 30 | ||
|
|
||
| # Manual-only (no auto_promote in semaphore.yml): trigger from the Semaphore UI to | ||
| # (re)build the GKE secondary-boot-disk image preloading calico/go-build. The work | ||
| # runs on a throwaway builder VM via Google's gke-disk-image-builder (see | ||
| # preload-disks/ci-cache/build-preload-disk.sh); this agent drives gcloud + `go run`. | ||
| global_job_config: | ||
| env_vars: | ||
| # The project holding the GKE clusters that consume the disk. Building it here | ||
| # keeps the image with its consumer and needs no cross-project IAM; a pool CAN | ||
| # reference an image elsewhere, but then both the cluster's compute SA and its | ||
| # GKE service agent need compute.imageUser -- see the build script's header. | ||
| - name: PROJECT | ||
| value: tigera-cc-dev | ||
| - name: ZONE | ||
| value: us-central1-a | ||
| # Daisy scratch/staging bucket the builder requires (uploads its workflow | ||
| # sources + streams the builder VM's serial log here); the SA needs write access. | ||
| # Must live in PROJECT: the builder VM writes its serial log, so a bucket | ||
| # elsewhere reintroduces the cross-project IAM that building here avoids. | ||
| - name: GCS_PATH | ||
| value: gs://gke-argo-disk-images-cc-dev | ||
| # Same secret the projectcalico/calico GCE jobs use; also feeds ADC, which | ||
| # gke-disk-image-builder reads. Needs compute instance/disk/image create+delete | ||
| # and log-bucket write in PROJECT. | ||
| secrets: | ||
| - name: google-service-account-for-gce | ||
| prologue: | ||
| commands: | ||
| - export GOOGLE_APPLICATION_CREDENTIALS=$HOME/secrets/secret.google-service-account-key.json | ||
| - gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS | ||
| - gcloud config set project $PROJECT | ||
| # gke-disk-image-builder is `go run ./cli`; pick the Go the tool's go.mod needs. | ||
| - sem-version go 1.24 | ||
| - checkout | ||
| # build-preload-disk.sh resolves the go-build tag from versions.yaml with yq. | ||
| - which yq || (sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 && sudo chmod +x /usr/local/bin/yq) | ||
|
|
||
| blocks: | ||
| - name: Build preload disk image | ||
| dependencies: [] | ||
| task: | ||
| jobs: | ||
| - name: build-preload-disk | ||
| commands: | ||
| - bash preload-disks/ci-cache/build-preload-disk.sh |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| // Copyright (c) 2026 Tigera, Inc. All rights reserved. | ||
|
|
||
| // Command scratch-utils bundles the CI VM helpers into one binary dispatched by | ||
| // subcommand -- one image, different args, not four binaries: | ||
| // | ||
| // scratch-utils createvm create the CI GCE VM (config from env) | ||
| // scratch-utils deletevm delete it by name, best-effort | ||
| // scratch-utils secret <ENV_VAR> <DEST_PATH> env var -> file | ||
| // scratch-utils runonvm [flags] <script> run a script on the VM over SSH | ||
| // | ||
| // No gcloud or bash dependency, so it runs from a distroless image. The ENTRYPOINT | ||
| // is this binary, so a pod passes the subcommand as an ARGUMENT -- `args: | ||
| // [createvm]`; a bare `command: [createvm]` replaces the entrypoint and fails. | ||
| package main | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "os" | ||
|
|
||
| "github.com/projectcalico/go-build/scratch-utils/subcmd/createvm" | ||
| "github.com/projectcalico/go-build/scratch-utils/subcmd/deletevm" | ||
| "github.com/projectcalico/go-build/scratch-utils/subcmd/runonvm" | ||
| "github.com/projectcalico/go-build/scratch-utils/subcmd/secret" | ||
| ) | ||
|
|
||
| func main() { | ||
| if len(os.Args) < 2 { | ||
| usage() | ||
| os.Exit(2) | ||
| } | ||
| sub := os.Args[1] | ||
| // Each subcommand then sees its own args as os.Args[1:]. | ||
| os.Args = append([]string{os.Args[0] + " " + sub}, os.Args[2:]...) | ||
|
|
||
| switch sub { | ||
| case "createvm": | ||
| os.Exit(createvm.Run()) | ||
| case "deletevm": | ||
| os.Exit(deletevm.Run()) | ||
| case "secret": | ||
| os.Exit(secret.Run()) | ||
| case "runonvm": | ||
| os.Exit(runonvm.Run()) | ||
| default: | ||
| fmt.Fprintf(os.Stderr, "scratch-utils: unknown subcommand %q\n", sub) | ||
| usage() | ||
| os.Exit(2) | ||
| } | ||
| } | ||
|
|
||
| func usage() { | ||
| fmt.Fprintln(os.Stderr, "usage: scratch-utils <createvm|deletevm|secret|runonvm> [args]") | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two things here:
=is recursive, so thisfindre-runs every time make expands the prerequisite list.:=runs it once.Bigger:
build:now depends onbin/scratch-utils-$(ARCH), andbuildis a prerequisite ofcalico-go-build-image(images/Makefile:67) andcalico-binfmt-image(:100). Every arch of the go-build image matrix now downloads and compilesgoogle.golang.org/api, gRPC and the OTel stack — roughly 20 modules those images never contain — before it can start its own Dockerfile.Give
bin/scratch-utils-$(ARCH)its own target that onlycalico-scratch-utils-imagedepends on.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.