Skip to content

Support Data Center precompiled driver container for Arm (Ubuntu 26.04) - #1010

Open
abrarshivani wants to merge 7 commits into
NVIDIA:mainfrom
abrarshivani:issue-275-precompiled-arm-ubuntu26.04
Open

Support Data Center precompiled driver container for Arm (Ubuntu 26.04)#1010
abrarshivani wants to merge 7 commits into
NVIDIA:mainfrom
abrarshivani:issue-275-precompiled-arm-ubuntu26.04

Conversation

@abrarshivani

@abrarshivani abrarshivani commented Sep 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Ubuntu 26.04 arm64 was the last precompiled target still pinned to amd64. #533 did the hard part by building the arm64 machinery, and since that machinery is distribution agnostic, enabling 26.04 is mostly a matter of removing pins.

Rebased on main now that #873 has landed. That change fixes the skopeo copy --all problem, so the commit I had for it here is dropped.

While enabling 26.04, a few other issues surfaced. They affect 24.04 as well, so I have included fixes here, but I am happy to split them into a separate PR if you would rather review them on their own.

What this enables

  • 26.04 builds amd64 and arm64 for every flavor except azure-fde
  • 26.04 is added to the PR workflow matrix, so the image is exercised per PR rather than only overnight
  • adds the 26.04 arm64 CVE scan job
  • uses the sbsa CUDA repository on arm64
  • builds the kernel discovery base image for amd64 only, since the kernel version is the same on both architectures

Issues found along the way

  • Existing tags were never upgraded. The push was skipped whenever the destination tag already existed, without checking which architectures it contained, so once an amd64-only tag was there arm64 could never be published. Every later run saw the tag, skipped, and reported success.
  • My first attempt at that check was wrong. regctl manifest get --platform only selects within a manifest list; against a plain manifest it succeeds for any platform. Comparing what regctl image config reports works instead.
  • The check only covered multi-arch targets. For 22.04 and azure-fde it had no platforms to loop over, so it fell back to asking whether the tag existed at all, which is the behaviour it was meant to replace. It now names amd64 for those too.
  • GitLab's emulator was too old for 26.04. image.yaml already selects per distribution, since the newer QEMU that runs resolute breaks the jammy glibc, so .common-ci.yml now matches that approach.

A note on packages

#533 skips linux-signatures-nvidia and libnvidia-fbc1 on arm64 for 24.04. That does not appear to hold for resolute:

  • both are in the archive as arm64 binaries, with the same per-flavor coverage as amd64
  • both install in an arm64 container
  • CI downloaded both as arm64 debs

I have left them in, since skipping them would ship an arm64 image without Canonical's secure boot signatures. Happy to be corrected if there is context I am missing here. azure-fde is genuinely amd64 only and stays excluded.

Testing

  • CI was green across 73 jobs on the previous head, including six new 26.04 jobs. The logs showed --platform=linux/amd64,linux/arm64 and arm64 debs being fetched.
  • A PR build never reaches the publish path, so I tested the registry check against a local registry, against seeded amd64-only, arm64-only, complete, and missing tags, for each dist and flavor.
  • Built the image natively for arm64 and pushed it to a personal registry, to confirm the packages resolve outside CI.
  • The GPU e2e already covers driver load, and the matrix now includes 26.04 arm64 on g5g.xlarge. It has not run yet only because that workflow is nightly.

Not included

  • 64k page size kernel flavors on arm64, which have no amd64 equivalent and are not tested today
  • the azure-fde row in the arm64 CVE scan, which cannot resolve for arm64; this matches 24.04's current behaviour

Both are pre-existing, so I have left them for separate changes. Happy to pick either up here if you would prefer.

@abrarshivani abrarshivani self-assigned this Sep 15, 2026
@abrarshivani
abrarshivani marked this pull request as ready for review September 15, 2026 00:16
@abrarshivani
Support Data Center precompiled driver container for Arm (Ubuntu 26.04)
007dd4e 
PR NVIDIA#533 added arm64 support to the precompiled pipeline but enabled it
only for ubuntu24.04. The shared plumbing it introduced (per-platform
kernel matrix, OCI archive artifacts, skopeo publish, arm64 Holodeck
instance) is distribution agnostic, so enabling ubuntu26.04 is mostly a
matter of removing the amd64 pins.

The precompiled workflow pinned binfmt to qemu-v6.2.0 for every
distribution. That emulator hangs on the 26.04 arm64 userland, so it now
uses the same qemu-v10.1.3 pin that image.yaml already applies to 26.04.

Unlike ubuntu24.04, no package guards are needed. linux-signatures-nvidia
and libnvidia-fbc1-<branch>-server are published for arm64 on resolute for
every kernel flavor in the build matrix, and the CUDA sbsa repo carries
nvidia-fabricmanager, libnvidia-nscq, nvidia-imex and nvlsm. Only
azure-fde is amd64 only, and build-kernel-matrix.sh already drops it from
the arm64 flavor list.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
@abrarshivani
Build ubuntu26.04 precompiled images in the PR workflow
8dbfe56 
The pre-compiled job in image.yaml covered ubuntu22.04 and ubuntu24.04
only, so nothing exercised the 26.04 image until the nightly precompiled
workflow ran. Adding it here builds the 26.04 image on every push to a
pull-request branch.

26.04 needs the 7.0 kernel line, which the lts_kernel axis did not carry.
The excludes mirror exclude_precompiled_build_matrix in
precompiled-matrix-config.json so the two workflows agree on which
dist, kernel and driver combinations are valid.

The base image step gained the resolute target. Without it BASE_TARGET is
empty for 26.04 and the build-base- target does not exist.

This takes the job count from 32 to 38. The six new jobs are driver 595
on kernel 7.0, one per flavor; the five non azure-fde ones build arm64.
The ispr flavor reduction does not apply to them, because the workflow
has no pull_request trigger and so github.event_name is always push.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
@abrarshivani
Register a newer QEMU for ubuntu26.04 on GitLab
39f6650 
BUILD_MULTI_ARCH_IMAGES is true for every GitLab job, so enabling arm64
for signed_ubuntu26.04 makes image-precompiled-ubuntu26.04 build it
under emulation. That job sets binfmt up through .buildx-setup, which
registers multiarch/qemu-user-static untagged, and that image tracks a
QEMU too old to run the 26.04 arm64 userland.

image.yaml hit the same problem and settled it by selecting the emulator
per distribution, because the newer QEMU that runs resolute segfaults in
libc-bin on jammy. This does the same rather than moving every job to
the newer image: ubuntu26.04 gets tonistiigi/binfmt:qemu-v10.1.3 and
everything else keeps the emulator it builds with today.

tonistiigi/binfmt has no equivalent of --reset and --install leaves an
already registered handler alone, so the handlers are uninstalled first
to make sure a runner that has emulated arm64 before picks up the
pinned version.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
@abrarshivani
Build the kernel discovery base image for amd64 only
08e916f 
The base image exists so the workflow can read kernel_version.txt out of
it, and that version is the same on both architectures. image.yaml built
it for every platform in DOCKER_BUILD_PLATFORM_OPTIONS, which meant an
arm64 copy was produced under emulation and then only ever used to read
one file. Adding 26.04 makes that emulated build slower still.

precompiled.yaml already pins the base build to amd64 for this reason.
It also publishes the base image from a single architecture
docker-archive, so the two paths disagreed about whether the base tag
was a manifest list; pinning here makes them consistent.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
@abrarshivani
Promote a precompiled tag that is missing an architecture
b1c4cf9 
pushImage skipped the push whenever the destination tag already existed,
and imageExists only asked regctl whether the tag resolved at all.
regctl manifest get --list succeeds for any manifest list, no matter
which platforms it contains, and FORCE_PUSH is not set anywhere in this
repository, so the skip always won.

That is harmless while a distribution is amd64 only, but ubuntu24.04 and
now ubuntu26.04 build both architectures. Once an amd64-only tag existed
at the destination, the arm64 instance could never be published: every
later run saw the tag, skipped, and reported success.

The check now probes each platform the build targets with regctl
manifest get --platform, which fails when the manifest list has no entry
for it, and pushes when any of them is missing. push runs as a separate
invocation from build, so the platform set has to be recomputed rather
than carried over; both paths now read it from targetPlatforms so they
cannot disagree about which distributions and kernel flavors are
multi-arch.

Distributions that stay single-arch keep the old behaviour, since
targetPlatforms is empty for them and only the tag existence check runs.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
@abrarshivani
Compare the image config when checking for a platform
6007f78 
The platform check used regctl manifest get --platform, which only
selects an entry inside a manifest list. Against a plain manifest it
succeeds whatever platform is asked for: an arm64-only tag answers yes
to linux/amd64, and to linux/ppc64le for that matter.

The tags this check exists for are plain manifests. skopeo copy without
--all writes a single manifest, so an amd64-only tag left behind by an
earlier run is exactly the case that slipped through, and the check went
on skipping the push it was added to allow.

regctl image config resolves the platform and reports the architecture
it actually found, so comparing that against the requested platform
distinguishes the two.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
@abrarshivani
Check the architecture of single architecture targets too
4975567 
targetPlatforms only named platforms for the multi-arch combinations and
was empty for ubuntu22.04 and for azure-fde. The registry check loops
over what it returns, so for those the loop had nothing to iterate and
the check collapsed back to asking whether the tag resolved at all.

That is the behaviour the check was added to replace. A ubuntu22.04 tag
carrying the wrong architecture would satisfy it and the push would be
skipped, which is the same silent success the multi-arch case no longer
has. Naming amd64 for those configurations makes the name accurate and
gives every build the same guarantee.

The platform set is unchanged for every configuration: the Makefile
already resolved these to linux/amd64 on its own, so exporting it
explicitly builds exactly what it built before.

Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
@abrarshivani
abrarshivani force-pushed the issue-275-precompiled-arm-ubuntu26.04 branch from f833a8a to 4975567 Compare September 16, 2026 02:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@abrarshivani abrarshivani

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@abrarshivani