Repository navigation
CI: give GPU unit-test containers a working NVIDIA Vulkan driver - #813
Merged
Merged
Conversation
test_viz_shutdown (added in openvdb#810) creates a Vulkan device, and on the GPU runners it fails with VkResult -9 (VK_ERROR_INCOMPATIBLE_DRIVER). The runner host's driver includes Vulkan, but the unit-test containers could not load it: - The CUDA images request NVIDIA_DRIVER_CAPABILITIES=compute,utility, so the container toolkit does not mount the Vulkan driver. - libGLX_nvidia links against libXext and opens libEGL at run time, and neither is in the slim CUDA images. Set NVIDIA_DRIVER_CAPABILITIES=all in the three unit-test jobs and install the Vulkan loader, libXext, and libEGL. The conda job also points its conda-provided loader at the mounted driver manifest. Verified by hand on the runner AMI (Ubuntu 24.04, driver 580.65.06, L4): with these settings vulkaninfo inside nvidia/cuda:13.0.2-cudnn-runtime-ubuntu22.04 lists the NVIDIA L4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/unit/test_viz_shutdown.py::test_viewer_shutdown_with_gpu_buffers, added in #810, needs a Vulkan device. On the GPU runners it fails withFailed to create Vulkan instance. VkResult: -9(VK_ERROR_INCOMPATIBLE_DRIVER). #810's unit-test jobs were skipped, so #812 is the first PR to hit it. Every PR based on current main will fail the same way.This PR gives the three GPU unit-test jobs a working NVIDIA Vulkan driver.
Cause
The runner AMI is fine. Its 580.65.06 driver includes the Vulkan driver, and
vulkaninfoon the host lists the L4. The unit-test containers could not load that driver, for two reasons:NVIDIA_DRIVER_CAPABILITIES=compute,utility. The container toolkit therefore does not mount the Vulkan driver (libGLX_nvidiaand/etc/vulkan/icd.d/nvidia_icd.json).libGLX_nvidialinks againstlibXext.so.6.libEGL.so.1at run time.vkCreateInstance, and instance creation fails with -9.Change
The fvdb Unit Tests jobs in
tests.yml(conda),cu130.ymlandcu132.ymlnow set up Vulkan in their containers:NVIDIA_DRIVER_CAPABILITIES: all.libvulkan1 libxext6 libegl1.libXext libglvnd-egl;VK_DRIVER_FILES=/etc/vulkan/icd.d/nvidia_icd.json, in case the conda-provided Vulkan loader doesn't search/etc/vulkan.The gtest and docs jobs are unchanged.
Test plan
nvidia/cuda:13.0.2-cudnn-runtime-ubuntu22.04container has no NVIDIA Vulkan driver files.NVIDIA_DRIVER_CAPABILITIES=allbut withoutlibxext6, the loader skips the driver (libXext.so.6: cannot open).libxext6but withoutlibegl1, the driver loads but returns novkCreateInstance.libvulkan1 libxext6 libegl1,vulkaninfo --summarylistsNVIDIA L4.pull_request_target, which usesmain's workflow definitions. This PR's own unit-test jobs will therefore still fail on the viewer test. The fix takes effect once it merges; the next PR, or a rerun of Support empty JaggedTensor structures and validate raw constructors #812, should then passtest_viz_shutdown.VK_DRIVER_FILESsetting is untested. If that job still fails after merge, its loader output will show whether the variable is needed or not enough.🤖 Generated with Claude Code