Add LLVM 23, drop LLVM 20, and stop emitting broken device debug info - #1419
Open
pvelesko wants to merge 25 commits into
Open
Add LLVM 23, drop LLVM 20, and stop emitting broken device debug info#1419pvelesko wants to merge 25 commits into
pvelesko wants to merge 25 commits into
Conversation
Both SPIR-V producers reachable on LLVM 21 and 22 emit debug info that spirv-val rejects. The SPIRV-LLVM-Translator encodes DebugTypeComposite with a Parent operand, which forms a genuinely cyclic forward reference, and the LLVM 22 SPIR-V backend emits DebugTypePointer with a DebugInfoNone base type. IGC then reports an internal error on Arc. 0002-preserve-device-debug-info was the only thing removing the DebugInfoKind = NoDebugInfo guard from HIPSPVToolChain::adjustDebugInfoKind, so dropping the patch restores stock clang's behaviour: no DICompileUnit is generated at all and nothing downstream can orphan debug metadata. Its other hunk (SPV_KHR_non_semantic_info / SPV_INTEL_optnone / nonsemantic-shader-200 under -g) is dead once debug info is off. Device debug info is delivered on LLVM 23 instead, where the in-tree SPIR-V backend avoids the cycle structurally. Refs: #1004
LLVM 23 is pinned to the upstream tag llvmorg-23.1.0-rc2 plus two patches; all four patches were verified to apply with git apply against a pristine rc2 checkout and llvm_release_230, and the patched HIPSPV.cpp and SPIRVSubtarget.cpp compile. 0001 backports llvm/llvm-project#213052 (merged 2026-07-31 as 7ef0ca2b13f9, a reland of #206910), which landed after release/23.x was cut. It makes the HIPSPV toolchain emit device SPIR-V with the in-tree SPIR-V backend by default and fall back to llvm-spirv under -fno-integrated-objemitter or when the SPIR-V target was not built. The upstream commit sits on a main that already had the extension list and the debug-info translator flags; release/23.x has neither, so both are folded in and a patched 23 matches main exactly. 0002 honors -g for device code, but only when the in-tree backend is the effective emitter. The translator encodes DebugTypeComposite with a Parent operand, producing a cyclic forward reference that spirv-val rejects and IGC mis-handles, so -g must keep being stripped there. The remaining llvm-21/llvm-22 patches are unnecessary here: the SPIR-V version and extension selection, static device library unbundling, the data layout and the macOS Mach-O support are all already in 23. Refs: #1004
23 maps to the upstream tag llvmorg-23.1.0-rc2 (not release/23.x, which moves) with the translator branch llvm_release_230 and the patches in llvm-patches/llvm-23/. LLVM 23 removed the LLVM_ENABLE_PROJECTS=openmp build mode, so openmp moves to LLVM_ENABLE_RUNTIMES there. This has to stay conditional: 21 and 22 still expect openmp as a project, and CI hard-fails when omp.h is missing from the resulting install.
The supported set becomes 21, 22, and 23. Removes llvm-patches/llvm-20/, the 20 option in configure_llvm.sh, and raises the LLVMCheck.cmake floor to 21.0.0. Documentation examples that named LLVM 20 are moved to 22. llvm-patches/README.md kept describing llvm-20 as the reference set every other version was trimmed down from, so llvm-21 takes that role and its table is spelled out in full.
Profiling.md claimed the preserve-device-debug-info patch is applied for every supported LLVM. It is now applied only on 23, and there only when the in-tree SPIR-V backend is the effective emitter.
…module LLVM 22 has no in-tree HIPSPV backend support, so its two producers need two separate toolchains and --variant picks between the llvm/22.0-translator and llvm/22.0-native modules. From LLVM 23 on chipStar patches in llvm#213052, so one toolchain built with the SPIRV target serves both: the translator is built either way and the producer is chosen per compilation. There is a single llvm/23.0 module and --variant becomes -DCHIP_LLVM_USE_INTERGRATED_SPIRV instead of a different module name. Also drop llvm-20 from the version error message.
Adds an LLVM 23 row to all three matrices in test-llvm-patches.yml and removes the LLVM 20 rows. LLVM 23 needs only one build. chipStar patches in llvm#213052, so a toolchain built with the SPIRV target also carries the translator and the producer is selected per compilation by -f[no-]integrated-objemitter. Hence a single module-name of 23.0 rather than the -native/-translator pair LLVM 22 needs. The omp.h guard in step2 resolves lib/clang/23/include/omp.h, which the LLVM_ENABLE_RUNTIMES=openmp build installs.
configure_llvm.sh no longer accepts --version 20, so presubmit.yml's matrix still asking for it fails at configure time. Move its three pins to 21, the new oldest supported version. Also drop the unit-tests-llvm-20-debug and unit-tests-llvm-20-release jobs from the x86 workflow. They are workflow_dispatch only so they did not fail on a PR, but they call unit_tests.sh with llvm-20 and nothing depends on them.
configure_llvm.sh builds LLVM with LLVM_LINK_LLVM_DYLIB=ON and CMAKE_INSTALL_RPATH pinned to the final install prefix. The staged (DESTDIR) binaries are therefore thin drivers whose RUNPATH points at $HOME/install/llvm/<ver>/lib, so a staged clang loads libclang-cpp and libLLVM from the previously installed toolchain. Since essentially all driver logic lives in those libraries, the "Build LLVM N and test chipStar" lanes were exercising the installed compiler while reporting on the newly patched one. That makes the workflow structurally unable to validate a change to llvm-patches/, which is the only reason it exists. It also explains why these lanes went red without any corresponding change to their inputs: the installed toolchains changed underneath them. Verified on meatloaf with the staged LLVM 21 from this branch. The same clang binary, compiling a HIP device TU with -g: default (installed libs): -debug-info-kind=constructor emitted staged libs pinned: no -debug-info-kind emitted Pin the loader to the stage and persist it to the test steps, and fail the Linux lane loudly if the pin does not take effect rather than silently validating the wrong toolchain again. The macOS side gets the same pin as a diagnostic, since its linkage layout is not verified here.
chipStar targets spirv64v1.2, and on LLVM 23 the integrated SPIR-V
backend is the default producer. The backend refuses to emit the
GroupNonUniform* capabilities below SPIR-V 1.3, so any warp-level
primitive kills the whole compilation:
GroupNonUniformShuffle(65) requires SPIR-V version 1.3 or later
fatal error: error in backend: Unable to meet SPIR-V requirements
for this target.
This is the same deviation the SPIRV-LLVM-Translator has carried for
years via spirv-translator/0001-pretend-subgroup-caps-are-spirv-1.2.patch;
the backend simply never had an equivalent, which is why LLVM 23 was the
first configuration to hit it.
Reproduced in isolation on LLVM main, no chipStar involved:
spirv64v1.2 + sub_group_shuffle -> Unable to meet SPIR-V requirements
spirv64v1.3 + sub_group_shuffle -> ok
The backend gates these capabilities in two independent places and both
must be relaxed; relaxing only the .td leaves them unavailable and fails
identically. The whole GroupNonUniform family is relaxed because chipStar
emits shuffle, shuffle-relative, ballot, arithmetic min/max and quad ops,
and all of them also require the parent GroupNonUniform capability, which
is itself gated at 1.3.
Commit 0947b0e fixed two occurrences but missed line 136. LLVM 23 removed Constant::isZeroValue(); use isNullValue() which is equivalent. (cherry picked from commit a127229e5ba19aca93f3bdf4086a0462df926d40)
The constant_fold_lgamma_r test targets amdgcn-amd-amdhsa and requires ROCm device libraries. It is not applicable to SPIR-V builds and always fails with "cannot find ROCm device library". Exclude it via CTestCustom.cmake. (cherry picked from commit 8bfd2ca56b3bdfea40dbbf87bffd6f4f40a5c2c6)
The promoteInt validity check converted the post-link bitcode to SPIR-V with llvm-spirv, which isn't built when LLVM uses the in-tree SPIR-V backend (native build). Fall back to 'clang --target=spirv64*-unknown- chipstar -x ir' when llvm-spirv is unavailable, and pass CLANG into the test environment from CMake. (cherry picked from commit 78545a46e028cf235fe5ff1f077d9ffc91e4ac34)
The hip_sycl_interop and hip_sycl_interop_no_buffers samples link their executables with -fsycl using chipStar's clang++ (not Intel's icpx). Starting with LLVM 23 the upstream -fsycl driver makes clang-linker-wrapper inject a dependency on libLLVMSYCL.so, which is only produced by an LLVM build that enables the SYCL project. chipStar builds LLVM with LLVM_ENABLE_PROJECTS="clang;clang-tools-extra" (no SYCL), so the library does not exist and the link fails with: /usr/bin/ld: cannot find libLLVMSYCL.so: No such file or directory Only build these two -fsycl-linked samples on LLVM < 23. The sycl_hip_interop sample (a static library, no -fsycl link) is unaffected and keeps building. (cherry picked from commit c28c1ffce9bca79c28d3b53db3b4fd970f728123)
LLVM 23's HIPSPV toolchain now defaults to the in-tree SPIR-V backend when built, so translator mode must request llvm-spirv explicitly. (cherry picked from commit 9c740105cc6ac1a13f5c7b506d170b661e3dc4e4)
CHIPUninitializeCallOnce accessed PerThreadDefaultQueue (a thread_local unique_ptr) from an atexit handler. TLS destruction order relative to atexit is implementation-defined; with clang 23 the unique_ptr is already destroyed, causing a SEGFAULT on the dangling pointer. Remove the per-thread queue sync since TLS destruction handles cleanup. (cherry picked from commit 74aed1ed7363630368295da14337fd4e5642f8e3)
The early-out in HipPrintfToOpenCLPrintfPass treated *any* module with a
single printf use as already-lowered ("only used by _cl_printf"). That is
only true once the pass has created its _cl_print_str helper. A module whose
sole printf call is a genuine, not-yet-lowered call -- e.g. the device-side
__assert_fail printf from spirv_hip.hh, which is the only printf in a kernel
that just uses assert() -- also has exactly one use, so the pass skipped it.
Skipping leaves that printf's format string in a non-constant address space.
The SPIR-V translator must then emit SPV_EXT_relaxed_printf_string_address_space,
which the runtime consumer (IGC on Intel GPUs, the llvm-spirv reverse path on
macOS/pocl) rejects with "InvalidModule ... disabled by --spirv-ext option".
The kernel module fails to load and the assert test hangs (200s timeout).
Only short-circuit when the _cl_print_str helper already exists. With the
helper absent, fall through and lower the printf so its format string ends up
in the constant address space and the extension is no longer required.
Fixes Unit_Assert_Positive_Basic_KernelPass_AssertionTest timing out on
LLVM 21 / 22-native (Linux, Intel Arc B570) and LLVM 22 native/translator
(macOS). Verified end-to-end on a B570: the test now passes and device-side
assert prints its message and aborts correctly.
(cherry picked from commit d043503846518a53727aee182787d1dab2144d9e)
Commit 3724d3c dropped 0002-preserve-device-debug-info.patch from llvm-21 and llvm-22 because the in-tree SPIR-V backend emits DebugTypePointer with a DebugInfoNone base type, which spirv-val rejects and IGC mis-handles. The patch was kept for llvm-23 on the assumption that the LLVM 23 backend had fixed this. It has not: spirv-val --target-env spv1.2 main_di.spv error: NonSemantic.Shader.DebugInfo.100 DebugTypePointer: expected operand Base Type is not a valid debug type %105 = OpExtInst %void %85 DebugTypePointer %104 %uint_5 %uint_0 %104 = OpExtInst %void %85 DebugInfoNone Because CMakeLists.txt:38 appends -gdwarf-4 unconditionally and CI configures CMAKE_BUILD_TYPE=Debug, every device module on LLVM 23 carried this invalid debug info. It only surfaced on the rtdevlib path because that is the sole path using clCompileProgram + clLinkProgram; everything else takes the clBuildProgram branch, which tolerates it. Hence: Device library link step failed (CHIPBackendOpenCL.cc:1247) clLinkProgram failed: -17 CL_LINK_PROGRAM_FAILURE (Intel GPU) clLinkProgram failed: -44 CL_INVALID_PROGRAM (Intel CPU) taking down every test that links rtdevlib: ballot, atomics, printf, TestSnakeMiscompileO2 and TestHeCBenchLebesgue. The failures are not subgroup-related; the CPU device reports hasWarpBallot: 0 and so never links ballot_native, yet the link still fails. Documented in issue #1420.
This reverts commit edca004.
The SPIR-V translator, which is the producer chipStar uses when the user
passes -g, emits debug information that is not valid SPIR-V: a
DebugTypeComposite member references a DebugTypePointer whose base type
is that very composite, emitted as a forward reference. Only IGC on
Intel Data Center GPU Max tolerates it. spirv-opt is not a usable repair
either, it fails to parse these modules ("Id is 0" for the
OpenCL.DebugInfo.100 form) and crashes on the NonSemantic form.
Add a raw-word filter over the instruction stream that drops OpExtInst
instructions belonging to a debug extended instruction set, the
OpExtInstImport declaring those sets, and OpExtension
"SPV_KHR_non_semantic_info" when no other non-semantic set survives.
Debug set names are matched by prefix since they are versioned;
OpenCL.DebugInfo.100 is the form the default translator invocation
emits. OpString, OpName, OpSource and OpLine are kept, chipStar resolves
kernels by name and OpLine is plain valid SPIR-V.
The filter returns its input unchanged on anything it cannot safely walk
(bad size, wrong magic number, out-of-bounds word count, unterminated
literal string). Stripping is a portability workaround, so it must never
turn a binary the driver could have consumed into one it cannot.
On real chipStar modules this takes 74376 bytes down to 22436 and turns
spirv-val rc=1 into rc=0, with all 4 OpEntryPoint and 26 OpName intact.
Modules without debug information come back byte-identical.
(cherry picked from commit 745f884cce919bbaa54b1f00ba3d46f2bfa6dc51)
Strip SPIR-V debug information at both backend hand-off points unless the device is an Intel Data Center GPU Max (Ponte Vecchio). IGC there tolerates the invalid debug information the SPIR-V translator emits for -g, which is what makes gdb-oneapi work on Aurora; every other driver we target either rejects the module or miscompiles it. The strip cannot live in SPVRegister::getFinalizedSource(), which produces one binary shared by all devices, so it goes in CHIPModuleOpenCL::compile and CHIPModuleLevel0::compile. Both call sites sit before the module cache key is computed, so a stripped and an unstripped module can never share a cache entry. Detection is by PCI device ID, not device name. 0x0BD0 (PVC-XL) has no marketing name and reports as "Intel(R) Graphics [0x0bd0]", so a name test misses it, and "Data Center GPU Flex" is DG2 rather than PVC, so a loose name test false-positives. OpenCL reads the ID via CL_DEVICE_ID_INTEL behind a cl_intel_device_attribute_query guard and Level Zero from ze_device_properties_t::deviceId, both falling back to the full "Data Center GPU Max" name for drivers without an ID. The result is cached on the device at populateDevicePropertiesImpl() time, so no extra driver queries happen per module load. chipStar had no PVC gate before this; the existing "Data Center GPU Max" mentions in the backends are comments on unconditional workarounds. (cherry picked from commit 2c143d82159ea8c7d321558e3d4322ed96c9ed72)
embed_spirv_in_cpp() compiled the rtdevlib bitcode with
--target=spirv64v${MAX_SPIRV_VERSION}-unknown-chipstar. The "chipstar" OS
component selects the HIPSPV toolchain, whose -c action emits LLVM
bitcode for the offload driver to link later rather than SPIR-V. clang
exits 0 and writes the file, so every rtdevlib ".spv" silently contained
LLVM bitcode ("BC\xc0\xde") instead of SPIR-V ("\x03\x02\x23\x07"), and
that bitcode was embedded into libCHIP and handed to the OpenCL driver:
clLinkProgram failed: -17 CL_LINK_PROGRAM_FAILURE (Intel GPU)
clLinkProgram failed: -44 CL_INVALID_PROGRAM (Intel CPU)
"Device library link step failed."
Measured with the LLVM 23 toolchain:
--target=spirv64v1.2-unknown-chipstar -> 4243c0de (LLVM bitcode)
--target=spirv64v1.2 -> 03022307 (SPIR-V)
Only LLVM 23 is affected: OFFLOAD_TRIPLE has no "chipstar" OS component
before 23, so the rtdevlib was built correctly there. This is why the 21
and 22 lanes are green while every LLVM 23 test that links the rtdevlib
(ballot, atomics, printf) failed.
Verified on meatloaf against the LLVM 23 toolchain: all rtdevlib modules
now carry the SPIR-V magic and pass spirv-val, and the previously failing
ballot and atomics tests pass on both Intel GPU and Intel CPU OpenCL with
no clLinkProgram errors.
IGC's optimizer miscompiles chipStar kernels, which TestSnakeMiscompileO2 catches. The failure is not in chipStar or in LLVM: the same chipStar build, the same LLVM 23 toolchain and the same SPIR-V module produce the correct answer on the Intel CPU runtime and on any Intel GPU when IGC's optimizer is disabled, and the wrong answer otherwise. Measured, varying only the IGC library: IGC 2.36.3 (meatloaf, latest in apt) Arc A380 FAIL IGC 2.38.2 (cupcake stock) Arc B570 FAIL IGC 2.38.2 (cupcake stock) UHD 770 FAIL igc/all-fixes-2026.07.30 Arc A380 PASS igc/all-fixes-2026.07.30 Arc B570 PASS igc/all-fixes-2026.07.30 UHD 770 PASS IGC 2.38.2 + -cl-opt-disable Arc B570 PASS Intel CPU OpenCL runtime (same spv) PASS The bug is unfixed in both released IGCs, so load the local build that carries the fixes. Full dGPU OpenCL suite on meatloaf with LLVM 23 and this IGC: 100% tests passed, 0 failed out of 974. The module is installed at ~/modulefiles/igc/all-fixes-2026.07.30 on the Linux X64 runner and only prepends LD_LIBRARY_PATH and PATH, so it is scoped to these steps and does not disturb the system driver.
Backport of llvm/llvm-project#206998, merged upstream 2026-07-21 as b1d21c6d3121, after release/23.x was cut and therefore missing from llvmorg-23.1.0-rc2. LLVM 23 runs ExpandVariadics in the SPIR-V backend IR pipeline for non-shader targets, i.e. OpenCL kernels. Its guard skips SPIR-V builtins by testing the demangled name against the prefix "printf(", but an unmangled C printf demangles to plain "printf" with no argument list, so OpenCL/HIP printf slipped through and had its arguments packed into a vararg buffer: LLVM 22: %87 = OpExtInst %uint %1 printf %fmt %arg0 %arg1 %arg2 LLVM 23: %99 = OpExtInst %uint %1 printf %fmt %vararg_buffer Device printf then printed garbage and crashed, on Intel dGPU, iGPU and the Intel CPU OpenCL runtime alike, unaffected by -cl-opt-disable. This is what cuda-simplePrintf and PrintfDynamic were catching on the LLVM 23 lane; chipStar's own printf lowering is correct and the IR reaching the backend is identical on LLVM 22 and 23. Drop this patch once release/23.x picks the fix up.
SPIRVEmitIntrinsics::insertPtrCastOrAssignTypeInstr() bounds its operand
loop by the call's argument count while indexing the callee's declared
parameters:
for (unsigned OpIdx = 0; OpIdx < CI->arg_size(); OpIdx++)
...
Argument *CalledArg = CalledF->getArg(OpIdx);
For a variadic callee those counts differ and getArg() asserts:
Function.h:861: llvm::Function::getArg(unsigned):
Assertion `i < NumArgs && "getArg() out of range!"' failed.
getArg() is only reached for pointer-typed operands that are neither
Instructions nor Arguments, i.e. pointer constants, so triggering it
needs a variadic call whose variadic argument is a constant pointer.
OpenCL printf with a "%s" argument is exactly that: printf declares one
parameter and the call carries several, the extra one pointing at a
string literal.
It does not reproduce while ExpandVariadics lowers printf away before
this pass runs, which is why it only appeared after backporting
llvm/llvm-project#206998 (patch 0004). Reproducer: chipStar's
hip-tests catch/unit/printf/printfFlags_exe.cc, which crashed clang
during hipspv-link.
The same unguarded indexing is present on llvm-project main, so a kernel
calling printf("%s", ...) should assert there too; this needs reporting
upstream rather than being a release-branch gap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes CI. Every PR has been red since 2026-07-31 because device debug info is enabled on toolchains that cannot emit it correctly.
Cause
Merging #1414 promoted 0002-preserve-device-debug-info.patch into the tracked patch set. That patch deletes DebugInfoKind = NoDebugInfo from HIPSPVToolChain::adjustDebugInfoKind, which was the only thing keeping device debug info off. Combined with the unconditional -gdwarf-4 in CMakeLists.txt, every build emits device debug info, Release included.
Both SPIR-V producers emit invalid SPIR-V when they do. The translator emits a cyclic forward reference in DebugTypeComposite Parent; the LLVM 22 backend emits DebugTypePointer with a DebugInfoNone base type. Measured with spirv-val on LLVM 22.1.8 and on upstream main. IGC hits an internal compiler error on the result, which is why whole test suites segfault.
The cycle is not fixable by reordering. The spec answer, SPV_KHR_relaxed_extended_instruction, is rejected by every driver we use: Intel dGPU, iGPU and CPU OpenCL, POCL, Arc B570, AMD rusticl, and Mali.
Changes
LLVM 21 and 22 stop applying 0002-preserve-device-debug-info.patch, so stock clang forces NoDebugInfo again and no device debug info is generated. This is the pre-#1414 behaviour that worked for years, and it retires #1004 on those versions: that path was emitting SPIR-V that spirv-val rejects, and only appeared to work on Aurora because its IGC tolerates malformed debug info where Arc ICEs.
LLVM 23 is added, pinned to llvmorg-23.1.0-rc2, with a backport of llvm#213052 which adds in-tree HIPSPV SPIR-V backend support. That backport had to be re-authored against rc2 rather than applied, because upstream assumes llvm#210504 and two extensions that are not on release/23.x. Only two LLVM patches are needed on 23: the other three are already upstream there.
On 23 the debug-info patch is kept but gated on useIntegratedBackend(), so -g is honoured only when the integrated backend is actually in use and a user passing -fno-integrated-objemitter still gets the strip.
LLVM 20 is dropped. Supported set becomes 21, 22, 23.
Why one llvm/23.0 module and not a -native/-translator pair
With llvm#213052 patched in, a single toolchain built with the SPIRV target serves both producers: the translator is built either way, and the producer is chosen per compilation by -f[no-]integrated-objemitter. So --variant becomes a cmake flag rather than a different module. This halves LLVM build time on every CI machine.
Verification
Device debug info was measured working on all five device and backend combinations with the integrated backend on LLVM 24: dgpu-l0, dgpu-ocl, igpu-l0, igpu-ocl, cpu-ocl. All four LLVM 23 patches apply with zero fuzz to a pristine llvmorg-23.1.0-rc2 and to llvm_release_230, and HIPSPV.cpp and SPIRVSubtarget.cpp compile with them in place. A full LLVM 23 build through configure_llvm.sh --version 23 is running as of opening this.
Note the lanes that module load an installed toolchain will still fail on this PR, because they resolve the currently installed compilers. They go green once this merges and install-llvm-on-main promotes the rebuilt toolchains. The lanes to judge this PR by are the Test LLVM Patches ones, which build and test against the freshly staged toolchain.
Depends on #1417, already merged, which stops install-llvm-on-main destroying a live toolchain when no build is staged.