Require numba and verify its threading layer at startup - #1253
Merged
Conversation
essreduce interpolates the wavelength lookup table with a Numba-compiled kernel and silently falls back to a SciPy implementation when numba is not importable. On a 1000x200 table the fallback costs 32-113x depending on point count (1e6 points: 139 ms vs 2.1 ms); even pinned to a single thread numba is 8-9x faster. Nothing declared the dependency, so which implementation ran was a property of the environment rather than of the release. That kernel is entered from several threads at once, since JobManager runs jobs in a pool (--job-threads defaults to 5). Numba's workqueue threading layer is unsafe under that access pattern and crashes the process (scipp/ess#705); it is selected only when neither OpenMP nor TBB can be loaded. Backend services now resolve the layer at startup and refuse to run on an unsafe one, so a missing OpenMP runtime surfaces as a clear startup error rather than as a segfault under load. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ess.reduce.unwrapinterpolates the wavelength lookup table with a Numba-compiled kernel and silently falls back to a SciPy implementation whennumbais not importable. Nothing declared the dependency, so which implementation ran was a property of the environment rather than of the release. Neither the production conda env nor the devcontainer hadnumbainstalled, so production has been running the SciPy fallback.The fallback is expensive. Measured on a 1000x200 table:
The kernel is entered from several Python threads at once, because JobManager runs jobs in a pool (
--job-threads, default 5). Numba'sworkqueuethreading layer is not safe under that access pattern and crashes the process (scipp/ess#705, scipp/ess#707). It is selected only when neither OpenMP nor TBB can be loaded — on Linuxlibgomp.so.1gives usomp, confirmed on the production host and in a barepip install numbavenv. Backend services now resolve the layer at startup and refuse to run on an unsafe one, so a missing OpenMP runtime surfaces as a startup error instead of a segfault under load, and a silent 30-100x slowdown cannot recur.Because nothing had
numbabefore, this also adds it to the dashboard process, which never interpolates a lookup table: ~68 MB of resident memory for numba/llvmlite, measured on the dashboard import (219 MB -> 287 MB). Two upstream changes remove it again -- scipp/ess#707 moves the numba import inside_get_interpolator_class, and makingess.reduce.__init__lazy stops the dashboard reachingunwrapat all. Neither is a blocker here: the backend needs numba either way, and a backend-only extra would trade the memory back for the silent-fallback failure mode this PR exists to close.Consequence worth knowing: on a Mac without
tbborllvm-openmpin the environment,numbaselectsworkqueueand backend services will now refuse to start. That is deliberate — running the workflows there risks the crash — and the error names the fix.conda install llvm-openmp(ortbb) resolves it.Not addressed here: thread oversubscription (5 concurrent jobs x numba defaulting to all cores) and
NUMBA_CACHE_DIR, which avoids re-JITting (~0.8 s) at every service start when the cache cannot be written next to the installedessreduce. Both are deployment-environment settings and this repo does not hold that config.Test plan
pytest -n autopasses withnumbainstalled, i.e. with the Numba interpolator actually in usepython -m ess.livedata.services.monitor_data --instrument dummy --checklogs the selected layer and exits cleanlyomp