Skip to content

Require numba and verify its threading layer at startup - #1253

Merged
SimonHeybrock merged 1 commit into
mainfrom
numba-threading-dep
Aug 26, 2026
Merged

Require numba and verify its threading layer at startup#1253
SimonHeybrock merged 1 commit into
mainfrom
numba-threading-dep

Conversation

@SimonHeybrock

@SimonHeybrock SimonHeybrock commented Aug 26, 2026

Copy link
Copy Markdown
Member

ess.reduce.unwrap interpolates the wavelength lookup table with a Numba-compiled kernel and silently falls back to a SciPy implementation when numba is not importable. Nothing declared the dependency, so which implementation ran was a property of the environment rather than of the release. Neither the production conda env nor the devcontainer had numba installed, so production has been running the SciPy fallback.

The fallback is expensive. Measured on a 1000x200 table:

points scipy numba (24 threads) numba (1 thread)
1e5 13.0 ms 0.41 ms 1.5 ms
1e6 139 ms 2.1 ms 16 ms
1e7 1502 ms 13.3 ms 166 ms

The kernel is entered from several Python threads at once, because JobManager runs jobs in a pool (--job-threads, default 5). Numba's workqueue threading layer is not safe under that access pattern and crashes the process (scipp/ess#705, scipp/ess#707). It is selected only when neither OpenMP nor TBB can be loaded — on Linux libgomp.so.1 gives us omp, confirmed on the production host and in a bare pip install numba venv. Backend services now resolve the layer at startup and refuse to run on an unsafe one, so a missing OpenMP runtime surfaces as a startup error instead of a segfault under load, and a silent 30-100x slowdown cannot recur.

Because nothing had numba before, this also adds it to the dashboard process, which never interpolates a lookup table: ~68 MB of resident memory for numba/llvmlite, measured on the dashboard import (219 MB -> 287 MB). Two upstream changes remove it again -- scipp/ess#707 moves the numba import inside _get_interpolator_class, and making ess.reduce.__init__ lazy stops the dashboard reaching unwrap at all. Neither is a blocker here: the backend needs numba either way, and a backend-only extra would trade the memory back for the silent-fallback failure mode this PR exists to close.

Consequence worth knowing: on a Mac without tbb or llvm-openmp in the environment, numba selects workqueue and backend services will now refuse to start. That is deliberate — running the workflows there risks the crash — and the error names the fix. conda install llvm-openmp (or tbb) resolves it.

Not addressed here: thread oversubscription (5 concurrent jobs x numba defaulting to all cores) and NUMBA_CACHE_DIR, which avoids re-JITting (~0.8 s) at every service start when the cache cannot be written next to the installed essreduce. Both are deployment-environment settings and this repo does not hold that config.

Test plan

  • pytest -n auto passes with numba installed, i.e. with the Numba interpolator actually in use
  • python -m ess.livedata.services.monitor_data --instrument dummy --check logs the selected layer and exits cleanly
  • Production host verified to select omp

essreduce interpolates the wavelength lookup table with a Numba-compiled
kernel and silently falls back to a SciPy implementation when numba is not
importable. On a 1000x200 table the fallback costs 32-113x depending on point
count (1e6 points: 139 ms vs 2.1 ms); even pinned to a single thread numba is
8-9x faster. Nothing declared the dependency, so which implementation ran was
a property of the environment rather than of the release.

That kernel is entered from several threads at once, since JobManager runs
jobs in a pool (--job-threads defaults to 5). Numba's workqueue threading
layer is unsafe under that access pattern and crashes the process
(scipp/ess#705); it is selected only when neither OpenMP nor TBB can be
loaded. Backend services now resolve the layer at startup and refuse to run
on an unsafe one, so a missing OpenMP runtime surfaces as a clear startup
error rather than as a segfault under load.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@SimonHeybrock SimonHeybrock left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@SimonHeybrock
SimonHeybrock merged commit a98c654 into main Aug 26, 2026
16 checks passed
@SimonHeybrock
SimonHeybrock deleted the numba-threading-dep branch August 26, 2026 08:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant