Fix silent GPU transfer failures disabling CPU threading fallback - #176
Open
bhuvan-somisetty wants to merge 2 commits into
Open
Fix silent GPU transfer failures disabling CPU threading fallback#176bhuvan-somisetty wants to merge 2 commits into
bhuvan-somisetty wants to merge 2 commits into
Conversation
to_gpu() caught any exception from the GPU transfer and fell back to NumPy with no logging or warning at all. BaseFitter.fit_image() made it worse: use_gpu was decided once before the transfer and never re-checked, so after a silent fallback, chunk sizing still used GPU-oriented sizing and the CPU multi-threaded fallback stayed disabled (use_threading = not use_gpu evaluated False). A real GPU failure (OOM, driver hiccup) would silently degrade fitting to serial, single-threaded CPU with no way to tell why. - to_gpu() now warns (UserWarning) when the transfer fails, matching the existing GPU-OOM warning pattern already used in batch.py. - fit_image() re-derives use_gpu from the actual array returned by to_gpu() instead of trusting the pre-transfer decision, so chunk sizing and the CPU threading fallback reflect what's actually running. Fixes OSIPI#175
Contributor
Author
|
cc @ltorres6 @MohamedNasser8 for review |
Collaborator
|
Hello @bhuvan-somisetty, thanks for all your work, I'll have a look soon. |
ltorres6
requested changes
Aug 30, 2026
ltorres6
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for catching this silent fallback and your fix attempt.
I think what we actually want here is to throw an error instead of falling back if we
a) have a detected GPU
b) are not using force_cpu, and
c) transfer fails for any reason.
This will also require a change in batch.py since that deliberately falls back to numpy if we fill memory.
Can you make those changes?
| except Exception: | ||
| # Fallback to NumPy if GPU transfer fails | ||
| except Exception as e: | ||
| warnings.warn( |
Collaborator
There was a problem hiding this comment.
Please use the logging package to log INFO/WARN/ERRORS.
…r failure Per review feedback: to_gpu() now raises GPUTransferError instead of warning and silently returning a NumPy array when a GPU is detected, force_cpu is not set, and the transfer fails for any reason. Callers that decided GPU-vs-CPU behavior before the transfer (chunk sizing, threading) can no longer be left assuming GPU execution that never happened. BatchProcessor.map() deliberately wants to keep falling back to CPU when GPU memory fills mid-run, so it now explicitly catches GPUTransferError from to_gpu() and continues its existing retry-on-CPU behavior, while other exceptions from the batch function itself are still filtered by the existing memory-error heuristic. Also switches the transfer-failure log from warnings.warn to the logging package, as requested.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #175
to_gpu()inosipy/common/backend/array_module.pycaught any exception from the actual GPU transfer and fell back to NumPy with zero logging or warning. That's already not great, a real CUDA OOM or driver error just vanishes, butBaseFitter.fit_image()(osipy/common/fitting/base.py) makes it worse:use_gpuis decided once before the transfer and never re-checked afterward, so on a silent fallback:chunk_size.use_threading = not use_gpu and ...staysFalsebecauseuse_gpuis stillTrue, so the CPU multi-threaded fallback that would normally compensate is disabled too.End result: a GPU transfer failure silently degrades fitting to serial, single-threaded CPU execution with nothing telling you why.
osipy/common/backend/batch.pyalready treats an analogous GPU-OOM case as worth awarnings.warn(...), this just bringsto_gpu()in line with that existing pattern.Fix
to_gpu()now warns (UserWarning) when the transfer fails, before falling back.fit_image()re-derivesuse_gpufrom the actual arrayto_gpu()returns (hasattr(..., "__cuda_array_interface__")) instead of trusting the pre-transfer decision, so chunk sizing and the CPU threading fallback reflect what's actually running.Testing
TestToGpu::test_transfer_failure_warns_and_falls_backintests/unit/common/backend/test_array_module.py: mocks a GPU transfer failure and confirms aUserWarningis raised and a usable NumPy array is still returned.tests/unit/common/fitting/test_base.py(new file, mirrors theosipy/common/fitting/package): confirms that whento_gpu()falls back,fit_image()re-enables the CPU threaded path (previously it silently stayed disabled), plus a sanity check that a successful GPU transfer still skips CPU threading as expected.PackageNotFoundErrorCLI failures already tracked/fixed separately in [BUG] osipy CLI crashes on every invocation when package metadata isn't discoverable, not just --version #173 / PR Fix CLI crashing on every invocation when package metadata is missing #174, unrelated to this change (this branch was cut frommainbefore that fix landed).ruff check/ruff format --checkclean on all changed/new files.mypyon both changed source files: no new errors introduced (verified by diffing mypy output againstmain, same 3 pre-existing findings inarray_module.pyshifted by one line from the added import;fitting/base.pyis clean).