Repository navigation
Fix NCESTrainer DataLoader worker-pool deadlock under coverage - #614
Merged
Merged
Conversation
NCESTrainer.train() creates a fresh DataLoader(num_workers=...) worker pool per architecture in a loop over self.synthesizer.model. The existing os.cpu_count()-based cap on num_workers only engaged when cpu_count <= num_workers, so on any machine with more cores than the default of 8 it silently did nothing, leaving num_workers at 8. Reproduced locally: coverage run -m pytest against test_nces_trainer.py hung indefinitely with num_workers=8 (confirmed via /proc/<pid>/wchan showing genuine futex/socket waits, not just slow computation); num_workers of 1/2/4 all completed reliably in under a minute. This matches CI's multi-hour hangs on PR #607 (issue #610). Cap num_workers unconditionally to min(requested, cpu_count-1, 4) in both NCES.train() and NCESTrainer.__init__ (defense in depth for direct NCESTrainer usage). Fixes #610. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018iKbU9gaut8SBBPnXcZhZA
This was referenced Oct 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NCESTrainer.train()creates a freshDataLoader(num_workers=...)worker pool per architecture in a loop overself.synthesizer.model. The existingos.cpu_count()-based cap onnum_workersonly engaged whencpu_count <= num_workers, so on any machine with more cores than the default of 8 it silently did nothing — the requested value passed straight through uncapped.coverage run -m pytestagainsttests/test_nces_trainer.pyhung indefinitely withnum_workers=8— confirmed via/proc/<pid>/wchanshowing genuinefutex_wait_queue/unix_stream_data_wait/do_pollblocking (a real deadlock, not slow computation: main process CPU time moved ~1s over 8 minutes of wall time).num_workersof 1/2/4 all completed reliably in under a minute under the same conditions.tests/test_nces_trainer.pyis where the CI job stalled). Full root-cause writeup in CI hangs for hours in NCESTrainer.train() — repeated num_workers=8 DataLoader creation deadlocks under coverage #610.num_workersunconditionally tomin(requested, cpu_count-1, 4)in bothNCES.train()(ontolearn/learners/nces.py) andNCESTrainer.__init__(ontolearn/nces_trainer.py, defense in depth for directNCESTrainerusage bypassingNCES.train()).Fixes #610.
Test plan
tests/test_nces_trainer.py(previously hung 40+ min undercoverage run, now passes in ~30-45s) — verified both in isolation and as part of the full suite.coverage run -m pytest -p no:warnings -x --durations=0, CI-equivalent command) against a full local mirror of the CI environment (live Fuseki + mutagenesis dataset, KGs/LPs/NCESData/CLIPData present, clean venv matchingpip install -e .["full"]): 191 passed in 20:48, no hangs, no failures.coverage report -mruns cleanly afterward.🤖 Generated with Claude Code
https://claude.ai/code/session_018iKbU9gaut8SBBPnXcZhZA