Skip to content

Fix NCESTrainer DataLoader worker-pool deadlock under coverage - #614

Merged
Demirrr merged 1 commit into
developfrom
fix-nces-trainer-dataloader-hang
Aug 7, 2026
Merged

Demirrr merged 1 commit into
developfrom
fix-nces-trainer-dataloader-hang

Conversation

@Demirrr

@Demirrr Demirrr commented Aug 7, 2026

Copy link
Copy Markdown
Member

Summary

  • NCESTrainer.train() creates a fresh DataLoader(num_workers=...) worker pool per architecture in a loop over self.synthesizer.model. The existing os.cpu_count()-based cap on num_workers only engaged when cpu_count <= num_workers, so on any machine with more cores than the default of 8 it silently did nothing — the requested value passed straight through uncapped.
  • Reproduced locally: coverage run -m pytest against tests/test_nces_trainer.py hung indefinitely with num_workers=8 — confirmed via /proc/<pid>/wchan showing genuine futex_wait_queue/unix_stream_data_wait/do_poll blocking (a real deadlock, not slow computation: main process CPU time moved ~1s over 8 minutes of wall time). num_workers of 1/2/4 all completed reliably in under a minute under the same conditions.
  • This matches the multi-hour CI hangs observed on PR Fix AttributeError in TDL's no-features-extracted error message #607's runs (tests/test_nces_trainer.py is where the CI job stalled). Full root-cause writeup in CI hangs for hours in NCESTrainer.train() — repeated num_workers=8 DataLoader creation deadlocks under coverage #610.
  • Fix: cap num_workers unconditionally to min(requested, cpu_count-1, 4) in both NCES.train() (ontolearn/learners/nces.py) and NCESTrainer.__init__ (ontolearn/nces_trainer.py, defense in depth for direct NCESTrainer usage bypassing NCES.train()).

Fixes #610.

Test plan

  • tests/test_nces_trainer.py (previously hung 40+ min under coverage run, now passes in ~30-45s) — verified both in isolation and as part of the full suite.
  • Full local suite (coverage run -m pytest -p no:warnings -x --durations=0, CI-equivalent command) against a full local mirror of the CI environment (live Fuseki + mutagenesis dataset, KGs/LPs/NCESData/CLIPData present, clean venv matching pip install -e .["full"]): 191 passed in 20:48, no hangs, no failures.
  • coverage report -m runs cleanly afterward.

🤖 Generated with Claude Code

https://claude.ai/code/session_018iKbU9gaut8SBBPnXcZhZA

NCESTrainer.train() creates a fresh DataLoader(num_workers=...) worker
pool per architecture in a loop over self.synthesizer.model. The
existing os.cpu_count()-based cap on num_workers only engaged when
cpu_count <= num_workers, so on any machine with more cores than the
default of 8 it silently did nothing, leaving num_workers at 8.

Reproduced locally: coverage run -m pytest against test_nces_trainer.py
hung indefinitely with num_workers=8 (confirmed via /proc/<pid>/wchan
showing genuine futex/socket waits, not just slow computation);
num_workers of 1/2/4 all completed reliably in under a minute. This
matches CI's multi-hour hangs on PR #607 (issue #610).

Cap num_workers unconditionally to min(requested, cpu_count-1, 4) in
both NCES.train() and NCESTrainer.__init__ (defense in depth for
direct NCESTrainer usage).

Fixes #610.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018iKbU9gaut8SBBPnXcZhZA
@Demirrr
Demirrr merged commit 5abd1f9 into develop Aug 7, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI hangs for hours in NCESTrainer.train() — repeated num_workers=8 DataLoader creation deadlocks under coverage

1 participant