You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CI: NCES tests intermittently hang for 1h+ in GitHub Actions #622
tests/test_nces.py, tests/test_nces2.py, and tests/test_nces_trainer.py intermittently hang in CI for well over an hour instead of completing in their normal ~1-2 minutes, eventually requiring the run to be manually cancelled.
Evidence
Same commit, two CI runs triggered by the same push (this repo's test.yml runs on both push and pull_request):
This is not new: a similar multi-hour stall (auto-cancelled around 6h) has happened on a prior PR's pull_request run as well.
Suspected cause
All three test files load a pretrained NCES model (load_pretrained=True) or otherwise pull in NCES model weights. The likely culprit is a flaky/slow external download (NCESData.zip from files.dice-research.org, or a pretrained-model fetch inside NCES(...) construction) that occasionally stalls indefinitely instead of timing out.
Workaround (temporary)
These three test files are excluded from the CI test run via --ignore flags in .github/workflows/test.yml (see PR referencing this issue) until the root cause is fixed. They still run fine locally/manually.
Suggested fix
Add a request/connect timeout around whatever network call is stalling (model/data download), so it fails fast and loudly instead of hanging until the GitHub Actions job timeout.
Consider caching NCESData.zip (and other external data downloads) as a GitHub Actions cache to reduce reliance on files.dice-research.org being consistently responsive.
Once fixed, remove the --ignore flags added for these test files in .github/workflows/test.yml.
Maybe it would make sense to cache the external downloads (KGs.zip, LPs.Zip, Nces data,...) in Github actions.
I have noticed, that sometimes the download speed is extremely slow, but it seems to be a GH issue.
There seems to be caching support with this: https://github.com/actions/cache
Closing: the hang was root-caused and fixed, and CI has been stable since.
Root cause (not the download, as suspected above): NCES.fit_one/fit_from_iterable and NCES2's equivalents built inference-only DataLoaders with a multiprocessing worker pool just to fetch one or a few batches. Under coverage run this deadlocks, each time right after the previous test finished cleanly. It is the same pattern fixed for the training path in #614.
Defense in depth: timeout=600 on the dicee embedding-training subprocess, and timeout-minutes: 90 on the CI job so any future hang fails fast instead of running for ~6h.
Evidence: all 8 develop pushes since (Aug 13 to Sep 16) are green in 31 to 38 min; none of the recent failures is a hang or timeout.
The actions/cache suggestion is split out to #637.
Activity
TheSheep-stack commentedon Aug 18, 2026
Maybe it would make sense to cache the external downloads (KGs.zip, LPs.Zip, Nces data,...) in Github actions.
I have noticed, that sometimes the download speed is extremely slow, but it seems to be a GH issue.
There seems to be caching support with this: https://github.com/actions/cache
Demirrr commentedon Oct 12, 2026
Closing: the hang was root-caused and fixed, and CI has been stable since.
Root cause (not the download, as suspected above):
NCES.fit_one/fit_from_iterableand NCES2's equivalents built inference-onlyDataLoaders with a multiprocessing worker pool just to fetch one or a few batches. Undercoverage runthis deadlocks, each time right after the previous test finished cleanly. It is the same pattern fixed for the training path in #614.Fixes
7e661a4):num_workers=0at all four inference call sites; the three NCES test files are re-enabled (no--ignoreflags remain intest.yml).f2208d0): same for theauto_trainconvenience path.timeout=600on the dicee embedding-training subprocess, andtimeout-minutes: 90on the CI job so any future hang fails fast instead of running for ~6h.Evidence: all 8
developpushes since (Aug 13 to Sep 16) are green in 31 to 38 min; none of the recent failures is a hang or timeout.The
actions/cachesuggestion is split out to #637.