Skip to content

CI: NCES tests intermittently hang for 1h+ in GitHub Actions #622

Closed

Description

@Demirrr

Summary

tests/test_nces.py, tests/test_nces2.py, and tests/test_nces_trainer.py intermittently hang in CI for well over an hour instead of completing in their normal ~1-2 minutes, eventually requiring the run to be manually cancelled.

Evidence

Same commit, two CI runs triggered by the same push (this repo's test.yml runs on both push and pull_request):

This is not new: a similar multi-hour stall (auto-cancelled around 6h) has happened on a prior PR's pull_request run as well.

Suspected cause

All three test files load a pretrained NCES model (load_pretrained=True) or otherwise pull in NCES model weights. The likely culprit is a flaky/slow external download (NCESData.zip from files.dice-research.org, or a pretrained-model fetch inside NCES(...) construction) that occasionally stalls indefinitely instead of timing out.

Workaround (temporary)

These three test files are excluded from the CI test run via --ignore flags in .github/workflows/test.yml (see PR referencing this issue) until the root cause is fixed. They still run fine locally/manually.

Suggested fix

  • Add a request/connect timeout around whatever network call is stalling (model/data download), so it fails fast and loudly instead of hanging until the GitHub Actions job timeout.
  • Consider caching NCESData.zip (and other external data downloads) as a GitHub Actions cache to reduce reliance on files.dice-research.org being consistently responsive.
  • Once fixed, remove the --ignore flags added for these test files in .github/workflows/test.yml.

Activity

  1. TheSheep-stack commented on Aug 18, 2026

    @TheSheep-stack
    Collaborator

    Maybe it would make sense to cache the external downloads (KGs.zip, LPs.Zip, Nces data,...) in Github actions.
    I have noticed, that sometimes the download speed is extremely slow, but it seems to be a GH issue.
    There seems to be caching support with this: https://github.com/actions/cache

  2. Demirrr commented on Oct 12, 2026

    @Demirrr
    MemberAuthor

    Closing: the hang was root-caused and fixed, and CI has been stable since.

    Root cause (not the download, as suspected above): NCES.fit_one/fit_from_iterable and NCES2's equivalents built inference-only DataLoaders with a multiprocessing worker pool just to fetch one or a few batches. Under coverage run this deadlocks, each time right after the previous test finished cleanly. It is the same pattern fixed for the training path in #614.

    Fixes

    Evidence: all 8 develop pushes since (Aug 13 to Sep 16) are green in 31 to 38 min; none of the recent failures is a hang or timeout.

    The actions/cache suggestion is split out to #637.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions