test(dist): assert reload replays new_group timeout and options - #2284
Open
EazyReal wants to merge 1 commit into
Open
test(dist): assert reload replays new_group timeout and options#2284EazyReal wants to merge 1 commit into
EazyReal wants to merge 1 commit into
Conversation
The WORLD reload tests only check that a reloaded group can still barrier. That still passes if reload drops timeout/pg_options and falls back to {ranks, backend=nccl}, which is the offload/heal watchdog bug.
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
CPU regression test that
reload_process_groups()callsnew_groupwith the originaltimeout,pg_options, andgroup_desc, not{ranks, backend="nccl"}.Why
Follow-up to #2095. The constructor replay landed in #2208; this is the assertion #2095 carried that did not.
#2208's multiprocess test checks that a reloaded group can still
barrier. That still passes if reload drops a 2-hour NCCL timeout and falls back to PyTorch's default — the offload/heal watchdog bug. This test spies on the constructor.Verified locally: passes on current main; fails if reload is patched back to
old_new_group(ranks=..., backend="nccl").Test
pytest tests/test_reloadable_process_group_world.py::test_reload_replays_original_new_group_timeout_and_options -qMade with Cursor