Skip to content

feat: bound Megatron checkpoint retention - #2290

Open
skirdey wants to merge 1 commit into
THUDM:mainfrom
skirdey:codex/checkpoint-retention
Open

feat: bound Megatron checkpoint retention#2290
skirdey wants to merge 1 commit into
THUDM:mainfrom
skirdey:codex/checkpoint-retention

Conversation

@skirdey

@skirdey skirdey commented Aug 20, 2026

Copy link
Copy Markdown

Summary

  • add opt-in --save-retain-count for long-running RL jobs
  • prune only checkpoint directories at or below Megatron latest completed iteration marker
  • never remove in-flight directories newer than the marker, symlinks, release checkpoints, or unrelated paths
  • support synchronous and asynchronous checkpoint saves

Motivation

Long jobs currently retain every multi-gigabyte Megatron checkpoint unless an external supervisor deletes them. This can exhaust local disks even when the job only needs a small resumable history. The retention operation runs only after synchronous completion, or after maybe_finalize_async_save confirms an asynchronous save.

The option is disabled by default.

Test plan

  • PYTHONPATH=$PWD uv run --no-project --with pytest --with pyyaml --with omegaconf pytest -q tests/utils/test_checkpoint_retention.py tests/test_megatron_argument_validation.py
  • 24 passed
  • python3 -m py_compile slime/utils/checkpoint_retention.py slime/utils/arguments.py slime/backends/megatron_utils/actor.py tests/utils/test_checkpoint_retention.py
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant