[sglang-miles] RDT/NIXL weight sync support for Ray scheduler actors - #27723
Merged
maocheng23 merged 11 commits intoAug 23, 2026
Conversation
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
github-actions
Bot
added
the
dependencies
yueming-yuan
force-pushed
the
sglang-miles
branch
from
June 18, 2026 21:02
1dbbf3a to
6a4ffcc
Compare
xyuzh
force-pushed
the
xinyu/rdt-weight-sync
branch
5 times, most recently
from
June 29, 2026 21:25
8fabe26 to
3d7ac3f
Compare
xyuzh
added a commit
to xyuzh/sglang
that referenced
this pull request
Jun 29, 2026
xyuzh
marked this pull request as ready for review
June 29, 2026 21:44
xyuzh
requested review from
CatherineSue,
Fridge003,
JustinTong0323,
Qiaolin-Yu,
Ying1123,
hnyls2002,
ispobock,
merrymercy and
slin1237
as code owners
June 29, 2026 21:44
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
yueming-yuan
force-pushed
the
sglang-miles
branch
from
July 7, 2026 06:43
f8cfad3 to
fbc2623
Compare
yueming-yuan
requested review from
ch-wan,
fzyzcjy,
key4ng,
kpham-sgl,
mickqian,
pyc96,
yhyang201 and
yuan-luo
as code owners
July 7, 2026 06:43
yueming-yuan
requested review from
DarkSharpness,
HydraQYH,
celve and
sogalin
as code owners
July 7, 2026 06:43
xyuzh
force-pushed
the
xinyu/rdt-weight-sync
branch
from
July 27, 2026 01:14
3d7ac3f to
90b4d28
Compare
yueming-yuan
force-pushed
the
sglang-miles
branch
from
July 29, 2026 20:58
f7ea06e to
3003d70
Compare
yueming-yuan
requested review from
Alisehen,
Kangyan-Zhou,
OrangeRedeng,
bingxche,
ishandhanani and
yctseng0211
as code owners
July 29, 2026 20:58
xyuzh
force-pushed
the
xinyu/rdt-weight-sync
branch
2 times, most recently
from
August 6, 2026 19:21
213a656 to
7b5206c
Compare
Add the sglang-side support for RDT (Ray Direct Transport / NIXL) weight sync, used by the miles trainer to push weights to rollout engines via a zero-copy RDMA pull instead of NCCL broadcast. - scheduler_actor: add pull_weights(), which uses ray.experimental set_target_for_ref to RDMA pre-sharded weight buckets directly into the model's param.data buffers (no intermediate receive buffers / copies). - ray/engine: register SchedulerActors as detached named actors with the http port baked into the name, so the trainer (a different Ray job) can discover them via list_named_actors even when several engines are co-located on one node; raise max_concurrency so a concurrent pull_weights is not starved while run_event_loop blocks; set RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1 so the absolute GPU id from get_accelerator_ids() stays valid. - server_args: add enable_engine_info_bootstrap, plus the needs_engine_info_bootstrap() / registers_parallelism_config() predicates that fold it in alongside the existing transfer-engine conditions. - engine / model_runner: gate the bootstrap server and the parallelism-config registration on those predicates, so RDT can get /parallelism_config WITHOUT the mooncake/verbs P2P transfer-engine seeding. Requires ray>=2.55.1 for ray.experimental.set_target_for_ref, which is already the floor in python/pyproject.toml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eight-sync Detached lifetime and threaded actors were applied to every Ray-mode engine. Add a dedicated flag that implies --enable-engine-info-bootstrap and gate them on it. Bind the device in pull_weights, which runs off the event-loop thread.
…ackend launch_server() unconditionally cleared server_args.placement_group, so a Ray-backend server launched from a parent job could not join an existing placement group and would auto-create a second one, double-booking the rollout GPUs. Honor a caller-supplied placement_group instead, and connect to the running cluster with the caller's runtime env when the launching process has no Ray context of its own (the mp.Process child loses it). Add SchedulerActor.register_weight_for_rdt() so the destination pins its model parameters with NIXL once, rather than re-pinning and re-handshaking on every RDT flush. Skipped under --enable-memory-saver, where parameter storages are not stably resident. Bump the ray extra to >=2.56.0 for ray.experimental.register_nixl_memory.
…e actors The SchedulerActor name format was inlined in _create_scheduler_actor, so an external trainer holding the placement bundles could only find the actors by scanning list_named_actors for substring matches. Move the format into a get_scheduler_actor_name() helper and re-export it from sglang.srt.ray, so a caller that knows (node ip, dp/pp/tp rank, port, bundle index) can rebuild the name and ray.get_actor() it directly. The placement-group hex is dropped from the name: it is not knowable from the caller side, and the http port already disambiguates engines co-located on one node.
… cluster The Ray-backend launch path connects with ray.init(address="auto"), which lands in a fresh anonymous namespace when none is given. SchedulerActors then register there instead of in the namespace the rest of the deployment uses, so discovery by name from another job has to scan every namespace. Honor a caller-supplied ray_namespace, alongside the placement group and runtime env already threaded through server_args.
Callers hosting the HTTP server in their own Ray actor can now call launch_engine to get the SchedulerActor handles back, then run serve_http on a thread. With handles returned directly, the schedulers no longer need detached lifetimes or name-based discovery, so the actor names key on the placement group ID instead of the server port.
launch_engine cleared server_args.placement_group before launching, so SchedulerActors fell back to their own scheduling instead of the group the caller reserved. Honor the override the caller set.
xyuzh
force-pushed
the
xinyu/rdt-weight-sync
branch
from
August 19, 2026 18:16
32d793c to
783651b
Compare
launch_engine and RayEngine now accept placement_group directly and carry it to _launch_subprocesses through a contextvar, replacing the dynamic server_args.placement_group attribute that had to be manually re-attached after every dataclasses.replace. Pairs with the separate PR that has RayDataParallelController accept is_custom_pg, which this passes.
xyuzh
force-pushed
the
xinyu/rdt-weight-sync
branch
from
August 19, 2026 18:22
783651b to
1fcbc8b
Compare
yueming-yuan
approved these changes
Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
sglang-side support for RDT (Ray Direct Transport / NIXL) weight sync — lets the miles trainer push RL weights to rollout engines via a zero-copy RDMA pull instead of NCCL broadcast. Paired with the miles-side PR (radixark/miles#1313), which has the perf write-up.
Changes
ray/scheduler_actor.py— addpull_weights(), which usesray.experimental.set_target_for_refto RDMA pre-sharded weight buckets directly into the model'sparam.databuffers (no intermediate receive buffers / copies).ray/engine.py+ray/data_parallel_controller.py— registerSchedulerActors as detached named actors with the http port baked into the name, so the trainer (a different Ray job) can discover them vialist_named_actorseven when several engines share a node; raisemax_concurrencyso a concurrentpull_weightsis not starved whilerun_event_loopblocks; setRAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1so the absolute GPU id fromget_accelerator_ids()stays valid.server_args.py— addenable_engine_info_bootstrapto start the engine-info bootstrap server and register per-rank parallelism config without the mooncake/verbs P2P transfer-engine seeding.entrypoints/engine.py+model_executor/model_runner.py— honorenable_engine_info_bootstrap; make the P2P transfer-engine memory registration non-fatal so EFA/verbs clusters where it fails don't crash the scheduler (RDT only needs the parallelism config).model_loader/loader.py— makepost_load_weightspublic so the RDT path can invoke it.pyproject.toml— requireray>=2.55.1(ray.experimental.set_target_for_ref).Notes
Targeting the
sglang-milesintegration branch; rebased onto currentsglang-miles(#28001, #29339).CI States
Latest PR Test (Base): ❌ Run #32602261107
Latest PR Test (Extra): ❌ Run #32602260844
Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.