Repository navigation
Conversation
The communication library nodes were MPI-specific. They now live in dace.libraries.comm, and every node with an NCCL equivalent gains an NCCL implementation. The same implementation uses RCCL on the HIP backend, which exposes the identical nccl* API. - Send/Recv map to ncclSend/ncclRecv. NCCL has no tags, so messages to a peer match in order; strided 2-D buffers move row by row. - Isend/Irecv join one open NCCL group, which the next Wait or blocking operation launches. Wait/Waitall also synchronize the streams involved. - Bcast, Reduce, Allreduce and Allgather use the native collectives. Alltoall, Scatter and Gather go through dace_ccl.h helpers that call ncclAlltoAll/ncclGather/ncclScatter where available (NCCL >= 2.28; RCCL, which names the all-to-all ncclAllToAll before its 2.28 API) and fall back to grouped send/recv otherwise. - Communicators are bootstrapped over MPI: the world one at initialization, process-grid ones in the SDFG init code after the grid itself, because creating them lazily would deadlock point-to-point calls. - The state struct holds a pointer to the NCCL context, so the GPU translation unit needs only a forward declaration (dace_ccl_fwd.h) and no MPI headers. library.comm.default_implementation (default MPI) selects the backend of portable nodes. Dummy, Redistribute, BlockScatter and BlockGather rely on MPI datatypes and stay MPI-only. Reduction operators accept MPI names or ReductionType names. dace.libraries.mpi remains as a deprecated alias, including its submodule paths and the environment names in saved SDFGs. Communication nodes read host control values next to GPU buffers, which made schedule inference report a conflict, e.g. for Isend with its request array. LibraryNode.bridges_storage replaces the Copy/Fill special case in infer_types and now also covers communication nodes. The Python frontend also stops placing rank and tag scalars in the buffer's storage, where a GPU buffer made them unreadable from the host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n nodes A communication node communicates over the communicator given to its optional _comm connector, an integer handle of an MPI_Comm or ncclComm_t, before its process grid or the world. Applications that own their processes (and their NCCL bootstrap) can then run DaCe programs on their own communicators. - Every MPI expansion selects its communicator through mpi_communicator; the Allgather, Gather, Scatter, Send and Recv expansions no longer hardcode MPI_COMM_WORLD and so respect _grid like their NCCL counterparts. - NCCL expansions derive from NcclExpansion. The NCCL environment no longer depends on MPI: the world communicator is created in the program's init code only when a node uses it, and only expansions that bootstrap over MPI add the MPI environment. The MPI environment defines DACE_WITH_MPI, which guards the MPI parts of dace_ccl.h. - dace.libraries.comm.communicators: nccl_unique_id, nccl_init_rank and nccl_destroy (ctypes on the NCCL/RCCL library the environment links) and mpi_handle for mpi4py communicators. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g connectors
- ReduceScatter: MPI_Reduce_scatter_block / ncclReduceScatter.
- Alltoallv: per-process symbolic send and receive counts; MPI_Alltoallv, or
grouped ncclSend/ncclRecv (dace::comm::AlltoAllv).
- The average ("avg") as a reduction operator: ncclAvg, or a sum divided by the
number of processes in the MPI expansions of Allreduce, Reduce and
ReduceScatter.
- Wait and Waitall accept ordering-only connectors (the buffer of a pending
receive or send), which the expansions ignore, and write only the status
outputs that are connected.
- NCCL has no requests: the _request connectors of the NCCL Isend, Irecv, Wait
and Waitall expansions are optional, so point-to-point programs over a given
communicator need no MPI types.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NCCL operations have no requests. An integer container connected to _request is set to zero, so it orders a nonblocking NCCL operation before its Wait without MPI types; MPI_Request containers are still set to MPI_REQUEST_NULL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The expansions communicate a count of elements from a pointer. Simplification may hand a collective a strided view (e.g., a transpose) in place of the contiguous copy a frontend wrote, and a packed permutation is contiguous memory but not in row-major order. pack_noncontiguous_buffers routes such buffers through contiguous transients, with one transient for in-place buffers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI runs the MPI tests on two ranks. The subgroup of every rank but the last has one member there, and send/recv runs over the world in reverse order, so communicator ranks still differ from world ranks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The c_char array field stopped at the first zero byte of the binary ID. Tested on a GPU: collectives and a self-exchange over an NCCL communicator created in Python, with no MPI in the program. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rank 0 broadcasts its compilation error instead of a build folder, and the final barrier becomes an exchange of load outcomes, so a failure no longer leaves the other ranks waiting. A rank that holds an SDFG checks that it is the one rank 0 compiled. Only Get_rank, bcast and allgather are used, so non-mpi4py communicators work too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Allreduce, Allgather, ReduceScatter, Bcast, Alltoall and Alltoallv derive from NonblockingCommNode, whose nonblocking property starts the operation and writes a request that a Wait completes. MPI calls the MPI_I* variant (Alltoallv keeps its count arrays alive until completion); NCCL launches the operation right away and tracks its stream for the Wait, with a token request. Nonblocking nodes need contiguous buffers (a packed copy would not outlive the operation), and MPI cannot average without blocking. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dace.libraries.comm.schedule.overlap_communication moves nonblocking starts as early, and their waits as late, as data, symbol and view dependencies allow within each scope; a wait inherits the buffers of the starts whose requests it completes. Starts keep their order relative to other starts and blocking operations (collectives start in the same order on every process), and waits keep theirs. Every communication node then gets a state of its own, which state fusion keeps (communication has side effects), so code generation runs start, independent work and wait in this order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A start moves ahead of the work before it together with the nodes it depends on, so that independent work that precedes its producers also runs while the communication is in flight; a wait likewise moves behind the work that does not depend on it. Relative orders within each group are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A collective runs asynchronously on the stream of its node already, so a nonblocking one is launched right away like any other and needs no request, stream tracking or Wait: code generation orders its readers on the device and runs independent work on other streams. A Wait (and Waitall) connected to GPU memory makes its stream wait for the pending point-to-point streams through events, instead of synchronizing them on the host, which also keeps code generation from forcing it onto the default stream. Isend accepts ordering output connectors, so that a send can be ordered before its wait through the sent buffer: a request or token in host memory makes the host wait for the stream. The overlap pass leaves NCCL nodes in the states of their neighbors, whose end would make the host wait for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Packing copies into and out of contiguous buffers are ordered on the GPU stream of the node, so stream-ordered (NCCL or RCCL) expansions pack the strided buffers of a nonblocking node like those of any other; only MPI, whose Wait completes the operation later, still requires contiguous buffers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The requirement that a nonblocking node have contiguous buffers holds for MPI only, so it moves out of pack_noncontiguous_buffers (and its stream_ordered escape hatch) into NonblockingCommNode.validate_nonblocking, which the MPI expansions call before packing, together with the request check. NCCL/RCCL expansions pack as for any node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main replaced all_states() with a recursive states() (#2652). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rotocol distributed_compile now ends with an allgather of every rank's outcome and broadcasts (folder, digest, error) rather than the folder alone. The fake communicator gains allgather in place of the unused Barrier, and the test compares against the broadcast folder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The communication library nodes were MPI-specific. They now live in dace.libraries.comm, and every node with an NCCL equivalent gains an NCCL implementation. The same implementation uses RCCL on the HIP backend, which exposes the identical nccl* API.
library.comm.default_implementation (default MPI) selects the backend of portable nodes. Dummy, Redistribute, BlockScatter and BlockGather rely on MPI datatypes and stay MPI-only. Reduction operators accept MPI names or ReductionType names. dace.libraries.mpi remains as a deprecated alias, including its submodule paths and the environment names in saved SDFGs.
Communication nodes read host control values next to GPU buffers, which made schedule inference report a conflict, e.g. for Isend with its request array. LibraryNode.bridges_storage replaces the Copy/Fill special case in infer_types and now also covers communication nodes. The Python frontend also stops placing rank and tag scalars in the buffer's storage, where a GPU buffer made them unreadable from the host.