Restrict UCX transports in CI to work around mpich 5 default - #186
Merged
Conversation
mpich 5 is built as ch4:ucx,ofi in conda-forge, making UCX the default
netmod (4.x was ofi-only). Every MPI_Init() now goes through UCX,
including the one inside ESMF_RegridWeightGen for serial, single-rank
mapping-file generation.
On Azure-hosted GitHub runners, UCX enumerates the MANA adapter and
fails to open an RDMA verbs interface on it:
UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed: Address not valid
MPIDI_UCX_init_worker(86): ucx function returned with failed status
MPI_Init(argc=(nil), argv=(nil)) failed
ESMF_RegridWeightGen then aborts with exit 143, failing all the
remapping tests. Runners without a verbs device are unaffected, which
is why this does not reproduce on typical development machines.
Set UCX_TLS at the workflow level so it covers both the test job and the
rattler-build package job. This is deliberately not set in pixi.toml's
activation, since forcing IB traffic over TCP would hurt real parallel
regridding with mapParallelExec on HPC machines.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Without this, CI is failing.