CI: exercise the flexible-rank RMA path on two MPI implementations

[DLWoodruff]

Closes the last release gate for flexible (unequal) rank assignments.

The flexible-rank path is the only place mpi-sppy uses passive-target MPI RMA (Lock/Get/Put), the least portable corner of the spec. doc/designs/flexible_rank_assignments.md therefore gates recommending the feature for production on exercising it against "at least two MPI implementations (e.g. OpenMPI and MPICH) and more than one mpi4py / MPI version".

Until now the one job that runs those tests — test-cylinders — did a bare conda install mpi4py, which resolves to MPICH. So only MPICH was ever covered, and nothing in the logs said which implementation had been used.

The matrix

test-cylinders becomes a three-cell include matrix. Resolved versions, from dry-run solves:

cell	MPI	mpi4py
mpich	MPICH 5.0.1	4.1.2
openmpi	OpenMPI 5.0.10	4.1.2
mpich, mpi4py 3.1	MPICH 4.3.2	3.1.6

Two implementations, two MPICH versions, two mpi4py versions. fail-fast: false, because the point is to learn which builds the RMA path works on — every cell has to report rather than being cancelled by a sibling.

A new step prints mpiexec --version and MPI.Get_library_version(), so the job log is the evidence the gate asks for instead of an assumption about what conda picked.

conda-forge, exclusively

The job now sets channels: conda-forge with conda-remove-defaults. conda-forge is the channel that maintains per-implementation MPI variants: asking for openmpi there also pulls the openmpi-flavored mpi4py through the mpi metapackage, so each cell exercises the MPI it names.

On the defaults channel the mpi4py build is not variant-selected, so an "openmpi" cell could install OpenMPI beside an mpi4py still linked against MPICH — a cell that looks like coverage and is not.

A bug this turned up

-oversubscribe is OpenMPI-only (MPICH rejects it) and the np=6 runs ask for more ranks than a runner has cores, so it has to be applied conditionally. The existing detection in run_coverage.bash greps the mpiexec --version banner for Open MPI/OpenRTE.

That breaks on OpenMPI 5, which launches through PRRTE and no longer reliably prints either string. The openmpi cell would have silently dropped the flag and failed at np=6 with "not enough slots available".

Both the workflow and run_coverage.bash now ask mpi4py which library it is linked against (MPI.Get_library_version()), which is stable across versions. Verified locally on OpenMPI 4.1.6: detection fires, and single-dash -oversubscribe works at np=6.

Notes for review

• The check name changes from tests on some cylinders to tests on some cylinders (mpich) / (openmpi) / (mpich, mpi4py 3.1). If branch protection lists the old name as a required check, that entry needs updating.
• Each cell needs a distinct coverage artifact name (upload-artifact@v4 errors on duplicates); the aggregator globs coverage-*, so they are still collected.
• The design doc's release-gate paragraphs are flipped, but only after the matrix was observed green (see below).

Follow-ups folded in after the first run

The first run failed in the openmpi cell — not on RMA. It died in the pre-existing equal-rank step (test_with_cylinders.py at np=2), which then skipped the flexible-rank step, leaving the question this job exists to ask unanswered.

test_xhatshuffle_coverage compares the value its helper extension records on iteration 0 against iteration 1, so the hub has to run both. It asks for max_iterations=2 but never overrides cfg.rel_gap, which defaults to 0.05, and the hub stops as soon as the inter-cylinder gap drops below it. The log shows Terminating based on inter-cylinder relative gap 4.595% immediately before the IndexError. That is a race between spoke and hub, so it fires on some MPI/host combinations and not others. rel_gap is now pinned to 0 in that test, which is what the assertion already assumed; guarding on len(v) >= 2 would have hidden the check instead of fixing it.

The flexible-rank step also gets if: !cancelled(), so a pre-existing equal-rank failure can no longer conceal the flexible-rank result. Both still fail the job.

Result

All three cells green, and the OpenMPI cell's log carries coherence diagnostic lines from PHHub, LagrangianOuterBound, XhatShuffleInnerBound and CGHub — those only print when multi-source reads actually happen, so the unequal-rank path demonstrably ran there rather than being skipped.

That was the last item on the "finish before recommending it" list, so the design doc now records that no release gates remain, and states what the matrix covers instead of what someone still has to do.

With the feature no longer gated, flexible ranks gets its first user-facing page, doc/src/flexible_ranks.rst (Advanced Topics). Until now the feature was documented only in doc/designs/, which is not part of the Sphinx build. The page covers the eight --<spoke>-rank-ratio options, the largest-remainder apportionment, the fact that all-1.0 ratios take the same code path as before (which is what makes reverting a complete fallback), and the limitations — reduced_costs has no ratio option, and any other unsupported field fails at startup rather than mis-assembling. The worked example was run end to end first: 14 ranks at 1.0/0.5/0.25 does print per-cylinder rank counts [8, 4, 2].

The read-outcome diagnostic from #816 also gets an entry in doc/src/secretmenu.rst: the always-on finalization summary, what each of the five buckets means, how to tell a coherence problem from a slow upstream sender, and the coherence_diagnostics_period knob. Three review caveats are documented rather than fixed, since they affect how the numbers read and not whether they are collected — counters aggregate per Field with no peer attribution, the periodic line's buckets sum to total - 1, and equal-rank runs print nothing.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.83%. Comparing base (ebc0d78) to head (45f51da).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #827   +/-   ##
=======================================
  Coverage   76.82%   76.83%           
=======================================
  Files         177      177           
  Lines       23498    23498           
=======================================
+ Hits        18053    18055    +2     
+ Misses       5445     5443    -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.