Flex ranks: coherence read-outcome diagnostic (#742)

[DLWoodruff]

Coherence read-outcome diagnostic for the unequal-rank (flexible-ranks) path, localized to the multi-source reader.

Closes #742.

(This PR originally also replaced the unequal-rank layout exchange with a two-level scheme (#726). That change has been dropped: the flat fullcomm.allgather runs once at startup, its latency is O(log N), and its O(N) per-rank data volume is the result itself — every rank needs every peer's layout, so any replacement still delivers O(N) data per rank and only changes the collective's participant pattern. At even 10,000 total ranks the flat exchange moves a few MB per rank, milliseconds against hours of solves. #726 is closed won't-fix with that rationale, and the design doc's Option D note, Phase-2 bullet, and release-gate list now record it.)

Coherence read-outcome diagnostic (#742)

The unequal-rank multi-source reader now counts every multi-source read (>= 2 sources; a single source cannot straddle a publish) into one of five buckets at the reduce_source_write_ids call site:

• new_accepted — coherent, advanced, used
• not_new — coherent, but the sender has not published (slow sender)
• rejected_incoherent — this rank's sources disagreed (the fundamental coherence miss: the read straddled a publish)
• rejected_cross_reader — this rank's sources agreed, the collective cross-reader check rejected (typically the shadow of some other reader rank's straddle)
• accepted_mixed — relaxed field, blended assembly used

The reported miss rate is every read a straddled publish cost — rejected_incoherent + rejected_cross_reader + accepted_mixed, equivalently everything that was neither a clean accept nor a clean nothing-to-take. Counting only the locally-detected misses would divide the rate by the reader-rank count, since one straddle on an R-rank reader records one rejected_incoherent and R-1 rejected_cross_reader.

so an infrequently-reporting bounds cylinder can be diagnosed as coherence problem (rejected_incoherent/accepted_mixed dominate) vs. slow upstream sender (not_new dominates). Counting is always-on (two integer increments per multi-source read); each cylinder prints a per-field summary at finalization (aggregated across its ranks with one gather, rank-0-gated, only for fields that did multi-source reads — equal-rank runs print nothing). An opt-in periodic line (coherence_diagnostics_period in the cylinder's opt_kwargs options) supports live debugging, and the counters are exposed as SPCommunicator.coherence_counters. Sample from the farmer 4+2 integration test:

coherence diagnostic [LagrangianOuterBound] DUALS: total=1812, new_accepted=38, not_new=1664, rejected_incoherent=50, rejected_cross_reader=60, accepted_mixed=0, miss rate=6.07%

Tests

• test_flex_coherence_policy.py: the deterministic stub reader now also pins the counter buckets (strict mixed → rejected_incoherent, relaxed mixed → accepted_mixed, coherent no-advance → not_new, buckets partition the total).
• test_flexible_rank_duals.py: after both split directions, asserts the buckets partition the total, that the strict DUALS field never accepts a blended read, and that the 4+2 bound came from accepted multi-source reads (in 2+4 the spoke's slices can nest inside the hub's halves, leaving every read single-source, so counter presence is only guaranteed in 4+2). All existing flex MPI tests (cylinders, duals, xhat, xfeas, xhat_multistage, extension_fields, spwindow_multisource) pass locally at -np 6.

The design doc moves the #742 diagnostic from "possible future work" into the coherence section.

Deliberately not in this PR: #749 (reduced_costs spoke at flexible ratios, Phase 7) — behavioral work on the spoke itself, better reviewed separately.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 95.34884% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.83%. Comparing base (99cd706) to head (d2edf5c).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
mpisppy/cylinders/spcommunicator.py	95.23%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #816      +/-   ##
==========================================
+ Coverage   76.79%   76.83%   +0.04%     
==========================================
  Files         177      177              
  Lines       23456    23498      +42     
==========================================
+ Hits        18012    18054      +42     
  Misses       5444     5444              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.