Umbrella follow-up from the public adversarial review of the #399/#400/#351/#386 batch (2026-07-24). That batch makes starved-rank meshes constructible and solvable (PETSc handles empty ranks fine), fixes the nearest-centroid particle mis-route, and makes the kd-tree sentinel guards honest. The next layer breaks at the first point evaluation:
uw.function.evaluate on a starved rank: points_in_domain calls get_max_radius, which does self._radii.max() on an empty array → ValueError on the starved rank while populated ranks sit in gather_data → deadlock. get_min_radius fails identically and feeds every solver's estimate_dt and penalty scaling. Needs empty-safe local reductions with defined semantics (e.g. points_in_domain short-circuits to all-False with no radii access; min/max radius return identity elements and callers reduce globally).- Consequence for field-valued metrics:
mesh_metric_mismatch skips evaluate on starved ranks (it must — see 1), so a metric containing MeshVariable data deadlocks at np>1 with a starved rank. Documented as a known limit in the function; lifts automatically once (1) is fixed.
gather_data silently strips NaN rows (utilities/_utils.py): any consumer whose local reduction produces NaN on an empty rank gets a compacted table whose row index no longer equals rank — this was the mechanism behind the particle mis-route. Either stop stripping (return the full table and let callers mask) or make the stripping opt-in and loudly documented.- Observed hazard class: when a rank-asymmetric raise occurs and user code catches it, the survivors' pending collective pairs with the next collective the failed ranks enter — both 'succeed' with garbage (observed live: a 1-row centroid table with no error). Worth a note in the parallel-patterns doc; the cure is the same as everywhere — no rank-asymmetric raises around collectives.
- Smaller members of the family:
MeshVariable's rbf path and the monotone-clamp kNN (functions_unit_system.py) index empty source arrays without a guard (SwarmVariable's rbf path IS guarded).
Related: #314 (global_evaluate np4 hang on empty-interior ranks) is likely the same family seen from the DMInterpolation side — a fix for (1) should be validated against #314's reproducer.
Underworld development team with AI support from Claude Code
Umbrella follow-up from the public adversarial review of the #399/#400/#351/#386 batch (2026-07-24). That batch makes starved-rank meshes constructible and solvable (PETSc handles empty ranks fine), fixes the nearest-centroid particle mis-route, and makes the kd-tree sentinel guards honest. The next layer breaks at the first point evaluation:
uw.function.evaluateon a starved rank:points_in_domaincallsget_max_radius, which doesself._radii.max()on an empty array → ValueError on the starved rank while populated ranks sit ingather_data→ deadlock.get_min_radiusfails identically and feeds every solver'sestimate_dtand penalty scaling. Needs empty-safe local reductions with defined semantics (e.g.points_in_domainshort-circuits to all-False with no radii access; min/max radius return identity elements and callers reduce globally).mesh_metric_mismatchskipsevaluateon starved ranks (it must — see 1), so a metric containing MeshVariable data deadlocks at np>1 with a starved rank. Documented as a known limit in the function; lifts automatically once (1) is fixed.gather_datasilently strips NaN rows (utilities/_utils.py): any consumer whose local reduction produces NaN on an empty rank gets a compacted table whose row index no longer equals rank — this was the mechanism behind the particle mis-route. Either stop stripping (return the full table and let callers mask) or make the stripping opt-in and loudly documented.MeshVariable's rbf path and the monotone-clamp kNN (functions_unit_system.py) index empty source arrays without a guard (SwarmVariable's rbf path IS guarded).Related: #314 (global_evaluate np4 hang on empty-interior ranks) is likely the same family seen from the DMInterpolation side — a fix for (1) should be validated against #314's reproducer.
Underworld development team with AI support from Claude Code