fix(geak): forward the run's actual GPU pin in the handoff

docs/components/geak.md

@@ -38,6 +38,67 @@ GEAK's `e2e_workflow` recursively drives `kernel_workflow` to author and tune th
 individual hot kernels worth fixing. See
 [Hyperloom optimization loop](../conceptual/optimization-loop.md).
 
+## GPU pinning in the handoff
+
+GEAK launches full servers out-of-process (baseline, profile, config-tuning
+validation) and writes a visible-devices mask for each one, so the handoff has
+to say which cards the run owns. Two fields carry that, in two different
+coordinate systems:
+
+| Field | Coordinate system | Value |
+|-------|-------------------|-------|
+| `gpu_ids` | HIP-level device list — HIP indexes into the ROCr-visible set | logical positions inside an *inherited* ROCr-level mask, capped at `tp` and at the mask width (`ROCR=6` → `"0"`); a HIP-level mask nested inside that slice is already logical and is forwarded instead (`ROCR=4,5,6,7` + `HIP=2,3` → `"2,3"`, i.e. cards 6 and 7); any other mask passes through uncapped (`HIP=4,5` → `"4,5"`); `0..tp-1` when the run is unpinned. Never empty — a falsy `gpu_ids` sends GEAK back to its own `0..tp-1` fallback |
+| `gpu_ids_space` | — | `"logical"` when `gpu_ids` indexes into an inherited ROCr mask, `"absolute"` when they are whole-machine ids, `"none"` when the mask is set but empty. The one field that makes the coordinate systems distinguishable from the payload alone |
+| `gpu_pin` | absolute device ids | `{"var", "value", "ids", "count", "source"}` for the winning mask, plus `"inner"` (the same shape) when a HIP-level mask is nested inside a ROCr-level one. Omitted entirely only when no mask is set anywhere, which means "whole machine visible", not "pinned to card 0"; a mask that is *set but empty* is reported with `count: 0` (zero devices visible). `value` is the mask as authored, only whitespace-trimmed and (for a YAML sequence) comma-joined, so a UUID mask can be re-exported as-is; `ids` are its numeric ids and `count` how many devices it exposes — both derived from the same effective token list, with duplicate and negative ordinals dropped, so `count >= len(ids)` always and they differ only for a (partly) non-numeric mask |
+
+The mask is resolved variable-major, over the full ROCm precedence chain in
+`common/visible_devices.py`: the ROCr-level masks (`ROCR_VISIBLE_DEVICES`, then
+its legacy spelling `HSA_VISIBLE_DEVICES`) before the HIP-level ones
+(`HIP_VISIBLE_DEVICES`, `CUDA_VISIBLE_DEVICES`, `GPU_DEVICE_ORDINAL`). Within
+each variable the process environment comes before the materialized baseline
+recipe's `benchmark.envs`.
+
+That module is also the single definition of the tuple and the mask parser that
+`orchestrator/bus/gpu_pool.py`, `orchestrator/policy/gate.py`,
+`actions/executors/_ray_serving.py` and `common/env_safety.py` use. Those layers
+read the narrower `COUNTING_VISIBLE_DEVICE_VARS` on purpose — they answer "how
+many GPUs does this process have", and widening that would change GPU accounting
+repo-wide — while this resolver answers "where is this run pinned", and a run
+pinned with a legacy spelling is really pinned.
+
+The process env comes first because the recipe's ROCR key is not evidence of a
+pin: `materialize_config_with_envs` autofills `ROCR_VISIBLE_DEVICES=0..tp-1`
+into every materialized recipe when the mask is absent or narrower than `TP`.
+A recipe ROCR value byte-identical to that default is therefore ignored, so a
+`HIP`-pinned or genuinely unpinned run is not silently re-pinned to cards
+`0..tp-1`. A recipe mask that differs from the default *is* honoured — but its
+ids are forwarded absolute, not logical, because the GEAK child inherits the
+process environment and never sees that mask.
+
+`tp` in the handoff is read from the same resolved recipe as `gpu_ids`, so the
+two cannot disagree when the materializer clamps `TP` to the visible GPU count.
+
+A consumer that re-exports `HIP_VISIBLE_DEVICES` and leaves ROCr alone should
+use `gpu_ids`; that is correct in both coordinate systems, because the child
+inherits the same ROCr mask this process runs under.
+
+A consumer that writes `ROCR_VISIBLE_DEVICES` itself must use
+`gpu_pin["value"]` — writing `gpu_ids` there resets the child to physical card
+0 regardless of the run's pin — and must then renumber: it has just made the
+device set `0..count-1` from the child's point of view, so the inner HIP mask
+is `0..count-1`, **not** `gpu_ids`. Re-applying an absolute `gpu_ids` on top of
+a ROCr mask it also wrote yields out-of-range ordinals (`ROCR=4,5` plus
+`HIP=4,5` indexes positions 4 and 5 of a two-element set). `gpu_ids_space` is
+how a consumer tells the cases apart without inferring it from `source`.
+
+`gpu_ids_space: "none"` is the third case: the mask is set but empty, so the run
+has no visible devices and no id list can be truthful. `gpu_ids` still carries
+`0..tp-1` — a falsy `gpu_ids` is read as "unset" and falls back to exactly those
+ids anyway (`interface/run_e2e.py`), so an empty string would buy nothing and
+lose the ability to say why. A consumer must treat those ids as placeholders and
+not launch on them; the orchestrator also logs the condition at ERROR, because a
+GEAK run that dies on an invalid device ordinal is otherwise unexplainable.
+
 ## GEAK documentation
 
 For detailed documentation on GEAK, see [GEAK on ROCm Docs](https://rocm.docs.amd.com/projects/geak/en/latest/).

src/hyperloom/common/env_safety.py

@@ -10,6 +10,8 @@
 
 from __future__ import annotations
 
+from hyperloom.common.visible_devices import GPU_MASK_ENV_NAMES as _GPU_MASK_ENV_NAMES
+
 import os
 import re
 from collections.abc import Mapping
@@ -208,15 +210,8 @@
 )
 
 # GPU visibility masks: setting one selects the hardware rather than tuning it.
-GPU_MASK_ENV_NAMES: frozenset[str] = frozenset(
-    {
-        "CUDA_VISIBLE_DEVICES",
-        "GPU_DEVICE_ORDINAL",
-        "HIP_VISIBLE_DEVICES",
-        "HSA_VISIBLE_DEVICES",
-        "ROCR_VISIBLE_DEVICES",
-    }
-)
+# Single definition in ``hyperloom.common.visible_devices``.
+GPU_MASK_ENV_NAMES = _GPU_MASK_ENV_NAMES
 
 # Env names an untrusted external source (reference recipe, framework-switch
 # manifest) may never set: shell-unsafe vars plus the workload/benchmark keys the

src/hyperloom/common/visible_devices.py

@@ -0,0 +1,173 @@
+# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc.
+# SPDX-License-Identifier: MIT
+"""Visible-device mask names and parsing, shared by every layer that reads one.
+
+The same tuple of mask variables and the same "split on ``,``/``;``, keep the
+first occurrence of each non-negative int" parser had accumulated five separate
+copies (``bus/gpu_pool``, ``policy/gate``, ``actions/executors/_ray_serving``,
+``common/env_safety``, ``loop/coordinator_helpers``), and their empty-mask
+semantics had already drifted apart. This module is the single definition; it
+imports nothing outside the standard library so the pure-helper layers can use
+it without dragging in the SQLite connection ``gpu_pool`` owns.
+
+Two var tuples, deliberately distinct:
+
+* :data:`COUNTING_VISIBLE_DEVICE_VARS` — what ``gpu_pool`` / ``gate`` consult to
+  answer "how many GPUs does this process have". Identical to what those layers
+  always used; widening it would change GPU accounting repo-wide. It is DERIVED
+  from the chain below by subtracting an explicit exclusion set, so the two can
+  never drift: a new var is counted unless someone names it as uncounted, and a
+  test asserts every chain member is classified.
+* :data:`VISIBLE_DEVICE_VARS` — the full ROCm pin-resolution chain, used when
+  answering "where is this run pinned". ``HSA_VISIBLE_DEVICES`` is ROCr's legacy
+  name and ``GPU_DEVICE_ORDINAL`` is the legacy HIP-level filter; a run pinned
+  with either is really pinned, and omitting them left it reported as unpinned.
+
+The ROCr-level and HIP-level groups are exposed separately because the
+distinction is load-bearing: a ROCr-level mask renumbers the devices the child
+sees (so ids inside it are LOGICAL), while a HIP-level mask indexes into
+whatever ROCr already exposed (so its ids are absolute unless a ROCr mask is
+also in force).
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+#: ROCr-level masks: these slice the device set and renumber it ``0..N-1``.
+#: ``HSA_VISIBLE_DEVICES`` is the legacy spelling; ``ROCR_VISIBLE_DEVICES``
+#: wins when both are set.
+ROCR_LEVEL_VARS: tuple[str, ...] = (
+    "ROCR_VISIBLE_DEVICES",
+    "HSA_VISIBLE_DEVICES",
+)
+
+#: HIP-level masks: these index INTO whatever ROCr exposed.
+HIP_LEVEL_VARS: tuple[str, ...] = (
+    "HIP_VISIBLE_DEVICES",
+    "CUDA_VISIBLE_DEVICES",
+    "GPU_DEVICE_ORDINAL",
+)
+
+#: Full pin-resolution precedence: ROCr level before HIP level, canonical
+#: spelling before legacy within each level.
+VISIBLE_DEVICE_VARS: tuple[str, ...] = ROCR_LEVEL_VARS + HIP_LEVEL_VARS
+
+#: Vars the capacity-counting layers deliberately do NOT read.
+#:
+#: Both are legacy aliases of a var already in the counting set, and both are
+#: honoured only when their modern spelling is absent — so counting them would
+#: not find a GPU the modern spelling missed, it would only change the answer on
+#: hosts that happen to export the legacy name. That is a repo-wide GPU
+#: accounting change, not a bugfix, so it stays out until someone makes it
+#: deliberately.
+_UNCOUNTED_VISIBLE_DEVICE_VARS: frozenset[str] = frozenset(
+    {
+        "HSA_VISIBLE_DEVICES",
+        "GPU_DEVICE_ORDINAL",
+    }
+)
+
+#: The subset the capacity-counting layers read — ``gpu_pool``, ``policy.gate``,
+#: ``actions.executors._ray_serving``. DERIVED from
+#: :data:`VISIBLE_DEVICE_VARS` rather than re-listed, so a var added to the
+#: precedence chain is counted by default and can only be left out by naming it
+#: in :data:`_UNCOUNTED_VISIBLE_DEVICE_VARS`. The two tuples cannot silently
+#: drift apart: ``test_visible_devices.py`` asserts every chain member is
+#: classified exactly once.
+COUNTING_VISIBLE_DEVICE_VARS: tuple[str, ...] = tuple(
+    var for var in VISIBLE_DEVICE_VARS if var not in _UNCOUNTED_VISIBLE_DEVICE_VARS
+)
+
+#: Every name that selects hardware rather than tuning it (superset of the
+#: precedence chain), for env scrubbing.
+GPU_MASK_ENV_NAMES: frozenset[str] = frozenset(VISIBLE_DEVICE_VARS)
+
+
+def is_rocr_level(var: str) -> bool:
+    """Does ``var`` slice and renumber the device set (rather than index it)?
+
+    Args:
+        var: A visible-devices env var name.
+
+    Returns:
+        ``True`` for the ROCr-level masks, whose member ids are logical
+        positions from the child's point of view.
+    """
+    return str(var or "") in ROCR_LEVEL_VARS
+
+
+def mask_tokens(raw: Any) -> list[str]:
+    """Split a visible-devices mask into its device tokens.
+
+    Tokens are NOT required to be numeric: ROCm accepts GPU UUID masks
+    (``ROCR_VISIBLE_DEVICES=GPU-a1b2c3,GPU-d4e5f6``), and those still say how
+    MANY devices the child will see, which is all the logical-index arithmetic
+    needs.
+
+    Args:
+        raw: A ``,``/``;``-separated mask, or an already-parsed YAML sequence.
+
+    Returns:
+        Non-empty tokens in order, duplicates preserved.
+    """
+    if isinstance(raw, (list, tuple)):
+        parts = [str(p) for p in raw]
+    else:
+        parts = str(raw if raw is not None else "").replace(";", ",").split(",")
+    return [tok for tok in (p.strip() for p in parts) if tok]
+
+
+def effective_mask_tokens(raw: Any) -> list[str]:
+    """The devices a mask actually exposes, in the order the runtime sees them.
+
+    :func:`mask_tokens` is the LITERAL split; this is the *effective* set. ROCm
+    exposes ``ROCR_VISIBLE_DEVICES="3,3,2"`` as two devices, not three, and
+    drops a negative ordinal. Deriving both the device COUNT and the forwarded
+    id list from this one function is what keeps them from disagreeing: counting
+    literal tokens inflates the count (logical index 2 of a 2-device set is
+    invalid), while re-serializing from the parsed ints deflates it.
+
+    Non-numeric tokens are kept — a UUID mask names real devices — so this
+    cannot filter out a genuinely invalid non-numeric entry; it removes only the
+    two forms that are unambiguously not extra devices.
+
+    Args:
+        raw: A ``,``/``;``-separated mask, or an already-parsed YAML sequence.
+
+    Returns:
+        Tokens with duplicates and negative ordinals removed, first-seen order.
+    """
+    out: list[str] = []
+    for tok in mask_tokens(raw):
+        if tok in out:
+            continue
+        try:
+            if int(tok) < 0:
+                continue
+        except ValueError:
+            pass
+        out.append(tok)
+    return out
+
+
+def parse_device_list(raw: Any) -> list[int]:
+    """Parse a visible-devices mask into absolute NUMERIC device ids.
+
+    Args:
+        raw: A ``,``/``;``-separated mask (``"4,5,6,7"``) or a YAML sequence;
+            ``None`` and malformed entries are tolerated.
+
+    Returns:
+        Unique non-negative ids in first-seen order; ``[]`` for an empty mask
+        and for a well-formed but non-numeric one (e.g. a UUID mask), which is
+        why callers that need a device COUNT must use
+        :func:`effective_mask_tokens`.
+    """
+    out: list[int] = []
+    for tok in effective_mask_tokens(raw):
+        try:
+            out.append(int(tok))
+        except ValueError:
+            continue
+    return out

src/hyperloom/inference_optimizer/breakdown/collectors/geak.py

@@ -11,6 +11,7 @@
 
 from __future__ import annotations
 
+from collections.abc import Mapping
 from datetime import datetime, timezone
 from pathlib import Path
 from typing import Any
@@ -614,6 +615,34 @@ def _geak_accepted_kernels_from_integrate_results(
     return accepted
 
 
+def _handoff_gpu_fields(handoff: Mapping[str, Any] | None) -> dict[str, Any]:
+    """Pull the device set + absolute GPU pin out of a handoff, if it has them.
+
+    A GEAK baseline that reads ``no_gain``/``incomplete`` because its servers
+    landed on a foreign tenant's card is otherwise indistinguishable from a
+    real result, so the breakdown records what the handoff told GEAK about the
+    run's cards (issue #1312).
+
+    Keys are inserted only when present, mirroring the writer's ``if gpu_pin:``
+    guard. An explicit ``null`` would conflate three different things: a
+    genuinely unpinned run, a pin that resolved empty, and a pre-v3 handoff on
+    disk (still produced by any session resumed from before this change).
+
+    Args:
+        handoff: The parsed ``geak/handoff.json``, or ``None``.
+
+    Returns:
+        A mapping with ``gpu_ids`` / ``gpu_ids_space`` / ``gpu_pin`` for
+        whichever keys the handoff carries; ``{}`` when it carries none.
+    """
+    out: dict[str, Any] = {}
+    for key in ("gpu_ids", "gpu_ids_space", "gpu_pin"):
+        val = (handoff or {}).get(key)
+        if val is not None:
+            out[key] = val
+    return out
+
+
 def _geak_reconstruct_from_disk(
     session_dir: Path,
     warnings: list[str],
@@ -671,6 +700,7 @@ def _load_json(p: Path) -> dict[str, Any]:
             "workload": handoff.get("workload"),
             "accepted_flags": handoff.get("accepted_flags"),
             "raw_baseline_tput": _to_float(handoff.get("raw_baseline_tput")),
+            **_handoff_gpu_fields(handoff),
         }
 
     # 2) a flushed-but-unpromoted result.json (absent or non-ok status).
@@ -1003,9 +1033,30 @@ def _rel_if_under(p: Any) -> Any:
         else:
             accepted_kernels = []
 
+    # The cards GEAK was told to use. Read from the handoff on disk rather than
+    # from ``geak_result``, which never carried them — and recorded on THIS
+    # path, not just the crash-recovery one, because the outcome that needs
+    # disambiguating (`no_gain`) is a completed run.
+    # An absent handoff is the normal shape for a run that never reached the
+    # handoff write (and for every pre-v3 session on disk), so it must not cost
+    # a stat+read or raise a warning — only a handoff that EXISTS and cannot be
+    # parsed is worth reporting.
+    _handoff_path = session_dir / "geak" / "handoff.json"
+    _gpu_fields: dict[str, Any] = {}
+    if _handoff_path.is_file():
+        _gpu_fields = _handoff_gpu_fields(
+            read_json(
+                _handoff_path,
+                default={},
+                require_dict=True,
+                on_error=lambda exc: warnings.append(f"geak: handoff read failed: {exc}"),
+            )
+        )
+
     section: dict[str, Any] = {
         "engaged": True,
         "status": status,
+        **_gpu_fields,
         # Failure provenance (None on success).
         "error_class": result.get("error_class"),
         "error": result.get("error"),