Feat/rpoornac/partition mode sweep

CHANGELOG.md

@@ -35,6 +35,36 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
   hipBLASLt, and the fusion backend. `deploy/` is also absent -- every file in
   it targets the retired repository.
 
+- **`scripts/partition_mode_sweep.py` measures which compute-partition mode a
+  workload wants.** Sets each mode on one card in turn, runs the same benchmark
+  on every partition that mode creates, sums the throughput, and restores the
+  card's entry mode on the way out — including after a failure or a Ctrl-C.
+  Modes whose partitions provably cannot hold the configured streams are skipped
+  with the arithmetic shown rather than run into an out-of-memory failure.<br/>
+  The fan-out is the substance of it. A benchmark that loads one partition and
+  ignores the rest measures a fraction of the card, which reports `CPX` as eight
+  times worse than it is; every figure here is the sum over a mode's partitions
+  with all of them loaded together, and a mode is reported only when every one of
+  its partitions returned a measurement. Partitions are selected by matching CU
+  count within the swept card's PCI bus, never by device index: `amd-smi` orders
+  by PCI address while HSA/HIP enumerates whole cards first, so on an 8-card
+  MI355X node with card 0 in `CPX` the two tools disagree about which devices the
+  partitions are — 0-7 against 7-14.<br/>
+  This is where the privileged `amd-smi set` lives, and the only place it does.
+  A card-wide mutation that evicts every GPU context is reasonable between
+  benchmarks in a script an operator ran on purpose, and unreasonable inside an
+  optimization loop that also runs agent-authored code, so `optimize` continues
+  to only read the mode. Together the two halves are a boundary: the sweep
+  chooses the shape, the session asserts it.<br/>
+  Because that set evicts work, the check standing in front of it fails closed:
+  an `amd-smi` process listing in a shape the parser does not model is a refusal,
+  not an empty one, since the only wrong answer that destroys anything is reading
+  a busy node as free. It is scoped to the card being swept, so a neighbour's
+  benchmark on a shared node no longer forces `--allow-busy` and with it the loss
+  of the guard on the target card. Every exit from a started sweep runs the
+  restore and the report, including on an error the script does not model — which
+  exits `4`, keeps the modes already measured, and still yields `3` if the card
+  could not be put back.
 - **The card's compute-partition shape is now recorded, checked, and published.**
   An MI300-series card can be split into independent partitions (`SPX`, `DPX`,
   `QPX`, `CPX`), and splitting one trades per-request latency for aggregate

README.md

@@ -86,6 +86,12 @@ feedback on how to improve Hyperloom by completing the
 - Main agent instructions: [`src/hyperloom/inference_optimizer/SKILL.md`](src/hyperloom/inference_optimizer/SKILL.md)
 - CLI entry point: `python -m hyperloom.inference_optimizer.cli optimize`
 - Operator tools: `python -m hyperloom.inference_optimizer.tools.*`
+- Compute-partition sweep: `python3 scripts/partition_mode_sweep.py` — sets each
+  AMD partition mode (`SPX`/`DPX`/`QPX`/`CPX`) on one card in turn, runs the same
+  benchmark on every partition that mode creates, sums the throughput and restores
+  the entry mode. Answers which shape a workload wants before a session commits to
+  one; `optimize` itself only ever reads the mode. Needs privilege for the set, so
+  it is a script rather than part of the loop.
 - Platform tuning audit: `python3 scripts/platform_audit.py` — checks the host CPU
   tuning that silently changes benchmark results. Judges Core Performance Boost and
   the cpufreq governor against [AMD's BIOS & Workload Tuning Guide for EPYC 9004][58011];

docs/reference/environment-variables.md

@@ -501,6 +501,52 @@ drive, and passing the flags with one warns. Its shape is still recorded in the
 report and the fingerprint — that is provenance, not a hand-off — and the report
 states plainly that the figure cannot be read as an aggregate.
 
+### Choosing the mode: `scripts/partition_mode_sweep.py`
+
+`optimize` treats the mode as fixed and asserts it. Deciding *which* mode to be
+in is a separate job, done before the session, by
+`python3 scripts/partition_mode_sweep.py`. It sets each mode on one card in
+turn, runs the same benchmark on every partition that mode creates, sums the
+result, and restores the card's entry mode on the way out.
+
+```bash
+# what it would do, nothing set
+python3 scripts/partition_mode_sweep.py --benchmark-config bench.yaml --dry-run
+
+# sweep every mode the card reports, skipping any that cannot hold the workload
+python3 scripts/partition_mode_sweep.py \
+    --benchmark-config bench.yaml --output-dir /shared/sweep \
+    --per-stream-gib 20.7 --sudo
+```
+
+The fan-out is the point rather than a detail: a benchmark that loads one
+partition and ignores the rest measures a fraction of the card, which makes
+`CPX` look eight times worse than it is. The sweep therefore launches every
+partition at once, and reports a mode only when all of its partitions returned
+a measurement — a mode with six of eight reporting is unmeasured, not slow.
+
+It publishes the same `HYPERLOOM_PARTITION_*` variables as a session, so a
+benchmark entrypoint written against the table above works unchanged under
+either. It pins each process with `ROCR_VISIBLE_DEVICES` and removes any
+inherited `HIP_VISIBLE_DEVICES`, because two masks apply in sequence and the
+second indexes into the first.
+
+The privileged `amd-smi set` lives here and nowhere else. An operator-run script
+between benchmarks is a reasonable place for a card-wide mutation that evicts
+every GPU context; an optimization loop that also runs agent-authored code is
+not. Before setting anything it refuses if a process holds a context on the card
+being swept — and only that card, since no other card is repartitioned. A
+neighbour's benchmark on a shared node is not a reason to stop. If `amd-smi`
+reports its process list in a shape the script cannot read, that is also a
+refusal rather than an assumption that the card is idle: `--allow-busy` is the
+way past both, and `--dry-run` never asks.
+
+Exit codes: `0` swept, `1` nothing measurable, `2` refused before anything
+changed, `3` swept but the card could not be restored to its entry mode, `4`
+stopped on an error it does not model. Every path out of a started sweep goes
+through the restore and the report, so a mode that fails unexpectedly costs its
+own result and nothing else.
+
 ---
 
 ## Multi-node / prefill-decode (PD)