Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,49 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

## [Unreleased]

### Added

- **The card's compute-partition shape is now recorded, checked, and published.**
An MI300-series card can be split into independent partitions (`SPX`, `DPX`,
`QPX`, `CPX`), and splitting one trades per-request latency for aggregate
throughput. Until now nothing in a session recorded which shape a
number came from, so two runs of the same configuration on the same card in
`SPX` and in `CPX` were indistinguishable in the history — different
experiments filed under one name.<br/>
The observed mode now goes into the platform fingerprint alongside NPS, the
session report names it on partitioned runs, and the shape is published to
the environment for the benchmark entrypoint to fan work out across
partitions. That entrypoint lives outside this repository, so until it reads
them a session on a split card measures one partition rather than the total;
the recorded shape is still what stops a `CPX` number being filed as though
it were `SPX`. The published variables are set only for the scriptable
frameworks whose benchmarks can fan out; a serving session records the shape
but is handed no fan-out contract, and its report says the figure cannot be
read as an aggregate.<br/>
Two optional flags configure it. `--compute-partition-mode` **asserts** the
mode the card is already in and refuses the session if it is in another one,
or if the card cannot be read — the flag exists to catch an external set that
did not take, so an unverifiable assertion is treated as a failed one.
`--streams-per-partition` (default `2`) is how many concurrent streams go on
each partition; a value below `1` is refused rather than quietly replaced by
the default, since `0` is far more likely to be a mistake than a request.<br/>
**The optimizer does not change the mode.** Setting it is privileged and
disrupts every process holding a GPU context, which is not something an
optimization loop should do between benchmark rounds. The card must be in its
mode before `optimize` starts: the shape is checked and recorded at launch, so
a mode applied later — by the benchmark entrypoint, for instance — is too late
to be either. Nothing added here needs privilege: every probe is an
unprivileged read, and a host without `amd-smi` behaves exactly as before.<br/>
**Operator note**: launch now refuses a session whose streams provably will
not fit one partition, sized from the checkpoint's weight bytes as a lower
bound. The arithmetic costs milliseconds and the failure it replaces is an
out-of-memory crash hours in. When the checkpoint cannot be sized the session
runs and says so. The refusal applies where streams will actually share a
partition — a scriptable framework, or an operator who named the flags — and
not to a serving session that merely happens to start on a card someone else
left split. Multi-node sessions record no shape, since the readable card is
not the benchmark's.

### Changed

- **BREAKING: the EXPLORE phase is merged into FRAMEWORK_AGENT.** The chain is
Expand Down
77 changes: 77 additions & 0 deletions docs/reference/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -425,6 +425,83 @@ multi-node runs or when the Ray backend is disabled.

---

## Compute partitioning (AMD)

An MI300-series card can be split into independent partitions (`SPX`, `DPX`,
`QPX`, `CPX`). Splitting it trades per-request latency for aggregate throughput,
so a partitioned measurement is not comparable with a whole-card one.

**The optimizer does not set the mode.** Changing it is privileged, disruptive to
every process holding a GPU context, and not something an optimization loop
should be doing between benchmark rounds. The card is put in its mode before
launch — by the operator or the provisioning platform — and `optimize` only
observes what it is in, refuses a session that cannot work in that shape, and
hands the shape to the benchmark entrypoint that places work across partitions.

Two CLI flags configure this, both optional:

- `--compute-partition-mode {SPX,DPX,QPX,CPX}` **asserts** the mode the card is
already in. It is a check, not a request: if the card is in a different mode
the session is refused rather than silently measuring the wrong topology. If
the card cannot be read at all, a declared mode is also a refusal — the flag
exists to catch an external set that did not take, and an unverifiable
assertion is not a satisfied one.
- `--streams-per-partition N` (default `2`) is how many concurrent streams the
benchmark places on each partition. One stream leaves each partition idle
through the fixed per-pass cost; beyond two, on the workloads measured so far,
only queueing is added. A value below `1` is refused rather than replaced by
the default, so `0` is a usage error instead of a silent `2`.

At launch the per-stream HBM footprint is checked against one partition's
memory. A workload that provably will not fit is refused in milliseconds instead
of failing out of memory hours in. The footprint is the checkpoint's weight bytes
— a lower bound, since each stream holds its own copy of the weights, which is
why a "does not fit" verdict from it is trustworthy and a "fits" verdict proves
nothing. When the checkpoint cannot be sized the session runs with a warning.

The check only applies where streams will actually share a partition: with a
serving framework and no partition flags, the shape is recorded and nothing is
refused, because nothing in that session places work per partition and, since
whole cards enumerate before partitions, its benchmark may not even land on one.

Multi-node sessions (`--nodes >= 2`) record no shape at all. The card this
process can read is not the card the benchmark runs on, and a shape recorded from
the wrong node is the exact mislabelling this feature exists to prevent. A
declared mode there is a usage error rather than a silently unchecked assertion.

| Variable | Default | Description |
|----------|---------|-------------|
| `HYPERLOOM_PARTITION_GPU` | `0` | Which GPU's partition state describes this session. A value that is not a GPU id falls back to `0` with a warning rather than silently — the fallback reads a different card, and every number the session files afterwards would carry that card's shape. |

### Runtime hand-off

Published once at launch by the CLI and read by the benchmark entrypoint. Do not
set these by hand: they are overwritten at every launch, and clearing them first
is what stops a second session in the same shell from inheriting a shape that
was not asked for.

The first three describe the card and are published for every session on a
readable card, because `platform_fingerprint()` reads them back from here — it
runs on the crash path, where spawning `amd-smi` is not acceptable. The last two
are instructions to a benchmark that places work on each partition, so they are
published only when one will.

| Variable | Published | Description |
|----------|-----------|-------------|
| `HYPERLOOM_PARTITION_MODE` | Always | The observed mode. Also recorded in the platform fingerprint, so a result is never filed under a topology it was not measured on. |
| `HYPERLOOM_PARTITION_COUNT` | Always | Partitions the card presents in that mode. |
| `HYPERLOOM_PARTITION_CU` | Always | Compute units per partition. The entrypoint selects partition devices by matching this exactly — HIP enumerates whole cards before partitions, so an index list computed at launch would be wrong in the one case that matters, and wrong invisibly. |
| `HYPERLOOM_PARTITION_STREAMS_PER_PARTITION` | Fan-out only | Streams to place on each partition. |
| `HYPERLOOM_PARTITION_TOTAL_STREAMS` | Fan-out only | `count × streams`; the total concurrency the entrypoint should drive if it fans out across every partition. |

Only scriptable frameworks (`xdit`, `custom`) place work per partition. A serving
session is handed no fan-out instruction rather than a concurrency nothing will
drive, and passing the flags with one warns. Its shape is still recorded in the
report and the fingerprint — that is provenance, not a hand-off — and the report
states plainly that the figure cannot be read as an aggregate.

---

## Multi-node / prefill-decode (PD)

Use CLI flags for multi-node topology and prefill-decode configuration:
Expand Down
1 change: 1 addition & 0 deletions src/hyperloom/agents/robustness/role/envelope.py
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ class IntentType(str, Enum):
"model_path",
"model_name",
"model_class",
"compute_partition",
"start_ts",
"resumed_ts",
"max_minutes",
Expand Down
Loading
Loading