GLM-5.3 series: engine image, example kits, and parked e2e cases - #151
Open
dorado269 wants to merge 27 commits into
Open
GLM-5.3 series: engine image, example kits, and parked e2e cases#151dorado269 wants to merge 27 commits into
dorado269 wants to merge 27 commits into
Conversation
…t moved GLM-5.3 and GLM-5.3-MXFP4 are glm_moe_dsa / GlmMoeDsaForCausalLM, field for field identical to GLM-5.2 except transformers_version, so the released engine already serves them through glm4_moe.py and this image needs no new model source. Only the bump was needed. It cost two anchor repairs, both verified in-image against lmsysorg/sglang:v0.5.18-rocm720-mi35x rather than assumed. draft_cuda_graph_dp_vote.diff applies 6 of its 7 dp_attn.py hunks on v0.5.18. Hunk 4 -- the min()-reduce -- rejects because upstream collapsed the per-field device reads into one D2H copy and renamed the tensor with it (tp0_info[:, N].min().item() -> tp0_info_cpu[:, N].min()). Losing that one hunk does not degrade gracefully, which is why it gets a port script rather than a shrug. It is the only place the gathered column is reduced back onto self.can_run_draft_cuda_graph; without it that attribute keeps the value the constructor put there, i.e. THIS RANK's answer. The vote silently never happens and the DP group splits across graph-replay and eager exactly as it did unpatched. The apply script's own header warns about this shape of failure: an inert 04 looks exactly like a working one until load. It is also why the existing dp_attn.py:can_run_draft_cuda_graph marker cannot gate it -- that identifier is present after six hunks, so the marker passes on an inert patch. patch_draft_cuda_graph_dp_vote_v0518.py re-asserts all six sibling edits before writing, and exits non-zero if any is missing. The second anchor is in the sglang_disagg set, which had no skip gate and so failed the build outright: mooncake/conn.py's typing import grew `Set` while disaggregation/common/utils.py's did not, and serving_responses.py's `background=` argument gained `and not request.stream`. Both fixes are still absent upstream, so the patches are still needed; only their anchors are stale. reanchor_sglang_disagg_glm53.sh already retargets exactly these two into a copy -- verified that both read identically on stock v0.5.18 and on the GLM-5.3 branch -- so it is reused here rather than duplicated. Its name still says glm53 because that is where it was first needed; its header now records that it is not GLM-5.3-specific. Verified in the built image: all four DSA bytecode markers present, including the hunk-4 min-reduce, and the two disagg patches applied. Signed-off-by: yihou <yihou@amd.com>
…s first commit GLM-5.3-Flash and -Flash-MXFP4 are model_type glm5_next -- a different architecture from the big pair, not a smaller version of it: hybrid KDA-linear plus DSA attention, mHC, no-RoPE MLA, a compressed indexer k-pool and a vision encoder. It exists in NO released sglang. Verified absent from lmsysorg/sglang:v0.5.18-rocm720-mi35x AND from the vendor-validated lmsysorg/sglang-rocm:v0.5.18-rocm724-mi35x-20260822, both checked inside the images. So this image fetches the model source at build time, which no other Dockerfile here does -- and that is exactly why it is a separate file rather than build args on Dockerfile.sglang: folding it in would put a fetch of unreleased source into the default build path for Kimi-K3 and GLM-5.2. These three files are adopted from the in-flight Infera PR #143, copied rather than branched from, since that PR is open, is someone else's, and its author records that the image has never been built. This one has been built and has served. THE PIN IS CHANGED from #143's 9e692c92 to c821c425, and the difference is not cosmetic. 9e692c92 is #36607's FIRST commit; five of the nine after it are the ones this image exists for, including "[AMD] Support mixed Quark MXFP4 and block-FP8 loading" and "[AMD] Enable AITER mHC for GLM on gfx95". Content, same commands across both refs: glm5_next.py 1834 -> 1942 lines, aiter references 3 -> 12, quantization/quark/quark.py 1103 -> 1172. Building on the earlier pin silently takes the pre-mHC path -- which upstream measured at 4.3-5.4x slower and which no log calls out -- and cannot load a Quark MXFP4 checkpoint at all. c821c425 is also the safest pin available rather than merely the newest: #36607 MERGED into #36507's branch on 2026-08-28, so its head is frozen. pull/36507/head is NOT a substitute -- it carried the AMD work between 08-28 and 08-30 and then LOST it in its 08-31 rebase; as of 09-01 it is back to aiter 3 / quark.py 1103. Known-good fallback: 7fa1924c. The header carries its own delete criterion -- a copy-pasteable probe and what to do once glm5_next appears in a release -- so this does not quietly become permanent. Signed-off-by: yihou <yihou@amd.com>
One row per checkpoint in the sglang PD-mixed grid, all four enable=False, plus the four model-id constants. The enable flag is what "disabled by default in CI" means here: expand_cases skips the row, so nothing new is collected -- verified, the grid still expands to the same 3 params and none of them is a GLM-5.3 id. They are parked for a resource reason, not a confidence one. Each recipe was brought up and smoke-checked on 8xMI355X; each needs 300-700 GB of weights pre-staged and four GPUs for ten-plus minutes of cold start, which no runner here has. Flipping the first field is the whole activation step, and the point of the row is that the recipe -- which took real debugging to establish -- is in the tree rather than in someone's shell history. Each row records what would otherwise have to be rediscovered: - Which IMAGE it needs. The two Flash rows require Dockerfile.sglang.glm53; the two big rows run on the stock image. Crossing them fails at CONFIG LOAD with "model type `glm5_next` but Transformers does not recognize this architecture" -- a message that names transformers and invites the wrong fix. - Why flash-mxfp4 passes --disable-shared-experts-fusion: sglang PR #36607 opened the gfx950 fusion gate without deepseek_v2's quant_blocks_shared_experts_fusion guard, so the checkpoint's BF16 shared expert lands in routed slot 288 of an MXFP4-packed FusedMoE and weight load dies on a 256-vs-512 mismatch. Upstream #37268 is the same bug on NVFP4. - Why the same flag on big-mxfp4 is insurance rather than a fix, and why it is still defaulted on: that checkpoint's shared experts are themselves MXFP4 so the precondition is absent, but upstream #25261 shows this class failing SILENTLY with wrong output when the shapes happen to line up. - That the DSA-on-ROCm env block is mandatory for the big pair: without it the model serves, returns 200s, and returns garbage. - That the vendor card's --cuda-graph-max-bs 2 --max-running-requests 2 is a concurrency-2 accuracy config and must not be copied into a benchmark arm. flash-fp8's row is honest about being the one recipe not yet validated in this tree: whether it also needs the fusion flag is marked unverified, with the specific index-count check that settles it. Signed-off-by: yihou <yihou@amd.com>
One kit with a VARIANT switch rather than four near-duplicate directories: the recipes differ in about six flags, and merging them makes the load-bearing fact impossible to miss -- GLM-5.3 is two unrelated architectures sharing a product name, and which one you serve decides which engine image you need. env.sh is the only file a site edits. worker.sh carries the tuned recipe and contains no addresses or paths; up.sh does container -> etcd -> worker -> router, waiting on health at each step; smoke.sh is six blocks each chosen because it goes red when a specific feature is SILENTLY absent, which is the failure mode this model family specialises in: - two AITER mHC lines per rank. Without them the server starts, answers correctly, and is 4.3-5.4x slower with nothing in any log saying so. - "Shared experts fusion optimization enabled." must be ABSENT -- a health check that is the absence of a line. - decode lines carrying BOTH "full token usage" and "mamba usage", because the Flash family keeps a KDA state pool as well as the paged KV pool. - a fault scan that excludes torch._dynamo/metrics_context telemetry, which appears in every healthy run, so a real traceback stays visible. The README states validation per variant rather than implying it: flash-mxfp4 and both big variants validated end to end, flash-fp8 recipe-carried but not yet validated in this kit, PD not covered at all. No PD kit ships here on purpose -- that shape has not been run for GLM-5.3, and for the big pair sglang_1p1d_glm5.2 is the validated reference. Three traps are baked into the scripts rather than left to the README: up.sh binds `realpath` output, because where the weights path crosses an NFS mount boundary binding the symlink's parent gives the container an empty directory whose failure surfaces minutes later as an unrelated-looking processor error; teardown names containers explicitly and never by pattern, and the GPU check is advisory rather than a kill, because this runs on shared nodes; and bench.sh uses sglang's own bench_serving rather than a shell fan-out, after a bash loop of 32 concurrent curls measured 350 output tok/s against an engine reporting 2398 with an empty queue. Signed-off-by: yihou <yihou@amd.com>
Two operators hit this independently, on different nodes and different ports: overriding only --listen-peer-urls leaves --initial-cluster at its default default=http://localhost:2380 while --initial-advertise-peer-urls is derived from the detected host, and etcd exits 1. A script that hardcodes 2380 never sees it, which is exactly why it bites the first time you land somewhere 2380 is taken. The three peer flags already moved together here; this records why. Signed-off-by: yihou <yihou@amd.com>
…forking it GLM-5.3 (big) is glm_moe_dsa and its config.json is identical to GLM-5.2's field for field except transformers_version, so the engine recipe is not similar to the GLM-5.2 one -- it IS the GLM-5.2 one. sglang_1p1d_glm5.2/engine/leg.sh already carries it, validated on two clusters and both fabric types. Copying those ~600 lines here to change a model path would create a second source of truth that drifts the first time either is fixed, so this ships wrappers only and points KIT_DIR at that kit, exactly as its own cluster/*.sh do. Two shapes. cluster.2node.sh is the validated topology with GLM-5.3 weights substituted. cluster.singlenode.sh splits one 8-GPU box into TP4 prefill on cards 0-3 and TP4 decode on 4-7 -- worth having because it makes PD testable without a second node, and because the mission explicitly allows it. Neither has been run yet, and the README says so per shape rather than implying readiness. The single-node one carries a real unknown that is called out rather than glossed: both legs share a host, so the KV handoff is a loopback RDMA transfer, and whether mooncake does that -- and at what speed -- is not established. That matters because the failure mode here is silent: mooncake falls back to a transport that works and is merely 5-20x slower, and nothing raises. smoke checks MC_FORCE_TCP and "GID is NULL" in both leg logs for exactly that reason. Fabric preconditions ARE verified first-hand on the reference node: 8 ionic devices all PORT_ACTIVE, ib_peer_mem loaded (so registration mode A, the best of the three), libionic 54.0-187-1 = ABI 4 in the image, and ibv_devinfo reporting 8 HCAs from inside the container. The README records the trap found while checking that: without --device=/dev/infiniband a container on that host reports ZERO HCAs, which is indistinguishable from an ABI mismatch and is the same reading that means RDMA has silently degraded to TCP. GLM-5.3-Flash is deliberately out of scope -- different architecture, separate question, no claim made. Signed-off-by: yihou <yihou@amd.com>
All three were measured on this stack today, and each is the kind of thing that is invisible until it costs you a run. CACHE_REPORT, default off, must be on for any cache-sensitive workload. Without --enable-cache-report the server answers normally and simply reports nothing: usage.prompt_tokens_details comes back null through the router and the engine logs "#cached-token: 0". That is indistinguishable from a genuine 0% hit rate, so an agentic replay reads as "the cache never worked" when what failed was the counter. Verified by sending the same prompt twice through a live router and watching both come back uncached. The KDA-pool clamp is real and it fires. On every rank at TP4: max_running_requests is capped to 200 by the mamba state cache (max_mamba_cache_size=1000, 5 state slots per request) The note now says to read the RESOLVED value out of the worker log rather than out of server_args -- server_args records what was REQUESTED, and reading it instead is exactly how one bring-up concluded the clamp had not fired when it had. smoke.sh surfaces the resolved cap as its own line so the next operator sees it without knowing to look for it. Decode CUDA-graph capture costs 15.4-17.4 GB per rank at TP4, against ~1.4 GB measured at TP8 on the same model -- fewer ranks means each one's graphs cover a larger shard. Anyone budgeting VRAM from a TP8 figure at TP4 will be surprised by an order of magnitude, so the comment says not to carry it across. Also: big-* now passes --max-running-requests explicitly rather than leaving it to the engine's memory-derived default, which varies with whatever VRAM happens to be free -- so two runs of the same recipe on differently-loaded nodes get different admission limits, and the benchmark then measures the limit. Signed-off-by: yihou <yihou@amd.com>
Under DP-attention the engine prints PER-RANK values at startup while /get_server_info reports the global ones: ask for --max-running-requests 256 and --chunked-prefill-size 65536 at dp8 and the startup line says 32 and 8192. That is 256/8 and 65536/8 -- a division, not a clamp -- but the line reads exactly like a clamp, and it already sent one reader looking for a cap that was not there. It also explains an apparent internal inconsistency in a reference packup, where the prose said 65536 and the server dump said 8192: both correct. Separately, --cuda-graph-max-bs is deprecated on v0.5.18 in favour of --cuda-graph-max-bs-decode. It still resolves, so nothing breaks today, but the vendor recipes and our earlier scripts emit the old spelling and will break on the next base bump. This kit already uses the current flags. Signed-off-by: yihou <yihou@amd.com>
MoEGate.__init__ allocates e_score_correction_bias as bf16 whenever a quant_config is present and _use_aiter, and biased_grouped_topk_gpu casts it down again at the aiter call. GLM's bias is a narrow band at a large offset, so bf16 cannot hold it. Measured on the checkpoints themselves: GLM-5.3 and GLM-5.3-MXFP4 collapse 238 distinct fp32 values to 8, and GLM-5.3-Flash-MXFP4 collapses 282 to 11. noaux_tc selects experts by sigmoid(logits) + bias, so this reorders top-k routing. Both edits are applied together or not at all. aiter's launcher dispatches on gating_output.dtype() and then reinterpret_casts the bias pointer to that same scalar_t without checking the bias tensor's own dtype, so an fp32 bias under a bf16 gating tensor is read as fp32 bytes reinterpreted as bf16 -- applying the deepseek_v2.py half alone is worse than the defect. The gate is not upstream sglang#37133's. That PR tests the architecture strings, which is False for the whole Flash family: Glm5NextForConditionalGeneration passes config.text_config down, and Flash's text_config has architectures: None. This predicate is a union of moe_router_dtype == "float32" (declared by all four GLM-5.3 configs and ignored entirely by sglang), model_type, and upstream's arch test kept as a superset. Checked against all 38 checkpoints on this host: 8/8 GLM keep fp32, 30/30 non-GLM byte-identical. Carried, not applied. No Dockerfile copies or runs patches/sglang_carried/, which is why the script is not in patches/sglang_rocm/ beside its siblings: those directories are consumed by an unconditional *.py loop, so a file placed there is wired into every image that copies it. VERIFIED AS TEXT AND AS PREDICATE LOGIC ONLY. It applies to copies of v0.5.18, c821c425 and the live source from infera/engine-sglang:v0518-glm53 (byte-identical result to the patched v0.5.18 tree), is idempotent, and both drift cases exit 1 having written neither file. It has never been executed on a GPU, and no accuracy or throughput delta has been measured. Wiring it mid-campaign would make our arm the only one running the fixed router and destroy the like-for-like property the alignment ratios rest on. Signed-off-by: yihou <yihou@amd.com>
Three updates, all from runs on 8xMI355X/gfx950 today. 1. big-mxfp4 carries numbers rather than a green tick: the TP8+DPA+MTP fixed-length sweep lands at 0.92 / 1.06 / 0.89 / 1.10 x the GLM-5.2 MIX baseline at concurrency 1/8/16/24. 2. flash-fp8 loads and serves. More usefully, the open question of whether it needs --disable-shared-experts-fusion is answered: it does not. It was brought up with fusion ENABLED and loaded all 62 shards cleanly, and the reason was predicted beforehand from the checkpoint index (129 .weight / 129 .weight_scale_inv, uniformly block-FP8, so the mixed-precision precondition is absent). That makes it the control arm proving the gfx950 fusion path itself works -- what #36607 shipped unguarded is the decision to use it. 3. bench.sh documents a limitation of its own shape. At the default OSL 320 the output is clean; at 3300 it is 40-65 % repetition loops and at 17000 it is 96-100 %, measured with a 10-gram check on both GLM-5.3 and GLM-5.2. The tok/s arithmetic is unaffected -- loopers were measured to decode at the same speed as clean requests -- so what is wrong is what the tokens are, not how many per second. The cause is NOT established: the leading candidate is this script's own --dataset-name random with no --apply-chat-template, and a second is the ROCm silent greedy fallback in EAGLE verify. Both were present in every arm measured and they are not mutually exclusive. Recorded because acceptance length does not detect this -- an aggregate of 2.96 with 1.25 % at the 4.00 ceiling was measured while 54 % of the same requests were looping -- so a user of this kit has no signal unless told. Signed-off-by: yihou <yihou@amd.com>
The README said the same-host KV handoff is a loopback RDMA transfer and that
the risk is silent slowness. Both are wrong, and the correction changes what a
first bring-up should check.
Read from the mooncake tree and build cache in the shipped image: the pinned
commit 01d1eb2a IS the single-node disaggregation commit (rdma+hip
multi-protocol segments, validated by its author on MI355X). The image builds
USE_HIP=ON with ENABLE_MULTI_PROTOCOL=ON, auto_discover composes hip on top of
rdma so the segment advertises "rdma,hip", registration fans out to both, and
selectTransport routes per request by priority hip 4 > rdma 2. KV therefore
moves over HIP IPC across XGMI with no NIC in the path. Note MOONCAKE_PROTOCOL
defaults to "rdma" and does NOT select the transport on this build.
The real risk is different and is now stated: the kit gives the two legs
disjoint HIP_VISIBLE_DEVICES, so neither can see the other's GPUs. Whether
hipIpcOpenMemHandle imports across that boundary is UNKNOWN and deliberately not
predicted. If it fails it fails loudly at transfer time ("Failed to get kvcache
from prefill instance"), so single-node PD breaks rather than silently
degrading -- provided hip is installed. A probe that settles it in seconds
without loading weights is included.
Also documents two properties of the logging that the existing smoke check
missed: the TCP fallback has no success log at all (detect it by requiring the
positive rdma line), and no log line anywhere reports which transport a transfer
actually used. MC_FORCE_TCP and 'GID is NULL' are cross-host signals and do not
cover the single-node path; the missing check is requiring 'HIP transport
installed for intra-node GPU P2P' in both leg logs.
Signed-off-by: yihou <yihou@amd.com>
…IPC unknown Both negative, recorded so the next person does not repeat them. The ROCm 7.2 header says hipIpcOpenMemHandle can attempt to enable peer access and points at hipDeviceCanAccessPeer to test it -- but that call takes visible ordinals, and under disjoint HIP_VISIBLE_DEVICES the importer cannot name the exporter's device at all. The doc is silent on that case. Mooncake's own three HIP harnesses are all single-process and single-device, and grep -rn HIP_VISIBLE_DEVICES across that repo returns nothing -- so the pinned commit's 'prefill GPU0 / decode GPU1' validation is not reproducible from the tree, and its test suite does not exercise the configuration this kit ships. Together these are the argument for running the two-process probe rather than reasoning further about it. Signed-off-by: yihou <yihou@amd.com>
Two measurements close two open questions in the kits. flash-fp8 now carries numbers: 99.70 tok/s at conc 1 and 456.68 at conc 8 (isl 7400 / osl 320, TP4, engine port). Against the flash-mxfp4 arm's 111.0 and 561.0 on the identical shape and harness, MXFP4 is 1.11x and 1.23x faster. That is the first direct FP8-vs-MXFP4 comparison on this architecture at fixed topology, and it says the quantization is not paying for itself in memory alone. The 1P1D single-node IPC unknown is answered by measurement rather than left open. Two processes in one container with disjoint HIP_VISIBLE_DEVICES: the exporter wrote a byte pattern to its cuda:0, the importer -- which cannot see that physical GPU -- imported the handle and read the pattern back correctly. So the kit's disjoint-GPU topology is fine and the --base-gpu-id fallback is not needed. Worth recording how that was nearly a false positive: the first attempt returned IMPORT OK and was not believed, because the handle records device index 0 while the importer's own ordinal 0 is a different physical GPU -- a purely local mapping would also return OK. Only the byte pattern distinguishes them. Anyone repeating this should check the data, not the return code. Also noted: torch.cuda.cudart() does not expose cudaIpcGetMemHandle in this build; the storage IPC path is what carries HIP IPC handles here. Both README validation tables now state what was measured rather than what was expected. Signed-off-by: yihou <yihou@amd.com>
Three defects found by source read, before any hardware. As shipped, `bash cluster.singlenode.sh up` could not work, and one of the three put the deployment in the silent-slow case the README promises is loud. 1. Both GLM-5.3 wrappers exported IMAGE, but engine/up.sh and common.sh require INFERA_IMAGE (the GLM-5.2 wrappers export that name; IMAGE is preflight_rdma.sh's). `up` died at its first require_env, before a container was started. Export both, and add INFERA_IMAGE to the placeholder guard. 2. leg.sh's KV_PUB_PORT/KV_SNAP_PORT were never forwarded by up.sh, and nothing a wrapper exports reaches leg.sh otherwise -- on() runs a fresh remote shell. Two legs on one host therefore both bound 5557/8801 and the second died at bind, the same "port_base at N is not available" common.sh's reap() warns about across restarts. Forward them per leg, defaulting to leg.sh's own values, and give the two legs distinct values in the single-node wrapper only. 3. leg.sh hardcoded MC_DISABLE_HIP_TRANSPORT=1 with no override, which disables the transport the single-node shape depends on. With hip absent the local segment advertises "rdma" only and KV between two legs on ONE host takes loopback RDMA -- silently, since there is no log line for that fallback and a same-host transfer never touches a GID, so neither the MC_FORCE_TCP nor the GID-is-NULL check covers it. Make it overridable and set it to 0 in the single-node wrapper. THE REASON FOR THE ORIGINAL DISABLE IS UNKNOWN. It was deliberate and we could not establish why, so the default stays 1 and the two-node path does not move by a byte. If you disabled it for a reason we have missed, say so -- this change is scoped to the single-node wrapper precisely so it can be objected to cheaply. Supporting measurement, on this hardware: HIP IPC works across disjoint HIP_VISIBLE_DEVICES. An importer that could not see the exporter's physical GPU mapped its memory and read back the correct byte pattern, so the disjoint-GPU topology is not an obstacle and the --base-gpu-id fallback is not needed. A bare "IMPORT OK" would not have shown this -- the handle records device index 0 and the importer's own ordinal 0 is a different physical GPU, so a local mapping also returns OK. The data pattern is what rules that out. Verified without hardware, against a stub engine/up.sh: the single-node wrapper resolves to KV 5557/8801 + 5558/8802 and MC_DISABLE_HIP_TRANSPORT=0, while the two-node wrapper still resolves to 5557/8801 on both legs and MC_DISABLE_HIP_TRANSPORT=1. All four scripts pass bash -n. Required check once this runs: `HIP transport installed for intra-node GPU P2P` must appear in BOTH leg logs, and `hipIpcOpenMemHandle failed` in neither. Signed-off-by: yihou <yihou@amd.com>
Measured on smci355-ccs-aus-n01-33 with preflight_rdma.sh mode, which
returns registration mode A (peer-mem present, 8 x ionic 400 Gb/s, KV
pool full, nothing pinned, MC_GID_INDEX=1 on a routable GID).
Two values that preflight asks for and the wrappers were not supplying:
RDMAV_FORK_SAFE. Recommended in all three of preflight's modes, and
engine/leg.sh honours it only when it is passed in
([ "${RDMAV_FORK_SAFE:-0}" = "1" ]) -- so with the wrapper silent it
never reached the engine. Defaulted to 1 in the SINGLE-NODE wrapper
only. Deliberately not defaulted on for two-node: that shape was
validated end to end without it, and a validated path should not move on
the strength of a recommendation rather than a measurement. Left there
as a commented line with that reasoning.
RDMA_IB_DEVICES. The single-node placeholder read <ionic_0,ionic_1,...>,
which is the cross-node recipe. preflight states the rule for this shape
directly, under mode A: "all active rails (cross-node; mooncake pairs by
GID subnet). Single-node loopback: pin ONE device on both legs instead."
Placeholder and comment now say one device.
That second one interacts with the hip transport enabled in the previous
commit, and the combination is deliberate. preflight emits
MC_DISABLE_HIP_TRANSPORT=1 in all three of its modes and models
single-node PD as loopback RDMA, which contradicts this kit's README --
so there are now two independent sources disagreeing about the intended
single-node transport, and that disagreement may be the unestablished
reason behind leg.sh's original hardcode. Running hip ON with a
correctly pinned single rail covers both readings: if hip installs, KV
goes over XGMI and the rails do not matter; if it does not, the fallback
is the loopback RDMA preflight recommends rather than a misconfigured
eight-rail one. `HIP transport installed for intra-node GPU P2P` in the
leg logs distinguishes the two, and either outcome is a result about the
single-node shape.
Verified against a stub engine/up.sh, no hardware: two-node resolves
RDMAV_FORK_SAFE unset and MC_DISABLE_HIP_TRANSPORT=1 with KV 5557/8801
on both legs -- unchanged from before either commit. Single-node
resolves RDMAV_FORK_SAFE=1, MC_DISABLE_HIP_TRANSPORT=0, KV 5557/8801 and
5558/8802. Both pass bash -n.
Also corrects a claim I made earlier from an inherited note rather than a
test: common.sh's `docker run <etcd-image> etcd --flags` form is fine on
this image -- `docker run --rm quay.io/coreos/etcd:v3.5.14 etcd
--version` prints "etcd Version: 3.5.14". etcd is on PATH. The separate
peer-port collision on 2380 was real when measured, but that port is
free on this node now; it returns only if the foreign etcd does.
Signed-off-by: yihou <yihou@amd.com>
The single-container probe left an untested variable: real PD legs are two containers, not two processes in one. Repeated across two containers with the importer on --ipc=host, same disjoint HIP_VISIBLE_DEVICES split, same byte pattern -- CROSS-CONTAINER IMPORT OK, READ BACK matches. So the disjoint-device topology is safe at both levels and the --base-gpu-id fallback is not needed. Caught by flash-fp8-0529, which noticed the shape mismatch in the original probe spec. Limits unchanged and still stated: cards 4,5 vs 6,7 rather than the literal 0-3 / 4-7, 1 MiB buffer, one handle, and it exercises the raw HIP IPC call rather than mooncake's registration path that wraps it. The probe says the door is unlocked; only 'HIP transport installed for intra-node GPU P2P' in both leg logs says anyone walked through it. Signed-off-by: yihou <yihou@amd.com>
The kit said PD for GLM-5.3-Flash was a question it made no claim about. It now has an answer, from a link-by-link read of the pinned ref c821c425. The reason to expect it broken: glm5_next keeps two pools, paged KV and a KDA recurrent-state pool, while PD hands off after prefill. A decode leg starting its linear-attention layers from a zero state fails as subtly wrong output, not as a crash. That premise does not hold. sglang's PD path is not KV-only -- it carries a generic state-component mechanism, StateType.MAMBA is one of its members, and glm5_next resolves onto the pool that mechanism reads. Ten links recorded with file:line, from hybrid_arch.py recognising the model type through disaggregation/utils.py duck-typing on get_state_buf_infos() to mooncake/conn.py iterating state_types. No guard refuses the combination, and that absence was grepped rather than assumed. So the worry relocates rather than disappearing: from 'not transferred' to 'transferred, but unverified'. Whether the KDA buffers survive the round-trip bit-exact on gfx950 is untested, and that is still the failure mode that produces wrong output with nothing logged. Two checks are recorded in order -- a runtime confirmation that StateType.MAMBA lands in kv_args.state_types, then the decisive one: same prompt to Flash MIX and Flash PD, greedy, diff the token sequences. The MIX reference side is already validated. Three open upstream items in the same area are cited, including #37276 on PD + mamba + speculative decoding, which is why running Flash PD with MTP off sidesteps a known bug. Signed-off-by: yihou <yihou@amd.com>
The single-node section builds its load-bearing unknown on the two legs having disjoint HIP_VISIBLE_DEVICES. As shipped they had identical ones: up.sh forwards a fixed list of per-leg variables through on(), which runs a fresh remote shell, and GPUS was not among them -- so both legs fell through to leg.sh's seq default and landed on the same four cards. The IPC probe answered a real question; that question was simply never reached. Recorded here rather than only in the fix, because of how it failed. The prefill leg died with 'Raise --mem-fraction-static above 0.773' -- arithmetically correct and diagnostically wrong. Taking the engine's advice would have let two legs coexist on four cards and produced a deployment that ran, with every subsequent number meaningless. The README's existing OOM note covers the failure direction but not this form. So the check to run before trusting any memory error is now stated: base_gpu_id must differ between the leg logs (0 and 4). Cheaper and less ambiguous than reading VRAM, and it is what distinguishes a tuning problem from a topology one. Also corrects the leg.sh line reference from :153 to :159. Signed-off-by: yihou <yihou@amd.com>
…r-leg interface
One design error with six faces. The GLM-5.2 kit's interface is strictly
per-leg -- PREFILL_DPA/DECODE_DPA, PREFILL_MTP/DECODE_MTP,
PREFILL_KVD/DECODE_KVD -- and engine/up.sh forwards exactly the names it
knows, because on() runs a fresh remote shell and nothing else reaches
engine/leg.sh. The GLM-5.3 wrappers invented simplified single knobs
(MTP, DPA, GPUS, EXTRA_ENGINE_ARGS) that the kit reads under no name at
all. Each silently did nothing and fell back to a plausible default, so
none produced an error and two would have produced a believable
benchmark number against a configuration the wrapper file claims is
different.
What each actually did, measured on a live bring-up rather than inferred:
MTP=0 -> decode leg launched with mtp=1 (up.sh default
DECODE_MTP:-1). Caught in flight from the
resolved-args line; the run was killed 3 min in.
GPUS -> PREFILL_GPUS/DECODE_GPUS read by nobody, so
leg.sh:26 fell back to `seq 0..TP-1` on BOTH
legs. Both loaded onto GPUs 0-3: measured
263.8 GB per card on 0-3 with 4-7 at 0.3 GB.
The prefill leg then died on "Loaded weights
leave no GPU memory for the KV cache under
--mem-fraction-static=0.7", which reads as a
GMU tuning problem and is not one -- raising it
would have produced two legs sharing four cards.
The decode leg meanwhile came up healthy and
registered in etcd, so a router would have
found one worker and served: a "PD deployment"
that is quietly a single aggregated leg.
EXTRA_ENGINE_ARGS -> leg.sh hardcoded EXTRA_ARGS=(--enable-cache-report),
so --disable-shared-experts-fusion never reached
the engine.
DPA=1 -> read by nobody, but up.sh's defaults are
PREFILL_DPA:-0 / DECODE_DPA:-1, which is the
intended configuration. Silently CORRECT by
luck. Worse than visibly broken: DPA=0 for a
single-variable round would have produced dp8
anyway with nothing said.
INFERA_IMAGE, KV ports -> fixed in 4493e33.
Fixes, all preserving today's behaviour where a path is already validated:
up.sh forwards GPUS per leg with ${PREFILL_GPUS:+GPUS=...}, injection
only when set. NOT a :- default: that would push a literal GPU list
into the two-node path, where leg.sh's `seq 0..TP-1` is correct and
stays correct if TP changes.
up.sh forwards EXTRA_ENGINE_ARGS through COMMON_ENV, quoted so several
flags survive as one value. leg.sh appends it unquoted to EXTRA_ARGS so
they word-split. Empty when unset, so the GLM-5.2 kit's own wrappers --
which do not define it -- are unaffected.
Both GLM-5.3 wrappers now export the per-leg names the kit reads, with
the single knob kept only as a convenience seed (PREFILL_MTP:-$MTP).
cluster.2node.sh's PREFILL_GPUS/DECODE_GPUS are now UNSET rather than a
literal 0..7. Now that up.sh forwards them, pinning eight cards would
silently contradict TP=4; unset keeps leg.sh's TP-derived default.
Verified by running the real engine/up.sh with SSH_CMD=echo, so every
per-leg command is printed instead of executed -- no node, no container,
no GPU:
single-node prefill GPUS=0,1,2,3 decode GPUS=4,5,6,7
MTP 0/0, DPA 0/1, KV 5557/8801 + 5558/8802,
MC_DISABLE_HIP_TRANSPORT=0, EXTRA_ENGINE_ARGS forwarded
two-node GPUS absent from both legs (leg.sh default applies),
MTP 0/0, DPA 0/1, KV 5557/8801 both, hip disable 1
All four scripts pass bash -n.
For the record: this is the fifth and sixth defect of this class in a kit
that had never been run end to end in this shape, which is what "not yet
run" means rather than a criticism of the kit. The GLM-5.2 wrappers have
no dead exports -- all 28 are read -- so the validated two-node GLM-5.2
runs took their configuration and nothing here casts doubt on them. The
defect is ours. Static greps under-detect it: up.sh contains the literal
string "MTP=${PREFILL_MTP:-0}", so a word-boundary search for MTP matches
an occurrence that is being assigned to, not read. The reliable test is
to launch and read the resolved-args line.
Signed-off-by: yihou <yihou@amd.com>
Corrects a check this file recommended one commit ago (e4ab8fe). base_gpu_id is an index into the VISIBLE device set, and HIP_VISIBLE_DEVICES=4,5,6,7 renumbers the decode leg's devices to 0-3 -- so it reads 0 on both legs when the split is broken and 0 on both legs when it is correct. It does not discriminate at all, and following the previous advice would have produced a confident all-clear on a broken topology. The VRAM read is the check. It is more expensive, it is the only unambiguous one here, and 'more expensive' was the wrong reason to look for a substitute. Caught by the operator running the relaunch, who checked the proposed check against a now-correct deployment rather than assuming it worked. Signed-off-by: yihou <yihou@amd.com>
Note 3 taught two failure directions for --mem-fraction-static, both under load: prefill activation OOM at low token usage means lower it, decode retract at high token usage means raise it. The single-node bring-up hit a third form that neither covers, and in which the engine's own suggested fix is a trap. A leg aborts during startup profiling, weights loaded, no request served: ValueError: Loaded weights leave no GPU memory for the KV cache under --mem-fraction-static=0.7. Raise --mem-fraction-static above 0.773 0.773 is arithmetically correct -- it is derived from the memory actually free at that instant. It is diagnostically wrong, because the reason there is none is that the other leg is on the same cards. Measured: GPUs 0-3 at 263.8 GB each, 4-7 at 0.3 GB. Following the advice yields a WORKING deployment on the wrong topology, with every subsequent number meaningless and nothing logged to say so. The note now says to check the cards before touching the knob, names the two checks that do NOT work (--showmemuse VRAM%, which does not fall when memory is released; and per-leg base_gpu_id, which is an index into the visible set and reads 0 on both legs whether the split is broken or correct -- see 635fc51), and gives the discriminator between all three forms: this one aborts at startup profiling, the classic prefill form aborts under load at low token usage, the decode form retracts under load at high token usage. Written from the bring-up that hit it. The underlying GPU-split cause is fixed in 1b5ea46; this note is for the next person whose legs collide for some other reason. Signed-off-by: yihou <yihou@amd.com>
leg.sh:60 exports MC_DISABLE_HIP_TRANSPORT=1 and unsets MC_ENABLE_HIP_TRANSPORT. Neither name exists in the shipped mooncake binary -- exact-match against mooncake/engine.*.so finds MC_DISABLE_HIP, MC_USE_HIP_IPC and MC_FORCE_TCP, and zero matches for either _TRANSPORT spelling. Confirmed behaviourally, which is the stronger half: a run launched with MC_DISABLE_HIP_TRANSPORT=1, verified present in the process environment via /proc, still logged 'HIP transport installed for intra-node GPU P2P' 4x per leg -- identical to a run without it. So the line has never had an effect in either path. It is not evidence that anyone deliberately disabled hip, which is likely why no reason for it could be established: there was no effect to reason about. INFERRED that the author intended to disable hip and used a name that does not exist. Practical consequence for anyone testing the transport: an A/B must set MC_DISABLE_HIP. Using the _TRANSPORT spelling yields a guaranteed-zero differential that reads as a null result rather than a broken experiment -- the same trap as base_gpu_id, one level up. The discriminator to check is 'HIP transport installed for intra-node GPU P2P' going 4/4 to 0/0. Signed-off-by: yihou <yihou@amd.com>
…ISABLE_HIP
leg.sh has set MC_DISABLE_HIP_TRANSPORT=1 and unset
MC_ENABLE_HIP_TRANSPORT since it was written. Neither name exists in the
shipped mooncake build, so neither line has ever had an effect -- in the
two-node shape or the single-node one.
Exact-match against mooncake/engine.cpython-310-x86_64-linux-gnu.so in
infera/engine-sglang:v0518-glm53:
MC_DISABLE_HIP 1 <- the real knob
MC_DISABLE_HIP_TRANSPORT 0
MC_ENABLE_HIP_TRANSPORT 0
MC_USE_HIP_IPC 1 <- a second gate, uninvestigated
MC_FORCE_TCP 1
The behavioural evidence came first and is the stronger half. A
single-node pair launched with MC_DISABLE_HIP_TRANSPORT=1 -- confirmed
present in /proc/<pid>/environ for both engine processes -- still logged
"HIP transport installed for intra-node GPU P2P" 4 times per leg, with
zero install failures, exactly as the run with it set to 0. The strings
table only names the reason.
Consequences, including for a claim made earlier in this branch:
- The single-node shape was NEVER at risk of silently falling back to
loopback RDMA through that knob. Any statement that "as shipped, KV
would have moved over loopback RDMA" is WRONG and is retracted here.
hip installs regardless.
- Making MC_DISABLE_HIP_TRANSPORT overridable (earlier on this branch)
is therefore NOT a fix. It makes a dead knob configurable and changes
nothing. It is kept only so it reads as the pair to the live name.
- INFERRED: whoever wrote the original hardcode probably intended to
disable hip and used a name that does not exist. That would explain
why no reason for the line could be established -- there was no effect
to reason about. It also retires the apparent conflict with
preflight_rdma.sh, which recommends the same dead name.
The fix is to forward the name that works. MC_DISABLE_HIP is passed
through COMMON_ENV in up.sh and into the leg's docker exec env in
leg.sh, both with ${VAR:+...} so an unset variable injects nothing and
the two-node path is byte-identical. cluster.singlenode.sh exports it
unset by default -- hip on is what that shape wants.
Verified by running the real engine/up.sh with SSH_CMD=echo, no node:
MC_DISABLE_HIP=1 appears exactly twice (once per leg); unset, it appears
zero times on both the single-node and two-node wrappers. All four
scripts pass bash -n.
Practical rule, recorded in both files: a hip-on/hip-off A/B MUST use
MC_DISABLE_HIP and MUST confirm the discriminator flipped -- "HIP
transport installed for intra-node GPU P2P" going from 4 per leg to 0 --
before benchmarking. With the dead name it produces a guaranteed-zero
differential that reads as a null result rather than a broken
experiment. That is the same shape as the base_gpu_id check retired in
635fc51: a control that returns the same value in both states.
Signed-off-by: yihou <yihou@amd.com>
… selection not installation Three corrections to the transport section, all from a re-read at the commit that actually built the shipped .so. Two-node PD never uses hip regardless of configuration. selectTransport calls isHipReachableTarget() and skips hip buffers for any off-host target, with the in-source rationale that the intra-node fast path and the cross-node path should work from one multi-protocol segment 'without requiring the operator to set MC_DISABLE_HIP'. So every hip question in this kit is a single-node question and none of it applies to cluster.2node.sh. The file previously implied otherwise. Installation is unconditional -- transfer_engine_impl.cpp:402-414 is a bare #ifdef USE_HIP. Selection is what MC_DISABLE_HIP gates, at multi_transport.cpp:489, demoting hip from priority 4 to 0 so rdma wins for the device KV pool. That distinction is why 'HIP transport installed for intra-node GPU P2P' cannot serve as a hip-off discriminator: it is an install-time log and the variable acts at selection time, so it reads 4/4 in both states forever. A correct hip-off deployment was discarded unmeasured on the strength of that non-flip. Verifying such an arm needs MC_DISABLE_HIP=1 in /proc/<pid>/environ on both legs plus the source read; no log line will confirm it. Also records that MC_DISABLE_HIP_TRANSPORT -- which leg.sh:60 sets -- and MC_ENABLE_HIP_TRANSPORT are absent from the binary entirely. Two dead names, one live name whose effect is invisible to the natural check. Signed-off-by: yihou <yihou@amd.com>
dorado269
requested review from
JohnQinAMD,
jiejingzhangamd,
limou102 and
xiaobochen-amd
as code owners
September 3, 2026 04:48
The section established that the KDA state is transferred and read as though a Flash PD run were planned. It is not, and the file should not imply otherwise. Retitled and headed with what the section actually is: a source read showing the shape is possible, never a validation, with no Flash PD deployment ever brought up. Recorded so the next person starts from the code reads instead of repeating them, and so 'we did not do this' is not read as 'this does not work'. Signed-off-by: yihou <yihou@amd.com>
Raised by an automated review of the packup scripts; the same patterns are in the live kit, which is what people will actually run. The scripts use --network=host and bind on all interfaces. That is fine for a single-tenant benchmark box and wrong for a shared one, and the reference cluster is shared -- three unrelated users logged into one of these nodes during a single day of this work, and a colleague's job claimed all eight GPUs of another without notice. etcd is the one that matters: unauthenticated, on 0.0.0.0, holding the worker discovery keys that tell the router where to send traffic. The router and KV event sockets are also raw. Left as validated rather than quietly re-bound, because changing a bind changes a recipe that was measured -- but the README now says so explicitly and gives the fix, which is cheap since router and workers are colocated. Also records that --trust-remote-code is not removable (glm5_next is in no released transformers), so the exposure is the checkpoint directory rather than the flag: /apps/data/models is a shared NFS mount, and anyone who can write there can execute code in your container. Signed-off-by: yihou <yihou@amd.com>
…that candidate The header named a missing chat template as the leading candidate for long-output repetition, on the strength of apply_chat_template=False in the benchmark args dump. That reading is wrong and the candidate is eliminated, not merely untested. On this backend the flag is parsed and never consumed -- grep for it in sglang's benchmark/serving.py returns one hit, an assignment that fires only for the image/mmmu datasets. And --backend sglang-oai-chat posts messages to /v1/chat/completions, which applies the model's chat template server-side by definition. The template has been applied in every arm all along. A flag that is parsed but unread reads exactly like a setting, which is why the args dump was misleading rather than wrong. What remains is stated: the prompt CONTENT is synthetic seeded-ShareGPT filler, which is a different thing from missing framing, and the ROCm greedy fallback in EAGLE verify -- noting that every arm measured as degenerate ran MTP ON, so that path was live in all of them. The stale advice to add --apply-chat-template for defensible long-OSL numbers is replaced with a note that it will not help. Signed-off-by: yihou <yihou@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds GLM-5.3-series support: engine image, patches, aggregated and
disaggregated example kits, and e2e cases parked off by default.
The fact that shapes the whole change: these are two unrelated architectures
They share a product name and almost nothing else, verified from the
checkpoints' own
config.json:model_typeglm_moe_dsaglm5_nextThe big models need no engine change — their
config.jsonis identical toGLM-5.2's field for field except
transformers_version, so they reuse theGLM-5.2 recipe. The Flash family exists in no released sglang, so it needs a
source overlay, which is why there are two Dockerfiles rather than one.
What is here
deploy/docker/Dockerfile.sglang— base moved tov0.5.18-rocm720-mi35x,with the one DSA patch hunk that drifted re-anchored (upstream renamed
tp0_info→tp0_info_cpu). Losing that hunk does not degrade gracefully: itis the only place the gathered column is
min()-reduced, so without it everyrank reads its own answer and the DP vote silently never happens.
deploy/docker/Dockerfile.sglang.glm53— same base plus the sglang #36607overlay pinned at
c821c425, for the Flash family. Kept separate so anunreleased-source fetch does not enter the default build path for Kimi-K3 and
GLM-5.2.
examples/sglang_mix_glm5.3/— aggregated kit covering all four checkpoints.examples/sglang_1p1d_glm5.3/— PD wrappers that drive the GLM-5.2 kitrather than forking it, so there is one source of truth for the engine recipe.
tests/e2e/— four GLM-5.3 cases, allenable=False; verified the existingparameter expansion is unchanged.
deploy/docker/patches/sglang_carried/— a MoE gate-bias fp32 fix, carriedand deliberately not wired into any image. The directory is unglobbed so
"carried, not applied" is structural rather than a comment someone can miss.
Measured on 8×MI355X / gfx950
Fixed-length, matched configuration both sides, benched through the router.
Alignment against the GLM-5.2 MIX baseline — the mission's bar:
0.89–1.11× across 8 matched points, TTFT better at 7 of 8, TPOT within ±10 % at
6 of 8. Conservative if anything: engine-side counters show our p50 arm did
11 points more prefill work than the baseline.
The p50 rows carry a measured quality statement, not an assumed one: scoring
490 generations at osl 320 gives 0 looping, worst 10-gram ×4 — throughput of
coherent text. The p90 rows are different: 54.7 % of 490 generations loop at
osl 3300, so those are a valid throughput measurement of the engine generating
repetitive text and must not be read as a quality result. The p90 ratio may
still be fair; the baseline's long-output arms cannot be checked the same way,
because its packup saved no generations.
MXFP4 vs FP8 on Flash, identical shape and topology: MXFP4 is 1.11× at
concurrency 1 and 1.23× at 8. First direct comparison on this architecture.
PD single-node 1P1D, feature-matched against aggregated on the same eight
GPUs: 1.10 / 1.26 / 0.87 / 0.87 at concurrency 1/8/16/24. PD is 10-26 %
ahead at low concurrency and ~13 % behind above it — at concurrency 1 its
TTFT p50 is 199 ms against the aggregated arm's 407 ms.
Getting to a number that could be attributed took three arms, and the first
two were each individually plausible: an unmatched comparison said 1.15× ahead,
a feature-matched one at concurrency 24 alone said 0.86× behind, and only the
arm that isolated decode-side DP-attention showed the crossover belonged to DPA
— worth 1.33× on its own — rather than to disaggregation. A single concurrency
point does not characterise a topology.
Turning the HIP transport off makes single-node PD faster: 1.23× throughput
and 3.4× better TTFT p50 at concurrency 8, read against a measured ~5 % noise
floor. That inverts the kit's own premise, which described HIP-over-XGMI as the
fast path and loopback RDMA as the degraded fallback.
Two engine-script fixes, and why they are here
The GLM-5.3 PD wrappers used single-knob names (
MTP,DPA,GPUS,EXTRA_ENGINE_ARGS) against a strictly per-leg interface, so six knobs silentlydid nothing and each degraded to a plausible default.
MTP=0launched decode atmtp=1;GPUSput both legs on the same four cards. None errored.Fixing that required two changes in
sglang_1p1d_glm5.2/— forwardingGPUSand the KV-event ports per leg. Both use
${VAR:+...}so an unset variableinjects nothing, and two-node resolution was verified byte-identical before and
after.
Known and stated
Documented in the kits rather than left for rediscovery:
osl 320, 40–65 % at 3300, on GLM-5.3 and GLM-5.2. Throughput arithmetic is
unaffected (loopers decode within 1 % of clean requests). Cause not
established, but two candidates are now eliminated by a single arm: run
with the chat template applied and MTP off — so no ROCm argmax branch — it
still looped at 60 %, and scoring the TP4 control arm — 490 generations, MTP
off throughout — puts that at 54.7 % against 54 % on the MTP-on
arms. The template was never missing (the flag is parsed and never consumed;
sglang-oai-chatapplies it server-side regardless), and turning MTP offdoes not change the rate at all, so
eagle_utils.py:726is eliminated ratherthan merely unnecessary — the three open upstream ROCm PRs are not on this
project's critical path for it. The surviving candidate, INFERRED, is that the
prompts are repetitive by construction:
benchmark/datasets/random.py:130-134reaches a long ISL by repeating one ShareGPT conversation and truncating, so an
isl-15500 prompt is roughly a 50× repetition of a few hundred tokens.
at the 4.00 ceiling was measured while 54 % of the same requests were looping.
--dataset-name randomdoes not build independent prompts. It drawsseeded ShareGPT, so each arm's list is a strict prefix of the next and higher
concurrency mechanically buys a higher cache-hit rate. Source predicts
12.5/50.0/66.7 %; four independent datasets measured 12.38/49.51/66.05 %.
than installation — so the natural discriminator reads 4/4 in both states.
Not included
GLM-5.3-Flash PD is out of scope. The kit records that the KDA state is
transferred (ten links with file:line) so the shape is feasible, and equally
that nobody has run it.