fix(router): say so when a missing tokenizer disables cache-aware routing

examples/recipes/glm5.2-fp8-gfx942/README.md

@@ -69,6 +69,21 @@ the build log.
 
 ## 2. Prerequisites
 
+```{admonition} kv-aware needs the tokenizer on the ROUTER, not just the workers
+:class: warning
+The server pod mounts `/models` and passes `--router-tokenizer-path` a **local
+path** — both manifests already do this, and both matter. kv-aware tokenizes each
+request on the router to compare its prefix against what the workers cached, so
+the router needs the same tokenizer files the engines load.
+
+Point it at a hub id instead and it resolves an HF cache directory that may hold
+only `tokenizer_config.json`. The load fails, every request hashes to zero
+blocks, and kv-aware quietly becomes least-loaded — `--kv-overlap-weight` then
+has no effect at any value. Nothing else shows it: the server starts, `/health`
+is green, requests succeed. Watch for `kv-aware DEGRADED` in the server log and
+`infera_cache_locality_skipped_total{reason="no_tokenizer"}`.
+```
+
 **Hardware.** Two nodes, 8× gfx942 each, on a mutually routable RoCE fabric — the
 KV handoff is RDMA with no TCP fallback. The prefill node carries three Pods and so
 needs ~**670 GiB** of free host RAM: 512 for the engine (256 GB of hicache host tier

examples/recipes/glm5.2-fp8-gfx942/disaggregated-kvd/deploy.yaml

@@ -59,6 +59,13 @@ spec:
             imagePullPolicy: IfNotPresent
             command: ["python3","-m","infera.server",
                       "--host","0.0.0.0","--port","8000","--router-backend","python",
+                      # kv-aware needs the router to tokenize exactly as the workers do, so this
+                      # is a LOCAL path into the same model volume the workers mount -- not a hub
+                      # id. Point it at a hub id and the router resolves a cache directory that may
+                      # be missing the model's custom tokenizer files; the load then fails,
+                      # hash_for returns no blocks, and kv-aware silently degrades to least-loaded
+                      # with --kv-overlap-weight having no effect. Watch for `kv-aware DEGRADED` in
+                      # the server log and infera_cache_locality_skipped_total{reason="no_tokenizer"}.
                       "--router-tokenizer-path","/models/GLM-5.2-FP8",
                       "--discovery-backend","kubernetes",
                       "--request-transport","http","--kv-event-transport","zmq",

examples/recipes/glm5.2/README.md

@@ -41,6 +41,21 @@ payload missing Mooncake comes up green and serves with no KV transfer at all.
 
 ## 2. Prerequisites
 
+```{admonition} kv-aware needs the tokenizer on the ROUTER, not just the workers
+:class: warning
+The server pod mounts `/models` and passes `--router-tokenizer-path` a **local
+path** — both manifests already do this, and both matter. kv-aware tokenizes each
+request on the router to compare its prefix against what the workers cached, so
+the router needs the same tokenizer files the engines load.
+
+Point it at a hub id instead and it resolves an HF cache directory that may hold
+only `tokenizer_config.json`. The load fails, every request hashes to zero
+blocks, and kv-aware quietly becomes least-loaded — `--kv-overlap-weight` then
+has no effect at any value. Nothing else shows it: the server starts, `/health`
+is green, requests succeed. Watch for `kv-aware DEGRADED` in the server log and
+`infera_cache_locality_skipped_total{reason="no_tokenizer"}`.
+```
+
 **Hardware.** 8× MI355X (or MI300X+) on one node for `aggregated`; two such nodes on a
 shared RoCE fabric for `disaggregated`. ~700 GB of host RAM if you enable kvd — its L2 arena is
 pinned host memory charged to the sidecar's limit.

examples/recipes/glm5.2/aggregated-kvd/deploy.yaml

@@ -56,6 +56,13 @@ spec:
             imagePullPolicy: IfNotPresent
             command: ["/overlay/bin/infera-exec","python3","-m","infera.server",
                       "--host","0.0.0.0","--port","8000",
+                      # kv-aware needs the router to tokenize exactly as the workers do, so this
+                      # is a LOCAL path into the same model volume the workers mount -- not a hub
+                      # id. Point it at a hub id and the router resolves a cache directory that may
+                      # be missing the model's custom tokenizer files; the load then fails,
+                      # hash_for returns no blocks, and kv-aware silently degrades to least-loaded
+                      # with --kv-overlap-weight having no effect. Watch for `kv-aware DEGRADED` in
+                      # the server log and infera_cache_locality_skipped_total{reason="no_tokenizer"}.
                       "--router-tokenizer-path","/models/GLM-5.2-MXFP4",
                       "--request-transport","http","--kv-event-transport","zmq"]
             env:

examples/recipes/glm5.2/aggregated/deploy.yaml

@@ -45,6 +45,13 @@ spec:
             imagePullPolicy: IfNotPresent
             command: ["/overlay/bin/infera-exec","python3","-m","infera.server",
                       "--host","0.0.0.0","--port","8000",
+                      # kv-aware needs the router to tokenize exactly as the workers do, so this
+                      # is a LOCAL path into the same model volume the workers mount -- not a hub
+                      # id. Point it at a hub id and the router resolves a cache directory that may
+                      # be missing the model's custom tokenizer files; the load then fails,
+                      # hash_for returns no blocks, and kv-aware silently degrades to least-loaded
+                      # with --kv-overlap-weight having no effect. Watch for `kv-aware DEGRADED` in
+                      # the server log and infera_cache_locality_skipped_total{reason="no_tokenizer"}.
                       "--router-tokenizer-path","/models/GLM-5.2-MXFP4",
                       "--request-transport","http","--kv-event-transport","zmq"]
             env:

examples/recipes/glm5.2/disaggregated-kvd/deploy.yaml

@@ -67,6 +67,13 @@ spec:
             imagePullPolicy: IfNotPresent
             command: ["/overlay/bin/infera-exec","python3","-m","infera.server",
                       "--host","0.0.0.0","--port","8000",
+                      # kv-aware needs the router to tokenize exactly as the workers do, so this
+                      # is a LOCAL path into the same model volume the workers mount -- not a hub
+                      # id. Point it at a hub id and the router resolves a cache directory that may
+                      # be missing the model's custom tokenizer files; the load then fails,
+                      # hash_for returns no blocks, and kv-aware silently degrades to least-loaded
+                      # with --kv-overlap-weight having no effect. Watch for `kv-aware DEGRADED` in
+                      # the server log and infera_cache_locality_skipped_total{reason="no_tokenizer"}.
                       "--router-tokenizer-path","/models/GLM-5.2-MXFP4",
                       "--request-transport","http","--kv-event-transport","zmq"]
             env:

examples/recipes/glm5.2/disaggregated/deploy.yaml

@@ -76,6 +76,13 @@ spec:
             imagePullPolicy: IfNotPresent
             command: ["/overlay/bin/infera-exec","python3","-m","infera.server",
                       "--host","0.0.0.0","--port","8000",
+                      # kv-aware needs the router to tokenize exactly as the workers do, so this
+                      # is a LOCAL path into the same model volume the workers mount -- not a hub
+                      # id. Point it at a hub id and the router resolves a cache directory that may
+                      # be missing the model's custom tokenizer files; the load then fails,
+                      # hash_for returns no blocks, and kv-aware silently degrades to least-loaded
+                      # with --kv-overlap-weight having no effect. Watch for `kv-aware DEGRADED` in
+                      # the server log and infera_cache_locality_skipped_total{reason="no_tokenizer"}.
                       "--router-tokenizer-path","/models/GLM-5.2-MXFP4",
                       "--request-transport","http","--kv-event-transport","zmq"]
             env:

examples/recipes/kimi-k3-optimized/README.md

@@ -452,6 +452,7 @@ Each row is a failure that was hit and diagnosed on this hardware, not a prefere
 | `attention_backend: ROCM_AITER_MLA` | in the speculative config. The upstream quick-start says `FLASHINFER_MLA`, which is CUDA-only; **omitting the key entirely is not the fix** — this is its ROCm counterpart |
 | `--gpu-memory-utilization 0.88` | the draft's weights land after the KV budget is computed. At `0.95` the run dies with 998 MB free trying to allocate 2.32 GiB |
 | `INFERA_ENGINE_READY_TIMEOUT=7200` | infera's 1800 s default is generous for local NVMe and impossible for anything slower; the worker then kills itself mid-load and restarts forever, which reads as a crash loop rather than as slow storage |
+| the **server** pod mounts `/models` too, and `--router-tokenizer-path` is a **local path** | kv-aware tokenizes each request on the router to compare its prefix against what the workers cached, so the router needs the same tokenizer files the engines load. Point it at a hub id and it resolves an HF cache directory that may hold only `tokenizer_config.json`: the load fails, every request hashes to zero blocks, and kv-aware quietly becomes least-loaded with `--kv-overlap-weight` having no effect at any value. Nothing else shows it — the server starts, `/health` is green, requests succeed. Seen on a fleet whose router had been moved to a CPU node with no model volume. Watch for `kv-aware DEGRADED` in the server log and `infera_cache_locality_skipped_total{reason="no_tokenizer"}` |
 | PD: each node needs its **own local** copy | the `model` volume is a `hostPath`. The fleet's obvious shared mount was an NFS export of the peer's array — one side then loads for ~95 min and restarts forever |
 | PD: never point a client at it that sends requests the engine will reject | the engine validates **after** prefill, so a rejected request has already had its KV computed and queued for transfer. 84 requests rejected for one bad field left 424 aborted Mooncake transfers — 53 requests × 8 TP ranks — and stalled *valid* traffic for ~20 minutes with `MooncakeXferMetadata transfer failed: Resource temporarily unavailable`, which reads exactly like a broken fabric. It was a broken client |
 

examples/recipes/kimi-k3-optimized/aggregated-dspark/deploy.yaml

@@ -73,6 +73,14 @@ spec:
           - 0.0.0.0
           - --port
           - '8000'
+          # kv-aware needs the router to tokenize exactly as the workers do, so
+          # this is a LOCAL path into the same model volume the workers mount --
+          # not a hub id. Point it at a hub id and the router resolves a cache
+          # directory that may be missing the model's custom tokenizer files;
+          # the load then fails, hash_for returns no blocks, and kv-aware
+          # silently degrades to least-loaded with --kv-overlap-weight having no
+          # effect. Watch for `kv-aware DEGRADED` in the server log, and for
+          # infera_cache_locality_skipped_total{reason="no_tokenizer"}.
           - --router-tokenizer-path
           - /models/Kimi-K3
           - --request-transport

examples/recipes/kimi-k3-optimized/aggregated/deploy.yaml

@@ -68,6 +68,14 @@ spec:
           - 0.0.0.0
           - --port
           - '8000'
+          # kv-aware needs the router to tokenize exactly as the workers do, so
+          # this is a LOCAL path into the same model volume the workers mount --
+          # not a hub id. Point it at a hub id and the router resolves a cache
+          # directory that may be missing the model's custom tokenizer files;
+          # the load then fails, hash_for returns no blocks, and kv-aware
+          # silently degrades to least-loaded with --kv-overlap-weight having no
+          # effect. Watch for `kv-aware DEGRADED` in the server log, and for
+          # infera_cache_locality_skipped_total{reason="no_tokenizer"}.
           - --router-tokenizer-path
           - /models/Kimi-K3
           - --request-transport

examples/recipes/kimi-k3-optimized/disaggregated-dspark/deploy.yaml

@@ -82,6 +82,13 @@ spec:
           imagePullPolicy: IfNotPresent
           command: ["/overlay/bin/infera-exec","python3","-m","infera.server",
                     "--host","0.0.0.0","--port","8000",
+                    # kv-aware needs the router to tokenize exactly as the workers do, so this
+                    # is a LOCAL path into the same model volume the workers mount -- not a hub
+                    # id. Point it at a hub id and the router resolves a cache directory that may
+                    # be missing the model's custom tokenizer files; the load then fails,
+                    # hash_for returns no blocks, and kv-aware silently degrades to least-loaded
+                    # with --kv-overlap-weight having no effect. Watch for `kv-aware DEGRADED` in
+                    # the server log and infera_cache_locality_skipped_total{reason="no_tokenizer"}.
                     "--router-tokenizer-path","/models/Kimi-K3",
                     "--request-transport","http","--kv-event-transport","zmq"]
           env:

examples/recipes/kimi-k3-optimized/disaggregated/deploy.yaml

@@ -66,6 +66,13 @@ spec:
           imagePullPolicy: IfNotPresent
           command: ["/overlay/bin/infera-exec","python3","-m","infera.server",
                     "--host","0.0.0.0","--port","8000",
+                    # kv-aware needs the router to tokenize exactly as the workers do, so this
+                    # is a LOCAL path into the same model volume the workers mount -- not a hub
+                    # id. Point it at a hub id and the router resolves a cache directory that may
+                    # be missing the model's custom tokenizer files; the load then fails,
+                    # hash_for returns no blocks, and kv-aware silently degrades to least-loaded
+                    # with --kv-overlap-weight having no effect. Watch for `kv-aware DEGRADED` in
+                    # the server log and infera_cache_locality_skipped_total{reason="no_tokenizer"}.
                     "--router-tokenizer-path","/models/Kimi-K3",
                     "--request-transport","http","--kv-event-transport","zmq"]
           env:

examples/recipes/kimi-k3/README.md

@@ -152,6 +152,7 @@ on this hardware.
 | `--enable-prefix-caching` is safe here | Kimi-K3 is hybrid, so vLLM runs it in Mamba cache `align` mode and calls that experimental. Measured: 72.7% hit rate on a shared prefix, output still correct. An earlier note claimed it fails engine init — that was an older base image. |
 | `--load-format auto` | `fastsafetensors` needs GDS; without it the load stalls in 30-second queue waits. |
 | `startupProbe`, not readiness | weight load outlives any sane readiness deadline. |
+| the **server** pod mounts `/models` too, and `--router-tokenizer-path` is a **local path** | kv-aware tokenizes each request on the router to compare its prefix against what the workers cached, so the router needs the same tokenizer files the engines load. Point it at a hub id and it resolves an HF cache directory that may hold only `tokenizer_config.json`: the load fails, every request hashes to zero blocks, and kv-aware quietly becomes least-loaded with `--kv-overlap-weight` having no effect at any value. Nothing else shows it — the server starts, `/health` is green, requests succeed. Seen on a fleet whose router had been moved to a CPU node with no model volume. Watch for `kv-aware DEGRADED` in the server log and `infera_cache_locality_skipped_total{reason="no_tokenizer"}` |
 
 ## Validation status
 

examples/recipes/kimi-k3/aggregated-kvd/deploy.yaml

@@ -55,6 +55,13 @@ spec:
             imagePullPolicy: IfNotPresent
             command: ["/overlay/bin/infera-exec","python3","-m","infera.server",
                       "--host","0.0.0.0","--port","8000",
+                      # kv-aware needs the router to tokenize exactly as the workers do, so this
+                      # is a LOCAL path into the same model volume the workers mount -- not a hub
+                      # id. Point it at a hub id and the router resolves a cache directory that may
+                      # be missing the model's custom tokenizer files; the load then fails,
+                      # hash_for returns no blocks, and kv-aware silently degrades to least-loaded
+                      # with --kv-overlap-weight having no effect. Watch for `kv-aware DEGRADED` in
+                      # the server log and infera_cache_locality_skipped_total{reason="no_tokenizer"}.
                       "--router-tokenizer-path","/models/Kimi-K3",
                       "--request-transport","http","--kv-event-transport","zmq"]
             env:

examples/recipes/kimi-k3/aggregated/deploy.yaml

@@ -53,6 +53,13 @@ spec:
             imagePullPolicy: IfNotPresent
             command: ["/overlay/bin/infera-exec","python3","-m","infera.server",
                       "--host","0.0.0.0","--port","8000",
+                      # kv-aware needs the router to tokenize exactly as the workers do, so this
+                      # is a LOCAL path into the same model volume the workers mount -- not a hub
+                      # id. Point it at a hub id and the router resolves a cache directory that may
+                      # be missing the model's custom tokenizer files; the load then fails,
+                      # hash_for returns no blocks, and kv-aware silently degrades to least-loaded
+                      # with --kv-overlap-weight having no effect. Watch for `kv-aware DEGRADED` in
+                      # the server log and infera_cache_locality_skipped_total{reason="no_tokenizer"}.
                       "--router-tokenizer-path","/models/Kimi-K3",
                       "--request-transport","http","--kv-event-transport","zmq"]
             env:

examples/recipes/kimi-k3/disaggregated-kvd/deploy.yaml

@@ -67,6 +67,13 @@ spec:
             imagePullPolicy: IfNotPresent
             command: ["/overlay/bin/infera-exec","python3","-m","infera.server",
                       "--host","0.0.0.0","--port","8000",
+                      # kv-aware needs the router to tokenize exactly as the workers do, so this
+                      # is a LOCAL path into the same model volume the workers mount -- not a hub
+                      # id. Point it at a hub id and the router resolves a cache directory that may
+                      # be missing the model's custom tokenizer files; the load then fails,
+                      # hash_for returns no blocks, and kv-aware silently degrades to least-loaded
+                      # with --kv-overlap-weight having no effect. Watch for `kv-aware DEGRADED` in
+                      # the server log and infera_cache_locality_skipped_total{reason="no_tokenizer"}.
                       "--router-tokenizer-path","/models/Kimi-K3",
                       "--request-transport","http","--kv-event-transport","zmq"]
             env:

examples/recipes/kimi-k3/disaggregated-sglang/deploy.yaml

@@ -50,6 +50,13 @@ spec:
             image: lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727@sha256:3c01f73fe23aebf4a8853de0899a70b75c2af6c0409d2331353847aac4d3f906      # libionic (ABI 4) baked
             imagePullPolicy: IfNotPresent
             command: ["/overlay/bin/infera-exec","python3","-m","infera.server","--host","0.0.0.0","--port","8000",
+                      # kv-aware needs the router to tokenize exactly as the workers do, so this
+                      # is a LOCAL path into the same model volume the workers mount -- not a hub
+                      # id. Point it at a hub id and the router resolves a cache directory that may
+                      # be missing the model's custom tokenizer files; the load then fails,
+                      # hash_for returns no blocks, and kv-aware silently degrades to least-loaded
+                      # with --kv-overlap-weight having no effect. Watch for `kv-aware DEGRADED` in
+                      # the server log and infera_cache_locality_skipped_total{reason="no_tokenizer"}.
                       "--router-tokenizer-path","/models/Kimi-K3",
                       "--request-transport","http","--kv-event-transport","zmq"]
             env: