Merge pull request #6264 from cardstack/cs-13037-the-link-shape-ladders-load-reading-counts-computations-not

Decide the link-shape ladder on search requests in flight, not computations

Author: habdelra

.claude/skills/indexing-diagnostics/SKILL.md

@@ -940,7 +940,7 @@ It is **not** persisted in `boxel_index.diagnostics` (that's the client side). S
 ```
 corr=<id> job=<jobId> handler=Nms parse=… resolveRealms=… sql=… stringify=… coalescedWait=… | results=…    (realm:search-timing)
 --> QUERY <accept> <url>: 200 [job: <jobId>] corr=<id> dur=Nms                                                                                            (realm:requests)
-eventLoopLagMs(mean/p99/max)=…/…/… inFlightSearch=… heapMB=…                                                                                              (realm:health)
+eventLoopLagMs(mean/p99/max)=…/…/… inFlightSearch=… searchRequests=… searchLoad=… heapMB=…                                                                                              (realm:health)
 ```
 
 The `|`-section is the **sequential wall-clock timeline** (these sum to ≈ `handler`).

.claude/skills/realm-load-harness/SKILL.md

@@ -464,11 +464,14 @@ per-render total:
 
 ## Getting a run to a threshold, and telling whether it did
 
-Two realm-server mechanisms engage at a level of concurrency: the admission gate
-bounds in-flight searches at a cap, and the link-shape policy degrades a live
-read's link closure one rung earlier, at a time-weighted mean of the same count
+Two realm-server mechanisms engage at a level of concurrency, on two different
+counts: the admission gate bounds concurrent search _computations_ at a cap,
+and the link-shape policy degrades a live read's link closure once a
+time-weighted mean of concurrent search _requests_ — joiners included, which
+the gate stops counting at the cache decision — crosses one of its rungs
 (120-second half-life by default, `LINK_SHAPE_LOAD_HALF_LIFE_MS` per
-deployment). Both are **per replica**, so a fleet of N tasks needs N times the
+deployment). The rungs are placed against sustained request load, not below
+the point where the gate starts shedding. Both are **per replica**, so a fleet of N tasks needs N times the
 load one process would.
 
 The gate does not shed at the cap: an arrival above it queues and is answered
@@ -482,16 +485,15 @@ could not have engaged. Two different questions, two different answers:
 
 - **Is the threshold where real traffic can reach it?** That needs load, and
   the credential pool is the ceiling: one session per CSV row, so a 20-row file
-  caps a run at 19 readers. At `--derive-page-size 0` that held a peak
-  120-second mean of **6.3** searches in flight. Read it against the rungs and
-  the cap in `packages/runtime-common/search-bounds.ts`: that run cost about
-  three readers per unit of mean, so the lower rung at 4 wants roughly 13
-  readers and the upper one at 12 roughly 36, **per replica**. So a pool of
-  this size already clears the lower rung, while the upper rung and the
-  admission cap still need one several times larger. Those are a floor, not an
-  estimate — the scaling is linear only while service time holds, and service
-  time is what rises first as a realm saturates. Growing the pool is the only
-  fix, and it is what puts the admission queue under enough pressure to shed.
+  caps a run at 19 readers. At `--derive-page-size 0` against a two-replica
+  deployment, that held the policy's reading — the 120-second mean of search
+  requests in flight, per replica, joiners included — at a p50 of about 14-17
+  and a peak of about 21-22. Read it against the rungs in
+  `packages/runtime-common/search-bounds.ts`: a pool of this size clears the
+  lower rung at 14 and does not reach the upper one at 28. The scaling is
+  linear only while service time holds, and service time is what rises first
+  as a realm saturates. Growing the pool is the only fix, and it is what puts
+  the admission queue under enough pressure to shed.
 
   The reading is also a count and not a cost: `inFlight` moves once per
   admitted search whatever that search is doing. An unbounded derived workload

.claude/skills/search-shape-diagnosis/SKILL.md

@@ -176,7 +176,7 @@ packages/observability/scripts/tail-logs.sh --env staging --service realm-server
 
 ## The link-shape policy
 
-How much of each result's link graph a live read carries is decided per realm from the load the process is under, one rung below admission control: a saturated process degrades a response before it refuses a request. Two channels record it — the per-request fields above, and a low-volume transition channel, `boxel:link-shape-policy`, on the same `| json` convention.
+How much of each result's link graph a live read carries is decided per realm from the load the process is under: under sustained request load, a process degrades a response rather than serving its full link closure. This is independent of admission control, which refuses requests on bursts against its cap and can do so at readings below the lower rung. Two channels record it — the per-request fields above, and a low-volume transition channel, `boxel:link-shape-policy`, on the same `| json` convention.
 
 A policy that reacts to load is only debuggable if it records the conditions it reacted to, because by the time anyone asks "why was this page slow" the concurrency that caused the decision is gone. These are the two questions an operator actually arrives with.
 
@@ -233,18 +233,18 @@ A `dwellMs` of `null` is a realm's first move, not a zero-length dwell — it ha
 ### The thresholds: what they mean and how to set them
 
 Four values decide when a realm changes rung. All four are **thresholds on the
-load reading** — the time-weighted mean of in-flight searches, per replica —
+load reading** — the time-weighted mean of search requests in flight, joiners included, per replica —
 and none of them selects a link shape. Which shape a rung serves is fixed by
 the ladder; these only decide when a realm arrives at that rung. The
 `_THRESHOLD` suffix is there because a bare `…_ENGAGE` reads as a switch that
 turns a shape on, which is the one thing it does not do.
 
 | Parameter                                | Ships as | Crossing it                                                               |
 | ---------------------------------------- | -------: | ------------------------------------------------------------------------- |
-| `LINK_SHAPE_MULTI_ROW_ENGAGE_THRESHOLD`  |        4 | reads that may return more than one row start shedding their link closure |
-| `LINK_SHAPE_MULTI_ROW_RELEASE_THRESHOLD` |        2 | those reads carry it again                                                |
-| `LINK_SHAPE_ALL_ENGAGE_THRESHOLD`        |       12 | every live read sheds it, including a single-card read                    |
-| `LINK_SHAPE_ALL_RELEASE_THRESHOLD`       |        6 | single-row reads carry it again                                           |
+| `LINK_SHAPE_MULTI_ROW_ENGAGE_THRESHOLD`  |       14 | reads that may return more than one row start shedding their link closure |
+| `LINK_SHAPE_MULTI_ROW_RELEASE_THRESHOLD` |        7 | those reads carry it again                                                |
+| `LINK_SHAPE_ALL_ENGAGE_THRESHOLD`        |       28 | every live read sheds it, including a single-card read                    |
+| `LINK_SHAPE_ALL_RELEASE_THRESHOLD`       |       14 | single-row reads carry it again                                           |
 
 Three more shape the same mechanism without being rungs:
 `LINK_SHAPE_LOAD_HALF_LIFE_MS` (120000, how far back the mean reaches),
@@ -255,7 +255,7 @@ All seven are settable the same way.
 Four properties decide whether a change to any of these does what you expect:
 
 1. **Per replica, no shared store.** Each process reads only its own
-   admissions, so a fleet-wide in-flight of 12 spread over N tasks is ~12/N
+   requests, so a fleet-wide in-flight of 28 spread over N tasks is ~28/N
    per replica. A threshold is only meaningful stated together with the fleet
    size it was chosen against, and adding tasks raises the fleet-wide load
    needed to engage roughly linearly. Read the current count rather than
@@ -271,9 +271,10 @@ Four properties decide whether a change to any of these does what you expect:
 3. **Every rung change costs a realm its cached validators.** The link mode is
    folded into both response validators and keys the card+json response cache,
    so a lower engage buys earlier protection and pays in cache fragmentation
-   landing exactly when the server is busiest. That trade is the reason the
-   upper rung sits at 12: lowering it to 8 raised a quiet control window's
-   degraded time from 0.3% to 3.8%.
+   landing exactly when the server is busiest. The upper rung, which degrades
+   even single-card reads, is placed above every load it has been observed on,
+   because nothing has measured it helping; `search-bounds.ts` records the
+   evidence for each rung.
 
    The fragmentation _rate_, though, does not track the engage as directly as
    that trade suggests, so "lower engage, more flapping" is not a safe thing to
@@ -349,7 +350,7 @@ Three of these cost real time on the investigation that produced the policy, and
 - **`busyMs(parallel-sum)` is not CPU.** It sums concurrently-awaited work, so it double-counts waiting — it has read 46× wall clock on a single-threaded loop. It cannot be used to argue the process was or was not saturated.
 - **The `realm:search-timing` stage lines carry no job marker.** A `loadLinks` share computed from them mixes worker traffic into live: in one 55-minute window 533 of 1,355 searches were `prerender`/job traffic. Separate them with this channel's `jobId` (via `correlationId`), not with the stage lines alone.
 
-One more, specific to the policy: the load it reads is the **sustained** in-flight count — a time-weighted mean with a multi-minute half-life — not the instantaneous `inFlightSearch` the health line prints. They differ by design and by a lot: on a production window whose raw samples repeatedly touched the admission cap of 30, the sustained reading peaked at 17. Comparing a `linkShapeLoad` against a health line's `inFlightSearch=` at the same timestamp will look like a contradiction and is not one.
+One more, specific to the policy: the load it reads is the **sustained count of search requests in flight** — every admitted request from admission until its response ends, joiners included, as a time-weighted mean with a multi-minute half-life. It is not the health line's `inFlightSearch`, which differs from it twice over: `inFlightSearch` is an instantaneous sample, and it counts admission slots — computations — which a request the live-search cache answers from another's computation hands back early, so it runs below the request count by the cache's miss rate. The health line carries the policy's own quantity as `searchLoad=` (and its instantaneous form as `searchRequests=`); compare `linkShapeLoad` against that, never against `inFlightSearch=`.
 
 ## Authoring a replay
 

docs/realm-server-health-signals.md

@@ -71,9 +71,16 @@ those cases, but it is a reason not to over-trust a 200 here on its own.
 lag crosses a threshold or a search is in flight:
 
 ```
-eventLoopLagMs(mean/p99/max)=… inFlightSearch=… heapMB=…
+eventLoopLagMs(mean/p99/max)=… inFlightSearch=… searchRequests=… searchLoad=… heapMB=…
 ```
 
+`inFlightSearch` counts admission slots — concurrent search _computations_,
+since a request the live-search cache answers from another's computation hands
+its slot back early. `searchRequests` counts every admitted search request until
+its response ends, and `searchLoad` is that count's time-weighted mean, which is
+the reading the link-shape policy's thresholds are compared against. The first
+two differ by the live-search cache's miss rate.
+
 Silent on an idle, healthy server — so its presence in the logs is itself the
 signal. Lag climbing in step with `inFlightSearch` is the fingerprint of the loop
 being starved by concurrent search serialization.

packages/observability/provisioning/alerting/realm-server-availability-group.json

@@ -482,7 +482,7 @@
           "title": "Realm Server Search Saturation",
           "condition": "B",
           "annotations": {
-            "summary": "A realm-server process reached 15+ concurrent _search / _federated-search requests at least once in the last 5 minutes (baseline is single digits). Sourced from the periodic health line (inFlightSearch=N, which counts both routes — including the in-render _search round-trips prerender issues). Concurrent search load drives heap growth toward OOM — this is the early-warning signal; find which client/realm is issuing the searches."
+            "summary": "A realm-server process reached 15+ concurrent _search / _federated-search computations at least once in the last 5 minutes (baseline is single digits). Sourced from the periodic health line (inFlightSearch=N: admission slots, which counts both routes — including the in-render _search round-trips prerender issues — but not requests the live-search cache answers from another computation; searchRequests=N counts those too). Concurrent search load drives heap growth toward OOM — this is the early-warning signal; find which client/realm is issuing the searches."
           },
           "data": [
             {

packages/realm-server/handlers/handle-search.ts

@@ -560,7 +560,9 @@ async function respondWithJobScopedSearchCache(
       // A joiner or a hit holds no result document of its own, so it stops
       // counting toward the search admission ceiling here rather than when
       // its response ends; the ceiling is then a bound on concurrent
-      // computations, which is what holds the heap.
+      // computations, which is what holds the heap. The request itself stays
+      // counted toward the link-shape policy's reading until its response
+      // ends, because it goes on waiting for the computation it joined.
       onOutcome: (decided) => {
         recordCacheOutcome(decided);
         if (decided !== 'miss') {

packages/realm-server/health-sampler.ts

@@ -1,6 +1,10 @@
 import { monitorEventLoopDelay, type IntervalHistogram } from 'node:perf_hooks';
 import { logger } from '@cardstack/runtime-common';
-import { getSearchInFlight } from './search-inflight.ts';
+import {
+  getSearchInFlight,
+  getSearchRequestLoad,
+  getSearchRequestsInFlight,
+} from './search-inflight.ts';
 import {
   heapTelemetry,
   formatHeapTelemetry,
@@ -49,6 +53,16 @@ export function startHealthSampler(
     let p99LagMs = toMs(histogram.percentile(99));
     histogram.reset();
     let inFlightSearch = getSearchInFlight();
+    // The link-shape policy's own inputs, beside the lag they are placed
+    // against. `inFlightSearch` counts admission slots, which a request the
+    // live-search cache answers from another's computation gives back early;
+    // `searchRequests` counts every admitted request until its response ends,
+    // and `searchLoad` is that count's sustained mean, the number the ladder's
+    // thresholds are compared against. A threshold derived from one of these
+    // and applied to another is off by the cache's miss rate, or by whatever
+    // the smoothing takes off a burst.
+    let searchRequests = getSearchRequestsInFlight();
+    let searchLoad = getSearchRequestLoad();
     // Reuse the prerender heap-telemetry helpers so the realm-server health
     // line carries the same `heapUsedMB=… heapLimitMB=…` fields (one spelling
     // of the quantity, and the effective V8 limit read from the running
@@ -57,7 +71,8 @@ export function startHealthSampler(
     let heap = heapTelemetry();
     log.info(
       `eventLoopLagMs(mean/p99/max)=${meanLagMs.toFixed(0)}/${p99LagMs.toFixed(0)}/${maxLagMs.toFixed(0)} ` +
-        `inFlightSearch=${inFlightSearch} ${formatHeapTelemetry(heap)}`,
+        `inFlightSearch=${inFlightSearch} searchRequests=${searchRequests} ` +
+        `searchLoad=${searchLoad.toFixed(2)} ${formatHeapTelemetry(heap)}`,
     );
   }, intervalMs);
   // Don't keep the process alive solely for sampling.

packages/realm-server/middleware/index.ts

@@ -28,6 +28,7 @@ import {
 import {
   admitSearch,
   admitSearchUnconditionally,
+  beginSearchRequest,
   getSearchAdmissionLimit,
   getSearchInFlight,
   getSearchShedCount,
@@ -249,8 +250,11 @@ const SEARCH_ADMISSION_RELEASE = 'searchAdmissionRelease';
 // ceiling; only the request doing the computing keeps its slot until its
 // response ends. That holds for an indexing-lane admission too: an in-render
 // search served from another request's computation holds no document either,
-// and the count is of computations, whichever lane admitted them. Idempotent,
-// and a no-op for requests the gate never saw.
+// and the count is of computations, whichever lane admitted them. Only the slot
+// is handed back: the request goes on counting toward the link-shape policy's
+// reading until its response ends, since it is still waiting on the
+// computation it joined. Idempotent, and a no-op for requests the gate never
+// saw.
 export function releaseSearchAdmission(ctxt: Koa.Context): void {
   let release = ctxt.state[SEARCH_ADMISSION_RELEASE];
   if (typeof release === 'function') {
@@ -263,7 +267,9 @@ export function releaseSearchAdmission(ctxt: Koa.Context): void {
 // (parse → SQL → serialize → send), which is the window in which it holds heap
 // and in which a saturated event loop would leave it unserviced; one that the
 // live-search cache serves from another's computation hands the slot back
-// early via `releaseSearchAdmission`. Mounted after CORS so that a shed
+// early via `releaseSearchAdmission`. Every admitted request, whichever it
+// is, is also counted as a request in flight until its response ends, which is
+// the reading the link-shape policy decides on. Mounted after CORS so that a shed
 // response carries the headers a cross-origin client needs to read its status
 // and Retry-After; before the body is parsed so that a shed costs nothing.
 export async function searchAdmission(ctxt: Koa.Context, next: Koa.Next) {
@@ -283,17 +289,27 @@ export async function searchAdmission(ctxt: Koa.Context, next: Koa.Next) {
   let arrivedAt = Date.now();
   let closed = false;
   let release: (() => void) | undefined;
+  let endRequest: (() => void) | undefined;
   let releaseSlot = () => {
     release?.();
     release = undefined;
   };
+  // The slot and the request end separately: a request the live-search cache
+  // serves from another's computation hands its slot back early, but it is
+  // still a request in flight — waiting on that computation — until its
+  // response ends, and that is the count the link-shape policy reads.
+  let end = () => {
+    releaseSlot();
+    endRequest?.();
+    endRequest = undefined;
+  };
   ctxt.state[SEARCH_ADMISSION_RELEASE] = releaseSlot;
   // `finish` fires on a fully-sent response; `close` covers a connection
-  // torn down before that, so a slot can't leak on an abort.
-  ctxt.res.on('finish', releaseSlot);
+  // torn down before that, so neither count can leak on an abort.
+  ctxt.res.on('finish', end);
   ctxt.res.on('close', () => {
     closed = true;
-    releaseSlot();
+    end();
   });
 
   if (isIndexing) {
@@ -313,6 +329,13 @@ export async function searchAdmission(ctxt: Koa.Context, next: Koa.Next) {
     }
     release = admitted;
   }
+  // Only a request whose connection is still open starts counting. The
+  // `close` listener is what ends the count, so a request that began after
+  // its close had already fired would never be ended — and a leaked request,
+  // unlike a leaked slot, holds the replica's load reading up until restart.
+  if (!closed) {
+    endRequest = beginSearchRequest();
+  }
   return next();
 }