wingstransit measures wingsmith's data path: public client → gateway → tunnel → agent → the local
OpenAI-compatible frontend. It answers two questions, in this order: where is the bottleneck
and how much can one gateway replica carry. It answers them without a GPU, without
Kubernetes, and without a model.
It is not wingsbench. That measures an engine on real hardware. This measures the plumbing, with the
engine replaced by a stopwatch.
One binary, one generator, one report shape, so a laptop number, a CI number and a production number can be put side by side.
| Mode | What is real | Blast radius | Safe to run against production? |
|---|---|---|---|
synthetic |
gateway binary, agent binary, tunnel, both reverse proxies | this host only: child processes it started, temporary ports, no network egress | n/a; it touches nothing shared |
live |
everything, including a real engine | offered load against a shared serving path | only at a concurrency somebody has agreed to; see below |
path |
gateway binary, agent binary, tunnel, in front of a frontend you run | this host only | n/a: it touches nothing shared |
upstream / plane |
one fake, on its own | a port | yes, it serves nothing of ours |
live against production. It is a real caller doing real completions, which cost real GPU
seconds and share a queue with real customers. The default ramp is 1,2,4 and any stage above
-max-concurrency 8 is refused outright, so the default posture is a probe instead of a load test.
Raising it is a decision, and it should be taken with the same seriousness as pointing a load
generator at any customer path: pick a low-traffic window, watch the gateway's own
wingsmith_gateway_* series while it runs, and stop when the endpoint's latency moves. What is always
safe is the shape of it (one caller, a handful of streams, for twenty seconds), which is enough to
compare production's TTFT distribution against the lab's.
Numbers that do not compare across environments, and why.
- TTFT and inter-token latency. In synthetic mode they are the fake frontend's schedule plus the path; in live mode they are a real engine's prefill and decode, which dominate everything the transport does. The comparable quantity is the overhead over baseline, which only synthetic mode can produce, because only there is the frontend's own contribution known.
- Throughput and tokens/minute. Synthetic mode has no GPU and will happily carry a concurrency no cluster could serve. Its throughput number is a ceiling for the transport, never a capacity for the product.
- Per-leg and per-process columns. They need a shared clock and child processes, so they are present in synthetic mode and empty in live mode. Empty, not estimated.
- Latency floor. Loopback is roughly 0.05 ms and a WAN is 30–80 ms. Every synthetic figure is missing one round trip's worth of everything.
Per stage, per request: TTFT, every inter-token gap, total duration, and the failure bucket. Aggregated to p50/p90/p99, inter-token jitter, request rate, token rate, and success rate.
TTFT runs to the first frame that carries generated content. The role-only chunk an engine sends before prefill finishes, empty deltas and the usage chunk are not tokens, and none of them enter the inter-token gaps either.
When a stage's hold ends, the generator stops starting requests and gives the ones in flight
-drain (30s by default) to finish. Cancelling them at the boundary would drop the slowest requests
of every stage. Anything still running when the drain ends is reported as censored: it is
neither a success nor a failure, the success rate is taken over the rest, and it enters the TTFT
percentiles at the time it was cut off, which is a lower bound on its real TTFT. Rates are per
second of the stage's elapsed time, drain included.
Attribution comes from four independent signals, so the named bottleneck is evidence instead of a guess:
- Per-leg split. The fake frontend stamps the wall clock when it received a request
(
transit_received_nsin the body, andX-Transit-Received-Ns, which the gateway's response header allowlist drops) and when it wrote each token (transit_write_nsin the chunk). One host means one clock, so TTFT decomposes into the inbound leg (gateway parse, routing, tunnel, agent, local proxy) and the return leg. This needed no change to the gateway or the agent. - Per-process cost. CPU and RSS of the gateway and each agent, polled every second. CPU is the
process's accumulated user and system time differenced between polls, so each figure is the share
of a core used in that second; mean and peak are reported per stage. The gateway's own
wingsmith_gateway_tunnelsgauge is read at the end of each stage and printed beside the number of tunnels the agents were configured to open. - Where requests were held. The frontend's own peak in-flight count against the concurrency offered. Fewer means something between the caller and the frontend held them (stream limits, flow control, or a proxy), and it says so.
- Baseline subtraction. The first stage is also run straight at the frontend, so the whole path's cost is a subtraction instead of an assertion.
The one thing this cannot separate is gateway-internal time from tunnel time from agent time on
the inbound leg. Splitting those needs a stamp inside the gateway and the agent. The
non-invasive way to get it is a Server-Timing response header written by each hop: additive, off
by default, and readable by anything. That is a proposal, not something this harness did silently.
just transit # the default ramp, 8/32/128
just transit "4,16,64,256,512" # a ramp of your own
just transit-smoke # the CI shape: short, no baseline, JSON out
just transit-live https://gateway.example.com my-modelwingstransit path -frontend <url> builds the same gateway and agent and holds them open in front
of a frontend you run, with no load of its own. The gateway serves its public listener on
-listen (default 127.0.0.1:8080) for one tenant whose key is WINGSMITH_API_KEY, and routes
-model (default gpt-oss-120b) to the frontend. just conformance-gateway uses it to run the
conformance probes through a real gateway.
Every mode writes the whole report and then exits 1 when no stage held the objective, so
transit-smoke fails a CI job when the path cannot carry its small ramp. synthetic builds the
gateway and agent from the checkout above the working directory; -repo points it elsewhere.
The fake plane answers the three routes the plane allows the gateway's service principal:
GET /v1/endpoints/resolved, GET /v1/gateway/tenants (an empty tenant table) and
POST /v1/clusters/tunnel-auth. The harness starts the gateway with
WINGSMITH_GATEWAY_SERVICE_TOKEN set to the token it accepts. Its wire types are checked against the
gateway's plane client by go test, and WINGSMITH_TRANSIT_E2E=1 go test ./internal/lab/ boots the
real gateway and agent against it.
live sends WINGSMITH_API_KEY as its bearer token: a tenant API key for the endpoint under
test.
Useful flags on synthetic:
| Flag | Asks |
|---|---|
-streams 16 -conns 1 |
what happens at the per-tunnel stream limit |
-agents 3 -conns 2 |
many tunnels to one gateway against one carrying everything |
-kill-agent-every 3s |
what a caller sees when an agent dies mid-stream, with no GOAWAY |
-ttft 2s -itl 50ms -tokens 512 |
a long generation, where the stream is held open for minutes |
-stream=false |
the non-streaming path |
-slo-ttft-p99, -slo-success |
the condition the capacity number is reported under |
-hold 15s -drain 30s |
how long a stage starts requests, and how long its stragglers get to finish |
- Concurrent streams per tunnel before anything degrades, and what the far side of the limit
looks like. This is first because the limit is admission, not queueing: past it the pool answers
ErrSaturatedand the gateway sheds with a 503. - Cost of shedding. A rejection is far cheaper than a completion, so a saturated cluster can
cost more CPU than a served one. Ask it with
-streams 16 -conns 1and a ramp past 16, and read the gateway's CPU against thehttp_503count per stage. - How much one gateway replica carries at a stated success rate and TTFT band, and what saturates next.
- Many agents against one carrying everything. Coverage is per-replica and the pool is least-loaded, so this is a question about the pick, not about bandwidth.
- An agent dying mid-stream. Requests whose headers have already gone out cannot be retried by design; the number worth knowing is how many that is per kill.
- Head-of-line blocking on one HTTP/2 connection, visible as inter-token jitter, not as mean latency.
- Memory per in-flight request, from RSS against concurrency.
- Reconnect storms. Every agent redials when the gateway rolls; the fake plane counts the token verifications that produces.
- It is not a network. No WAN latency, no packet loss, no MTU, no TLS on the tunnel by default, and every process competes for one laptop's cores. Absolute throughput here is optimistic and the latency floor is fictional.
- It is not a model. The frontend never runs out of KV cache, never queues, and never gets slower under load. Everything a real cluster does when it is full is absent.
- It is one gateway replica. Multi-replica coverage, the Service in front of it, and the load balancer's choice of replica are not exercised.
- The control plane is a fixture. It proves nothing about the plane, the table the plane really derives, or what happens when the plane is slow.
- CPU is per second, not per instant. Each figure is a one-second average from accumulated CPU time, so a burst shorter than that is spread over its second; the peak is printed beside the mean for that reason.