Repository navigation
Benchmarking: measure ate-api-server cold resume capacity on the fake data plane - #2686
Open
haiyanmeng wants to merge 17 commits into
Open
haiyanmeng wants to merge 17 commits into
haiyanmeng wants to merge 17 commits into
Conversation
haiyanmeng
requested review from
Julian Gutierrez Oschmann (juli4n) and
Max Smythe (maxsmythe)
October 14, 2026 21:38
Max Smythe (maxsmythe)
left a comment
Max Smythe (maxsmythe)
left a comment
Collaborator
There was a problem hiding this comment.
Left some high level comments.
The majority of them are mostly about how we need to separate Substrate manifest shape from benchmarking code (which requires install that allows custom params).
I'm not sure what the best solution is here. I'm open to some kind of interim shim to unblock things since customizeable install story is complex, but we should have some idea of how we re-converge if necessary.
Other comment is just around where analysis code belongs.
| ) | ||
|
|
||
|
|
||
| def shape(cfg: dict[str, Any], runner_cpu: Any) -> None: |
Collaborator
There was a problem hiding this comment.
This seems like a shim because we have no story for configuring Substrate's footprint during install. I am assuming that is something we will want to fix in the future. What is the plan for this code if/when we get there? How are we trying to de-risk needing to carry this code foreward?
Collaborator
Author
There was a problem hiding this comment.
Agreed, it's a shim. The plan:
- The
ateapi:block intests.yamlis the contract, andcontrolplane.pyis only today's implementation of it. The sizing knobs (replicas, CPU, memory, pool size, Postgres settings) are what an install config would expose. - When Substrate's footprint becomes configurable at install time,
controlplane.pyturns the block into install config instead of patching after the deploy, and the patch code is deleted without touchingtests.yaml. - I'll rename the block in this PR (per your other comment) so the contract only changes once.
Until then, the risk in carrying it is silent drift: a manifest change leaves a patch with no effect and the run measures the wrong shape. I added a TODO in controlplane.py for two guards:
- Tests that read
manifests/ate-installand fail when what the patches assume changes. - A check after
shape()that the running pods carry the requested resources/settings.
haiyanmeng
force-pushed
the
fake-dataplane-test-1
branch
from
October 19, 2026 15:50
bf5e127 to
efc7b65
Compare
Bowei Du (bowei)
added
the
kind/feature
haiyanmeng
force-pushed
the
fake-dataplane-test-1
branch
from
October 19, 2026 23:55
b722bd2 to
82b6600
Compare
ate-controller resolves the SandboxConfig each sandbox class of a WorkerPool uses and writes it to the pool's status.sandboxClasses, which ate-api-server reads to boot the pool's actors. fake-workersync must write the same status on a --fake-data-plane install, where ate-controller does not run, and the resolution lived in ate-controller's internal package, out of its reach. Move the resolution functions, which read only a pool and a list of configs, to internal/workerpoolsandbox, with their unit tests. The claim and confirm steps that protect a config before a running pool switches to it stay in ate-controller, which now calls the package. The finalizer check uses slices.Contains in place of controllerutil, so the package does not depend on controller-runtime. No behavior changes.
ate-api-server boots an actor with the SandboxConfig its Worker's pool resolved, read from the pool's status.sandboxClasses, which ate-controller writes. On a --fake-data-plane install ate-controller does not run and fake-workersync wrote only the replica counts, so no golden actor could start and no actor template ever got its golden snapshot. fake-workersync now resolves each of a pool's sandbox classes with internal/workerpoolsandbox, as ate-controller does, and writes the result with the replica counts. It skips ate-controller's claim of a new config, which keeps the config from being deleted under running worker pods; fake Workers run nothing. It watches SandboxConfigs to reconcile when one changes, and its ClusterRole reads them.
RecordServerLatency adds locust_server_duration_milliseconds, the server's own elapsed time for a request as the server reports it, for a user class to record beside the client latency. The gap between the two is network and client time. Its buckets start at 0.1 ms: a fast control-plane RPC takes around a millisecond on the server, so a first bucket at 1 ms would hide its p50. locust_request_duration_milliseconds is unchanged. resumepaused records it for each RPC that succeeds, from the trailer that ateinterceptors.ServerUnaryInterceptor sets. A failure stays out: the server histogram has no status, and a call cut off by its deadline brings back no trailer.
boomer inside an automation runner Job served its Prometheus metrics on port 8001, but nothing scraped them. The runner container now declares the port, and monitoring.yaml's boomer-worker job keeps it on substrate-benchmark-runner pods too, so a run started by the automation has the same client and server latency histograms in Prometheus as one started by hand.
A capacity run of ate-api-server reads cores and latency per request rate, which means nothing unless ateapi's CPU is fixed and nothing else competes for it, and it has to say whether Postgres or ate-api-server bound first. As installed, ateapi is BestEffort and shares nodes with the load generator, and nothing scrapes Postgres or loads pg_stat_statements. A locust test in tests.yaml can now carry an ateapi: block. pre_test, after the deploy: - with pgStats, on by default, loads pg_stat_statements, passed as a -c flag so the installed config is left alone, and adds a postgres_exporter sidecar that reads the server over the pod's unix socket: one patch, thus one Postgres restart. monitoring.yaml scrapes the exporter as job postgres-exporter: connections by state and wait event, commits, cache hits, and per-statement calls and execution time. Prometheus discovers it through a ClusterRole, since a Role in ate-system would go with the namespace, which every automated test deletes and recreates. postgresArgs adds Postgres settings as further -c flags in the same restart, so a test sets Postgres without a new install profile. - with pinNodes, on by default, labels one node for ate-api-server and another for the runner, both away from the Postgres node, and pins each there with a node affinity, so the load generator is never ateapi's neighbor. Every ateapi replica shares its node, so the node must be able to allocate all of their CPU, and the runner's node the runner's; a node that cannot is passed over, and pre_test fails if none can, rather than leaving a pod Pending. The roles are picked after the Postgres restart, since on size0 nothing pins Postgres and the restarted pod may land on another node. The runner takes a node of the pool reserved for load generators when there is one, as its Job tolerates that pool's taint; ate-api-server takes an untainted node, which a --cordon-control-plane install never gives it, so validate rejects that combination. - with poolMaxConns, sets ATE_API_POSTGRES_POOL_MAX_CONNS in the ate-api-server-envvars ConfigMap, which ateapi reads over any pool_max_conns in the DSN. It sizes the read/write pool; the owner and watch pools stay at 2 and 3 connections per replica. - sets ate-api-server's replicas, and CPU and memory with requests equal to limits, so Go sizes GOMAXPROCS from the CPU limit, and waits for the rollout. It prints where ateapi, Postgres and the runner landed, since a step compares across runs only when the placement matches. Every test redeploys substrate, so nothing is unpatched afterwards. The shaping lives in controlplane.py, apart from the locust hooks, which only validate the block, call it from pre_test, and pin the runner, so another test type can take the same block.
analyze_ladder.py summarizes a ladder run per step: given a run's downloaded results, it reads the step boundaries from logs.txt and queries the benchmarking Prometheus for each step's steady window, printing one JSON object per step with throughput, client and server latency, ate-api-server CPU overall and per replica, Postgres CPU, connections and their waits, commits, and memory. A step logged with target_rps=unchanged keeps the previous step's rate, and a step too short to leave a steady window after the trim is skipped with a message. A query that fails, or gets no answer in 30 s, stops the run rather than reporting the figure as missing.
… tests ateapi_cold_resume_pause_fake_2cpu drives resumepaused in isolate mode, a ladder of cold resume rates from 100 to 1500/s held 3 minutes a step, to find where one ate-api-server replica at 2 CPUs, or Postgres, stops a cold resume and pause cycle. fake-atelet answers each call, restore and checkpoint alike, after 150 ms on every atelet node, and 200 fake Workers of 1000 actors each mean placement never binds. Each user owns one actor, so the 1000 users have 1000 actors. The test pins ate-api-server and the runner to their own nodes and collects Postgres statistics, through the ateapi: block. ateapi_cold_resume_pause_fake_smoke runs the same setup with 50 users on 2 fake Workers, at 20 and then 50 cold resumes/s for 2 minutes each, to check a change end to end in 6 minutes rather than 23. The benchmarking README now says how to summarize such a ladder run per step with automation/analyze_ladder.py: fetch the run's logs.txt, port-forward the benchmarking Prometheus, and run the script.
pinNodes picked ate-api-server's node and the runner's by comparing the CPU each asks for with the node's allocatable CPU. DaemonSets and ate-system pods already request CPU on every node, so a node could pass and still leave the pod Pending, and the run failed only when rollout status timed out after 600 s. label_nodes now lists the pods and passes pick_role_nodes the CPU already requested on each node, so a node must have the CPU free: its containers' requests plus its sidecars', from pods that are scheduled and not finished. ate-api-server's own pods are left out, since the pin replaces them. Pods scheduled after the pick, such as the benchmark workloads, can still take the runner's CPU.
requested_millis summed a pod's containers and sidecars, which is less than the scheduler reserves: it also takes the largest init container, with the sidecars started before it, when that is more, and adds the RuntimeClass overhead, 250m a pod for microvm. A node could thus look free to pinNodes when it was not. _pod_cpu_millis now computes the reservation the way Kubernetes does, and matches the CPU requests that kubectl describe node reports.
The ateapi block's pinNodes labeled one node for ate-api-server and another for the runner, picking them by replaying the scheduler's CPU accounting over every pod in the cluster. That tied the benchmarking code to the shape of the control plane manifests and duplicated the isolation the install already offers: --cordon-control-plane keeps ate-api-server and Postgres on their own tainted pools, away from the nodes that run the load. Drop pinNodes and the node picking, and have the cold resume capacity and smoke tests install with --cordon-control-plane. The ateapi block now only sizes ate-api-server, the pool, and Postgres.
controlplane.py patches ate-api-server, Postgres, and the ateapi env ConfigMap after the deploy, so each patch assumes names and args from the install manifests. A manifest change can leave a patch without effect and the run measuring the wrong shape. Record, next to shape(), the tests and post-rollout check that would make such drift fail the run instead.
The ateapi: block mixed ate-api-server and Postgres knobs at one level, so a knob such as cpu did not say which component it sized, and the block had no room for other components' footprints. Replace it with a controlPlane: block holding an apiServer: section (cpu, memory, replicas, poolMaxConns) and a postgres: section (stats, args), each validated on its own with errors that name the full knob path.
The TestType docstring described pre_test through what each test type does with it, so every type's details leaked into the generic abstraction. Describe only the hook's purpose; each type documents its own pre_test.
The per-step ladder summary was a script run by hand after the test: it needed the run's logs.txt downloaded and a port-forward to the cluster's Prometheus, and it worked only while that Prometheus still held the run's samples. Every other analysis runs on the runner pod and uploads its output with the run. Move it into the locust runner as an additive step after the server telemetry harvest, whose wait covers the last step's samples. It reads the runner's own log, queries the in-cluster Prometheus, and appends a ladder_step_NNN row per step to stats.jsonl, its measurements flattened to scalars like the other rows. A failed query adds no step rows and leaves the run's other results alone.
resumepaused now pauses an actor with SuspendActor(ASYNCHRONOUS) and records it as SuspendActorAsync, since PauseActor is gone. The ladder step queries still asked for PauseActor, so every pause figure in a step's row would have come back empty.
A fake Worker reports its node's allocatable as its capacity, and each benchmark actor books 1 CPU of it by default, so on a 4-CPU node a fake Worker held about three actors. The smoke test's two Workers then had room for a handful of its 50 actors, and most cold resumes failed with ResourceExhausted. Fake actors run nothing: book 10m CPU each, so placement never binds on any node size.
fake-atelet runs in atelet's DaemonSet under atelet's container name, so the ladder step query for container="fake-atelet" matched nothing and fake_atelet_cores was always empty.
haiyanmeng
force-pushed
the
fake-dataplane-test-1
branch
from
October 20, 2026 00:28
82b6600 to
a981433
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds what it takes to measure ate-api-server's and Postgres's cold resume capacity on the fake data plane from the benchmark automation, plus the first test that uses it.
boomer: record server-side latency.
resumepausedrecordslocust_server_duration_millisecondsfrom thex-server-elapsed-ustrailer thatateinterceptors.ServerUnaryInterceptorsets, next to the client latency. The gap between the two is network and client time.Scrape boomer in automation runner Jobs. The runner container declares port 8001 and
monitoring.yaml'sboomer-workerjob keepssubstrate-benchmark-runnerpods, so automated runs get the same latency histograms as manual ones.Shape ate-api-server and Postgres for capacity runs. A locust test can carry an
ateapi:block. It fixes ate-api-server's replicas, CPU and memory, pins ate-api-server and the runner to their own nodes, sizes the Postgres pool, and loadspg_stat_statementswith apostgres_exportersidecar that Prometheus scrapes.Summarize ladder runs per step.
benchmarking/automation/analyze_ladder.pyreads a run's step boundaries fromlogs.txtand queries Prometheus for each step's steady window. It prints throughput, client and server latency, CPU, memory and Postgres figures per step.Cold resume tests.
tests.yamladds two tests:ateapi_cold_resume_pause_fake_2cpu: a 100–1500/s cold resume ladder against one 2 CPU replica.ateapi_cold_resume_pause_fake_smoke: the same setup at 50 actors and 20 → 50/s, a quick end-to-end check.The benchmarking README now explains how to analyze a run.
Addresses #2261
go test -race ./internal/benchmarking/boomer/..., thebenchmarking/automationPython tests,hack/verify-all.sh)