Repository navigation
benchmarking: add a burst benchmark that resumes and suspends many actors at once - #2399
Sairaj Pokale (sairajp-rewind) wants to merge 2 commits into
Conversation
SpawnUser's CreateActor and ResumeActor each had their own copy of the retry-with-backoff loop. Move it into retryWithBackoff, and the rule that AlreadyExists on a retry means an earlier attempt landed into createLanded, so the burst benchmark can reuse both. Spawn's behavior does not change: the same errors are terminal, with the same backoff.
Add BurstUser, which measures how fast Substrate resumes and suspends many actors at the same moment. Setup creates --total-actors actors from the glutton template. Each batch then resumes every actor at once, runs the workloads for one live window, and suspends every actor at once, with a wait window between batches. Batch 1 resumes from the golden snapshot, so it books its own First rows. Each burst books per-actor times, the 10% to 100% marks, and the time until the last success. An actor that fails a phase is deleted right away and left out of later batches, so later batches measure the actors that still work. Failures never stop the run: at the density ceiling they are the result. Like SpawnUser, a worker runs one burst, driven by its first user, and a stop from the web UI cancels it and deletes the cohort in one pass. The actors reuse gluttonActor unchanged; only the readiness ping, which needs an error return, is local. Rows booked by the reused code keep GluttonUser's Prometheus label and are not dropped after a stop, while burst's own rows are.
| ) | ||
|
|
||
| func init() { | ||
| userclass.Add(userclass.Entry{ |
There was a problem hiding this comment.
Heads-up rather than a defect: #2160 (approved, about to land) replaces the shared dynconfig.Config with a per-class knobs struct registered here as Entry.Config, and removes Holder.Load() returning Config, NewHolder(Config) and uniformWait. Burst reads TotalActors, ActorDeadline, the wait and live windows, mem and CPU knobs and MaxPingsPerWake through those, so whichever of the two merges second has to adapt. A trial merge against today's main is clean; against #2160's head this file will not compile. Burst will want its own knobs struct (WaitTime, Lifecycle, live window, mem, CPU, pings, total_actors, actor_deadline) and dynconfig.Get[knobs] in place of cfg.Dyn.Load(), and the frozen copy becomes dynconfig.Static(knobs).
|
|
||
| // resumeOne resumes a and waits until it answers through the router. | ||
| func (b *burstRun) resumeOne(ctx context.Context, a *burstActor) error { | ||
| if err := retryWithBackoff(ctx, func(int) error { return a.resume(ctx) }); err != nil { |
There was a problem hiding this comment.
At the density ceiling the failure this benchmark exists to find is ResourceExhausted ("no free workers available"), and isSpawnTerminalError does not treat it as terminal, so every actor past the ceiling retries ResumeActor with backoff for the whole actor deadline. The ceiling then reads as 120 s latencies and late ActorTimeToResume failures, and the TimeTo*_<k>pct marks are taken over a cohort that is partly spinning in backoff rather than failing. Spawn is a creation benchmark where riding out capacity makes sense; for a burst I would make ResourceExhausted terminal in the resume phase (or make the policy a flag) so the ceiling shows up as failures at the moment they happen, and keep the retries for Unavailable and Aborted.
| } | ||
| boomerutil.LogSampledTrace(span, "GluttonReadyPing", latency, boomerutil.SourceClient, err) | ||
| if err != nil { | ||
| b.recordFailure("http", "GluttonReadyPing", latency, err.Error()) |
There was a problem hiding this comment.
Each 100 ms poll until the actor answers books a GluttonReadyPing failure, so the row's failure count is ten per second of restore time per actor, a function of how long the wake took rather than of anything failing. On a 1,000 actor burst that is tens of thousands of failures on a row operators will read as errors. Book only the final outcome on this row (plus an attempts count or the time-to-ready, which ActorTimeTo*Resume already carries), or send the polls to a row named for what they are.
| // Gap between readiness pings once ResumeActor has returned. | ||
| burstReadyPingGap = 100 * time.Millisecond | ||
| // Budget for one round of DeleteActor calls. | ||
| burstDeleteTimeout = 60 * time.Second |
There was a problem hiding this comment.
60 s for a whole round at concurrency 32 does not scale with the cohort: an any_state delete of a RUNNING actor tears the sandbox down, about a second each, so a 2,000 actor cohort needs over a minute and a 5,000 one about two and a half, after which teardown logs the leak warning and stops. Since a leaked cohort keeps holding workers for the next run, size the budget from the cohort (for example actorDeadline + N/concurrency × per-delete) or loop rounds until nothing is left or deletes stop making progress.
Bowei Du (bowei)
added
area/benchmarking
kind/feature
3 participants
Adds
BurstUser, a benchmark that measures how fast Substrate resumes andsuspends many actors at the same moment, and how many fail, up to node density
ceilings.
SpawnUsercreates a batch once and never suspends it;GluttonUsercycles each actor on its own timer, so actors never burst together.
How it works
--total-actorsactors from thegluttontemplate, all suspended.then suspends every actor at once. A wait window separates the batches, which
repeat until the run is stopped.
Firstrows. Laterbatches resume from each actor's own snapshot.
last success. Locust merges later batches into one row, so the worker also logs
one
burst: batch doneline per batch with p50, p90 and max.--mem-target(and the CPU flags) set, batch 1 fills RAM and starts the CPUload; later batches read the RAM the snapshot kept, then churn part of it.
Failures never stop the run: at the density ceiling they are the result.
-u 1). Stop cancels itand deletes the cohort.
Commits
benchmarking: extract spawn's retry loop: moves spawn's retry loop intoretryWithBackoffandcreateLandedso burst can reuse them. Spawn's behaviordoes not change.
benchmarking: add the burst suspend/resume benchmark:burst.goand its tests,burst.py, atests.yamlentry, and a README section.For review
gluttonActoras it is.The trade is that rows booked by the reused methods (
CreateActor,ResumeActor,SuspendActor,DeleteActor,GluttonPing, the RAM and CPUrows,
CrashCount) keepuser_class="GluttonUser"in Prometheus and are notdropped after a stop. Burst's own rows (
ActorTimeTo*,*_<k>pct,TimeToAll*,GluttonReadyPing,Retired*) carryBurstUserand are droppedafter a stop.
stats.csvdoes not split rows by user class, so it isunaffected.
Follow-ups (not in this PR)
gluttonActor. Burst's reused rows could then carryBurstUser.On both pools, no
burst-*actors remained after Stop, and a second Start didnothing until the worker restarted.