feat: add http_server.tcp_rejected_rate metric for connection-cap saturation

[ycombinator]

What is the problem this PR solves?

Fleet Server's existing autoscaling metric, http_server.tcp_active, is a utilization signal: EPA uses it to maintain headroom by targeting N active connections per pod. This works well at steady state but understates demand when connections are short-lived — specifically during large-scale agent enrollment, where each enrollment connection completes in under a second.

tcp_active is a point-in-time gauge: it reflects the instantaneous count of open HTTP connections at the moment EPA polls /stats. A burst of enrollment connections that arrive and clear between two EPA polls leaves no trace in the metric. The result is that pods plateau well below the theoretical requirement, agents receive HTTP 429s, and enter exponential backoff — turning a short enrollment phase into a multi-hour convergence tail.

How does this PR solve the problem?

Adds http_server.tcp_rejected_rate to /stats: a gauge, in rejections/sec, derived from an internal counter that increments whenever the chi Throttle middleware returns HTTP 429 due to the per-pod connection cap (max_connections). The rate is sampled every 10 s by a new RunConnRejectionRateSampler goroutine.

tcp_rejected_rate vs tcp_active

	tcp_active	tcp_rejected_rate
Signal type	Utilization	Saturation
When it fires	Active connections approach the per-pod target	Pod is already at its connection ceiling and returning 429s
Enrollment bursts	Understates demand — burst clears before next EPA poll	Catches the burst; rejections accumulate in the counter until sampled
Steady state	Correct — long-held checkin connections register reliably	Zero — pod is not at ceiling, no 429s
EPA behavior	Scale to maintain headroom	Scale the moment a pod is overwhelmed

Because EPA takes the maximum desired replica count across all configured metrics, wiring both allows proactive scaling (tcp_active: headroom at steady state) and reactive scaling (tcp_rejected_rate: act immediately under a burst). Neither metric suppresses the other.

The raw rejection count is kept as an unexported atomic.Uint64 — only the derived rate gauge is exposed in /stats, since that is the only value EPA consumes. The chi Throttle middleware behaviour is unchanged; the wrapper only observes the response status code.

How to test this PR locally

1. Configure Fleet Server with max_connections: N and start it.
2. Send more than N concurrent requests (e.g. with hey or wrk).
3. curl localhost:5066/stats | python3 -m json.tool | grep tcp_rejected — confirm tcp_rejected_rate becomes non-zero after ~10 s.
4. Let load drop — tcp_rejected_rate should return to 0 within the next sample interval.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Relates https://github.com/elastic/fleet-controller/pull/638

[mergify]

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[swiatekm]

The logic looks right to me, but calculating the rate of change this way within the application looks like an antipattern. Could we emit it as a total counter and calculate the delta downstream, either in a library or the monitoring solution?

[ycombinator]

The logic looks right to me, but calculating the rate of change this way within the application looks like an antipattern. Could we emit it as a total counter and calculate the delta downstream, either in a library or the monitoring solution?

I had looked into this previously (for a similar trigger that we may/may not need: https://github.com/elastic/fleet-controller/issues/626). Unfortunately, Elastic Pod Autoscaler (EPA) doesn't support computing rates from a counter. We could implement this around fleet-controller, but that would require it to poll the /stats API on every Fleet Server pod, retain per-pod counter history, compute the rate, and expose the derived metric for EPA to consume — responsibilities it does not have today.

That would be a larger architectural change, so this PR keeps the rate calculation in Fleet Server for now.

[UPDATE] I've created https://github.com/elastic/elastic-pod-autoscaler/issues/745 so the EPA maintainers can discuss and decide if they want to support deriving rate metrics from counter metrics in the future.

[macdewee]

We could also add test for tcp_rejected_rate should return to 0 within the next sample interval

[ycombinator]

We could also add test for tcp_rejected_rate should return to 0 within the next sample interval

Added in 47aa51d.

[mergify]

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.