Skip to content

operator-endpoint.md

Latest commit

 

History

History
526 lines (426 loc) · 29.3 KB

operator-endpoint.md

File metadata and controls

526 lines (426 loc) · 29.3 KB
Raw

The per-pool operator endpoint

Namespace note (v0.10.0, issue #386): the four directives on this page used to be spelled pm.status_path, pm.metrics_path, pm.status_listen and pm.metrics_listen. They are now operator.*, with operator.metrics / operator.status as shorthands. The old spellings are refused by name. A pool.type = gateway exposes every pool's pages under <base>/<pool name> on its public port since issue #389 with http.operator = yes; see gateway.md.

A cron, supervisor, gateway or http-direct pool has nothing in front of it that could answer a monitoring scrape. A fastcgi pool does — the web server that speaks FastCGI to it — which is why upstream FPM answers pm.status_path inside a request and why that arrangement is left alone here. pm.status_path stays an upstream directive answered on the pool's own FastCGI socket.

A FastCGI pool may separately opt in to the operator listener using operator.*; that does not move or replace pm.status_path. The operator pages are still served by the internal HTTP listener, never by a handler on the FastCGI socket. For pools that expose a page, php-fpm-ng serves the answer from a small HTTP listener of its own:

[tick]
pool.type = cron
cron.schedule = */5 * * * *
cron.script = /srv/app/bin/tick.php

operator.status_listen = 127.0.0.1:9253
operator.status_path   = /tick/status
operator.metrics_listen = 127.0.0.1:9253
operator.metrics_path   = /tick/metrics
Directive Meaning Default
operator.status_path Path answering JSON for this pool. Unset: off. unset = off, except gateway (/status; operator.status = off to disable)
operator.metrics_path Path answering Prometheus text for this pool. Unset: off. unset = off, except gateway (/metrics; operator.metrics = off to disable)
operator.status Shorthand: expose status at /status/<pool name>. off
operator.metrics Shorthand: expose metrics at /metrics/<pool name>. off
operator.status_listen Where operator.status_path binds. 127.0.0.1:9253
operator.metrics_listen Where operator.metrics_path binds. 127.0.0.1:9253

operator.status = on and operator.status_path = … are two spellings of the same page. Setting the flag and an explicit path for the same format is a startup error naming both; setting neither leaves that format off. A path directive with no path (operator.status_listen alone) binds nothing — the address is where an endpoint would go, not an instruction to open one.

The default is a loopback address on purpose. These pages describe the inside of your process tree, and listen.allowed_clients is a weaker boundary than not binding the port outward at all. The port is the Prometheus registry's PHP-FPM exporter port, chosen so a scraper that already knows it needs no mapping.

The old names are refused, not aliased

pm.status_path on any type but fastcgi, and pm.status_listen, pm.metrics_path and pm.metrics_listen on every type, are startup errors that name the replacement:

[pool tick] 'pm.metrics_path' was renamed to 'operator.metrics_path' (issue #386); update the configuration

Aliasing them would keep two names alive for one mechanism, which is the thing the rename removes. fastcgi is the single exception for pm.status_path: it keeps upstream's meaning (a path on the pool's own FastCGI socket, for the web server in front). Since issue #383, a FastCGI pool may also opt into the separate operator.* listener; those paths are answered over HTTP and do not replace pm.status_path.

Path-safe pool names

A pool that exposes a page carries its name into a URL: operator.metrics = on derives /metrics/<pool name>, and a gateway later forwards <base>/<pool name>. Such a pool's section name must therefore contain only the characters [alphanum]/_-.~. A name with anything else ([bad name], a space) is refused at startup, naming the pool and the offending character. A pool that exposes nothing keeps whatever name it had.

One socket, many pools

Pools that name the same listen address share one socket and one process; their paths are routes on it. That process is created by php-fpm-ng, not by you: it is not a pool you can name, configure or see in pool.type's list of known types. It runs no PHP and speaks no FastCGI.

What is refused is two answers for one URL. An address, a port and a path identify one endpoint, so:

[a]
operator.status_listen = 127.0.0.1:9253
operator.status_path   = /status

[b]
operator.status_listen = 127.0.0.1:9253
operator.status_path   = /status

fails at startup, naming both pools. Give one of them a different path, or a different address. The same path on two different addresses is fine, and so is one pool serving both formats from one address.

A request for a path no pool claimed gets a 404 listing the paths that listener does answer.

A gateway is the one type whose paths default to being set (/status and /metrics, issue #388). Two gateways with no operator.*_listen therefore both want those paths on the default 127.0.0.1:9253 -- and that must still start, because the defaults are offered, not demanded. The first gateway to register a default path owns it; a later gateway whose derived path collides drops that page and logs a NOTICE naming the pool that owns it. An explicit operator.status_path / operator.metrics_path is not treated that way: if the operator wrote it, a collision is still a startup error. Give the second gateway a path or a listen address of its own to expose it, or operator.status = off / operator.metrics = off to say it has none. Note that an explicit off is honoured: it is not overwritten by the default.

Two masters on one host

The sharing above happens inside one master. The default address 127.0.0.1:9253 is global to the host, not derived from the pool's own listen (issue #561): a TCP port can be bound once, so a second master whose pools use the default fails at startup with unable to bind listening socket ... 9253. Gateways are affected by default, because their /status and /metrics are on unless turned off; any other type is affected as soon as it exposes a page (operator.status = on, operator.metrics = on, or an operator.status_path / operator.metrics_path) without a *_listen. The default is deliberately fixed so one scraper target covers the box (#386); the cost is that every additional master needs its own address. In each master after the first, either:

  • set operator.status_listen / operator.metrics_listen to an unused address (for example 127.0.0.1:9254), or
  • turn the pages off with operator.status = off / operator.metrics = off (and set no explicit path), so no operator listener is bound at all.

The phpt suite works around it with a shared --CONFLICTS-- key (build/phpt-parallel.sh), so no two gateway tests run at the same time.

One socket is one process, so it can have only one identity. The user, group, listen.owner, listen.group and listen.mode of the pools sharing an address must agree; if they do not, startup fails naming the directive and both pools. Give one of them its own address. This matters more than it looks, because the default address is the same for every pool: sharing is the normal case, not the unusual one.

Replacing a pool.type = status pool

That pool type is gone (issue #278). It was a listener of its own that reported on every other pool in the master — which is what the operator endpoint does, so keeping both meant two ways to configure the same listener and one page whose contents depended on a pool that was not the one you were reading about.

A configuration that still names it does not start: the master says pool.type 'status' no longer exists and names what to set instead, rather than ignoring the pool and leaving you with a port nobody answers on.

To migrate, delete the pool and put the two paths on the pools you actually want to watch, pointing them at the address the status pool used:

; before
[monitor]
listen = 127.0.0.1:9001
pool.type = status

; after
[api]
; …
operator.status_listen = 127.0.0.1:9001
operator.status_path = /api/status
operator.metrics_path = /api/metrics

[worker]
; …
operator.status_listen = 127.0.0.1:9001
operator.status_path = /worker/status
operator.metrics_path = /worker/metrics

Both *_listen directives default to 127.0.0.1:9253, so the four paths above share the one port the scraper was already pointed at. What changes for that scraper is the path: one target per pool instead of one target holding every pool. The JSON body is unchanged in shape — still {"pools":[…]} — but the array is one element long, so a client that iterated it keeps working.

Upgrading an http pool that already set pm.status_path

On the classic pool.type = http this directive used to be answered by upstream FPM's in-child handler, on the pool's public listener — reachable whenever http.front_controller was empty, because that is what makes SCRIPT_NAME the request path. On the types that carry an operator endpoint it is answered on the operator listener instead, and the public listener hands the path to your application like any other. (The classic http type is retired in favour of pool.type = gateway, #388; this paragraph is for configurations predating both changes.)

The page is not the same page: it is the per-pool JSON described below, not upstream's text/html/json/xml status body. A scraper pointed at the public listener has to move to the operator address. A FastCGI pool still keeps pm.status_path with its upstream meaning in full; issue #383 separately lets it opt into operator.* without moving that FastCGI-socket page.

Upgrading an http-direct pool that already set pm.status_path

The page is the same page — the one described in docs/http-direct.md, with the per-connection counters, the direct schema version and the per-child rows on ?full, unchanged field for field and byte for byte. What changed is the socket it is on: the operator listener instead of the pool's own, so a scraper moves from http://<listen>/status to http://<operator.status_listen>/status and keeps parsing exactly what it parsed before. ?json and ?full work there too.

Two consequences worth knowing before you compare numbers across the upgrade:

  • The scrape is no longer one of the pool's own requests. It used to arrive on the pool's listener and be counted — accepted conn and non-php requests both included it, and it appeared in access.log. Now it reaches a different process entirely, so those counters describe your traffic and nothing else.
  • The path is free on the public listener again. A request for /status there goes to your application like any other URL.

operator.status_listen is accepted on this type since the move; it used to be refused, because under its upstream meaning it asked for a second FastCGI socket a direct child has nowhere to put.

What the pages contain

Both formats report the pool that configured the path, and only that pool.

The status page is the pool type's own where the type has one: http-direct answers the page described above, and every other type answers the per-pool JSON described here. The metrics page is the same exposition format on every type, on purpose — a scraper reads one endpoint and compares labelled series across pools.

A pool that serves requests (fastcgi, http-direct) reports its worker counts and request total. On pool.type = http-direct with pool.executor = worker that request total is a real, per-request count since issue #333 — see http-direct.md for what else that executor does and does not report, including the two extra gauges (fpmng_pool_worker_pending, fpmng_pool_worker_watchers) it adds on top of the shape below, and Per-slot worker metrics below for a further, more detailed set. A fastcgi pool and an http-direct pool also report the saturation numbers in Saturation (issue #644). A pool that does not serve requests (cron, supervisor) reports its state, when it last started, how many consecutive failures it has had, its last exit code, and — for cron — when it next runs. A cron pool with cron.expect_within set also reports stale, and a supervisor pool whose script has called fpmng_supervisor_heartbeat() also reports heartbeat_age — both purely observational (issue #327); see cron.md and supervisor.md.

A pool.type = gateway runs no PHP child at all, so it has no state to report: its page carries fpmng_pool_info and its baseline counter, and no state block. Since issue #390 both come from the gateway's own shared-memory segment, which the master allocates in .init_main and the operator child renders: the baseline counter is the requests the gateway accepted, and the page adds fpmng_gateway_{upstreams_used,upstreams_max,requests_total,rejected_total}{pool, target}, one row per http.route[] target plus target="operator" for forwarded operator pages (#389) and target="-" for local answers (including every request on http.plain_listen), the fpmng_gateway_connections_open gauge (per gateway process, summed by the renderer) and the pool-wide fpmng_gateway_ping_total counter, and one fpmng_gateway_exposed_pool line per pool the gateway forwards for. The page also has the flow-control pair fpmng_gateway_responses_paused, a gauge of the paused responses of all gateway processes, and fpmng_gateway_responses_paused_total, a pool-wide counter of pauses (issue #706). Each target also has a request-duration histogram, fpmng_gateway_request_duration_seconds (issue #652). Its buckets and timing rules are in gateway.md. /status on the gateway is the same numbers as JSON, one row per target plus the pool row. The pool row also has responses_paused and responses_paused_total.

Known upstream bug: keep-alive counters (php/php-src#18956)

This applies to pool.type = fastcgi pools, including the ones behind a gateway (the gateway always sends FCGI_KEEP_CONN). It does not apply to http-direct pools. Pristine fastcgi.c calls its on_read() hook before the blocking read on a kept connection, so a worker that only starts waiting for the next request on a kept connection is already counted as reading headers. Effects on a gateway pool (the gateway keeps connections open, and drops idle ones after http.idle_timeout, 500 ms by default):

  • accepted conn (the pool's requests) and the per-process requests count one extra request for every kept connection the client closes without sending another request. This is a lasting counter error.
  • A worker that waits on a kept connection is shown as Reading headers in the per-process rows, and its request duration includes the idle wait.
  • max active processes stays too high, and idle processes / active processes can be wrong for up to one maintenance heartbeat (about 1 s).
  • request_terminate_timeout can hit an idle kept-alive worker, but only with http.idle_timeout = 0, a large value, or an external proxy using fastcgi_keep_conn on; with the default 500 ms and a whole-second timeout it cannot happen. request_slowlog_timeout is not affected (it fires only in the executing stage).
  • pm scaling is not affected: the maintenance loop recounts from each child's request stage every heartbeat.

This repository carries no php-src patch for it; the numbers are right once GH-18956 lands upstream. (Static reading, not measured.)

The baseline counter

Every pool reports one counter of its own invocations whether or not its PHP code ever calls fpm_metric_*(). What an invocation is depends on the type, so the counter's name does too:

Pool type Counter Counts
fastcgi, http-direct requests Requests served.
gateway requests Requests accepted (its own shm segment since #390).
cron runs Scheduled runs started.
supervisor restarts Times the supervised script was started again.

The name is the JSON key on the status page and, as fpmng_pool_<name>_total, the series on the metrics page — one name, two spellings of it, and they cannot disagree.

All three are monotonic and survive a worker being replaced. A pm.max_requests recycle does not reset requests, and a supervised child exiting does not reset restarts — the numbers live with the pool, not with the process.

restarts counts every start of the supervised script past the pool's first supervisor.processes of them — the starts that were meant to happen are not restarts, and a pool that has been up since boot without its script ever exiting reads zero.

restarts counts restarts, not failures. A script that exits 0 and is brought back by supervisor.restart = always leaves consecutive_failures at zero forever while restarts climbs; that combination is the signature of a pool that is flapping silently, and reading the two together is how you see it.

Per-slot worker metrics (issue #339)

On pool.type = http-direct with pool.executor = worker, operator.metrics_path also carries a set of series keyed by slot (this pool's scoreboard index, one worker child per slot) or by a reason/type label, on top of the pool-wide fpmng_pool_worker_pending and fpmng_pool_worker_watchers gauges issue #333 already added. They exist to answer three questions issue #333's two gauges cannot: which slot is under pressure, why a worker was refused or recycled, and how long its event loop has gone quiet.

operator.status_path stays unsupported on this executor (see http-direct.md) — only operator.metrics_path reports these.

Per-slot gauges. Each is only meaningful for a slot currently holding a live worker; a slot with no worker in it (not yet spawned, or between an exit and its replacement) reports 0 rather than a stale number from whichever worker had the slot before:

Series Labels Meaning
fpmng_pool_worker_queued pool, slot Requests accepted but not yet dispatched to the script (fw.ready_count).
fpmng_pool_worker_pending_oldest_seconds pool, slot Age, in seconds, of the oldest request this worker has accepted but not yet answered. 0 when nothing is pending.
fpmng_pool_worker_loop_stall_seconds_max pool, slot Longest gap seen so far between two consecutive fpmng_worker_loop() calls — how long the script's own code, not libevent, has ever held the loop.
fpmng_pool_worker_watchers pool, slot, type (read, write, timer) Libevent watchers this worker currently has registered, split by kind. The sum across type for one slot equals that slot's fpmng_pool_worker_watchers without the split, minus nothing — the two counts agree by construction.
fpmng_pool_worker_memory_bytes pool, slot This worker's peak RSS (getrusage(RUSAGE_SELF).ru_maxrss), the same sample worker.max_memory (issue #334) already computes for its own recycling decision.
fpmng_pool_worker_responses_unflushed pool, slot Bytes this worker has queued to a client but not yet confirmed written to the socket (worker.send_buffer_limit, issue #332).

Per-slot counter. Unlike the gauges above, this one is owned by the slot, not by whichever worker currently occupies it — it keeps counting across a pm.max_requests recycle instead of resetting to 0 when a new worker takes the slot:

Series Labels Meaning
fpmng_pool_worker_accepted_total pool, slot TCP connections accepted by this slot, ever. Counts connections, not requests — keep-alive means this can be lower than fpmng_pool_<name>_total's per-request count for the same traffic.

Pool-wide counters, one number for the whole pool rather than one per slot, because what they count outlives any one worker occupying any one slot:

Series Labels Meaning
fpmng_pool_worker_refused_total pool, reason (saturated, stopping, acl, bad_request) Requests this pool answered with an error instead of dispatching to a script, by reason. saturated: every ready slot was full. stopping: the worker holding the connection was already draining. acl: listen.allowed_clients rejected the peer. bad_request: the request itself was malformed.
fpmng_pool_worker_recycles_total pool, reason (max_requests, saturation, max_memory, max_lifetime, script_returned, signal) Times a worker in this pool was told to stop, by reason. max_requests/max_memory/max_lifetime are the matching pm.max_requests/worker.max_memory/worker.max_lifetime directives; saturation is this worker recycling itself after refusing a request it had no room for; script_returned is a script that exited on its own without ever being asked to stop; signal is every other stop signal (shutdown, reload, FPM\Tester::terminate(), …). max_lifetime is an addition beyond the metric's original proposal in issue #339, added because worker.max_lifetime (issue #334) is an existing, distinct recycle trigger the reason list would otherwise have no way to report.
fpmng_pool_worker_abandoned_total pool Requests this pool never answered at all: still pending when a worker's shutdown drain gave up on them, or a streamed response whose bytes never reached the socket before the worker exited.
fpmng_pool_worker_client_gone_total pool Requests whose client disconnected before this pool had sent any answer for them.

Saturation (issue #644)

A fastcgi pool and an http-direct pool with pool.executor = classic report three numbers that show how close the pool is to its limits. The master process writes them to the pool's scoreboard. The operator page reads them and does not change them. The worker executor does not report them. Use the per-slot series above for that executor.

Each series is present only when the pool can give a correct number:

Series JSON key (fastcgi status page) Type Present when
fpmng_pool_listen_queue{pool} listen_queue gauge The listener is TCP and the build reads TCP_INFO (Linux).
fpmng_pool_listen_queue_max{pool} listen_queue_max gauge The same as listen_queue.
fpmng_pool_listen_queue_length{pool} listen_queue_length gauge The same as listen_queue.
fpmng_pool_max_children_reached_total{pool} max_children_reached counter pm is dynamic or ondemand.
fpmng_pool_slow_requests_total{pool} slow_requests counter request_slowlog_timeout is set.

An absent series means "not measured". It does not mean zero.

The classic status page uses other key names: listen queue, max listen queue and listen queue length. It keeps its existing max children reached and slow requests lines and prints them on every classic pool.

Listen queue

The master samples the kernel's listen queue about once per second. Only a TCP listener has this queue. A unix socket listener never shows the three listen queue series.

  • listen_queue: connections that wait for accept() now.
  • listen_queue_max: the highest listen_queue since the pool started.
  • listen_queue_length: the backlog that the kernel allows for the listener.

A build without TCP_INFO does not sample the queue. A macOS development build is one example. On that build the three series are absent.

max_children_reached

The counter rises once for each episode. An episode starts when the master finds the pool at pm.max_children. It ends when the pool can start a child again. A pool that stays at its limit adds one, not one per heartbeat.

The series is absent when pm is static. A classic pool must use pm = static in a shipped build, so it never shows this series. On its status page the max children reached line reads 0. That 0 means the count does not apply.

slow_requests

The counter rises for each request that runs longer than request_slowlog_timeout. It can rise while the request still runs. A pool without request_slowlog_timeout does not trace requests, so the series is absent.

Selective reload

Under reload.selective = yes, a pool that the reload spares keeps its values. The master that starts after the reload counts episodes from zero. So max_children_reached can rise again while the pool is still at its limit.

Application metrics on a per-pool metrics path

operator.metrics_path carries the series your PHP code registered with fpm_metric_register() and fed with fpm_metric_inc(), fpm_metric_set() and fpm_metric_observe(), appended to the pool metrics above in the same exposition format.

It reports the scraped pool's workers and nobody else's. The store is one shared region with a slot per worker, and each pool owns a contiguous run of slots, so a per-pool scrape aggregates that run alone. Two pools that register a series under the same name get two independent numbers on two endpoints; a pool whose code registered nothing adds no lines at all.

Every series keeps its automatic pool="…" label, even though on a per-pool endpoint that label never varies. It is what lets a scraper that reads several pools' endpoints put the series side by side once they are in one database — without it, the same series name would mean a different pool depending on which port it was collected from.

There is no aggregate endpoint. Until issue #278 a pool.type = status pool answered /metrics with every pool's series at once; it was removed, because one pool that reported on all the others is the same listener the operator endpoint already runs, with a second configuration language for it. A scraper that wants several pools reads several paths — on one port if they share an operator.metrics_listen, which is the usual arrangement.

Turning metrics off

For every type except gateway, unset operator.metrics_path (and do not set operator.metrics). That is the off switch, and it is a real one: the path is not answered, and if nothing else on that address needs a listener the port is not bound at all.

On pool.type = gateway that is not the off switch (issue #491). Its paths default to being set (/status and /metrics, the paragraph above), so leaving both operator.metrics_path and operator.metrics unset leaves /metrics answered. Turn it off explicitly with operator.metrics = off, or with an explicitly empty operator.metrics_path =. (The same holds for operator.status and operator.status_path.) Verified on a running gateway: with both unset, GET /metrics on the public port returns the real metrics page; with operator.metrics = off it is not answered — and because the format is off and nothing else needs the listener, the operator metrics listener is not bound either — while /status and ordinary application traffic are untouched.

Turning the format off also turns off the gateway's forwarding of it: with http.operator = yes, <base>/<pool> for that format stops being served too (fpm_http_operator_base()).

It does not necessarily make the public URL return 404. With http.operator = yes the gateway stops treating the path as an operator page, but http.route[] still sees the request, so a route that claims the prefix (for example /) answers it as an ordinary application request. A 200 with the application's body is not a metrics response — assert on the body, not the status, when checking that metrics are off.

What none of this switches off is the API. fpm_metric_register() and the rest keep working in every pool, keep writing to the same shared slots, and keep costing exactly what they cost. A script cannot tell whether its pool exposes a metrics path, which is deliberate: turning off an endpoint is an operator's decision about exposure, not a change to what the application may call.

?openmetrics is not an alias

Upstream FPM emits OpenMetrics as a query flag on the status page. php-fpm-ng does not. GET <operator.status_path>?openmetrics is the status page being asked for a variant it does not have, exactly like ?json or ?xml; metrics live on operator.metrics_path and nowhere else. Two paths are what gives the two pages independent on/off switches, since "on" means "the path is set".

What is not here yet

  • ping.path stays on the pool's own listener on fastcgi and http-direct. It is a liveness probe for whatever is in front of the pool, so that is where it belongs — and on http-direct "in front of the pool" already is the pool's own listener, so it was answered locally from the start. On pool.type = gateway (the retired pool.type = http before issue #388), "in front of the pool" is the gateway process, and since issue #382 that is exactly who answers it: the gateway matches ping.path itself, before routing and before the request ever reaches a worker, so a locally answered ping never touches the scoreboard, pm.max_requests or a queue counter — it is not a request of the pool. See docs/gateway.md and docs/http-direct.md.
  • pool.type = fastcgi can opt into this listener with operator.* (issue #383); pm.status_path remains on its FastCGI socket.