Add deterministic trace replay benchmark

.env.example

@@ -0,0 +1,35 @@
+# Replay executor configuration. Copy to `.env` and adjust.
+# Real environment variables always override values in `.env`.
+
+# --- Redis ---------------------------------------------------------------
+
+# Which client to benchmark: phpredis | relay | predis (predis not yet supported)
+REPLAY_REDIS_CLIENT=phpredis
+
+REPLAY_REDIS_HOST=127.0.0.1
+REPLAY_REDIS_PORT=6379
+REPLAY_REDIS_PASSWORD=
+REPLAY_REDIS_DB=0
+REPLAY_REDIS_TLS=0
+# Reuse one connection per worker (recommended, matches production drop-ins)
+REPLAY_REDIS_PERSISTENT=1
+REPLAY_REDIS_TIMEOUT=2.0
+
+# --- Corpus --------------------------------------------------------------
+
+# Directory of captured traces (NNNNN.json + manifest.json)
+REPLAY_CORPUS=./corpus
+
+# --- SQL -----------------------------------------------------------------
+
+# sleep   = reproduce each query's captured duration as a usleep (no DB needed)
+# execute = run the captured queries against MySQL (needs a matching schema/data)
+REPLAY_SQL_MODE=sleep
+
+# Only used when REPLAY_SQL_MODE=execute
+REPLAY_DB_HOST=127.0.0.1
+REPLAY_DB_PORT=3306
+REPLAY_DB_NAME=
+REPLAY_DB_USER=
+REPLAY_DB_PASSWORD=
+REPLAY_DB_CHARSET=utf8mb4

.gitignore

@@ -0,0 +1,5 @@
+# Local secrets / config
+.env
+
+# Captured corpus working dir (keep the committed example traces)
+corpus/raw/

CAPTURE.md

@@ -0,0 +1,153 @@
+# Building the request-capture mechanism
+
+> Handoff instructions for a future Claude session. Read this top to bottom,
+> then read `corpus/00000.json` and `corpus/manifest.json` — those are the
+> canonical, hand-authored examples of the exact output you must produce.
+
+## Goal
+
+Capture real WordPress request data from a live site so it can be **replayed**
+later without WordPress or MySQL (see `TODO.md` and `replay.js`). For each of
+~10–100 sampled requests, record the interleaved timeline of **PHP compute
+gaps**, **Redis commands** (reads + writes, verbatim), and **SQL queries**
+(timing + full text), and write one JSON trace file per request into `corpus/`.
+
+The replay side already exists in skeleton form (`stubs/replay-server.php`
+serves traces by `?id=N`; `replay.js` drives it). Your job is the **capture**
+side and a small **manifest build** step. Do **not** change the replay/metrics
+behavior of `replay.js` or `lib/metrics.js`.
+
+## Output format (authoritative)
+
+The format is fully specified by the examples in this repo:
+- `corpus/00000.json` — one captured request (a homepage hit: alloptions read,
+  an options miss → SQL → SET, a main-query SQL, a pipelined write, a pipelined
+  read). **Match this schema exactly.**
+- `corpus/manifest.json` — the corpus index.
+
+Key rules (all illustrated in the example):
+- **One file per request**, zero-padded id (`00000.json`), id matches `?id=N`.
+- `timeline` is an **ordered, interleaved** list of events. Durations there are
+  **integer microseconds** (`us`). `summary` uses **float milliseconds**
+  (`ms_*`) and field names that match the existing metric vocabulary in
+  `lib/metrics.js` (`hit_ratio`, `bytes`, `sql_queries`, `redis_reads`/`writes`
+  ↔ `store-reads`/`store-writes`, `ms_total` ↔ `ms-total`).
+- Event types: `php` (compute gap → replay `usleep`), `redis` (one network
+  round-trip carrying 1+ commands), `sql` (query → replay `usleep`, text kept
+  for analysis).
+- `redis.mode` ∈ `single` | `pipeline` | `multi` — preserves round-trip
+  structure (round trips dominate latency; do not flatten pipelines).
+- **Verbatim Redis values**, **binary-safe**: each element of `cmd.args` is a
+  plain JSON string when it is valid UTF-8, otherwise the object
+  `{ "b64": "<base64>" }`. Keys stay readable; serialized/compressed payloads
+  (igbinary, lz4/zstd) are base64. Also record `value_bytes`/`ttl` on writes and
+  `reply_bytes`/`hit` on reads.
+
+Consider also emitting a `corpus/schema.json` (JSON Schema) and validating
+output against it.
+
+## What to build
+
+1. A capture **mu-plugin** modeled on `stubs/k6-metrics.php` (same coding
+   style, same `wp_object_cache` detection). Suggested:
+   `stubs/k6-capture.php`.
+2. A small **manifest build** step that scans the written trace files and emits
+   `corpus/manifest.json`. Suggested: `bin/build-corpus.php` (or `.sh`). Keep
+   capture and manifest separate so capture stays race-free (see below).
+
+## Where the data comes from
+
+### Redis commands + per-op timing — investigate first, then wrap
+
+There is no portable per-command hook in PhpRedis. Options, in order of
+preference:
+1. **Check OCP first.** Object Cache Pro may already expose command logging /
+   an analytics log you can tap (the user maintains OCP — confirm the current
+   API rather than guessing). If it yields ordered commands with timing, use it.
+2. **Decorate the client.** Otherwise wrap the drop-in's Redis client
+   (`$wp_object_cache->redis_instance()` for Redis Object Cache; OCP's client
+   accessor) in a logging proxy that, for every command, records
+   `{ cmd, args, t_start, t_end }` using `hrtime(true)`.
+   - For **pipelines/MULTI**, wrap the client's `pipeline()` / `multi()` so
+     commands buffered until `exec()` are grouped into one `redis` event with
+     the right `mode`; the round-trip `us` is measured across the `exec()`.
+   - Capture `hit` from read replies (non-null/found), `reply_bytes` from the
+     raw reply length, `value_bytes`/`ttl` from write args.
+
+Do **not** use Redis `MONITOR` — it drops under load, hides pipelining, and
+skews timing.
+
+### SQL — `$wpdb->queries`
+
+- Ensure `define('SAVEQUERIES', true)` for capture runs (document this; it adds
+  overhead so it must be capture-only).
+- After the request, read `$wpdb->queries`; each entry is
+  `[0] => SQL, [1] => seconds (float), [2] => caller`. Convert seconds → `us`.
+  `rows` is not in `$wpdb->queries` — leave it out or best-effort.
+
+### Timing totals & PHP gaps
+
+- `summary.ms_total` from `$timestart`/the OCP footnote (`ms-total`).
+- **PHP gaps are derived, not measured directly:** order all Redis + SQL ops by
+  their `hrtime` start, then a `php` event's `us` = gap between the previous
+  op's end and the next op's start. Prepend the bootstrap gap (request start →
+  first op) and append the tail gap (last op end → response). The sum of all
+  `php` + `redis` + `sql` durations should ≈ `ms_total`.
+
+## Sampling / triggering
+
+Capture must be **opt-in per request** so you control exactly which URLs become
+traces and avoid capturing admin/cron/REST noise:
+- Trigger on a request header/cookie/query flag (e.g. `X-K6-Capture: 1` or
+  `?k6_capture=1`), set only by the operator driving the capture (curl/k6 over a
+  fixed URL list). Bail early otherwise.
+- Reuse the bail-out guards from `stubs/k6-metrics.php` (skip WP-CLI, cron,
+  AJAX, REST, JSON, robots, etc.).
+- Write the trace on the `shutdown` hook (after the response, like
+  `k6-metrics.php`'s `maybePrint`).
+
+## Writing files & id assignment (avoid races)
+
+Multiple PHP-FPM workers may capture concurrently — do **not** assign sequential
+ids at write time. Instead:
+- Capture writes to a raw dir with a collision-free name, e.g.
+  `corpus/raw/<unixnanos>-<pid>-<rand>.json`, written atomically (temp file +
+  `rename()`).
+- `bin/build-corpus.php` later sorts raw files (by capture time or a provided
+  URL order), assigns sequential ids `0..N-1`, writes `corpus/NNNNN.json`, and
+  emits `corpus/manifest.json`. This is also where you can dedupe by URL or cap
+  the count.
+
+## Gotchas
+
+- **Binary values**: OCP serializes (igbinary) and may compress (lz4/zstd) —
+  values are binary; you MUST base64 them per the encoding rule. Verify a
+  round-trip (`base64_decode` reproduces the exact bytes).
+- **Persistent connections / object cache state**: capture reflects whatever was
+  warm at request time; the `hit`/`miss` flags record that. Replay's warm-vs-cold
+  handling is a separate concern (`TODO.md`).
+- **SAVEQUERIES overhead**: capture-only; never enable on the benchmarked SUT.
+- **Don't let capture itself pollute the trace**: exclude the plugin's own
+  Redis/file I/O from the recorded ops.
+- Keep file sizes sane: a homepage `alloptions` blob can be large; verbatim +
+  base64 inflates ~33%. Fine for ~100 requests; gzip (`*.json.gz`) only if
+  needed.
+
+## Verification
+
+1. Stand up (or point at) a WordPress site with an object cache drop-in.
+2. Drive a known URL with the capture trigger; confirm a raw file appears.
+3. Run `bin/build-corpus.php`; confirm `corpus/NNNNN.json` + `manifest.json`
+   are produced and that a produced trace **validates against the example
+   schema** (same keys/shape as `corpus/00000.json`).
+4. Round-trip a `{b64}` value: `base64_decode` equals the original bytes.
+5. Sanity-check timing: `sum(php+redis+sql us) ≈ ms_total * 1000`.
+6. End-to-end: once `stubs/replay-server.php` is updated (later phase) to read
+   `corpus/`, confirm `k6 run replay.js` drives the real corpus.
+
+## Out of scope (later phases)
+
+- The real replay executor (firing the captured commands via Relay/PhpRedis in
+  `stubs/replay-server.php`).
+- The PhpBench subject (separate track in `TODO.md`).
+- Do **not** modify `replay.js` or `lib/metrics.js`.

README.md

@@ -44,6 +44,83 @@ k6 run woo-customer.js --env SITE_URL=https://example.com --env BYPASS_CACHE=1
 
 This script requires [seeded users](#seeding-users).
 
+### `replay.js`
+
+Replays a fixed corpus of captured requests ("traces") against a stateless
+replay endpoint, instead of driving a real WordPress site. Every run executes
+the *identical* workload — the same traces in the same per-VU order — so
+results are comparable across runs and across horizontally scaled load
+generators. k6 owns which trace runs when; the server just executes whatever
+`id` it is handed.
+
+By default it fires 100 unique traces, each repeated until 10,000 requests are
+reached, as fast as the system under test allows (closed-loop).
+
+```bash
+k6 run replay.js --env SITE_URL=http://localhost:8080
+k6 run replay.js --env SITE_URL=http://localhost:8080 --env VUS=200
+```
+
+The load profile is selectable via `--env SCENARIO=`:
+
+```bash
+# fixed (default): exactly TOTAL requests, each trace run equally often
+k6 run replay.js --env SITE_URL=http://localhost:8080 --env SCENARIO=fixed --env TOTAL=10000
+
+# ramping: ramp concurrency up/down to find the saturation point
+k6 run replay.js --env SITE_URL=http://localhost:8080 --env SCENARIO=ramping \
+  --env STAGES="30s:50,1m:200,2m:200,30s:0"
+
+# constant: hold VUS for a duration
+k6 run replay.js --env SITE_URL=http://localhost:8080 --env SCENARIO=constant --env VUS=100 --env DURATION=2m
+```
+
+Tunable via env vars: `SCENARIO` (`fixed`/`ramping`/`constant`/`shared`),
+`TRACES` (unique traces, default 100), `TOTAL` (total requests, default 10000),
+`VUS` (concurrency, default 100), `STAGES`, `DURATION`, `START_VUS`,
+`MAX_DURATION`, `REPLAY_PATH` (endpoint path, default `/render`).
+
+Metrics are collected the same way as the other scripts — from the [Object
+Cache Pro footnote](lib/metrics.js) comment in the response body (enable OCP's
+`analytics.footnote` on the real SUT).
+
+To scale across multiple load-generator machines, use k6 execution segments —
+the trace mapping is segment-safe, so coverage stays complete and reproducible:
+
+```bash
+# machine 1 of 4
+k6 run replay.js --env SITE_URL=http://lb \
+  --execution-segment "0:1/4" \
+  --execution-segment-sequence "0,1/4,2/4,3/4,1"
+```
+
+There are two endpoints to drive with `replay.js`:
+
+**Real executor — [`replay/index.php`](./replay/index.php)** replays captured
+traces (`corpus/NNNNN.json`) against a real Redis using the configured client
+(phpredis/relay), and emits a measured Object Cache Pro-style footnote so the
+metrics above are real. Configure it with environment variables — copy
+[`.env.example`](./.env.example) to `.env` and adjust:
+
+```bash
+cp .env.example .env          # set REPLAY_REDIS_* (and REPLAY_REDIS_CLIENT)
+php -S 0.0.0.0:8080 replay/index.php
+k6 run replay.js --env SITE_URL=http://localhost:8080
+```
+
+SQL is reproduced as a sleep by default (`REPLAY_SQL_MODE=sleep`, no database
+needed); set `REPLAY_SQL_MODE=execute` with `REPLAY_DB_*` to run the captured
+queries against MySQL instead. For real benchmarking, serve via nginx +
+PHP-FPM so the corpus and connections are preloaded/persisted per worker.
+
+**Dummy — [`stubs/replay-server.php`](./stubs/replay-server.php)** needs no
+Redis: it fakes the work and prints a fake footnote, useful for exercising
+`replay.js` and the metrics pipeline offline.
+
+```bash
+php -S 0.0.0.0:8080 stubs/replay-server.php
+```
+
 ## Reset WooCommerce
 
 ```

TODO.md

@@ -0,0 +1,99 @@
+# WordPress benchmark simulation — TODO
+
+## Goal
+
+Benchmark a WordPress site's cache/Redis layer **without** standing up
+WordPress, MySQL or doing real page rendering. We capture a small corpus of
+real requests once, then *replay* them: sleep for the captured PHP compute
+time and fire the captured Redis commands. This isolates the Redis/Relay layer
+and lets every benchmark run execute an **identical, reproducible workload** so
+systems (Relay vs PhpRedis vs Predis, Redis vs Valkey/KeyDB, single vs
+clustered, etc.) can be compared apples-to-apples — including horizontally
+scaled setups.
+
+## Architecture
+
+```
+  (1) capture          (2) corpus            (3) replay endpoint        (4) driver
+  ───────────          ──────────            ───────────────────        ──────────
+  real WP request  →   N traces of      →    stateless PHP-FPM     ←──   k6 (replay.js)
+  (one-off)            Redis cmds +          endpoint: id in,             fires ?id=N,
+                       compute time +        replay trace, return         owns ordering
+                       TTLs + batches        metrics                      & determinism
+```
+
+- **The server is dumb / stateless.** k6 decides which trace runs and in what
+  order via `?id=N`; the server just executes trace N. All determinism lives in
+  the driver, which keeps runs comparable and lets us shard across machines.
+- **Closed-loop** by default (run as fast as the SUT allows) for capability
+  comparison; ramping/constant scenarios available for saturation testing.
+
+## What's done
+
+- [x] `replay.js` — k6 driver. Deterministic, segment-safe trace selection;
+      selectable scenarios (`fixed` / `ramping` / `constant` / `shared`) via
+      `--env SCENARIO=`; configurable `VUS`, `TOTAL`, `TRACES`, `STAGES`, etc.
+- [x] `stubs/replay-server.php` — dummy endpoint that fakes the work and returns
+      the metric JSON shape `replay.js` parses. Placeholder for the real
+      executor.
+- [x] README section documenting usage.
+
+## What needs to be done
+
+### Capture (not the hard part — do later)
+- [ ] Instrument the real Redis client (wrap Relay/PhpRedis as used by the
+      object-cache drop-in) to log per request: ordered commands + args,
+      **pipeline/MULTI batch boundaries**, TTLs, response sizes, and the
+      **per-batch timing gaps** (not just one total). Tag each with a request id.
+- [ ] Decide capture mechanism — client wrapper preferred over Redis `MONITOR`
+      (MONITOR drops under load, hides pipelining, skews timing).
+- [ ] Capture 100 representative requests → corpus file(s) (one per id).
+- [ ] Decide corpus format (JSON/MessagePack) and how the server preloads it
+      (static array / APCu) so request handling is pure CPU + Redis.
+
+### Replay executor (replace the dummy)
+- [ ] Implement the real replay in `stubs/replay-server.php` (or a small app):
+      load trace by id, `usleep` the captured compute/gap times, fire the
+      captured Redis commands via the **production client**, preserving batch
+      structure & TTLs, and **consume responses** (so client-side
+      deserialization / Relay's in-memory layer is exercised).
+- [ ] Run behind real **nginx + PHP-FPM** (not `php -S`) for a realistic
+      process model. Pin `pm.max_children` (sleeping workers hold slots) and
+      use persistent Redis connections to match production.
+- [ ] Decide warm vs cold cache: pre-warm pass, or discard first N iterations.
+- [ ] Emit real metrics in the response (total_ms, redis_ms, cmd_count, hits,
+      misses, bytes) — `replay.js` already parses these.
+
+### Driver / orchestration
+- [ ] Validate distributed runs with execution segments across >1 machine.
+- [ ] Optional `bin/` wrapper for a concurrency sweep (e.g. VUS=50/100/200/400)
+      to produce a capability curve.
+- [ ] Decide metrics sink: k6 JSON summary to start, Prometheus/Grafana later.
+
+## Separate track: PhpBench micro-benchmark of the Redis fake workload
+
+In addition to the k6 / HTTP path above, we want to run the **same captured
+Redis fake workload under [PhpBench](https://phpbench.readthedocs.io)**, as a
+standalone, single-process micro-benchmark. This gives us tight statistical
+rigor (revs/iterations, mean/mode/stdev, retry until stable) on the *pure*
+replay cost — no HTTP, nginx or FPM in the path — which complements the k6
+capability/concurrency numbers.
+
+- [ ] Add a PhpBench `*Bench.php` subject that loads a trace and replays its
+      Redis commands via the production client (reuse the same replay code as
+      the endpoint so both paths exercise identical logic).
+- [ ] Parameterize over the corpus (`@ParamProviders`) so each trace is its own
+      benchmarked subject; report per-trace and aggregate.
+- [ ] Pin revs/iterations/warmup; commit a `phpbench.json` config.
+- [ ] Compare clients/backends (Relay / PhpRedis / Predis) as separate runs.
+- [ ] Keep this isolated-Redis benchmark distinct from the k6 end-to-end one —
+      they answer different questions (raw call cost vs system capability).
+
+## Open decisions
+
+- [ ] Corpus multiplicity: 100 unique traces × 100 repeats = 10,000 (current
+      default). Confirm or adjust per real capture.
+- [ ] CPU contention: `sleep` yields the CPU (conservative, isolates Redis).
+      Switch to a calibrated busy-loop only if PHP CPU cost must be modeled.
+- [ ] Key namespacing: replay captured keys verbatim (realistic shared-cache
+      contention) vs per-worker namespacing. Default: verbatim.