Measured 2026-09-18 with benchmarks/run.sh (Locust 100 users, 60s per mode, RATE_LIMIT=off).
Server: docker compose, 1 uvicorn worker, ONNX Runtime CPU. Client (Locust) ran on the same machine, so it competes for CPU.
Hardware: AMD Ryzen 7 5800H with Radeon Graphics | 16 logical cores | Windows-10-10.0.26200-SP0 | Docker version 29.8.0, build 88096ef
| Mode | Throughput (req/s) | P50 (ms) | P99 (ms) | Requests | Failures |
|---|---|---|---|---|---|
| Naive (MAX_BATCH_SIZE=1, no Redis) | 18 | 4500 | 10000 | 1079 | 38 (3.5%) |
| Dynamic batching (max 32, 50ms, no Redis) | 37 | 2300 | 5200 | 2213 | 0 (0.0%) |
| Batching + Redis cache (100% hits after warm-up) | 653 | 130 | 210 | 38412 | 0 (0.0%) |
Raw Locust CSVs are in benchmarks/results/ (gitignored); rerun the script to regenerate.