Skip to content

Latest commit

 

History

History
14 lines (10 loc) · 871 Bytes

results.md

File metadata and controls

14 lines (10 loc) · 871 Bytes

Benchmark results

Measured 2026-09-18 with benchmarks/run.sh (Locust 100 users, 60s per mode, RATE_LIMIT=off). Server: docker compose, 1 uvicorn worker, ONNX Runtime CPU. Client (Locust) ran on the same machine, so it competes for CPU.

Hardware: AMD Ryzen 7 5800H with Radeon Graphics | 16 logical cores | Windows-10-10.0.26200-SP0 | Docker version 29.8.0, build 88096ef

Mode Throughput (req/s) P50 (ms) P99 (ms) Requests Failures
Naive (MAX_BATCH_SIZE=1, no Redis) 18 4500 10000 1079 38 (3.5%)
Dynamic batching (max 32, 50ms, no Redis) 37 2300 5200 2213 0 (0.0%)
Batching + Redis cache (100% hits after warm-up) 653 130 210 38412 0 (0.0%)

Raw Locust CSVs are in benchmarks/results/ (gitignored); rerun the script to regenerate.