Skip to content

Commit b62ca23

Browse files
committed
docs(bench): record Ubuntu benchmark artifact
1 parent 50d5dee commit b62ca23

3 files changed

Lines changed: 81 additions & 105 deletions

File tree

‎.superpowers/sdd/2026-09-21-parity-bench-binary/task-4-report.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,3 +132,37 @@ Because stderr from the parser is redirected to `/dev/null`, the workflow surfac
132132
## Parser fix: 2026-09-21
133133

134134
Replaced the incompatible f-string expression in `scripts/bench.sh` with `str.format()` using the same five JSON fields and numeric format specifiers. This removes the escaped-quote syntax error without changing benchmark semantics or output format. Added `scripts/test-bench-parser.sh`, which validates the recorded JSON shape and confirms malformed JSON and missing required fields still fail.
135+
136+
## Complete rerun result: 2026-09-21
137+
138+
### Status
139+
140+
Completed. The Ubuntu benchmark workflow succeeded, and the required artifact was downloaded and inspected. Placeholder benchmark evidence was replaced with the artifact's actual Ubuntu data. Multi-platform coverage remains deferred. No quantization-specific benchmark was run.
141+
142+
### Provenance
143+
144+
- Branch: `feat/batched-inference`
145+
- Branch SHA: `50d5dee10da4db7b7820082da7a3ea0eaab5f346`
146+
- Workflow: `.github/workflows/bench.yml`
147+
- Runner: `ubuntu-latest`, platform reported by artifact as `linux-x86_64`
148+
- Dispatch command: `gh workflow run bench.yml --ref feat/batched-inference -f variant=small -f repeats=5 -f threads=2`
149+
- Run: [35629935462](https://github.com/espetro/dinov2.cpp/actions/runs/35629935462)
150+
- Run head SHA: `50d5dee10da4db7b7820082da7a3ea0eaab5f346`
151+
- Artifact: `bench-results-small-35629935462/benchmark_results.txt`
152+
- Artifact generated: `2026-09-21T17:08:06Z`
153+
154+
### Actual measurement
155+
156+
The artifact reported the small f16 GGUF on the CPU backend with one warmup, five timed runs, one 224x224 image, batch 1, and two OpenMP threads:
157+
158+
- Mean: `212.8 ms`
159+
- Standard deviation: `9.1 ms`
160+
- Minimum: `204 ms`
161+
- Maximum: `225 ms`
162+
- Peak RSS: `104 MB`
163+
164+
### Repository outcome
165+
166+
- `benchmark_results.txt`: replaced with the inspected artifact data plus explicit run and commit provenance.
167+
- `docs/benchmarks.md`: updated to report only the actual Ubuntu small-model measurement and deferred multi-platform coverage.
168+
- No models, build outputs, or release artifacts were added to the repository.

‎benchmark_results.txt‎

Lines changed: 15 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,16 @@
11
# DINOv2 inference benchmark (forward pass wall time, ms; lower is better).
2-
#
3-
# Generated by scripts/bench.sh. Order-of-magnitude seed values from the plan
4-
# file at .agents/plans/2026-09-18-post-v0.2.0.md. Real numbers will be
5-
# regenerated by the bench.yml manual-trigger workflow once PR C lands.
6-
#
7-
# Numbers here are placeholders, not measurements. Do not cite them as
8-
# performance claims until the bench.yml workflow runs them.
9-
#
10-
# Methodology: 1 warmup + 5 timed runs, single 224x224 image (batch=1),
11-
# --threads=min(4, hw_concurrency), f16 GGUF unless noted.
12-
# Peak RSS is for the ViT-S column; it grows roughly linearly with model size
13-
# (B ~3x, L ~10x, g ~40x). All rows exercise the CPU backend (Metal/CUDA
14-
# are deferred -- see .agents/plans/2026-09-18-post-v0.2.0.md section 1).
15-
# | Platform | Arch | ggml | ViT-S | ViT-B | ViT-L | ViT-g | Peak RSS (ViT-S) |
16-
# |--------------------|--------|------|--------|--------|--------|---------|------------------|
17-
# | macos-arm64 (M1) | arm64 | cpu | 72 | 198 | 580 | 1980 | 110 MB |
18-
# | ubuntu-x64 (Xeon) | x86_64 | cpu | 165 | 410 | 1190 | 4020 | 110 MB |
19-
# | ubuntu-arm64 | arm64 | cpu | 140 | 360 | 1050 | 3550 | 110 MB |
20-
# | windows-x64 | x86_64 | cpu | 170 | 420 | 1210 | 4080 | 110 MB |
21-
#
22-
# Quantization sweep (ViT-S on ubuntu-x64 baseline):
23-
# | Quant | mean (ms) | peak RSS |
24-
# |-------|-----------|----------|
25-
# | f16 | 165 | 109 MB |
26-
# | q4_0 | 50 | 49 MB |
27-
# | q4_1 | 50 | 52 MB |
28-
# | q5_0 | 59 | 54 MB |
29-
# | q5_1 | 57 | 57 MB |
30-
# | q8_0 | 51 | 70 MB |
31-
#
32-
# Notes:
33-
# - Numbers are seed values from order-of-magnitude estimates. Real numbers
34-
# will be regenerated by the bench.yml manual-trigger workflow (see
35-
# .agents/plans/2026-09-18-post-v0.2.0.md for the workflow design).
36-
# - All rows exercise the CPU backend (Metal/CUDA CI is deferred).
37-
# - Single 224x224 image (assets/tench.jpg or equivalent), batch=1,
38-
# 1 warmup + 5 timed runs each cell, --threads=min(4, hw_concurrency).
2+
# Actual artifact from GitHub Actions run 35629935462:
3+
# https://github.com/espetro/dinov2.cpp/actions/runs/35629935462
4+
# Branch: feat/batched-inference
5+
# Commit: 50d5dee10da4db7b7820082da7a3ea0eaab5f346
6+
# Workflow inputs: variant=small, repeats=5, threads=2
7+
# Artifact: bench-results-small-35629935462/benchmark_results.txt
8+
# Generated by scripts/bench.sh on 2026-09-21T17:08:06Z.
9+
# Platform: linux-x86_64 on ubuntu-latest; CPU backend; f16 GGUF only.
10+
# Methodology: 1 warmup + 5 timed runs, one 224x224 image (batch=1), 2 OpenMP threads.
11+
12+
| Model | mean (ms) | stddev (ms) | min (ms) | max (ms) | peak RSS (MB) |
13+
|:------|----------:|------------:|---------:|---------:|---------------:|
14+
| small | 212.8 | 9.1 | 204 | 225 | 104 |
15+
16+
# Multi-platform measurements are deferred. No quantization-specific benchmark was run.

‎docs/benchmarks.md‎

Lines changed: 32 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,27 @@
11
# Benchmarks
22

3-
Numbers below are order-of-magnitude estimates, regenerated by
4-
[`scripts/bench.sh`](../scripts/bench.sh) running across CI runners
5-
(`ubuntu-latest`, `ubuntu-24.04-arm`, `macos-latest`, `windows-latest`).
6-
The committed `benchmark_results.txt` ships as a placeholder; the
7-
real per-platform tables are produced by the `.github/workflows/bench.yml`
8-
manual-trigger workflow once PR C lands.
9-
10-
Headline: dinov2.cpp is **3x faster than native PyTorch inference on CPU**
11-
for the small model and roughly **1.5-2x faster** for the rest, with
12-
substantially lower memory use. (See "vs PyTorch" below for the
13-
historical comparison numbers.)
3+
The table below contains the actual Ubuntu benchmark artifact produced by
4+
[`scripts/bench.sh`](../scripts/bench.sh) in GitHub Actions. Multi-platform
5+
measurements are deferred.
6+
7+
The benchmark ran on `.github/workflows/bench.yml` run
8+
[35629935462](https://github.com/espetro/dinov2.cpp/actions/runs/35629935462)
9+
from `feat/batched-inference` at commit
10+
`50d5dee10da4db7b7820082da7a3ea0eaab5f346`. The workflow inputs were
11+
`variant=small`, `repeats=5`, and `threads=2`; the uploaded artifact was
12+
`bench-results-small-35629935462/benchmark_results.txt`.
1413

1514
## Forward pass wall time (CPU backend)
1615

1716
Forward pass only -- the timed block excludes image load, preprocess,
1817
and PCA visualization. Numbers in milliseconds (lower is better).
1918

20-
| Platform | Arch | ggml | ViT-S | ViT-B | ViT-L | ViT-g | Peak RSS (ViT-S) |
21-
|--------------------|--------|------|------:|------:|------:|------:|------------------|
22-
| macos-arm64 (M1) | arm64 | cpu | 72 | 198 | 580 | 1980 | 110 MB |
23-
| ubuntu-x64 (Xeon) | x86_64 | cpu | 165 | 410 | 1190 | 4020 | 110 MB |
24-
| ubuntu-arm64 | arm64 | cpu | 140 | 360 | 1050 | 3550 | 110 MB |
25-
| windows-x64 | x86_64 | cpu | 170 | 420 | 1210 | 4080 | 110 MB |
26-
27-
Seed values from the plan file at
28-
[`.agents/plans/2026-09-18-post-v0.2.0.md`](../.agents/plans/2026-09-18-post-v0.2.0.md#3-benchmark-infrastructure--workstream-b-in-progress).
29-
Treat as placeholders until bench.yml produces real numbers.
30-
31-
Quantized GGUFs (q4_0 / q4_1 / q5_0 / q5_1 / q8_0) are pulled from
32-
[`dinov2-cpp-core/<variant>-gguf`](https://huggingface.co/dinov2-cpp-core);
33-
the `dinov2-cli` binary loads them transparently and a per-quant bench
34-
table can be produced by pre-staging those GGUFs under
35-
`models/<variant>/model.<q>.gguf` and running
36-
`scripts/bench.sh --models small --repeats 5` with the desired quant
37-
selected by which GGUF you pre-place.
19+
| Platform | Arch | ggml | Model | mean (ms) | stddev (ms) | min (ms) | max (ms) | Peak RSS |
20+
|-----------------|--------|------|--------|----------:|------------:|---------:|---------:|---------:|
21+
| ubuntu-latest | x86_64 | cpu | ViT-S | 212.8 | 9.1 | 204 | 225 | 104 MB |
22+
23+
This is the only current platform measurement. Other platform rows remain
24+
deferred, and no quantization-specific benchmark was run.
3825

3926
## How to read the tables
4027

@@ -65,40 +52,23 @@ cold CPU cache) is in the loop.
6552
linearly with the patch grid.
6653
- **Warmup:** 1 discarded run (`--bench-warmup 1`).
6754
- **Timed runs:** 5 (`--bench-runs 5`).
68-
- **Threads:** `min(4, hw_concurrency)` -- the same default as
69-
`dino_params::n_threads`.
70-
- **Batch:** 1 (`--batch` exists for multi-image runs; the tables bench
55+
- **Threads:** 2 OpenMP threads, supplied by the workflow input.
56+
- **Batch:** 1 (`--batch` exists for multi-image runs; this table benches
7157
one image per pass).
72-
- **Backend:** CPU only -- the macos-arm64 row does NOT exercise Metal.
73-
- **Quantization:** f16 baseline. Pre-quantized GGUFs can be benched by
74-
pre-staging them under `models/<variant>/model.<q>.gguf` (download from
75-
`dinov2-cpp-core/<variant>-gguf`); `dinov2-cli` loads them transparently.
58+
- **Backend:** CPU only.
59+
- **Precision:** f16 GGUF.
7660
- **Build:** `cmake --build build --target dinov2-cli` in
7761
`Release` configuration (no `DEBUG`/`RelWithDebInfo`).
7862

79-
The current `benchmark_results.txt` is regenerated by
80-
`scripts/bench.sh` writing a markdown table per platform, then
81-
`--aggregate`'d into a single file with platform headers.
63+
The committed `benchmark_results.txt` is the downloaded artifact from the
64+
Ubuntu workflow run identified above. Additional platform measurements are
65+
deferred.
8266

8367
## Hardware
8468

85-
Four CI runners, all on the GitHub-hosted matrix:
86-
87-
- **macos-arm64 (M1)** -- Apple M1, 8 cores (4 perf + 4 efficiency), 16 GB.
88-
Uses the Accelerate-backed BLAS in ggml's CPU backend. The fastest
89-
per-core row in the matrix.
90-
- **ubuntu-x64 (Xeon)** -- `ubuntu-latest`, Intel Xeon Platinum 8275CL
91-
@ 3.0 GHz, 2 vCPU visible to the runner (the rest are billed but not
92-
scheduled). Single-threaded matmul performance is roughly 60% of the
93-
M1's perf cores.
94-
- **ubuntu-arm64** -- `ubuntu-24.04-arm`, AWS Graviton3 (Neoverse V1),
95-
4 vCPU. The Neon SIMD backend is competitive with AVX2 on Skylake for
96-
the matmul shapes ViT uses.
97-
- **windows-x64** -- `windows-latest`, Intel Xeon E5-2673 v4 @ 2.3 GHz,
98-
2 vCPU. MSVC-compiled binary; the slowest per-core row.
99-
100-
CPU specs above are GitHub-hosted runner specifications as of 2025;
101-
they may drift without notice.
69+
The evidence currently covers one GitHub-hosted `ubuntu-latest` runner. The
70+
runner's exact hardware may drift without notice; the artifact records the
71+
platform as `linux-x86_64`.
10272

10373
## Caveats
10474

@@ -121,24 +91,18 @@ they may drift without notice.
12191
- **No KV cache.** DINOv2 is a vision encoder with no autoregressive
12292
decode step -- there is no KV cache to populate. The "no KV cache"
12393
caveat in some other engines doesn't apply here.
124-
- **Metal/CUDA rows deferred.** The `macos-arm64` row above is the CPU
125-
backend, not Metal. Adding Metal is a one-line cmake flag change in
126-
`release.yml`; tracked in
127-
[the plan, section 1](../.agents/plans/2026-09-18-post-v0.2.0.md#1-gpu-ci--deferred).
128-
CUDA needs a larger GitHub-hosted runner (Tesla T4) which is not yet
129-
configured for the org.
130-
- **CI runner noise.** The `stddev` column on regenerated runs is
131-
expected to be 1-3% of the mean; >5% means the runner was loaded.
132-
- **Numbers are placeholders.** The committed `benchmark_results.txt`
133-
is seed data; bench.yml will overwrite it.
94+
- **Multi-platform coverage is deferred.** This evidence contains only the
95+
Ubuntu CPU run described above. Metal and CUDA measurements are not included.
96+
- **CI runner noise.** The `stddev` column is sensitive to load on the shared
97+
runner; the recorded run measured 9.1 ms of standard deviation.
13498
- **Embeddings flags are single-shot only.** `--print-embeddings` and
13599
`--print-patch-tokens` exist for output inspection; the bench path
136100
ignores them and reports timings only.
137101

138102
## Reproduce locally
139103

140-
The committed `benchmark_results.txt` will eventually be a CI artifact;
141-
in the meantime, you can reproduce locally:
104+
The committed `benchmark_results.txt` records the Ubuntu CI artifact;
105+
you can reproduce the measurement locally with:
142106

143107
```bash
144108
# Build the dinov2-cli binary first.

0 commit comments

Comments
 (0)