Skip to content

Commit c08cfda

Browse files
authored
Merge pull request #11 from espetro/feat/batched-inference
feat: add batched and multi-image inference
2 parents e1dae6c + 6db2e3a commit c08cfda

9 files changed

Lines changed: 892 additions & 197 deletions

File tree

‎.agents/memory/dinov2_cpp.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,20 @@
11
# dinov2.cpp fork — project notes
22

3+
## 2026-09-21 — embeddings-dropin PR 1 complete (branch feat/embeddings-dropin, 5cd839d..16a5cb6)
4+
5+
- Goal: drop-in replacement for HF PyTorch DINOv2. Swap-score was 2.5/10; PR 1 closes the hard blockers (embeddings output, classify parity, agent-facing CLI/docs).
6+
- `dino_output` now carries cls, pooled `[cls‖mean(patches)]`, patch tokens (feature mode), `preds` (class idx) + `pred_scores`. Registers excluded from patch view in BOTH modes — HF bug #37817 parity; the earlier classify pooling included registers and diverged ~0.0066 prob on with-registers checkpoints.
7+
- Classify preprocessing is 256-shortest-edge + 224-crop per HF `preprocessor_config.json` — NOT the GGUF `img_size` 518 (that's backbone-native; resizing to 518 breaks parity). Gap report's suggested 518 fix was wrong.
8+
- Two latent bugs the gap report missed: `preds[i]` stored probability-as-uint32 (always ~0) not class idx; `forward_head` divided by `n_img_embd²` (1369 at img_size 518) instead of actual patch count — pooled features ~5.3× under-scaled at 224 input.
9+
- CLI: `--print-embeddings` (JSON stdout: model/image/n_patches/hidden/cls/pooled, +patches w/ --print-patch-tokens, +topk w/ -c), `--l2-normalize`, `--version` (CMake `DINOV2_VERSION` define from PROJECT_VERSION, "dev" fallback), `-o` now opt-in for PCA PNG (was default `pca_visual.jpg`, actually PNG bytes), unknown-arg exits 1, argv bounds on all value flags, data on stdout / logs on stderr.
10+
- Docs: `docs/cli.md` is the full agent-facing reference; `--help` links the raw.githubusercontent URL. Model filename is `model.gguf` (HF `dinov2-cpp-core/*-gguf` repos), not `ggml-model.gguf`.
11+
- `scripts/parity_check.py`: HF AutoModel vs CLI JSON. Gates: cls/pooled/flat-patch cosine ≥0.999, top-1 match, |Δprob|<0.05. Per-token-min is diagnostic only (f16 tail ~0.98 expected); strict opt-in via `--patches-token-min-threshold`. Observed: no-reg cls 0.999911 / flat 0.999570; with-reg cls 0.999984 / flat 0.999847.
12+
- CMakePresets.json now has buildPresets — `cmake --build --preset <name>` works (docs always claimed it; it silently couldn't before).
13+
- Deferred minors in `.superpowers/sdd/2026-09-21-embeddings-dropin/progress.md` (json_escape control chars, nan/inf in %.6g, div-by-zero guard, nothing-to-do hint omits --bench).
14+
- PR 2 pending: `--batch N` batching. Known hardcoded-1 sites: dinov2.cpp input tensor (~472), flash reshape (~390), non-flash reshape (~403-404); attention partially batch-aware (`B = cur->ne[3]`).
15+
- Process note: no GitHub Project configured for this repo — user approved proceeding without one; raise later.
16+
- Norma MCP unavailable this session (server dropped) — live_check never ran; don't claim it.
17+
318
## Repo identity
419

520
- Upstream: `https://github.com/lavaman131/dinov2.cpp` (no releases, no CI)

‎ARCHITECTURE.md‎

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -144,16 +144,20 @@ Model:
144144
-t N, --threads number of threads to use during computation (default: 4)
145145
146146
Input:
147-
-i FNAME, --inp input image file (default: ../assets/tench.jpg)
147+
-i FNAME, --inp input image file; repeat or comma-separate for several
148+
(default: ../assets/tench.jpg)
148149
-s N, --seed RNG seed (default: 42)
150+
--batch N max images per forward pass; inputs run in chunks of N
151+
(default: 1, max: 64)
149152
150153
Output modes:
151-
-c, --classify classify the image and print top-k labels (default: off)
154+
-c, --classify classify each input image and print top-k labels (default: off)
152155
-k N, --topk top k classes to print (default: 5)
153-
--print-embeddings emit one JSON object on stdout with cls/pooled embeddings
156+
--print-embeddings emit embeddings JSON on stdout, one object per input image (JSONL)
154157
--print-patch-tokens include per-patch token vectors in the JSON output
155158
--l2-normalize L2-normalize emitted embedding vectors
156-
-o FNAME, --out write PCA visualization of patch features to FNAME (default: off)
159+
-o FNAME, --out write PCA visualization of patch features to FNAME; with multiple
160+
inputs FNAME is a directory for <input-stem>.pca.png files
157161
158162
Benchmark:
159163
--bench enable bench loop (default repeats=5, warmup=1); skips PCA image output
@@ -170,6 +174,8 @@ Workflows:
170174
dinov2-cli -m model.gguf -i img.jpg --print-embeddings # embeddings JSON on stdout
171175
dinov2-cli -m model.gguf -i img.jpg --print-embeddings --print-patch-tokens
172176
# + per-patch tokens
177+
dinov2-cli -m model.gguf -i a.jpg -i b.jpg --batch 2 --print-embeddings
178+
# batch: one JSON line per image
173179
dinov2-cli -m model.gguf -i img.jpg -o pca.png # PCA viz of patch features
174180
dinov2-cli -m model.gguf -i img.jpg --bench --bench-json # benchmark, JSON lines
175181
@@ -185,14 +191,14 @@ flowchart TD
185191
load_img --> load_model["dino_model_load<br/>(gguf -> ggml tensors)"]
186192
load_model --> alloc["ggml_gallocr_new"]
187193
alloc --> branch{--bench?}
188-
branch -- no --> single["single-shot predict<br/>+ JSON / top-k / PCA outputs"]
189-
branch -- yes --> loop["warmup x N + bench x M<br/>emit JSON or stderr row"]
194+
branch -- no --> single["chunked single-shot predict (n_batch per pass)<br/>+ JSONL / top-k / PCA outputs"]
195+
branch -- yes --> loop["warmup x N + bench x M<br/>each run covers all inputs<br/>emit JSON or stderr row"]
190196
single --> cleanup["free ctx + buffer + backend"]
191197
loop --> cleanup
192198
cleanup --> exit([return 0])
193199
```
194200

195-
Every flag maps to one `dino_params` field (`dinov2.h:61-79`).
201+
Every flag maps to one `dino_params` field (`dinov2.h:67-88`).
196202
`--print-embeddings` and `--bench-json` are the stable machine-readable
197203
outputs; `scripts/bench.sh` parses `--bench-json` lines one by one.
198204

@@ -230,14 +236,15 @@ struct dino_model {
230236

231237
struct dino_params {
232238
uint32_t seed = 42, topk = 5;
239+
uint32_t n_batch = 1; // max images per forward pass (--batch)
233240
bool enable_flash_attn = false;
234241
uint32_t n_threads = std::min(4u, std::thread::hardware_concurrency());
235242
bool classify = false;
236243
bool print_embeddings = false;
237244
bool print_patch_tokens = false;
238245
bool l2_normalize = false;
239246
std::string model = "../model.gguf";
240-
std::string fname_inp = "../assets/tench.jpg";
247+
std::vector<std::string> fnames_inp = {"../assets/tench.jpg"};
241248
std::string image_out = ""; // PCA visualization is opt-in via -o
242249
float eps = 1e-6f;
243250
uint32_t bench_repeats = 0; // --bench default: 5
@@ -270,6 +277,11 @@ std::vector<float> interpolate_pos_embed(ImgSize img_size,
270277
struct ggml_cgraph *build_graph(ImgSize img_size, struct ggml_context *ctx_cgraph,
271278
const dino_model &model, const dino_params &params);
272279

280+
// batch form: 1..n_batch same-dims images -> one output per image
281+
std::vector<dino_output> dino_predict(const dino_model &model,
282+
const std::vector<ImageF> &imgs,
283+
const dino_params &params, ggml_gallocr_t allocr);
284+
// single-image convenience wrapper around the batch form
273285
std::unique_ptr<dino_output> dino_predict(const dino_model &model, const ImageF &img,
274286
const dino_params &params, ggml_gallocr_t allocr);
275287

‎README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,14 @@ huggingface-cli download dinov2-cpp-core/dinov2-small-gguf --local-dir models
5858

5959
Emits `cls`/`pooled` vectors on stdout as one JSON object.
6060

61+
**Batch inference:** repeat `-i` (or comma-separate paths) and set `--batch`;
62+
each input gets one JSON line on stdout, identical to running it alone
63+
(see [docs/cli.md](docs/cli.md#batch-inference)):
64+
65+
```bash
66+
./bin/dinov2-cli -m models/model.gguf -i a.jpg -i b.jpg --batch 2 --print-embeddings
67+
```
68+
6169
That's it. Up to **3x faster than PyTorch on CPU** with up to **4x less memory** (see [docs/benchmarks.md](docs/benchmarks.md)).
6270

6371
## Features

0 commit comments

Comments
 (0)