RAG Eval Harness

A local-first RAG pipeline with an evaluation harness that scores retrieval quality, answer correctness, and hallucination across three swappable retrievers — BM25, dense (pgvector), and hybrid (RRF fusion) — over the HotpotQA benchmark.

The point of the project is not the RAG pipeline; it's the harness that tells you which retrieval strategy wins on which metric, and where the system breaks.

Architecture

flowchart LR
    HQA[HotpotQA distractor] --> L[loader: dedupe paragraphs<br/>into global corpus]
    L --> C[(Chunks<br/>id = sha1 title)]
    C --> BM25[BM25 index]
    C --> PG[(pgvector<br/>384-dim cosine)]
    Q[Question] --> R{Retriever<br/>bm25 / dense / hybrid}
    BM25 --> R
    PG --> R
    R --> G[Ollama generation<br/>qwen3.5:4b]
    G --> T[Tracer: ids + answer + latency]
    T --> S[Scorers]
    S --> A[Aggregator] --> J[(runs/latest.json)]
    J --> API[FastAPI /metrics + dashboard]

    subgraph Scorers
      direction TB
      RM[retrieval: hit_rate, ctx precision/recall<br/>from gold supporting facts]
      AM[answer: HotpotQA EM / F1]
      FM[faithfulness: LLM judge]
    end

Loading

Retrieval metrics are deterministic — HotpotQA ships gold supporting-fact titles, so hit rate / context precision / recall need no LLM judge. Only faithfulness uses one.

Stack

Python 3.13 · sentence-transformers (all-MiniLM-L6-v2) · pgvector · Ollama (qwen3.5:4b) · FastAPI. Fully local — no paid APIs, no keys.

Quickstart

python3 -m venv .venv && source .venv/bin/activate
pip install -e .
docker compose up -d                 # Postgres + pgvector
ollama pull qwen3.5:4b               # generation + judge model

python -m eval --dataset hotpotqa --retriever hybrid --n 20 --top-k 5
uvicorn api.main:app --port 8000     # dashboard at http://localhost:8000

--no-judge skips the (slow) faithfulness LLM calls for fast iteration.

Query your own PDFs

The retrievers are source-agnostic — they index any list[Chunk] — so the same BM25 / dense / hybrid stack works over PDF text. ingest/pdf.py extracts text per page (via pypdf), segments each page at heading cues into sections, and chunks within them (falling back to a paragraph index when no heading is detected). Every chunk carries a citation — source · page · section — so answers are traceable:

python -m query --pdf docs/sample.pdf \
  --question "How many paid time off days do new employees get?" --top-k 2
# Retrieved 2 chunks from 6 total:
#   [sample.pdf · p1 · "Paid Time Off"] New employees accrue eighteen days ...
# A: 18
# Sources:
#   - sample.pdf · p1 · "Paid Time Off"
#   - sample.pdf · p1 · "Remote Work Policy"

Section detection is heuristic — standalone heading lines and inline Title Case. leads — so it degrades to p<page> · ¶<n> when a document has no clear headings. The citation is stored in the chunk title, so it survives even the dense path (which only persists titles).

--pdf accepts a single file or a directory of PDFs. Default --retriever is bm25 (in-memory, zero infra). dense/hybrid also work but share the pgvector chunks table with the eval corpus, so run them on a clean table for PDF-only results. Scanned / image-only PDFs produce no extractable text and are skipped.

Results

20 HotpotQA questions, top_k=5, qwen3.5:4b. Higher is better.

Metric	BM25	Dense	Hybrid
Hit rate@5	0.85	1.00	0.90
Context precision	0.25	0.33	0.29
Context recall	0.63	0.83	0.73
Exact match	0.25	0.35	0.30
Answer F1	0.39	0.49	0.46
Faithfulness	0.70	0.85	0.70

Latency (p50/p95)	BM25	Dense	Hybrid
Retrieval p50	~2 ms	~380 ms	~330 ms
Generation p50	~3.8 s	~3.6 s	~3.5 s
Total p95	~7.0 s	~6.1 s	~5.4 s

On this sample dense retrieval wins across the board — HotpotQA questions are paraphrase- heavy, which favors embeddings over lexical overlap. Hybrid trails dense here because RRF gives equal weight to BM25's weaker ranking; on a larger or more keyword-heavy corpus the ordering can flip, which is exactly what the harness is for.

What breaks

• Generation, not retrieval, dominates latency. Retrieval is milliseconds; the LLM is seconds. Total-latency SLOs live and die on the generation step.
• Dense "retrieval" latency is mostly query embedding. The pgvector lookup is fast; encoding the query with sentence-transformers is the ~380 ms. Caching/quantizing the encoder matters more than tuning the ANN index at this scale.
• EM is brutal and honest. The model answers "German" when gold is "Prussian" — right topic, wrong token. EM=0, F1 partial. This is why faithfulness (grounding) and correctness (EM/F1) are tracked separately: a fluent, grounded answer can still be wrong.
• Faithfulness depends on a 4B local judge. It returns coarse scores and occasionally prose instead of a number; the scorer regex-extracts and clamps, defaulting to 0. Treat it as a directional signal, not ground truth.
• Title-level retrieval metrics. Gold is HotpotQA's paragraph titles, so retrieval is scored at paragraph granularity, not sentence — generous to the retriever by design.

Layout

ingest/    Chunk model + HotpotQA loader + PDF loader/chunker
retrieval/ base · bm25 · dense (pgvector) · hybrid (RRF k=60) · registry
generate/  Ollama client + prompt pipeline
eval/      dataset · tracer · scorers/ · aggregator · __main__ (CLI)
query/     ask questions over a PDF (__main__ CLI)
api/       FastAPI /metrics + static dashboard