PyVectorHound

Diagnose why your RAG retrieval is returning the wrong documents.

Not to be confused with PyAgentHound: PyVectorHound diagnoses RAG retrieval failures; PyAgentHound traces and root-causes AI agent execution failures.

Which project do I need?

Part of the Mullassery open-source family. These tools have similar names — here is the short version of who does what:

If you need to…	Use
Build a repository's dependency graph and match it against vulnerability advisories (Python, Rust, Node, containers)	PyDeepMesh
Understand why a Python dependency exists, who added it, and whether it is actually used	PyDependencyCheck
Statically analyze a dbt project (risky tests, test blast radius, cost)	PyDBTGuard
Statically analyze Apache Beam / Dataflow pipelines (cost, hot keys, reliability)	PyBeamGuard
Validate Airflow DAGs and dbt runs	PyAirflowTester
Simulate cascading-failure blast radius across infrastructure, data pipelines, and AI systems	PyBlastRadius
Check the quality of the data itself at runtime (schema, drift, anomalies)	StatGuardian
Discover devices, services, and vulnerabilities on a network	PyNetworkIntel
Discover and score API security risk (real traffic, agent/MCP, Cedar policy)	PyAPICheck
Trace and root-cause AI agent failures	PyAgentHound
Diagnose RAG retrieval failures	PyVectorHound
Orchestrate local, dependency-aware tasks (DAG scheduler + content-addressed cache)	PyTaskWireCheck

Browse all projects →

What it is

PyVectorHound is a component-level diagnostic engine for retrieval-augmented generation (RAG) pipelines. Point it at a set of search results (from your own pipeline, or from a live Qdrant/Chroma/Milvus/pgvector/Weaviate instance) and it isolates which stage is failing — embedding quality or vector search ranking — and gives you plain-English, ranked recommendations. It does not run an embedding model, a reranker, or BM25 for you, and it is not a vector database: it's a diagnostic layer that sits on top of retrieval results you already have and tells you, with real computed metrics, what's wrong.

Features

• Embedding-space diagnostics (isotropy, coverage, distinctiveness) computed by a Rust extension (pyvectorhound._core, built via PyO3) from the real per-document embeddings your database adapter's get_embeddings() returns.
• Vector search accuracy (precision, recall, MRR) computed against expected_docs you supply as ground truth.
• LLM-as-judge faithfulness / contradiction checking — pass document_texts and an llm_judge_fn (no LLM client bundled) and Diagnosis flags retrieved documents that score well on embedding similarity but are actually irrelevant to or contradict the query.
• Root cause + ranked recommendations in plain English, combining the above.
• Concurrent batch evaluation via Hound.diagnose_batch(), with automatic retry/backoff on embed_fn/llm_judge_fn calls.
• Model-agnostic quality thresholds — Hound.track_metric() switches GOOD/MODERATE/WEAK status from a fixed cutoff to your own historical baseline.
• Ranking & diversification via rank_and_diversify(results, top_k), combining BM25, semantic, and recency signals, then penalizing near-duplicates using real cosine similarity. compute_reranker_metrics(...) reports MRR/NDCG/Precision@K/Recall@K/diversity against your ground truth.
• int8 scalar quantization (src/quantization.rs) for shrinking large local vector indices (4x memory reduction, bounded reconstruction error).

Who it's for

• Teams whose RAG pipeline is returning irrelevant documents and who don't know whether it's the embeddings or the ranking — Diagnosis/Hound.diagnose() isolates which stage is at fault instead of forcing you to eyeball cosine-similarity scores.
• Anyone catching retrieved documents that look similar but contradict the query — the LLM-as-judge faithfulness check, when pure distance metrics would score them as a good match.
• Engineers comparing embedding models or database configs objectively — track metrics over time with Hound.track_metric() so quality thresholds are relative to your own baseline, not a hardcoded cutoff.

What it is not

• Not an embedding model, reranker, or vector database. It brings none of those, and won't silently generate a random vector and pretend the resulting diagnosis means something: pass an embed_fn or a precomputed query_embedding.
• Not a BM25/keyword-search or reranker diagnostic. Both components report UNKNOWN — not implemented, not guessed — because Diagnosis doesn't yet accept external BM25/reranker scores.
• Not a full retrieve-then-generate evaluator. Unlike Ragas, PyVectorHound never requires an LLM to generate a final answer, so it has no visibility into a generation-layer failure.
• Not fully integrated into storage/retrieval. The int8 quantization functions are low-level quantize/dequantize primitives, not wired into Hound's storage/retrieval path.
• Some methods remain unimplemented: Hound.compare_metrics() and Hound.detect_drift() raise NotImplementedError; ModelComparison/Hound.compare_models() reports real published cost/latency metadata but cannot measure quality unless you pass quality_fn.

Project status

Production/Stable per the package classifier (version 1.6.0), with the honest gaps above. If a component doesn't have enough input to measure honestly (no adapter, fewer than 2 documents with embeddings, or no expected_docs), it's reported as UNKNOWN with an explanation of what to supply — never a made-up number. See ROADMAP_HONEST.md for a plain list of what's not built, what's untested, and known technical debt. Changes are tracked in CHANGELOG.md.

Installation

pip install pyvectorhound

Optional vector database clients (only install the one(s) you use):

pip install pyvectorhound[qdrant]     # Qdrant
pip install pyvectorhound[chroma]     # Chroma
pip install pyvectorhound[milvus]     # Milvus
pip install pyvectorhound[weaviate]   # Weaviate
pip install pyvectorhound[pgvector]   # PostgreSQL + pgvector

Requires Python 3.8+. PyPI currently carries a macOS ARM64 / CPython 3.9 wheel only (no sdist and no abi3 build for 1.6.0), so on other platforms or interpreter versions, install straight from the repo instead (a Rust toolchain is required):

pip install git+https://github.com/Mullassery/PyVectorHound.git

Quick start

Diagnose results you already have

This is the fastest way to try it — no live database or embedding model needed. Diagnosis fetches per-document embeddings for you via a small adapter object (anything with a get_embeddings(doc_ids) -> dict method); without one, the embedding component honestly reports "UNKNOWN" instead of a fabricated score.

from pyvectorhound import Diagnosis

class InMemoryAdapter:
    """Anything with get_embeddings(doc_ids) works -- swap in your own
    QdrantAdapter/ChromaAdapter/etc., or a wrapper around your pipeline."""
    def __init__(self, embeddings_by_id):
        self._embeddings_by_id = embeddings_by_id

    def get_embeddings(self, doc_ids):
        return {d: self._embeddings_by_id[d] for d in doc_ids if d in self._embeddings_by_id}

results = [
    {"id": "pricing.pdf", "score": 0.91},
    {"id": "onboarding.md", "score": 0.84},
    {"id": "faq.md", "score": 0.79},
]

diagnosis = Diagnosis(
    query="What's your return policy?",
    results=results,
    expected_docs=["returns.pdf", "policy.md"],  # ground truth
    adapter=InMemoryAdapter(my_document_embeddings),
)
diagnosis.analyze()

print(diagnosis.root_cause())
for rec in diagnosis.recommendations():
    print(f"[{rec['priority']}] {rec['action']}")

print(diagnosis.hunt())  # full plain-English report

A runnable version (with synthetic embeddings so it works with no setup) is in examples/retrieval_debug.py.

Diagnose against a live vector database

from pyvectorhound import Hound

hound = Hound(
    db="qdrant",                      # qdrant | chroma | milvus | weaviate | postgres
    endpoint="localhost:6333",
    index_name="documents",
    # PyVectorHound doesn't ship an embedding model -- wrap whatever you use:
    embed_fn=lambda text: my_embedding_client.embed(text),
)

diagnosis = hound.diagnose(
    query="What's your return policy?",
    expected_docs=["returns.pdf", "policy.md"],
    top_k=5,
)
print(diagnosis.hunt())

Hound connects lazily — constructing it doesn't require a live server, only calling diagnose() (or another querying method) does.

Usage

Diagnostics it runs

Component	What it measures	Requires
Embedding	Isotropy, coverage, distinctiveness of the retrieved documents' real embeddings	An adapter with get_embeddings(), and ≥2 retrieved documents
Vector search	Precision, recall, MRR	expected_docs (ground truth)
Faithfulness	LLM-judge contradiction/relevance check	document_texts + llm_judge_fn
BM25 (keyword)	Not implemented — reports UNKNOWN	n/a
Reranker	Not implemented — reports UNKNOWN	n/a

Every measured component is computed for real from the input you give it; nothing is guessed when the input isn't there.

Faithfulness checking and batch evaluation

from pyvectorhound import Hound

def judge(query: str, doc_texts: list[str]) -> dict:
    # Wrap whatever LLM client you already use -- PyVectorHound doesn't bundle one.
    response = my_llm_client.judge_faithfulness(query, doc_texts)
    return {
        "contradiction_score": response.score,
        "contradicted_count": response.contradicted,
        "reasoning": response.explanation,
    }

hound = Hound(db="qdrant", embed_fn=my_embed_fn, llm_judge_fn=judge)

diagnosis = hound.diagnose(
    query="What's your return policy?",
    document_texts={"returns.pdf": "...", "policy.md": "..."},  # doc_id -> text
)
print(diagnosis.metrics()["faithfulness"])

# Evaluate many queries concurrently instead of one at a time:
diagnoses = hound.diagnose_batch(
    queries=["query 1", "query 2", "query 3"],
    document_texts=[{"a": "..."}, None, {"b": "..."}],  # per-query, optional
    max_workers=8,
)

Track a few diagnoses over time and quality-status classification switches from a fixed cutoff to your own historical baseline automatically:

hound.track_metric("vector_search_precision", diagnosis.metrics()["vector_search"]["precision"])
# After ~5+ tracked points, later diagnose() calls classify status
# (GOOD/MODERATE/WEAK) relative to that baseline instead of a fixed number.

Ranking & diversification

from pyvectorhound import rank_and_diversify, compute_reranker_metrics

results = [
    ("doc_1", {"semantic": 0.9}, [1.0, 0.0, 0.3]),     # (doc_id, scores, embedding)
    ("doc_2", {"semantic": 0.85}, [0.98, 0.1, 0.25]),  # near-duplicate of doc_1
    ("doc_3", {"semantic": 0.6}, [0.0, 1.0, 0.0]),
]
ranked = rank_and_diversify(results, top_k=3)
metrics = compute_reranker_metrics(
    [r["document_id"] for r in ranked],
    [r["diversity_score"] for r in ranked],
    relevant_docs=["doc_1", "doc_3"],
    k=3,
)

int8 scalar quantization

src/quantization.rs maps each f32 vector linearly to the i8 range (-128..=127) using that vector's own min/max, alongside a (scale, offset) pair needed to dequantize it. This is scalar quantization, not product quantization, and it isn't SIMD/GPU accelerated. Storing i8 instead of f32 is a 4x memory reduction, at the cost of bounded reconstruction error (max absolute error per element is at most (max - min) / 255). Constant vectors (including all-zero) round-trip exactly.

from pyvectorhound import _core

result = _core.py_quantize_vector([0.12, -0.87, 0.5, 0.0])
# {"values": [i8, ...], "scale": float, "offset": float, "min": float, "max": float}
restored = _core.py_dequantize_vector(result["values"], result["scale"], result["offset"])

Other tools

• hound.quality_scorer() — QualityScorer for scoring an embedding's validity and, given corpus neighbors via the adapter, real isotropy/coverage/distinctiveness.
• hound.benchmark() — latency percentiles and database/embedding-model comparisons.
• hound.analyze_trends() — track metrics over time and detect drift, regressions, and anomalies from real tracked values.
• hound.tracer() / hound.replayer() — capture a retrieval pipeline run and replay it under different configurations to compare recall/latency.

vs Ragas

Ragas is the closest OSS competitor — the standard for RAG evaluation metrics. We ran both against the same real corpus (168 real chunks from 3 live English Wikipedia articles), real embeddings (nomic-embed-text via a local Ollama instance), and a real local LLM judge/generator (qwen2.5:7b-instruct via Ollama — no cloud API keys used or needed), across 4 real queries including 2 deliberately hard cases.

Query	PyVectorHound vector_search	PyVectorHound faithfulness	Ragas context_precision	Ragas context_recall	Ragas faithfulness
"easy" (attention mechanism)	WEAK — 0% precision/recall	GOOD	0.9999	1.0	1.0
"cross_chunk" (how RAG reduces hallucination)	GOOD — 100% precision/recall	GOOD	0.9999	1.0	0.0
"no_answer" (a date not in corpus)	UNKNOWN (correctly declines to guess)	GOOD	0.0	1.0	1.0
"distractor" (self- vs cross-attention)	WEAK — 50% precision	GOOD	0.9999	1.0	1.0

Two real methodological lessons came out of this, not bugs: (1) document-identity ground truth is brittle — in the "easy"/"distractor" cases the retrieved chunks (from a different article) still contained the answer, which answer-based metrics like Ragas's aren't fooled by; (2) the "cross_chunk" row shows the real scope difference — PyVectorHound judges the retrieved passages, Ragas judges the generated answer against context, so Ragas caught a generation-layer failure PyVectorHound has no visibility into. That's the tradeoff: diagnose the retrieval layer standalone (no generation step needed), or evaluate the full retrieve-then-generate pipeline end to end.

Documentation

• docs/ARCHITECTURE.md — architecture.
• docs/GUIDE.md — usage guide.
• ROADMAP_HONEST.md — what's not built, what's untested, and known technical debt.
• CHANGELOG.md — release history.
• examples/ — runnable scripts.

Contributing

git clone https://github.com/Mullassery/PyVectorHound.git
cd PyVectorHound
pip install maturin
maturin develop --release   # builds the Rust extension in place
pip install -e ".[dev]"
pytest tests/ -v

CI (the badge above) is green and runs the full test suite. OpenTelemetry / LangChain / LlamaIndex / MCP integrations, the CLI, and the REST server exist and have passing tests but have seen far less real-world use than the core Hound/Diagnosis path.

Related projects

Other tools from the same family, for adjacent parts of the RAG/LLM stack:

• PyAgentHound — debug AI systems by tracing execution, detecting failures, identifying root causes, and explaining bad agent results.
• PyStreamPDF — reduce RAG costs by extracting only what matters from PDFs: selective retrieval, structure analysis, token-efficient extraction.
• PyStreamMCP — intelligence layer for AI agents: query planning, context discovery, and cost optimization.
• PyTokenCalc — count tokens and estimate LLM costs across 20+ providers before you send a request.
• OpenAnchor — token intelligence middleware for multi-provider LLM usage: 6D attribution, pattern detection, recommendations, OTEL export.
• StatGuardian — fast, declarative data quality framework (Rust): schema validation, drift detection, anomaly detection.

Browse all projects →

License

This project is licensed under the Apache License 2.0.