reference.md

Reference

Configuration, the HTTP API, the capability matrix, and operational troubleshooting.

Environment variables

The backend loads .env on the host. Compose services load .env.docker. Compose itself reads .env for ${...} interpolation. Do not copy .env.docker into .env — its hosts are Compose service names that do not resolve outside the stack.

Read by the backend

Variable	Required/default	Purpose
POSTGRES_USER	required	Database user
POSTGRES_PASSWORD	required	Database password
POSTGRES_DB	required	Database name
POSTGRES_HOST	localhost	Database host
POSTGRES_PORT	5432	Database port
RABBITMQ_USER	required	Broker user
RABBITMQ_PASSWORD	required	Broker password
RABBITMQ_HOST	localhost	Broker host
RABBITMQ_PORT	5672	AMQP port
RABBITMQ_VHOST	/	Broker virtual host
QDRANT_URL	http://localhost:6333	Vector API
UI_PORT	5173	Read by the server only to build the dev CORS allowlist
SOURCE_ROOT	sources	Root exposed by the source browser and seeder
DATASETS_DIR	/tmp/datasets	Dataset-owned raw + canonical parsed artifacts
PIPELINE_RUNS_DIR	/tmp/pipeline_runs	Per-run logs and diagnostics
LOG_DIR	logs	Fallback logging root when no run is active
INSECURE_TLS_DOMAINS	empty	Comma-separated TLS-verification exceptions
PARSER_METADATA_MAX_BYTES	262144	Reject parser metadata over this size
PARSER_INTERMEDIATE_MAX_BYTES	8388608	Reject parser intermediate data over this size
CHUNK_METADATA_MAX_BYTES	262144	Reject chunk metadata over this size

The five required variables are the minimum for a host run; every host defaults to localhost.

Read by Compose and tooling

Variable	Default	Purpose
SERVER_PORT	8000	Uvicorn --port and the published port for the server service
RABBITMQ_MANAGEMENT_PORT	15672	Published RabbitMQ management port
VITE_SERVER_URL	http://localhost:8000	API base URL used by the UI
SERVER_URL	http://server:8000	API base URL used by the mcp service
MCP_PORT	8080	Port the MCP server listens on

SERVER_PORT and UI_PORT are interpolated into docker-compose.yml from .env (never .env.docker), which also re-injects UI_PORT into the server and ui containers — so .env alone is enough to move either port. Vite reads UI_PORT in ui/vite.config.ts, and the server derives its dev CORS allowlist from UI_PORT through UI_PORT + 2 because Vite increments when its port is taken.

Provider form defaults

There are no embedding-provider environment variables. A provider, its model, and its endpoint are chosen when the provider is created, and each pipeline snapshots them into actor_configs.embed_chunks.provider. Only the form defaults for new providers are env-sourced:

Variable	Fallback
OLLAMA_URL	http://localhost
OLLAMA_PORT	11434
OPENAI_BASE_URL	https://api.openai.com/v1
RERANKER_URL	http://localhost
RERANKER_PORT	8012
RERANKER_PATH	v1/rerank

Ports and URLs

Port	Service
5173	UI (Vite), overridable via UI_PORT
8000	FastAPI server, overridable via SERVER_PORT
6333 / 6334	Qdrant REST / gRPC
5432	PostgreSQL, overridable via POSTGRES_PORT
5672	RabbitMQ AMQP, overridable via RABBITMQ_PORT
15672	RabbitMQ management UI, overridable via RABBITMQ_MANAGEMENT_PORT
8080	mcp service — MCP endpoint at /mcp

URL	What
http://localhost:5173	Web UI
http://localhost:8000	REST API
http://localhost:8000/docs	Interactive OpenAPI docs
http://localhost:6333/dashboard	Qdrant dashboard
http://localhost:15672	RabbitMQ management UI

Volumes and directories

Compose defines four named volumes — postgres_data, qdrant_data, rabbitmq_data, and an unused ollama_data. Artifacts use bind mounts rather than named volumes so they are visible on the host.

Mount	Host path	Contents
SOURCE_ROOT	sources/ → /app/sources	Local XML inputs
DATASETS_DIR	tmp/datasets/ → /tmp/datasets	Dataset-owned raw + canonical parsed documents
PIPELINE_RUNS_DIR	tmp/pipeline_run/ → /tmp/pipeline_runs	Per-run logs and diagnostics

DATASETS_DIR must be shared by every service: preparation writes it once and every pipeline indexing that dataset reads the same files. See the layout in Filesystem.

HTTP API

The backend is a FastAPI application (title Embeddings Matcher API). Its interactive, always-current OpenAPI documentation is the authoritative schema:

• Swagger UI: http://localhost:8000/docs
• OpenAPI JSON: http://localhost:8000/openapi.json

Request and response models use camelCase aliases; nested run snapshots, prep configs, and plugin FieldSpec.key values stay snake_case. There is no authentication.

Health

Method	Path	Purpose
GET	/health	Liveness check

Datasets and the prepared corpus

Described in full, with curl examples, in Datasets.

Method	Path	Purpose
GET	/datasets	List datasets, newest first
POST	/datasets	Create a dataset
GET	/datasets/{id}	Fetch a dataset
PUT	/datasets/{id}	Replace a dataset (source changes invalidate the corpus)
DELETE	/datasets/{id}	Delete a dataset and its artifact tree
GET	/datasets/{id}/prep-config	Read the fully resolved preparation config
PUT	/datasets/{id}/prep-config	Replace it; invalidates the corpus. 409 while preparing
POST	/datasets/{id}/prepare	Purge the old corpus and start preparation. 409 if already running, or if relevance judgments would be lost without force=true
GET	/datasets/{id}/targets	Paginated per-source preparation progress (limit, offset)
GET	/datasets/{id}/source-artifacts	Paginated per-source outcomes and artifact paths, plus outcomeCounts. Optional outcome= filter
GET	/datasets/{id}/source-artifacts/{targetId}/content	Raw or parsed source text (kind=raw|parsed, optional start/end)
GET	/datasets/{id}/corpus	Current corpus provenance (generation, fingerprint)
GET	/datasets/{id}/documents	Paginated canonical documents (limit ≤ 100, offset)
GET	/datasets/{id}/documents/{documentId}	One canonical document's metadata
GET	/datasets/{id}/documents/{documentId}/content	Canonical text, optionally sliced by start/end offsets

Providers

Method	Path	Purpose
GET	/providers	List providers
POST	/providers	Create a provider
GET	/providers/{id}	Fetch a provider
PUT	/providers/{id}	Replace a provider
DELETE	/providers/{id}	Delete a provider
GET	/providers/configs	Discovered provider adapters and form metadata
POST	/providers/discover-models	Scan the server described by {providerType, config} for the models it serves. 400 if the type does not support discovery, 502 if the server is unreachable

Pipelines

Method	Path	Purpose
GET	/pipeline-runs	List runs, newest first
POST	/pipeline-runs	Create a pending run (snapshot only)
GET	/pipeline-runs/{id}	Fetch a run with chunk progress
POST	/pipeline-runs/{id}/launch	Launch or relaunch. 409 if running or the dataset is not prepared
PATCH	/pipeline-runs/{id}	Manually set run status
GET	/pipeline-runs/{id}/documents	Paginated per-document indexing progress (limit, offset)
DELETE	/pipeline-runs/{id}	Delete a run and its pipeline-owned data

Plugin metadata and sources

Method	Path	Purpose
GET	/actor-configs	Every plugin-backed actor's strategy field metadata
GET	/chunkers	Chunker field metadata
GET	/source-files	List directories and .xml files under path

Search

Method	Path	Purpose
POST	/search	Run a staged retrieval/fusion/reranking/output pipeline for one or more queries
GET	/searches	List persisted executions (without hits)
GET	/searches/{id}	One execution with its ordered final results and stage snapshot
POST	/compare	Manual text-to-text embedding comparison
GET	/search-queries	List queries (savedOnly, default true)
POST	/search-queries	Create a saved query, reusing an identical one
GET	/search-queries/{id}	One query with case/execution reference counts
PUT	/search-queries/{id}	Update a query (never rewrites history)
DELETE	/search-queries/{id}	Delete; refused when referenced by an evaluation case

Projection

Method	Path	Purpose
GET	/projection/algorithms	Reduction algorithms this server can run, with tunables and availability
POST	/projection	Lay a query and its retrieved chunks out in 2D/3D alongside the surrounding corpus

/projection reduces server-side and returns coordinates, not vectors. PCA needs only numpy; t-SNE needs scikit-learn (shipped in the server extra) and UMAP needs umap-learn (not shipped). An algorithm whose dependency is missing is reported available: false by /projection/algorithms and answers 501 if requested anyway, so a slim container still serves the view with whatever it does have.

Evaluation

Method	Path	Purpose
GET	/evaluation-sets	List evaluation sets
POST	/evaluation-sets	Create a set bound to one prepared dataset
GET	/evaluation-sets/{id}	Fetch a set with counts and corpus state
PATCH	/evaluation-sets/{id}	Update name/description
DELETE	/evaluation-sets/{id}	Delete a set
POST	/evaluation-sets/{id}/duplicate	Copy a set
GET	/evaluation-sets/{id}/staleness	Which judgments broke and why
POST	/evaluation-sets/{id}/repin	Explicitly re-pin to the current corpus
GET	/evaluation-sets/{id}/pipelines	Pipelines eligible to be evaluated
POST	/evaluation-sets/{id}/candidates	Suggest candidate passages from existing pipelines
GET	/evaluation-sets/{id}/cases	List cases
POST	/evaluation-sets/{id}/cases	Create a case from a search query
PATCH	/evaluation-cases/{id}	Update a case (notes, noRelevantPassage, position)
DELETE	/evaluation-cases/{id}	Delete a case
POST	/evaluation-cases/{id}/judgments	Add a graded span; the quote is re-verified server-side
PATCH	/judgments/{id}	Update grade or note
DELETE	/judgments/{id}	Delete a judgment
GET	/evaluation-sets/{id}/runs	List evaluation runs
POST	/evaluation-sets/{id}/runs	Execute the set against selected pipelines
GET	/evaluation-runs/{id}	One run with aggregate and per-case results
DELETE	/evaluation-runs/{id}	Delete a run

Common errors

Status	When
400	Invalid provider, run, search, or reranker configuration
404	Unknown record
409	Launching a running pipeline; launching against an unprepared dataset; preparing an already-preparing dataset; changing prep config mid-preparation

Malformed generic actor blocks during pipeline creation are logged and replaced with defaults rather than rejected.

Supported capabilities

Statuses: Supported — implemented and usable. Experimental — present but partial. Not supported — out of scope today.

Sources and formats

Capability	Status	Notes
Web source (single seed URL)	Supported	Same-origin child links optional
Child-link following	Supported	follow_child_links; restricted to the parent origin
max_depth	Supported	Root sources are depth 0; targets at the limit do not expand
follow_cross_domain	Not supported	Stored but not enforced; crawling remains same-origin
Local XML files/directories	Supported	*.xml only, under SOURCE_ROOT
Other local file types	Not supported	Local ingestion is XML-only

Parsers

Format	Status	Notes
HTML / XHTML	Supported	Web responses
Plain text	Supported	Web responses
XML	Supported	Web and local
Estonian legal XML (Juurakt)	Supported	Used by the legal_xml chunker
PDF	Not supported	Listed in the selector; no parser plugin exists

Chunkers

Chunker	Status
text_markdown (default), text_fixed, text_recursive, text_section, text_sentence, text_sliding_window	Supported
legal_xml	Supported

Providers

Provider type	Capability	Status	Notes
mock	embedding	Supported	Random vectors; flow validation only
mock	cross-encoder	Supported	Random permutation; exercises the rerank stage without a server
ollama	embedding	Supported	External/host service
openai	embedding	Supported	OpenAI-compatible; no explicit client HTTP timeout
rerank_server	cross-encoder	Supported	Jina/Cohere-style HTTP contract; reranks vector, keyword, or hybrid candidates
ollama	cross-encoder	Not supported	Ollama serves no rerank route; use rerank_server

Retrieval

Strategy or capability	Status	Notes
Vector / semantic (dense)	Supported	Qdrant; semantic and embedding remain legacy request names
Keyword (BM25)	Supported	PostgreSQL pg_textsearch
Hybrid	Supported	Independent vector and keyword candidate limits
RRF fusion	Supported	Configurable rank constant; default 60
DBSF fusion	Supported	Per-retriever z-score normalization and sum
Weighted score fusion	Supported	Vector/keyword weights; min-max or z-score normalization
Union Only	Supported	Deduplicated pool intended for downstream reranking
Cross-encoder reranking	Supported	Optional after vector, keyword, or hybrid retrieval
Stage-specific limits	Supported	Vector candidates, keyword candidates, fusion pool, and final results
Stage snapshots	Supported	Per-retriever, fusion, reranking, final rankings, counts, and latency

See Retrieval for the nested request shape, formulas, legacy-field mapping, score orientation, and configuration constraints.

Evaluation

Capability	Status	Notes
Reusable search queries	Supported	Shared by search, comparison, and evaluation
Corpus-anchored relevance judgments	Supported	Graded character spans in canonical documents
Hit Rate, Recall, MRR, nDCG, Passage Coverage	Supported	At K = 1, 3, 5, 10
Precision@K	Supported	Computed per query; not a headline metric
Staleness detection and blocking	Supported	Corpus fingerprint plus per-judgment quote hash
Answer-generation evaluation	Not supported	Deliberately out of scope

Storage, infrastructure, and interfaces

Capability	Status	Notes
PostgreSQL 17 (system of record)	Supported	Custom image builds pg_textsearch
Qdrant (vector store)	Supported	One point per chunk
RabbitMQ + Dramatiq	Supported	One process and one thread per actor
Elasticsearch / OpenSearch	Not supported	—
Web UI	Supported	—
REST API	Supported	Unauthenticated
Plugin extension	Supported	Chunkers selectable in the UI; other actors need a frontend change
MCP server	Supported	Full lifecycle as agent tools; see MCP server
LangGraph agent	Experimental	MCP-dependent prototype
Docker Compose (local/trusted network)	Supported	The only supported runtime
Authentication / authorization	Not supported	—
Public-internet / multi-user deployment	Not supported	Not hardened

Troubleshooting

For ingestion-specific issues — stalled preparations or pipelines, skipped sources — see Troubleshooting ingestion.

The stack does not start

docker compose ps
docker compose logs postgres
docker compose logs rabbitmq

Workers depend on a healthy RabbitMQ and a started Postgres. The most common local causes are missing .env credentials and occupied host ports.

Workers do not wait on migrate — it is a separate, on-demand job (scripts/migrate.sh). A worker that starts against an un-migrated database comes up and then fails on its first query with a missing-relation error; run the migrations and let it retry.

Ports already in use

The stack publishes 5173, 8000, 6333, 6334, 5432, 5672, and 15672. Free the conflicting port, or move it via UI_PORT, SERVER_PORT, POSTGRES_PORT, RABBITMQ_PORT, or RABBITMQ_MANAGEMENT_PORT in .env. The Qdrant ports are fixed in docker-compose.yml.

RabbitMQ connection failures

The backend needs the RabbitMQ variables at import time. Confirm the rabbitmq service is healthy and that .env (host runs) or .env.docker (containers) has matching credentials and vhost.

PostgreSQL connection problems

Check docker compose logs postgres. Most database dependencies use service_started rather than a health check, and the migrate entry point retries up to 60 times at one-second intervals. If connections are exhausted, stop extra host processes with their own pools — the server and each worker maintain their own.

Qdrant connectivity or collection mismatch

Confirm Qdrant is reachable at QDRANT_URL and open http://localhost:6333/dashboard. Collections are named per dataset and model (<QDRANT_COLLECTION_BASE>_<dataset-name>_<provider>_<model>_d<dim>) and are created only when absent — an existing collection is not recreated with a new vector size or distance. Because the model and its measured dimension are part of the name, changing models — or re-pulling a different model under the same Ollama tag — creates a new collection rather than clashing with the old one. Run creation returns 503 if the provider cannot be reached to report its dimension. A run pointed at a collection of another dimension fails with VectorDimensionMismatch, naming both sizes; delete the collection and re-index, or create the run with the original model.

Collections created before model-aware naming keep their old BASE_<dataset-name>_qwen_embed_8b name. They are still found and cleaned up by a dataset reset (which scans Qdrant for everything under the dataset's prefix), but no new run will write to them.

Ollama not reachable from containers

The embedding worker runs inside Docker, where localhost is the worker container itself. For Ollama on a Docker Desktop host, configure the provider with http://host.docker.internal and port 11434.

Embedding model errors

Embedding runs with exactly the provider snapshot stored on the pipeline and fails loudly if there is none. Confirm the model is served and reachable from the worker container, and that its vector dimension matches the target collection. The OpenAI client sets no explicit HTTP timeout; the Ollama client relies on its library's default.

Keyword search returns nothing

If the migration that builds the BM25 extension and index failed, the keyword path cannot run. Re-run scripts/migrate.sh and read its output, plus docker compose logs postgres. The custom Postgres image is required because it builds and preloads pg_textsearch.

Reset local state

docker compose down -v      # remove named Compose volumes
scripts/full-clean.sh       # purge queues, drop Qdrant collections, reset schema, rebuild workers

full-clean.sh is destructive to local data.

MCP server

launch_pipeline returns 409 Dataset '<name>' is not prepared when the dataset has not been prepared, or was reset by a update_prep_config call. Run prepare_dataset and poll get_dataset_status until indexable is true.

A search with reranker_provider_id set can succeed without actually reranking: a rerank endpoint that is unreachable or returns an error degrades to the un-reranked ordering by design. The stage snapshot records it (reranking.degraded: true plus reranking.error), and the Search Lab renders the stage as degraded; the server logs carry the traceback under Reranking failed (provider=http_rerank ...).

The most common cause is a reranker pointed at a server with no rerank route, which answers No rerank route at <url>. Ollama is never a valid rerank endpoint — it serves no such route on any path. Use the rerank_server provider type and see Running a reranker locally.