architecture.md

Architecture

Embeddorium separates orchestration state, source artifacts, dense vectors, and message transport, and runs them as Docker Compose services on one trusted host.

Responsibilities

• PostgreSQL — system of record for datasets, providers, pipeline runs, crawl targets, documents, chunks, links, outbox events, search queries, search history, and evaluation.
• Filesystem — dataset-owned raw and canonical parsed text; per-run logs.
• Qdrant — one vector point per chunk.
• RabbitMQ — carries Dramatiq actor messages.
• FastAPI — management, retrieval, comparison, evaluation, and plugin-metadata APIs. Retrieval runs synchronously in the request.
• React/Vite UI — calls those APIs.

Ingestion is a set of single-purpose actors. Each actor has a handler.py with the stage logic and a launcher.py with Dramatiq, broker, logging, and process wiring. Downstream stage messages are normally committed to a SQL outbox with the data they depend on and published later by a standalone dispatcher.

Runtime topology

Service	Responsibility	Host exposure
postgres	PostgreSQL 17 plus pg_textsearch	${POSTGRES_PORT:-5432}
qdrant	Dense-vector storage	6333, 6334
rabbitmq	Dramatiq broker and management UI	${RABBITMQ_PORT:-5672}, ${RABBITMQ_MANAGEMENT_PORT:-15672}
migrate	Reapply ordered SQL migrations, then exit (migrate profile — launched on demand, not by docker compose up)	none
server	FastAPI and synchronous retrieval	${SERVER_PORT:-8000}
ui	Vite development server	${UI_PORT:-5173}
worker-outbox-dispatcher	Publish committed outbox events	none
worker-validate-source	Normalize, validate, dedup a source	none
worker-fetch-source	Fetch web bytes or read a local file	none
worker-filter-documents	Keyword relevance gate	none
worker-parse-source	Produce the canonical document and discovered links	none
worker-schedule-links	Feed the crawl frontier, finalize targets	none
worker-track-dataset-preparation	Mark a dataset prepared	none
worker-chunk-document	Produce pipeline-owned chunks	none
worker-schedule-embeddings	Batch chunks into embed jobs	none
worker-embed-chunks	Embed and upsert into Qdrant	none
worker-track-pipeline-status	Complete a finished run	none
mcp	Lifecycle as MCP tools; HTTP client of server	8080

Concurrency and startup

Every Dramatiq service runs with --processes 1 --threads 1 and consumes one queue, so the shipped concurrency per stage — including embedding — is one message at a time. This is pinned deliberately so each launcher's SQL pool can be sized against a known bound. There is no application rate limiter; raising processes, threads, or replicas increases concurrent provider calls.

The outbox dispatcher is a single poll loop that reads up to 100 pending events and sleeps only when it publishes none.

Workers wait for RabbitMQ health and successful migration completion. Most database dependencies use service_started rather than a Postgres health check; the migration entry point retries the database up to 60 times at one-second intervals. The server and qdrant services have no Compose health check.

Frontend boundary

A React 19 single-page application (TypeScript, Vite 6) calling VITE_SERVER_URL (default http://localhost:8000).

Path	Header label	Purpose
/	Search	Four tabs over one workspace, selected by ?tab=: Search (manual embedding comparison or pipeline-scoped retrieval), Compare (?tab=compare, compare persisted search executions), History (?tab=history, browse saved searches and their stored results), and Queries (?tab=queries, manage reusable search queries). /search-comparison redirects to the Compare tab.
/datasets	Datasets	Create/view/delete datasets, edit prep config, prepare, inspect per-source outcomes and artifacts — see Datasets
/evaluation	Evaluation	Evaluation sets, annotation workspace, evaluation runs
/providers	LLM Providers	Create/view/delete provider configs
/ingestion-pipelines	Pipelines	Create, inspect, launch, relaunch, delete pipelines
/pipeline-runs	Indexing Runs	Poll and inspect per-document indexing progress

Actor and provider controls are partially metadata-driven from GET /actor-configs and GET /providers/configs. API modules under ui/src/api translate camelCase HTTP objects into UI types; nested snapshots stay snake_case because the backend returns raw JSON model dumps there. The Search form persists inputs and preferences (not results) in localStorage, and Indexing Runs polls running selections every 10 seconds.

The two workflows

Ingestion is two independent workflows with two different scopes. This is the central architectural decision; see the core idea.

	Dataset preparation	Pipeline indexing
Trigger	POST /datasets/{id}/prepare	POST /pipeline-runs/{id}/launch
Scope	dataset_id	pipeline_id
Unit of work	crawl_target (frontier entry)	pipeline_document (canonical document)
Config from	datasets.prep_config	pipeline_runs.actor_configs
Progress on	crawl_targets	pipeline_documents
Completed by	track_dataset_preparation	track_pipeline_status
May fetch/parse?	yes	never
May chunk/embed?	never	yes

Ownership boundaries

Concern	Owner	Where it lives
Source URLs / local paths, crawl frontier	Dataset	crawl_targets.dataset_id
Fetch, validation, filtering, parser selection, link config	Dataset	datasets.prep_config (DatasetPreparationConfig)
Canonical normalized documents	Dataset	documents.dataset_id; text under {DATASETS_DIR}/{dataset_uuid}/parsed/
Preparation lifecycle and corpus identity	Dataset	datasets.preparation_status, preparation_generation, corpus_fingerprint
Chunker + settings, embedding provider, batching, vector store	Pipeline	pipeline_runs.actor_configs (PipelineActorConfigs)
Chunks, embeddings, per-document index progress, the vector index	Pipeline	document_chunks.pipeline_id, pipeline_documents, Qdrant

Multiple pipelines read the same prepared documents; none of them can fetch or parse.

Preparation lifecycle

not_prepared -> preparing -> prepared | prepared_with_errors | failed

Only prepared and prepared_with_errors are indexable (INDEXABLE_PREPARATION_STATUSES). Two things invalidate a prepared corpus back to not_prepared:

• replacing the preparation config (PUT /datasets/{id}/prep-config), and
• changing the dataset's source identity — source_type, url, or paths.

Renaming a dataset does not invalidate it. Entering preparing bumps preparation_generation; completion records a recomputed corpus_fingerprint. Existing pipelines keep clear provenance of the prepared state they were built from; a changed dataset is never silently treated as safely indexed.

Invalidation only changes the status. Preparation itself is what purges: every run first deletes the dataset's documents, crawl targets, chunks, Qdrant points and on-disk artifacts, so re-preparing empties every index built from that dataset while leaving the pipeline runs themselves intact. See Invalidation and re-preparation.

Ingestion flow

Dataset preparation (scoped by dataset_id)

Web and local XML inputs use the same actor sequence. validate_source and fetch_source pick a web or local strategy from the dataset's source type, and both source types then pass through the keyword filter.

POST /datasets/{id}/prepare
  -> validate_source
  -> fetch_source
  -> filter_documents
  -> parse_source                 (canonical document + discovered links)
  -> schedule_discovered_links
       -> validate_source         (discovered web links, back to the frontier)
       -> track_dataset_preparation -> dataset marked prepared

Actor	Queue	Main result
validate_source	ingest.crawl.source.validate.v1	Normalize/validate/dedup per dataset, create a queued target
fetch_source	ingest.crawl.source.fetch.v1	Raw artifact and fetch provenance
filter_documents	ingest.crawl.file.filter.v1	Pass to filtered, or skip as not relevant
parse_source	ingest.crawl.source.parse.v1	Canonical document + discovered links
schedule_discovered_links	ingest.crawl.links.schedule.v1	Frontier events and target finalization
track_dataset_preparation	prepare.dataset.status.track.v1	Mark the dataset prepared

Preparation never chunks or embeds. It completes with no chunk, embedding, or vector collection in existence — which is exactly what makes the corpus reusable.

Discovered links are extracted by format parsers from the raw source, with canonical Markdown links retained as a fallback, then keyed to the source document by parse_source. Link discovery therefore never depends on a chunker, and changing the chunker cannot change which sources belong to the dataset.

Crawl target statuses:

queued -> fetching -> fetched -> filtering -> filtered -> parsing -> parsed
       -> scheduling -> processed

Crawl targets carry no chunk or embed progress; that is pipeline-owned state.

Pipeline indexing (scoped by pipeline_id)

POST /pipeline-runs/{id}/launch     (409 unless the dataset is prepared)
  -> for each prepared document: register a pipeline_documents row (idempotent)
  -> chunk_document                 (document_id, pipeline_id)
  -> schedule_embeddings
       -> embed_chunks -----------> track_pipeline_status -> run completed

Actor	Queue	Main result
chunk_document	ingest.crawl.document.chunk.v1	Pipeline-owned chunks
schedule_embeddings	ingest.crawl.embeddings.schedule.v1	Batched embed events
embed_chunks	ingest.embed.chunk.generate.v1	Qdrant points and embedded chunk states
track_pipeline_status	ingest.pipeline.status.track.v1	Complete a finished run

Pipeline document statuses:

pending -> chunking -> chunked -> embedding -> embedded    (or failed)

One row per (pipeline_id, document_id), so two pipelines index the same document concurrently without contending, and rebuilding one pipeline never touches another's rows or the dataset's documents. Documents that chunk down to nothing go straight to embedded.

Stage handlers use compact ID payloads and reload data and config from storage.

Transaction boundaries

Preparation and index launch enqueue their first messages directly, and after saving a target validate_source also enqueues fetch_source directly. These handoffs do not use the SQL outbox, so a process failure between persisting the target and publishing fetch can leave a queued target without a message. From fetch_source onward, a stage normally commits its data, status, and next event in one SQL transaction, and the dispatcher publishes the event later. Delivery is at-least-once.

Error handling

Every actor declares max_retries=3. Handlers raise for transient conditions (connection or timeout during fetch, or missing rows that may reflect ordering/visibility), and Dramatiq's default retry middleware supplies backoff. Permanent conditions update the target and return a terminal status:

Status	Cause
skipped_unsupported	Unsupported MIME type or no matching parser
skipped (reason not_relevant)	Keyword filter mismatch
failed_permanent	Permanent fetch error
failed_transient	Transient fetch error surfaced after retries

A dataset whose frontier drains with permanent failures becomes prepared_with_errors: the successfully parsed documents remain usable and the dataset stays indexable.

There is no custom dead-letter consumer; exhausted messages stay diagnosable through RabbitMQ, target state, and worker logs.

Idempotency

• Actors claim expected statuses with compare-and-set updates — crawl targets during preparation, pipeline_documents rows during indexing.
• Outbox dedup_key is unique; indexing keys include pipeline_id, so two pipelines indexing one document never collide.
• Fetches, documents, chunks, discovered links, and pipeline_documents rows use stable/natural-key update logic, and Qdrant point IDs are chunk UUIDs.
• Run completion uses a conditional update from running to completed.

Retrieval flow

The search API orchestrates an explicit retrieval → fusion → optional reranking → output pipeline for each query in the request. The nested request configuration is validated by backend/server/search/service/config.py; older flat searchMethod / topK requests are migrated to the staged model before execution. See Retrieval for every method, fusion algorithm, parameter, and limit.

Pipeline scoping

A search selects a pipeline run, not a raw collection: the run supplies the exact provider snapshot, collection, and ID used to scope results.

• Dense — Qdrant points carry a pipeline_run_id payload field and are filtered on it.
• BM25 — document_chunks is filtered directly on its pipeline_id.

Both paths hydrate chunk text and document source URLs from PostgreSQL.

Stage execution

1. Retrieval runs vector search, BM25, or both. Vector and keyword retrievers have independent candidate limits. Keyword-only search does not resolve an embedding provider and never touches Qdrant.
2. Fusion runs only for hybrid retrieval. The selected registry strategy is RRF, DBSF, weighted score fusion, or Union Only. It deduplicates candidate IDs and caps the fused pool.
3. Reranking is optional for every retrieval mode. A current cross-encoder provider re-scores the active pool; it is deliberately not part of the indexing run snapshot.
4. Output truncates either the active retrieval/fusion ranking or the reranked ranking to output.resultLimit.

The response includes a stage snapshot keyed by query input ID: active retriever rankings, fusion and reranking outputs when present, final results, candidate counts, and per-stage latency. When result storage is enabled, the same snapshot and the effective nested configuration are persisted with the search execution.

Result shape

Every method normalizes hits to the same keys: query identity, score, chunk ID, document ID, chunk index and text, dataset name, source URL, and custom metadata.

Error handling

Missing or invalid run IDs, retrieval/fusion methods, positive limits, cross-stage limit relationships, weights, or reranker configuration are HTTP errors. Embedding, Qdrant, and BM25 runtime failures propagate from the request. Only reranker runtime failures degrade gracefully, falling back to the active input pool's order capped to output.resultLimit. Search-history persistence failures are logged and swallowed, so they never fail a search.

Evaluation runs go through search_db — the exact function serving interactive /search — so a benchmark measures the implementation users actually query.

Domain model and persistence

The UI and the backend do not always use the same name: the Pipelines page manages pipeline_runs, and there is no separate persisted "pipeline template" entity.

Entities

Product object	Table	Purpose
Dataset	datasets	Sources, prep config, preparation status, corpus identity
Provider	providers	Model runtime, capability, and validated config
Pipeline run	pipeline_runs	Dataset and actor-config snapshot plus lifecycle
Source work item	crawl_targets	Per-dataset URL/file status and lineage
Fetch	source_fetches	Retrieval provenance and raw artifact path
Document	documents	Canonical parsed provenance, structured output, text path
Pipeline document	pipeline_documents	One pipeline's indexing state for one document
Chunk	document_chunks	Searchable text, metadata, offsets, embed status
Discovered link	discovered_links	Link lineage and scheduling state
Search query	search_queries	Reusable information need, shared by search and evaluation
Search	searches	One execution: pipeline, strategy config, ordered result JSON
Evaluation set	evaluation_sets	Benchmark owned by one prepared dataset, pinned to its corpus identity
Evaluation case	evaluation_cases	One search query plus its expected evidence
Relevance judgment	relevance_judgments	A graded ground-truth span in a canonical document
Evaluation run	evaluation_runs / evaluation_run_results	One execution of a set against pipelines
Outbox event	outbox_events	Durable downstream message and publish state

erDiagram
    DATASET ||--o{ CRAWL_TARGET : scopes
    DATASET ||--o{ DOCUMENT : owns
    CRAWL_TARGET ||--o| SOURCE_FETCH : produces
    DOCUMENT ||--o{ DISCOVERED_LINK : discovers
    PIPELINE_RUN ||--o{ PIPELINE_DOCUMENT : tracks
    DOCUMENT ||--o{ PIPELINE_DOCUMENT : indexed_as
    PIPELINE_RUN ||--o{ DOCUMENT_CHUNK : owns
    PIPELINE_RUN ||--o{ SEARCH : scopes
    SEARCH_QUERY ||--o{ SEARCH : executed_as
    SEARCH_QUERY ||--o{ EVALUATION_CASE : asked_by
    EVALUATION_SET ||--o{ EVALUATION_CASE : contains
    EVALUATION_CASE ||--o{ RELEVANCE_JUDGMENT : expects
    DOCUMENT ||--o{ RELEVANCE_JUDGMENT : located_in

Loading

Providers are intentionally absent from the diagram: pipeline runs store JSON snapshots rather than foreign keys to those mutable records.

Snapshots and lifecycle

Creating a pipeline run copies the dataset and the selected embedding provider into JSON snapshots and resolves actor defaults. update_status is the only mutation method — dataset, provider, and actor_configs cannot be changed after creation. A run is created pending, launched to running, and completed by the status tracker when no active pipeline document or embedding batch remains. An operator can also PATCH a status; completed and failed runs can be relaunched.

Because a run holds snapshots, updating or deleting the original dataset or provider does not rewrite it. That is what keeps runs reproducible and comparable.

PostgreSQL

Migrations are hand-written numbered SQL files. The runner executes every file on every invocation, so migrations must be idempotent. The Compose Postgres image is PostgreSQL 17 and builds pg_textsearch (tag v1.3.1) for the BM25 index.

Filesystem

Two sibling roots, both bind-mounted from the host so artifacts survive container restarts:

{DATASETS_DIR}/                       (default /tmp/datasets -> host tmp/datasets)
  {dataset_uuid}/
    raw/{crawl_target_id}/content.{ext}      # fetched bytes
    parsed/{crawl_target_id}/content.txt     # canonical normalized document

{PIPELINE_RUNS_DIR}/                  (default /tmp/pipeline_runs -> host tmp/pipeline_run)
  {pipeline_run_id}/logs/...                 # run-specific logs and diagnostics only

The canonical corpus is never copied into a pipeline-run directory. The database stores only relative paths, so every container sharing the mount resolves the same artifact. Resolution tries the dataset root first and falls back to the legacy per-run tree, so pre-split rows keep resolving.

Qdrant

Collections are named <QDRANT_COLLECTION_BASE>_<dataset-name>_<provider>_<model>_d<dim>, resolved once at run creation from the selected provider — including its actual vector size, which the provider is asked for (mock reads mock_dim, remote providers probe once) — and stored in the run's actor_configs.vector_store. Ingestion and search both read that recorded name, so they cannot disagree. A point ID is the chunk UUID; payload fields include chunk and document IDs, chunk index and type, pipeline_run_id, and namespaced system/custom metadata. Re-upserting a chunk overwrites the same point. Collections are created only when absent, and a later pipeline reusing the same name cannot change their vector size or distance.

Delete and retention behavior

• Deleting a dataset cascades its crawl targets and documents and removes its on-disk artifact tree; pipeline-run logs are untouched.
• Deleting a pipeline run cascades its pipeline_documents, chunks, and searches.
• Its Qdrant points and search_queries rows are not removed; searches simply cannot select a deleted run.
• Deleting a query used only by search history sets query_id to NULL; deleting a query referenced by an evaluation case is refused.
• Deleting a dataset or provider does not rewrite existing run snapshots.
• Automatic retention, archival, and per-run Qdrant cleanup are not implemented. scripts/full-clean.sh purges queues, deletes all Qdrant collections, resets the schema, clears logs, and rebuilds workers.

Security and deployment

Embeddorium is configured as a trusted local development system, not a hardened deployment. The default Compose stack and unauthenticated API are intended for a trusted local machine or private network. Do not expose them to an untrusted network.

Deployment model

The stack owns four named volumes (postgres_data, qdrant_data, rabbitmq_data, and an unused ollama_data) and bind-mounts source and artifact directories so they are visible on the host. It is the only supported runtime.

PostgreSQL major-version changes require an explicit operator migration: the current image is PostgreSQL 17 and will not start on a volume initialized by PostgreSQL 16 without pg_dump/restore or pg_upgrade.

Current controls

• Web fetch TLS verification is on by default. A dataset's prep config can disable it, and INSECURE_TLS_DOMAINS can allowlist domains for insecure TLS.
• Web responses are capped at 10 MiB and restricted to supported text/XML MIME types.
• Discovered web links are restricted to the parent document's origin.
• The source browser resolves paths under SOURCE_ROOT and rejects traversal.
• BM25 queries bind query text and pipeline IDs as parameters.
• Provider API keys are accepted as config and sent as bearer credentials to compatible endpoints.
• Evaluation re-reads each annotated quote from the canonical document and rejects a mismatch, so a client-side offset bug fails loudly instead of corrupting a benchmark.

Known exposure

• FastAPI has no authentication or authorization.
• CORS allows credentials for localhost UI origins on UI_PORT through UI_PORT + 2 (by default 5173–5175).
• Qdrant, Postgres, RabbitMQ, RabbitMQ management, API, and UI ports are published to the host by Compose.
• .env.docker contains development credentials; they are not production secrets.
• Provider configuration, including API keys, is stored in PostgreSQL JSON and returned by provider GET endpoints so the UI can read it.
• Local dataset APIs accept absolute paths for administrative use. The browse endpoint is root-confined, but the seeder deliberately passes absolute paths through.
• There is no request rate limiting, audit-log policy, secret-redaction contract, network policy, encryption-at-rest setup, or CSRF protection.

A production threat model and hardening guide are not present. See Scope and limitations.