Skip to content

ingestion.md

Latest commit

 

History

History
321 lines (239 loc) · 13.8 KB

ingestion.md

File metadata and controls

321 lines (239 loc) · 13.8 KB
Raw

Ingestion

Getting content into Embeddorium is two steps: prepare a dataset into canonical documents, then index it with one or more pipelines. This page covers the second step — pipelines, chunking, and the walkthroughs that tie the two together. For the actor-level design, see Architecture.

Prepared datasets

A dataset is the reusable corpus boundary: it owns source URLs and paths, crawl scope, fetch and validation settings, filtering, parser selection, and the canonical parsed documents that come out of them. Chunking, embeddings and vector storage are not dataset concerns — they belong to pipelines.

curl -sS -X POST http://localhost:8000/datasets \
  -H 'Content-Type: application/json' \
  -d '{"name":"example","sourceType":"web","url":"https://example.com"}'

curl -sS -X POST http://localhost:8000/datasets/<dataset-uuid>/prepare

Preparation fetches, filters and parses every source and stores canonical documents under {DATASETS_DIR}/{dataset_uuid}/parsed/. It creates no chunks, embeddings or vector collections, and a pipeline cannot launch until the dataset reaches prepared or prepared_with_errors.

Datasets is the full reference:

Topic Where
Sources, URL normalization, crawl scope, the XML-only local limit Web datasets, Local datasets
The Datasets page, preparation results, the artifact viewer The Datasets page
The lifecycle, the actor chain, where artifacts land on disk Preparation
Every prep_config block, field and default Preparation config
What invalidates a corpus, and what re-preparing destroys Invalidation and re-preparation
Every dataset endpoint API reference

Pipelines

A pipeline is a persisted indexing configuration plus its execution state. The backend entity is PipelineRun. The UI presents creation and launch on Pipelines and execution detail on Indexing Runs.

The run snapshot

Creating a run resolves a saved actor_configs object:

Block Purpose
chunk_document Chunker name and its plugin-specific settings
schedule_embeddings batch_size — chunks per embed job (default 32)
embed_chunks Full embedding provider snapshot
vector_store Derived collection name and similarity metric

Fetching, parsing, crawling, and filtering are not pipeline settings. Any such block in a create request is ignored rather than recorded on a run that could never act on it.

actorSettings.embed_chunks.provider is required and must reference a provider whose modelType is embedding (400 otherwise). The collection name is derived from the dataset name, and similarity accepts cosine, dot, or euclid — anything else (the UI picker also offers euclidean and manhattan) falls back to cosine.

curl -sS -X POST http://localhost:8000/pipeline-runs \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "example-markdown-1200",
    "datasetId": "<dataset-uuid>",
    "actorSettings": {
      "embed_chunks": { "provider": "<embedding-provider-uuid>", "similarity": "cosine" },
      "chunk_document": { "chunker": "text_markdown", "settings": { "chunk_size": 1200, "chunk_overlap": 150 } },
      "schedule_embeddings": { "batchSize": 32 }
    }
  }'

The response status is pending; the server has already snapshotted the dataset and provider. A malformed generic actor block is logged and replaced with that model's defaults rather than rejected.

Launch

curl -sS -X POST http://localhost:8000/pipeline-runs/<run-uuid>/launch

Launch is refused with 409 unless the dataset's preparation_status is prepared or prepared_with_errors. It then enumerates the dataset's canonical documents, registers one pipeline_documents row per document, publishes one chunk_document message each, sets startedAt, clears any earlier finishedAt, and moves the run to running.

Launching an already-running run returns 409. Pending, failed, and completed runs may be launched or relaunched.

Monitor

curl -sS http://localhost:8000/pipeline-runs/<run-uuid>
curl -sS 'http://localhost:8000/pipeline-runs/<run-uuid>/documents?limit=50&offset=0'

The run completes when no pipeline document is still active and completed embedding batches reach the number scheduled. Failures stay visible after completion.

Because state and chunks are keyed by pipeline, two pipelines can index the same dataset concurrently, and rebuilding one never affects the other or the dataset's documents.

Chunking

Chunking is the one indexing stage the UI exposes as a user-selectable plugin list, and chunk boundaries strongly affect retrieval — so it is the natural first comparison.

Chunker Settings (defaults)
text_markdown (default) chunk_size (1200), chunk_overlap (150)
text_fixed chunk_size (1200), chunk_overlap (150)
text_recursive chunk_size (1200), chunk_overlap (150)
text_sentence chunk_size (1200), chunk_overlap (150)
text_sliding_window window_size (200), step_size (100)
text_section none
legal_xml target_tokens (600), max_tokens (1100), min_tokens (100)

Read the authoritative field keys, defaults, and bounds from the API rather than guessing:

curl -sS http://localhost:8000/chunkers

Settings are nested under chunk_document.settings and use FieldSpec.key values verbatim (snake_case), unlike the surrounding camelCase actor settings:

curl -sS -X POST http://localhost:8000/pipeline-runs \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "chunker-compare-recursive",
    "datasetId": "<dataset-uuid>",
    "actorSettings": {
      "embed_chunks": { "provider": "<embedding-provider-uuid>", "similarity": "cosine" },
      "chunk_document": { "chunker": "text_recursive", "settings": { "chunk_size": 800, "chunk_overlap": 100 } }
    }
  }'

Comparing chunking choices

Create several pipelines over the same prepared dataset, differing only in the chunk_document block. Each records its chunker and settings in its snapshot, so results stay attributable. Compare them on the Compare tab or score them with an evaluation run.

Because the Qdrant collection is named per dataset, this works only while the embedding dimension and distance are unchanged. Comparing incompatible embedding models needs a distinct dataset name or a collection cleanup.

A newly added chunker appears only after the server and worker-chunk-document services restart — discovery is cached per process. To write one, see Add a chunker.

Walkthrough: a website

  1. Create the web dataset.

    curl -sS -X POST http://localhost:8000/datasets \
  -H 'Content-Type: application/json' \
  -d '{"name":"example","sourceType":"web","url":"https://example.com"}'

  2. Bound the crawl scope for a first run, so you are not waiting on a site-wide crawl to see whether the rest works.

    curl -sS -X PUT http://localhost:8000/datasets/<dataset-uuid>/prep-config \
  -H 'Content-Type: application/json' \
  -d '{"schedule_discovered_links": {"follow_child_links": false}}'

  3. Prepare, and wait for preparationStatus to reach prepared.

    curl -sS -X POST http://localhost:8000/datasets/<dataset-uuid>/prepare
curl -sS http://localhost:8000/datasets/<dataset-uuid>

  4. Create an embedding provider, then create and launch a pipeline as shown in Pipelines.

  5. Monitor with GET /pipeline-runs/{id}/documents, then run a search.

Walkthrough: local XML

  1. Add XML files under sources/ (configurable via SOURCE_ROOT). Compose bind-mounts this directory into every worker.

  2. Create the local dataset, selecting a file or folder in the UI source browser, or by API:

    curl -sS -X POST http://localhost:8000/datasets \
  -H 'Content-Type: application/json' \
  -d '{"name":"local-laws","sourceType":"local","paths":["xml.2026.en"]}'

    A directory is searched recursively for *.xml; an individual path is queued directly.

  3. Prepare exactly as for a web dataset. Preparation publishes one validate_source message per matched XML file.

  4. Choose a chunker. For Estonian legal acts, legal_xml emits section-oriented chunks using the Juurakt-aware reader and falls back to markdown chunking when the input is not a recognized act. Other XML uses the format parser plus a text chunker.

  5. Create and launch a pipeline, then search.

Troubleshooting ingestion

For stack-level problems — services not starting, port conflicts, broker or database connectivity — see Troubleshooting.

A dataset stays preparing

Confirm the preparation workers and worker-outbox-dispatcher are running:

docker compose logs worker-outbox-dispatcher
docker compose logs worker-parse-source
docker compose logs worker-track-dataset-preparation

Downstream stages are published from the transactional outbox, so a stopped dispatcher stalls the workflow without failing it. Use GET /datasets/{id}/targets to find the source still in an active state.

A pipeline stays running

docker compose logs worker-chunk-document
docker compose logs worker-embed-chunks
docker compose logs worker-track-pipeline-status

Use GET /pipeline-runs/{id}/documents or Indexing Runs to find the document still in an active state. A failed pipeline document is terminal and does not by itself block the run.

A source was skipped

Permanent conditions update the crawl target instead of retrying:

Status Meaning
skipped_unsupported Unsupported MIME type or no matching parser
skipped (reason not_relevant) The keyword filter excluded it
failed_permanent Permanent fetch error
failed_transient Transient fetch error surfaced after retries

Every actor declares max_retries=3; transient conditions retry with backoff before becoming terminal. The Preparation results tab groups these statuses into five outcomes and explains each one — see Preparation results.

Unsupported content

Built-in parsing supports HTML/XHTML, plain text, and XML. The parser selector lists pdf, but no PDF parser plugin exists and application/pdf is not a supported content type, so a PDF source is rejected at fetch time and its target ends at skipped_unsupported. Local ingestion accepts *.xml only. Web responses over 10 MiB are rejected.

Launching a pipeline returns 409

Either the pipeline is already running, or its dataset is not prepared. Check preparationStatus on GET /datasets/{id} — it must be prepared or prepared_with_errors.

Keyword search returns nothing

If the migration that builds the BM25 extension and index failed, keyword search cannot run. Re-run scripts/migrate.sh and inspect its output and the postgres service logs.

Inspect artifacts

Under Compose, canonical artifacts are bind-mounted to tmp/datasets/{dataset_uuid}/ with raw/ and parsed/ subtrees keyed by crawl target ID. Per-run logs are under tmp/pipeline_run/{pipeline_run_id}/logs/. Comparing parsed text against the chunks in the database shows exactly what each stage produced. The same raw and parsed text is readable without a shell from the Source artifacts tab.

Reset local state

docker compose down -v      # remove the named Compose volumes
scripts/full-clean.sh       # purge queues, drop Qdrant collections, reset schema, rebuild workers

full-clean.sh is destructive to local data.

How pipelines and searches use a provider

Two very different bindings, and the difference matters when you edit or delete a provider:

Consumer Capability Bound Effect of editing the provider afterwards
Ingestion pipeline (embed_chunks) embedding Snapshot — the whole provider row is copied into actor_configs.embed_chunks.provider at run creation None. The run keeps embedding with what it recorded
Search over an indexed dataset embedding Read from the selected run's snapshot None — and that is the point: the query is embedded with exactly the model the collection was indexed with
Reranking stage of a search cross-encoder Live, by provider id at search time Takes effect on the next search
Compare mode and vector-space projection embedding Selected per request / read from the run Compare uses the provider you pick in the form

Creating a pipeline requires a provider whose capability is embedding — anything else is a 400. Creation then probes the provider for its vector dimension, because the Qdrant collection name carries it (<QDRANT_COLLECTION_BASE>_<dataset>_<provider>_<model>_d<dim>). An unreachable provider therefore fails the request with 503 rather than guessing a dimension and splitting one model's vectors across two collections.

Reranking is validated up front too: a provider id that does not resolve, or that is not a cross-encoder, is a request error. A reranker that resolves but whose endpoint fails at search time degrades instead — you get the retrieval ordering back, marked as a degraded stage in the Search Lab rather than silently passed off as reranked.