knowledge-flow-backend eagerly loads Docling/PaddleOCR (onnxruntime) at startup — split into a slim API image + a dedicated worker image

[dimitri-tombroff]

Observation

knowledge-flow-backend (the API service) uses ~1Gi RSS at complete idle — no
uploads, no active work — both on a GKE deployment and on a local docker-compose
stack. That's surprising: the backend hands off actual document content
processing to knowledge-flow-worker via Temporal; it isn't supposed to need
heavy ML runtime itself.

Confirmed via startup logs: the backend emits the same onnxruntime
pthread_setaffinity_np errors as the worker, and the same Docling plugin-load
messages — i.e. it's loading the full Docling/OCR stack on boot, not just the
worker.

Root cause (runtime memory)

ApplicationContext.__init__ (application_context.py:296-306, shared
unconditionally by both the backend and the worker entrypoints) calls
validate_profile_input_processor_config()
(application_context.py:184-216). That function importlib.import_module()s
every configured input-processor class across all three profiles — fast,
medium, and rich — not just the active default_profile — to fail fast on
a broken class_path.

The medium/rich PDF processor chain has two modules with module-top-level
imports of heavy ML libraries:

• core/processors/input/pdf_markdown_processor/docling_processor.py — imports
  docling.backend.pypdfium2_backend, docling.datamodel.* (pulls in
  onnxruntime)
• core/processors/input/common/ocr/paddle_ocr.py — from paddleocr import PaddleOCR

Since Python executes a module's top-level code on import, merely resolving
these classes for validation purposes triggers full Docling/onnxruntime and
PaddleOCR initialization — in every process that constructs an
ApplicationContext, regardless of whether that process (or that profile) is
ever actually used.

Confirmed this isn't scheduler-conditional: ingestion_service.py::extract_metadata
(called directly from the backend's own request path,
ingestion_controller.py::_stream_upload_process) resolves and constructs the
processor for whatever profile the caller requests, synchronously, before
anything touches Temporal — so gating eager loading on "is Temporal the
scheduler backend" would risk breaking metadata extraction for whichever
profile is actually in active use.

Also confirmed the class constructors are already properly lazy —
PdfMarkdownProcessor.__init__ just makes a temp folder; PaddleOCR(...) /
DocumentConverter(...) construction happens inside the actual
content-extraction methods, not __init__, not process_metadata(). And
process_metadata() itself (the base-class method the backend calls
synchronously — base_input_processor.py:211) never touches Docling/PaddleOCR
for any profile, it's a lightweight file-validity check + metadata read. So
the entire eager cost is exactly the two module-level import lines above —
nothing more needs to move.

Root cause (image size — separate from the memory issue)

apps/knowledge-flow-backend/dockerfiles/Dockerfile-prod bakes Docling/PaddleOCR
model weights into the image at build time, unconditionally, for the one
image shared by both backend and worker:

RUN HF_HUB_OFFLINE=0 TRANSFORMERS_OFFLINE=0 python /workspace/scripts/download_models.py --models-dir ${PADDLE_MODELS_DIR}

download_models.py fetches real model files — layout detection (ONNX), a
figure classifier, PP-OCR detection/recognition models. pyproject.toml lists
docling==2.102.2, paddleocr>=3.6.0, paddlepaddle>=3.3.1 as plain,
unconditional dependencies (no extras group, no optional marker). Fixing the
runtime-memory issue above (lazy imports) does not fix this — the image
itself still ships these packages and model weights to every backend replica
for nothing: wasted pull time, wasted disk, unnecessary attack surface.

Fix — two parts, must ship together, not sequenced

Part 1 (prerequisite — makes Part 2 possible at all). Move the
docling/paddleocr imports in docling_processor.py and paddle_ocr.py
from module top-level into the method/constructor that actually needs them.
Preserves validate_profile_input_processor_config's fail-fast guarantee (the
class path still resolves and type-checks on import) without paying ML runtime
init cost until a document of that profile is genuinely processed.

Part 2 (the actual win — smaller image, faster pulls, quieter restarts,
smaller attack surface). Split into two images:

• pyproject.toml: move docling, paddleocr, paddlepaddle into an
  optional dependency group (e.g. [project.optional-dependencies] worker = [...])
  instead of unconditional deps.
• A second Dockerfile (or a build-arg-gated stage in the existing one) for the
  backend that skips both the heavy deps and the download_models.py step.
• One more job in Build-and-push-docker.yml, publishing a dedicated worker
  image alongside the (now genuinely slim) knowledge-flow-backend image.

Critical ordering constraint — this is why the two parts must land in the
same coordinated change, not as independently sequenced fixes: a slim image
built before Part 1 lands would fail to start. validate_profile_input_processor_config
unconditionally tries to import every profile's processor class at boot,
including medium/rich's Docling-based one, and explicitly raises
ImportError(f"Input Processor '{entry.class_path}' could not be loaded: {e}")
when the package isn't installed. Shipping Part 2 without Part 1 first (or in
the same change) means every backend pod hits that error on every startup.

No fred-deployment-factory chart changes needed to consume this once shipped —
knowledgeFlow.image.repository and knowledgeFlowWorker.image.repository are
already independent Helm values (currently both point at the same image; that's
purely because only one image exists today). Pointing the backend at the new
slim image is a one-line, reversible values change on the deployment side.

Why this matters beyond this one fix

This is also a deliberate first step toward more clearly separated,
independently-scalable file-processing workers — smaller/quieter API pods,
dedicated heavy-lifting worker images that can be scaled, resourced, and
rolled out independently of the API surface. Worth treating as a real
direction, not just a one-off image-size optimization.

Impact

• Every knowledge-flow-backend replica pays ~1Gi+ of unnecessary baseline
  runtime memory for ML runtimes it may never use.
• Every knowledge-flow-backend replica ships Docling/PaddleOCR model weights
  in its image for nothing — slower pulls, more disk, larger attack surface.
• Same behavior observed both on GKE and on a local docker-compose stack,
  confirming it's inherent to ApplicationContext initialization and image
  packaging, not deployment-specific.