JosePDFs

Turn static PDFs into structured, API-ready REDCap data — on your own machine.

In the story, Joseph took Pharaoh's abstract, unstructured dreams and turned them into the concrete, structured plans that governed Egypt. He was the mediator who made two worlds talk to each other.

JosePDFs does the same thing for research data: it reads unstructured clinical/administrative PDFs and turns them into validated records ready to import into a REDCap project — bridging the world of scanned documents and the world of the database.

JosePDFs = Joint OCR & Semantic Extraction Pipeline for Document→Form structuring.

---

What it does

1. Reads a PDF (embedded text, or Tesseract OCR for scans).
2. Extracts fields with a local LLM + deterministic parsers — all defined by your domain module.
3. Validates every value against your live REDCap data dictionary — a value is written only if the field exists, is on the target event, its branching logic allows it, and any coded value is a valid option. Correct by construction.
4. Resolves the record: new baseline vs. a new/existing follow-up instance, matched against live data (no duplicate visits).
5. Carries ongoing state forward on a new follow-up (a device or therapy that persists until changed), flagging anything it inherits.
6. Previews first — nothing is imported until you pass --commit.

Everything runs locally. Your documents and API token never leave your machine.

Measure your accuracy

You can't improve what you don't measure. benchmark.py runs the pipeline over your PDFs and compares each result, field by field, to the row a human already entered in REDCap — reporting per-field agree / mismatch / extra so you see exactly where extraction is strong and where it needs work (dry-run, nothing is imported):

python benchmark.py path/to/pdfs/ --limit 20

Design: a generic engine + a private domain

The engine (josepdfs/) is completely study-agnostic — it contains no field names, no prompts, no disease. All project-specific knowledge lives in a DomainConfig you write and keep local and gitignored:

josepdfs/            the public engine
  ocr.py             PDF -> text
  llm.py             local OpenAI-compatible / Ollama client
  redcap.py          schema-driven validation, resolution, import
  pipeline.py        orchestration
  domain.py          the DomainConfig interface + loader
domains/
  example.py         a FICTIONAL demo domain (a plant-care registry)
domains_local/       YOUR real domain lives here (gitignored, never pushed)
cli.py               command-line entry point

This split is the whole point: you can open-source the methodology and engineering so others reproduce it for their own projects, while your own CRF structure, prompts, and data stay private.

Quick start

pip install -r requirements.txt
cp .env.example .env      # fill in your REDCap URL/token and LLM endpoint
# run the fictional example domain against a demo project:
python cli.py path/to/document.pdf

Set your endpoints via environment variables (see .env.example): REDCAP_API_URL, REDCAP_TOKEN, LLM_BASE_URL, LLM_MODEL, JOSEPDFS_DOMAIN.

Try it in 30 seconds (offline — no install, no REDCap, no LLM)

Just clone and run — the offline engine, demo and tests use only the Python standard library (no pip install needed):

python demo_offline.py         # watch a document's fields get validated into a
                               # REDCap row; bad values are dropped with reasons
python tests/test_engine.py    # 8 engine tests (or: pip install pytest && pytest -q)

demo_offline.py uses an in-memory mock data dictionary, so you can see the core behaviour — schema validation, ISO date normalization, event restriction, new-vs-follow-up resolution, duplicate-visit detection — without any server or model.

Try the real extraction (needs a local LLM, still no REDCap)

Install Ollama, pull a small model, and run the bundled fictional sample PDF through the actual OCR + LLM extraction:

ollama pull qwen2.5:3b-instruct
set LLM_MODEL=qwen2.5:3b-instruct        # macOS/Linux: export LLM_MODEL=...
python extract_demo.py                    # reads examples/sample_plant_note.pdf

Expected output — the note's contents turned into structured fields:

{ "_patient_id": "PLANT-042", "_visit_date": "14/05/2024",
  "log_date": "14/05/2024", "height_cm": 37, "health": 3, "watered": 1 }

That's the "PDF → structured data" half working locally. Add your REDCap credentials (below) and the same values get validated against your live dictionary and imported.

Write your own domain

Copy domains/example.py to domains_local/mine.py, adapt it, and set JOSEPDFS_DOMAIN=domains_local.mine. A domain declares:

• structural config: event names, the identifier field, the follow-up date field;
• mapping hints: field renames, values to drop, numeric sanity ranges, fields whose state persists across visits;
• extract_fields(text, schema) -> {field: value}: your extraction brains (LLM prompts + deterministic parsers), returning REDCap-coded values plus _patient_id / _visit_date;
• optional segment_visits(...) for longitudinal reconstruction.

The engine validates whatever you return against the live dictionary, so a mistake in a domain can never write an invalid field.

Requirements

• Python 3.10+
• Tesseract (for scanned PDFs) and Poppler (for pdf2image), installed separately
• A local LLM served over an OpenAI-compatible API (e.g. Ollama)
• A REDCap project with API access

License

MIT — see LICENSE.