feat: add best_effort option and text_unreliable flag

[abimaelmartell]

What

Adds an opt-in best_effort option that returns whatever native text can be read, even where the library is not confident in it, and a text_unreliable field on the result that says when it did.

• Rust: PdfOptions::best_effort(bool), PdfProcessResult::text_unreliable
• Python: process_pdf(path, pages=None, best_effort=False) and process_pdf_bytes(...), PdfResult.text_unreliable, stub updated

Why

Today a Scanned or ImageBased document returns no Markdown, even when it has a readable text layer (or an invisible one behind the scan). A Mixed or TextBased document also loses its Markdown when the text is judged garbled or its fonts undecodable. Callers that want a best-effort result had no way to get it. On a 200-PDF structure benchmark, one such page (165 characters of real text, classified image-based) scored 0.00 and now scores 0.91, lifting the corpus overall score from 0.870 to 0.874.

Behavior

With best_effort set:

• Scanned and ImageBased documents are extracted instead of returning early, with the same invisible-text-layer fallback Mixed documents already use. Extraction failure there is non-fatal, as for Mixed.
• The Markdown dropped for a Mixed document upgraded to Scanned, a garbled TextBased document, and an all-GID-encoded document is kept.
• Detection still sets the OCR flags (pdf_type, pages_needing_ocr, ocr_reasons_by_page, has_encoding_issues), so they remain available alongside the text.
• A TextBased page whose text reads as garbage is kept too (the per-page garbage strip is skipped), and marks the result text_unreliable.
• text_unreliable is true when the returned Markdown is text the default would have withheld, so a caller does not have to infer it from the OCR flags. It is false without best_effort, and for a document whose text was never withheld.

Default behavior is unchanged.

Notes for reviewers

• PdfProcessResult gains a public field, so code that builds it with a struct literal needs text_unreliable: false. Two in-repo uses are updated.
• Not exposed in the CLI, Node or WASM bindings yet.
• Test: test_force_extraction_keeps_scanned_text (invisible text layer over a covering image: no Markdown and text_unreliable false by default; Markdown, text_unreliable true and the same OCR page flags with the option).
• cargo fmt, cargo clippy -- -D warnings and cargo test pass. I did not run the pdf-evals regression suite; the default path is untouched.

🤖 Generated with Claude Code

[cubic-dev-ai]

All reported issues were addressed across 5 files

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 6 files (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 3 files (changes from recent commits).

<sub>Shadow auto-approve: would require human review. AI evaluation unavailable; defaulting to comment-only for safety.
Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

Confidence score: 5/5

• In src/lib.rs, the reclassification only runs when markdown is generated, so Mixed PDFs are not reclassified in ProcessMode::Analyze. Qualify the statement to modes that generate markdown.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/lib.rs">

<violation number="1" location="src/lib.rs:352">
P2: This reclassification only runs when markdown exists; `ProcessMode::Analyze` sets it to `None`, so a Mixed PDF is not reclassified there. Qualify the statement to modes that generate markdown.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved P0–P2 issue from previous reviews.

Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would require human review. Adds opt-in best_effort extraction and text_unreliable flags (Rust/Python); default unchanged. A dismissed finding says the TextBased/OCR-garbage path still drops text; the relevant diff is truncated, leaving that behavior unverified.

Turn on auto-fix | Re-trigger cubic</sub>