Skip to content

Decode UTF-8 BOM text strings in PDF metadata - #1422

Open
hyperammonemia wants to merge 1 commit into
jsvine:developfrom
hyperammonemia:fix/utf8-text-strings
Open

hyperammonemia wants to merge 1 commit into
jsvine:developfrom
hyperammonemia:fix/utf8-text-strings

Conversation

@hyperammonemia

Copy link
Copy Markdown

decode_text recognizes a UTF-16BE byte order mark but treats a UTF-8 BOM as PDFDocEncoding. A PDF 2.0 UTF-8 text string such as b"\xef\xbb\xbfCaf\xc3\xa9" therefore becomes Café instead of Café.

Recognize the EF BB BF prefix and decode the remaining bytes as UTF-8. This uses the same errors="ignore" policy as the existing UTF-16BE branch. Unmarked PDFDocEncoding, UTF-16BE, and string inputs retain their existing handling.

The PDF Association encoding explanation describes this BOM. Its public test PDF at a fixed revision reproduces the problem: the current metadata Author is Peter Wyatt; after the change it is Peter Wyatt, and the multilingual Title decodes correctly. The external PDF is not added to this PR.

Tests cover BOM-marked ASCII and multibyte text, empty and malformed UTF-8 payloads, and legacy encoding controls. An in-memory PDF with an xref table tests actual metadata parsing with UTF-8, UTF-16BE, and PDFDocEncoding fields together. With the old decoder, six cases fail and seven legacy controls pass; all thirteen pass with the change.

Validation: the complete fork workflow runs the full suite and package build on Python 3.10–3.14, plus Black, isort, flake8, and strict mypy, at commit 141237ed940da5fc60a083eb9465dc2b9ea5a332. All checks pass. The suite contains 186 cases, including the 13 added here.

On Windows, a broader local run had 107 passes and four CLI newline failures. Those same four failures reproduce in the previous checkout without this change; all 186 cases pass in the Linux workflow above.

This changes PDF text-string decoding. It does not claim complete PDF 2.0 support or change page-glyph text extraction.

AI assistance: OpenAI Codex investigated the issue, prepared the patch and tests, and wrote this description. It executed validation and used a separate agent for review; no human review is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant