Skip to content

Commit 8e7f232

Browse files
committed
Fix bytes serializer dropping non-UTF-8 values to null
Serializer.do_bytes returned None on the first encoding that failed to decode, so the latin-1/utf-16/utf-16le fallbacks in ENCODINGS_TO_TRY never ran. Any non-UTF-8 bytes attribute was therefore serialized to null by to_json/to_csv/to_dict -- silent data loss (e.g. a UTF-16BE annotation Contents value). Continue to the next encoding instead, restoring the behavior from before the type-annotation refactor (9587cc7); latin-1 decodes any byte string, so values survive serialization.
1 parent 4cf8241 commit 8e7f232

3 files changed

Lines changed: 39 additions & 2 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ All notable changes to this project will be documented in this file. The format
66

77
### Fixed
88
- Initialize PDFium's form environment in `get_page_image` so that filled AcroForm field content is included when rendering pages via `Page.to_image()`. ([#1367](https://github.com/jsvine/pdfplumber/issues/1367))
9+
- Fix `Serializer.do_bytes` to continue to the next entry in `ENCODINGS_TO_TRY` instead of returning `None` on the first failed encoding, so non-UTF-8 `bytes` attributes (e.g. an annotation's `Contents`) are no longer dropped to `null` by `.to_json()`/`.to_csv()`/`.to_dict()`.
910

1011
## [0.11.10] — 2026-06-14
1112

‎pdfplumber/convert.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -119,8 +119,8 @@ def do_bytes(self, obj: bytes) -> Optional[str]:
119119
for e in ENCODINGS_TO_TRY:
120120
try:
121121
return obj.decode(e)
122-
except UnicodeDecodeError: # pragma: no cover
123-
return None
122+
except UnicodeDecodeError:
123+
continue
124124
# If none of the decodings work, raise whatever error
125125
# decoding with utf-8 causes
126126
obj.decode(ENCODINGS_TO_TRY[0]) # pragma: no cover

‎tests/test_convert.py‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,42 @@ def test_additional_attr_types(self):
173173
c = json.loads(pdf.to_json())
174174
assert len(c["pages"][0]["images"])
175175

176+
def test_serialize_bytes_encoding_fallback(self):
177+
# Serializer.do_bytes must try each entry in ENCODINGS_TO_TRY until one
178+
# decodes, instead of returning None on the first failure. latin-1 (the
179+
# second entry) decodes any byte string, so no bytes value is ever lost
180+
# to None. Regression test for the broken encoding-fallback loop.
181+
from pdfplumber.convert import ENCODINGS_TO_TRY, Serializer
182+
183+
assert ENCODINGS_TO_TRY[0] == "utf-8"
184+
assert "latin-1" in ENCODINGS_TO_TRY
185+
serialize = Serializer().serialize
186+
# Valid utf-8 (including ascii) still passes through unchanged.
187+
assert serialize(b"plain ascii") == "plain ascii"
188+
assert serialize("café".encode("utf-8")) == "café"
189+
# Bytes that are not valid utf-8 fall back to latin-1 rather than None.
190+
assert serialize(b"caf\xe9") == "café"
191+
for raw in (b"\xe9", b"\x80\x81", b"\xff\xfe\x00", b"\x91\x92\x93"):
192+
decoded = serialize(raw)
193+
assert decoded is not None
194+
assert isinstance(decoded, str)
195+
assert decoded == raw.decode("latin-1")
196+
197+
def test_json_non_utf8_bytes_preserved(self):
198+
# The annotation in this shipped fixture carries a non-UTF-8 (UTF-16BE)
199+
# Contents value. Both the top-level "contents" (decoded via page.py)
200+
# and the serialized "data.Contents" (via Serializer.do_bytes) must
201+
# survive to_json as strings instead of being dropped to null.
202+
path = os.path.join(HERE, "pdfs/issue-463-example.pdf")
203+
with pdfplumber.open(path) as pdf:
204+
annot = json.loads(pdf.to_json(object_types=["annot"]))["pages"][0][
205+
"annots"
206+
][0]
207+
assert annot["contents"] is not None
208+
data_contents = annot["data"]["Contents"]
209+
assert data_contents is not None
210+
assert isinstance(data_contents, str)
211+
176212
def test_csv(self):
177213
c = self.pdf.to_csv(precision=3)
178214
assert c.split("\r\n")[9] == (

0 commit comments

Comments
 (0)