fix(tounicode): read cmap-less core Windows subsets by their own glyph order, and their blank glyphs as spaces

[abimaelmartell]

Summary

#522 decodes a TrueType subset that has no cmap, no glyph names and no ToUnicode through the standard Macintosh glyph order. Two gaps were left for subsets of the core Windows fonts, which Word embeds this way. Ten Sourcewell contract packets, embedded in city agenda PDFs that search-index flagged for garbage text, all show both gaps.

• The Windows variant of the order. Arial, Times New Roman, Courier New, Verdana, Georgia and Tahoma differ from the Macintosh order from slot 172 on: no nonbreakingspace at 172 and no apple at 210, Euro in the place of currency, and overscore and middot at the end. Read through the Macintosh order, every glyph from 172 on came out as its neighbour: “N/A” as —N/A“, company’s as company‘s, en dashes as œ.
  • The font's own advances choose the variant. An accented letter advances like its base letter, which both orders keep at the same slots, so each order is scored on how many of its accented slots from 172 on fit their base letter and how many do not.
  • The Windows order is taken only when it fits more slots and misfits fewer. Equal evidence, as in a subset that zeroed its dropped glyphs' metrics, keeps the Macintosh order, which is today's behaviour.
  • Checked against the fonts shipped with macOS and every program embedded in the fulton packet: each Windows font scores 35 fits and 0 misfits under the Windows order and 10–24 misfits under the Macintosh one; Helvetica and Times score the other way round.
• Blank glyphs past the order. A glyph past the 258 named slots has no name to read. One that paints nothing but advances now reads as a space. The core Windows fonts draw U+2000–U+200A as a one-point outline, and Word justifies lines with the en and em spaces (glyphs 3024 and 3025 in Arial). These came out as U+FFFD between every word (Proposer�Legal�Name), and the pages went to OCR as suspected_garbled_text.
  • Only glyphs with an outline record count. A glyph the subset dropped has no record, though it often keeps its advance, so those slots stay unmapped.

Results

On the 11 PDFs from the report that could be downloaded (the ten Sourcewell packets, plus the Texas House rules PDF, which was already clean):

	Before	After
U+FFFD share	1–6%	0.0% (santafe 0.1%, an unrelated font)
fulton pages in pages_needing_ocr	40	0
fulton “ / ”	48 / 26	48 / 48
stray œ	11–28 per document	0

On the pdf-evals corpus the fix moves one snapshot: the Navigating AI book's chatbot figures, set in an ArialMT subset of this kind, read OpenAI’s, don’t, Anne’s and seven more where they read ‘. It is synced in firecrawl/pdf-evals#114. A control run of main's binary moves nothing, and no curated score row moves.

Monospaced Windows fonts such as Courier New still fail the existing i/l against m/w corroboration, so they never reach this path.

Tests

• Unit tests:
  • a core Windows subset reads “N/A”, –, ’ and € through the Windows order;
  • a core Macintosh subset still reads through the Macintosh order;
  • a subset that zeroed its dropped glyphs' metrics keeps the Macintosh reading;
  • the Windows names first depart from the Macintosh ones at 172;
  • a one-point blank glyph past the order reads as a space, while a dropped slot, a zero-width blank and an outlined glyph past the order stay unmapped.
• The test font builder can now draw a glyph as a single point.

These pass:

• cargo fmt
• cargo clippy -- -D warnings
• cargo test
• bench.py test in pdf-evals (one snapshot moved, synced in firecrawl/pdf-evals#114)

🤖 Generated with Claude Code

---

Summary by cubic

Fixes cmap-less TrueType decoding for core Windows font subsets, which previously read every glyph from slot 172 on as its neighbor (“N/A” as —N/A“) and rendered blank justification spaces as U+FFFD garbage.

• The font's own glyph advances now select the Windows variant of the standard glyph order; equal evidence keeps the Macintosh order.
• Glyphs past the order that paint nothing but advance now decode as spaces, while dropped slots stay unmapped.
• On the reported Sourcewell city agenda PDFs, U+FFFD share drops from 1–6% to 0.0%, and 40 pages previously sent to OCR as garbled now read as text.

^{Written for commit baa841b. Summary will update on new commits.}

[cubic-dev-ai]

All reported issues were addressed across 3 files

<sub>Shadow auto-approve: would auto-approve with 1 open P3 issue. Fixes cmap-less TrueType decoding for core Windows font subsets by selecting the Windows glyph order via advances and mapping blank glyphs to spaces, with tests; bounded bug fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would auto-approve. Fixes decoding of cmap-less TrueType subsets for core Windows fonts: selects the Windows glyph-order variant by advances and maps blank glyphs to spaces, with comprehensive tests and changelog.

Re-trigger cubic</sub>