feat(post): two-pass streaming split — book-scale documents render on commodity RAM - #477
Merged
Conversation
…he whole-DOM parse The last whole-DOM stage of post (Split's one-time parse) OOM'd a 31 GB laptop on the 131 MB witness's 2.68 GB core XML (>26.6 GB mid-parse, exit 137, zero pages). latexml_post::stream_split partitions the file with a TextReader pull-parse: pages assembled as text and spilled at close, wrapper subtrees (back-matter shells) via mini-DOM descent, then a pre-order Scan sweep that preserves Scan's order-sensitive semantics. Auto-engages for file input >= 1 GiB (LATEXML_POST_STREAM_SPLIT forces, LATEXML_POST_STREAM_THRESHOLD tunes); oversized in-memory handoffs now spill at the same threshold so the single-invocation .tex -> .htm flow streams too. Page spills live beside the destination, not the (possibly tmpfs) system temp dir. Parity gate 118_streaming_split_parity: byte-identical rendered pages vs the DOM split across run adjacency, TOC suppression, inlist=toc lookahead, wrapper descent, unnamed pages, template copies, inherited attributes. The gate caught a latent DOM-split defect, also fixed: the inherited xml:lang copy silently never fired (namespaced-attribute read; get_attribute returns None for xml:* on parsed docs). Requires libxml 0.3.18 (attributes_qname/value/is_empty_element/event/ outer_xml + the expand_to_document default:-prefix mint fix) — TEMPORARY [patch.crates-io] path override in Cargo.toml until 0.3.18 is published; REMOVE before merge. Vendored-native audit pins refreshed (libxml 0.3.18, ar_archive_writer 0.5.2 — verdicts re-verified). Suite 1853/1853, clippy -D warnings clean, rustdoc clean, lint.sh green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The descent probe matched '<index' inside '<indexmark' (and any name-prefix collision), sending every \index-marked paragraph through a mini-DOM descent — correct but a per-paragraph expand+copy storm on index-heavy documents. The probe now requires a name-boundary character after the localname. A second top-level element (recover-mode oddity) now fails loud instead of clobbering the root spill slot. Guards: element_probe_respects_name_boundaries; parity + handoff + post suites green; A/B on a 300 MB slice of the Nasser witness: 892 pages byte-identical between the DOM and streaming paths (diff -r clean), peak RSS 7.1 GB (DOM) vs 771 MB (streamed). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dth results --splitat A/B on a 300 MB witness slice: byte-identical DOM-vs-streaming trees at subsubsection (892 pages), section (17) and chapter (5) — the chapter case failing identically on both paths at a ~260 MB page (libxslt ceiling, reproduced bit-for-bit on the pre-branch rc4 binary; not a regression). Unsplit giant documents hit the same ceiling and always have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Published (KWARC/rust-libxml#211); cargo tree confirms the registry resolution. Suite 1854/1854 against the published crate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…xml 0.3.19 floor
The compile-only Windows check job runs on a TeX-less runner, so kpathsea's
build-script toolchain check ('no usable TeX backend') killed it before
checking anything — the lane has NEVER been green, including the PRs that
introduced it (#465/#466). Set KPATHSEA_SKIP_TOOLCHAIN_CHECK=1 there, which
is that check's documented bypass for build-without-TeX machines; the
build+test job installs TeX Live and keeps the guard armed.
libxml floor 0.3.18 → 0.3.19: 0.3.18 fails to LINK on windows-msvc (xmlFree
is not a linkable data symbol there; LNK2019) — fixed upstream via the
crate's per-target free shim (KWARC/rust-libxml#212), 0.3.19 published,
0.3.18 yanked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lockfile is gitignored, so CI and dev boxes can resolve different compatible versions of the same crate during a release window — ar_archive_writer flip-flopped 0.5.2/0.5.3 between this laptop and the CI runner, each side failing the version-exact audit the other passed. An entry's version field now accepts a tuple listing every audited version (both 0.5.2 and 0.5.3 carry the same uncompiled reference/*.cpp verdict); unaudited versions still fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The 131 MB witness's single-invocation
.tex → .htmdied at the core→post handoff (Document too large for i32, #471 fixed that), which exposed the real wall: post's Split needs the ENTIRE document parsed as one DOM — >26.6 GB for this witness's 2.68 GB core XML, an OOM with zero pages written on a 31 GB host. No configuration reached success.What this does
latexml_post::stream_split— theSTREAMING_POST_DESIGN_2026-07-06.md§3 front-end, whose "revisit if a <64 GB target appears" trigger has fired. ATextReaderpull-parse assembles each page as text and spills it at page close (wrapper subtrees like back-matter shells take a mini-DOM descent); a pre-order Scan sweep then re-parses one spilled page at a time, preserving Scan's order-sensitive semantics exactly. Peak = the open ancestor chain + one content subtree. Auto-engages for file input ≥ 1 GiB (LATEXML_POST_STREAM_SPLITforces,LATEXML_POST_STREAM_THRESHOLDtunes); oversized in-memory handoffs spill at the same threshold so the single-invocation flow streams too. Page spills live beside the destination (not a possibly-tmpfs system temp dir).Fixed en route
XML_PARSE_HUGE: without it, libxml2's per-document dictionary cap poisons the ID table from ~1.47 GB (the witness's 237,732 bogusID … already definedreports — each id occurs exactly once) and the parse dies at ~1.71 GB.xmllint --streamreproduces;--hugeclears.xml:lang(namespaced-attribute read returned None; the copy silently never fired) — caught by the parity gate.<indexmarkno longer triggers an<indexmini-DOM descent per paragraph).Evidence
--splitat=subsubsection --max-memory=26000. Was: exit 137, 0 pages.118_streaming_split_parity— byte-identical rendered pages vs the DOM split (run adjacency, TOC suppression,inlist=toclookahead, wrapper descent, unnamed pages, template copies, inherited attributes). A/B on a 300 MB witness slice: byte-identical trees atsubsubsection(892 pages),section(17) andchapter(5) — the chapter case failing identically on both paths at a ~260 MB page (pre-existing libxslt ceiling, reproduced on the rc4 binary).Follow-ups (documented in the design doc, not here): parallel page rendering (XSLT is 60 % of post wall), flat handoff serialization, the render loop's ~150 KB/page retention.
🤖 Generated with Claude Code