Streaming-split reader support + expand_to_document default-namespace fix (0.3.18) - #211
Merged
Conversation
…mpty_element, outer_xml outer_xml dumps the reader-owned expanded node directly instead of going through xmlTextReaderReadOuterXml: that API deep-copies the node parentless, and xmlNewReconciledNs then mints a 'default:' prefix onto default-namespace content. Guard test pins the exact serialization. 0.3.18 version bump + changelog. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
node_type() maps end-element and both whitespace events to None; a structure-reconstructing caller needs them apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
xmlReconciliateNs mints xmlns:default= for a NULL-prefix namespace inherited from an un-copied ancestor, making every element serialize as <default:x> — and a dumped subtree of the copy never re-parses into the right namespace. Restore minted declaration prefixes to the source element's prefixes (clash-checked). Guard added to the reconciliation test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dginev
added a commit
to dginev/latexml-oxide
that referenced
this pull request
Aug 1, 2026
Published (KWARC/rust-libxml#211); cargo tree confirms the registry resolution. Suite 1854/1854 against the published crate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dginev
added a commit
to dginev/latexml-oxide
that referenced
this pull request
Aug 1, 2026
… commodity RAM (#477) * feat(post): two-pass streaming split — huge documents split without the whole-DOM parse The last whole-DOM stage of post (Split's one-time parse) OOM'd a 31 GB laptop on the 131 MB witness's 2.68 GB core XML (>26.6 GB mid-parse, exit 137, zero pages). latexml_post::stream_split partitions the file with a TextReader pull-parse: pages assembled as text and spilled at close, wrapper subtrees (back-matter shells) via mini-DOM descent, then a pre-order Scan sweep that preserves Scan's order-sensitive semantics. Auto-engages for file input >= 1 GiB (LATEXML_POST_STREAM_SPLIT forces, LATEXML_POST_STREAM_THRESHOLD tunes); oversized in-memory handoffs now spill at the same threshold so the single-invocation .tex -> .htm flow streams too. Page spills live beside the destination, not the (possibly tmpfs) system temp dir. Parity gate 118_streaming_split_parity: byte-identical rendered pages vs the DOM split across run adjacency, TOC suppression, inlist=toc lookahead, wrapper descent, unnamed pages, template copies, inherited attributes. The gate caught a latent DOM-split defect, also fixed: the inherited xml:lang copy silently never fired (namespaced-attribute read; get_attribute returns None for xml:* on parsed docs). Requires libxml 0.3.18 (attributes_qname/value/is_empty_element/event/ outer_xml + the expand_to_document default:-prefix mint fix) — TEMPORARY [patch.crates-io] path override in Cargo.toml until 0.3.18 is published; REMOVE before merge. Vendored-native audit pins refreshed (libxml 0.3.18, ar_archive_writer 0.5.2 — verdicts re-verified). Suite 1853/1853, clippy -D warnings clean, rustdoc clean, lint.sh green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(stream-split): boundary-checked element probe + multiple-root guard The descent probe matched '<index' inside '<indexmark' (and any name-prefix collision), sending every \index-marked paragraph through a mini-DOM descent — correct but a per-paragraph expand+copy storm on index-heavy documents. The probe now requires a name-boundary character after the localname. A second top-level element (recover-mode oddity) now fails loud instead of clobbering the root spill slot. Guards: element_probe_respects_name_boundaries; parity + handoff + post suites green; A/B on a 300 MB slice of the Nasser witness: 892 pages byte-identical between the DOM and streaming paths (diff -r clean), peak RSS 7.1 GB (DOM) vs 771 MB (streamed). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(streaming-post): record the libxslt page-size ceiling + A/B breadth results --splitat A/B on a 300 MB witness slice: byte-identical DOM-vs-streaming trees at subsubsection (892 pages), section (17) and chapter (5) — the chapter case failing identically on both paths at a ~260 MB page (libxslt ceiling, reproduced bit-for-bit on the pre-branch rc4 binary; not a regression). Unsplit giant documents hit the same ceiling and always have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * build: libxml 0.3.18 from crates.io — drop the temporary path override Published (KWARC/rust-libxml#211); cargo tree confirms the registry resolution. Suite 1854/1854 against the published crate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci(windows): un-wedge the check lane (kpathsea toolchain check) + libxml 0.3.19 floor The compile-only Windows check job runs on a TeX-less runner, so kpathsea's build-script toolchain check ('no usable TeX backend') killed it before checking anything — the lane has NEVER been green, including the PRs that introduced it (#465/#466). Set KPATHSEA_SKIP_TOOLCHAIN_CHECK=1 there, which is that check's documented bypass for build-without-TeX machines; the build+test job installs TeX Live and keeps the guard armed. libxml floor 0.3.18 → 0.3.19: 0.3.18 fails to LINK on windows-msvc (xmlFree is not a linkable data symbol there; LNK2019) — fixed upstream via the crate's per-target free shim (KWARC/rust-libxml#212), 0.3.19 published, 0.3.18 yanked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * tools(audit): an AUDITED entry may list multiple audited versions The lockfile is gitignored, so CI and dev boxes can resolve different compatible versions of the same crate during a release window — ar_archive_writer flip-flopped 0.5.2/0.5.3 between this laptop and the CI runner, each side failing the version-exact audit the other passed. An entry's version field now accepts a tuple listing every audited version (both 0.5.2 and 0.5.3 carry the same uncompiled reference/*.cpp verdict); unaudited versions still fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three commits preparing 0.3.18, driven by latexml-oxide's streaming post-split (splitting a multi-GB XML into subdocuments without materializing large subtrees):
TextReaderadditions —attributes_qname()(document-order attributes incl. namespace declarations, without expanding the subtree),value(),is_empty_element(), andouter_xml()(serialize the current subtree exactly as input).outer_xmldeliberately does not usexmlTextReaderReadOuterXml: that API deep-copies the node parentless, andxmlNewReconciledNsthen mints adefault:prefix onto default-namespace content. Dumping the reader-owned node directly keeps ancestor declarations reachable, so nothing is fabricated.ReaderEvent— losslessxmlReaderTypesaccess.node_type()maps end-element and both whitespace events toNone; a structure-reconstructing caller needs them distinguished.expand_to_documentminteddefault:prefixes.xmlReconciliateNsturns a NULL-prefix namespace inherited from an un-copied ancestor intoxmlns:default="…", so every element serialized as<default:x>— and a dumped subtree of the copy never re-parsed into the right namespace. Minted declaration prefixes are now restored to the source element's prefixes (clash-checked). Guard added to the reconciliation test.Validation
🤖 Generated with Claude Code