feat(post): two-pass streaming split — book-scale documents render on commodity RAM

.github/workflows/windows-ci-manual-trigger.yml

@@ -163,6 +163,14 @@ jobs:
       RUST_BACKTRACE: "1"
       VCPKG_TRIPLET: x64-windows-static-md
       VCPKGRS_TRIPLET: x64-windows-static-md
+      # This job is COMPILE ONLY on a TeX-less runner, which is precisely the
+      # case kpathsea's build-script toolchain check exists to catch at
+      # runtime-deployment time — and precisely its documented bypass:
+      # "Building on a machine without TeX for deployment to one with it?"
+      # Without this, the job dies in kpathsea's build script ("no usable TeX
+      # backend") before checking anything — it has NEVER been green.
+      # The `build + test` job installs TeX Live and keeps the check armed.
+      KPATHSEA_SKIP_TOOLCHAIN_CHECK: "1"
     steps:
       - uses: actions/checkout@v6
       - uses: dtolnay/rust-toolchain@nightly

Cargo.toml

@@ -221,3 +221,4 @@ lto = false
 incremental = false
 debug = false
 strip = "symbols"
+

docs/SYNC_STATUS.md

@@ -64,7 +64,8 @@ Re-verify a row before planning on it (rule 1).
 | **R9** | Deep deferred families (`.bst`, xy-pic, mode-frame, …) | parked; several carry explicit "do NOT start". The `.bst` row's "`.bst` files *vendor macro definitions*" premise was **RETRACTED 2026-07-27** (`alpha.bst` has zero `Dbar`; the macro is `mathscinet.sty`'s) — it survives on label style / sort order / **field selection**, and the prerequisite is a corpus measurement of the `.bib`+`.bst`-with-no-`.bbl` population | **family** | [`DEFERRED_FAMILIES.md`](parity/DEFERRED_FAMILIES.md), and R9-BST below |
 | — | `\gls`/`\acrshort` in math mode (1705.10306) | **PARITY, blocked** on unrunnable Perl | — | do not chase; Open items |
 | — | **Streaming CORE (fragmented conversion)** | **IMPLEMENTED + PRODUCTION-PROVEN on `feat-streaming-xml-core`** (2026-07-30, one PR pending): stomach yield → spill/placeholder → pass 2 → recursive splice; `--streaming` + auto-when-doomed; byte-identical to eager (suite 1782/0, 8 sweep suites, 19.8 MB witness `cmp`-equal at 613,104,457 B). Leak fix halved eager (rust-libxml unlink never frees → `Node::free_subtree`, libxml **0.3.17 published**; eager 63.7→34.9 GB; streamed@24 GB-cap 16.97 GB/8:28). **The 131 MB Nasser witness CONVERTS: exit 0, 28.1 GB peak @ 48 GB cap, 2.66 GB well-formed XML, 1:10 wall** (node_boxes stale sweep, nested spills stay nested, spilled-id dedup, malloc_trim/mi_collect, token backstop ×200/byte). Remaining error: `{nowrap}` (issue 297 binding). **Perf consolidation 2026-07-31 (`perf-streaming-pass2-segment-coalescing`): 1:10:29 → 32:56 wall (−53 %), log −99.2 %, output md5-identical to base; soft-yield floor + flat spill + shared label index + streaming telemetry phases — see STREAMING_CORE_DESIGN "PERF CONSOLIDATION". Measured split: MathParse 41 % is the only remaining ≥40 % block.** | — | [`STREAMING_CORE_DESIGN_2026-07-29.md`](performance/STREAMING_CORE_DESIGN_2026-07-29.md) |
-| — | **POST half: page-major rendering** | **LANDED 2026-07-30 (PR #451)**: the driver was phase-major, so every page stayed alive at every boundary (~1.6 MB/page of `xmlDoc` overhead) — 614 MB core XML with `--splitat` went **80 GB/exit 137/0 pages → flat 15.98 GB/exit 0/40,201 pages/2.25 GB HTML/0 errors**. Same PR: whole-document `//X[pred]` answered by traversal (six queries had been FAILING silently → no MathML, no crossrefs, 0-byte HTML, exit 0), and an unanswerable query is now an `Error`. Markup verified against Perl 0.8.8 on a 10-page fragment. Remaining lever: Split still parses the whole document once (~16 GB here, ~70 GB for a 2.66 GB core XML) → the two-pass streaming split. | — | [`STREAMING_POST_DESIGN_2026-07-06.md`](performance/STREAMING_POST_DESIGN_2026-07-06.md) §2a |
+| — | **POST half: page-major rendering** | **LANDED 2026-07-30 (PR #451)**: the driver was phase-major, so every page stayed alive at every boundary (~1.6 MB/page of `xmlDoc` overhead) — 614 MB core XML with `--splitat` went **80 GB/exit 137/0 pages → flat 15.98 GB/exit 0/40,201 pages/2.25 GB HTML/0 errors**. Same PR: whole-document `//X[pred]` answered by traversal (six queries had been FAILING silently → no MathML, no crossrefs, 0-byte HTML, exit 0), and an unanswerable query is now an `Error`. Markup verified against Perl 0.8.8 on a 10-page fragment. | — | [`STREAMING_POST_DESIGN_2026-07-06.md`](performance/STREAMING_POST_DESIGN_2026-07-06.md) §2a |
+| — | **POST half: two-pass streaming split** | **IMPLEMENTED + WITNESS-PROVEN on `feat-streaming-post-split` (2026-07-31)**: the 131 MB witness's post half now COMPLETES on the 31 GB laptop — 2.68 GB core XML → **115,519 pages / 11 GB HTML / exit 0 / 37:31 wall / 17.4 GB peak** (`--splitat=subsubsection --max-memory=26000`; was: OOM during the whole-DOM parse, 0 pages). Mechanism: the last whole-DOM stage (Split's one-time parse — >26.6 GB for the witness's 2.68 GB core XML, OOM on a 31 GB laptop with 0 pages) replaced by a `TextReader` stream that spills pages as text + a pre-order Scan sweep; peak = open-ancestor chain + one content subtree. Byte-identical to the DOM split (guard `118_streaming_split_parity`); auto ≥ 1 GiB file input, `LATEXML_POST_STREAM_SPLIT` forces. Oversized in-memory handoffs also spill at the threshold so the single-invocation `.tex → .htm` flow streams too. Fixed en route: DOM split dropped inherited `xml:lang` (namespaced-attr read); **post parses now pass `XML_PARSE_HUGE`** — without it libxml2's per-document dictionary cap poisons the ID table from ~1.47 GB (the witness's 237,732 bogus `ID … already defined` reports, each id occurring exactly once — the rc4 mystery) and the parse dies at ~1.71 GB (`xmllint --stream` reproduces; `--huge` clears); libxml 0.3.18 (`expand_to_document` `default:`-prefix mint + 5 reader APIs — **needs crates.io publish before merge**). A mid-stream split failure now fails LOUD instead of falling back into a guaranteed whole-DOM OOM. Follow-up (perf): flat handoff serialization (50.9 % of witness core XML is indentation). | — | [`STREAMING_POST_DESIGN_2026-07-06.md`](performance/STREAMING_POST_DESIGN_2026-07-06.md) §3 |
 
 ### R3 mini plan — the remaining bibliography failures
 

docs/performance/STREAMING_POST_DESIGN_2026-07-06.md

@@ -1,14 +1,58 @@
 # Streaming post-processing for very large split documents — design + staged plan
 
-**Date:** 2026-07-06
-**Status:** foundation landed, **and the eager path's time bottleneck is now
-fixed** (CrossRef O(n²)→O(n), commit `4ec2587993`): a full `index.xml` run
-dropped **42 min 50 s → 2 min 18 s**. The two-pass streaming split is **deferred
-indefinitely** (user decision 2026-07-06): keep the simplicity of eager
-whole-DOM loading for as long as we can — the reporter's target has >64 GB RAM,
-so the ~21.6 GB peak is a non-issue for now, and the run is now fast as well as
-correct. **Revisit only if a <64 GB target appears.** The rest of this doc is
-the preserved design for that eventual work.
+**Date:** 2026-07-06 (§3 status updated 2026-07-31)
+**Status:** **the two-pass streaming split (§3) is IMPLEMENTED** —
+`latexml_post/src/stream_split.rs` + the front-end selection in
+`latexml_oxide/src/post.rs` (branch `feat-streaming-post-split`). The
+deferral's revisit trigger ("a <64 GB target appears") fired on 2026-07-31:
+the 131 MB witness's 2.68 GB core XML OOM'd a 31 GB laptop *during the
+whole-DOM parse* (>22 GB and >26.6 GB under two caps, exit 137, zero pages) —
+no configuration reaches success there. The implementation deviates from the
+§3.2 sketch in one structural way: instead of expanding page subtrees and
+running Scan inline (post-order), the stream **assembles each page's XML as
+text and spills it at page close**, then a separate **pre-order Scan sweep**
+re-parses one spilled page at a time — preserving Scan's order-sensitive
+semantics (SITE_ROOT from the first page, ancestor-before-descendant parent
+inference, `children` list order) byte-for-byte. Wrapper subtrees (non-page
+elements containing pages, e.g. back-matter) take a mini-DOM descent via
+`TextReader::expand_to_document`. Parity gate (§3.3):
+`latexml_oxide/tests/118_streaming_split_parity.rs` — byte-identical rendered
+pages vs the DOM split on a fixture exercising run adjacency, TOC
+suppression, `inlist="toc"` lookahead, wrapper descent, unnamed pages,
+template copies and inherited attributes. Gate: auto for file input ≥ 1 GiB
+(`LATEXML_POST_STREAM_SPLIT=1/0` forces; `LATEXML_POST_STREAM_THRESHOLD`
+tunes). The gate also exposed and fixed a latent DOM-split defect (inherited
+`xml:lang` copy silently skipped — namespaced-attribute read) and a
+rust-libxml one (`expand_to_document` minted a `default:` prefix onto
+default-namespace content; fixed in libxml 0.3.18 with three new reader
+APIs: `attributes_qname`, `value`/`is_empty_element`/`event`, `outer_xml`).
+**Witness proof (2026-07-31, this laptop, maxperf):** `flat_index.xml`
+(2.68 GB core XML) `--splitat=subsubsection --max-memory=26000` → exit 0,
+**115,519 pages, 11 GB HTML, 37:31 wall, 17.4 GB peak RSS** (split ~1 min at
+~0.6 GB; Scan sweep to ~3 GB; the render loop dominates and retains
+~150 KB/page — follow-up below). Baseline: the whole-DOM parse alone
+exceeded 26.6 GB with zero pages written.
+
+**Ceilings that remain (pre-existing, verified identical on the rc4 binary,
+2026-08-01):** a page (or an unsplit whole document) too large for libxslt
+fails its transform ("XSLT transformation failed: Unknown error applying
+stylesheet") — measured at a ~260 MB chapter page (`--splitat=chapter`) and
+at a 300 MB unsplit document; both split paths fail IDENTICALLY (parity
+holds in the failure case: same 5 of 6 pages written, same Error). The
+unsplit-giant case additionally stacks the whole-DOM RAM cost and the
+2 GiB `xmlBuffer` output ceiling. Splitting at a granularity that keeps
+pages libxslt-sized (section and below for this witness) is the supported
+mode at book scale. Note also the driver's standing signal-integrity wart:
+a failed post run writes an EMPTY destination with process exit 0 (the
+Error is in the log and the status code) — improving the CLI exit policy is
+a separate decision.
+
+Follow-ups (perf only, not correctness): (a) the render loop's ~150 KB/page
+retention (ObjectDB is ~3 GB of it; the per-page `DocOwnedNode` drip on
+id-dense math pages is the suspect — a `set_linked()` relink API in
+rust-libxml would remove it); (b) serialize the core→post handoff flat
+(`spill_flat`-style) to halve the streamed bytes — measured 50.9 % of the
+witness's core XML is decorative indentation.
 **Supersedes the resume half of** the original `HANDOFF.md` (large-index-database
 hardening). Companion to `docs/reproducers/` witness `~/scratch/nasser/index.xml`
 (614 MB, ~7M nodes, 40 000 one-equation sections, `--splitat=section`).

latexml_contrib/Cargo.toml

@@ -36,7 +36,7 @@ latexml_codegen = {path = "../latexml_codegen", version = "0.4.0"}
 latexml_engine = {path = "../latexml_engine", version = "0.5.0"}
 latexml_package = {path = "../latexml_package", version = "0.5.0"}
 rustc-hash = "2.0.0"
-libxml = "0.3.17"
+libxml = "0.3.19"
 regex = { version = "1.7.1", default-features = false, features = ["std", "perf", "unicode-case", "unicode-gencat", "unicode-perl", "unicode-script"] }
 # Optional — only compiled under `runtime-bindings`. Pure Rust, no native deps.
 rhai = { version = "1", optional = true, features = ["no_module", "no_time"] }

latexml_core/Cargo.toml

@@ -57,7 +57,7 @@ regex = { version = "1.7.1", default-features = false, features = ["std", "perf"
 log = "0.4"
 quote = { version = "1.0", optional = true }
 proc-macro2 = { version = "1.0", optional = true }
-libxml = "0.3.17"
+libxml = "0.3.19"
 # (replaced `dirs` 2026-05-17 — its single use was `dirs::home_dir()`,
 # which on Linux is exactly `std::env::var_os("HOME")`. On Windows
 # dirs falls back to `SHGetKnownFolderPath`; not relevant for us.)

latexml_engine/Cargo.toml

@@ -29,7 +29,7 @@ token-locators = ["latexml_core/token-locators"]
 once_cell = "1.17.0"
 regex = { version = "1.7.1", default-features = false, features = ["std", "perf", "unicode-case", "unicode-gencat", "unicode-perl", "unicode-script"] }  # DEP-10
 log = "0.4"
-libxml = "0.3.17"
+libxml = "0.3.19"
 chrono = { version = "0.4", default-features = false, features = ["clock", "std"] }
 unicode-normalization = "0.1.7"
 rustc-hash = "2.0.0"

latexml_math_parser/Cargo.toml

@@ -20,7 +20,7 @@ crate-type = ["lib"]
 token-locators = ["latexml_core/token-locators"]
 
 [dependencies]
-libxml = "0.3.17"
+libxml = "0.3.19"
 regex = { version = "1.7.1", default-features = false, features = ["std", "perf", "unicode-case", "unicode-gencat", "unicode-perl", "unicode-script"] }  # DEP-10
 log = "0.4"
 # dginev/marpa fork, published as `marpa-asf`; `package` alias keeps `marpa::`.

latexml_oxide/Cargo.toml

@@ -151,7 +151,7 @@ zip = { version = "8", default-features = false, features = ["deflate-flate2", "
 indexmap = "=2.14.0"
 tempfile = "3"
 log = "0.4"
-libxml = "0.3.17"
+libxml = "0.3.19"
 once_cell = "1.17.0"
 regex = { version = "1.7.1", default-features = false, features = ["std", "perf", "unicode-case", "unicode-gencat", "unicode-perl", "unicode-script"] }  # DEP-10
 glob = { version = "0.3", optional = true }