Stream Ptex a tile at a time behind ptex-rs's cache - #137
Open
doubleailes wants to merge 10 commits into
Open
doubleailes wants to merge 10 commits into
doubleailes wants to merge 10 commits into
Conversation
Upstream gained a tile-level read path and a thread-safe reader over it: `tile_layout` / `tile_info` / `get_tile` describe and fetch one tile of one face at one resolution, the mipmap helpers (`face_num_levels`, `res_for_level`, `is_res_stored`) answer which resolutions a face holds without any I/O, and `SharedReader` wraps all of it behind `&self` with an LRU of decoded pixels under a byte budget. That last one is the thing "Known incomplete work" said streaming Ptex was waiting on, and it deliberately lives upstream rather than in crust-assets: a `.ptx` is already a per-face mip pyramid addressed randomly on disk, so the cache belongs next to the reader that addresses it, exactly as the C++ Ptex library ships `PtexCache`. Nothing in crust changes with the bump — no `use` moved and the whole suite passes — because the rename in the same range (`ptex-rs` -> `ptex-rust`) touched the package and not the lib. Cargo names a renamed dependency by its key rather than by its lib target, so the key becomes `ptex`, which is both the lib name and what every `use` in the workspace already said. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LgzLA1uRAYXcxNWCBzSUCf
`CRUST_PTEX_STREAM=1` swaps the fully-decoded `PtexColor` for a `PtexStream` that pages one tile of one level of one face out of the `.ptx` under a byte budget (`CRUST_PTEX_CACHE_MB`, default 1024 to match `CRUST_TEX_CACHE_MB` and so OIIO's). Preloading stays the default and the oracle, and any file that declines to stream falls back to it — turning residency on can make a render slower, never break one. This is the residency half of the Ptex problem, and it is what retires `CRUST_PTEX_MAX_LOG2` as a *requirement*. That cap was never a tuning knob: the island's 2 576 238 faces are 4.58 GiB at 32x32 and 494 GiB at full resolution, so the cap is the thing that makes the island loadable, at the price of discarding authored detail permanently. A cache scales with itself instead. The cap still applies when explicitly set — that is what makes the two backends comparable at a resolution both hold — and streaming is otherwise uncapped, which is the point of it. The cache is upstream's, deliberately. `SharedReader` is a `&self` reader over an LRU of decoded blocks, which is the `PtexCache` equivalent "Known incomplete work" named as the fix, and it belongs next to the reader that already addresses the on-disk pyramid rather than beside the `.tx` cache, whose keys and layout answer a different question. So this module is a sampler over that reader: level selection, tile addressing, the colour decode, and a per-thread microcache. Four microcache slots, not the two `tiled::cache` keeps, and the difference is measured rather than assumed. A `.tx` grids once over the texture so a tap straddling a seam is rare; a `.ptx` grids per face, and the faces that get tiled are the large ones a streamed render lives in. A lookup on a four-tile corner needs four tiles for its four taps, and with two slots each tap evicts one the same lookup is about to want: that case hit 0.000 of 1600 fetches against 0.999 for a tap inside a tile. Four slots take the corner to 0.998 and leave the interior at 0.999. The invariant, measured end to end on the new sample scene at 16 spp with both backends capped alike: 0 of 57 600 pixels differ. Unit-level, the same equality holds texel for texel across four fixtures, every face, and caps from 0 to authored — including the capped-reduction arm, where a "tile" is the whole face. The `u8` decode table is pinned bit-for-bit against the scalar decode so that comparison can be an equality rather than a tolerance. Where the two are *supposed* to differ is the mip chain, and that is measured too rather than asserted away: a preloaded pyramid is reduced in linear light, a streamed one comes off disk reduced in the file's encoding. Convexity makes the streamed chain the darker one, by up to 0.147 on the tiled fixture and brighter by exactly 0. Cost, interleaved min-of-7 on the worst case this scene is built to be (two textured planes filling frame at depth 3, so nearly every shading call is a fetch): 0.123s preloaded against 0.342s streamed, with peak RSS 18.36 MiB against 10.27 MiB at a 4 MiB budget — and that 4 MiB render is bit-identical to the 1 GiB one, so the budget moves residency and not the image. `samples/ptex_quads.usda` is the repository's first scene to bind a `.ptx` at all, so `check_images.sh` now covers Ptex by its `samples/*.usda` glob. The four fixtures under `samples/textures/` are the reference C++ writer's, kept in one place so the unit invariant and the rendered one cannot be checked against different bytes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LgzLA1uRAYXcxNWCBzSUCf
`docs/ptex_streaming.md` is the measurements and the reasoning: the invariant and how it is checked, the mip-chain divergence and its direction, the four-slot microcache table, the cost, and the gaps. CLAUDE.md gets the switches, the A/B recipe, and — more to the point — its "Ptex does not stream" caveat rewritten, since that paragraph asked for exactly the upstream `PtexCache` equivalent this now consumes. One finding from building the sample scene is recorded in both, because it is a trap rather than a limitation of this change: a `crust:openpbr` material cannot bind Ptex at all. `inputs:surfaceMap` is consulted only for `UsdPreviewSurface` and `PxrDisneyBsdf`, so a Ptex material authored the native way renders on its constant `baseColor` — indistinguishable from `CRUST_PTEX=0`, and nothing warns. Unrelated to residency and left for its own change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LgzLA1uRAYXcxNWCBzSUCf
Two gaps a Moana island run on this branch exposed, neither visible on the sample scene. **The budget was per file.** `ptex::SharedReader` owns its cache, which is right for a library — a `.ptx` is a self-contained pyramid — but it means N textures opened at `CRUST_PTEX_CACHE_MB` each hold N times it. The `.tx` path never had this: every streaming texture there shares one `TileCache`, so the total is the budget by construction. On a stage that binds Ptex per element, which the island does across its 20, the default 1 GiB would have become tens of GiB — the feature whose whole purpose is to bound residency, unbounded in the texture count. `FileAssets` now divides one budget over the streamed textures as they arrive, floored at 4 MiB a share because a zero budget in `CacheOptions` disables caching outright and overshooting the total beats silently turning the cache off. An even split rather than OIIO's demand-driven pool: that needs a second cache here, which is the design "Known incomplete work" ruled out, and the property that matters holds either way. The test asserts the naive total *does* multiply first, so the sharing is testing something. **`--stats` was silent about Ptex.** It reported the `.tx` tile cache and nothing else, so a render could not say which Ptex backend produced it. That matters precisely on the island, where peak RSS is dominated by geometry and the SBVH build transient and the Ptex residency is buried inside a much larger number — the one figure a streamed run and a preloaded one differ by is the one the report did not print. The new block reports for *both* backends: `backend` (including `N streamed, M preloaded (fell back)`, which is not a bug but is when you want to be told), textures and faces, preloaded resident, and for a streamed run the live resident against the shared budget, the three fetch tiers and evictions — with the same "raise the budget" hint the `.tx` block earns when evictions run with the misses. The render invariant is unchanged: 0 differing pixels at a shared cap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LgzLA1uRAYXcxNWCBzSUCf
A full island render's log settles what the sample scene could not: it binds **3 618** `.ptx` totalling **5.98 GiB** preloaded, and they are Pareto-distributed. 167 of them hold 97% of the bytes, the top 25 hold 88%, the top two (`trunk0001`, referenced twice at 1.29 GiB each) hold 42%, and the median texture is under a kilobyte. That breaks the even split this branch shipped one commit ago. Dividing one budget over 3 618 readers hands the four textures holding half the bytes a 0.3 MiB cache each — smaller than one of their faces, so every read comes back `oversized` and nothing caches at all — while 3 451 sub-kilobyte files each sit on a slot they can never fill. And the 4 MiB floor meant to stop that multiplies out to **14.1 GiB**, worse than the preload it replaces. The fix was right about where the budget belongs and wrong about how to divide it. Admission is therefore per texture and priced against the alternative: a texture smaller than the cache slot it would occupy should just be preloaded. `PtexStream::preload_bytes` answers what preloading would cost from the parsed header alone — `face_infos()` carries every face's resolution and no pixel data — so the test is exact and free rather than a heuristic. Below `DEFAULT_STREAM_MIN_MB` (8 MiB, `CRUST_PTEX_STREAM_MIN_MB`), preload. On the island that admits ~39 readers at ~26 MiB each, a real working set, and preloads 0.54 GiB of small ones: **~1.5 GiB against 5.98 GiB**. A lower threshold wins on paper (1 MiB gives 1.18 GiB) but starves each reader, and a total is worthless if nothing caches. `MIN_PTEX_SHARE` drops to 1 MiB and is now only a backstop — admission is what keeps the count small, and raising the floor treats the symptom. One consequence worth knowing, and now documented in the scene itself: the sample fixtures are kilobytes, so `CRUST_PTEX_STREAM=1` alone preloads them and the A/B would compare nothing. The recipe takes `CRUST_PTEX_STREAM_MIN_MB=0`, and `--stats` names the backend either way. With it, the invariant is unchanged — 0 of 57 600 pixels differ at a shared cap. The island bucket table in the test is derived from the 3 618 real rows, so it reproduces the asset's count and total exactly rather than modelling a guess. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LgzLA1uRAYXcxNWCBzSUCf
A streamed island render says the backend line was wrong: it read "39 streamed, 3 579 preloaded (fell back)", and those 3 579 did not fall back — they were declined by the size threshold, working exactly as designed. On a production stage the mixed report is the *normal* case, so calling it a fallback reads as 3 579 errors. The two reasons are now counted and named apart: "preloaded under the size threshold" says the admission rule worked, "PRELOADED BECAUSE STREAMING FAILED" says a file is broken. Streaming being off is a third case, distinguished because a user who turned it on and got a bare "preloaded" needs to be told why. The run itself is the measurement every claim on this branch was missing — everything until now was fixtures and a synthetic scene. At 640x360 / 8 spp against the same build preloading: Ptex resident 5.98 GiB -> 0.61 GiB (9.8x) Ptex decode 84.8 s -> 14.0 s (Load assets 01:40.7 -> 27.3 s) Traverse RSS 47.34 -> 41.48 GiB (-5.86) peak RSS 51.28 -> 47.08 GiB (-4.20) Render 13:57.0 -> 14:06.8 (+1.2%) total 20:07.4 -> 18:58.7 Three things worth keeping. The render cost is **+1.2%**, not the 2.8x the sample scene shows — that scene is built to be the worst case and the island is traversal-bound; both are honest and the gap is the point. Streaming also made the render *start* faster, by 71 s of avoided decode, which is why the whole run finished sooner despite rendering slightly slower. And peak fell by less than residency did because peak lands at the SBVH build transient, not at texture load — the figure this feature moves is the traverse RSS. The cache held **3.28 MiB of a 2 GiB budget with zero evictions**. That is the mechanism, not a disappointment: the ray cone asks for a coarse level of each face at this framing, and a coarse level is a few texels, so streaming reads only the resolution the frame resolves — which is structurally what preloading cannot do. The budget is a ceiling rather than an allocation, so it is not to be lowered on that number; `evictions` running with the misses is the signal that it is too small, and it is zero. Also corrects the long-standing 4.58 GiB figure for the island's Ptex: that is the 32x32 base without the mip pyramid, and what a run actually holds is 5.98 GiB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LgzLA1uRAYXcxNWCBzSUCf
Contributor
PR Summary by QodoStream Ptex tiles through a bounded reader cache
AI Description
Diagram
High-Level Assessment
Files changed (18)
|
Contributor
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
Comment on lines
+409
to
+412
| fn base_res(&self, res: ptex::Res) -> ptex::Res { | ||
| let Some(cap) = self.cap else { | ||
| return res; | ||
| }; |
Contributor
There was a problem hiding this comment.
3. Streaming bypasses the default cap 📘 Rule violation ≡ Correctness
max_log2_from_env_opt returns None for both an absent and malformed CRUST_PTEX_MAX_LOG2, and PtexStream::base_res interprets that value as permission to retain each face’s authored resolution instead of applying the default log2 edge cap of 5. With CRUST_PTEX_STREAM=1, either an unset value or a typo such as CRUST_PTEX_MAX_LOG2=bad reaches uncapped streamed sampling, while preloading, fallback handling, and admission pricing use the 32×32 default.
Agent Prompt
## Issue description
The streaming Ptex backend treats both an absent and malformed `CRUST_PTEX_MAX_LOG2` as `None`, which `PtexStream` interprets as no resolution ceiling. Ensure missing or invalid values use the required default log2 edge cap of 5 so streamed sampling, preloading, fallback handling, and admission pricing remain consistent.
## Fix Focus Areas
- crates/crust-assets/src/ptex_texture.rs[469-495]
- crates/crust-assets/src/ptex_stream.rs[35-35]
- crates/crust-assets/src/ptex_stream.rs[278-284]
- crates/crust-assets/src/ptex_stream.rs[409-418]
## Recommended Fix
Pass the validated, defaulted value from `max_log2_from_env()` into `PtexStream::open` so both absent and invalid environment values resolve to `DEFAULT_MAX_LOG2`. Alternatively, change the parsing API or caller to substitute `Some(DEFAULT_MAX_LOG2)` for either case, while retaining the invalid-value warning and the existing per-face clamping logic.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+249
to
+252
| eprintln!( | ||
| "microcache rate: {:.3} inside a tile, {:.3} on a four-tile corner", | ||
| stats.micro_rate(), | ||
| corner_stats.micro_rate() |
Contributor
There was a problem hiding this comment.
4. Test metrics bypass configured logging 📘 Rule violation ◔ Observability
the_microcache_absorbs_most_taps and two other new tests emit measured diagnostics with eprintln! instead of a tracing macro. Running the integration suite with output enabled sends cache-rate, mip-divergence, and admission metrics directly through stderr outside the configured tracing subscriber.
Agent Prompt
## Issue description
Three new Ptex integration tests use `eprintln!` for diagnostic measurements instead of the required tracing-based logging path.
## Fix Focus Areas
- crates/crust-assets/tests/ptex_stream.rs[249-253]
- crates/crust-assets/tests/ptex_stream.rs[394-397]
- crates/crust-assets/tests/ptex_stream.rs[624-631]
## Recommended Fix
Remove these informational stderr emissions because the assertions already retain failure diagnostics, or replace them with appropriate `tracing` events if the measurements must remain observable.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+383
to
+390
| loop { | ||
| floats += w * h * 3; | ||
| if w == 1 && h == 1 { | ||
| break; | ||
| } | ||
| w = (w / 2).max(1); | ||
| h = (h / 2).max(1); | ||
| } |
Contributor
There was a problem hiding this comment.
6. Small textures stream when mips are off 🐞 Bug ≡ Correctness
PtexStream::preload_bytes always sums every halved mip level even when its mip field is false. With CRUST_PTEX_MIP=0, PtexColor allocates only level zero, so the inflated estimate can admit a texture to streaming although its actual preload size is below CRUST_PTEX_STREAM_MIN_MB.
Agent Prompt
## Issue description
Ptex streaming admission compares `preload_bytes` with the stream threshold, but `preload_bytes` always includes a complete mip pyramid. When `CRUST_PTEX_MIP=0`, the fallback `PtexColor` contains only its base level, so admission is comparing the threshold to a larger representation than the renderer would actually preload.
## Fix Focus Areas
- crates/crust-assets/src/ptex_stream.rs[373-392]
- crates/crust-assets/src/ptex_texture.rs[182-193]
- crates/crust-assets/src/lib.rs[452-455]
## Recommended Fix
Make `preload_bytes` stop after the base level when `self.mip` is false, matching `PtexColor::open_with`. Add an admission test that opens a stream with mipmapping disabled and verifies its predicted size against the no-mip preloaded texture.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+61
to
+63
| Err(_) => DEFAULT_CACHE_MB, | ||
| }; | ||
| mb * 1024 * 1024 |
Contributor
There was a problem hiding this comment.
7. Large cache settings can crash asset loading 🐞 Bug ☼ Reliability
cache_budget_from_env and stream_min_bytes_from_env accept any positive usize and multiply it by 1,048,576 without checked arithmetic. A sufficiently large accepted environment value overflows during configuration, panicking in checked builds or wrapping to an unintended cache budget or admission threshold in release builds.
Agent Prompt
## Issue description
The new Ptex cache and stream-threshold environment parsers validate only that the value is a positive integer, then perform unchecked conversion from MiB to bytes. Values that parse as `usize` but exceed the representable byte count can overflow, turning accepted configuration into a panic or an unintended small value.
## Fix Focus Areas
- crates/crust-assets/src/ptex_stream.rs[50-64]
- crates/crust-assets/src/ptex_stream.rs[104-119]
## Recommended Fix
Use `checked_mul(1024 * 1024)` after parsing in both helpers. On overflow, log the same validation warning style used for malformed values and return the corresponding safe default; add tests for oversized numeric input.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Review catch, and a real one: `rebudget_ptex` gave every streamed reader `max(budget / n, MIN_PTEX_SHARE)`, so whenever the admitted count exceeded the budget in MiB the assigned total exceeded the budget. At `CRUST_PTEX_CACHE_MB=8` with 39 admitted readers that is 39 MiB against 8 — the setting whose entire job is to bound residency, not bounding it. The floor looked like prudence: never hand a reader a share too small to be useful. But it fails in the direction that matters, overshooting *more* the smaller the budget gets, so it breaks down exactly where the budget is being taken most seriously. And it could not be fixed by lowering it — any positive floor multiplies by the reader count. So the same 1 MiB is now read as a **capacity** rather than a floor. At most `budget / MIN_PTEX_SHARE` readers may stream, and the budget divides *exactly* among them. Both properties then hold by construction: every admitted reader gets at least 1 MiB, and `n * (budget / n) <= budget` because integer division floors. A texture arriving past the cap is preloaded, counted as a third policy reason (`budget_full`) and reported by `--stats` with the one hint that fixes it — distinct from `below_threshold`, because these are textures big enough to want streaming that the budget could not seat. That is the honest answer rather than a compromise: there is no cache left to give such a reader, and a share too small to hold one block caches nothing anyway — upstream returns that read `oversized`. Admission is checked before the open, so a texture the budget cannot seat costs no file I/O at all. Costs the default path nothing: 1 GiB seats 1 024 readers and the island wants 39, so the measured island numbers are unchanged. It engages only when the budget is genuinely small. `the_total_budget_holds_when_readers_outnumber_megabytes` sweeps budgets of 1, 2, 3, 4, 8 and 64 MiB against four fixtures, asserting the assigned total never exceeds the budget, that no admitted reader is given less than a usable share, and that each still reads bit-identically to the preloaded oracle at whatever share it got — a bounded budget must cost detail-per-second, never correctness. End to end the invariant holds with the cap engaged too: forcing a mixed backend at a 1 MiB budget still renders 0 of 57 600 pixels different. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LgzLA1uRAYXcxNWCBzSUCf
Second review catch on the same invariant, and the sharper one. The per-thread microcache holds `ptex::PixelData` — decoded tiles the reader's byte budget does not know about — and retained them unconditionally. The case that bites is not an ordinary tile (128x128 at four channels is 64 KiB, four per thread is nothing) but a block upstream has *refused*: a face too big for the budget comes back `oversized`, deliberately uncached, and this put it straight back into residency, four slots deep, on every worker thread, entirely off the books. Reader eviction and re-budgeting could not release it, and `--stats` could not see it. Three changes, all needed: **A slot has a ceiling.** `MICRO_SLOT_MAX` is 256 KiB — above any real Ptex tile (256x256 at four channels) and below the whole-face reads that are the oversized case. A tile over it is handed to the caller and dropped, which is upstream's own rule: the microcache must never re-admit what the reader declined. **The allowance comes out of the budget.** `micro_reserve` is `threads * MICRO_SLOTS * MICRO_SLOT_MAX`, clamped to half the budget, and `FileAssets` subtracts it before dividing the rest among the readers — so the two halves of Ptex residency sum to `CRUST_PTEX_CACHE_MB` rather than the readers alone matching it. Below the clamp the slots shrink instead of the budget being exceeded, and once a slot falls under a real tile the microcache retains nothing: every tap goes to the reader, slower and still correct. **It is reported.** `--stats` prints `thread tiles / reserve` beside the reader's resident figure, because a number nobody can see is a number nobody checks against the budget. Measured at a 1 MiB budget on 4 threads: reserve 512 KiB, slot 32 KiB, and a whole-face read of the 1024x512 fixture is 512 KiB — retention grows by **0 bytes** over 16 900 taps, every one served by the reader. At the default 1 GiB the reserve is 4 MiB and a slot 256 KiB, so ordinary tiles are kept exactly as before and the sample scene still shows a 100% microcache rate against a 1020 MiB reader budget. `a_tile_larger_than_a_slot_is_never_retained` drives that case and asserts on the process-wide retained total, then checks the refused tile still reads bit-identically to the preloaded oracle — a bounded microcache costs re-reads, never texels. `the_microcache_allowance_comes_out_of_the_budget` pins the reserve arithmetic across budgets including the clamped ones. Known and now documented rather than hidden: a tile stays in its slot until that slot is reused, so dropping a texture does not immediately release its tiles. That is bounded by the reserve rather than by the texture, which is the property the budget is about; releasing on drop would need upstream to account for live handles. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LgzLA1uRAYXcxNWCBzSUCf
A `.ptx`'s stored mip levels were reduced before crust ever saw the file, in the file's own encoding; crust decodes Ptex by 2.2 afterwards. So a streamed coarse level is `g(mean e)` where the preloaded pyramid gives `mean g(e)`, and convexity makes the streamed chain the darker of the two by up to 0.147 on the tiled fixture. That is the same mismatch `crust:mipspace` guards against for `.tx`, and there the answer is refusal rather than a footnote -- for the reason the mismatch is dangerous rather than for tidiness: level 0 stays perfectly correct and only minification is wrong, so it never presents as a colour bug but as a filtering one. Describing a defect nobody can see is not a control. The streaming Ptex path accepted it with a doc note, which was an inconsistency with the project's own standard. So it is refused here too. `MipSpace::Linear` is the default and declines to stream a texture whose lookups could reach such a level, preloading it instead, reported as its own `backend` reason. The gate asks about the texture (`PtexStream::chain_is_exact`), not the switch, so the two configurations with no chain to get wrong still stream: `CRUST_PTEX_MIP=0` -- exact *and* uncapped, the base level being the bit-identical one -- and a texture whose every face is one texel under the cap. `CRUST_PTEX_STREAM_MIPSPACE=file` is the opt-in that takes the file's chain. It is what the C++ PtexCache does and what every measurement in docs/ptex_streaming.md was taken with, the island's included -- so the cost of the default is worth stating plainly: with a pyramid on, `CRUST_PTEX_STREAM=1` alone now streams nothing on a normal render, and the island's 5.98 -> 0.61 GiB needs the opt-in. Deliberate, and one variable to reverse. Neither remedy the review named is available here. Building the linear chain from streamed base tiles needs a second pyramid cache -- exactly the design "Known incomplete work" ruled out -- and would read level 0 to answer a coarse lookup, defeating streaming where the island uses it. The honest fix is upstream: a reader that reduces in a declared working space would let both backends share one chain and retire both gates. Two new tests pin the gate rather than the wording: every mipmapped fixture is declined and the same file admitted with the pyramid off (where streamed and preloaded then agree bit for bit at every footprint), and a texture capped to one texel per face is admitted. The divergence test stays, reframed as what the opt-in accepts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LgzLA1uRAYXcxNWCBzSUCf
The Moana section documented the resolution cap as the only answer to 494 GiB of authored Ptex. Streaming replaces it, but on this scene it takes two variables rather than one, and the second is the easy one to miss: a mipmapped .ptx is declined by default because its stored levels were reduced in the file's own encoding, so `CRUST_PTEX_STREAM=1` alone reproduces the preloaded numbers exactly. Adds the command line, the measured before/after, and why the default declines. Also corrects the "Ptex textures are still fully resident" limitation, which this no longer is -- the remaining restriction is the mip chain, not residency. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LgzLA1uRAYXcxNWCBzSUCf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integrates the tile-streaming API
ptex-rsjust gained, and tests it — on fixtures, on a new sample scene, and on the Moana island.CRUST_PTEX_STREAM=1swapsPtexColorfor aPtexStreamthat pages one tile of one level of one face out of the.ptxunderCRUST_PTEX_CACHE_MB. Preloading stays the default, the oracle, and the fallback.Why this could not be done before
"Known incomplete work" has been specific for a while about what was missing and where it belonged:
Upstream now ships exactly that:
SharedReaderis a&selfreader over an LRU of decoded blocks under a byte budget, andtile_layout/tile_info/get_tileaddress one tile without materialising its face. Socrust-assets/src/ptex_stream.rsis a sampler over that reader — level selection, tile addressing, the colour decode, a per-thread microcache — and not a cache.Unlike the
.txpath there is no conversion step.maketxexists because a.pngis not tiled or mip-mapped; a.ptxis already both. The missing piece was never a format.The bump itself is a no-op: no
usemoved and the suite passed unchanged. The package renamedptex-rs→ptex-rustin the same range, which only needed the dependency key to becomeptex.Measured on the Moana island
640x360 / 8 spp, against the same build preloading:
Load assetsTraverse primsRSSRenderThree things worth reading off that. The render cost is +1.2%, not the 2.8x the deliberately texture-bound sample scene shows — the island is traversal-bound, and both numbers are honest. Streaming also made the render start faster, by 71 s of avoided decode. And peak fell by less than residency did because peak lands at
Commit acceleration structure, the SBVH build transient; the figure this feature moves is the traverse RSS.The cache held 3.28 MiB of a 2 GiB budget with zero evictions. That is the mechanism, not a disappointment: at this framing the ray cone asks for coarse levels, and a coarse level of a face is a few texels — streaming reads only the resolution the frame resolves, which is structurally what preloading cannot do. The budget is a ceiling rather than an allocation, so it should not be lowered on that number.
CRUST_PTEX_MAX_LOG2also stops being load-bearing: unset now means uncapped.The invariant
At a resolution both backends hold, streaming changes where the texels live and nothing else.
samples/ptex_quads.usdaat 16 spp with both capped alike: 0 of 57 600 pixels differ. Texel for texel, across four fixtures, every face, ~1 400 sample points each, and caps from 0 to authored.Where they are supposed to differ is measured rather than asserted away: a streamed mip level comes off disk reduced in the file's encoding while a preloaded one is reduced in linear light, so convexity makes the streamed chain the darker — by up to 0.147 on the tiled fixture, and never brighter.
Three problems the island surfaced, all fixed here
Each was invisible on the sample scene and is now pinned by a test.
The budget was per file.
SharedReaderowns its cache, so N textures each held the full budget. The island binds 3 618.ptx, so the default 1 GiB would have become tens of GiB — a residency feature unbounded in the texture count. One budget is now divided across the streamed readers.An even split was still wrong. The island's Ptex is Pareto-distributed: 167 textures hold 97% of the bytes and the median is under a kilobyte. Splitting evenly gives the four textures holding half the bytes 0.3 MiB each — below one face, so nothing caches — and flooring the share instead multiplies to 14.1 GiB. So admission is per texture and priced against the alternative:
preload_bytescosts what preloading would from the header alone (no pixel I/O), and anything underCRUST_PTEX_STREAM_MIN_MB(8 MiB) is preloaded. On the island that is 39 streamed, 3 579 preloaded.--statswas silent about Ptex, so an island run could not say which backend produced it — and the first version of the block then called those 3 579 policy decisions a fallback, which reads as 3 579 errors. Both fixed: the block reports for either backend and names "preloaded under the size threshold" apart from "PRELOADED BECAUSE STREAMING FAILED".Also corrects a long-standing figure: the island's Ptex at the default cap is 5.98 GiB, not 4.58 GiB — that number is the base without the mip pyramid.
What's in the diff
18 files, +2 422 / −52.
crust-assets/src/ptex_stream.rs— the streaming backendcrust-assets/tests/ptex_stream.rs— 13 integration tests (the invariant, tile-seam taps, concurrency, the budget, admission)samples/ptex_quads.usda+ four.ptxundersamples/textures/— the repository's first scene binding a.ptx, socheck_images.shnow covers Ptex by its existing glob. Fixtures are the reference C++ Ptex writer's, copied fromptex-rs(MIT).crust-core/src/stats.rs— thePtexreport blockdocs/ptex_streaming.md— measurements and reasoning, plus the gapscargo fmt,clippy -D warningsand all 29 test targets pass.Worth knowing before merging
crust:openpbrmaterial cannot bind Ptex at all, streamed or not —inputs:surfaceMapis read only forUsdPreviewSurfaceandPxrDisneyBsdf. Unrelated to this change, found while writing the sample scene, documented and left for its own PR.🤖 Generated with Claude Code
https://claude.ai/code/session_01LgzLA1uRAYXcxNWCBzSUCf
Generated by Claude Code