Skip to content

pixelpipe: clearing the darkroom caches gives both pipes new keys, so they recompute - #1639

Merged
Jiyone merged 1 commit into
masterfrom
clear-darkroom-caches
Nov 14, 2026
Merged

Jiyone merged 1 commit into
masterfrom
clear-darkroom-caches

Conversation

@Jiyone

@Jiyone Jiyone commented Nov 14, 2026

Copy link
Copy Markdown
Member

Closes #1638.

Run > Clear darkroom pipeline caches recomputed nothing: it freed memory, and the darkroom kept the frames it already had.

Cause. dt_dev_pixelpipe_reset_all() flushes the cache, then rebuilds both darkroom pipes. Since 73ec46a31b the flush spares every line somebody holds, so that it never frees pixels under their holder.

  • The backbuffer keepalives hold the frames on screen, and the scopes hold their inputs.
  • Both pipes resync to the same keys.
  • The exact hit taken before a run (dt_dev_pixelpipe_cache_ref_host_entry_by_hash() in pixelpipe_hb.c) finds the frame on screen, and no module runs.

Reproduce on master: start with -d dev -d perf, open an image in the darkroom and click Run > Clear darkroom pipeline caches. No Started … pipeline recompute line follows, and no processed `…' line.

Change. Recomputing a held key would mean unpublishing its line while keeping it alive, and the cache cannot. The table holds one line per key, and raster masks, the raw detail mask and a run's final output are released by hash, so a second line under the same key would take their release. auto_destroy does not help either: dt_dev_pixelpipe_cache_get() and dt_dev_pixelpipe_cache_get_writable() still serve a flagged line as an exact hit while it is held. So the keys move instead.

  • dt_dev_pixelpipe_t gets a cache_generation, folded into the seed every key of the pipe derives from (_default_pipe_hash()): the base buffer's, the global and mask hashes, upstream_hash. Generation 0 leaves every key as it was.
  • dt_dev_pixelpipe_reset_all() bumps it before the rebuild, on both darkroom pipes and to the same value, so the full pipe still reuses what the preview computes.

The lines the flush spared stay valid for their holders under their own keys: the screen keeps the old frame until the new one is published, and nothing looks them up again.

Tested in the GUI under Xephyr (1600×1000), on a NEF with its XMP, with -d dev -d perf -d pipecache. An LD_PRELOAD shim activated the menu item (gtk_menu_item_activate() on the item found by its label).

  • Before: the flush freed 13 lines (1853 → 864 MiB) and spared six held ones. Both pipes then ended on the exact hit, in 7 ms and 1 ms, without running a module.
  • After: the preview reran its 15 modules from basebuffer, and the full pipe ran gamma on the preview's new lines. The old frame stayed on screen through 7 exposes until the new one replaced it.
  • A second click 300 ms into that recompute abandoned it in highlights, the usual kill-switch abort, and restarted it. A third click, at rest, freed the lines the first had spared, released since (1922 → 864 MiB), and recomputed everything again.
  • No reference count went below zero in any run.

ctest passes 32/32, and so do the static gates (layering, module boundaries, return types, list order, include guards and cycles, statelessness, unused and conditional includes). The touched files build without a warning. The build measured was 6a9005ef9c plus this change, in a working tree that also carried unrelated uncommitted colour-picker work, which touches no cache, hash or pipe code.

What it leaves.

  • A spared line, once released, waits for the LRU or the next flush, because the age-based collection keeps a line that was hit. Here that was 22 MiB: the old frames and the scopes' inputs.
  • After a clear, the darkroom shares no key with an export, a snapshot or a thumbnail of the same image for the rest of the session, the retouch memo included. A miss costs only the work being done anyway.

The finding and the mechanism are in doc/pipeline-cache.md, in the new section "Clearing the darkroom caches moves every key to a new generation".

🤖 Generated with Claude Code

… they recompute

Run > Clear darkroom pipeline caches recomputed nothing. dt_dev_pixelpipe_reset_all() flushes the cache and rebuilds both darkroom pipes, but the flush spares every line somebody holds, the frames on screen first, so as not to free pixels under their holder. Both pipes resynced to the same keys, and the exact hit taken before a run found the frame on screen.

Recomputing a held key would mean unpublishing its line while keeping it alive, and the cache cannot: the table holds one line per key, and raster masks, the raw detail mask and a run's final output are released by hash. dt_dev_pixelpipe_t gets a cache_generation instead, folded into the seed every key of the pipe derives from in _default_pipe_hash(), and dt_dev_pixelpipe_reset_all() bumps it on both darkroom pipes to the same value, so the full pipe still reuses what the preview computes. Generation 0 leaves every key as it was. The spared lines stay valid for their holders under their own keys: the screen keeps the old frame until the new one is published, and nothing looks them up again.

Measured in the GUI under Xephyr on a NEF with its XMP, the menu item activated by an LD_PRELOAD shim, with -d dev -d perf -d pipecache. Before, both pipes ended on the exact hit in 7 ms and 1 ms without running a module. After, the preview reran its 15 modules from basebuffer and the full pipe ran gamma, while the old frame stayed on screen through 7 exposes. A click 300 ms into that recompute abandoned it in highlights and restarted it, and a third at rest freed what the first had spared (1922 → 864 MiB). No reference count went below zero. ctest passes 32/32 and the static gates pass. doc/pipeline-cache.md records the measurement and the mechanism in a new section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jiyone
Jiyone force-pushed the clear-darkroom-caches branch from 38043a8 to 916d3bf Compare November 14, 2026 11:59
@sonarqubecloud

Copy link
Copy Markdown

@Jiyone
Jiyone merged commit 8e77b45 into master Nov 14, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

Task: Clear darkroom pipeline caches recomputes nothing, since the flush spares the frames on screen

1 participant