Skip to content

Latest commit

 

History

History
949 lines (791 loc) · 67.4 KB

image-mipmap-cache.md

File metadata and controls

949 lines (791 loc) · 67.4 KB

The image and mipmap caches

Verified against 42eca0e8fe on 2026-09-29.

When a thumbnail is regenerated, what releasing an image cache entry actually returns, and the race a duplicate runs against its own thumbnail.

Each finding below is dated with the commit that established it. A finding is only as good as its hash: before acting on one older than the code you are changing, re-measure it — and re-date it here when you do. Where an earlier version of a claim was wrong, that is recorded rather than quietly corrected: how a claim was wrong is usually the more useful thing to know.

Where the on-disk thumbnail cache lives, and the one way to ask

Established 2026-09-30 against aac2ee1321.

dt_loc_cachedir() is the answer, always — or dt_loc_default_user_cache_dir() (src/common/file_location.c) for the two callers that run before dt_loc_init() has set the global. Never g_get_user_cache_dir() directly, for three independent reasons:

  • It is the wrong directory on Windows. GLib returns FOLDERID_InternetCache there — %LOCALAPPDATA%\Microsoft\Windows\INetCache, the shell folder still labelled "Temporary Internet Files" — a shell-managed container for the browser cache, not a general-purpose per-user cache. Storage Sense empties it whenever it runs, of every file not written in the last week or so however recently it was read (measured below), so a thumbnail cache written there — written once, read for months — is deleted behind the user's back, which reads as "the disk cache never works on Windows". It also carries the Hidden and System attributes, so Explorer does not show it even with hidden files shown, and a thumbnail cache or a log written there is invisible to a user who goes looking (#1473). The default is now %LOCALAPPDATA%\cache\ansel on Windows and ~/.cache/ansel elsewhere, with an absolute XDG_CACHE_HOME winning on every platform. On Windows it sits beside the config folder %LOCALAPPDATA%\ansel (g_get_user_config_dir() is %LOCALAPPDATA% there), not inside it, as ~/.cache/ansel sits beside ~/.config/ansel: a cache nested in the config goes with every backup or deletion of anselrc and library.db. The branch first had ansel\cache; it was changed on 2026-10-07, before it shipped, so only test builds wrote there. %LOCALAPPDATA% is read from the environment rather than through g_get_user_data_dir(), which resolves to the same folder but takes XDG_DATA_HOME first — a variable that says where data goes and must not steer a cache.

    Measured 2026-10-07 against ea59057346 plus #1493, on one Windows 11 machine with Storage Sense on (StoragePolicy 01=1, temporary files 04=1) at its default cadence (2048=0, "when disk space is low"), by running it by hand — "Run Storage Sense now", recycle bin and Downloads set to "Never" — over the 118 files an older build had left in INetCache\ansel (backed up first):

    • every last-access time set back to 2026-07-01: it deleted the 109 files last written 2026-04-30 .. 2026-09-28 and kept the 9 written the day before (the log, two OpenCL kernels, six thumbnails);
    • the 109 restored with their write and creation times unchanged and their last-access time set to the minute before the run: it deleted the same 109 again and kept the same 9.

    So the criterion is the last write, somewhere between one and nine days old, and a recent read does not protect a file. The attributes were read on the same machine (INetCache: Hidden, System), with Explorer showing hidden files but not protected system ones (Hidden=1, ShowSuperHidden=0). Disk Cleanup was not tried.

    How this claim went wrong, then right. The first version of this section said Storage Sense clears the folder "by default", without measuring it. The review of #1493 then called that unestablished, because those same 118 files had sat there for five months, and an earlier commit of the PR took it out. The runs above put it back. The five months were the cadence, not Storage Sense sparing the folder: at "when disk space is low" it had most likely never run on that machine — which also means "by default" overstates how often it happens, not what happens.

  • It ignores --cachedir. That is how libs/textnotes.c came to build a download's path under g_get_user_cache_dir()/ansel/downloads while creating the directory under dt_loc_cachedir()/downloads: with --cachedir given, it made one directory and wrote into another, which did not exist.

  • It takes XDG_CACHE_HOME literally, relative included, and keeps its first answer. Established 2026-10-07 against ea59057346 plus #1493. g_build_user_cache_dir() (glib/gutils.c) copies the variable whenever it is non-empty, on every platform, with no g_path_is_absolute() test; g_get_user_cache_dir() memoises that for the life of the process. Read from the GLib source (main) and measured on Windows with a probe (GLib 2.90, MSYS2 UCRT64): with XDG_CACHE_HOME=relative-cache it answers relative-cache, and a second call after changing the variable answers the same. So the non-Windows default is built from g_get_home_dir() + .cache, which is what GLib answers for an unset or empty variable — the only cases left once an absolute one has returned. The first version of this branch had it wrong: it kept g_get_user_cache_dir()/ansel for Linux and macOS behind an "absolute only" guard, and a relative value walked around the guard and came back through that call as relative-cache/ansel. Its test could not see it: the empty-variable case ran first and froze GLib on ~/.cache.

tests/unittests/test_loc_cache_dir.c pins the XDG precedence and that the default is not a shell-managed temporary folder. Its relative-variable case runs first, before anything in the process can have asked GLib for its cache directory, so a return to g_get_user_cache_dir() would fail it on Linux. The thumbnail cache path itself is built on top of this by dt_mipmap_cache_get_filename(), which hashes the library's absolute path into the directory name so two libraries do not share thumbnails.

What the eviction write persists, and how it handles its files

Established 2026-09-30 against aac2ee1321.

dt_mipmap_cache_deallocate_dynamic() is the only place a thumbnail reaches the disk cache — on LRU eviction and, for everything still resident, from dt_cache_cleanup() at exit. Four things about it were wrong, and each one is a way to end up with a cached file that should not exist or cannot be removed.

The write predicate is about the entry's state, not its size. It used to be dsc->width > 8 && dsc->height > 8 and nothing else, which caught the skull and missed the other two cases:

  • An entry that was never generated carries DT_MIPMAP_BUFFER_DSC_FLAG_GENERATE over an uninitialised dt_alloc_align() block, at the mip's full dimensions — so the size test passed and the JPEG encoder was handed whatever the allocator returned. _generate_blocking() deliberately leaves such an entry behind whenever the shutdown atomic is raised mid-render. Only the async surface fetcher passes one (dt_view_image_get_surface_async(): the preview window, slideshow, map, print, studio capture), and it raises it on a resize, an image change, or when its widget goes; lighttable and filmstrip thumbnails go through dt_view_image_get_surface(), which passes none, and never abort. The next get of the same key renders over the entry again, so an aborted entry survives only when nothing asks that size of that image again — a preview window enlarged and then closed before its render ended. Eviction and exit wrote it out as a thumbnail of uninitialised heap at the box's full dimensions, which the next session then loaded as valid. The first version of this section called this the common case and said scrolling a filmstrip produced them; the filmstrip never aborts a render (measured below).
  • An invalidated entry (..._FLAG_INVALIDATE) unlinked its file and then, with no else between the two branches, immediately wrote the very buffer that had just been declared stale. The flag is set on a thumbnail only by dt_mipmap_cache_remove_at_size() with flush_disk TRUE, which unlinks a second time once dt_cache_remove() has returned, so no stale file outlived the call. What it cost was a JPEG encode and write per removed size, and a window: dt_cache_remove() releases the cache lock before that second unlink, and a get of the same key in between runs the allocate callback, which reads the disk cache — the stale file — under that lock. Read from src/caches/cache.c against 2fadf8a366, not measured. The first version of this section said that with flush_disk FALSE the stale thumbnail was persisted; with flush_disk FALSE the flag is never set, and the write is the ordinary eviction write.

An entry still flagged GENERATE is absent to a DT_MIPMAP_TESTLOCK get as well. Keeping it off the disk is not enough on its own: unless the embedded JPEG is used (the default for an unedited raw), _init_8() builds a size from the next larger one a TESTLOCK get finds, and that get handed back an aborted entry's buffer at the box's full dimensions. The smaller size was downsampled from nothing and became an ordinary thumbnail, shown and written to disk. A TESTLOCK get now releases such an entry and answers as if it were not there, which is also what the other caller, the darkroom's "is the cache ready for this size" test, means by it.

Measured 2026-10-23 against 2fadf8a366 plus #1495, with a program linked against libansel that requests size 2 of one NEF with the shutdown atomic already raised, then size 1 (lighttable/embedded_jpg=0, --disable-opencl, fresh --configdir, --cachedir and --library):

  • 2fadf8a366: a TESTLOCK get of size 2 returns 1440×900 of zeros; size 1 is downsampled from it ("generate mip size 1 for image 1 from mip size 2"), 720×450 and all zeros; both are written to the disk cache, black.
  • With the eviction write fixed only: size 2 is withheld ("never generated"), and the black size 1 is still written.
  • With the TESTLOCK change too: the get returns no buffer, size 1 is rendered from scratch (300×450, mean byte value 127) and that is what is written.

The buffer read as zeros here, a fresh allocation; nothing guarantees that in a long session. With ansel-generate-cache and lighttable/embedded_jpg=0 (one size rendered from scratch, two downsampled from a larger one), the three files are bit-identical with and without the change.

g_unlink() must come after fclose(), in both halves. On Windows DeleteFileW refuses a file that any handle still holds open without FILE_SHARE_DELETE, which the CRT's fopen never sets. The read path unlinked a file it had judged corrupt while its own read handle was open, and the write path's error branch unlinked while holding a handle of its own — so on Windows a bad disk-cache entry could never be cleared: it was re-read, re-rejected and re-logged on every start, permanently. This is invisible on POSIX, where unlinking an open file is ordinary.

The output file was opened twice. dt_imageio_jpeg_write() opens, writes and closes the path itself; the caller also opened it "wb" purely to test writability, which truncated it before the free-space check ran and held a second handle across the whole write. That outer open is gone — the function's return value is the only answer needed.

A free-space probe that cannot answer must not veto the write. It ran statvfs() on the output file (so the file had to exist first, which is why the redundant open was there) and treated an unanswerable probe as "no room". src/win/statvfs.c built a drive root out of path[0], which is right for C:\... and wrong for a UNC share, a redirected %LOCALAPPDATA% or a drive mounted as a folder — and there every thumbnail write failed silently. The probe now runs on the directory before anything is created, returns TRUE when it cannot answer, and the Windows shim uses GetDiskFreeSpaceExW, which takes any directory rather than a drive root. Its documentation asks a UNC path to end in a backslash, and the cache directory does not, so on a UNC share the probe may still fail — now without consequence, since the write goes ahead. Not tried on Windows. That shim reports its counts in bytes with f_frsize == 1: every consumer computes f_frsize * f_bavail, which is satisfied exactly, but the true cluster geometry is gone — do not add a consumer that wants it without a second query.

The skip is logged under -d cache ("is not written to the disk cache: never generated | a skull"), because how often an aborted render leaves an entry behind is a question only a real session answers.

Measured 2026-10-23 in the GUI against 2fadf8a366 plus #1495: twelve hardlinks of one NEF, lighttable/embedded_jpg=0, fresh --configdir, --cachedir and --library, -d cache, aborts counted by their DT_IMAGEIO_ABORTED ("error 11") line. Scrolling the lighttable and the filmstrip for 27 minutes at five thumbnails per row aborted nothing. The preview window opens at 350×350, a size the lighttable usually holds already, so it shows at once and renders nothing; enlarging it asks for size 2. Opening it on an image the window had not shown yet, enlarging it and closing it within a second, on three images, at three thumbnails per row:

  • With master's src/caches/mipmap_cache.c: 14 aborts; size 2 of images 2, 6 and 8 never completed, and all three were written at exit as 1440×900 JPEGs of zeros, where a real render of that image is 600×900; size 1 of images 6 and 8, which the window asked for first and which was aborted too, was written as 720×450 of zeros, where a real one is 300×450.
  • With #1495: 11 aborts; the four entries they left behind (size 2 of images 2, 4 and 6, size 1 of image 6) are logged "never generated" and not written, and the 13 files written are all real images.

Neither session built a size from an aborted larger one, so the TESTLOCK half is measured only by the program above.

Measured 2026-10-23 against 2fadf8a366 plus #1495, with ansel-generate-cache --min-mip 0 --max-mip 2 on one NEF, fresh --configdir, --cachedir and --library, --disable-opencl: the three thumbnails are written and the skip line appears 0 times; a second run leaves them byte-identical; a thumbnail truncated to 0 bytes is reported ("Reason: empty file"), unlinked and rewritten identically; and the three files are bit-identical (sha256) to the ones a build of src/caches/mipmap_cache.c from 2fadf8a366 writes for the same image.

A thumbnail read from the disk cache is not written back

Established 2026-10-24 against 2fadf8a366.

dt_mipmap_cache_deallocate_dynamic() writes a thumbnail to the disk cache on LRU eviction and, for everything still resident, from dt_cache_cleanup() at exit. The read path used to leave no trace on the entry (flags = 0), so that write could not tell a thumbnail read from its file from a fresh render, and re-encoded it at every session, at database_cache_quality (89 by default). The comment above the write said "Don't write existing files as both performance and quality (lossy jpg) suffer", and the FIXME under it that it did anyway.

The entry now carries DT_MIPMAP_BUFFER_DSC_FLAG_ON_DISK from the read until new pixels are written into it, and the write is skipped while it does. Only dt_mipmap_cache_swap_at_size(), the darkroom's copy, writes new pixels into an entry read from disk, and it clears the flag; generation runs only on an entry flagged GENERATE, which a read never leaves, and dt_mipmap_cache_write_get() has no caller. A file deleted from outside behind a resident entry so flagged is not written again at eviction; the next session renders it.

Measured 2026-10-23 and 2026-10-24, with a program linked against libansel that asks sizes 0 to 2 of one NEF and quits, run in a loop on one cache (lighttable/embedded_jpg=0, --disable-opencl, fresh --configdir, --cachedir and --library):

  • Before the flag (2fadf8a366, 30 sessions, and a branch writing the same bytes session for session, 150 sessions): every session from the second on read the three files and wrote them back. Sizes 0 and 1 reached a fixed point: the file stopped changing after session 8 and session 12, at 54.6 and 54.9 dB PSNR from the first file. Size 2 did not: one 8×8 block (x 352–359, y 736–743 of 600×900) gained one level per session, 155 to 254, and stopped only once near white, after about 100 sessions.
  • With the flag (2fadf8a366 plus this change): the three files were byte-identical over 30 sessions, and the first one byte-identical to 2fadf8a366's; an entry read from disk and then overwritten through dt_mipmap_cache_swap_at_size() was written back with the new pixels.

A failed thumbnail is retried once, and a skull is never a source

Established 2026-09-30 against aac2ee1321; re-measured 2026-10-25 on top of the eviction rewrite and the embedded-JPEG mode snapshot (8f8b83d8bf). The write lock, the full-size reload and the DT_MIPMAP_TESTLOCK status filter measured 2026-10-25 against 51de724f6e.

_init_8() and _init_f() report a failure as a zero-sized image. If _generate_blocking() cleared DT_MIPMAP_BUFFER_DSC_FLAG_GENERATE on it, the failure would be latched on the spot: every later request takes the "already in cache, no I/O needed" branch and gets the 8×8 skull back, and dt_view_image_get_surface() reports an 8×8 buffer as DT_VIEW_SURFACE_OK — so the widget commits it and never asks again. One transient fault would cost an image its thumbnail for the whole session, clearable only through "Purge selected thumbnails from cache".

The retry is bounded, and the descriptor must be restored before it runs. GENERATE|RETRIED means "failed once, try again on the next request"; RETRIED alone means "failed twice, stopped". The trap: a failed attempt leaves the skull's 8×8 in dsc->width/height, and _init_8() opens with if(size >= DT_MIPMAP_F || dsc->width < 16 || dsc->height < 16) return DT_IMAGEIO_OK; — so re-entering with those dimensions does nothing at all, silently, and the skull would be kept as a thumbnail that generated. _generate_blocking() puts back what dt_mipmap_cache_update_buffer_addresses() asks for first.

The retry runs under a write lock. A 'r' get of an entry that is not in the cache yet comes back write-locked (the allocator has to fill it), but one that is already there comes back read-locked — and an entry left at GENERATE|RETRIED is there. Rendered as it came, the retry wrote the pixels, the descriptor and the flags under a read lock, with any number of other readers in the same entry. dt_mipmap_cache_get_with_caller_and_shutdown() now takes such an entry again with 'w' before _generate_blocking(), which re-reads the flags, so a second reader that waited behind the first finds nothing left to do. Measured with a probe on the lock's writer field, in the GUI: a fresh library of two copies of one NEF, one overwritten with random bytes after import, embedded_jpg=0, every thumbnail refreshed twice through dt_thumbtable_refresh_thumbnail_real(…, -1, 1):

51de724f6e with the write lock
generations under a read lock 2 0
generations under a write lock 4 6
will try again / failed … twice 2 / 2 2 / 2

The two read-locked generations were both retries (flags GENERATE|RETRIED), at mip 0 and at the full size.

The full size keeps its count across the reload. dt_imageio_open() reaches dt_mipmap_cache_alloc(), which calls dt_mipmap_cache_update_buffer_addresses(), which resets the flags to GENERATE alone — on an allocation failure too. A retry that got that far forgot it was the retry, failed, and was granted another, on every request. _generate_blocking() carries RETRIED over the call. From the code, not measured: a random-bytes file fails in the loader before any allocation, so the repro above does not reach it.

A skull is not a downsample source. _init_8() fills a size from the first larger size present in RAM, through a DT_MIPMAP_TESTLOCK get. An entry that failed twice is no longer flagged GENERATE, and has the dead image as its buffer: returned to that get, it would fill the smaller mips, marked as generated without ever attempting their own generation. So the DT_MIPMAP_TESTLOCK get returns neither an entry flagged GENERATE nor one whose dsc->status is not DT_IMAGEIO_OK — every failure path of _init_8() returns a non-OK status, and the full size stores the loader's. Its other caller, the darkroom's preview handover, reads an entry dt_mipmap_cache_swap_at_size() has just written with DT_IMAGEIO_OK, and that swap also clears RETRIED: new pixels leave no failure to retry.

An earlier account of this failure was wrong, and the measurement is the reason to record it. It said the propagated skull, "being larger than 8×8, then cleared the 'don't write skulls' guard on eviction and reached the disk cache, where it survived restarts". It does not. dt_iop_flip_and_zoom_8() never upscales, so the copy comes out 8×8 as well (generate mip size 1 for image 1 from mip size 2 (8x8->8x8)) and the guard holds — measured on aac2ee1321 with a corrupt source: 2 mips filled from the skull, 0 JPEGs written. The harm is the latching of the smaller sizes, not disk contamination.

Reproducing this needs a failure that happens after the downsample loop, which rules out the obvious repro: !input_exists returns early, above the loop. Replace an imported file's contents with random bytes instead, and set lighttable/embedded_jpg=0 — otherwise use_embedded_jpg is TRUE for an unaltered image and the loop is skipped entirely.

Measured that way (a 200 kB random-bytes NEF, ansel-generate-cache --min-mip 0 --max-mip 2 -d cache, --disable-opencl, fresh --configdir, --cachedir and --library):

8f8b83d8bf with the retry
could not process thumbnail 1 3
will try again — 4
generate mip size … from mip size 2 0
JPEGs on disk 0 0

Without the retry, one generation is attempted and the two smaller sizes are filled from its skull; with it, each size attempts its own. Every failed size still carries GENERATE in this run, so it does not exercise the status filter: ansel-generate-cache asks once per size and never reaches a size that has failed twice. With the write lock and the status filter on top of 51de724f6e, the same run gives the same four lines: 3, 4, 0, 0.

Mipmap invalidation is explicit, not hash-driven

Found 22f623c0be, 2026-06-25.

The mipmap cache get path (_generate_blocking in caches/mipmap_cache.c) does NOT compare history_hash vs mipmap_hash to detect staleness. Regeneration only happens after an explicit dt_mipmap_cache_remove(imgid, TRUE). (This file gave that call a leading cache argument until 2026-09-29; the cache is file-static and the documented call did not compile — caches/mipmap_cache.h:203.)

Every operation that mutates an image's history/development MUST explicitly:

  1. dt_mipmap_cache_remove
  2. Refresh the cached image metadata so history_items is correct (_write_mipmap_to_disk uses img->history_items > 0 as the "altered" flag for the embedded-JPEG-vs-raw decision)
  3. dt_thumbtable_refresh_thumbnail

The darkroom/paste path does this via dt_dev_history_notify_change (dev_history.c). Paths that write history straight to DB (XMP load, dt_image_set_flip) bypass it and need the fix pattern: dt_image_cache_get_reload, then remove mipmap + refresh thumbnail.

Do NOT refresh the filmstrip from darkroom write paths — it competes with the realtime main preview pipeline. Lighttable ops may refresh both.

dt_mipmap_cache_remove() drops the THUMBNAILS, never the decoded raw. Its loop stops at DT_MIPMAP_F, and dt_mipmap_cache_remove_at_size() refuses DT_MIPMAP_F/DT_MIPMAP_FULL outright, so those two — the unprocessed input, RAM-only, every disk write being gated on mip < DT_MIPMAP_F — are reachable only through dt_mipmap_cache_remove_all_sizes(). That is the right default for the list above: a development change does not invalidate the decoded raw, and dropping it on every history commit would re-read and re-demosaic the file per slider tick. The one DT_MIPMAP_FULL entry it does drop holds no raw: a load that failed (see "A purge tries a failed load again" below).

An image LEAVING the library is the other case, and the only caller of the all-sizes form. Its input buffer otherwise outlives the row, with nothing but memory pressure to reclaim it, and basebuffer — which slices that buffer — is handed the stale entry when the image comes back on Ctrl+Z. It reports invalid cache entry size 0 for module basebuffer, the mipmap get path answers with an 8x8 husk, and no later render replaces it. Only a developed image shows this: an unaltered one is drawn from the embedded JPEG and never asks for the input at all, which is why the symptom reads as "one broken thumbnail" rather than as a cache bug.

The embedded-JPEG mode is what the cache was told, not what conf says

Established 2026-10-24 against 2fadf8a366.

The cache never reads lighttable/embedded_jpg. _init_8() decides between the embedded JPEG and the raw from dt_mipmap_cache_settings_t.embedded_jpg, which dt_mipmap_cache_set_settings() sets at startup and on DT_SIGNAL_PREFERENCES_CHANGE. The lighttable's thumbnail menu (_jpg_combobox_changed(), gui/actions/display.c) changes the mode without that signal, so it tells the cache itself, and does so before it purges: a thumbnail regenerated between the purge and the change would be rendered in the old mode and cached.

The purge drops the thumbnails on disk too (flush_disk = TRUE). A file in the disk cache does not record which mode rendered it, and the disk read in the get path takes it as it is.

Measured with -d cache on a copy of a 51-image collection, switching from "Always process the RAW" to "Always use embedded JPG". When the menu wrote only the conf key, the purged thumbnails were rendered again from the raw (fetch … float32 … from original file I/O, none from embedded jpeg), for more than four minutes, one pipe at a time. With the cache told, 21 came from the embedded JPEG, one image with none became a skull (embedded JPEG mode forbids raw processing), no raw was decoded, and all were done 1.1 s after the purge.

A render in flight across the change is not kept

Established 2026-10-25 against a8d4f8e203.

The purge cannot reach a thumbnail that is being rendered. _generate_blocking() runs under the entry's write lock, and dt_mipmap_cache_remove_at_size() takes the entry with dt_cache_testget(..., 'w'): a locked entry is neither flagged nor removed, only its file on disk is unlinked. The render cannot be stopped either. The lighttable asks for its thumbnails through dt_view_image_get_surface(), which passes no shutdown flag, and the refresh after the purge only orphans the job, whose cache entry the render still fills. A worker waiting on dt_pipeline_threadsafe_mutex() already holds its entry and has already chosen the raw, so a change usually finds several renders in flight.

_generate_blocking() therefore reads the mode once, hands it to _init_8(), and reads it again when _init_8() returns. If it changed, the result is dropped the way a stopped pipe's is: the descriptor is restored, the buffer invalidated, DT_MIPMAP_BUFFER_DSC_FLAG_GENERATE kept, and the next request renders it in the new mode (-d cache prints rendered in embedded JPEG mode 0, now 2: not kept). Waiting for the renders in the purge instead would hold the GUI thread for as long as the slowest of them, more than a minute here. The renders still run to their end; only their result is discarded.

Measured with -d cache on the same copy, a session started in "Always process the RAW" on an empty cache, switched from gdb as the menu does (conf, dt_mipmap_cache_set_settings(), purge of the 51 images, thumbtable refresh) once the first raw thumbnail was done. Both runs had seven DT_MIPMAP_0 renders in _generate_blocking() at the switch. Without the check, all seven finished from the raw 108 to 131 s after the switch and were kept, and the lighttable then read each of them from RAM (will skip disk I/O, found in RAM cache). With it, all seven finished 14 to 104 s after the switch, were dropped, and were rendered again from the embedded JPEG 0.05 to 0.66 s after the drop; no raw thumbnail was kept.

Two windows remain, of the same kind and present before: a change between the second read and the end of _generate_blocking(), a few microseconds, and an entry a reader holds read-locked while the purge passes, which any purge skips.

An RGB file in an embedded mode shows the image before any edit

Established 2026-10-25 against 831833157f with ignore_history added on top.

What an embedded preview shows of a raw is the camera's picture, before any edit. A PNG, a TIFF or another RGB file usually has no embedded preview, and in an embedded mode (1 or 2) _init_8() then renders it through the pipe with ignore_history set in dt_imageio_export_with_flags(): the history end is set to 0 after the history is read, so every node gets its module's default params and enabled state -- the same picture an embedded preview would be. A raw (may_be_raw) is never rendered this way: its defaults include the auto-applied presets a raw needs to look like a photo.

Measured with ansel-generate-cache at mip 2 in mode 2, on a 640x427 gradient PNG given a 29-item history (exposure, filmic, crop, flip): the thumbnail's mean went from 0.809 -- the same as in mode 0, which renders the edits -- to 0.514, against 0.499 for the source file.

The darkroom copies a frame into the thumbnail only when it carries no overlay

Found ea59057346, 2026-10-07.

At the end of a darkroom run, dt_dev_darkroom_pipeline() (develop/develop.c) copies the backbuffer into the mipmap cache (dt_dev_resync_mipmap_cache(), then dt_mipmap_cache_swap_at_size()) whenever the run has the preview's output size. Until 07aea1a673 (2026-06-19) only the preview pipe did this. Since then the main pipe does it too at zoom == fit, where both pipes render the same frame. 8f7c553fb6 (2026-06-21) kept mask previews out of that copy, but not the overlays that only the main pipe draws:

  • the clipping and raw clipping indicators (overexposed and rawoverexposed disable themselves on any other pipe);
  • soft proof and gamut check (colorout applies the proofing mode to the FULL pipe only).

After an edit at fit, the main pipe finished after the preview pipe and overwrote its clean copy. The disk cache then kept the overlay across restarts. A user reported clipping colours in a thumbnail.

requested_overlay now gates the copy. Like requested_mask_preview, it is a snapshot taken before the run.

Measured with -d perf -d cache on a 5208x3912 raw at fit. A copy prints will fit a mip size 1 followed by is synchronized from pipeline. A lone will fit comes from _preview_pipe_finished() (views/darkroom.c) answering the finished signal of a run that copied nothing.

  • Before (-d cache alone): two copies per edit, about 50 ms apart (106.91/106.97 s, 113.21/113.27 s, 114.49/114.69 s, 124.49/124.54 s).
  • After, an edit with an overlay on: the pipes run preview, main, preview. The main run copies nothing and the preview run after it does (23.20/23.28 s, 24.77/24.84 s, 30.61/30.68 s, 31.82/31.87 s).
  • After, toggling the overlay: the toolbox resyncs the main pipe alone, in 1-34 ms runs. Each switch-off produces a copy (25.93, 34.86, 40.60, 43.71 s) and each switch-on produces none (27.29, 39.71, 42.31 s).
  • After, an edit with no overlay: both pipes copy, as before (36.76/36.80 s).

The thumbnails stayed clean. Two limits, read from the source:

  • The snapshot reads the GUI state when the run starts, while the modules read it when they commit, earlier. An overlay switched off between the two lets one overlay frame through. The resync that the switch-off queues replaces it on the next run.
  • A thumbnail polluted before the fix stays until something rewrites it. Opening its image in the darkroom does: the first run on entry copies a clean frame.

The second size in a synchronized line is the entry's size before the downscale. It reads 720x450 for an entry that was just allocated and the fitted size otherwise, so it does not mean the image was stretched.

A failed database read is not a missing image

Measured 2026-10-26 on b0f10525fd (the probe on f6f82cbf45, which differs only in the exit path), and 2026-10-27 with this change on top of b0f10525fd. RelWithDebInfo, clang 19. A probe linked to libansel.so, with the library on disk and a second SQLite connection taking BEGIN EXCLUSIVE; and the lighttable on Xephyr with 40 images, driven by an LD_PRELOAD shim that calls the real actions on the GUI thread, with an outside Python writer and /proc/locks sampled every 2 ms.

dt_image_repository_load() tells three outcomes apart:

sqlite3_step() img stderr
SQLITE_ROW filled from the row —
SQLITE_DONE: no such row, the image was removed id = UNKNOWN_IMAGE, the rest as it was —
anything else: SQLITE_BUSY, an I/O error as it was one line

An id left at UNKNOWN_IMAGE is not a passing state. The image cache keeps the entry, and dt_image_cache_get() answers NULL for that image for as long as the entry stays cached; the thumbnail path reads that NULL as "File not found" (see "Why a thumbnail is a skull" below), and "Purge selected thumbnails from cache" does not help, since it drops thumbnails, not image cache entries: on b0f10525fd, where a failed read did write the id, every visible thumbnail (20) failed to generate again after it. So a failed read leaves the struct alone: dt_image_cache_get_reload() reloads a live entry in place, with no dt_image_init(), and the entry keeps the last values that did come from the database.

The allocate path is not covered. dt_image_cache_allocate() runs dt_image_init() before the load, so a read that fails there still leaves an invalid entry: in the probe, an image first asked for during a 2 s lock stayed NULL after it.

When a read can fail. Ansel's connection holds a SHARED lock on library.db almost all the time — 40.0 s out of 40.1 s in the lighttable — because the cached statement of dt_image_repository_load(), stepped to SQLITE_ROW and not reset, keeps its read transaction open. While it does, no read can fail with SQLITE_BUSY, and an outside writer never gets its EXCLUSIVE lock. A load that finds no row runs the statement to SQLITE_DONE and ends the transaction; until the next successful load, an outside lock fails every read. Removing an image from the lighttable did not open that window: the lock went to RESERVED for the removal and back to SHARED within 3 ms. In the window (an absent id read, then an 8 s EXCLUSIVE lock, then two stars on 40 images), all 40 reloads failed; their entries stayed valid and the purged thumbnails regenerated. The rating itself was lost, as every write of Ansel's is while that lock is held.

Ansel's writes fail as well, and silently. While an outside writer waits for the lock (PENDING), every write of Ansel's fails at once and nothing is shown: with a client committing in a loop, one of two 40-image rotations was lost; with an 8 s lock attempt, a two-star rating on 40 images.

There is no busy timeout, on purpose. SQLite does not call the busy handler when a connection already in a read transaction asks for the write lock, since that could deadlock, and it is Ansel's usual state: with sqlite3_busy_timeout(5000), the same rotation and the same rating were lost. The timeout does act in the window above, where the rating then waited for the lock and was kept, but on the GUI thread: it stalled 4.57 s for an 8 s lock, and 26.61 s for a 30 s one with three retries of the read on top. And the allocate path runs under the image cache's global lock: during such a wait, another thread asking for an image already in cache waited 14.89 s.

Testing it. A second connection in the same process locks the file as another program would; it takes no second process. It does take a library on disk, since on ":memory:" each connection has a database of its own. And nothing of Ansel's may hold a read transaction when the lock is taken, or BEGIN EXCLUSIVE itself fails: one successful dt_image_repository_load() is enough. tests/unittests/test_image_repository.c pins both cases. The absent one starts from a struct that looks loaded, since from UNKNOWN_IMAGE the assertion would hold without the write. The success path reads conf (dt_image_film_roll_name()); test_image_cache_flags_writeback.c, whose fixture initialises it, covers that.

What a caller does with the NULL. Counted on 27a9aacb2b and 1442b277c6, with the same result, and on top of 1442b277c6 with the change that made every caller test the answer. dt_image_cache_get() and dt_image_cache_get_reload() answer NULL for an id ≤ 0, for an image whose row is gone, for an image whose first read failed — while other queries, group members and duplicates among them, still list that image — and for an entry that could not be allocated (see "Image cache" below). Before the change, 29 of the 97 calls outside caches/image_cache.c dereferenced the answer untested. A script flags every assignment whose variable is dereferenced or passed on before a NULL test, and each flag was then read; the call in common/variables.c, inside a conditional expression, was found by reading. On top of the change, it flags 2 of 95 calls, neither a dereference: _get_thumbnail_input() (caches/mipmap_cache.c) hands the answer to dt_image_choose_input_path(), which tests it, and _image_remove() (common/image.c) only releases it. What a missing image means depends on the caller:

Caller With no image
downscaled float input (_init_f()), Exif blob of an export, $(IMAGE.EXIF) that piece fails: a zero-sized input, no Exif blob (stderr), an empty expansion
import: the group's representative, a member the new image leads its own group; the member keeps its group
import: the new image itself its row is removed again and the import of that file fails — a row whose Exif was never read would block a new import of the file
dt_grouping_*() the member is not written through the cache, but an update of the rows still moves it with the others; a group is never named after it; the image itself unreadable answers -1, "no change"
location, date/time, monochrome setters the image is left as it is (stderr), with no undo step; undo and redo skip it
duplicate the copy of the original's row, group_id included, is its group
move or rename a duplicate's row stays on the old film roll, and a "File operation failed" window gives the new path
local copy creating one: the "cannot create local copy." window (the copy is written but unflagged, and asking again flags it); removing one: stderr only, the copy being gone and a stale flag falling back to the original
GUI print view and preview window refuse to open, a zoomed thumbnail is not available, the jump to the film roll does nothing, focus peaking and the deflicker histogram are skipped
HDR merge stops, with the "Image could not be opened" window
ansel-cli with an XMP file says the image could not be read from the database

tests/unittests/test_image_cache_unreadable.c puts one image in the state a failed first read leaves — the second connection of "Testing it" holds the lock while the image is asked for, before anything else of the test reads — and pins grouping, the setters and a move. Measured 2026-10-28, Debug, clang 19, on the change merged with the allocation-failure work as it stood just before landing in 1442b277c6 (its code differs by one split assignment in caches/mipmap_cache.c): the four tests pass, and with the callers put back as they were, each ends in SIGSEGV. On top of c99e5d42fc: dt_grouping_change_representative() left the unreadable member alone in a group named after the old representative, which had moved to the new group; the test asserting the whole group under the new representative fails there, and passes once the rows are moved. An allocation failure reaches the import: ansel-cli importing a PNG, under an LD_PRELOAD shim whose posix_memalign() answers ENOMEM once inside dt_image_cache_allocate(). With the callers as they were, SIGSEGV on img->group_id in _image_import_internal(); with them, the import of the file fails, the library keeps no row of it, and the same command run again imports and exports it.

Claims of an earlier version of this section, found wrong: that provoking SQLITE_BUSY takes a second process; that the purge brings back an image whose read failed; that a moment's lock by an indexer or a backup agent could cost a thumbnail at any time — outside the window above, it cannot fail a read; and that a busy timeout waits such a lock out — not for a connection already reading.

Releasing an image cache entry returns the LOCK, not the image

Found 2a0a1efe91, 2026-09-06.

dt_image_cache_read_release() and dt_image_cache_write_release() (caches/image_cache.c) guard on a NULL pointer and nothing else. They used to read if(IS_NULL_PTR(img) || img->id <= 0) return; — which is dt_image_invalid() spelled out — and that skipped the release for precisely the entries most likely to have one outstanding.

An entry whose row has gone stays in the cache with id == UNKNOWN_IMAGE (-1): the allocator runs dt_image_repository_load(), that fails with no more rows available, and dt_image_init() has already left the id there. Anything holding such an entry then called release, got nothing, and left it locked forever. dt_cache_get() spins on trywrlock with a g_usleep(5) retry, and try* locks report busy even on same-thread reentry (see the rwlock section below), so the next writer hangs the GUI thread with no error and no stack anywhere else — every other thread sits idle in dt_pthread_cond_wait. Measured: a whole film roll removed and undone froze in dt_image_history_changed() waiting on an entry nobody held.

dt_image_cache_testget() is the other half and now carries the validity check its two siblings (dt_image_cache_get(), dt_image_cache_get_reload()) always had: handing out a LOCKED invalid image is what creates the leak, because the caller has no way to release what it was told is not an image.

This is reachable whenever a row disappears while the GUI still refers to it — removal, and the lighttable refreshing a thumbnail right after. Grouped images make it far likelier, since _add_thumbnail_group_borders() re-reads every member.

Duplicating an image races its own thumbnail generation against the history copy

Found 48f8e58e0a, 2026-07-31.

Lighttable "Duplicate" (dt_control_duplicate_images_job_run, control_jobs.c) creates the new DB row via dt_image_duplicate(), then copies the source's history onto it via dt_history_copy_and_paste_on_image(..., DT_HISTORY_MERGE_REPLACE, ...). dt_image_duplicate() (common/image.c) used to call dt_collection_update_query(..., DT_COLLECTION_CHANGE_RELOAD, ...) unconditionally, right after inserting the row — i.e. before the caller had copied any history onto it. That reload makes the new image visible to the lighttable grid, which can create its thumbnail widget and request a render immediately, against the row's momentary real state: zero history.

Confirmed with -d cache -d history -d lighttable: for a freshly duplicated image, the first [mipmap_cache] compute mip size 0 ... from original file log line landed ~650ms before the matching [dt_dev_write_history_ext] writing history for image N line. The mipmap cache is not hash-driven (previous section), so once that first, historyless render is cached, only an explicit dt_mipmap_cache_remove + refresh recovers — and even when that recovery path runs correctly and a second, correct render finishes and gets cached, nothing guarantees a timely repaint of it (the thumbnail widget can be left showing the first render under a permanent "busy" overlay for several seconds, until an unrelated GUI event forces a redraw). Patching the recovery/notification side (adding a missing GUI-thread redraw request on one early-return path in gui/dtgtk/thumbnail.c's _get_image_buffer()) did not fix this reliably and was reverted — the actual fix is to not let the race start in the first place.

Fixed by dt_image_duplicate_no_reload() (common/image.c): same as dt_image_duplicate() but skips the immediate collection reload. Both call sites that duplicate-then-copy-history (dt_control_duplicate_images_job_run in control_jobs.c, _history_style_apply's duplicate-and-apply-style branch in history_actions.c) now use it and trigger exactly one dt_collection_update_query(..., DT_COLLECTION_CHANGE_RELOAD, ...) themselves, after the history copy/delete completes — so the very first time the duplicate becomes visible, it already carries its final history. Any future caller of dt_image_duplicate() that will mutate the new image's history afterward (a style, a batch edit, ...) should do the same rather than let the default immediate reload race its own follow-up write.

A cache entry that cannot be allocated

Measured 2026-10-28 on top of 0ba2521c6f, by an LD_PRELOAD shim that fails posix_memalign() (where dt_alloc_align() ends) only while the stack holds a given allocate callback, both being exported from libansel.so: on the ansel-cli export of tools/check_it_runs.sh, and on ansel-generate-cache --min-mip 2 -m 2 over the library that export leaves. Leaks counted by memcheck 3.22, which intercepts the allocation functions of every object, the shim's included, unless run with --soname-synonyms=somalloc=nouserintercepts.

dt_cache_get() answers NULL when the key was not cached and the allocate callback left entry->data NULL. The entry goes back to g_slice_free1(), its allocator, and its LRU cell with it: nothing else would free the cell, since the entry was never inserted (memcheck: 24 bytes definitely lost per failure while it stayed). A cached key never takes that path, so the demote re-gets — of an entry marked _lock_demoting, which _cache_gc_locked() and dt_cache_remove() skip — cannot answer NULL and do not check.

Mipmap cache. dt_mipmap_cache_allocate_dynamic() leaves entry->data NULL when dt_alloc_align() fails. A blocking get then answers status CACHE_FULL with a NULL cache_entry: no buffer for a DT_MIPMAP_F or DT_MIPMAP_FULL input, and for a thumbnail the 8x8 skull itself, _skull_8, shared and read-only since no entry holds a copy (releasing it is a no-op, as for any buffer without an entry). The status matters to the export, which hands it back: OK with no buffer would read as a success there — _init_8() would then copy into the descriptor the output size the export sets only after its pipe ran (measured: the disk-cache write of that thumbnail fails on libjpeg's "Maximum supported image dimension is 65500 pixels").

A NULL cache_entry with CACHE_FULL is a failure of this very request, where a failed render or decode leaves an entry behind. The two callers that see it open a "Not enough memory" window, which caches/ cannot do itself, sitting below control/: _view_image_get_surface_internal() (views/view.c) for a thumbnail, whose skull reads "Out of RAM" like any CACHE_FULL skull, and _dt_dev_mipmap_prefetch_full() (develop/develop.c) for a full-size input, with the message dt_imageio_open() gives a decode that ran out of memory, so both land in one window. Seen in the GUI on top of 600b680296, the same day, all thumbnail allocations refused: three skulls labelled "RAM insuffisante" and one window listing the three files; ansel-cli prints the full-size window on stderr. Nothing is cached, so every new request for that thumbnail tries the allocation again (a zoom change: 24 refusals, 8 per image, measured on top of 600b680296), and a dt_control_alert() would bring the window back at each one once OK is clicked. So both callers, and dt_imageio_open() for its decode, raise it with dt_control_alert_once() (on top of c2a2a33088): the window lists every file refused while it is up, and once closed it does not open again this session for that message -- the skulls keep saying it on the thumbnails. Seen in the GUI on top of c2a2a33088, 2026-10-28, under the same refusals: the window listed the three files, and after OK a zoom change left the skulls and did not bring it back. An alert raised without once under the same title and message still opens the window, and the once items that come while it is up are listed in it.

In an export, the first get to fail is _dt_dev_mipmap_prefetch_full(), through dt_dev_load_image(), whose failure the export does not check. One failure and the run goes on to fail in the pipe, basebuffer reading a 0x0 input: PROCESSING_FAILED (measured, error 10 on the thumbnail). Two in a row and the export's own get fails: CACHE_FULL (error 3). basebuffer fails its run on a NULL buffer as well, which takes the entry being evicted after the export's get and not re-allocated (by reading: the shim cannot order that eviction).

Image cache. dt_image_cache_allocate() returns early when dt_alloc_align() fails, saying so on stderr ([image_cache] could not allocate a cache entry for image N), and dt_image_cache_get() and dt_image_cache_get_reload() answer NULL. No window says it, since caches/ cannot open one: each caller decides what a missing image means where it is, and what it says, the same as for a vanished row or a failed read (see "What a caller does with the NULL" in "A failed database read is not a missing image"). Found wrong on 2026-10-28: this paragraph counted 18 of the 97 calls dereferencing the answer untested, on top of 314f2a3875; there were 29, and the count had missed 11.

Why a thumbnail is a skull, and how it says so

Found 8bef6908d9, 2026-10-05, by reading every path that zeroes a thumbnail's size (#1531).

A skull is one 8x8 pattern, _skull_8 (caches/mipmap_cache.c). _paint_skulls() copies it (dead_image_8()) over any thumbnail whose descriptor came out of generation as 0x0; a thumbnail whose entry could not be allocated is handed the pattern itself (see "A cache entry that cannot be allocated"). Every cause of a failed render therefore goes through zeroing dsc->width/height, and until this change the cause was lost there — stderr only got could not process thumbnail!. The causes, each now kept as a dt_imageio_retval_t in dsc->status and handed to readers in dt_mipmap_buffer_t.status:

Where Cause Status
_init_8 neither the original nor a local copy exists, or the image left the image cache FILE_NOT_FOUND
_init_8 embedded-JPEG mode "always" (2), an image that is a raw -- or, before its first decode, has a raw extension (dt_image_ext_is_raw(), so .dng but not .tif, which is RGB whether LDR or HDR) -- and no embedded/companion JPEG could be read: the pipe is forbidden. A PNG, a TIFF or another RGB file goes through the pipe instead -- it usually has no embedded preview NO_EMBEDDED_THUMBNAIL
_generate_blocking, DT_MIPMAP_FULL every decoder failed dt_imageio_open()'s reason: see "Which loader failure a skull names"
_generate_blocking, DT_MIPMAP_FULL no source any more: the file went after _init_8's check, or the image left the image cache FILE_NOT_FOUND
dt_imageio_open the path exists but is not a regular file FILE_NOT_FOUND
dt_mipmap_cache_get, a thumbnail size the thumbnail's own entry could not be allocated: the shared skull, with no entry behind it CACHE_FULL
dt_imageio_export_with_flags the full-size input's mipmap entry could not be allocated: see "A cache entry that cannot be allocated" CACHE_FULL
dt_imageio_export_with_flags pipe init failed, or the pipe returned an error / no backbuf PROCESSING_FAILED
dt_imageio_export_with_flags the pipe returned an error after the pixelpipe cache refused it memory CACHE_FULL
dt_imageio_export_with_flags the pipe was stopped (dt_dev_pixelpipe_process returns the same 1 for a kill-switch stop as for a failure) ABORTED
dt_imageio_export_with_flags the pipe's output was gone from the pixelpipe cache before it could be referenced PROCESSING_FAILED
dt_imageio_export_with_flags the output buffer could not be allocated CACHE_FULL
dt_imageio_export_with_flags format->write_image failed IOERROR

Not causes: an external stop (shutdown_ext, a thumbnail scrolled out of view) restores the descriptor and keeps the entry flagged for generation, so it never leaves a skull; the disk cache never holds one (only > 8x8 entries are written, and an unreadable .jpg is deleted and regenerated); _init_8's early return for width < 16 does not zero the size.

"File not found" is about the source, never the cached thumbnail

Found c1c4123fcb, 2026-10-20, by reading every path that returns DT_IMAGEIO_FILE_NOT_FOUND; the table's two FILE_NOT_FOUND rows after the first come from this reading. The EXISTS and IS_REGULAR columns below are measured with g_file_test(), GLib 2.80.0, Linux, as uid 1000 — not root, which passes every permission check. The labels are by reading.

FILE_NOT_FOUND reaches a thumbnail by three paths, all about the image's source file: the original, or its local copy.

  1. _init_8(), before anything else: _write_mipmap_to_disk() asks dt_image_choose_input_path() for a source and none exists. The candidates, in order: the local copy (then its legacy path) when the image is flagged DT_IMAGE_LOCAL_COPY, then fullpath, then the local copy again as a last chance. An image that left the image cache gives no dt_image_t, hence no path either.
  2. _generate_blocking() at DT_MIPMAP_FULL, reached through the export pipe _init_8() starts: the same lookup fails there, so the file went in between — a drive unplugged mid-render. dt_imageio_export_with_flags() hands the full-size buffer's status back as is.
  3. dt_imageio_open(): a path that exists but is not a regular file returns !DT_IMAGEIO_OK, which is 1, which is FILE_NOT_FOUND by its position in the enum rather than by choice.

"Exists" is g_file_test(G_FILE_TEST_EXISTS), which is access(F_OK) on POSIX:

The source path EXISTS IS_REGULAR Label
absent: deleted, moved, renamed, drive unplugged, share not mounted 0 0 File not found (path 1)
a broken symlink 0 0 File not found (path 1)
a file in a directory without search (x) permission 0 0 File not found (path 1)
a file without read permission (mode 000) 1 1 a decoder's failure, not "File not found"
a directory named like an image 1 0 File not found (path 3)

Two FILE_NOT_FOUND returns never reach a thumbnail: dt_imageio_open_j2k()'s (it cannot open the file or read its header) runs inside dt_imageio_open_raster(), whose chain ends on FILE_CORRUPTED; and the dt_imageio_open_libraw() stub of a build without LibRaw is never called, since dt_imageio_open_raw() calls it only under HAVE_LIBRAW.

The disk cache never produces it. In the cache's allocate callback, a missing .jpg leaves the entry flagged for generation, and so does an unreadable one, which is deleted; _init_8() then renders from the source. A .jpg that loads is shown without the source being looked at, and so is an entry already in RAM: a missing original keeps showing its last thumbnail, and the skull appears only when no thumbnail is left to show.

Like every skull, it stays in RAM until the entry is evicted or invalidated. The file coming back does not change that: the full-size entry stays flagged for generation, but the thumbnail entry that asked for it does not, and the mount signal libs/collect.c listens to refreshes the folders' accessibility (dt_film_set_folder_status()), not the mipmap cache.

A pipe that failed for lack of memory says so

Established on 8bef6908d9 with this change, 2026-10-05; the cache's side measured by tests/unittests/test_pipe_cache_alloc_refusals.c, the export's side not run.

A run the pixelpipe cache refuses a buffer fails like any other: the module or dt_dev_pixelpipe_cache_get_writable() gets NULL and the pipe returns 1. Telling the two apart afterwards needs the cache, which is the only one that knows. It counts its refusals per thread in dt_pixelpipe_cache_get_alloc_refusals(), at the two places where it already tells the user "the pipeline cache is full":

  • _free_space_to_alloc() returning an error: the budget is spent and every line is in use, so nothing can be evicted;
  • _log_arena_allocation_failure(): the arena has no free run long enough, or _system_memory_pressure_valve() refused because the system is out of RAM.

The export reads the count before dt_dev_pixelpipe_process() and again on failure. The pipe runs on its caller's thread, so a change is this run's refusal, not another pipe's. The test measures that: a granted buffer is not counted, a refusal is counted exactly once on each of the two paths, and a refusal on another thread leaves this one's count unchanged.

Still PROCESSING_FAILED: memory a module allocates outside the cache (plain dt_alloc_align), allocations on OpenMP worker threads, and OpenCL allocations. Device memory is not the cache's, and a device failure falls back to the CPU anyway. One known mislabel goes the other way: a refusal the run recovered from, followed by an unrelated failure in the same run, reads as CACHE_FULL.

A skull stays in RAM until the entry is evicted or invalidated, whatever the cause — a transient one (CACHE_FULL) included. The label makes that visible; it does not change it. dt_mipmap_cache_remove() invalidates it, together with the failed full-size load behind it.

_view_image_get_surface_internal() (views/view.c) paints the label: every consumer (lighttable, filmstrip, preview window, map, print, slideshow) goes through it. It runs in a worker thread, so it builds its own Pango font description rather than borrowing bauhaus's, which dt_bauhaus_load_theme() frees from the GUI thread.

Which loader failure a skull names

Measured 2026-10-20 on c1c4123fcb, before and with this change, by a scratch program linked against libansel.so: it imports a good file into a :memory: library, breaks the file, asks for a DT_MIPMAP_2 thumbnail and the DT_MIPMAP_FULL buffer and prints their status, repairs the file, calls dt_mipmap_cache_remove(), and asks again. --disable-opencl, lighttable/embedded_jpg=0, so every thumbnail goes through the export. The table's broken files are made from one Nikon Z 6 NEF, which rawspeed decodes. Labels from view.c. Seen in the lighttable the same day, in French, on the same files imported whole and then broken on disk: "Boîtier non supporté", "Erreur d'E/S" twice, "Fichier corrompu" twice.

When every decoder fails, dt_imageio_open() returns the first failure that says why a decoder which recognised the file could not load it — UNSUPPORTED_FEATURE, UNSUPPORTED_CAMERA, LOAD_FAILED or IOERROR — and FILE_CORRUPTED otherwise (_decoder_reason()). Each chain (dt_imageio_open_raster/_raw/_hdr()) applies the same rule to its own decoders.

The other two failures say nothing once the whole cascade has run. Every raster and HDR decoder, and LibRaw, answer FILE_CORRUPTED both to another format and to a broken file in their own; rawspeed answers UNSUPPORTED_FORMAT to a file it cannot parse, a cut header included. The cascade tries every decoder on every file, so kept, these would make a truncated JPEG read "Unsupported format": rawspeed, tried after the raster chain, does not know JPEG. "Unsupported format" therefore never reaches a skull.

Broken file rawspeed LibRaw Before Now
model renamed to an unknown NIKON Z 0 UNSUPPORTED_CAMERA decodes it no skull no skull
the same, cut to 4 MB UNSUPPORTED_CAMERA I/O error Corrupted file Unsupported camera
0 bytes FileIOException: IOERROR fails Corrupted file I/O error
mode 000 FileIOException: IOERROR fails Corrupted file I/O error
cut to 4 MB RawDecoderException: UNSUPPORTED_FORMAT fails Corrupted file Corrupted file
cut to 300 bytes, inside the TIFF header IOException fails Corrupted file Corrupted file
1 MB of A no decoder: UNSUPPORTED_FORMAT fails Corrupted file Corrupted file

Before, the table row above that says "the loader's" status was wrong in effect. The loader's value reached dsc->status, but every chain ended on return DT_IMAGEIO_FILE_CORRUPTED and the cascade on dt_imageio_open_hdr()'s, so a failed load always read "Corrupted file".

LibRaw follows rawspeed on every raw, not only on CR3 (dt_imageio_open_raw()). A camera or a feature rawspeed does not support leaves a skull only when LibRaw fails too: the first row.

rawspeed's IOException is FILE_CORRUPTED, not IOERROR. rawspeed throws it on reading past the end of the copy FileReader::readFile() made of the whole file, so it means cut or corrupt; failing to read the file throws FileIOException. decodeRaw(), decodeMetaData() and checkSupport() rewrap it as a RawDecoderException, so only RawParser::getDecoder() lets it out, on a header cut short. Cut to 60 %, an ARW, a CR2, a RAF, an ORF, an RW2 and a DNG all failed in rawspeed through RawDecoderException, before and now. LibRaw then decoded the ARW, the ORF and the DNG anyway (under gdb, dt_imageio_open_libraw() returned DT_IMAGEIO_OK): the first two gave a picture, the DNG a PROCESSING_FAILED skull from the pipe.

Reachable by reading, not measured: UNSUPPORTED_FEATURE (rawspeed: a sample type or size it does not handle, a RawDecoderException that mentions "supported", a CameraMetadataException) and LOAD_FAILED (rawspeed: a CFA image with more than one component per pixel).

Two statuses of the table above never reach a skull, by reading. IOERROR from format->write_image: the thumbnail's format copies into the cache buffer and returns 0. ABORTED: the thumbnail's pipe is local to dt_imageio_export_with_flags(), so only its shutdown_ext can stop it, and that external stop restores the entry.

A purge tries a failed load again

Measured 2026-10-20 on c1c4123fcb, before and with this change, as above. Seen in the lighttable the same day: once the five broken files were repaired, "Purge selected thumbnails from cache" brought them back without a restart.

dt_mipmap_cache_remove() — "Purge selected thumbnails from cache", an embedded-JPEG mode change, any history change — drops the thumbnails, and also the DT_MIPMAP_FULL entry when its status is not DT_IMAGEIO_OK. Without that, the purged thumbnail was regenerated, but the export behind it asked for DT_MIPMAP_FULL and read the failure back. Fourteen broken files whose load failed — the NEF renamed and cut, emptied, made unreadable, cut to 4 MB and filled with A; the NEF cut at six sizes from 100 bytes to 50 kB; and a CR2, a RAF and an RW2 cut to 60 % — were repaired before the purge: the thumbnail stayed a skull with the same status for all fourteen. Only eviction or dt_mipmap_cache_remove_all_sizes(), an image leaving the library, dropped that entry. Now all fourteen give their picture back after the purge.

Eviction does retry a failure by chance once enough other images are loaded. The full-size cache holds worker_threads + 1 entries and is collected above 80 % of that, by reading: in the lighttable, with seven images and worker_threads=4, each broken file was decoded twice before any purge, once per thumbnail request. A failure stuck in the cache is therefore the case of a few images, and the purge is what retries it there.

The decoded raw of a healthy image still survives a purge: over the same import, thumbnail, purge, thumbnail sequence, -d cache logs one from original file I/O, before and now.

So does a raw decoded from a broken file, since its load succeeded. LibRaw decodes the DNG cut to 60 % and the pipe fails on it: the PROCESSING_FAILED skull came back after the repair and the purge, before and now. The cut ARW and ORF gave a picture, which a purge keeps in the same way. Nothing in the cache notices that the file changed.

The failure stays cached until then rather than not being cached at all. A thumbnail's export loads the image through dt_dev_load_image(), which already asks for DT_MIPMAP_FULL (_dt_dev_mipmap_prefetch_full()), then asks for it again itself. Uncached, every thumbnail of a broken file would run the whole decoder cascade twice, and again for each thumbnail size.

The label wraps onto half the skull at most

Measured on 8bef6908d9 with this change, 2026-10-05, outside the application: a standalone pangocairo program running the label block of view.c line for line over dead_image_8(), scaled with the nearest filter as view.c does, at square sizes from 32 to 160 px (FIT scales the 8x8 skull uniformly, so the surface is always square). Pango 1.58.2, cairo 1.18.6, the Win32 font map; "sans bold" at 12 px resolved to DejaVu Sans Bold, one line 15 px tall. The final code ran on ten labels at eight sizes from 32 to 128 px: four English ones; "No embedded thumbnail" in French, Japanese, Chinese, Thai and Russian (the strings are new, so these were not taken from po/); and "Speicherzugriffsfehler", for a word wider than the thumbnail. Not yet seen in the application.

The label wraps onto as many lines as half the thumbnail holds (pango_layout_set_height(layout, img_height / 2 * PANGO_SCALE)), and Pango ellipsizes the last. Half keeps the skull's eyes clear: they end at 3/8 of its height. At this font size, half holds a single line below 56 px, which is the ellipsized line of before. Over the 80 label × size pairs, the band reached the eyes once, by half a pixel: Chinese at 60 px, whose two lines take 34 px.

Half is approximate, by a pixel or two. 27 px holds one line, 28 px already two (30 px), and three start at 45 px. 28 is twice the font's metric height, 13.97 px, against 15 px for a laid-out line. A line count instead (pango_layout_set_height(layout, -n), with n from a first one-line measure) was tried and dropped. Over 35 label × size pairs it gave the same lines as the pixel height in all but one: Thai at 60 px, one line instead of two, the eyes clear either way. Nor is it more exact: the one line it measures need not be as tall as the lines that wrap. Chinese at 60 px measured 15 px and wrapped onto 34.

PANGO_WRAP_WORD_CHAR, not PANGO_WRAP_WORD. With WORD, a word wider than the thumbnail stays on one line, overflows both edges and is not ellipsized: "Speicherzugriffsfehler" at 72 px was drawn from −41 to 113 px, "Unsupported" from −9 to 81. WORD_CHAR breaks such a word inside, with a hyphen, and still breaks between words where it can.