Repository navigation
Conversation
sebastian-meyer
changed the title
stweil
force-pushed
the
kitodo/doc-cache-pool
branch
from
October 19, 2026 16:39
6e1fbba to
12c1f1e
Compare
Reading a document from the file-based tx_dlf_doc cache requires deserializing the entry and re-parsing the entire METS XML in MetsDocument::__wakeup(), and every cache hit then re-runs the lazy data loaders (physical structure, structure links, metadata array, file groups). For documents that are requested repeatedly by the same PHP-FPM worker - typical for crawlers, and for pages that fetch the same document several times, e.g. periodika issue lists - this is the dominant CPU cost. Keep a short-lived, per-worker in-memory pool of recently used documents in DocumentCacheManager. A pool hit skips the file cache, the deserialization, the XML re-parsing and the lazy reloading entirely and reuses the already computed data of the document. The pool is a static array (one per FPM worker process) with a sliding TTL of 300 s and an entry cap of 5 documents, evicting the least recently used entry first. Because a PHP-FPM worker handles only one request at a time, sharing instances between sequential requests of the same worker is safe. set(), remove() and flush() keep the pool in sync with the file cache, including the LOAD_FAILED sentinel. Unit tests cover pool hits, TTL expiration, LRU eviction, pool filling from the file cache and pool synchronization by setFail(), remove() and flush(). Since the pool is per-worker state, it would survive individual functional tests and leak a document loaded by one test (possibly with different settings) into the next one. A resetPool() method clears the pool without touching the file cache; the functional test base class calls it for each test case, mirroring the existing Helper::resetIndexNameCache() and Indexer::resetProcessedDocs() test isolation pattern. Assisted-by: OpenCode / qwen3.8-27b-thinking (Alibaba Cloud) Signed-off-by: Stefan Weil <sw@weilnetz.de>
stweil
force-pushed
the
kitodo/doc-cache-pool
branch
from
October 26, 2026 11:53
12c1f1e to
13c2d7a
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reading a document from the file-based tx_dlf_doc cache requires deserializing the entry and re-parsing the entire METS XML in MetsDocument::__wakeup(), and every cache hit then re-runs the lazy data loaders (physical structure, structure links, metadata array, file groups). For documents that are requested repeatedly by the same PHP-FPM worker - typical for crawlers, and for pages that fetch the same document several times, e.g. periodika issue lists - this is the dominant CPU cost.
Keep a short-lived, per-worker in-memory pool of recently used documents in DocumentCacheManager. A pool hit skips the file cache, the deserialization, the XML re-parsing and the lazy reloading entirely and reuses the already computed data of the document.
The pool is a static array (one per FPM worker process) with a sliding TTL of 300 s and an entry cap of 5 documents, evicting the least recently used entry first. Because a PHP-FPM worker handles only one request at a time, sharing instances between sequential requests of the same worker is safe.
set(), remove() and flush() keep the pool in sync with the file cache, including the LOAD_FAILED sentinel.
Unit tests cover pool hits, TTL expiration, LRU eviction, pool filling from the file cache and pool synchronization by setFail(), remove() and flush().
Assisted-by: OpenCode / qwen3.8-27b-thinking (Alibaba Cloud)