Failure: stateless / 00157_cache_dictionary — INSERT timeout when randomizer picks enable_filesystem_cache=0

[CarlosFelipeOR]

Summary

00157_cache_dictionary times out when inserting 1M rows from test.hits into test.hits_1m. The failure is intermittent because the INSERT only times out when the CI randomizer rolls enable_filesystem_cache=0, which happens ~50% of the time. On Altinity runners (and local machines), the unsupported read path is too slow for the test's 300s limit.

Affected tests:

• tests/queries/0_stateless/00157_cache_dictionary.sql (upstream stateless suite, stateful test that requires the hits dataset)

Affected jobs / branches:

• Stateless tests (amd_coverage, 3/8) on antalya-26.1 — already mitigated via broken_tests.yaml in Altinity/ClickHouse#1604.
• Stateless tests (amd_debug, sequential) on antalya-26.3 and stable-26.3 — 64 FAILs / 308 commits in last 60 days (~21% job-level fail rate after retries).
• Stateless tests (arm_binary, sequential) on antalya-26.3 and stable-26.3 — 57 FAILs / 285 commits in last 60 days (~20%).
• Stateless tests (amd_ubsan, sequential) on antalya-26.3 — 7 FAILs / 115 commits (~6%).
• All other Altinity checks SKIP this test via existing tags (stateful, no-tsan, no-msan, no-asan, no-parallel).

Affected files:

• tests/queries/0_stateless/00157_cache_dictionary.sql — does not constrain enable_filesystem_cache in the randomizer, while explicitly forcing max_threads=1 on the INSERT.

Failing runs (samples):

• antalya-26.3, amd_debug sequential, SHA 59595eb83: https://altinity-build-artifacts.s3.amazonaws.com/json.html?REF=antalya-26.3&sha=59595eb833e646555197d9f891693acff3a99644&name_0=MasterCI&name_1=Stateless+tests+%28amd_debug%2C+sequential%29
• stable-26.3 - PR #1867, amd_debug sequential (failed first attempt, passed on retry): https://altinity-build-artifacts.s3.amazonaws.com/PRs/1867/0f1a2bdd63003e2d54b41fcaf7d71c020de69cca/stateless_tests_amd_debug_sequential/job.log

Description

00157_cache_dictionary fails intermittently on multiple antalya-26.3 / stable-26.3 non-coverage checks with:

Code: 159. DB::Exception: Timeout exceeded: elapsed ~300000–340000 ms, maximum: 300000 ms:
While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED)
(in query: INSERT INTO test.hits_1m SELECT * FROM test.hits LIMIT 1000000
 SETTINGS min_insert_block_size_rows = 0, min_insert_block_size_bytes = 0,
          max_block_size = 8192, max_insert_threads = 1, max_threads = 1, max_parallel_replicas=1;)

The failure happens during the INSERT that prepares the source table, before the CACHE dictionary validation queries run. The 4 SELECTs that actually exercise the dictionary cache (the purpose of the test) never execute when the failure triggers.

Same test on the same builds in upstream CI: 0 FAILs in 16,064 runs on Stateless tests (amd_debug, sequential) in last 90 days. The Altinity-fork code is not the cause — the test source, the randomizer (tests/clickhouse-test), the random_settings limits, and the test config (tests/config/users.d/*) are byte-identical between Altinity/ClickHouse antalya-26.3 and ClickHouse/ClickHouse 26.3. The discrepancy is explained by runner hardware/environment, not by code (see "Why upstream CI sees 0 failures" below).

Root Cause

A single randomized setting, enable_filesystem_cache, when sorted to 0, combined with the .sql's hardcoded max_threads=1, max_insert_threads=1 on the source-table INSERT, makes the INSERT execution-time uncertain depending on runner hardware/load. On hardware that is borderline relative to the test's 300s max_execution_time, the INSERT consistently overflows.

Evidence — local reproduction (30 runs, upstream 26.3 build, no overrides)

17 OK  / 13 FAIL  → 43% per-attempt fail rate

Across the 13 failing runs, the Settings used in the test block shows:

Setting	Value across all 13 failures	Pattern
enable_filesystem_cache	0 0 0 0 0 0 0 0 0 0 0 0 0	constant (100%)
page_cache_inject_eviction	mixed True/False	irrelevant
use_page_cache_for_local_disks	mixed True/False	irrelevant
local_filesystem_read_method	all 4 values appear	irrelevant
max_threads	1, 2, 3, 32 (overridden by .sql SETTINGS to 1)	irrelevant
enable_filesystem_cache_on_write_operations	mixed 0/1	irrelevant

The randomizer rule (identical upstream and Altinity, tests/clickhouse-test:1381):

"enable_filesystem_cache": lambda: random.randint(0, 1),

gives P(toxic value) = 50%. The 43% observed per-attempt fail rate matches expectation within n=30 noise.

Evidence — failure reproduces on the upstream binary too

The intermittent timeout is not Altinity-binary-specific. Reproduced in 4 independent setups on the same workstation:

Build	Test source	Result
Altinity stable-26.3 debug	Altinity/ClickHouse@stable-26.3 00157_cache_dictionary.sql	Reproduces (timeout on INSERT)
Upstream 26.3.13.30 debug	Altinity/ClickHouse@stable-26.3 00157_cache_dictionary.sql	Reproduces
Upstream 26.3.13.30 debug	ClickHouse/ClickHouse@26.3 (upstream) 00157_cache_dictionary.sql	Reproduces (43% per-attempt fail rate over 30 runs — see local reproduction table above)

In every reproduction the failing INSERT had enable_filesystem_cache=0 in the randomized settings dump. In every reproduction the failure occurred during MergeTreeSelect, before any CACHE dictionary code was exercised.

This rules out Altinity-fork patches as a cause. The bug is in the interaction between (a) the test's hardcoded max_threads=1 on a multi-GB read, (b) the randomizer's 50% chance of disabling the filesystem cache, and (c) runner hardware/IO performance.

Evidence — CI database

Source	Total rows	Distinct commits	Fail rate (job-level, after retry)	Implied per-attempt fail rate (sqrt)
Altinity 26.3 amd_debug, sequential	308	297	21%	~46%
Altinity 26.3 arm_binary, sequential	285	—	20%	~45%
Upstream amd_debug, sequential	16,064	—	0%	0%

The implied per-attempt fail rate on Altinity (~46%) is statistically indistinguishable from the local notebook (43%) and very close to the theoretical 50% from the randomizer. Upstream's 0% indicates the upstream runner hardware completes the INSERT within 300s even when enable_filesystem_cache=0 is rolled.

Why this test specifically fails

• The .sql forces max_threads=1, max_insert_threads=1, max_parallel_replicas=1 on the INSERT to test the CACHE dictionary under deterministic thread conditions.
• test.hits is multi-GB and the LIMIT 1000000 pull still requires reading many parts.
• With enable_filesystem_cache=0, each MergeTreeSelect read goes to the underlying storage without ClickHouse's filesystem cache, in a single thread.
• On runners where the unfavored read path is borderline (Altinity CI, consumer notebooks), the INSERT consistently approaches/exceeds the test's max_execution_time = 300.
• The randomized setting has no relationship to what the test is intended to validate (CACHE dictionary correctness under cache thrashing with SIZE_IN_CELLS=1).

Why upstream CI sees 0 failures despite the bug existing in upstream code

ClickHouse/ClickHouse CI runs this test only on m7i-flex.2xlarge instances:

• 8 vCPU, 32 GiB RAM, 40% baseline CPU performance (burstable; credits absorb spikes)
• EBS-only storage (no local NVMe), up to 10 Gbps EBS bandwidth, gp3 default 3k IOPS / max 16k IOPS
• Dedicated instance, fresh container per job, zero concurrent workload, no swap, no userspace processes competing for page cache

In this environment, even with enable_filesystem_cache=0 and max_threads=1, the INSERT consistently completes in well under 300s. The combination of clean kernel page cache (32 GiB free for the OS) and the consistent EBS throughput is enough to keep the single-threaded MergeTreeSelect under the limit.

On any runner that is not a fresh, idle, well-provisioned cloud instance — Altinity CI runners (containerized, possibly co-tenanted, ~21% per-commit fail rate), local developer machines (Docker bind-mount overhead, co-existing desktop processes, fragmented page cache, 43% per-attempt fail rate) — the same INSERT pushes past 300s.

The test does not announce or document a hardware floor; it just silently relies on it. Any environment hitting the borderline range trips the timeout. This is a latent fragility in the test, not a property of upstream CI being immune.

Why retries hide ~half the failures in the CI database

The CI database stores one row per (commit, check) after retries. clickhouse-test reruns the failing test with a new randomizer seed (no --no-random-settings is passed). The retry has a fresh ~50% chance of avoiding enable_filesystem_cache=0, so the DB-reported fail rate is approximately the square of the per-attempt rate: 0.46² ≈ 21%. Job-level retries account for the observed gap between the per-attempt rate seen locally and the 21% reported in the DB.

Suggested Solution

Three viable approaches, in increasing order of upstream-alignment friction:

Option A — Surgical fix in the test (Random settings limits, recommended)

Add one line to tests/queries/0_stateless/00157_cache_dictionary.sql:

-- Tags: stateful, no-tsan, no-msan, no-asan, no-parallel
-- Random settings limits: enable_filesystem_cache=(1, 1)

• Restricts exactly the one toxic setting, preserves randomization of ~99 other settings.
• Uses the existing test framework mechanism (already used by ~20 other tests like 02346_text_index_hint_profileevents.sql, 03381_lazy_materialization_limit_offset.sql).
• The test does not validate filesystem cache behavior, so restricting it cannot mask a real bug.
• Possible to land upstream-first (clean PR with the data above as justification) and inherit on rebase, or alternatively cherry-pick into both antalya-26.1 and antalya-26.3 and skip the upstream PR if approval is faster.

Option B — Altinity-fork-only patch

Same .sql change but applied only to Altinity branches (antalya-26.1, antalya-26.3, stable-26.3). Faster path if waiting on upstream review is undesirable. Will need to be carried as a divergence until the upstream PR (Option A) lands.

Option C — broken_tests.yaml workaround

Add the test to broken_tests.yaml with a strict timeout-only message filter, scoped to the affected check_types. Status:

Branch	Status	Coverage
antalya-26.1	applied via Altinity/ClickHouse#1604	sanitize-coverage only
antalya-26.3	not applied	—
stable-26.3	not applied	—

Both antalya-26.3 and stable-26.3 still need a broken_tests.yaml entry covering at least debug and arm_binary (and likely sanitize-coverage again) if Options A or B are not pursued.

Hides the symptom but does not fix the root cause; ~21% of stateless job runs continue to waste CI time on the timeout + retry cycle on the affected branches.

Applied Solution

Partial mitigation, antalya-26.1 only:

• broken_tests.yaml entry for sanitize-coverage check_type: Altinity/ClickHouse#1604.

antalya-26.3 and stable-26.3 remain unmitigated. Options A, B, or C still to be decided for those branches.

References

• Test source: tests/queries/0_stateless/00157_cache_dictionary.sql
• Randomizer (identical upstream + Altinity): tests/clickhouse-test, class SettingsRandomizer, line ~1381.
• Existing Random settings limits precedent: e.g. tests/queries/0_stateless/03381_lazy_materialization_limit_offset.sql (-- Random settings limits: index_granularity=(None, 60000)).
• Historical upstream hardening PRs of the same test, same kind of issue:
  • ClickHouse#67893 — original coverage timeout report
  • ClickHouse#69205 — added max_parallel_replicas=1 to the INSERT
• Upstream stateless results query: play.clickhouse.com / default.checks WHERE test_name = '00157_cache_dictionary' AND check_name = 'Stateless tests (amd_debug, sequential)' → 16,064 OK / 0 FAIL in last 90 days.
• Altinity CI query: `gh-data`.checks WHERE test_name LIKE '%00157_cache_dictionary%' AND (base_ref LIKE '%26.3%' OR head_ref LIKE '%26.3%')
• Local reproduction logs (30 runs, upstream 26.3 build): /home/carlos/upstream/ClickHouse/ci/tmp/00157_cache_dictionary_run_*.log and 00157_cache_dictionary_30_runs_summary.log.
• Existing mitigation PR for antalya-26.1: Altinity/ClickHouse#1604.