Repository navigation
fix(clickhouse): request LZ4 responses from ClickHouse 26.9+ - #371
Merged
Merged
Conversation
ClickHouse 26.9 frames `compress=1` HTTP responses with ZSTD by default, while the clickhouse crate 0.14 only decodes LZ4 native frames, so every sink query failed with "decompression error: incorrect magic number". Setting network_compression_method=lz4 restores LZ4 framing on 26.9 and is a no-op on older servers.
The test wrote the stale and canonical versions of block 1 in a single INSERT, which ReplacingMergeTree collapses on insert, so the two-version precondition never held. Write the stale version separately into another monthly partition; stopping merges instead would also block the repair's DELETE mutation.
emmajam
approved these changes
Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ClickHouse 26.9 switched the codec of
compress=1HTTP responses to ZSTD (ClickHouse#108786), but theclickhousecrate 0.14 only decodes LZ4 native frames. Againstclickhouse-server:latest, which all of our compose files use, every sink query fails withdecompression error: incorrect magic number, starting with the version check inensure_schema.The sink client now sets
network_compression_method=lz4. Every 26.9 stable release honors it forcompress=1responses (ClickHouse#119825), and older servers already respond with LZ4. The upstream crate fix (clickhouse-rs#482) is unreleased; it avoids this setting because read-only users cannot change it, which doesn't matter for the sink since it needs write access. The query engine insrc/clickhouse.rsuses plain reqwest withoutcompress=1and was not affected.This also fixes
test_backfill_replays_blocks_with_extra_block_versions, which failed on every server version. It wrote the stale and canonical versions of block 1 in one INSERT, which ReplacingMergeTree collapses on insert, so the two-version precondition never held. The product check is correct (canonical_blocks_presentreads withoutFINAL). The test now writes the stale version separately into another monthly partition so it survives until the backfill repairs it. Stopping merges would not work here because it also blocks the repair'sALTER TABLE ... DELETEmutation.The ClickHouse integration tests pass against both 26.9.8.3 and 25.12.11.4.
🤖 Generated with Claude Code