query: default Lz4 SELECTs are ~2x slower on main because the server compresses the HTTP response with LZ4HC (regression from #482) #490

Describe the bug

Since #482 (on main, not released yet; #487 prepares 0.15.3), a SELECT with the default Compression::Lz4 is about 2x slower than with 0.15.2.

#482 replaced the HTTP parameter compress=1 with enable_http_compression=1 + Accept-Encoding: lz4. With compress=1, the server compressed native blocks with the LZ4 codec in fast mode. With Accept-Encoding: lz4, the server writes an LZ4 frame stream (Lz4DeflatingWriteBuffer) at the level from the setting http_zlib_compression_level. That setting defaults to 3. The client does not send a level for Lz4, so the server compresses every Lz4 response at level 3, which is LZ4HC. LZ4HC is much slower than fast LZ4.

The new client-side frame decoder is not the cause. With http_zlib_compression_level=1, the same main build is as fast as 0.15.2.

Steps to reproduce

1. Server 26.4.5.143. Load 1M rows of ClickBench hits (105 columns, 748 MB as RowBinary):
   CREATE TABLE hits ENGINE = MergeTree ORDER BY tuple() AS SELECT * FROM url('https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_0.parquet')
2. Read all rows with the default client (Lz4), once with 0.15.2 and once with main (code example below).
3. Compare the elapsed time.

Expected behaviour

main reads at the same speed as 0.15.2.

Actual (same server and table, median of 5–8 runs):

The server alone shows the same result. curl, output to /dev/null, so there is no client cost:

?compress=1                                          2.54-2.68 s   151,664,849 bytes   (0.15.2 path)
?enable_http_compression=1  + Accept-Encoding: lz4   5.99-6.21 s   120,461,841 bytes   (main path, level 3 = default)
  ... + &http_zlib_compression_level=2               3.28-3.35 s   137,638,039 bytes
  ... + &http_zlib_compression_level=1               2.47-2.97 s   151,694,238 bytes
  ... + &http_zlib_compression_level=0               2.52-2.92 s   151,699,223 bytes

Levels 0 and 1 give the same size and speed as compress=1 (fast LZ4). Levels 2 and 3 give smaller bodies and are slower (LZ4's HC modes).

ClickHouse's nightly client benchmark shows the same regression on 2026-10-11. In that run, both builds ran on the same host, against server 26.4.1.2509, with the same benchmark revision:

main at fcb8b02 (before #482) measured 1650-1865 ms on SelectHits on 2026-09-29 and 2026-09-30.

Code example

use std::time::Instant;

#[tokio::main]
async fn main() -> clickhouse::error::Result<()> {
    let client = clickhouse::Client::default().with_url("http://localhost:8123");
    // Add `.with_option("http_zlib_compression_level", "1")` to get 0.15.2 speed on `main`.
    for _ in 0..5 {
        let t = Instant::now();
        let mut cursor = client.query("SELECT * FROM hits").fetch_bytes("RowBinary")?;
        let mut n = 0;
        while let Some(chunk) = cursor.next().await? {
            n += chunk.len();
        }
        println!("{} ms, {n} bytes decoded, {} bytes received", t.elapsed().as_millis(), cursor.received_bytes());
    }
    Ok(())
}

Error log

None. The queries succeed, but they are slower.

Query log

system.query_log, query_duration_ms per run of the code example. The server-side duration is equal to the client elapsed time, so the time is spent on the server:

caller                 http_zlib_compression_level      query_duration_ms per run
clickhouse-rs/0.15.2   not sent                         2652 2704 2707 2915 2939 2949 2988 3886
clickhouse-rs/0.15.3   not sent (server default 3)      6034 6099 6117 6257 6266 6269 6294 6322
clickhouse-rs/0.15.3   1                                2768 2808 2879 3006 3022

(0.15.3 is the version in Cargo.toml on main.)

Root cause

• src/query.rs:271-277, Query::do_execute: the Compression::Lz4 | Compression::Lz4Hc(_) arm sends enable_http_compression=1 and Accept-Encoding: lz4, but no http_zlib_compression_level. The Zstd arm (lines 279-287) sends its level.
• Server: Lz4DeflatingWriteBuffer::initialize (src/IO/Lz4DeflatingWriteBuffer.cpp) gives the HTTP compression level to the LZ4F preferences. LZ4F uses its HC compressor when the level is LZ4HC_CLEVEL_MIN or higher. In the server's bundled lz4 the HC modes start at level 2 (measured above).
• 0.15.2 sent compress=1. Its response compression did not depend on http_zlib_compression_level.

Suggested fix

For Compression::Lz4, send http_zlib_compression_level with a value below 2 (for example 1). This is the same thing the Zstd arm does with its level. With level 1, main reads as fast as 0.15.2 (measured above).

Constraint: http_zlib_compression_level is a setting. A readonly = 1 user can send a setting only with its current value. Verified on 26.4.5.143 (SELECT 42, Accept-Encoding: lz4):

readonly = 1 user, &http_zlib_compression_level=1  ->  Code: 164. DB::Exception: Cannot modify 'http_zlib_compression_level' setting in readonly mode. (READONLY)
readonly = 1 user, &http_zlib_compression_level=3  ->  OK (value not changed)
readonly = 2 user, &http_zlib_compression_level=1  ->  OK

This is the same constraint as in #486, and as for network_compression_method in #476. If the client always sends the level, every Lz4 SELECT fails for readonly = 1 users. So the fix for this issue and the fix for #486 must be chosen together.

Contrast cases to keep:

• Do not go back to compress=1 for Lz4. That brings back Default (LZ4) compression breaks on ClickHouse 26.9+: Lz4Decoder hardcodes magic 0x82 but the server default codec is now ZSTD(3) #468.
• Compression::Zstd(level) keeps the level that the user selected.
• Inserts do not change. The client compresses insert bodies itself, and InsertHits did not move.

Trade-off: at level 3 the body is about 21% smaller. In this test, level 3 is faster only on a link slower than about 25 MB/s (about 200 Mbit/s).

Workaround until a fix (writable users and readonly = 2 users): Client::with_option("http_zlib_compression_level", "1").

Configuration

Environment

• Client version: main at ca0ddbf (Cargo.toml version 0.15.3). Regression from d72ea54 (fix: use HTTP compression when requesting Lz4 for result sets #482). 0.15.2 is not affected.
• OS: Linux (Docker, Rust 1.89)

ClickHouse server

• ClickHouse Server version: 26.4.5.143 (local reproduction); 26.4.1.2509 (nightly benchmark)
• ClickHouse Server non-default settings, if any: none. http_zlib_compression_level = 3 and enable_http_compression = 1 are the server defaults.
• CREATE TABLE statements for tables involved: see "Steps to reproduce".

Found by automated analysis (Polyglot AI) of ClickHouse's nightly client benchmark. On 2026-10-11 it flagged clickhouse-rs SelectHits (+102%), SelectCollections (+166%) and SelectJSON (+80%). Verified against a live 26.4.5.143 server with the client built from main and from 0.15.2, not by inspection.

Related: #482, #486, #468, #487.