Repository navigation
docs: add a performance tuning section and updates for 1.4.0 (#491) - #558
Conversation
Collects the tuning advice that was spread across the reference tables, and adds the parts that were not documented anywhere, in order of how much each one usually gives you. New material, all of it from the macro-benchmark suite: - Materialization path. QueryAsync<T> is the box-free path; MapTo<T> and the ADO accessors are not. 1,372 MB against 3,133 MB on a 1M-row, 105-column read. ORMs drive the ADO reader, so they get the boxed path. - Batch size, with a warning. Large batches used to be a pessimisation and are now the fastest option, so tuning done against 1.3.0 or earlier is now inverted. Small batches cost far more on Cloud than on local disk. - Compression by direction. Responses: compress. Requests: measure first, as it cost time at every distance measured up to 27 ms RTT. - Server GC, which is worth up to 49% on insert-heavy work and nothing at all on reads, with the pause-length trade-off stated. - Cold start decomposed into its three round trips, to show that no driver setting moves it and only connection reuse does. - How to measure without misleading yourself, since several of these results reverse between loopback and a real network. Also warns against GZipCompressor's default level. It constructs at CompressionLevel.Fastest, and the server ingests that stream slowly enough that a large insert can pass the 120 s HttpClient timeout. InsertOptions' own documentation steers proxy-bound users toward gzip, so the default constructor is easy to reach. Corrects the ReadBufferSize row in the settings table, which still documented the 8 KiB default and a large-object-heap warning. The default has been 64 KiB and pooled since #451. Co-Authored-By: Claude <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Assumes #499 (ADO typed column slots) lands before this. With it, the reader keeps storage of the column's own type, so GetInt32/GetInt64/GetGuid and the scalar GetFieldValue<T> read a field instead of unboxing. GetValue and GetValues still box, because they return object. That splits the ORM advice rather than flattening it: linq2db registers the typed accessors per column and so reads without boxing, while a path that goes through GetValue does not. Co-Authored-By: Claude <noreply@anthropic.com>
⚠ Now depends on #499 — do not merge before itUpdated the read-path guidance to assume #499 (ADO typed column slots) lands first. This makes the PR wrong if merged on its own. The section now tells readers that What changed in the section:
Also worth knowing for review#550 (stacked on #499) removes the last per-value allocations on UUID, array elements and |
There was a problem hiding this comment.
Pull request overview
Adds task-oriented performance tuning guidance for reads, inserts, compression, GC, buffering, and measurement.
Changes:
- Documents performance tradeoffs and benchmark results.
- Corrects
ReadBufferSizedocumentation. - Adds a changelog fragment.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
docs/overview.mdx |
Adds performance guidance and updates buffer settings. |
changelog.d/491-performance-tuning.docs.md |
Records the documentation update. |
Suppressed comments (2)
docs/overview.mdx:1441
- The current typed accessors delegate to
GetValue, so linq2db still consumes values thatRead()already boxed inCurrentRow. Claiming that typed-accessor ORMs are box-free contradicts the implementation and the PR description's note that #499 has not merged.
- **ORMs get the fast path when they use typed accessors.** linq2db registers `GetInt64`,
`GetDouble` and `GetDateTime` for each column, so it reads without boxing. Code that reads through
`GetValue` — including a `dynamic` result from Dapper — boxes each value. If an ORM query is hot
and reads through `GetValue`, use `QueryAsync<T>` for that one query.
docs/overview.mdx:1516
- The existing rough guide at
docs/overview.mdx:1015-1020recommendszstd:1for same-region inserts, but this table says compression costs 18% in that same deployment category and the at-a-glance table suggests disabling it. These opposing recommendations need to be reconciled or explicitly scoped to different bandwidth, codec, or parallelism conditions.
| Client to server | Effect of compression |
|---|---|
| Same host (loopback) | Costs 30% |
| Same cloud region | Costs 18% |
| One region away | Costs 4% |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…607) * docs: sync the POCO read and buffer notes from the public docs repo Brings across the changes that were made directly in clickhouse-docs and never came back to this repo, corrected against the current code: - Supported property types for QueryAsync<T>, plus the nullable, wrapper and composite rules. - Map columns: the pair forms read on the box-free path in either MapReadMode, and a Dictionary property throws under KeyValuePairs. - How read value converters behave on each read path. - When each path checks the property type against the column. - The Large Object Heap effect of raising ReadBufferSize. The draft in the docs repo predated MapReadMode (#538) and stated several things the code does not do, so this is a rewrite rather than a copy. * docs: document the 1.4.0 changes that shipped without docs An audit of the 77 pull requests merged since 1.3.0 found 17 user-visible changes with no documentation. This covers all of them. Compression: - Constructor, level and Content-Encoding for each built-in compressor. - IClickHouseCompressor as an extension point, and the two members an implementation must supply. - Compressor instances are shared and thread-safe. - Where Accept-Encoding can be set, and which place wins. - A warning that the insert codec changed from gzip to zstd, and how to restore the earlier behavior. ADO.NET: - Value accessors now throw when there is no current row; the full list of members that throw, and of those that still work. - TryGetEnumOrdinal. - Which columns GetSchemaTable() gives a precision and a scale. - GetSchema("Columns"): the collection name, the restrictions, the result. - Which read value converter overload each accessor uses. Errors: - A new error handling section covering empty error bodies and failures that arrive after the response has started, with the limits of each. Types and parameters: - JSON typed paths hold a null instead of disappearing. - ReadStringsAsByteArrays reaches nested strings, but not into JSON columns. - The lexical rules for rewriting @name placeholders. - TimeOnly for Time and Time64, and byte[] query parameters. Inserts: - Binary inserts are chunked and carry no Content-Length. - MemoryStreamManager is obsolete. - The raw insert paths take ownership of the stream and dispose it. Every statement was checked against the source on this branch rather than taken from the pull request descriptions, several of which were out of date. * docs: trim the POCO read and buffer notes Shortens the sections added in 9e1e142 and drops the bold emphasis in the supported property types table. Also removes four statements: when each read path checks the property type against the column, the boxed read for composite columns, the Dictionary throw under MapReadMode=KeyValuePairs, and the Large Object Heap effect of raising ReadBufferSize. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Update overview.mdx * Delete changelog.d/1-4-0-doc-gaps.docs.md --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
MDX reads a bare {name:type} in prose as a JavaScript expression and cannot
parse it, which fails the Mintlify check. Backticks make it inline code, as
the other placeholder forms in this file already are.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes #491.
The issue asked for buffer size configs, compression codecs, POCOs vs boxed
object[], and skippingschema probes. Those knobs were each documented in their own reference table already, so rather than
repeat them this adds a task-oriented section that orders them by payoff, and fills in the parts that
were not written down anywhere.
What is new
Beyond the four items in the issue:
QueryAsync<T>is box-free;MapTo<T>and the ADO accessors are not. 1,372 MB against 3,133 MB on a 1M-row, 105-column read. Nothing said which to reach for.GZipCompressordefault level warningTwo things worth a maintainer's eye
1. A warning about
GZipCompressor's default level. It constructs atCompressionLevel.Fastest(zlib level 1), and the server ingests that stream slowly enough that a large insert can pass the
120-second
HttpClienttimeout. This is reachable:InsertOptions.Compressor's own XML doc says"
gzipremains the safer choice behind such a tier", so a proxy-bound user is steered toward gzipand the default constructor selects exactly the bad level. The default insert compressor is
zstd,so the out-of-box path is unaffected.
The underlying behaviour looks server-side rather than driver-side — replaying captured wire bytes
with
curl, no .NET involved, reproduces it. Worth deciding separately whetherGZipCompressorshould default to
Optimal.2. The insert-compression guidance may sit oddly next to the existing table. The existing
#tuning-compression"rough guide by deployment" recommendszstd:1for same-region inserts. Themeasurements say uncompressed was fastest at every distance tested (0.1 ms, 8.7 ms, 26.8 ms RTT),
costing 30% / 18% / 4%. Those are not strictly contradictory — the existing table reasons from
bandwidth thresholds and single-thread encode rates, and notes that parallel inserts move the
threshold — but a reader could land on both and be confused. I deliberately did not rewrite the
existing section, since it was revised recently on purpose. Flagging it for you to reconcile.
Drive-by fix
The
ReadBufferSizerow in the settings table still documented an 8 KiB default and alarge-object-heap warning. Since #451 the default is 64 KiB and the buffer is pooled, so the
warning no longer applies and the number was wrong.
Provenance
Every figure comes from the August 2026 macro-benchmark package (ClickBench
hits, 1M rows,105 columns, ClickHouse 26.x) across four environments spanning a 270x range of network latency. Each
number is captioned with the environment it holds in, because several of them invert.
Checks
<Note>/<Warning>tags balanced; code fences balanced.main:GZipCompressor(CompressionLevel = Fastest, ...),InsertOptions.Compressoris nullable with aZstdCompressor.Defaultdefault,RegisterPocoType<T>().mainspecifically: Box-free POCO read fast path #449 has merged soQueryAsync<T>is box-free, butBox-free ADO read path: typed column slots (stacked on #449) #499 has not, so the ADO accessors still box. The section says so and does not overclaim.
dotnet run scripts/changelog.cs -- --checkpasses.Draft because of the two open questions above.
🤖 Generated with Claude Code