TCP converters 4a/6: write combinators for the converter layer by alex-clickhouse · Pull Request #671 · ClickHouse/clickhouse-cs

alex-clickhouse · 2026-10-09T22:59:18Z

This is part 4a of the stack of pull requests for ClickHouse/integrations#801, a converter layer for the native TCP
client. It is stacked on #670 (part 3, POCO reads on the converter layer), which is on #668, #667, #666 and #665. It adds
internal code only: no tier of the client calls the new writers yet, so the client behaves as before. Part 4b moves
InsertAsync with the caller's columns and ClickHouseTcpTypes.CanWrite onto them. There is no public API change, no
changelog fragment and no docs change in this part.

What

Write combinators under ClickHouse.Driver.Tcp/Types/Converters/. Each writes its streams across all the values
of a write, in the three phases of a column write (begin, state prefix, body), from a value source of one span or of
segments (the arrays of a jagged column):
- Nullable: the null map, then the child with each NULL marked, so the child writes its canonical placeholder there.
- Array: the offsets, then the child over the arrays of the rows as segments, with no copy of the elements.
- LowCardinality: a dictionary of the distinct canonical values of the leaf, and one key for each value. The
  dictionary of a fixed-width leaf interns the canonical value (FixedInterner); the dictionary of String and
  FixedString interns the canonical bytes (ClrKeyedByteInterner), which looks a string up by its CLR value first
  and stops that lookup after 1,024 lookups when more than half of them failed. Two values share an entry exactly when
  they encode to the same bytes. An interner is made only for a body that has values.
- Map (KeyValuePair<TKey, TValue>[]): the keys and the values of all the rows, then the offsets, the keys and the
  values.
- Tuple (ValueTuple of 1 to 7 elements): one writer for each field.
- Variant (from object) and Dynamic, QBit, the geo types and Tuple(), which are written from their canonical
  type through their codec. Dynamic keeps DynamicTypeInference and the state that its codec shares between the
  prefix and the body.
The write derivation: ConverterDerivation.Derive(type, context, T, Write) builds these trees, or refuses with a
reason. It writes what the codecs write today, and two writes that they refuse: LowCardinality(String) from byte[]
(ClickHouse/integrations#792) and FixedString(N) from string under every composite. Nested is written only from
its own dense column, so its derivation refuses.
IColumnCodec.WritesFromStorage(column): the condition of each codec's existing dense write (a decoded String
column with its raw bytes, a dense LowCardinality, Map, Tuple, Variant, Dynamic, Nested or QBit column). Part 4b routes
such a column to the codec and every other column to the converters.

How

Variant placement. A value whose CLR type is the canonical type of an alternative goes to that alternative, by the
rule of the Variant codec, and ClaimsValue breaks a tie between alternatives that share the canonical type. Only a
value whose type is the canonical type of no alternative is placed by the derivation of each alternative: an
alternative that is written from the value's type takes it, with the same tie rule. So every write that works today
goes where it goes today, and Variant(String, UInt64) also takes a byte[] value as a String. A leaf alternative
writes the values of two CLR types (string and byte[]) in row order; a composite alternative takes one CLR type in
one write. The scratch of a Variant write holds one entry for each value, whatever the number of alternatives: the
writer counts the values of each alternative, then puts them in one array, grouped by alternative and in row order
inside each alternative. The Variant codec's own scattered write rents one buffer of the row count for each
alternative.
Error texts. The combinators throw the texts of the codecs: a null Array or Map row names the column and the
row, and a value that a leaf refuses names its row. Under LowCardinality, a refused value is named by the dictionary slot
that it would take, as the current writer names it (a decimal past its precision, a FixedString value of the wrong
width). One text differs: a null string in LowCardinality(String) throws ArgumentNullException with the
parameter name value, as the String write of a null does; the current LowCardinality writer gives key, the
parameter name of its dictionary. No existing test pins either text.
A cached tree keeps no buffer sized by a type parameter: every buffer is pooled for one write.

Why

Today each composite codec has its own family of write shapes for each CLR type, and a LowCardinality dictionary needs
a key strategy for each pair of write type and inner type, so a missing pair is a refusal
(ClickHouse/integrations#792). With the converter layer, the leaf table converts each value to its canonical value, and
the combinators write any CLR type that converts to the leaf.

Tests

The differential tests of part 0 run the write combinators for every write facet of a case whose column type is not a
leaf (349 facets: the built and read-back columns, for all rows and for a tail that starts above row 0; a column that
the codec writes from its storage goes to the codec), and the derivation's CanWrite answer (573 facets), against the
old path. Three writes are declared changes: LowCardinality(String) and LowCardinality(Nullable(String)) from
byte[], and LowCardinality(FixedString(4)) from string. Each writes the bytes of the case's source column, and the
answer is true.
WriteCombinatorTests compares 56 columns of every combinator with the current codec, over a span, over segments and
under the marks of Nullable(Tuple(...)), for all rows and from row 2; zero values; 28 refused values by exception
type, message and parameter name; bytes that UTF-8 cannot spell in LowCardinality(String); padded FixedString text;
two lone surrogates in one dictionary entry; keys of 2 and 4 bytes.
WriteDerivationTests compares the derivation with the codecs' accepted CLR types for 51 composite types and 63 CLR
types, with the listed additions; it bounds the allocation of deriving a very wide FixedString under each composite,
and runs one tree on eight threads.
VariantWriterTests covers the placement by canonical type and by derivation, the ties, two CLR types in a leaf
alternative, the prefix of an alternative that no value selects, a Variant of 64 alternatives against the codec from
row 3, and the allocation of one write of 4,000 values with the shared pool drained.
One test changed on purpose: the test that pinned "has no converter" for composite writes tests the composite refusal
texts, because composite writes exist.
net9.0, full TCP suite with a server: 5773 passed on part 3, 6010 passed here (+237), 0 failed, 0 skipped. net8.0 and
net10.0, the offline suite (no server): 3718 passed each. The public API does not change.

Review: two rounds; fixed and pinned by tests: the Variant write scratch scaled with alternatives times rows.

Changes after review

A Variant alternative groups its values by CLR type, so a full placement cache does not split one CLR type into two runs (Copilot, Cursor, Polyglot).
The comment of the placement cache gives its bound past the limit.

🤖 Generated with Claude Code

codecov · 2026-10-10T09:26:46Z

Codecov Report

❌ Patch coverage is 95.54707% with 35 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...use.Driver.Tcp/Types/Converters/NullableWriters.cs	91.30%	5 Missing and 1 partial ⚠️
...House.Driver.Tcp/Types/Converters/VariantWriter.cs	97.46%	4 Missing and 1 partial ⚠️
...ckHouse.Driver.Tcp/Types/Converters/ArrayWriter.cs	92.00%	3 Missing and 1 partial ⚠️
...lickHouse.Driver.Tcp/Types/Converters/MapWriter.cs	94.52%	3 Missing and 1 partial ⚠️
...kHouse.Driver.Tcp/Types/Converters/WriteBuffers.cs	85.18%	3 Missing and 1 partial ⚠️
....Tcp/Types/Converters/ConverterDerivation.Write.cs	98.27%	1 Missing and 1 partial ⚠️
...e.Driver.Tcp/Types/Converters/DictionaryWriters.cs	96.87%	1 Missing and 1 partial ⚠️
...kHouse.Driver.Tcp/Types/Converters/TupleWriters.cs	98.47%	2 Missing ⚠️
....Driver.Tcp/Types/Codecs/FixedStringColumnCodec.cs	0.00%	1 Missing ⚠️
...e.Driver.Tcp/Types/Codecs/JsonStringColumnCodec.cs	0.00%	1 Missing ⚠️
... and 4 more

📢 Thoughts on this report? Let us know!

Copilot

🟡 Changes recommended

Variant grouping can falsely reject repeated composite values after its placement cache reaches capacity.

1 open finding

Rejects repeated uncached CLR types after placement cache fills · New

What changed in this PR

Adds internal write combinators and derivation infrastructure for the native TCP converter layer, preparing a later migration of TCP inserts.

Changes:

Adds composite writers for arrays, maps, tuples, nullable, LowCardinality, Variant, and canonical codec-backed types.
Adds codec storage-path detection and write derivation.
Adds differential, concurrency, allocation, error, and wire-format tests.

File	Description
`ClickHouse.Driver.Tcp.Tests/Types/Converters/ConverterDerivationTests.cs`	Updates composite refusal tests.
`ClickHouse.Driver.Tcp.Tests/Types/Converters/ConverterHarness.cs`	Propagates test column names.
`ClickHouse.Driver.Tcp.Tests/Types/Converters/VariantWriterTests.cs`	Tests Variant placement and allocation.
`ClickHouse.Driver.Tcp.Tests/Types/Converters/WriteCombinatorTests.cs`	Tests composite writer behavior.
`ClickHouse.Driver.Tcp.Tests/Types/Converters/WriteConverterRegistration.cs`	Registers differential writer coverage.
`ClickHouse.Driver.Tcp.Tests/Types/Converters/WriteDerivationTests.cs`	Tests write derivation and concurrency.
`ClickHouse.Driver.Tcp/Types/Codecs/ArrayColumnCodec.cs`	Identifies dense array storage.
`ClickHouse.Driver.Tcp/Types/Codecs/DynamicColumnCodec.cs`	Identifies dense Dynamic storage.
`ClickHouse.Driver.Tcp/Types/Codecs/FixedStringColumnCodec.cs`	Identifies compatible fixed storage.
`ClickHouse.Driver.Tcp/Types/Codecs/ILowCardinalityShape.cs`	Adds storage-write contract.
`ClickHouse.Driver.Tcp/Types/Codecs/JsonStringColumnCodec.cs`	Identifies raw JSON string storage.
`ClickHouse.Driver.Tcp/Types/Codecs/LowCardinalityColumnCodec.cs`	Delegates storage detection.
`ClickHouse.Driver.Tcp/Types/Codecs/LowCardinalityShape.cs`	Detects dense dictionaries.
`ClickHouse.Driver.Tcp/Types/Codecs/MapColumnCodec.cs`	Detects dense map storage.
`ClickHouse.Driver.Tcp/Types/Codecs/NestedColumnCodec.cs`	Marks writable Nested storage.
`ClickHouse.Driver.Tcp/Types/Codecs/NullableColumnCodec.cs`	Detects decoded nullable storage.
`ClickHouse.Driver.Tcp/Types/Codecs/NullableLowCardinalityShape.cs`	Detects nullable dictionaries.
`ClickHouse.Driver.Tcp/Types/Codecs/QBitColumnCodec.cs`	Validates dense QBit storage.
`ClickHouse.Driver.Tcp/Types/Codecs/StringColumnCodec.cs`	Detects raw string storage.
`ClickHouse.Driver.Tcp/Types/Codecs/TupleColumnCodec.cs`	Detects dense tuple storage.
`ClickHouse.Driver.Tcp/Types/Codecs/VariantColumnCodec.cs`	Detects compatible Variant storage.
`ClickHouse.Driver.Tcp/Types/Converters/ArrayWriter.cs`	Writes array offsets and elements.
`ClickHouse.Driver.Tcp/Types/Converters/BytesLeafWriter.cs`	Marks byte leaves as flat.
`ClickHouse.Driver.Tcp/Types/Converters/CodecWriter.cs`	Adapts canonical values to codecs.
`ClickHouse.Driver.Tcp/Types/Converters/ColumnWriter.cs`	Adds flat-body metadata.
`ClickHouse.Driver.Tcp/Types/Converters/ConverterDerivation.Write.cs`	Derives composite writers.
`ClickHouse.Driver.Tcp/Types/Converters/ConverterDerivation.cs`	Enables composite write derivation.
`ClickHouse.Driver.Tcp/Types/Converters/DictionaryWriters.cs`	Writes LowCardinality dictionaries.
`ClickHouse.Driver.Tcp/Types/Converters/FixedLeafWriter.cs`	Marks fixed leaves as flat.
`ClickHouse.Driver.Tcp/Types/Converters/MapWriter.cs`	Writes map streams.
`ClickHouse.Driver.Tcp/Types/Converters/NullableWriters.cs`	Writes null maps and placeholders.
`ClickHouse.Driver.Tcp/Types/Converters/TupleWriters.cs`	Writes tuple field streams.
`ClickHouse.Driver.Tcp/Types/Converters/ValueSource.cs`	Carries column names into errors.
`ClickHouse.Driver.Tcp/Types/Converters/VariantWriter.cs`	Places and groups Variant values.
`ClickHouse.Driver.Tcp/Types/Converters/WriteBuffers.cs`	Centralizes pooled write buffers.
`ClickHouse.Driver.Tcp/Types/IColumnCodec.cs`	Adds storage-write detection contract.

🧠 Review effort: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

alex-clickhouse · 2026-10-11T18:59:10Z

+            int end = first + count;
+            for (int i = first + 1; i <= end; i++)
+            {
+                if (i < end && ReferenceEquals(groupedChildren[i], groupedChildren[start]))


Confirmed and fixed in 811dc4f. When the placement cache was full, each value of an uncached CLR type got a new child, and the run split compared the children by reference. The run split now also compares the CLR type of the children, after the reference check, so a write with cached placements has the same cost and no new allocation. A regression test fills the cache with 1,024 refused types, then writes two byte[][] values to Variant(Array(String), UInt64) and checks the bytes.

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f6ec2aa. Configure here.}

polyglotAI-bot · 2026-10-11T15:36:08Z

Summary
This PR adds internal write combinators and storage-routing predicates for the native TCP client. The previous Variant placement-cache finding is fixed, with a regression test covering repeated derived values after the cache fills. I found no remaining issues requiring changes. ✅ Approve. This was a static review; builds and tests could not run in this sandbox.

Final verdict

✅ Approve

_{Automated review by Polyglot AI (gpt-6-astra) of 76b7254. Replies on the inline threads are read on the next review. A maintainer can ask for a new review by mentioning @polyglotAI-bot or requesting a review from it.}

A composite writer names the column in a message about a row (a null Array or Map row), so the value source carries the column name and the composites give it to the sources of their children. A leaf writer is flat: two writes one after the other give the body of one write, which a Variant alternative that takes values of two CLR types needs. Part of ClickHouse/integrations#801.

…ap and Tuple Each combinator writes its streams across all the values, from a value source of one span or of segments: - Nullable: the null map, then the child with each NULL marked, so the child writes its canonical placeholder there. A value type is copied out of T? into a pooled buffer. - Array: the offsets, then the child over the rows' arrays as segments. - LowCardinality: a dictionary of the canonical values of the leaf (FixedInterner, or ClrKeyedByteInterner made only for a body with values) and the keys. A refused value is named by the dictionary slot that it would take, as the current writer names it. - Map: the keys and the values of all the rows in pooled buffers. - Tuple: one writer for each field, arity 1 to 7, with the marks of the source in every field. No tier calls them yet. Part of ClickHouse/integrations#801.

… Dynamic through its codec VariantWriter places a value in the alternative whose canonical CLR type is the value's type, as the Variant codec does, with ClaimsValue for a tie between alternatives that share it. A value whose type is the canonical type of no alternative goes to the alternative whose derivation writes from its type, with the same rule for a tie. A leaf alternative writes values of two CLR types in row order; a composite alternative takes one CLR type in one write. CodecWriter writes Dynamic, QBit, the geo types and Tuple() from their canonical type through their codec: Dynamic keeps DynamicTypeInference and the state that its codec shares between the prefix and the body. Part of ClickHouse/integrations#801.

The derivation builds a write tree for Nullable (T? and a reference type), Array, LowCardinality (over a fixed-width or a byte-run leaf; the dictionary of LowCardinality(Nullable(X)) holds the bare X), Map, Tuple, and Variant (from object). Nested is written only from its own dense column, so its derivation refuses. Dynamic, QBit, the geo types and Tuple() are written from their canonical type through their codec. LowCardinality(JSON) is refused, as the codec refuses it. The derivation adds two writes that the codecs refuse: LowCardinality(String) from byte[] (ClickHouse/integrations#792), and FixedString from string under every composite. No tier calls it yet. ConverterDerivationTests.Derive_CompositeWrite_RefusesWithNoConverter becomes a test of the composite refusal texts, because composite writes exist now. Part of ClickHouse/integrations#801.

IColumnCodec.WritesFromStorage(column) is the condition of each codec's dense write: a decoded String or FixedString column (its raw bytes), a decoded Nullable column, a dense Array, LowCardinality, Map, Tuple, Variant (same alternatives), Dynamic, Nested or QBit column. The insert tier will route such a column to the codec and every other column to the converters. No behavior change. Part of ClickHouse/integrations#801.

WriteConverterRegistration registers two arms for the cases whose column type is not a leaf: 'Composite write converters: Write' (349 facets: the built and read-back inputs, all rows and a tail that starts above row 0; a column that the codec writes from its own storage goes to the codec) and 'Composite write converters: CanWrite' (573 facets). It declares the three writes that the old path refuses and the derivation accepts: LowCardinality(String) and LowCardinality(Nullable(String)) from byte[] (ClickHouse/integrations#792), and LowCardinality(FixedString(4)) from string. Each gives the bytes of the case's source column, and CanWrite says true. Part of ClickHouse/integrations#801.

WriteCombinatorTests: 56 columns of every combinator, over a span, over segments and under the marks of Nullable(Tuple(...)), whole and from row 2, against the current codec; zero values; 28 refused values (type, message, parameter name), the LowCardinality ones named by their dictionary slot; the added writes (bytes that UTF-8 cannot spell, padded FixedString text, two lone surrogates in one entry); key widths of 2 and 4 bytes. WriteDerivationTests: the derivation against CanWriteElementType for 51 composite types and 63 CLR types, with the listed additions; allocation bounds for a very wide FixedString under each composite; one tree on eight threads. VariantWriterTests: placement by canonical type first, by derivation next, the ties, two CLR types in a leaf alternative, the prefix of an alternative that no value selects. Part of ClickHouse/integrations#801.

The Variant writer rented a value buffer and a writer buffer of the row count for each alternative, so a Variant of 255 alternatives and a block of 50,000 rows reserved about 25 million references for each write. It counts the values of each alternative first, then puts the values together in one array, grouped by alternative and in row order inside each alternative, with one array for their writers. The scratch is three buffers of the row count, whatever the number of alternatives. Tests: the bytes of a Variant of 64 alternatives against the current codec, for all rows and from row 3; the allocation of one write of 4,000 values with the shared pool drained (4.3 MB with the old layout, under 512 KB with the new one). Review round 1, F1. Part of ClickHouse/integrations#801.

When the placement cache of a Variant writer holds 1,024 CLR types, a placement of a type that the cache does not keep makes a new writer of the alternative for each value. The runs of an alternative were split by writer identity, so two values of one composite CLR type made two runs, and the write refused them as values of two CLR types. In one alternative, the CLR type of a value selects its writer, so the runs are now split by CLR type. Children that are the same object still compare first, so the cost of a write with cached placements stays the same. Test: 1,024 refused values of distinct ValueTuple types fill the cache of a Variant(Array(String), UInt64) writer, then values of byte[][] give the bytes of one Array(String) run. Bot review of 2026-10-11, F9. Part of ClickHouse/integrations#801. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The GitHub review of the converter core (ClickHouse/integrations#801) asked about the limit of the derivation cache, whose count check does not lock. The cache of the Variant writer's placements has the same check, and its comment did not say that it can pass the limit. It now gives the bound: each call adds one entry at most after its check, so the cache passes the limit by fewer entries than the number of calls that run at the same time. The code does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This was referenced Oct 9, 2026

TCP converters 4b/6: columnar writes through the converter layer #672

Open

TCP converters 5/6: POCO and untyped writes through the converter layer #673

Open

TCP converters 6a/6: delete the old read members and the reference paths #674

Open

alex-clickhouse added this pull request to stack #675 October 10, 2026 09:23

This was referenced Oct 10, 2026

TCP converters 6b/6: write through the converter layer everywhere and delete the old write members #677

Open

TCP converters 6c/6: refuse array casts that change the sign of the elements #678

Open

alex-clickhouse marked this pull request as ready for review October 11, 2026 15:14

alex-clickhouse requested a review from mzitnik as a code owner October 11, 2026 15:14

Copilot AI balanced review requested due to automatic review settings October 11, 2026 15:14

Copilot started reviewing on behalf of alex-clickhouse October 11, 2026 15:14

View session

Copilot AI reviewed Oct 11, 2026

View reviewed changes

cursor Bot reviewed Oct 11, 2026

View reviewed changes

Comment thread ClickHouse.Driver.Tcp/Types/Converters/VariantWriter.cs Resolved

polyglotAI-bot reviewed Oct 11, 2026

View reviewed changes

Comment thread ClickHouse.Driver.Tcp/Types/Converters/VariantWriter.cs Resolved

alex-clickhouse and others added 10 commits October 11, 2026 17:23

alex-clickhouse force-pushed the alex-clickhouse/converter-4a-write-combinators branch from f6ec2aa to 76b7254 Compare October 11, 2026 18:57

Navigation Menu

Repository navigation

TCP converters 4a/6: write combinators for the converter layer - #671

TCP converters 4a/6: write combinators for the converter layer#671
alex-clickhouse wants to merge 10 commits into
alex-clickhouse/converter-3-poco-readsfrom
alex-clickhouse/converter-4a-write-combinators

alex-clickhouse commented Oct 9, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

alex-clickhouse Oct 11, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Uh oh!

polyglotAI-bot commented Oct 11, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

alex-clickhouse commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Why

Tests

Changes after review

Uh oh!

codecov Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

alex-clickhouse Oct 11, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

polyglotAI-bot commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Final verdict

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alex-clickhouse commented Oct 9, 2026 •

edited

Loading

codecov Bot commented Oct 10, 2026 •

edited

Loading

polyglotAI-bot commented Oct 11, 2026 •

edited

Loading