Skip to content

TCP converters 5/6: POCO and untyped writes through the converter layer - #673

Open
alex-clickhouse wants to merge 13 commits into
alex-clickhouse/converter-4b-columnar-writesfrom
alex-clickhouse/converter-5-poco-writes
Open

alex-clickhouse wants to merge 13 commits into
alex-clickhouse/converter-4b-columnar-writesfrom
alex-clickhouse/converter-5-poco-writes

Conversation

@alex-clickhouse

@alex-clickhouse alex-clickhouse commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

This is part 5 of the stack of pull requests for ClickHouse/integrations#801, a converter layer for the native TCP
client. It is stacked on #672 (part 4b, columnar writes through the converter layer), which is on #671, #670, #668, #667,
#666 and #665. It moves the row inserts, InsertRowsAsync<T> (POCO rows) and InsertRowsAsync with object[] rows,
onto the converter derivation, and gives every insert tier and ClickHouseTcpTypes.CanWrite the write rules of POCO
inserts. It fixes ClickHouse/integrations#800 (the untyped insert refuses converted values for composite columns).

What changes for a user

  • One set of write rules for every insert. InsertAsync,
    InsertRowsAsync<T>, InsertRowsAsync with object[] rows, and ClickHouseTcpTypes.CanWrite accept the same CLR
    types. In addition to the type mappings, four rules apply to the whole column type:
    • a CLR enum is written as its integer ordinal (an enum Status : sbyte into Int8 or Enum8(...));
    • a value is written as a type that it casts to with no conversion (a string into a Variant, which is written from
      object; a subclass into the column of its base class; and, as the .NET cast allows, a uint[] into
      Array(Int32), which keeps the bits of each element);
    • a value type is written into a column of its nullable type (an int into Nullable(Int32));
    • a nullable value type is written into a column that cannot hold NULL (an int? into Int32), and the insert
      throws an InvalidOperationException at the first NULL, which names the column, its type and the row.
      Before this part, only InsertRowsAsync<T> applied them. InsertAsync and CanWrite refused these types, and the
      untyped insert took no enum.
  • ClickHouse/integrations#800: the untyped insert writes a composite column from every CLR type that the column type
    is written from, for example DateTime[] for Array(DateTime), DateTime for LowCardinality(DateTime),
    DateTime?[] for Array(Nullable(DateTime)), string labels for Array(Enum8(...)), and byte[] inside Array,
    Map and Tuple. Before, it chose a write type only from the codec's preferred write types.
  • The rules apply to every CLR type that the column type is written from, not only to the codec's preferred write
    types. So InsertRowsAsync<T> also takes a DateTime? property for LowCardinality(DateTime) (it took DateTime), a
    DateTime property for LowCardinality(Nullable(DateTime)) (it took DateTime?), and a nullable tuple for
    Tuple(DateTime, String).
  • The writes of the converter layer reach the row inserts: LowCardinality(String) and
    LowCardinality(Nullable(String)) from byte[] (ClickHouse/integrations#792; with the cast rule also from sbyte[]
    and from arrays of byte-sized enums), FixedString(N) from string, a Variant value placed in the alternative that
    is written from its type (a byte[] into the String alternative of Variant(Int32, String), which the row inserts
    failed to write before), and a LowCardinality dictionary that holds each distinct encoded value once.
  • Where a NULL fails: the row inserts find a NULL that the column cannot hold when they convert the rows of its
    block, before they send it, with their messages unchanged: the blocks before it are stored and the connection stays
    usable. InsertAsync finds it while it writes the column, inside the block: as for every value that InsertAsync
    cannot write, the insert ends its connection.
  • Everything else is unchanged: every CLR type that InsertRowsAsync<T> accepted is accepted, with the same bytes and the
    same failure texts, and the untyped insert gives the same outcome for every value that it accepted.

How

  • Write rules in the derivation. When the column type's own writes refuse a CLR type, the derivation tries the rules,
    at the root only, each through the column type's own write of another CLR type: EnumWriter (the source span read as
    ordinals with MemoryMarshal.Cast, span by span, never by an array cast), NullableEnumWriter, NonNullWriter (finds
    the first NULL before the inner writer starts), AssignWriter (a span of references read in place; value types boxed
    into a pooled buffer), LiftWriter (for Nullable(X), a null map of zeros and X from the same values, with no copy).
    The order follows the old POCO rule. The cast rule takes its types from the cast rule of the reads
    (ReadRules.IsAssignable), so the CLR array casts that read elements as another type are decided in one place for
    reads and writes (see Open question).
  • POCO inserts. The write plan derives the tree of each column type for the property type; a refusal keeps its old
    message. The gather copies each property value into the buffer of its column and checks a NULL that the column cannot
    hold, with the old message; the insert plan writes the buffer through the tree. A T? property into a column that is
    written from T is gathered as T, so its values are read once, into one buffer. Where the runtime compiles expression
    trees the gather is one compiled loop for each column; on a runtime without dynamic code it reads each property through a
    getter delegate and compiles no code. Both give the same values and the same first failure in row order.
  • Untyped inserts. The CLR type of the first value of a column that is not null selects its write type through the
    derivation: the type that a cast rule writes the values as (object for Variant and Dynamic, so values of any type
    still go there), the nullable type of a value type for a column that holds NULL, else the type of the values. The
    messages are unchanged.
  • Insert plan. Every column of an insert goes through InsertColumnWrite.For (part 4b), also the columns that the row
    inserts build. ClickHouseTcpTypes.CanWrite is whether the derivation succeeds, so it gives the answer of every insert.
  • LowCardinality(String) writes (a cost of part 4b, see Benchmarks): the CLR-key lookup of the dictionary
    interner is inlined into the write loop. A value that the lookup finds costs one dictionary lookup and one decrement;
    a new value goes to a method that is not inlined, with the conversion, the byte lookup, the cleanup after a refused
    value and the dictionary size for its message. The bytes and the messages do not change.
  • The old POCO write plan stays in production as the reference of the differential tests (LegacyPocoColumnBuilderFactory,
    LegacyPocoWriteConversion, PocoWritePlan.BuildLegacy); the old untyped type choice is copied into the test project.
    The last part of the stack deletes them.

Why

The POCO insert had its own write rules (PocoWriteConversion), which the other tiers did not know, and the untyped
insert asked the codec's preferred-type list, which is incomplete for composite types (ClickHouse/integrations#800). With
the converter derivation, every insert asks one question, and CanWrite asks the same one.

Tests

  • Differential tests. Four new tiers, PocoWrite, PocoCanWrite, UntypedWrite and UntypedCanWrite, run every
    write facet of the case list (the built, decoded and read-back columns, for all rows and for a tail that starts above
    row 0) through the row inserts, against the old row inserts: the POCO plan in both gather tiers, the untyped rows, and
    ClickHouseTcpTypes.CanWrite in both answer tiers, so CanWrite gives the answer of the row inserts on every facet. The
    declared changes: the four writes of part 4 that reach the row inserts, the nullable rules for two
    LowCardinality(Nullable(DateTime)) answers, and 42 untyped facets of ClickHouse/integrations#800. In the columnar
    tiers, the write rules give 8 writes and 13 CanWrite answers, the mirror of the 13 read-rule readings of part 2. The
    reference outcome counts of the old tiers do not change.
  • WriteRulesTests, 51 column types x 54 CLR types (enums of each width, nullable sources and targets, casts to
    object and to a base class, the array casts, tuples, maps, calendar types): the derivation accepts every type that the
    old POCO plan accepted plus 16 listed additions; for each type that the old plan accepted, both gather tiers write the
    same bytes or fail with the same exception and text, with and without a NULL, for all rows and from row 1 (one listed
    change: a byte[] value into Variant(Int32, String)); the columnar insert accepts exactly what the POCO plan accepts
    and writes the same bytes; the untyped insert gives the old outcome for every value that it accepted and agrees with
    CanWrite, also for later values of the type that a first value is written as through a cast; the write-side array
    casts are listed.
  • Unit tests of each rule writer over segments, under marked positions and from a row above 0; the first NULL and
    its message, before any byte; the gather tiers, a property getter that throws against a NULL in both orders and in one
    column, against the old plan; two lone surrogates in one LowCardinality dictionary entry through both row inserts.
  • No dynamic code: a child process with System.Runtime.CompilerServices.RuntimeFeature.IsDynamicCodeSupported: false
    chooses the delegate gather by itself and writes the bytes of the compiled gather, for POCO and untyped rows, and fails
    at the same NULL.
  • Against the server (two rows to a block, so later blocks are slices): the three shapes of
    ClickHouse/integrations#800 through object[] rows (each throws the issue's error with the old type choice, checked by
    hand); InsertAsync through each rule; InsertRowsAsync<T> with byte[] into LowCardinality(String), text into
    FixedString, and the nullable rules; a NULL into Int32 through each tier (POCO and untyped keep the block before
    it and the connection; InsertAsync names the row and ends its connection); five more CanWrite candidates.
  • Tests changed on purpose:
    • PocoWritePlanTests: four tests asserted the CLR type of the gathered column (the plan converted values in the
      gather); they now assert the bytes of the insert against the old plan. The plan tests run again in the delegate tier
      (PocoWritePlanDelegateTests).
    • ConverterDerivationTests.Derive_CompositeWriteOfAnotherShape_RefusesWithTheShapeItIsWrittenFrom,
      LeafTableTests.Derive_EveryCandidateType_AgreesWithTheCurrentAnswers and
      WriteDerivationTests.Derive_EveryCandidateType_AgreesWithTheCurrentCodec compare the column type's own writes,
      without the rules at the root, with the codecs (three refusal texts of the first are now writes of a rule).
    • InternerTests: the ClrKeyedByteInterner tests call Intern(value), because the method no longer takes the
      position of the value (the interner reads the dictionary slot of a refused value itself). No assertion changes.
  • A T? property into a column with no NULL: a matrix test checks, for every column type and nullable CLR type, that
    the type that the gather writes gives the tree inside the NULL check; the plan tests check the gathered column and the
    bytes of the old plan for int?, DateTime? and an enum in both gather tiers.
  • net9.0, full TCP suite with a server: 6032 passed on part 4b, 6391 passed here (+359), 0 failed, 0 skipped. net8.0 and
    net10.0, the offline suite (no server): 4067 passed each. The public API does not change.

Open question

Resolved in #678: the stack now refuses, in every tier and in both directions, an array cast that gives the elements another meaning (an integer type of the other sign, an enum over another integer type). It keeps an enum array from an array of its exact underlying type, so Array(Enum8(...)) still reads into a MyEnum[].

The cast rule accepts the CLR array casts that read the elements as another type: a uint[] written into
Array(Int32) keeps the bits of each element, and an array of byte-sized enums goes into LowCardinality(String) as
bytes. The reads of part 2 accept the same casts. Both directions take them from one place (ReadRules.IsAssignable),
so refusing them in every tier, reads and writes, is a change in that one method. This part keeps them, as
InsertRowsAsync<T> and QueryAsync<T> did before.

Review

  • Self-review: no blockers or majors. Fixed: the comments of the reference classes, one wrong comment, a docs sentence
    on later untyped values of a cast target, and its test.
  • Round 1: changes requested, one blocker, one major, one minor, all fixed. The blocker was the performance gate on
    Write.Insert(LowCardinalityStringHighRepeat) (see Benchmarks). The major was a second scan and a second buffer for a
    T? property into a column with no NULL: the gather now copies the T, and two benchmark shapes measure it. The
    minor was the length of the changelog entry.
  • Round 2: approved, no findings.

Benchmarks

Gate against part 0. The converter benchmarks of part 0 (TcpConverterRead, TcpConverterWrite), this part
against part 0, 4 rounds on a shared machine, each round one run of part 0 and then one run of this part. "ratio" is the
median of the four ratios of the medians (this part / part 0, below 1 is faster); "pairs" is the ratio of each round; a
shape fails when it is above 1.05 in every round.

base 77ca7eeb, head e31377c2, 4 rounds (base then head in each), limit 1.05
  round 1: base load average 5.82 to 3.36, CPU busy before 42% and after 13%; head load average 3.09 to 1.99, CPU busy before 14% and after 11%
  round 2: base load average 1.68 to 1.72, CPU busy before 13% and after 13%; head load average 1.58 to 1.48, CPU busy before 11% and after 11%
  round 3: base load average 1.40 to 2.66, CPU busy before 16% and after 14%; head load average 2.45 to 2.80, CPU busy before 11% and after 11%
  round 4: base load average 2.57 to 2.28, CPU busy before 12% and after 15%; head load average 2.10 to 2.09, CPU busy before 11% and after 12%

shape                                                                   base ms  ratio  pairs            spread base/head  verdict
TcpConverterRead.Columnar(ArrayStringAsBytes)                             0.549  0.716  0.726 0.702 0.724 0.708    10.7%/  8.6%  ok
TcpConverterRead.Columnar(DateTimeAsOffset)                               0.265  0.701  0.706 0.699 0.697 0.703     1.7%/  0.8%  ok
TcpConverterRead.Columnar(LowCardinalityNullableStringAsString)           0.031  1.016  1.027 1.020 0.891 1.012    14.4%/  1.2%  ok
TcpConverterRead.Columnar(LowCardinalityStringAsBytes)                    0.113  0.347  0.233 0.369 0.332 0.361    48.8%/ 14.9%  ok
TcpConverterRead.Columnar(NullableDateTimeAsOffset)                       0.300  0.632  0.641 0.628 0.578 0.635     7.6%/  3.1%  ok
TcpConverterRead.Columnar(StringAsString)                                 0.224  1.007  1.008 1.024 0.998 1.006     1.2%/  2.0%  ok
TcpConverterRead.Columnar(Wide)                                           1.421  0.792  0.787 0.778 0.798 0.835     4.7%/  3.3%  ok
TcpConverterRead.Poco(ArrayStringAsBytes)                                 0.685  0.676  0.674 0.692 0.662 0.678     6.9%/ 10.3%  ok
TcpConverterRead.Poco(DateTimeAsOffset)                                   0.283  0.991  0.995 0.981 0.987 1.023     4.0%/  1.4%  ok
TcpConverterRead.Poco(LowCardinalityNullableStringAsString)               0.255  0.451  0.440 0.462 0.437 0.476     2.9%/  7.0%  ok
TcpConverterRead.Poco(LowCardinalityStringAsBytes)                        0.209  0.596  0.613 0.556 0.606 0.586     4.4%/  5.9%  ok
TcpConverterRead.Poco(NullableDateTimeAsOffset)                           0.294  0.877  0.869 0.877 0.888 0.877     2.7%/  1.5%  ok
TcpConverterRead.Poco(StringAsString)                                     0.341  0.967  0.910 0.964 0.970 0.980     5.6%/  7.8%  ok
TcpConverterRead.Poco(Wide)                                               1.559  0.749  0.746 0.736 0.770 0.751     4.7%/  3.1%  ok
TcpConverterWrite.Insert(ArrayStringFromText)                             0.634  0.949  0.941 0.913 0.957 1.017     1.0%/ 10.5%  ok
TcpConverterWrite.Insert(LowCardinalityFixedStringBytesHighRepeat)        0.352  0.991  0.991 0.992 0.972 1.019     2.6%/  3.9%  ok
TcpConverterWrite.Insert(LowCardinalityFixedStringBytesLowRepeat)         1.277  0.494  0.506 0.496 0.492 0.489     1.7%/  2.9%  ok
TcpConverterWrite.Insert(LowCardinalityStringHighRepeat)                  0.308  0.964  0.967 0.946 0.972 0.961     2.5%/  0.5%  ok
TcpConverterWrite.Insert(LowCardinalityStringLowRepeat)                   1.274  0.815  0.825 0.805 0.845 0.801     5.7%/  3.9%  ok
TcpConverterWrite.Insert(NullableDateTimeFromOffset)                      0.156  0.404  0.404 0.403 0.404 0.420     1.5%/  4.7%  ok

largest run-to-run spread on one side: 48.8%
gate: PASS (0 shapes slower than 1.05 in every pair)
  • Writes: 0.40x to 0.99x. LowCardinality(String) with many repeats is 0.964x (part 4b was 1.028x; see below).
  • No read code changes in this part; the reads are 0.35x to 1.02x, as in parts 2b to 4b.

Row inserts against part 4b. The first commit adds TcpConverterRowWrite, the POCO and untyped row inserts of the
spike's write shapes, in memory: one operation opens the column source over 10,000 rows, plans the columns, gathers the
rows and writes one data block. Against part 4b with that benchmark commit (the old POCO and untyped gathers, written by
the codecs), measured on the last commit before the review fixes, which only make the LowCardinality(String) write and
the T? gather faster (three rounds on the final commit: 0.99x for both LowCardinalityStringHighRepeat rows). 4
rounds, limit 1.05: pass.

Shape ratio pairs
Poco(ArrayStringFromText) 0.943 0.942 1.003 0.884 0.943
Poco(LowCardinalityStringHighRepeat) 1.020 0.988 1.000 1.064 1.040
Poco(LowCardinalityStringLowRepeat) 0.829 0.802 0.848 0.830 0.827
Poco(NullableDateTimeFromOffset) 0.571 0.555 0.575 0.577 0.567
Poco(Wide) 0.874 0.918 0.897 0.845 0.852
Untyped(ArrayStringFromText) 0.949 0.963 0.962 0.872 0.935
Untyped(LowCardinalityStringHighRepeat) 1.046 1.008 1.027 1.066 1.065
Untyped(LowCardinalityStringLowRepeat) 0.844 0.807 0.868 0.852 0.835
Untyped(NullableDateTimeFromOffset) 0.674 0.660 0.677 0.670 0.680
Untyped(Wide) 0.895 0.901 0.894 0.896 0.890

The gather alone. A later commit that changes only the benchmark file adds two shapes of a T? property into a
column with no NULL, and its plan step uses InsertColumnWrite.For, as this part's insert does. On a base with this
PR's benchmark file on part 4b, the base also writes the gathered columns through part 4b's converter writers, so this
gate compares only the old gathers with the new ones (and the faster LowCardinality(String) write). 4 rounds, limit
1.05: pass.

Shape ratio pairs
Poco(ArrayStringFromText) 1.034 1.066 1.056 1.011 0.949
Poco(LowCardinalityStringHighRepeat) 0.941 0.942 0.928 0.949 0.940
Poco(LowCardinalityStringLowRepeat) 1.012 0.977 1.034 1.008 1.016
Poco(NullableDateTimeFromOffset) 0.989 0.981 1.086 0.962 0.996
Poco(NullableDateTimePropertyIntoDateTime) 0.985 0.983 1.024 0.987 0.980
Poco(NullableIntPropertyIntoInt32) 0.962 0.999 1.023 0.880 0.926
Poco(Wide) 1.007 0.938 1.016 0.998 1.032
Untyped(ArrayStringFromText) 0.998 0.949 1.102 1.002 0.994
Untyped(LowCardinalityStringHighRepeat) 0.981 0.963 0.972 0.989 0.996
Untyped(LowCardinalityStringLowRepeat) 1.012 0.988 1.019 1.007 1.016
Untyped(NullableDateTimeFromOffset) 0.979 0.968 1.008 0.991 0.957
Untyped(NullableDateTimePropertyIntoDateTime) 0.994 0.990 1.011 0.999 0.988
Untyped(NullableIntPropertyIntoInt32) 0.999 0.978 1.025 1.018 0.979
Untyped(Wide) 0.983 0.986 0.977 0.980 0.996

tcp: keep the CLR-key lookup of a LowCardinality string write inline is a commit of this part that removes a cost
that part 4b introduced and that this part increased. Part 4b's LowCardinality(String) write looked each value up
through a method that was not inlined (it had a try/catch) and read the dictionary size for each row. On
Write.Insert(LowCardinalityStringHighRepeat) (100 distinct values), part 4b was 1.028x part 0 in its gate, and this
part before the commit was 1.088x part 0 (pairs 1.090, 1.099, 1.086, 1.081) and 1.049x part 4b. This part changes no
code on that path, and the compiled code of the hot methods is the same; with tiered compilation on (a console harness),
part 4b and this part both take 1.21x the time of part 0. After the commit: 0.964x part 0 in the gate above, 0.943x part
4b (three rounds), and 0.99x part 0 with tiered compilation on. All distinct values: 0.815x part 0, 1.01x part 4b. The
bytes and the messages do not change.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Oct 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.94595% with 18 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ickHouse.Driver.Tcp/Types/Converters/WriteRules.cs 87.67% 8 Missing and 10 partials ⚠️

📢 Thoughts on this report? Let us know!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The converter and serialization changes are broad and protocol-sensitive despite strong tests and benchmark evidence.

0 open findings

What changed in this PR

This TCP-client PR routes POCO and untyped row inserts through the shared converter derivation, unifying write rules with columnar inserts and fixing composite-value handling.

Changes:

  • Adds enum, cast, nullable-lifting, and non-null enforcement writers.
  • Migrates POCO and untyped inserts to converter-derived write plans.
  • Adds extensive differential, integration, no-dynamic-code, and performance coverage.
File Description
docs/​tcp.mdx Documents unified write rules.
ClickHouse.Driver.Tcp/​Types/​Converters/​ReadRules.cs Clarifies shared cast logic.
ClickHouse.Driver.Tcp/​Types/​Converters/​NullableWriters.cs Exposes the inner nullable writer.
ClickHouse.Driver.Tcp/​Types/​Converters/​DictionaryWriters.cs Uses revised interner API.
ClickHouse.Driver.Tcp/​Types/​Converters/​ConverterDerivation.WriteRules.cs Implements root write-rule derivation.
ClickHouse.Driver.Tcp/​Types/​Converters/​ConverterDerivation.cs Applies rules to refused writes.
ClickHouse.Driver.Tcp/​Types/​Converters/​ClrKeyedByteInterner.cs Optimizes repeated-value lookup.
ClickHouse.Driver.Tcp/​Types/​ClickHouseTcpTypes.cs Documents unified CanWrite behavior.
ClickHouse.Driver.Tcp/​Protocol/​ClickHouseTcpConnection.cs Uses converter plans for row columns.
ClickHouse.Driver.Tcp/​Poco/​UntypedRowColumns.cs Derives untyped write types.
ClickHouse.Driver.Tcp/​Poco/​PocoWritePlan.cs Builds converter-backed POCO plans.
ClickHouse.Driver.Tcp/​Poco/​PocoWriteErrors.cs Centralizes row-write errors.
ClickHouse.Driver.Tcp/​Poco/​PocoWriteConversion.cs Retains shared codec facts.
ClickHouse.Driver.Tcp/​Poco/​PocoTypeRegistry.cs Caches plans by gather tier.
ClickHouse.Driver.Tcp/​Poco/​PocoGatherTier.cs Defines compiled/delegate gathers.
ClickHouse.Driver.Tcp/​Poco/​PocoColumnBuilder.cs Implements converter-backed gathering.
ClickHouse.Driver.Tcp/​Poco/​LegacyPocoWriteConversion.cs Preserves the differential oracle.
ClickHouse.Driver.Tcp/​Poco/​LegacyPocoColumnBuilderFactory.cs Preserves legacy POCO gathering.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​WriteRuleWriterTests.cs Tests individual rule writers.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​WriteRulesRegistration.cs Registers differential rule changes.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​WriteDerivationTests.cs Separates intrinsic writes from rules.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​LeafTableTests.cs Tests intrinsic leaf writes.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​InternerTests.cs Updates interner coverage.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​ConverterDerivationTests.cs Tests intrinsic refusals.
ClickHouse.Driver.Tcp.Tests/​Program.cs Adds no-dynamic-code write mode.
ClickHouse.Driver.Tcp.Tests/​Poco/​RowWriteRegistration.cs Adds row-write differential tiers.
ClickHouse.Driver.Tcp.Tests/​Poco/​PocoWritePlanTests.cs Expands write-plan coverage.
ClickHouse.Driver.Tcp.Tests/​Poco/​PocoWritePlanDelegateTests.cs Repeats tests through delegate gathering.
ClickHouse.Driver.Tcp.Tests/​Poco/​PocoWriteNoDynamicCodeTests.cs Tests writes without dynamic code.
ClickHouse.Driver.Tcp.Tests/​Poco/​PocoReadNoDynamicCodeTests.cs Generalizes child-process execution.
ClickHouse.Driver.Tcp.Tests/​Poco/​PocoColumnBuilderFactoryTests.cs Tests tiers and failure ordering.
ClickHouse.Driver.Tcp.Tests/​Integration/​WriteRulesIntegrationTests.cs Verifies rules against ClickHouse.
ClickHouse.Driver.Tcp.Tests/​Integration/​ClickHouseTcpTypesIntegrationTests.cs Extends CanWrite integration cases.
ClickHouse.Driver.Tcp.Tests/​Differential/​RowWriteArms.cs Implements row-write candidate arms.
ClickHouse.Driver.Tcp.Tests/​Differential/​LegacyRowWrite.cs Implements legacy references.
ClickHouse.Driver.Tcp.Tests/​Differential/​Expectation.cs Supports cross-tier write expectations.
ClickHouse.Driver.Tcp.Tests/​Differential/​DifferentialTests.cs Pins new outcome counts.
ClickHouse.Driver.Tcp.Tests/​Differential/​DifferentialRegistry.cs Registers additional write tiers.
ClickHouse.Driver.Tcp.Tests/​Differential/​DifferentialCase.cs Defines row-write facets.
ClickHouse.Driver.Tcp.Tests/​Differential/​ClientArms.cs Registers client and reference arms.
ClickHouse.Driver.Tcp.Tests/​Differential/​Arms.cs Generalizes write and answer arms.
ClickHouse.Driver.Benchmark/​TcpConverterRowWrite.cs Benchmarks row insert paths.
changelog.d/​converter-5-write-rules.improvements.md Records unified write behavior.
changelog.d/​800-untyped-insert-composite-values.fixes.md Records the composite insert fix.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@polyglotAI-bot

polyglotAI-bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR moves the native TCP client's POCO and untyped row inserts onto the converter layer and unifies enum, cast, and nullable write rules across inserts and CanWrite. No actionable findings were identified in the supplied snapshot. ✅ Approve. This was a static review; builds and tests were not run in the sandbox.

Final verdict

✅ Approve

Automated review by Polyglot AI (gpt-6-astra) of 75126e6. Replies on the inline threads are read on the next review. A maintainer can ask for a new review by mentioning @polyglotAI-bot or requesting a review from it.

alex-clickhouse and others added 13 commits October 11, 2026 17:23
TcpConverterRowWrite prices InsertRowsAsync<T> (the POCO write plan) and
InsertRowsAsync(object[]) (the untyped row columns) for one block of
10,000 rows, with the write shapes of the converter spike as rows:
LowCardinality(String) from string with many repeats and with every
value distinct, Nullable(DateTime) from DateTimeOffset?, Array(String)
from string[], and a wide row of six columns. One operation opens the
column source of the insert, plans the columns as InsertAsync plans the
columns of a row insert, gathers the rows, writes the data block and
flushes, in memory. Same attributes and category as TcpConverterWrite.

The performance gate of the converter stack can then compare the row
inserts of the next part with the current ones.

Part of ClickHouse/integrations#801.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the own writes of the whole column type refuse a CLR type, the
write derivation tries the rules of POCO mapping, through the column
type's own write of another CLR type:

- a CLR enum is written as its ordinal (EnumWriter, a span cast);
- a nullable enum as its nullable ordinal (NullableEnumWriter);
- a nullable value type into a type written from its value type, which
  throws at the first NULL, before the inner writer starts
  (NonNullWriter, with a message that names the column, the type and
  the row of the insert);
- a value into a type that it casts to with no conversion, such as
  object, a base class, or an array that the CLR reads as another array
  (AssignWriter; the targets come from WriteRules.CastTargets, filtered
  by ReadRules.IsAssignable, so the array casts that read elements as
  another type are decided in one place for both directions);
- a value type into a type written from its nullable type (LiftWriter;
  for Nullable(X) a null map of zeros and X from the same values).

The rules apply to the root only, as the read rules do, and through
every CLR type that the column type is written from, not only through
the codec's preferred write types. InsertAsync and CanWrite take them
now: the differential tests declare the 8 writes and 13 answers that
mirror the 13 read-rule readings. Three tests compare the column types' own
writes with the codecs, with no rule at the root (DeriveNode).

Part of ClickHouse/integrations#801.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
InsertRowsAsync<T> and InsertRowsAsync(object[]) use the converter
derivation, with the write rules:

- POCO: the plan derives the tree of each column type for the property
  type. A refusal keeps the old text. The gather copies each property
  value into the buffer of its column, and checks a null that the column
  cannot hold before the block is written, with the old text (a
  reference type into a type with no NULL, a nullable value type whose
  tree writes the value type). The converter tree of the insert plan
  writes the gathered values, so the enum, nullable and cast rules are
  the derivation's, not the plan's.
- Untyped rows: the write type of a column comes from the derivation for
  the CLR type of its first value that is not null: the type that a
  cast rule writes the values as (object for a Variant), the nullable
  type of a value type for a type that holds NULL, else the type of the
  values. A composite column is written from every CLR type that it is
  written from (ClickHouse/integrations#800).
- The insert plan writes every column through InsertColumnWrite.For,
  also the columns that a row insert builds.
- The gather runs as a compiled loop (PocoGatherTier.Compiled) where the
  runtime compiles expression trees, else with a getter delegate for
  each row (PocoGatherTier.Delegate), which compiles no code. The
  registry can force the tier for the tests.

The old POCO plan stays as the reference of the differential tests
(PocoWritePlan.BuildLegacy, LegacyPocoColumnBuilderFactory,
LegacyPocoWriteConversion); the old untyped choice is copied into the
test project (LegacyRowWrite). The differential tests get four tiers,
PocoWrite, PocoCanWrite, UntypedWrite and UntypedCanWrite, which run
every write facet for all rows and from a row above 0, against the old
row inserts, with ClickHouseTcpTypes.CanWrite in both answer tiers. The
declared changes: the added writes that reach the row inserts
(FixedString from text, LowCardinality(String) from byte[]), the
nullable rules for CLR types that are not the codec's preferred types,
and ClickHouse/integrations#800's untyped values. Four plan tests that
asserted the CLR type of the gathered column assert the bytes of the
insert against the old plan; the plan tests run in both gather tiers.

Part of ClickHouse/integrations#801.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WriteRulesTests runs 51 column types and 55 CLR source types (enums of
each width, nullable sources and targets, casts to object and to a base
class, the array casts that read elements as another type, tuples,
maps, calendar types):

- the derivation accepts every source that the old POCO write plan
  accepted, and the listed additions (the cast into
  LowCardinality(String) from byte[], the nullable rules for CLR types
  that are not the codec's preferred types);
- for each source that the old plan accepted, both gather tiers write
  the same bytes or fail with the same exception and text, with and
  without a NULL, for all rows and from row 1; one listed change, a
  byte[] value into Variant(Int32, String), goes to the String
  alternative;
- the columnar insert accepts a source exactly when the POCO plan does
  and CanWrite says true, and writes the same bytes;
- the untyped insert gives the old outcome where the old one accepted
  the values, and accepts exactly the values that CanWrite accepts;
- the array casts that the write cast rule uses are listed.

Part of ClickHouse/integrations#801.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- WriteRuleWriterTests: the writer that each rule gives; the refusal
  text of a source that no rule writes; types that no writer takes
  (pointer, by-reference, ref struct, open generic, void) refused with
  no exception; each rule writer against the writer of the type it
  writes as, over segments, from a row above 0, and under marked
  positions; a NULL into a type with no NULL stops the write at the
  first NULL, before any byte, and names the column, the type and the
  row.
- PocoColumnBuilderFactoryTests: the tier choice; a property getter that
  throws and a NULL give the first failure of the old plan in both
  tiers; a property of a base class in both tiers; two lone surrogates
  share one LowCardinality dictionary entry in both row inserts,
  where the old plan wrote two.
- PocoWriteNoDynamicCodeTests: the row inserts in a child process with
  dynamic code off choose the Delegate tier and write the bytes of the
  compiled gather of the test process, and fail at the same NULL. The
  child-process launcher of the read test takes the argument of the
  test.

Part of ClickHouse/integrations#801.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WriteRulesIntegrationTests, two rows to a block:

- the three shapes of ClickHouse/integrations#800 through untyped rows:
  Array(DateTime) from DateTime[], LowCardinality(DateTime) from
  DateTime, Array(Nullable(DateTime)) from DateTime?[]. With the old
  write type choice each insert throws "was given values of type ...,
  which it cannot be written from" (checked by hand);
- InsertAsync from a column that only a write rule writes: an enum as
  its ordinal (Enum8 and Int32), a nullable enum, a value type into
  Nullable and into LowCardinality(Nullable), a nullable value type with
  no NULL, a cast to object (Variant), an array cast that reads the
  elements as another type;
- InsertRowsAsync<T> with the writes of the converter layer:
  LowCardinality(String) from byte[], FixedString from text, the
  nullable rules into LowCardinality(DateTime) and
  LowCardinality(Nullable(DateTime));
- a NULL into Int32: the POCO and the untyped inserts fail at the gather
  with their texts, keep the block before it and the connection; the
  columnar insert fails with the column, the type and the row, and ends
  its connection.

ClickHouseTcpTypesIntegrationTests gets five CanWrite candidates for the
rules (and Int64 from int, which stays false).

Part of ClickHouse/integrations#801.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TcpConverterRowWrite repeats the column plan of InsertAsync by hand,
because the plan is private. The row inserts write their gathered
columns through InsertColumnWrite.For now, so the benchmark plans them
that way too, and measures the write path of the insert. The base of
the performance gate runs its own copy of the benchmark, which plans the
columns with the codec's CanWrite.

Part of ClickHouse/integrations#801.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/tcp.mdx gets a "Write rules" section: a CLR enum as its ordinal, a
cast, a value type into a nullable column, and a nullable value type
into a column that cannot hold NULL, which throws at the first NULL. It
says where each insert finds that NULL: the row inserts before the block
is sent, with the connection kept; InsertAsync while it writes the
block, which closes the connection. The columnar, row and POCO insert
sections, the writing tables and the CanWrite paragraph refer to the
rules, and the row inserts section says that the first value of a
column selects how the column is written.

Changelog fragments for ClickHouse/integrations#800 and for the write
rules in every insert tier (ClickHouse/integrations#801).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Self-review of the row inserts:

- The reference classes of the POCO write plan and of the row inserts
  say what they are (the reference path over the codecs' preferred write
  types), not when they were the client's path.
- ReinterpretedValues says why a segmented source is copied: the rules
  apply to the whole column type, whose source is one span.
- NonNullWriter says which row its message names.
- PocoGatherTier.Delegate says that it compiles no code and that the plan
  still closes generic types at run time.
- docs/tcp.mdx: the later values of an untyped column can also have a
  type that casts to the type that the first value is written as.
- WriteRulesTests: an untyped column whose first value is written
  through a cast (an IPAddress subclass, uint[] into Array(Int32),
  sbyte[] into String) takes later values of the target type, as the old
  insert did.

Part of ClickHouse/integrations#801.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ClrKeyedByteInterner.Intern ran once for each row of a LowCardinality(String)
write and was not inlined into BytesDictionaryWriter.WriteBody: its try/catch
for a refused value kept it out. Each call also read the dictionary size for
an error message, and counted the probe with an increment and two compares.

The hit path is now one CLR lookup and one decrement, and Intern inlines. A
new value, and every value when the probe has turned the CLR lookup off, go
to InternNew, which is not inlined. The conversion and the byte lookup are
written in InternNew itself: with tiered compilation on, the JIT inlines them
there, as it did into the old Intern, and a column of many distinct values
keeps its speed (a separate small method for the byte path lost that inlining
and was about 7% slower on that shape). The probe counts down to zero, and
its end is a method of its own. The dictionary size that names a refused
value is read only for a new value, which is the only time the leaf can
refuse it.

The bytes and the messages do not change: a refused value is still named by
the dictionary slot that it would take (ByteInterner.Count is the entry count
that the caller passed before). Intern loses its position parameter, so the
interner tests call it with the value only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…no NULL

Two shapes of TcpConverterRowWrite: an int? property into Int32 and a
DateTime? property into DateTime('UTC'), with no null in the rows. The
untyped rows of the same shapes hold a boxed int and a boxed DateTime.

The commit changes only the benchmark file, so the file can be put on an
older commit to measure that commit's write of these shapes with the same
benchmark code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s no NULL

For a T? property into a column that the derivation writes from T (a
NonNullWriter<T>), the POCO write plan gathered the property into a column
of T?, and the insert wrote it through the NonNullWriter, which scanned the
values for a NULL again and copied them into a second, pooled buffer of T.
The gather already refuses a NULL there, with the message of POCO mapping.

The gather now copies the T of each value into a column of T, in both
gather tiers, and the insert writes that column through the derivation of
T, which is the tree inside the NonNullWriter. The values are read once,
into one buffer. The columnar InsertAsync keeps the NonNullWriter, which
throws at Begin.

NonNullWriter<T>.Inner exposes the inner tree, and a matrix test checks
for every column type and nullable source that the derivation of the value
type gives a tree of the same type. The plan tests check the gathered
column type and the bytes of the old plan for int?, DateTime? and an enum.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The details of the rules stay in docs/tcp.mdx and the pull request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alex-clickhouse
alex-clickhouse force-pushed the alex-clickhouse/converter-5-poco-writes branch from 41c99f2 to 75126e6 Compare October 11, 2026 18:57

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants