[alex-clickhouse]

This is part 6b of the stack of pull requests for ClickHouse/integrations#801, a converter layer for the native TCP client.
It is stacked on #674 (part 6a), which is on #673, #672, #671, #670, #668, #667, #666 and #665. Part 6a deleted the old read
members. This part moves the last production users of the codecs' write paths to the converter derivation, with the
same bytes, and then deletes the write members, so that IColumnCodec is the wire layer only. It also fixes the insert
of a column that a query read into a column of another DateTime64 or Time64 scale or Enum definition, which main
writes as raw counts or ordinals. Last, it trims the stack: code that kept the old failure order of a column with several
failing rows, and the plug-in machinery of the differential tests, go (see "Trim").

internal interface IColumnCodec
{
    string TypeName { get; }
    Type ElementType { get; }
    ValueTask ReadStatePrefixAsync(ClickHouseBinaryReader reader, CancellationToken cancellationToken) => default;
    ValueTask<IColumn> ReadColumnAsync(ClickHouseBinaryReader reader, string columnName, string columnType, int rowCount, CancellationToken cancellationToken);
    bool WritesFromStorage(IColumn column) => false;   // true only for a column that the codec writes from its storage
    bool ClaimsValue(object value) => true;   // breaks a tie between Variant alternatives
    IColumnWriteState BeginWrite(IColumn column, int start, int length) => null;
    void WriteStatePrefix(ClickHouseBinaryWriter writer, IColumn column, int start, int length, IColumnWriteState state) { }
    void WriteColumn(ClickHouseBinaryWriter writer, IColumn column, int start, int length, IColumnWriteState state);
}

A codec reads a column, and writes a column from its storage: the column class that a query of the type reads, or a
dense composite column whose child columns the child codecs write from their storage, with the same parameters where a
parameter gives a stored value its meaning. Such a write keeps the bytes that the server sent. The converter layer
writes every other column, in every insert.

What moves to the converter layer

• Dynamic: the inference of a type for each value, then one derived writer for each inferred type.
• Point, Ring, LineString, Polygon, MultiLineString, MultiPolygon and Geometry: the derived writers of
  their structures (Tuple, Array, Variant).
• QBit and Tuple(): their own writers.
• ClickHouseTcpColumn.CreateArray over a column of your own: the offsets, then the inner column through the element
  writer of the array's tree, with no copy of the rows into arrays.
• A column that implements IColumn<T> for more than one T: the first of those types in the list of CLR types that
  the insert's refusal message names for the target type.
• The NULL checks of the row inserts ask the type whether it holds NULL.

Before the move, a new test file pinned the bytes of every route: 97 column types and CLR types, for a write from row 0
and from a later row, and 21 row inserts with a null value, through the insert plan. The pins pass after the move,
except the two declared changes below.

What changes for a user

• Fix. InsertAsync with a column that a query read, into a column of another type: a DateTime64(3) column into a
  DateTime64(6) column was written as its counts, so the server stored other instants (milliseconds read as
  microseconds). Time64 of another scale and Enum8/Enum16 of another definition (an ordinal means another label)
  were the same. The bug is on main too, for the re-insert of a read column. Now:

  • the codec writes a read column from its storage only when the scale of a DateTime64 or Time64 and the members of
    an Enum are the same (a timezone does not count: a count is an instant);
  • else the insert converts the values through their meaning: it reads them as DateTimeOffset, TimeSpan or the
    label, also inside Nullable, LowCardinality, Array, Map and Tuple, and writes them with the rules of the
    target type (a label that the target does not declare, and an instant finer than its scale, are refused). In a
    composite only those parts are converted: the offsets, the null map and every other part are written as they were
    read, so a String next to them keeps bytes that are not UTF-8;
  • a column of a scale above 7 into another scale, and a Variant or Nested of such types into another type, are
    refused before the insert sends data, with the reason: no CLR type holds those values without a loss.

  The other parameters were checked: Decimal stores CLR values, which the codec encodes at the scale of the target;
  FixedString and QBit compare their width and layout and refuse a value of another width or dimension; DateTime
  and Date/Date32 store instants and dates. A server test reads a column from one table and inserts it into a table
  of another scale, timezone, enum definition or precision, then reads back the values.

• Fix. InsertAsync with a String column that a query read, into a LowCardinality(String),
  LowCardinality(Nullable(String)) or Nullable(String) column: the insert wrote the text of the values, so bytes that
  are not valid UTF-8 were stored as U+FFFD (main does the same). Such a column, and such a part of an Array, Map
  or Tuple, is written from its bytes. A FixedString(N) column, which main refuses for such a column, takes it when
  each value is exactly N bytes, as for any byte[]. A column of string values that you build keeps the text rules
  (a shorter FixedString value is padded), and a String into a String keeps the bytes that the server sent.

• Fix. InsertAsync with a column that a query read, whose NULL row hides a value that the target type cannot hold
  (for example a Nullable(Decimal(18, 2)) column after nullIf, into a Nullable(Decimal(9, 2)) column): the insert
  converted the hidden value and failed with an OverflowException (main does the same). The row gets the placeholder
  of the target, with no conversion and no check, also inside a Nullable(Tuple(...)) and in the Array and Map
  elements of such a row. A part of the same type keeps the bytes that the server sent.

• InsertAsync with a column that implements IColumn<T> for more than one T: a Nullable(Int32) column that also
  implemented IColumn<string> threw InvalidOperationException, and a FixedString(N) column of string values that
  also implemented IColumn<int> was refused. Both are written now, as int? and as string. The rule is in the docs
  ("Columnar inserts").

• A decoded Nullable column inserted into a column of a related type (for example Nullable(Enum8('a' = 1, 'b' = 2))
  into Nullable(Int8)) has the placeholder of the target type under each NULL, not the value that the source column
  held there. The values that the server stores are the same.

• An insert that refuses a QBit vector names its row in the column, also in a block after the first. The refusals of a
  Geometry value come from the Variant writer, with the type name Geometry; the exception types are the same.

• Writes of Dynamic, of the geo types and of CreateArray columns over a column of your own are faster (measured
  below).

What is deleted

• From IColumnCodec and the codecs: WritableElementTypes, CanWriteElementType, NullPlaceholder,
  NullPlaceholderAs, LowCardinalityKeyWriter, the WriteStatePrefix and WriteColumn overloads with no state, and
  the old CanWrite ("does the codec take a column of this CLR type"). The storage check keeps its name,
  WritesFromStorage.
• The write shapes of Array, Nullable, LowCardinality and Map, LowCardinalityValueWriter, LowCardinalityKeys,
  the substitute, concatenated, tuple-field and projected columns, ISpanWritableCodec, the write half of the composite
  element projections, the scattered writes of Variant and Dynamic, the transposition of QBit, and the conversions
  of every leaf codec write (with the helpers that only they used).
• ColumnCodecExtensions and ArrayColumn.OverBuffer, which only tests used after the move.

The Dynamic codec writes a decoded column only when the codec of each type writes its type column from its storage,
as the Variant and Nested codecs do.

Trim

Two cleanups at the end of this part, with no change to values, bytes, accepted types or the error text of a single
failing row:

• The failure order. The read paths kept the failure order of the old code when several rows of a column fail: a
  tuple reader, the map reader and the Fill tier of POCO mapping read the rows again one at a time after a bulk read
  failed, and a LowCardinality reader had two failure orders for its dictionary. These replays ran only on failure
  and existed only for that order, so they are gone (about 180 lines of production code). The write side had no such
  code. The visible change: a column with two or more failing values can report another one of them than main, and
  Block.ReadAs<T> no longer fails on a LowCardinality dictionary entry that no row uses (as QueryAsync<T> on
  main). A changelog fragment and docs/tcp.mdx say so.
• The differential tests. Their registry, arms with facet counts and declared outcomes were built to compare the old
  path with the new one, which part 6 removed. Four direct tests over the same case list now check what they still
  checked: the read tiers agree, the inserts agree (with an independent write that gathers each slice through the
  indexer, which checks the slicing of the insert), a decoded column written again decodes to its rows, and the
  scenario cases give their stated outcome; a fifth test pins the case list. Every tier and range follows one rule: each
  row is also read or written alone, and a read or a write fails exactly when one of its rows fails alone, with the
  failure of one of them, and else gives the values of its rows. 3,443 lines become 1,993, with the same line
  coverage (no line or branch lost) and with every hand mutation of the insert slicing, the Fill and Emit row
  handling, the POCO row numbers and the decoded-column checks still failing the suite.

The failure contract, in every tier:

• One failing row, dictionary entry or element: the read or the write throws its failure, with its exception type,
  message and row number. The tests pin these exactly.
• Several failing rows: the failure of one of them. Which one is not specified, and it can differ between
  Block.ReadAs<T>, the two tiers of POCO mapping, a slice and a full write. A message that names a row names the row of
  that value. The tests accept the failure of any one of the failing rows.
• A value that a NULL hides is never converted or checked, and a read or a write never succeeds when a row that it
  needs fails.
• A LowCardinality dictionary entry that has no reading fails a read only when a row of the read uses it.

Tests

The test layers follow one policy, written for this part: a test of a deleted member goes; a value test goes when an
InsertRoundTripCase case (a server round trip of the same type and CLR type) or the differential tests cover it; the
values that nothing covered become InsertRoundTripCase cases; bytes that no round trip can observe are pinned as
literals; and the tests of the codecs' dense writes stay.

• First, cases and pins, on the code before the deletions. 12 InsertRoundTripCase cases for the values that only
  codec tests held: a DateTime of DateTimeKind.Local, DateTime64 from DateTimeOffset at the scales 0, 6, 7 and 8,
  a DateTime64 wall clock in an hour that occurs twice, Decimal values at the declared precision (Decimal(1, 0),
  Decimal32(0), Decimal64(0), Decimal(19, 2), Decimal128(0)), and a float that BFloat16 cannot hold. One lift
  case, Nullable(Tuple(DateTime('UTC'), String)), for the differential tests. The converter tests that compared a
  derived write with the codec write compare with literal bytes and texts, with bytes computed from the plain wire form,
  or with the values that the codec reads back. These two commits pass with the codec write paths still in place, so
  every literal is the output of the write that it replaces.
• Then the deletions. The codec unit tests keep the reads, the type resolution, the storage check WritesFromStorage (rejections:
  a child that a caller built, another arity, another order of Variant alternatives, a Nothing child, a
  LowCardinality column without the NULL slot) and the dense writes, which go through a helper that fails when the
  codec does not write the column from its storage. A few tests of bytes or refusals that nothing else pins stay in the
  codec test files and write through the insert plan.
• An earlier commit of this part removes the test layers that duplicated the client baseline after part 6a: the read and
  answer arms of the converters, the declared outcomes, the second run of the POCO read server suite through the Fill
  tier, and a duplicate rule matrix (−381 tests; measured, no line or branch of coverage lost).
• Numbers, net9.0, the full TCP suite with a server: 6280 on part 6a; 6497 after the route pins and literal pins; 6116
  after removing the duplicate layers; 5610 after the deletions (the same on net8.0 and net10.0); 5747 with the three
  fixes; 5749 after the failure-order trim; 6603 after the fixture trim; 6614 at the tip, with the changes after review, 0 failed, 0 skipped. The last step adds tests although
  it removes lines: each of the three per-case differential tests is one test for each of the 416 cases. Of the codec unit tests, 475 cases go to 334; of the converter tests that used the codec write as the
  reference, 673 cases go to 376. The suite takes 2 m 25 s here (2 m 16 s on main for 4186 tests); the sum of the test
  durations is the same as before the deletions (132 s).
• The fix adds 91 tests: the storage checks and the comparison of meanings, the conversion through the insert plan
  (whole columns and slices, with the refusals), byte pins of the parts that keep their bytes (strings that are not
  UTF-8, NaN payloads, -0, a DateTime64(9), Decimal, UUID, IPv6, Enum, LowCardinality dictionaries, in
  Tuple, Map, Array, Nullable(Tuple)), and 20 server cases. The fix of the String bytes adds 28: byte pins of
  each source and target string type, also in Array, Map and Tuple, the read bytes against the text of a column
  that you build, and 12 server cases with unhex('FF'). The fix of the hidden values adds 15: byte pins with a hidden
  Decimal out of range (also at a row where a slice starts), hidden enum ordinals, a hidden DateTime64 finer than the
  target and hidden strings of another width, and 2 server cases with nullIf.
• Line coverage of ClickHouse.Driver.Tcp with the whole suite: 95.9%, the same before and after the trim of the
  differential tests.
• Review: three rounds, all findings fixed and pinned by tests. Round 1: a column that a query read as a composite with a
  part of another scale was converted as a whole, so a String part that was not UTF-8 was stored as U+FFFD; only the
  parts whose values mean other values are converted, and a read String into another type of a string is written from
  its bytes. Round 2: a value that a NULL hides was still converted by a part of another type; it gets the placeholder
  of the target. Round 3: approved; one more test pins a hidden value at the row where a slice starts. A fourth round on
  the trim: a row insert that failed counted as agreeing with the insert; the differential tests now require each row
  insert to give the outcome of the insert, and pin the one case where it legitimately differs. A fifth round: a
  Block.ReadAs failure was compared by its kind only; every tier and range now follows one rule against its rows read
  or written alone, and a mutation of each kind in each tier fails the differential tests. A sixth round approved; a
  failure of a row insert is now compared with the row of the source that it names, not with its numbers masked.
• ClickHouse.Driver.Tcp/AGENTS.md describes the two layers (where a leaf pair goes, how a combinator is built, the
  Fill/Emit rule, the differential tests) and what each test layer asserts.

Lines

Diff	Production (ClickHouse.Driver.Tcp)	Tests	Benchmarks	Docs	Changelog
This part	+2905 / −4106	+5688 / −7438	+190 / −2	+116 / −31	+26
of which the trim	+52 / −230	+1121 / −2470	0	+18 / −7	+5
The stack from main	+12009 / −5573	+12904 / −4282	+1035	+238 / −44	+57

Net, the stack from main: +16,344 lines, of which +6,436 production and +8,622 tests. Before the trim: +17,855, of
which +6,614 production and +9,971 tests.

Benchmarks

The route benchmarks that this part adds, against the code before this part (the first gate below, ratio new / old):
CreateArray of Nullable(Int32) 0.38, Dynamic of mixed kinds 0.82, LowCardinality(UUID) 0.68 (100 distinct values)
and 0.76 (all distinct), CreateArray of LowCardinality(String) 0.95, LowCardinality(UInt32) 1.03 (100 distinct).
Measured outside BenchmarkDotNet, 100,000 rows: Point 0.07, Ring 0.11, Geometry 0.46, QBit(Float32, 16) 0.82; every
fixed-width LowCardinality leaf at 0.31 to 1.05 of the codec write that it replaces.

Two performance gates, measured on b15ba584 (the later commit changes tests only), 4 rounds each, each run alternately
on base and head on the same machine. A shape fails when its ratio head/base is above 1.05 in every pair; check marks a
shape above 1.05 in some pairs only.

Against the code before this part: 0006c365 is a local commit (not pushed) of part 6a with the benchmark file of this part
(TcpConverterWriteRoutes.cs) added, so that the new route benchmarks have a base. PASS.

base 0006c365, head b15ba584, 4 rounds (base then head in each), limit 1.05
  round 1: base load average 1.26 to 2.01, CPU busy before 12% and after 12%; head load average 1.85 to 1.63, CPU busy before 14% and after 13%
  round 2: base load average 1.02 to 1.65, CPU busy before 13% and after 13%; head load average 1.52 to 2.45, CPU busy before 12% and after 13%
  round 3: base load average 2.25 to 4.31, CPU busy before 12% and after 13%; head load average 4.21 to 3.14, CPU busy before 11% and after 13%
  round 4: base load average 3.13 to 3.67, CPU busy before 13% and after 13%; head load average 3.53 to 2.29, CPU busy before 11% and after 11%

shape                                                                   base ms  ratio  pairs            spread base/head  verdict
TcpConverterRead.Columnar(ArrayStringAsBytes)                             0.373  1.061  1.127 1.064 1.058 0.955     3.4%/ 14.3%  check (slower in some pairs)
TcpConverterRead.Columnar(DateTimeAsOffset)                               0.191  0.992  0.985 0.996 0.987 1.002     1.2%/  1.0%  ok
TcpConverterRead.Columnar(LowCardinalityNullableStringAsString)           0.036  1.037  0.972 1.079 1.053 1.021     3.1%/  9.8%  ok
TcpConverterRead.Columnar(LowCardinalityStringAsBytes)                    0.036  0.994  1.015 0.975 1.005 0.983     5.9%/  6.0%  ok
TcpConverterRead.Columnar(NullableDateTimeAsOffset)                       0.189  0.998  0.999 0.994 0.996 1.064     1.4%/  7.8%  ok
TcpConverterRead.Columnar(StringAsString)                                 0.227  1.001  1.007 0.996 1.037 0.996     0.8%/  3.5%  ok
TcpConverterRead.Columnar(Wide)                                           1.155  0.997  0.994 0.969 1.030 1.000     1.8%/  4.8%  ok
TcpConverterRead.Poco(ArrayStringAsBytes)                                 0.444  1.009  1.048 0.835 1.033 0.986    24.6%/  1.8%  ok
TcpConverterRead.Poco(DateTimeAsOffset)                                   0.284  0.976  1.150 0.953 0.999 0.947     6.6%/ 17.6%  ok
TcpConverterRead.Poco(LowCardinalityNullableStringAsString)               0.123  0.922  0.976 0.936 0.909 0.896     2.5%/  7.8%  ok
TcpConverterRead.Poco(LowCardinalityStringAsBytes)                        0.124  0.909  0.927 0.912 0.907 0.882     3.4%/  8.7%  ok
TcpConverterRead.Poco(NullableDateTimeAsOffset)                           0.258  1.010  1.003 1.016 0.994 1.024     2.8%/  4.5%  ok
TcpConverterRead.Poco(StringAsString)                                     0.323  0.976  0.931 0.990 1.090 0.961     6.1%/ 10.3%  ok
TcpConverterRead.Poco(Wide)                                               1.181  0.996  1.023 0.991 0.995 0.998     5.6%/  5.6%  ok
TcpConverterRowWrite.Poco(ArrayStringFromText)                            0.685  0.993  0.965 1.021 1.034 0.934     3.3%/  7.7%  ok
TcpConverterRowWrite.Poco(LowCardinalityStringHighRepeat)                 0.344  0.969  0.979 0.966 0.951 0.971     2.4%/  1.6%  ok
TcpConverterRowWrite.Poco(LowCardinalityStringLowRepeat)                  1.105  0.987  0.985 0.998 0.973 0.988     1.7%/  1.7%  ok
TcpConverterRowWrite.Poco(NullableDateTimeFromOffset)                     0.099  0.976  0.944 0.978 0.980 0.975     3.9%/  3.1%  ok
TcpConverterRowWrite.Poco(NullableDateTimePropertyIntoDateTime)           0.084  1.053  1.077 1.058 1.002 1.049     2.4%/  4.8%  check (slower in some pairs)
TcpConverterRowWrite.Poco(NullableIntPropertyIntoInt32)                   0.023  1.056  1.031 1.081 0.992 1.130     7.4%/  6.9%  check (slower in some pairs)
TcpConverterRowWrite.Poco(Wide)                                           2.370  1.018  0.940 1.078 1.054 0.983     4.1%/ 16.5%  ok
TcpConverterRowWrite.Untyped(ArrayStringFromText)                         0.812  0.965  0.966 0.998 0.954 0.964     9.1%/  9.4%  ok
TcpConverterRowWrite.Untyped(LowCardinalityStringHighRepeat)              0.426  0.976  0.978 0.979 0.966 0.975     1.7%/  2.5%  ok
TcpConverterRowWrite.Untyped(LowCardinalityStringLowRepeat)               1.191  1.009  1.034 0.981 1.032 0.987     4.4%/  2.9%  ok
TcpConverterRowWrite.Untyped(NullableDateTimeFromOffset)                  0.177  0.995  0.998 0.992 1.022 0.976     2.1%/  5.3%  ok
TcpConverterRowWrite.Untyped(NullableDateTimePropertyIntoDateTime)        0.143  0.999  1.008 1.002 0.996 0.995     2.5%/  3.3%  ok
TcpConverterRowWrite.Untyped(NullableIntPropertyIntoInt32)                0.083  1.013  0.994 1.026 1.008 1.018     1.9%/  4.4%  ok
TcpConverterRowWrite.Untyped(Wide)                                        2.872  1.001  0.998 0.985 1.004 1.032     4.9%/  1.8%  ok
TcpConverterWrite.Insert(ArrayStringFromText)                             0.598  1.015  1.033 1.031 0.999 0.955     8.3%/ 11.8%  ok
TcpConverterWrite.Insert(LowCardinalityFixedStringBytesHighRepeat)        0.359  0.984  0.974 0.994 0.967 1.025     1.9%/  4.2%  ok
TcpConverterWrite.Insert(LowCardinalityFixedStringBytesLowRepeat)         0.636  0.999  1.003 1.037 0.994 0.993     0.9%/  3.7%  ok
TcpConverterWrite.Insert(LowCardinalityStringHighRepeat)                  0.299  0.983  0.964 0.973 0.992 1.282     2.1%/ 32.2%  ok
TcpConverterWrite.Insert(LowCardinalityStringLowRepeat)                   1.028  1.018  1.048 1.010 1.006 1.026     2.4%/  2.7%  ok
TcpConverterWrite.Insert(NullableDateTimeFromOffset)                      0.062  1.012  1.191 1.013 0.981 1.010     6.6%/ 20.9%  ok
TcpConverterWriteRoutes.Insert(CreateArrayLowCardinalityString)           0.643  0.948  0.938 0.950 0.954 0.947     1.5%/  0.8%  ok
TcpConverterWriteRoutes.Insert(CreateArrayNullableInt32)                  0.279  0.429  0.429 0.429 0.403 0.438     6.4%/  2.3%  ok
TcpConverterWriteRoutes.Insert(DynamicMixedKinds)                         1.272  0.817  0.831 0.808 0.805 0.826     2.2%/  1.4%  ok
TcpConverterWriteRoutes.Insert(LowCardinalityUInt32HighRepeat)            0.097  1.029  1.014 1.034 1.029 1.029     2.3%/  0.4%  ok
TcpConverterWriteRoutes.Insert(LowCardinalityUuidHighRepeat)              0.368  0.677  0.677 0.670 0.677 0.680     2.0%/  1.1%  ok
TcpConverterWriteRoutes.Insert(LowCardinalityUuidLowRepeat)               0.946  0.764  0.787 0.765 0.730 0.762     6.8%/  3.7%  ok

largest run-to-run spread on one side: 32.2%
gate: PASS (0 shapes slower than 1.05 in every pair)

The gate of the stack: the converter benchmarks of part 0 on part 0 (base) and on this part (head). One FAIL.

base 77ca7eeb, head b15ba584, 4 rounds (base then head in each), limit 1.05
  round 1: base load average 2.13 to 2.59, CPU busy before 13% and after 11%; head load average 2.38 to 1.60, CPU busy before 13% and after 13%
  round 2: base load average 1.47 to 1.83, CPU busy before 12% and after 12%; head load average 1.68 to 4.49, CPU busy before 12% and after 40%
  round 3: base load average 1.81 to 3.42, CPU busy before 41% and after 11%; head load average 3.15 to 2.61, CPU busy before 12% and after 14%
  round 4: base load average 2.40 to 2.41, CPU busy before 12% and after 15%; head load average 2.22 to 2.19, CPU busy before 12% and after 11%

shape                                                                   base ms  ratio  pairs            spread base/head  verdict
TcpConverterRead.Columnar(ArrayStringAsBytes)                             0.560  0.701  0.719 0.657 0.684 0.730     6.3%/ 10.3%  ok
TcpConverterRead.Columnar(DateTimeAsOffset)                               0.271  0.704  0.700 0.715 0.680 0.708     5.7%/  1.7%  ok
TcpConverterRead.Columnar(LowCardinalityNullableStringAsString)           0.031  1.153  1.145 1.112 1.213 1.161     5.1%/  4.5%  FAIL
TcpConverterRead.Columnar(LowCardinalityStringAsBytes)                    0.112  0.327  0.322 0.337 0.331 0.322     1.9%/  6.0%  ok
TcpConverterRead.Columnar(NullableDateTimeAsOffset)                       0.302  0.631  0.641 0.617 0.622 0.640     1.4%/  2.5%  ok
TcpConverterRead.Columnar(StringAsString)                                 0.225  1.001  0.984 1.018 1.002 1.000     3.4%/  1.8%  ok
TcpConverterRead.Columnar(Wide)                                           1.423  0.810  0.820 0.818 0.802 0.794     2.0%/  5.4%  ok
TcpConverterRead.Poco(ArrayStringAsBytes)                                 0.677  0.680  0.698 0.663 0.671 0.689     8.5%/  9.9%  ok
TcpConverterRead.Poco(DateTimeAsOffset)                                   0.282  0.975  0.964 1.019 0.918 0.986    10.2%/  3.1%  ok
TcpConverterRead.Poco(LowCardinalityNullableStringAsString)               0.259  0.448  0.418 0.461 0.443 0.453     3.5%/  9.1%  ok
TcpConverterRead.Poco(LowCardinalityStringAsBytes)                        0.212  0.531  0.535 0.523 0.532 0.530     1.4%/  2.7%  ok
TcpConverterRead.Poco(NullableDateTimeAsOffset)                           0.287  0.906  0.938 0.893 0.881 0.919     4.7%/  3.9%  ok
TcpConverterRead.Poco(StringAsString)                                     0.348  0.941  0.952 0.993 0.930 0.927     3.7%/  7.8%  ok
TcpConverterRead.Poco(Wide)                                               1.570  0.753  0.764 0.756 0.735 0.750     3.1%/  4.5%  ok
TcpConverterWrite.Insert(ArrayStringFromText)                             0.627  0.938  0.935 0.950 0.929 0.941     2.7%/  4.6%  ok
TcpConverterWrite.Insert(LowCardinalityFixedStringBytesHighRepeat)        0.363  0.984  0.985 1.754 0.973 0.983     3.3%/ 76.6%  ok
TcpConverterWrite.Insert(LowCardinalityFixedStringBytesLowRepeat)         1.295  0.492  0.492 0.513 0.481 0.491     2.3%/  4.6%  ok
TcpConverterWrite.Insert(LowCardinalityStringHighRepeat)                  0.307  0.950  0.942 2.547 0.954 0.945     0.8%/171.9%  ok
TcpConverterWrite.Insert(LowCardinalityStringLowRepeat)                   1.221  0.861  0.866 0.897 0.819 0.855     6.3%/  4.5%  ok
TcpConverterWrite.Insert(NullableDateTimeFromOffset)                      0.157  0.411  0.402 0.813 0.414 0.407     1.1%/102.9%  ok

largest run-to-run spread on one side: 171.9%
gate: FAIL (1 shapes slower than 1.05 in every pair)

The shapes that are not ok are not a cost of this part. The FAIL, Columnar(LowCardinalityNullableStringAsString) at
1.15x against part 0 (1.04x against part 6a), is the code-placement artifact that parts 3 and 6a found on this shape and
that is accepted for the stack: the method that reads the column has the same disassembly as on part 0 apart from
embedded addresses, it starts at offset 0 instead of 32 (mod 64) in the benchmark's processes, and with tiered
compilation on the two builds read at the same speed. Of the check shapes against part 6a,
Poco(NullableDateTimePropertyIntoDateTime) is placement: its hot methods have the same disassembly as on part 6a, start
32 bytes apart (mod 64), and that offset alone makes it 1.07x slower in one build. Poco(NullableIntPropertyIntoInt32)
(23 µs, above 1.05 in two pairs of four, a spread of 7% on each side) is a row write of the same POCO gather, which this
part does not change after its earlier gates (0.986 on an earlier commit of this part). Columnar(ArrayStringAsBytes)
(above 1.05 in three pairs, 0.955 in the fourth, a spread of 14% on the head side) is a read path that this part does not
change (ArrayReader over the String leaf read as bytes). The benchmarks do not change.

Changes after review

• The block writer names the class of a column that has no single CLR type in its refusal (Cursor).
• The comments of the Dynamic and HoldsNull caches give their bound past the limit.
• DifferentialTests checks that the nested Array samples hold values at every depth (the check of TCP converters 0/6: differential oracle and converter benchmarks #665, after the trim).
• The storage check of the codecs keeps the name WritesFromStorage that TCP converters 4a/6: write combinators for the converter layer #671 gives it.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 89.53846% with 34 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
ClickHouse.Driver.Tcp/Format/InsertColumnWrite.cs	87.20%	13 Missing and 19 partials ⚠️
ClickHouse.Driver.Tcp/Format/BlockWriter.cs	83.33%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit cbc1227. Configure here.}

[Copilot]

🔵 Needs a closer look

The broad wire-format rewrite and extensive removal of reference paths warrant final human review despite comprehensive tests and successful CI.

0 open findings

What changed in this PR

This TCP-client PR completes migration of writes to the converter layer, restricts codecs to storage-level serialization, and fixes conversions between related decoded column types.

Changes:

• Routes all insert forms through derived writers and removes legacy codec write machinery.
• Preserves semantic values, raw string bytes, and hidden nullable values when reinserting decoded columns.
• Simplifies failure ordering and differential tests while expanding integration, slicing, and byte-level coverage.

File	Description
docs/​tcp.mdx	Documents revised read/write behavior.
ClickHouse.Driver.Tcp/​Types/​TupleFieldColumn.cs	Removes legacy tuple projection.
ClickHouse.Driver.Tcp/​Types/​SubstituteColumn.cs	Removes nullable substitution views.
ClickHouse.Driver.Tcp/​Types/​ProjectedColumn.cs	Removes projected write columns.
ClickHouse.Driver.Tcp/​Types/​ISpanWritableCodec.cs	Removes legacy span-write interface.
ClickHouse.Driver.Tcp/​Types/​EnumMemberTable.cs	Adds enum-definition comparison.
ClickHouse.Driver.Tcp/​Types/​EnumColumn.cs	Exposes internal member metadata.
ClickHouse.Driver.Tcp/​Types/​Converters/​MapWriter.cs	Clarifies refusal diagnostics.
ClickHouse.Driver.Tcp/​Types/​Converters/​MapReader.cs	Removes failure-order replay.
ClickHouse.Driver.Tcp/​Types/​Converters/​LeafTable.cs	Uses typed enum placeholders.
ClickHouse.Driver.Tcp/​Types/​Converters/​FixedLeafWriter.cs	Exposes equality-conversion property.
ClickHouse.Driver.Tcp/​Types/​Converters/​FixedInterner.cs	Optimizes 128-bit keys.
ClickHouse.Driver.Tcp/​Types/​Converters/​EmptyTupleWriter.cs	Adds converter-layer empty-tuple writing.
ClickHouse.Driver.Tcp/​Types/​Converters/​DictionaryWriters.cs	Adds CLR-keyed fixed interning.
ClickHouse.Driver.Tcp/​Types/​Converters/​ConverterDerivation.cs	Adds nullability caching and simplifies derivation.
ClickHouse.Driver.Tcp/​Types/​Converters/​CodecWriter.cs	Removes codec-backed converter writer.
ClickHouse.Driver.Tcp/​Types/​Converters/​BytesLeafWriter.cs	Clarifies refusal diagnostics.
ClickHouse.Driver.Tcp/​Types/​Converters/​ArrayWriter.cs	Exposes derived element writer.
ClickHouse.Driver.Tcp/​Types/​ConcatColumn.cs	Removes flattening write view.
ClickHouse.Driver.Tcp/​Types/​CompositeElementProjections.cs	Removes legacy projection helper.
ClickHouse.Driver.Tcp/​Types/​ColumnCodecExtensions.cs	Removes obsolete codec write helpers.
ClickHouse.Driver.Tcp/​Types/​Codecs/​ValueNullableShape.cs	Removes nullable write behavior.
ClickHouse.Driver.Tcp/​Types/​Codecs/​UuidColumnCodec.cs	Restricts writes to decoded storage.
ClickHouse.Driver.Tcp/​Types/​Codecs/​StringColumnCodec.cs	Writes only stored raw bytes.
ClickHouse.Driver.Tcp/​Types/​Codecs/​ReferenceNullableShape.cs	Removes reference nullable writes.
ClickHouse.Driver.Tcp/​Types/​Codecs/​NothingColumnCodec.cs	Removes obsolete write metadata.
ClickHouse.Driver.Tcp/​Types/​Codecs/​LowCardinalityValueWriter.cs	Removes legacy dictionary writer.
ClickHouse.Driver.Tcp/​Types/​Codecs/​LowCardinalityShape.cs	Restricts writes to dense storage.
ClickHouse.Driver.Tcp/​Types/​Codecs/​LowCardinalityKeys.cs	Removes legacy key strategies.
ClickHouse.Driver.Tcp/​Types/​Codecs/​JsonStringColumnCodec.cs	Restricts JSON codec writes.
ClickHouse.Driver.Tcp/​Types/​Codecs/​INullableShape.cs	Reduces interface to reading.
ClickHouse.Driver.Tcp/​Types/​Codecs/​ILowCardinalityShape.cs	Reduces interface to storage checks.
ClickHouse.Driver.Tcp/​Types/​Codecs/​IArrayWriteShape.cs	Removes legacy array writing.
ClickHouse.Driver.Tcp/​Types/​Codecs/​GeoColumnCodecs.cs	Exposes geo structures to derivation.
ClickHouse.Driver.Tcp/​Types/​Codecs/​FixedWidthColumnCodec.cs	Restricts writes to primitive storage.
ClickHouse.Driver.Tcp/​Types/​Codecs/​EnumColumnCodec.cs	Validates matching enum definitions.
ClickHouse.Driver.Tcp/​Types/​Codecs/​EmptyTupleColumnCodec.cs	Restricts storage writes and shares placeholder.
ClickHouse.Driver.Tcp/​Types/​Codecs/​DynamicTypeInference.cs	Updates converter-layer documentation.
ClickHouse.Driver.Tcp/​Types/​Codecs/​DecimalColumnCodec.cs	Restricts writes to decoded storage.
ClickHouse.Driver.Tcp/​Types/​Codecs/​BFloat16ColumnCodec.cs	Adds chunked stored-value encoding.
ClickHouse.Driver.Tcp/​Types/​ArrayColumn.cs	Removes non-owning buffer wrapper.
ClickHouse.Driver.Tcp/​Protocol/​ClickHouseTcpConnection.cs	Plans converter and decoded-part writes.
ClickHouse.Driver.Tcp/​Poco/​UntypedRowColumns.cs	Uses derivation nullability.
ClickHouse.Driver.Tcp/​Poco/​PocoWritePlan.cs	Removes codec dependency from planning.
ClickHouse.Driver.Tcp/​Poco/​PocoWriteConversion.cs	Removes legacy nullability helper.
ClickHouse.Driver.Tcp/​Poco/​PocoScatterTier.cs	Documents relaxed failure order.
ClickHouse.Driver.Tcp/​Poco/​PocoGatherTier.cs	Documents relaxed failure order.
ClickHouse.Driver.Tcp/​Poco/​PocoColumnScatterFactory.cs	Removes failed-read replay.
ClickHouse.Driver.Tcp/​Poco/​PocoColumnBuilder.cs	Uses derivation nullability.
ClickHouse.Driver.Tcp/​Format/​InsertColumn.cs	Removes implicit codec write construction.
ClickHouse.Driver.Tcp/​Format/​BlockWriter.cs	Plans all writes through insert routing.
ClickHouse.Driver.Tcp.Tests/​Utilities/​NestedArrayShape.cs	Updates segment-write documentation.
ClickHouse.Driver.Tcp.Tests/​Utilities/​DecodedColumns.cs	Adds decoded-column fixture helper.
ClickHouse.Driver.Tcp.Tests/​Types/​UuidColumnCodecTests.cs	Tests storage UUID encoding.
ClickHouse.Driver.Tcp.Tests/​Types/​NothingColumnCodecTests.cs	Tests complete write refusal.
ClickHouse.Driver.Tcp.Tests/​Types/​IpColumnCodecTests.cs	Uses storage-write harness.
ClickHouse.Driver.Tcp.Tests/​Types/​FixedWidthColumnCodecTests.cs	Tests storage-only fixed writes.
ClickHouse.Driver.Tcp.Tests/​Types/​EnumColumnCodecTests.cs	Removes obsolete codec-write tests.
ClickHouse.Driver.Tcp.Tests/​Types/​DerivedColumnTests.cs	Stabilizes concurrency tests.
ClickHouse.Driver.Tcp.Tests/​Types/​DateColumnCodecTests.cs	Uses storage-write harness.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​WriteConverterRegistration.cs	Removes obsolete differential plug-in.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​RefusalMessageTests.cs	Updates explicit insert descriptors.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​LeafSamples.cs	Updates write terminology.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​LeafReaderTests.cs	Updates differential-test references.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​LeafConverterRegistrationTests.cs	Directly checks leaf coverage.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​DictionaryEntryCacheTests.cs	Updates readers and concurrency setup.
ClickHouse.Driver.Tcp.Tests/​Types/​Converters/​ConverterDerivationTests.cs	Tests nullability derivation and caching.
ClickHouse.Driver.Tcp.Tests/​Types/​ColumnReadScenario.cs	Updates scenario documentation.
ClickHouse.Driver.Tcp.Tests/​Types/​BFloat16ColumnCodecTests.cs	Tests storage-shape discrimination.
ClickHouse.Driver.Tcp.Tests/​Types/​ArrayColumnTests.cs	Removes deleted buffer-wrapper tests.
ClickHouse.Driver.Tcp.Tests/​Poco/​RowWriteRegistration.cs	Removes obsolete differential registration.
ClickHouse.Driver.Tcp.Tests/​Poco/​PocoReadRegistration.cs	Removes obsolete differential registration.
ClickHouse.Driver.Tcp.Tests/​Poco/​PocoReadPlanTests.cs	Updates decoded-column terminology.
ClickHouse.Driver.Tcp.Tests/​Poco/​PocoColumnScatterFactoryTests.cs	Tests relaxed failure ordering.
ClickHouse.Driver.Tcp.Tests/​Integration/​PocoReadTierIntegrationTests.cs	Expands multi-window tier coverage.
ClickHouse.Driver.Tcp.Tests/​Integration/​PocoReadFillIntegrationTests.cs	Removes duplicate integration fixture.
ClickHouse.Driver.Tcp.Tests/​Integration/​NestedArrayInsertIntegrationTests.cs	Updates segmented-write coverage.
ClickHouse.Driver.Tcp.Tests/​Integration/​GeometryIntegrationTests.cs	Builds decoded geo alternatives.
ClickHouse.Driver.Tcp.Tests/​Integration/​ColumnarReadSurfaceIntegrationTests.cs	Uses decoded dictionaries.
ClickHouse.Driver.Tcp.Tests/​Integration/​ClickHouseTcpConnectionInsertIntegrationTests.cs	Verifies dense storage routing.
ClickHouse.Driver.Tcp.Tests/​Integration/​ClickHouseTcpClientIntegrationTests.cs	Clarifies Dynamic JSON behavior.
ClickHouse.Driver.Tcp.Tests/​Format/​BlockWriterTests.cs	Tests explicitly planned writes.
ClickHouse.Driver.Tcp.Tests/​Differential/​Expectation.cs	Removes old expectation framework.
ClickHouse.Driver.Benchmark/​TcpConverterRead.cs	Benchmarks insert-plan routing.
changelog.d/​converter-6b-write-reroute.improvements.md	Records write-routing changes.
changelog.d/​converter-6b-failure-order.improvements.md	Records failure-order changes.
changelog.d/​converter-6b-decoded-string-bytes.fixes.md	Records raw-byte preservation fix.
changelog.d/​converter-6b-decoded-hidden-values.fixes.md	Records hidden-value fix.
changelog.d/​converter-6b-decoded-column-parameters.fixes.md	Records parameter-conversion fixes.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[polyglotAI-bot]

Summary
This PR moves the native TCP client’s remaining writes into the converter layer, fixes decoded-column conversions across related types, and simplifies the differential tests. I found no actionable issues in the current snapshot. ✅ Approve. This was a static review; builds and tests were not run in the sandbox.

Final verdict

✅ Approve

_{Automated review by Polyglot AI (gpt-6-astra) of f83c7ce. Replies on the inline threads are read on the next review. A maintainer can ask for a new review by mentioning @polyglotAI-bot or requesting a review from it.}