Repository navigation
TCP converters 6a/6: delete the old read members and the reference paths - #674
alex-clickhouse wants to merge 13 commits into
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
7b859a7 to
9a9e654
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The cross-cutting legacy-path removal and reported benchmark-gate failure warrant final human review despite passing CI and extensive replacement coverage.
0 open findings
What changed in this PR
Removes obsolete native TCP converter reference paths now that all read/write tiers use converter derivation, while preserving coverage through tier parity and literal behavior matrices.
Changes:
- Deletes legacy codec projections and POCO/reference implementations.
- Derives diagnostic type suggestions from converter rules.
- Reworks differential tests around client entry-point baselines and updates docs/changelog.
| File | Description |
|---|---|
ClickHouse.Driver.Tcp.Tests/Differential/Arms.cs |
Simplifies baseline-arm wrappers. |
ClickHouse.Driver.Tcp.Tests/Differential/ClientArms.cs |
Makes client entry points baselines. |
ClickHouse.Driver.Tcp.Tests/Differential/DifferentialCase.cs |
Updates baseline terminology. |
ClickHouse.Driver.Tcp.Tests/Differential/DifferentialCases.cs |
Pins literal read targets. |
ClickHouse.Driver.Tcp.Tests/Differential/DifferentialEngine.cs |
Compares candidates with baselines. |
ClickHouse.Driver.Tcp.Tests/Differential/DifferentialRegistry.cs |
Registers client baselines and outcomes. |
ClickHouse.Driver.Tcp.Tests/Differential/DifferentialSelfTests.cs |
Tests revised differential semantics. |
ClickHouse.Driver.Tcp.Tests/Differential/DifferentialTests.cs |
Adds cross-tier consistency checks. |
ClickHouse.Driver.Tcp.Tests/Differential/Expectation.cs |
Updates declared-outcome support. |
ClickHouse.Driver.Tcp.Tests/Differential/LegacyColumnarRead.cs |
Deletes legacy read reference. |
ClickHouse.Driver.Tcp.Tests/Differential/LegacyColumnarWrite.cs |
Deletes legacy write reference. |
ClickHouse.Driver.Tcp.Tests/Differential/LegacyPocoRead.cs |
Deletes legacy POCO-read reference. |
ClickHouse.Driver.Tcp.Tests/Differential/LegacyRowWrite.cs |
Deletes legacy row-write reference. |
ClickHouse.Driver.Tcp.Tests/Differential/Outcome.cs |
Updates baseline terminology. |
ClickHouse.Driver.Tcp.Tests/Differential/RowWriteArms.cs |
Simplifies client row-write arms. |
ClickHouse.Driver.Tcp.Tests/Integration/ColumnReadProjectionIntegrationTests.cs |
Tests derived calendar readers. |
ClickHouse.Driver.Tcp.Tests/Poco/LegacyPocoColumnScatterFactoryTests.cs |
Removes legacy scatter tests. |
ClickHouse.Driver.Tcp.Tests/Poco/PocoColumnBuilderFactoryTests.cs |
Compares gather tiers directly. |
ClickHouse.Driver.Tcp.Tests/Poco/PocoColumnScatterFactoryTests.cs |
Removes legacy-plan comparisons. |
ClickHouse.Driver.Tcp.Tests/Poco/PocoReadRegistration.cs |
Uses client POCO baseline. |
ClickHouse.Driver.Tcp.Tests/Poco/PocoReadRulesTests.cs |
Compares Fill and Emit tiers. |
ClickHouse.Driver.Tcp.Tests/Poco/PocoWritePlanTests.cs |
Compares row and columnar writes. |
ClickHouse.Driver.Tcp.Tests/Poco/RowWriteRegistration.cs |
Removes legacy row registrations. |
ClickHouse.Driver.Tcp.Tests/Types/ColumnReadProjectionTests.cs |
Exercises values through ReadAs. |
ClickHouse.Driver.Tcp.Tests/Types/CompositeElementProjectionTests.cs |
Removes obsolete projection tests. |
ClickHouse.Driver.Tcp.Tests/Types/CompositeLiftMatrixTests.cs |
Uses public read capability checks. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/ConverterHarness.cs |
Replaces old reads with ReadAs. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/LeafConverterRegistration.cs |
Removes legacy change declarations. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/LeafConverterRegistrationTests.cs |
Verifies client-baseline coverage. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/LeafReaderTests.cs |
Compares readers with ReadAs. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/LeafSamples.cs |
Updates reference terminology. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/LeafTableTests.cs |
Removes obsolete codec-list checks. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/ReadCombinatorTests.cs |
Updates reader parity coverage. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/ReadConverterRegistration.cs |
Declares nullable-tail outcomes. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/ReadRulesTests.cs |
Pins accepted read targets. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/RefusalMessageTests.cs |
Pins five diagnostic messages. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/SuggestedTypesTests.cs |
Tests derived suggestions. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/WriteConverterRegistration.cs |
Removes old-path comparisons. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/WriteRulesRegistration.cs |
Deletes obsolete declarations. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/WriteRulesTests.cs |
Pins accepted write sources. |
ClickHouse.Driver.Tcp.Tests/Types/FixedStringColumnCodecTests.cs |
Tests derived text reading. |
ClickHouse.Driver.Tcp.Tests/Types/NullableColumnCodecTests.cs |
Tests nullable write acceptance. |
ClickHouse.Driver.Tcp/Poco/LegacyPocoColumnBuilderFactory.cs |
Deletes legacy POCO builder. |
ClickHouse.Driver.Tcp/Poco/LegacyPocoColumnScatterFactory.cs |
Deletes legacy POCO scatter. |
ClickHouse.Driver.Tcp/Poco/LegacyPocoScatterTier.cs |
Deletes legacy tier enum. |
ClickHouse.Driver.Tcp/Poco/LegacyPocoWriteConversion.cs |
Deletes legacy conversion rules. |
ClickHouse.Driver.Tcp/Poco/PocoColumnBuilder.cs |
Uses derived write suggestions. |
ClickHouse.Driver.Tcp/Poco/PocoColumnScatterFactory.cs |
Uses derived read suggestions. |
ClickHouse.Driver.Tcp/Poco/PocoReadErrors.cs |
Accepts derived suggestion lists. |
ClickHouse.Driver.Tcp/Poco/PocoReadPlan.cs |
Removes legacy plan construction. |
ClickHouse.Driver.Tcp/Poco/PocoSpan.cs |
Deletes obsolete span helper. |
ClickHouse.Driver.Tcp/Poco/PocoValueProjection.cs |
Deletes legacy value projections. |
ClickHouse.Driver.Tcp/Poco/PocoWriteConversion.cs |
Removes preferred-type enumeration. |
ClickHouse.Driver.Tcp/Poco/PocoWriteErrors.cs |
Improves derived-type diagnostics. |
ClickHouse.Driver.Tcp/Poco/PocoWritePlan.cs |
Removes legacy write plan. |
ClickHouse.Driver.Tcp/Poco/UntypedRowColumns.cs |
Removes legacy type chooser. |
ClickHouse.Driver.Tcp/Protocol/ClickHouseTcpConnection.cs |
Uses derived insert suggestions. |
ClickHouse.Driver.Tcp/Types/Codecs/ArrayColumnCodec.cs |
Removes read projections. |
ClickHouse.Driver.Tcp/Types/Codecs/DateTime64ColumnCodec.cs |
Removes read projections. |
ClickHouse.Driver.Tcp/Types/Codecs/DateTimeColumnCodec.cs |
Removes read projections. |
ClickHouse.Driver.Tcp/Types/Codecs/EnumColumnCodec.cs |
Removes label projections. |
ClickHouse.Driver.Tcp/Types/Codecs/FixedStringColumnCodec.cs |
Removes text projections. |
ClickHouse.Driver.Tcp/Types/Codecs/LowCardinalityColumnCodec.cs |
Removes dictionary projections. |
ClickHouse.Driver.Tcp/Types/Codecs/MapColumnCodec.cs |
Removes map projections. |
ClickHouse.Driver.Tcp/Types/Codecs/NullableColumnCodec.cs |
Removes nullable projections/lists. |
ClickHouse.Driver.Tcp/Types/Codecs/StringColumnCodec.cs |
Removes byte projections. |
ClickHouse.Driver.Tcp/Types/Codecs/Time64ColumnCodec.cs |
Removes read projections. |
ClickHouse.Driver.Tcp/Types/Codecs/TimeColumnCodec.cs |
Removes read projections. |
ClickHouse.Driver.Tcp/Types/Codecs/TupleColumnCodec.cs |
Removes tuple projections. |
ClickHouse.Driver.Tcp/Types/ColumnReadProjections.cs |
Uses derived read suggestions. |
ClickHouse.Driver.Tcp/Types/ColumnValueProjections.cs |
Removes obsolete expression helpers. |
ClickHouse.Driver.Tcp/Types/CompositeElementProjections.cs |
Retains write-side array helper. |
ClickHouse.Driver.Tcp/Types/Converters/ConverterDerivation.Suggestions.cs |
Implements diagnostic suggestions. |
ClickHouse.Driver.Tcp/Types/Converters/LeafReaders.cs |
Updates wire-byte diagnostic reference. |
ClickHouse.Driver.Tcp/Types/IColumnCodec.cs |
Removes legacy read members. |
ClickHouse.Driver.Tcp/Types/LegacyColumnProjection.cs |
Deletes legacy projection infrastructure. |
ClickHouse.Driver.Tcp/Types/ProjectedReadColumn.cs |
Deletes legacy projected view. |
changelog.d/converter-2b-columnar-reads.improvements.md |
Consolidates prior stack wording. |
changelog.d/converter-4b-columnar-writes.improvements.md |
Consolidates insert scope. |
changelog.d/converter-4b-lowcardinality-bytes.fixes.md |
Clarifies all-insert coverage. |
changelog.d/converter-5-write-rules.improvements.md |
Consolidates write-rule wording. |
changelog.d/converter-6a-refusal-messages.improvements.md |
Records diagnostic changes. |
docs/tcp.mdx |
Documents shortened suggestion lists. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Summary Final verdict✅ Approve Automated review by Polyglot AI (gpt-6-astra) of |
… tests The old path of every tier is now only a reference for tests that the next commits remove, so the differential fixture stops comparing with it (ClickHouse/integrations#801). - The client's entry point of each tier is the first candidate of every facet, so it is the baseline. The converter arms (Fill and Emit), the Fill tier of the POCO read plan, the delegate gather of the POCO write plan and ClickHouseTcpTypes.CanWrite in the row tiers are compared with it. The reference arms and the declared changes against the old path go. - One family of outcomes stays declared: a nullable column read as a type that cannot hold NULL fails at the first NULL of the column in Block.ReadAs, and at the first NULL of the tail in a reader of the tail. - The column types of the read-projection cases are a literal table with the 56 types and targets of before, in place of the targets that came from ReadableElementTypes and the test attributes. - New cross-tier tests: CanRead agrees with ReadAs and POCO mapping, CanWrite with the insert, a decoded column writes the bytes it was read from, POCO mapping reads what ReadAs reads, and the row inserts write the bytes of the columnar insert. - The baseline outcome counts replace the reference outcome counts.
The POCO read plan has read through the converter derivation since part 3 of ClickHouse/integrations#801. The old scatter stayed only as the reference of the tests. Deleted: LegacyPocoColumnScatterFactory, LegacyPocoScatterTier, PocoValueProjection with PocoProjectionSite, PocoSpan and PocoReadPlan.BuildLegacy, and the test arm and tests of the old scatter. The tests that compared with the old plan compare without it: PocoReadRulesTests runs the Fill tier against the Emit tier over the read-rule matrix; the read-rule matrix of ReadRulesTests compares Emit and the client's POCO plan with Fill; the two dictionary failure-order tests state the failure text; the setter test runs the two tiers.
A derivation cannot list what it accepts, so the read refusals of Block.ReadAs and of the POCO read plan need a short list of CLR types to suggest. ConverterDerivation.SuggestedTypes gives it: the CLR types of a leaf's pairs in the leaf table, made nullable under Nullable, the same under LowCardinality and SimpleAggregateFunction, and the canonical type of any other type. It keeps only the types that the derivation accepts for the whole column type. ClickHouse/integrations#801. For every column type of the test suites the read lists are the ones that ReadableElementTypes gave, so the texts of the read refusals do not change. A test checks, for every leaf and for Nullable, LowCardinality and SimpleAggregateFunction around it, that each suggested type derives and that the leaf suggests every CLR type of its pairs.
The call was ambiguous on net8.0 (ClickHouse/integrations#801).
Block.ReadAs, CanRead and the POCO read plan read through the converter derivation, and the read refusals take their suggested types from it, so the column projections of the codecs have no caller (ClickHouse/integrations#801). Deleted: IColumnCodec.TryProjectRead, TryProjectColumnRead and ReadableElementTypes with every codec's implementation and helpers, LegacyColumnProjection with the ColumnReadProjection delegate and ProjectedViewCache, ProjectedReadColumn, the read half of CompositeElementProjections, ColumnValueProjections.RequireSourceType and TryLiftOverAbsent, StringColumnCodec.RowBytes, FixedStringColumnCodec.RowText and the test copy of the old columnar dispatch. Tests: the value and error scenarios of ColumnReadProjectionTests read each value through Block.ReadAs (the differential tests read the same scenarios); the tests of the projection API and of the read lists go. The leaf, lift matrix and combinator tests compare with Block.ReadAs, CanRead, literal values or the expected failing value. The calendar integration tests read through the derived reader. The edge cases of the composite projections are CanRead cases.
The row inserts have written through the converter derivation since part 5 of ClickHouse/integrations#801; the old plan stayed only as the reference of the tests. Deleted: LegacyPocoColumnBuilderFactory, LegacyPocoWriteConversion with PocoGatherSite, PocoWritePlan.BuildLegacy, the UntypedWriteTypeChooser overload of UntypedRowColumns.CreateSource, and the test copy of the old row inserts. Tests: the write rule matrix compares the two gather tiers, the columnar insert with the POCO write plan and CanWrite, and the untyped insert with CanWrite, in place of the old plan; an untyped column written through a cast gives the bytes of the columnar insert of the cast values; the plan tests that compared with the old plan state the bytes or compare with the columnar insert of the same values; the getter-and-NULL test states the first failure and compares the two tiers. The template arms of the row inserts in the differential tests are one client arm each.
The refusals of the POCO write plan, the untyped rows and the columnar insert listed the codec's preferred write types, which miss writes that the converter derivation takes: "It accepts System.Byte[]" for FixedString(4), which is also written from string, and no byte[] for LowCardinality(String) (ClickHouse/integrations#792). They now take the list from ConverterDerivation.SuggestedTypes, and the composite clause names what is true: an Array, Map or Tuple type also accepts the types that its element types accept. PocoWriteConversion.AcceptedWriteTypes goes. ClickHouse/integrations#801. Tests: the exact texts of the five refusals that suggest types (columnar and POCO reads, which do not change, and the three writes).
…exts The fragments of the converter parts describe the final behavior in the past tense: the read rules and the whole-column view of Block.ReadAs, the writes of the converter layer in every insert, the one set of write rules, and ClickHouse/integrations#792 in every insert. A new fragment covers the write refusals that name true CLR types. docs/tcp.mdx says what such a message lists. ClickHouse/integrations#801.
A boxed value is never a nullable value type, so for a column that holds NULL the refusal of the untyped insert named a type that no value has (System.Nullable`1[System.Int32]). It names the value type (System.Int32), which the insert takes. ClickHouse/integrations#801.
MapColumnCodec.Row read a row of the old Map projection, and the WritableElementTypes list of NullableColumnCodec served only the old refusal messages (its own CanWriteElementType asks the inner codec). The Nullable test of that list checks CanWriteElementType for the same types. Also comment and message wording that named the old projection or path, and two test parameters that no caller passes. ClickHouse/integrations#801.
With no old plan to compare with, the read-rule matrices compared the tiers only with each other, so a rule that started or stopped accepting a type would pass. Each matrix now pins the accepted types of every column type in a literal table, and LeafTableTests pins the read pairs of every sample type. The differential tests also check that the answers of the row tiers are whether the row inserts accept the column, that a column read back as another CLR type writes the bytes of the source column (except where the CLR type cannot hold the counts of a DateTime64 or Time64 of a scale above 7), and that the first candidate of each facet is the client's arm itself. ClickHouse/integrations#801.
The write of a decoded column (the columnar insert writes it from its storage) was compared with the source bytes for all rows only, and it is the only candidate of its facet, so a dense write that ignored the start of a slice passed. The row inserts of the decoded column had no comparison at all. Every write of the decoded column, in the three write tiers, for all rows and for the tail, now decodes to the values of the same rows of the source column (equal bytes need no decode; a slice of a decoded LowCardinality column keeps the dictionary of the whole column, so its bytes can differ). Self-tests: a dense write that ignores the start of Array(UInt8) is reported, the client's write is not. A dense Array write that ignores the element base of the slice fails the check on every decoded Array case. ClickHouse/integrations#801 (review round 1, F2).
9a9e654 to
9603d21
Compare
This is part 6a of the stack of pull requests for ClickHouse/integrations#801, a converter layer for the native TCP client.
It is stacked on #673 (part 5, POCO and untyped writes), which is on #672, #671, #670, #668, #667, #666 and #665. Every read
and write tier of the client goes through the converter derivation since part 5. This part deletes the old read members of
the codecs, the old POCO read and write plans and the old untyped type choice, which stayed only as the reference of the
differential tests, and removes the comparison with the old path from those tests. The codec write surface (the write
shapes, the LowCardinality value and key writers,
WritableElementTypes,CanWriteElementType,NullPlaceholderAs)is the subject of a later part, because some production paths still use it.
What changes for a user
Only the texts of three error messages, which suggest CLR types. They listed the codec's preferred write types, which
miss writes that the converter layer takes, and their composite clause was not exact. Each list now comes from the
derivation, and every type in it is accepted:
InsertRowsAsync<T>, a property that the column cannot be written from:Column 'code' (FixedString(4)) is filled from property 'Event.Code' of type System.Int32, which it cannot be written from. It accepts System.Byte[], and — for a composite type — rows whose elements are any type its element codecs accept. Give the property one of those types, or insert that column through the columnar API.... It accepts System.Byte[] or System.String, and an Array, Map or Tuple type also accepts rows of the types that its element types accept. Give the property one of those types, or insert that column through the columnar API.InsertRowsAsyncwithobject[]rows, values of a type that the column cannot be written from:Column 0 ('value', Array(DateTime('UTC'))) was given values of type System.Guid[], which it cannot be written from. It accepts System.UInt32[].... It accepts System.UInt32[], and an Array, Map or Tuple type also accepts values of the types that its element types accept.NULL, the list names the value type (System.Int32forNullable(Int32)), not the nullabletype that it named before (
System.Nullable`1[System.Int32]): a boxed value is never of a nullable type.InsertAsync, a column of an element type that the target cannot be written from:Column 'value' (FixedString(4)) was given a column of element type System.Int32, which it cannot be written from. It accepts System.Byte[], and — for a composite type — a column whose elements are any type its element codecs accept.... It accepts System.Byte[] or System.String, and an Array, Map or Tuple type also accepts a column whose elements are of the types that its element types accept.The lists change for these column types:
FixedString(N)(bare, inNullable, inLowCardinality) addsSystem.String;LowCardinality(String)andLowCardinality(Nullable(String))addSystem.Byte[];LowCardinality(DateTime...)andLowCardinality(Nullable(DateTime...))add theDateTimeOffsetandDateTimetypes.The read refusals of
Block.ReadAs<T>andQueryAsync<T>also take their lists from the derivation, and their texts do notchange: for every column type of the test suites the list is the one of before.
What is deleted
IColumnCodecand of every codec:TryProjectRead,TryProjectColumnRead,ReadableElementTypes, with their helpers;LegacyColumnProjectionwithProjectedViewCacheand theColumnReadProjectiondelegate;ProjectedReadColumn; the read half ofCompositeElementProjections; the expressionhelpers of
ColumnValueProjectionsthat only the projections used.LegacyPocoColumnScatterFactory,LegacyPocoScatterTier,PocoValueProjection,PocoSpan,PocoReadPlan.BuildLegacy.LegacyPocoColumnBuilderFactory,LegacyPocoWriteConversion,PocoWritePlan.BuildLegacy, theUntypedWriteTypeChooseroverload;PocoWriteConversion.AcceptedWriteTypesand theWritableElementTypeslist ofNullableColumnCodec, which only the old messages read.How
ConverterDerivation.SuggestedTypes(type, context, direction)gives the short list of an error message: the CLR types ofa leaf's pairs in the leaf table, made nullable under
Nullable, the same underLowCardinalityandSimpleAggregateFunction, and the canonical type of any other type. It keeps only the types that the derivation acceptsfor the whole column type. A derivation cannot list what it accepts, so the list leaves out the types of the read and write
rules and the other types inside a composite.
baseline: the
FillandEmitarms of the converters, theFilltier of the POCO read plan, the gather with no compiledcode of the POCO write plan, and
ClickHouseTcpTypes.CanWritein the row tiers are compared with it, for all rows and for atail that starts above row 0. The comparisons with the old path and their declared changes are gone. One family of
outcomes stays declared: a nullable column read as a value type fails at the first
NULLof the column inBlock.ReadAs, which converts the whole column, and at the firstNULLof the tail in a reader of the tail. New testsover the reports check, for every case:
CanReadagrees withReadAsand POCO mapping;CanWriteagrees with the insert(a type with a
Nestedin it is written only from a column of its own layout), and the answers of the row tiers agreewith the row inserts; a decoded column writes the bytes that it was read from, every write of a decoded column (also a
slice, also through the row inserts) gives the values of the same source rows, and a column read back as another CLR type
writes the bytes of the source column (except where the CLR calendar types cannot hold the counts of a
DateTime64orTime64of a scale above 7); POCO mapping reads whatReadAsreads; the row inserts write the bytes of the columnarinsert. The 56
column types of the read cases are a literal table with the types and targets of before, in place of targets that came
from
ReadableElementTypes; the case list is unchanged (403 cases).Tests
cache, the old scatter tiers. Their behavior is covered by the differential tests,
ReadRulesTests,ReadCombinatorTestsand new
CanReadcases for the shapes that a composite refuses.FillwithEmit, the two POCO read tiers, the twoPOCO gather tiers, the columnar insert with the POCO plan and
CanWrite, the untyped insert withCanWriteand with thecolumnar insert of the same values. Failure-order tests state the failing value or text. What the rules accept is pinned
in two literal tables (the accepted targets of 36 column types for reads, the accepted sources of 51 column types for
writes), and the read pairs of the leaf table in a literal list for each of 58 sample types.
ColumnReadProjectionTestsread each value throughBlock.ReadAs(renamed fromTryProjectRead_*toReadAs_*). The calendar integration tests read through the derived reader.SuggestedTypesTests(every leaf, withNullable,LowCardinalityandSimpleAggregateFunctionaround it, inboth directions: each suggested type derives, and a leaf suggests all its pairs) and
RefusalMessageTests(the exact textsof the five refusals that suggest types).
test cases of part 5, about 260 are gone with the members and comparisons they tested; about 690 have a new name or
compare with another tier; about 160 are new. net8.0 and net10.0, the offline suite (no server): 3967 passed each.
Review: two rounds. Round 1 found two items, fixed and pinned by tests: the writes of a decoded column were compared
for all rows only, so a dense write that ignored the start of a slice passed; and the gate FAIL needed the placement
check of the stack (below). Round 2 approved with no findings.
Lines
ClickHouse.Driver.Tcp)77ca7eeb^)mainThe write side of the codecs that the later part removes is still in these counts.
Where the speedup of the stack comes from
The converter layer does not convert values faster. Both the old and the new code call the same conversion code
for each value (for example
DateTimeToOffset, with the timezone resolved once when the codec is built, and thesame UTF-8 encode and decode). The gains come from work that the old structure did around each value:
TryProjectColumnReadgave aview, and
TryProjectReadgave an expression that was compiled into a delegate for each value. A composite codecput the view of its child inside its own view. So one converted value cost a delegate call, an interface cast,
and interface calls into the column (each of which built a span again), and, for a composite, the same again
for each element. The POCO scatter used its fast tier only for columns that implement
IStoredValuesColumn;String,Nullable(...)andLowCardinality(...)columns do not, so each row went through the column'sindexer. The writes read the caller's column one element at a time through
IColumn<T>[i], andNullableput asubstitute view around it. The new code derives one tree for the whole type.
Binddoes the casts and buildsthe spans once for each column (once for each window of 256 rows in a POCO read). The bulk
Fillloops aregenerics closed over struct conversions, so the conversion is inlined into the loop. A POCO read splices the
per-row expression of the whole tree into one compiled loop for each column. For example:
LowCardinality(String)read asbyte[]: a delegate call, an interface cast, two interface calls and atype-checked array store for each row, against two loads and a store.
Array(String)read asbyte[][]: about 120,000 dispatches for each block of 10,000 rows, against 30,000.Nullable(DateTime)written fromDateTimeOffset?: three interface calls for each row, against none.ReadAs<T>allocated a new array for the whole convertedcolumn in each block. At 10,000 rows a
DateTimeOffset[](160 KB) orDateTimeOffset?[](240 KB) is on the largeobject heap, and a reader that keeps going runs a garbage collection about once in 20 blocks. The new view uses a
pooled array that the block owns (about 300 bytes allocated for each block). The old LowCardinality writer
built a new
Dictionarywith no capacity in each block: for 10,000 distinct values that is 11 resizes, 941 KBfor each block (719 KB on the large object heap) and a generation 2 collection about once in 10 blocks, and it
hashed each new value two times. The new interners keep their tables in pooled buffers that grow with the
number of distinct values, and hash each value one time (1.8 KB for each block for
FixedString).ClickHouseBinaryWritercall foreach null byte, each
UInt32and each dictionary key. The new writes put the null map in one copy, convert into4 KB chunks on the stack, and write the keys in chunks.
The work that both versions share limits the gain. A read that converts a
Stringleaf only (Stringasstring,DateTimein a POCO) does the same work for each row in both versions. ALowCardinalitydictionary isconverted one time for each block in both. A repeated
stringintoLowCardinality(String)costs one dictionarylookup for each row in both.
Ratios in the default runtime mode (tiered compilation with dynamic PGO, the steady state), new time / old time
for a block of 10,000 rows, measured with the old and the new code side by side:
Block.ReadAs:DateTime('UTC')asDateTimeOffsetBlock.ReadAs:Nullable(DateTime('UTC'))asDateTimeOffset?Block.ReadAs:LowCardinality(String)asbyte[]Block.ReadAs:Array(String)asbyte[][]LowCardinality(Nullable(String))asstringLowCardinality(String)asbyte[]Array(String)asbyte[][]Nullable(DateTime('UTC'))asDateTimeOffset?Stringasstring,DateTime('UTC')asDateTimeOffsetInsertAsync:Nullable(DateTime('UTC'))fromDateTimeOffset?InsertAsync:LowCardinality(FixedString(16))frombyte[], all distinctInsertAsync:LowCardinality(String)fromstring, all distinctInsertAsync:LowCardinality(String)fromstring, 100 distinctInsertRowsAsync<T>:Nullable(DateTime('UTC'))fromDateTimeOffset?The gate in "Benchmarks" below runs with tiered compilation off, so that the code does not change during a run.
Its ratios differ from the ones above: in that mode the JIT cannot remove the old per-row interface calls of
ordinary methods, and it compiles the new loops without the inlining that tier 1 adds.
Benchmarks
The performance gate of the stack: the converter benchmarks of part 0 on part 0 (base) and on this part (head), run
alternately in 4 rounds on the same machine. A shape fails when its ratio head/base is above 1.05 in every pair. Measured on
9df449b0; the later commit changes tests only.The one
FAIL,Columnar(LowCardinalityNullableStringAsString)at 1.15x, is not a cost of this part. That shape reads aLowCardinality(Nullable(String))column asstring, whichReadAsreturns as the column itself, and this part does not changethat code. Part 3 found the same effect on the same shape: the benchmarks run with tiered compilation off, so the JIT compiles
each method once, at the next free place in its code memory, and the method that reads the column
(
NullableLowCardinalityReferenceColumn<T>.Values) is slower when it starts at offset 0 (mod 64) than at offset 32:StringColumn.Values, is the same on part 0 and on this part. Only the embeddedaddresses differ.
This part deletes code, so the setup compiles other methods before it.
first moves the method from offset 0 to 32 and the read from 35.0 µs to 30.4 µs a block (median of the minimums of 10 runs);
two extra methods move it back, and it takes 35.2 µs.
block (1.01x).
Decision: the shape is a code-placement artifact of the benchmark setup, not a cost of this part, and the benchmark does not
change.
🤖 Generated with Claude Code