Repository navigation
TCP converters 5/6: POCO and untyped writes through the converter layer - #673
alex-clickhouse wants to merge 13 commits into
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
e31377c to
41c99f2
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The converter and serialization changes are broad and protocol-sensitive despite strong tests and benchmark evidence.
0 open findings
What changed in this PR
This TCP-client PR routes POCO and untyped row inserts through the shared converter derivation, unifying write rules with columnar inserts and fixing composite-value handling.
Changes:
- Adds enum, cast, nullable-lifting, and non-null enforcement writers.
- Migrates POCO and untyped inserts to converter-derived write plans.
- Adds extensive differential, integration, no-dynamic-code, and performance coverage.
| File | Description |
|---|---|
docs/tcp.mdx |
Documents unified write rules. |
ClickHouse.Driver.Tcp/Types/Converters/ReadRules.cs |
Clarifies shared cast logic. |
ClickHouse.Driver.Tcp/Types/Converters/NullableWriters.cs |
Exposes the inner nullable writer. |
ClickHouse.Driver.Tcp/Types/Converters/DictionaryWriters.cs |
Uses revised interner API. |
ClickHouse.Driver.Tcp/Types/Converters/ConverterDerivation.WriteRules.cs |
Implements root write-rule derivation. |
ClickHouse.Driver.Tcp/Types/Converters/ConverterDerivation.cs |
Applies rules to refused writes. |
ClickHouse.Driver.Tcp/Types/Converters/ClrKeyedByteInterner.cs |
Optimizes repeated-value lookup. |
ClickHouse.Driver.Tcp/Types/ClickHouseTcpTypes.cs |
Documents unified CanWrite behavior. |
ClickHouse.Driver.Tcp/Protocol/ClickHouseTcpConnection.cs |
Uses converter plans for row columns. |
ClickHouse.Driver.Tcp/Poco/UntypedRowColumns.cs |
Derives untyped write types. |
ClickHouse.Driver.Tcp/Poco/PocoWritePlan.cs |
Builds converter-backed POCO plans. |
ClickHouse.Driver.Tcp/Poco/PocoWriteErrors.cs |
Centralizes row-write errors. |
ClickHouse.Driver.Tcp/Poco/PocoWriteConversion.cs |
Retains shared codec facts. |
ClickHouse.Driver.Tcp/Poco/PocoTypeRegistry.cs |
Caches plans by gather tier. |
ClickHouse.Driver.Tcp/Poco/PocoGatherTier.cs |
Defines compiled/delegate gathers. |
ClickHouse.Driver.Tcp/Poco/PocoColumnBuilder.cs |
Implements converter-backed gathering. |
ClickHouse.Driver.Tcp/Poco/LegacyPocoWriteConversion.cs |
Preserves the differential oracle. |
ClickHouse.Driver.Tcp/Poco/LegacyPocoColumnBuilderFactory.cs |
Preserves legacy POCO gathering. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/WriteRuleWriterTests.cs |
Tests individual rule writers. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/WriteRulesRegistration.cs |
Registers differential rule changes. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/WriteDerivationTests.cs |
Separates intrinsic writes from rules. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/LeafTableTests.cs |
Tests intrinsic leaf writes. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/InternerTests.cs |
Updates interner coverage. |
ClickHouse.Driver.Tcp.Tests/Types/Converters/ConverterDerivationTests.cs |
Tests intrinsic refusals. |
ClickHouse.Driver.Tcp.Tests/Program.cs |
Adds no-dynamic-code write mode. |
ClickHouse.Driver.Tcp.Tests/Poco/RowWriteRegistration.cs |
Adds row-write differential tiers. |
ClickHouse.Driver.Tcp.Tests/Poco/PocoWritePlanTests.cs |
Expands write-plan coverage. |
ClickHouse.Driver.Tcp.Tests/Poco/PocoWritePlanDelegateTests.cs |
Repeats tests through delegate gathering. |
ClickHouse.Driver.Tcp.Tests/Poco/PocoWriteNoDynamicCodeTests.cs |
Tests writes without dynamic code. |
ClickHouse.Driver.Tcp.Tests/Poco/PocoReadNoDynamicCodeTests.cs |
Generalizes child-process execution. |
ClickHouse.Driver.Tcp.Tests/Poco/PocoColumnBuilderFactoryTests.cs |
Tests tiers and failure ordering. |
ClickHouse.Driver.Tcp.Tests/Integration/WriteRulesIntegrationTests.cs |
Verifies rules against ClickHouse. |
ClickHouse.Driver.Tcp.Tests/Integration/ClickHouseTcpTypesIntegrationTests.cs |
Extends CanWrite integration cases. |
ClickHouse.Driver.Tcp.Tests/Differential/RowWriteArms.cs |
Implements row-write candidate arms. |
ClickHouse.Driver.Tcp.Tests/Differential/LegacyRowWrite.cs |
Implements legacy references. |
ClickHouse.Driver.Tcp.Tests/Differential/Expectation.cs |
Supports cross-tier write expectations. |
ClickHouse.Driver.Tcp.Tests/Differential/DifferentialTests.cs |
Pins new outcome counts. |
ClickHouse.Driver.Tcp.Tests/Differential/DifferentialRegistry.cs |
Registers additional write tiers. |
ClickHouse.Driver.Tcp.Tests/Differential/DifferentialCase.cs |
Defines row-write facets. |
ClickHouse.Driver.Tcp.Tests/Differential/ClientArms.cs |
Registers client and reference arms. |
ClickHouse.Driver.Tcp.Tests/Differential/Arms.cs |
Generalizes write and answer arms. |
ClickHouse.Driver.Benchmark/TcpConverterRowWrite.cs |
Benchmarks row insert paths. |
changelog.d/converter-5-write-rules.improvements.md |
Records unified write behavior. |
changelog.d/800-untyped-insert-composite-values.fixes.md |
Records the composite insert fix. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Summary This PR moves the native TCP client's POCO and untyped row inserts onto the converter layer and unifies enum, cast, and nullable write rules across inserts and Final verdict✅ Approve Automated review by Polyglot AI (gpt-6-astra) of |
TcpConverterRowWrite prices InsertRowsAsync<T> (the POCO write plan) and InsertRowsAsync(object[]) (the untyped row columns) for one block of 10,000 rows, with the write shapes of the converter spike as rows: LowCardinality(String) from string with many repeats and with every value distinct, Nullable(DateTime) from DateTimeOffset?, Array(String) from string[], and a wide row of six columns. One operation opens the column source of the insert, plans the columns as InsertAsync plans the columns of a row insert, gathers the rows, writes the data block and flushes, in memory. Same attributes and category as TcpConverterWrite. The performance gate of the converter stack can then compare the row inserts of the next part with the current ones. Part of ClickHouse/integrations#801. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the own writes of the whole column type refuse a CLR type, the write derivation tries the rules of POCO mapping, through the column type's own write of another CLR type: - a CLR enum is written as its ordinal (EnumWriter, a span cast); - a nullable enum as its nullable ordinal (NullableEnumWriter); - a nullable value type into a type written from its value type, which throws at the first NULL, before the inner writer starts (NonNullWriter, with a message that names the column, the type and the row of the insert); - a value into a type that it casts to with no conversion, such as object, a base class, or an array that the CLR reads as another array (AssignWriter; the targets come from WriteRules.CastTargets, filtered by ReadRules.IsAssignable, so the array casts that read elements as another type are decided in one place for both directions); - a value type into a type written from its nullable type (LiftWriter; for Nullable(X) a null map of zeros and X from the same values). The rules apply to the root only, as the read rules do, and through every CLR type that the column type is written from, not only through the codec's preferred write types. InsertAsync and CanWrite take them now: the differential tests declare the 8 writes and 13 answers that mirror the 13 read-rule readings. Three tests compare the column types' own writes with the codecs, with no rule at the root (DeriveNode). Part of ClickHouse/integrations#801. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
InsertRowsAsync<T> and InsertRowsAsync(object[]) use the converter derivation, with the write rules: - POCO: the plan derives the tree of each column type for the property type. A refusal keeps the old text. The gather copies each property value into the buffer of its column, and checks a null that the column cannot hold before the block is written, with the old text (a reference type into a type with no NULL, a nullable value type whose tree writes the value type). The converter tree of the insert plan writes the gathered values, so the enum, nullable and cast rules are the derivation's, not the plan's. - Untyped rows: the write type of a column comes from the derivation for the CLR type of its first value that is not null: the type that a cast rule writes the values as (object for a Variant), the nullable type of a value type for a type that holds NULL, else the type of the values. A composite column is written from every CLR type that it is written from (ClickHouse/integrations#800). - The insert plan writes every column through InsertColumnWrite.For, also the columns that a row insert builds. - The gather runs as a compiled loop (PocoGatherTier.Compiled) where the runtime compiles expression trees, else with a getter delegate for each row (PocoGatherTier.Delegate), which compiles no code. The registry can force the tier for the tests. The old POCO plan stays as the reference of the differential tests (PocoWritePlan.BuildLegacy, LegacyPocoColumnBuilderFactory, LegacyPocoWriteConversion); the old untyped choice is copied into the test project (LegacyRowWrite). The differential tests get four tiers, PocoWrite, PocoCanWrite, UntypedWrite and UntypedCanWrite, which run every write facet for all rows and from a row above 0, against the old row inserts, with ClickHouseTcpTypes.CanWrite in both answer tiers. The declared changes: the added writes that reach the row inserts (FixedString from text, LowCardinality(String) from byte[]), the nullable rules for CLR types that are not the codec's preferred types, and ClickHouse/integrations#800's untyped values. Four plan tests that asserted the CLR type of the gathered column assert the bytes of the insert against the old plan; the plan tests run in both gather tiers. Part of ClickHouse/integrations#801. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WriteRulesTests runs 51 column types and 55 CLR source types (enums of each width, nullable sources and targets, casts to object and to a base class, the array casts that read elements as another type, tuples, maps, calendar types): - the derivation accepts every source that the old POCO write plan accepted, and the listed additions (the cast into LowCardinality(String) from byte[], the nullable rules for CLR types that are not the codec's preferred types); - for each source that the old plan accepted, both gather tiers write the same bytes or fail with the same exception and text, with and without a NULL, for all rows and from row 1; one listed change, a byte[] value into Variant(Int32, String), goes to the String alternative; - the columnar insert accepts a source exactly when the POCO plan does and CanWrite says true, and writes the same bytes; - the untyped insert gives the old outcome where the old one accepted the values, and accepts exactly the values that CanWrite accepts; - the array casts that the write cast rule uses are listed. Part of ClickHouse/integrations#801. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- WriteRuleWriterTests: the writer that each rule gives; the refusal text of a source that no rule writes; types that no writer takes (pointer, by-reference, ref struct, open generic, void) refused with no exception; each rule writer against the writer of the type it writes as, over segments, from a row above 0, and under marked positions; a NULL into a type with no NULL stops the write at the first NULL, before any byte, and names the column, the type and the row. - PocoColumnBuilderFactoryTests: the tier choice; a property getter that throws and a NULL give the first failure of the old plan in both tiers; a property of a base class in both tiers; two lone surrogates share one LowCardinality dictionary entry in both row inserts, where the old plan wrote two. - PocoWriteNoDynamicCodeTests: the row inserts in a child process with dynamic code off choose the Delegate tier and write the bytes of the compiled gather of the test process, and fail at the same NULL. The child-process launcher of the read test takes the argument of the test. Part of ClickHouse/integrations#801. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WriteRulesIntegrationTests, two rows to a block: - the three shapes of ClickHouse/integrations#800 through untyped rows: Array(DateTime) from DateTime[], LowCardinality(DateTime) from DateTime, Array(Nullable(DateTime)) from DateTime?[]. With the old write type choice each insert throws "was given values of type ..., which it cannot be written from" (checked by hand); - InsertAsync from a column that only a write rule writes: an enum as its ordinal (Enum8 and Int32), a nullable enum, a value type into Nullable and into LowCardinality(Nullable), a nullable value type with no NULL, a cast to object (Variant), an array cast that reads the elements as another type; - InsertRowsAsync<T> with the writes of the converter layer: LowCardinality(String) from byte[], FixedString from text, the nullable rules into LowCardinality(DateTime) and LowCardinality(Nullable(DateTime)); - a NULL into Int32: the POCO and the untyped inserts fail at the gather with their texts, keep the block before it and the connection; the columnar insert fails with the column, the type and the row, and ends its connection. ClickHouseTcpTypesIntegrationTests gets five CanWrite candidates for the rules (and Int64 from int, which stays false). Part of ClickHouse/integrations#801. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TcpConverterRowWrite repeats the column plan of InsertAsync by hand, because the plan is private. The row inserts write their gathered columns through InsertColumnWrite.For now, so the benchmark plans them that way too, and measures the write path of the insert. The base of the performance gate runs its own copy of the benchmark, which plans the columns with the codec's CanWrite. Part of ClickHouse/integrations#801. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/tcp.mdx gets a "Write rules" section: a CLR enum as its ordinal, a cast, a value type into a nullable column, and a nullable value type into a column that cannot hold NULL, which throws at the first NULL. It says where each insert finds that NULL: the row inserts before the block is sent, with the connection kept; InsertAsync while it writes the block, which closes the connection. The columnar, row and POCO insert sections, the writing tables and the CanWrite paragraph refer to the rules, and the row inserts section says that the first value of a column selects how the column is written. Changelog fragments for ClickHouse/integrations#800 and for the write rules in every insert tier (ClickHouse/integrations#801). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Self-review of the row inserts: - The reference classes of the POCO write plan and of the row inserts say what they are (the reference path over the codecs' preferred write types), not when they were the client's path. - ReinterpretedValues says why a segmented source is copied: the rules apply to the whole column type, whose source is one span. - NonNullWriter says which row its message names. - PocoGatherTier.Delegate says that it compiles no code and that the plan still closes generic types at run time. - docs/tcp.mdx: the later values of an untyped column can also have a type that casts to the type that the first value is written as. - WriteRulesTests: an untyped column whose first value is written through a cast (an IPAddress subclass, uint[] into Array(Int32), sbyte[] into String) takes later values of the target type, as the old insert did. Part of ClickHouse/integrations#801. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ClrKeyedByteInterner.Intern ran once for each row of a LowCardinality(String) write and was not inlined into BytesDictionaryWriter.WriteBody: its try/catch for a refused value kept it out. Each call also read the dictionary size for an error message, and counted the probe with an increment and two compares. The hit path is now one CLR lookup and one decrement, and Intern inlines. A new value, and every value when the probe has turned the CLR lookup off, go to InternNew, which is not inlined. The conversion and the byte lookup are written in InternNew itself: with tiered compilation on, the JIT inlines them there, as it did into the old Intern, and a column of many distinct values keeps its speed (a separate small method for the byte path lost that inlining and was about 7% slower on that shape). The probe counts down to zero, and its end is a method of its own. The dictionary size that names a refused value is read only for a new value, which is the only time the leaf can refuse it. The bytes and the messages do not change: a refused value is still named by the dictionary slot that it would take (ByteInterner.Count is the entry count that the caller passed before). Intern loses its position parameter, so the interner tests call it with the value only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…no NULL
Two shapes of TcpConverterRowWrite: an int? property into Int32 and a
DateTime? property into DateTime('UTC'), with no null in the rows. The
untyped rows of the same shapes hold a boxed int and a boxed DateTime.
The commit changes only the benchmark file, so the file can be put on an
older commit to measure that commit's write of these shapes with the same
benchmark code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s no NULL For a T? property into a column that the derivation writes from T (a NonNullWriter<T>), the POCO write plan gathered the property into a column of T?, and the insert wrote it through the NonNullWriter, which scanned the values for a NULL again and copied them into a second, pooled buffer of T. The gather already refuses a NULL there, with the message of POCO mapping. The gather now copies the T of each value into a column of T, in both gather tiers, and the insert writes that column through the derivation of T, which is the tree inside the NonNullWriter. The values are read once, into one buffer. The columnar InsertAsync keeps the NonNullWriter, which throws at Begin. NonNullWriter<T>.Inner exposes the inner tree, and a matrix test checks for every column type and nullable source that the derivation of the value type gives a tree of the same type. The plan tests check the gathered column type and the bytes of the old plan for int?, DateTime? and an enum. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The details of the rules stay in docs/tcp.mdx and the pull request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
41c99f2 to
75126e6
Compare
This is part 5 of the stack of pull requests for ClickHouse/integrations#801, a converter layer for the native TCP
client. It is stacked on #672 (part 4b, columnar writes through the converter layer), which is on #671, #670, #668, #667,
#666 and #665. It moves the row inserts,
InsertRowsAsync<T>(POCO rows) andInsertRowsAsyncwithobject[]rows,onto the converter derivation, and gives every insert tier and
ClickHouseTcpTypes.CanWritethe write rules of POCOinserts. It fixes ClickHouse/integrations#800 (the untyped insert refuses converted values for composite columns).
What changes for a user
InsertAsync,InsertRowsAsync<T>,InsertRowsAsyncwithobject[]rows, andClickHouseTcpTypes.CanWriteaccept the same CLRtypes. In addition to the type mappings, four rules apply to the whole column type:
enum Status : sbyteintoInt8orEnum8(...));stringinto aVariant, which is written fromobject; a subclass into the column of its base class; and, as the .NET cast allows, auint[]intoArray(Int32), which keeps the bits of each element);intintoNullable(Int32));NULL(anint?intoInt32), and the insertthrows an
InvalidOperationExceptionat the firstNULL, which names the column, its type and the row.Before this part, only
InsertRowsAsync<T>applied them.InsertAsyncandCanWriterefused these types, and theuntyped insert took no enum.
is written from, for example
DateTime[]forArray(DateTime),DateTimeforLowCardinality(DateTime),DateTime?[]forArray(Nullable(DateTime)),stringlabels forArray(Enum8(...)), andbyte[]insideArray,MapandTuple. Before, it chose a write type only from the codec's preferred write types.types. So
InsertRowsAsync<T>also takes aDateTime?property forLowCardinality(DateTime)(it tookDateTime), aDateTimeproperty forLowCardinality(Nullable(DateTime))(it tookDateTime?), and a nullable tuple forTuple(DateTime, String).LowCardinality(String)andLowCardinality(Nullable(String))frombyte[](ClickHouse/integrations#792; with the cast rule also fromsbyte[]and from arrays of byte-sized enums),
FixedString(N)fromstring, aVariantvalue placed in the alternative thatis written from its type (a
byte[]into theStringalternative ofVariant(Int32, String), which the row insertsfailed to write before), and a
LowCardinalitydictionary that holds each distinct encoded value once.NULLfails: the row inserts find aNULLthat the column cannot hold when they convert the rows of itsblock, before they send it, with their messages unchanged: the blocks before it are stored and the connection stays
usable.
InsertAsyncfinds it while it writes the column, inside the block: as for every value thatInsertAsynccannot write, the insert ends its connection.
InsertRowsAsync<T>accepted is accepted, with the same bytes and thesame failure texts, and the untyped insert gives the same outcome for every value that it accepted.
How
at the root only, each through the column type's own write of another CLR type:
EnumWriter(the source span read asordinals with
MemoryMarshal.Cast, span by span, never by an array cast),NullableEnumWriter,NonNullWriter(findsthe first
NULLbefore the inner writer starts),AssignWriter(a span of references read in place; value types boxedinto a pooled buffer),
LiftWriter(forNullable(X), a null map of zeros and X from the same values, with no copy).The order follows the old POCO rule. The cast rule takes its types from the cast rule of the reads
(
ReadRules.IsAssignable), so the CLR array casts that read elements as another type are decided in one place forreads and writes (see Open question).
message. The gather copies each property value into the buffer of its column and checks a
NULLthat the column cannothold, with the old message; the insert plan writes the buffer through the tree. A
T?property into a column that iswritten from
Tis gathered asT, so its values are read once, into one buffer. Where the runtime compiles expressiontrees the gather is one compiled loop for each column; on a runtime without dynamic code it reads each property through a
getter delegate and compiles no code. Both give the same values and the same first failure in row order.
derivation: the type that a cast rule writes the values as (
objectforVariantandDynamic, so values of any typestill go there), the nullable type of a value type for a column that holds
NULL, else the type of the values. Themessages are unchanged.
InsertColumnWrite.For(part 4b), also the columns that the rowinserts build.
ClickHouseTcpTypes.CanWriteis whether the derivation succeeds, so it gives the answer of every insert.LowCardinality(String)writes (a cost of part 4b, see Benchmarks): the CLR-key lookup of the dictionaryinterner is inlined into the write loop. A value that the lookup finds costs one dictionary lookup and one decrement;
a new value goes to a method that is not inlined, with the conversion, the byte lookup, the cleanup after a refused
value and the dictionary size for its message. The bytes and the messages do not change.
LegacyPocoColumnBuilderFactory,LegacyPocoWriteConversion,PocoWritePlan.BuildLegacy); the old untyped type choice is copied into the test project.The last part of the stack deletes them.
Why
The POCO insert had its own write rules (
PocoWriteConversion), which the other tiers did not know, and the untypedinsert asked the codec's preferred-type list, which is incomplete for composite types (ClickHouse/integrations#800). With
the converter derivation, every insert asks one question, and
CanWriteasks the same one.Tests
PocoWrite,PocoCanWrite,UntypedWriteandUntypedCanWrite, run everywrite facet of the case list (the built, decoded and read-back columns, for all rows and for a tail that starts above
row 0) through the row inserts, against the old row inserts: the POCO plan in both gather tiers, the untyped rows, and
ClickHouseTcpTypes.CanWritein both answer tiers, soCanWritegives the answer of the row inserts on every facet. Thedeclared changes: the four writes of part 4 that reach the row inserts, the nullable rules for two
LowCardinality(Nullable(DateTime))answers, and 42 untyped facets of ClickHouse/integrations#800. In the columnartiers, the write rules give 8 writes and 13
CanWriteanswers, the mirror of the 13 read-rule readings of part 2. Thereference outcome counts of the old tiers do not change.
WriteRulesTests, 51 column types x 54 CLR types (enums of each width, nullable sources and targets, casts toobjectand to a base class, the array casts, tuples, maps, calendar types): the derivation accepts every type that theold POCO plan accepted plus 16 listed additions; for each type that the old plan accepted, both gather tiers write the
same bytes or fail with the same exception and text, with and without a
NULL, for all rows and from row 1 (one listedchange: a
byte[]value intoVariant(Int32, String)); the columnar insert accepts exactly what the POCO plan acceptsand writes the same bytes; the untyped insert gives the old outcome for every value that it accepted and agrees with
CanWrite, also for later values of the type that a first value is written as through a cast; the write-side arraycasts are listed.
NULLandits message, before any byte; the gather tiers, a property getter that throws against a
NULLin both orders and in onecolumn, against the old plan; two lone surrogates in one
LowCardinalitydictionary entry through both row inserts.System.Runtime.CompilerServices.RuntimeFeature.IsDynamicCodeSupported: falsechooses the delegate gather by itself and writes the bytes of the compiled gather, for POCO and untyped rows, and fails
at the same
NULL.ClickHouse/integrations#800 through
object[]rows (each throws the issue's error with the old type choice, checked byhand);
InsertAsyncthrough each rule;InsertRowsAsync<T>withbyte[]intoLowCardinality(String), text intoFixedString, and the nullable rules; aNULLintoInt32through each tier (POCO and untyped keep the block beforeit and the connection;
InsertAsyncnames the row and ends its connection); five moreCanWritecandidates.PocoWritePlanTests: four tests asserted the CLR type of the gathered column (the plan converted values in thegather); they now assert the bytes of the insert against the old plan. The plan tests run again in the delegate tier
(
PocoWritePlanDelegateTests).ConverterDerivationTests.Derive_CompositeWriteOfAnotherShape_RefusesWithTheShapeItIsWrittenFrom,LeafTableTests.Derive_EveryCandidateType_AgreesWithTheCurrentAnswersandWriteDerivationTests.Derive_EveryCandidateType_AgreesWithTheCurrentCodeccompare the column type's own writes,without the rules at the root, with the codecs (three refusal texts of the first are now writes of a rule).
InternerTests: theClrKeyedByteInternertests callIntern(value), because the method no longer takes theposition of the value (the interner reads the dictionary slot of a refused value itself). No assertion changes.
T?property into a column with noNULL: a matrix test checks, for every column type and nullable CLR type, thatthe type that the gather writes gives the tree inside the
NULLcheck; the plan tests check the gathered column and thebytes of the old plan for
int?,DateTime?and an enum in both gather tiers.net10.0, the offline suite (no server): 4067 passed each. The public API does not change.
Open question
Resolved in #678: the stack now refuses, in every tier and in both directions, an array cast that gives the elements another meaning (an integer type of the other sign, an enum over another integer type). It keeps an enum array from an array of its exact underlying type, so
Array(Enum8(...))still reads into aMyEnum[].The cast rule accepts the CLR array casts that read the elements as another type: a
uint[]written intoArray(Int32)keeps the bits of each element, and an array of byte-sized enums goes intoLowCardinality(String)asbytes. The reads of part 2 accept the same casts. Both directions take them from one place (
ReadRules.IsAssignable),so refusing them in every tier, reads and writes, is a change in that one method. This part keeps them, as
InsertRowsAsync<T>andQueryAsync<T>did before.Review
on later untyped values of a cast target, and its test.
Write.Insert(LowCardinalityStringHighRepeat)(see Benchmarks). The major was a second scan and a second buffer for aT?property into a column with noNULL: the gather now copies theT, and two benchmark shapes measure it. Theminor was the length of the changelog entry.
Benchmarks
Gate against part 0. The converter benchmarks of part 0 (
TcpConverterRead,TcpConverterWrite), this partagainst part 0, 4 rounds on a shared machine, each round one run of part 0 and then one run of this part. "ratio" is the
median of the four ratios of the medians (this part / part 0, below 1 is faster); "pairs" is the ratio of each round; a
shape fails when it is above 1.05 in every round.
LowCardinality(String)with many repeats is 0.964x (part 4b was 1.028x; see below).Row inserts against part 4b. The first commit adds
TcpConverterRowWrite, the POCO and untyped row inserts of thespike's write shapes, in memory: one operation opens the column source over 10,000 rows, plans the columns, gathers the
rows and writes one data block. Against part 4b with that benchmark commit (the old POCO and untyped gathers, written by
the codecs), measured on the last commit before the review fixes, which only make the
LowCardinality(String)write andthe
T?gather faster (three rounds on the final commit: 0.99x for bothLowCardinalityStringHighRepeatrows). 4rounds, limit 1.05: pass.
Poco(ArrayStringFromText)Poco(LowCardinalityStringHighRepeat)Poco(LowCardinalityStringLowRepeat)Poco(NullableDateTimeFromOffset)Poco(Wide)Untyped(ArrayStringFromText)Untyped(LowCardinalityStringHighRepeat)Untyped(LowCardinalityStringLowRepeat)Untyped(NullableDateTimeFromOffset)Untyped(Wide)The gather alone. A later commit that changes only the benchmark file adds two shapes of a
T?property into acolumn with no
NULL, and its plan step usesInsertColumnWrite.For, as this part's insert does. On a base with thisPR's benchmark file on part 4b, the base also writes the gathered columns through part 4b's converter writers, so this
gate compares only the old gathers with the new ones (and the faster
LowCardinality(String)write). 4 rounds, limit1.05: pass.
Poco(ArrayStringFromText)Poco(LowCardinalityStringHighRepeat)Poco(LowCardinalityStringLowRepeat)Poco(NullableDateTimeFromOffset)Poco(NullableDateTimePropertyIntoDateTime)Poco(NullableIntPropertyIntoInt32)Poco(Wide)Untyped(ArrayStringFromText)Untyped(LowCardinalityStringHighRepeat)Untyped(LowCardinalityStringLowRepeat)Untyped(NullableDateTimeFromOffset)Untyped(NullableDateTimePropertyIntoDateTime)Untyped(NullableIntPropertyIntoInt32)Untyped(Wide)tcp: keep the CLR-key lookup of a LowCardinality string write inlineis a commit of this part that removes a costthat part 4b introduced and that this part increased. Part 4b's
LowCardinality(String)write looked each value upthrough a method that was not inlined (it had a try/catch) and read the dictionary size for each row. On
Write.Insert(LowCardinalityStringHighRepeat)(100 distinct values), part 4b was 1.028x part 0 in its gate, and thispart before the commit was 1.088x part 0 (pairs 1.090, 1.099, 1.086, 1.081) and 1.049x part 4b. This part changes no
code on that path, and the compiled code of the hot methods is the same; with tiered compilation on (a console harness),
part 4b and this part both take 1.21x the time of part 0. After the commit: 0.964x part 0 in the gate above, 0.943x part
4b (three rounds), and 0.99x part 0 with tiered compilation on. All distinct values: 0.815x part 0, 1.01x part 4b. The
bytes and the messages do not change.
🤖 Generated with Claude Code