Repository navigation
Description
Describe the bug
The docs for InsertOptions.QueryPlacement = InsertQueryPlacement.Url name http_max_uri_size (1 MiB) as the server limit for a statement sent in the URL. The server applies a lower limit first: http_max_field_value_size (128 KiB by default) to each URL query-string value, and the query parameter is one of these values. On .NET 10 the .NET runtime does not limit the URI (the 65,519-character limit applies only to .NET 6 through .NET 9). Thus an INSERT statement with more than 131,072 bytes of URL-encoded text fails with HTML Form Exception: Field value too long, although the URI is far below 1 MiB.
The same claim is in three places:
docs/http.mdx:521("Insert query placement"): "ClickHouse'shttp_max_uri_sizeis 1 MiB by default, while an intermediary may impose a lower limit."ClickHouse.Driver/InsertOptions.cs:54(XML doc ofQueryPlacement): "The server'shttp_max_uri_sizeis 1 MiB by default."ClickHouse.Driver/InsertQueryPlacement.cs:19(XML doc ofUrl): "the server'shttp_max_uri_sizeis 1 MiB by default."
The text says "the effective limit is the lowest imposed by the .NET runtime, an intermediary and the server", but the server limit it gives is not the lowest one.
PostStreamAsync and InsertRawStreamAsync always send the SQL in the URL, so the same 128 KiB limit applies to them (verified for PostStreamAsync below). The docs do not state a limit for them.
Steps to reproduce
- On
net10.0, create a table whose column list makes theINSERT INTO ... FORMAT RowBinarystatement longer than 128 KiB when URL-encoded (below: 700 columns with 200-character names). - Call
InsertBinaryAsyncwithnew InsertOptions { QueryPlacement = InsertQueryPlacement.Url }. - The insert fails with
HTML Form Exception: Field value too long. The same insert withInsertQueryPlacement.Bodysucceeds.
Expected behaviour
The docs name the limit that the server applies first. Server evidence (26.9.1.1629, defaults):
SELECT name, value FROM system.settings WHERE name IN ('http_max_field_value_size','http_max_uri_size','max_query_size')
max_query_size 262144
http_max_uri_size 1048576
http_max_field_value_size 131072
Raw HTTP requests with no client, POST /?query=SELECT+length(%27aaa…%27). The length is the encoded query value:
encoded value 131072 bytes -> HTTP/1.1 200 OK
encoded value 131073 bytes -> HTTP/1.1 500 Code: 1000 ... HTML Form Exception: Field value too long
encoded value 131073 bytes + &http_max_field_value_size=1048576 -> HTTP/1.1 500 ... Field value too long
A query-level setting does not raise the limit, because the server parses the URL query string with its default settings. Server source: HTMLForm::readQuery in src/Server/HTTP/HTMLForm.cpp throws Field value too long when value.size() reaches max_field_value_size. It checks the raw value before Poco::URI::decode, so the limit applies to the encoded length.
Code example
var recorder = new Recorder { InnerHandler = new HttpClientHandler() }; // logs the encoded length of the query= value
using var client = new ClickHouseClient(connectionString, new HttpClient(recorder));
foreach (var (cols, placement) in new[] {
(500, InsertQueryPlacement.Url), (700, InsertQueryPlacement.Url), (700, InsertQueryPlacement.Body) })
{
var table = $"repro_url_limit_{cols}_{placement}".ToLowerInvariant();
var names = Enumerable.Range(0, cols).Select(i => $"c{i:D4}_" + new string('x', 194)).ToArray();
await client.ExecuteNonQueryAsync($"CREATE TABLE {table} ({string.Join(", ", names.Select(n => n + " UInt8"))}) ENGINE = Memory");
var row = names.Select(_ => (object)(byte)1).ToArray();
await client.InsertBinaryAsync(table, names, new[] { row }, new InsertOptions { QueryPlacement = placement });
}
// PostStreamAsync always puts the SQL in the URL
var sql = $"INSERT INTO FUNCTION null('x String') /* {new string('p', 140_000)} */ FORMAT TSV";
await client.PostStreamAsync(sql, new MemoryStream("a\n"u8.ToArray()), false, CancellationToken.None);Output:
cols=500 placement=Url encodedQueryValue=105051 uri=105303 -> OK, inserted 1, count()=1
cols=700 placement=Url encodedQueryValue=147051 uri=147303 -> ClickHouseServerException: Poco::Exception. Code: 1000, e.code() = 0, HTML Form Exception: Field value too long
cols=700 placement=Body -> OK, inserted 1, count()=1
PostStreamAsync sqlChars=120055 encodedQueryValue=120063 -> 200
PostStreamAsync sqlChars=140055 encodedQueryValue=140063 -> ClickHouseServerException: Poco::Exception. Code: 1000, e.code() = 0, HTML Form Exception: Field value too long
Error log
ClickHouse.Driver.ClickHouseServerException: Poco::Exception. Code: 1000, e.code() = 0, HTML Form Exception: Field value too long
Suggested fix
Docs only, no code change:
- In
docs/http.mdx:521and both XML docs, replace thehttp_max_uri_sizesentence with the limit that applies first. For example: "The server limits each URL parameter value, including the statement, tohttp_max_field_value_size(128 KiB by default, measured on the URL-encoded text). A query-level setting cannot raise it.http_max_uri_size(1 MiB) limits the whole URI." - Optionally, state the same limit for
PostStreamAsync/InsertRawStreamAsync, which always send the SQL in the URL.
Keep the rest unchanged: the .NET 6 through .NET 9 limit of 65,519 characters is correct and is lower than 128 KiB, so on those runtimes the driver's InvalidOperationException still fires first. The server limit is reached first only on .NET 10.
Configuration
Environment
- Client version:
main@ c8a72ab (the docs text has been there since 1.5.0, from feat(insert): option to send the INSERT statement in the query URL parameter #580) - Language version: C# (SDK 10.0.203)
- .NET version: net10.0 (runtime 10.0.14)
- OS: Linux (Docker, Debian-based image)
ClickHouse server
- ClickHouse Server version: 26.9.1.1629
- ClickHouse Server non-default settings, if any: none
CREATE TABLEstatements for tables involved: see code example (ENGINE = Memory, NUInt8columns)- Sample data for all these tables: one row of
1s
Found by automated analysis (Polyglot AI) while working on #700. Verified against a live ClickHouse 26.9 server through InsertBinaryAsync and PostStreamAsync, and with raw HTTP requests, not by reading code alone.
Activity