Skip to content

docs: InsertQueryPlacement.Url is limited to 128 KiB of encoded statement (http_max_field_value_size), not 1 MiB #702

Open

Description

@polyglotAI-bot

Describe the bug

The docs for InsertOptions.QueryPlacement = InsertQueryPlacement.Url name http_max_uri_size (1 MiB) as the server limit for a statement sent in the URL. The server applies a lower limit first: http_max_field_value_size (128 KiB by default) to each URL query-string value, and the query parameter is one of these values. On .NET 10 the .NET runtime does not limit the URI (the 65,519-character limit applies only to .NET 6 through .NET 9). Thus an INSERT statement with more than 131,072 bytes of URL-encoded text fails with HTML Form Exception: Field value too long, although the URI is far below 1 MiB.

The same claim is in three places:

  • docs/http.mdx:521 ("Insert query placement"): "ClickHouse's http_max_uri_size is 1 MiB by default, while an intermediary may impose a lower limit."
  • ClickHouse.Driver/InsertOptions.cs:54 (XML doc of QueryPlacement): "The server's http_max_uri_size is 1 MiB by default."
  • ClickHouse.Driver/InsertQueryPlacement.cs:19 (XML doc of Url): "the server's http_max_uri_size is 1 MiB by default."

The text says "the effective limit is the lowest imposed by the .NET runtime, an intermediary and the server", but the server limit it gives is not the lowest one.

PostStreamAsync and InsertRawStreamAsync always send the SQL in the URL, so the same 128 KiB limit applies to them (verified for PostStreamAsync below). The docs do not state a limit for them.

Steps to reproduce

  1. On net10.0, create a table whose column list makes the INSERT INTO ... FORMAT RowBinary statement longer than 128 KiB when URL-encoded (below: 700 columns with 200-character names).
  2. Call InsertBinaryAsync with new InsertOptions { QueryPlacement = InsertQueryPlacement.Url }.
  3. The insert fails with HTML Form Exception: Field value too long. The same insert with InsertQueryPlacement.Body succeeds.

Expected behaviour

The docs name the limit that the server applies first. Server evidence (26.9.1.1629, defaults):

SELECT name, value FROM system.settings WHERE name IN ('http_max_field_value_size','http_max_uri_size','max_query_size')
max_query_size             262144
http_max_uri_size          1048576
http_max_field_value_size  131072

Raw HTTP requests with no client, POST /?query=SELECT+length(%27aaa…%27). The length is the encoded query value:

encoded value 131072 bytes -> HTTP/1.1 200 OK
encoded value 131073 bytes -> HTTP/1.1 500  Code: 1000 ... HTML Form Exception: Field value too long
encoded value 131073 bytes + &http_max_field_value_size=1048576 -> HTTP/1.1 500  ... Field value too long

A query-level setting does not raise the limit, because the server parses the URL query string with its default settings. Server source: HTMLForm::readQuery in src/Server/HTTP/HTMLForm.cpp throws Field value too long when value.size() reaches max_field_value_size. It checks the raw value before Poco::URI::decode, so the limit applies to the encoded length.

Code example

var recorder = new Recorder { InnerHandler = new HttpClientHandler() }; // logs the encoded length of the query= value
using var client = new ClickHouseClient(connectionString, new HttpClient(recorder));

foreach (var (cols, placement) in new[] {
    (500, InsertQueryPlacement.Url), (700, InsertQueryPlacement.Url), (700, InsertQueryPlacement.Body) })
{
    var table = $"repro_url_limit_{cols}_{placement}".ToLowerInvariant();
    var names = Enumerable.Range(0, cols).Select(i => $"c{i:D4}_" + new string('x', 194)).ToArray();
    await client.ExecuteNonQueryAsync($"CREATE TABLE {table} ({string.Join(", ", names.Select(n => n + " UInt8"))}) ENGINE = Memory");
    var row = names.Select(_ => (object)(byte)1).ToArray();
    await client.InsertBinaryAsync(table, names, new[] { row }, new InsertOptions { QueryPlacement = placement });
}

// PostStreamAsync always puts the SQL in the URL
var sql = $"INSERT INTO FUNCTION null('x String') /* {new string('p', 140_000)} */ FORMAT TSV";
await client.PostStreamAsync(sql, new MemoryStream("a\n"u8.ToArray()), false, CancellationToken.None);

Output:

cols=500 placement=Url  encodedQueryValue=105051 uri=105303 -> OK, inserted 1, count()=1
cols=700 placement=Url  encodedQueryValue=147051 uri=147303 -> ClickHouseServerException: Poco::Exception. Code: 1000, e.code() = 0, HTML Form Exception: Field value too long
cols=700 placement=Body                                    -> OK, inserted 1, count()=1
PostStreamAsync sqlChars=120055 encodedQueryValue=120063 -> 200
PostStreamAsync sqlChars=140055 encodedQueryValue=140063 -> ClickHouseServerException: Poco::Exception. Code: 1000, e.code() = 0, HTML Form Exception: Field value too long

Error log

ClickHouse.Driver.ClickHouseServerException: Poco::Exception. Code: 1000, e.code() = 0, HTML Form Exception: Field value too long

Suggested fix

Docs only, no code change:

  • In docs/http.mdx:521 and both XML docs, replace the http_max_uri_size sentence with the limit that applies first. For example: "The server limits each URL parameter value, including the statement, to http_max_field_value_size (128 KiB by default, measured on the URL-encoded text). A query-level setting cannot raise it. http_max_uri_size (1 MiB) limits the whole URI."
  • Optionally, state the same limit for PostStreamAsync / InsertRawStreamAsync, which always send the SQL in the URL.

Keep the rest unchanged: the .NET 6 through .NET 9 limit of 65,519 characters is correct and is lower than 128 KiB, so on those runtimes the driver's InvalidOperationException still fires first. The server limit is reached first only on .NET 10.

Configuration

Environment

ClickHouse server

  • ClickHouse Server version: 26.9.1.1629
  • ClickHouse Server non-default settings, if any: none
  • CREATE TABLE statements for tables involved: see code example (ENGINE = Memory, N UInt8 columns)
  • Sample data for all these tables: one row of 1s

Found by automated analysis (Polyglot AI) while working on #700. Verified against a live ClickHouse 26.9 server through InsertBinaryAsync and PostStreamAsync, and with raw HTTP requests, not by reading code alone.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdocumentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions