UseFormDataParameters=true: SQL text over 128 KiB fails with "HTML Form Exception: Field value too long" #700

Description

With UseFormDataParameters=true, ClickHouseClient.PostSqlQueryAsync builds the request through BuildHttpRequestMessageWithFormData (ClickHouse.Driver/ClickHouseClient.cs:392), which puts the SQL text itself into a multipart form field named query (ClickHouse.Driver/ClickHouseClient.cs:411-413) alongside the param_<name> fields.

ClickHouse caps every form field at http_max_field_value_size (128 KiB by default), so in this mode any query whose SQL text exceeds 128 KiB fails before execution with:

Poco::Exception. Code: 1000, e.code() = 0, HTML Form Exception: Field value too long

The default mode (UseFormDataParameters=false) is unaffected: BuildHttpRequestMessageWithQueryParams sends the SQL as the raw text/sql request body and only the parameters go into the URI, so the body is bounded by max_query_size (256 KiB) rather than the form-field cap.

Realistic cases that hit this in form-data mode: an INSERT ... FORMAT JSONEachRow carrying its rows inline, a large inline IN (...) list, or a generated query with many branches. A single parameter value over 128 KiB hits the same cap.

Passing http_max_field_value_size as a URL setting does not help — the form is parsed before the setting applies — and a readonly user cannot set it anyway.

This matters beyond the opt-in flag: #334 proposes making form-data parameters the default to avoid HTTP 414 with large parameter values. Flipping that default as-is would make this failure the default behaviour, which is precisely the regression reported in clickhouse-rs (ClickHouse/clickhouse-rs#501, their 0.15.3). The two issues pull in opposite directions and the fix should account for both.

ClickHouse server version

26.10.2.103 (official build), reached over HTTP at localhost:8123. Verified against this running server.

Reproduction

NUnit test in ClickHouse.Driver.Tests. The first case is a control showing the default path works; the second is the bug.

using System.Threading.Tasks;
using ClickHouse.Driver.ADO.Parameters;
using ClickHouse.Driver.Tests.Utilities;
using ClickHouse.Driver.Utility;
using NUnit.Framework;

namespace ClickHouse.Driver.Tests;

public class LargeQueryTests
{
    private static ClickHouseParameterCollection Params()
    {
        var p = new ClickHouseParameterCollection();
        p.AddParameter("p", 1);
        return p;
    }

    // ~200 KiB of SQL text: over the 128 KiB form-field cap, under max_query_size (256 KiB).
    private static string BigSql() => "SELECT {p:Int32}, length('" + new string('x', 200_000) + "')";

    [Test]
    public async Task LargeParameterizedQuery_UrlParameters_ShouldSucceed()
    {
        using var client = TestUtilities.GetTestClickHouseClient(useFormDataParameters: false);
        var result = await client.ExecuteScalarAsync(BigSql(), Params());
        Assert.That(result, Is.EqualTo(1));
    }

    [Test]
    public async Task LargeParameterizedQuery_FormDataParameters_ShouldSucceed()
    {
        using var client = TestUtilities.GetTestClickHouseClient(useFormDataParameters: true);
        var result = await client.ExecuteScalarAsync(BigSql(), Params());
        Assert.That(result, Is.EqualTo(1));
    }
}

Expected: both tests pass; the SQL size should not depend on how parameters are transported.

Actual: Failed! - Failed: 1, Passed: 1, Skipped: 0, Total: 2

• LargeParameterizedQuery_UrlParameters_ShouldSucceed — passes.
• LargeParameterizedQuery_FormDataParameters_ShouldSucceed — fails:

ClickHouse.Driver.ClickHouseServerException : Poco::Exception. Code: 1000, e.code() = 0,
HTML Form Exception: Field value too long (version 26.10.2.103 (official build))
   at ClickHouse.Driver.ClickHouseClient.HandleError(...) in ClickHouse.Driver/ClickHouseClient.cs:line 1200
   at ClickHouse.Driver.ClickHouseClient.PostSqlQueryAsync(...) in ClickHouse.Driver/ClickHouseClient.cs:line 336

Equivalent curl shapes against the same server:

python3 -c "print('SELECT length(\'' + 'x'*200000 + '\')')" > big.sql
curl -s localhost:8123/ -F 'query=<big.sql' -F 'param_p=1'      # form-data shape: Field value too long
curl -s 'localhost:8123/?param_p=1' --data-binary @big.sql      # default shape: works

Suggested fix

ClickHouse.Driver/ClickHouseClient.cs:392-424 — the SQL does not have to travel as a form field. Options, roughly in order of how much they cost:

1. Keep the parameters in the multipart body but move the SQL out of it, e.g. into the query URI parameter via ClickHouseUriBuilder. That keeps the thing form-data mode exists for (large parameter values off the URI) while letting the SQL use the larger http_max_uri_size (1 MiB) budget. Note the SQL then counts against the URI limit, so this trades one cap for a looser one rather than removing it.
2. Choose the transport per request: use the form-data shape only when the SQL fits under the field cap, and otherwise fall back to the raw-body shape. This interacts with Query parameters sent in URI by default cause HTTP 414 with large IN-lists / vector embeddings #334, where the parameters are what is oversized.
3. At minimum, document the 128 KiB http_max_field_value_size cap on UseFormDataParameters in docs/http.mdx (the settings table at line 105 and the multipart note at line 1139) and in the ClickHouseClientSettings.UseFormDataParameters XML docs, and surface a clearer client-side error than the raw Poco exception.

A single parameter value over 128 KiB has the same problem in this mode and should be covered by whichever fix is chosen.

Link

Relayed from ClickHouse/clickhouse-rs#501 — ClickHouse/clickhouse-rs#501

Related: #334 (the opposite failure mode, oversized parameters in the default URI transport).