Skip to content

Fix UseFormDataParameters: send SQL over 128 KiB as several query form fields - #701

Open
polyglotAI-bot wants to merge 1 commit into
mainfrom
polyglot/form-data-large-sql-chunking
Open

polyglotAI-bot wants to merge 1 commit into
mainfrom
polyglot/form-data-large-sql-chunking

Conversation

@polyglotAI-bot

@polyglotAI-bot polyglotAI-bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Fixes #700.

With UseFormDataParameters=true, BuildHttpRequestMessageWithFormData sent the whole SQL text as one multipart form field named query. The server limits each form field to http_max_field_value_size (128 KiB by default), so any query with SQL text over 128 KiB failed with HTML Form Exception: Field value too long. This happened with or without parameters, on every entry point (ClickHouseClient and ClickHouseCommand both go through PostSqlQueryAsync).

For a multipart request, the server joins the values of all query fields in order (DynamicQueryHandler::getQuery in src/Server/HTTPHandler.cpp: full_query += it.second for each field with the query parameter name). The fix uses this: the SQL text is now sent as one or more query fields of at most 128 KiB each. A chunk boundary never splits a multi-byte UTF-8 character. A query under 128 KiB still produces one query field with the same text/plain; charset=utf-8 part header as before, so its request does not change.

Moving the SQL to the URL instead would not help: the server applies the same per-value limit to URL query-string values (HTMLForm::readQuery), and proxies often set a lower URL limit. Form-data mode exists to keep large values out of the URL (#334).

Changes

  • ClickHouse.Driver/ClickHouseClient.cs: new AddQueryFormFields splits the UTF-8 SQL text into query fields of at most 128 KiB (the server default of http_max_field_value_size).
  • ClickHouse.Driver.Tests/SQL/FormDataLargeQueryTests.cs: regression tests.
  • docs/http.mdx: the UseFormDataParameters row states the per-parameter limit (see below).
  • changelog.d/700-form-data-large-sql.fixes.md: changelog fragment.

Not changed: one parameter value over 128 KiB

The issue also notes that a single parameter value over 128 KiB hits the same limit. This is a server limit that applies in both modes, so the client cannot avoid it:

  • for a repeated param_<name> field the server keeps the first value, so a value cannot be split across fields;
  • the server applies http_max_field_value_size to URL query-string values too (HTMLForm::readQuery), so moving the value to the URL does not help: a 131,073-byte param_p URL value fails with the same Field value too long;
  • a query-level http_max_field_value_size does not raise it, because the server parses the form with its default settings.

All three verified on 26.9. The UseFormDataParameters row in docs/http.mdx now states this limit.

Test

FormDataLargeQueryTests. All 6 cases fail on main with Field value too long and pass with the fix, on ClickHouse 26.9 and 26.3:

  • ExecuteReaderAsync_FormDataWithSqlOverFieldLimit_SendsValidFieldsAndReturnsLiteralUnchanged: a parameterized SELECT with a ~200 KB string literal, sent to the real server through a forwarding HttpMessageHandler that records the query parts. It asserts:

    • the parameter and the exact literal come back;
    • each query part is at most 128 KiB. This holds on any server configuration, including one that raised the limit;
    • each part decodes as strict UTF-8, and the parts joined give the original SQL.

    The cases are ASCII, plus a 4-byte character with ASCII padding that puts the field boundary at each byte position inside the character. If the no-split loop is removed, the three cases with a boundary inside the character fail. The server alone cannot detect this, because it joins raw bytes.

  • ExecuteNonQueryAsync_FormDataInsertWithInlineRowsOverFieldLimit_InsertsAllRows: an INSERT ... FORMAT JSONEachRow with ~760 KiB of inline rows (six fields). It asserts the row count, sum(id), and that each row's s matches its id, so a lost or reordered field fails it.

The full ClickHouse.Driver.Tests suite (net10.0) passes against 26.9: 14,824 passed, 0 failed.

Pre-PR validation gate

  • Deterministic repro confirmed (new tests fail on main)
  • Root cause documented above
  • Fix targets the root cause (how the SQL is put in the form body)
  • Test fails without fix, passes with fix
  • No existing tests changed or broken
  • Convention compliance verified per AGENTS.md / ClickHouse.Driver/AGENTS.md (changelog fragment, docs, CreateTableName)
  • No public API change

Related: ClickHouse/clickhouse-rs#501 (same bug in the Rust client), #334.

🤖 Generated with Claude Code


Note

Medium Risk
Changes how multipart HTTP requests encode SQL for UseFormDataParameters, affecting large queries and bulk inline inserts; behavior for SQL under 128 KiB is intended to stay identical.

Overview
Fixes UseFormDataParameters failures when SQL text exceeds ClickHouse’s default http_max_field_value_size (128 KiB), which previously surfaced as HTML Form Exception: Field value too long.

ClickHouseClient now sends the SQL as one or more repeated query multipart fields (≤128 KiB UTF-8 bytes each), relying on the server concatenating them in order. Chunk boundaries never split a multi-byte UTF-8 character; queries under the limit still use a single field with the same content type as before.

Adds FormDataLargeQueryTests (large SELECT with captured form parts, plus a multi-field INSERT ... FORMAT JSONEachRow), a changelog note for #700, and docs clarifying the per-value 128 KiB limit for parameters in both URL and form-data modes (individual parameter values are not split by this change).

Reviewed by Cursor Bugbot for commit fc24565. Bugbot is set up for automated code reviews on this repo. Configure here.

…ataParameters

With UseFormDataParameters=true the whole SQL text went into one multipart
`query` field, which the server caps at http_max_field_value_size (128 KiB
by default), so larger queries failed with "Field value too long". The
server joins all `query` fields of a multipart request in order, so the SQL
is now split into fields of at most 128 KiB, never inside a multi-byte
UTF-8 character. Smaller queries produce the same request as before.

Fixes: #700

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 25, 2026 22:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The multipart transport change needs human confirmation of compatibility across supported frameworks, server versions, and Cloud beyond the reported test runs.

No open issues 🎉

What changed in this PR

Fixes #700 in the HTTP client by splitting large SQL text into repeated multipart query fields when UseFormDataParameters is enabled. The native TCP client is unchanged.

Changes:

  • Limits each SQL field to 128 KiB without splitting UTF-8 characters.
  • Adds regression tests for large literals, UTF-8 boundaries, and multi-field inline inserts.
  • Documents the remaining per-parameter size limit and adds a changelog fragment.
File Description
docs/​http.mdx Clarifies parameter-value limits in both transport modes.
ClickHouse.Driver/​ClickHouseClient.cs Splits SQL into size-limited multipart fields.
ClickHouse.Driver.Tests/​SQL/​FormDataLargeQueryTests.cs Tests field sizes, UTF-8 integrity, and large inserts.
changelog.d/​700-form-data-large-sql.fixes.md Records the large-SQL fix for #700.

Review effort: Balanced

@codecov

codecov Bot commented Oct 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UseFormDataParameters=true: SQL text over 128 KiB fails with "HTML Form Exception: Field value too long"

2 participants