Skip to content

feat(arrow): add RecordBatch bulk insert via C Data Interface - #31

Merged
wudidapaopao merged 3 commits into
chdb-io:mainfrom
hyperbyte-cloud:feat/arrow-bulk-insert
Jul 30, 2026
Merged

wudidapaopao merged 3 commits into
chdb-io:mainfrom
hyperbyte-cloud:feat/arrow-bulk-insert

Conversation

@austin-barrington

@austin-barrington austin-barrington commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Add high-throughput Arrow ingest for RecordBatch and RecordBatchReader, with automatic fallback between direct libchdb FFI (chdb_insert_arrow_*) and a register → INSERT … SELECT FROM ArrowStream('name') → unregister SQL path when the symbols are absent at link time.

Breaking change: ArrowStream, ArrowSchema, and ArrowArray now wrap Arrow C Data Interface pointers (FFI_ArrowArrayStream/Schema/Array). Registered names must be queried via ArrowStream('name'), not as bare table identifiers. Add arrow_stream_table_sql() to build the correct table expression.

Also includes named column lists on the SQL insert path (maps by column name, not position), InsertOptions tuning, session helpers, example 08, integration tests, and docs. Bumps crate version to 1.4.0.

Note

Add Arrow RecordBatch bulk insert to Session via C Data Interface

  • Adds insert_record_batch, insert_record_batches, and insert_record_batch_reader methods to Session and Connection, using Arrow's FFI (C Data Interface) to insert in-memory Arrow data into chDB tables without writing to disk.
  • Two insert paths are supported: a registration-based SQL path using the ArrowStream('name') table function (always available), and a direct FFI path via chdb_insert_arrow_array/chdb_insert_arrow_stream (enabled when the detected libchdb exports these symbols, gated by cfg(direct_arrow_insert)).
  • Adds InsertOptions in src/arrow_options.rs to configure ClickHouse insert settings (e.g. max_threads, max_insert_block_size) passed as a SQL SETTINGS clause or direct FFI options.
  • The arrow feature is enabled by default (pulling in Arrow 59), so downstream crates that set default-features = false will lose Arrow support.
  • Behavioral Change: package version bumps from 1.3.1 to 1.4.0 and the arrow crate becomes a default dependency, increasing compile times and binary size for all users who do not opt out.

Macroscope summarized ee60642.

Add high-throughput Arrow ingest for RecordBatch and RecordBatchReader,
with automatic fallback between direct libchdb FFI (chdb_insert_arrow_*)
and a register → INSERT … SELECT FROM ArrowStream('name') → unregister
SQL path when the symbols are absent at link time.

Breaking change: ArrowStream, ArrowSchema, and ArrowArray now wrap Arrow
C Data Interface pointers (FFI_ArrowArrayStream/Schema/Array). Registered
names must be queried via ArrowStream('name'), not as bare table identifiers.
Add arrow_stream_table_sql() to build the correct table expression.

Also includes named column lists on the SQL insert path (maps by column
name, not position), InsertOptions tuning, session helpers, example 08,
integration tests, and docs. Bumps crate version to 1.4.0.
@CLAassistant

CLAassistant commented Jul 27, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@wudidapaopao

Copy link
Copy Markdown
Contributor

Thanks, this is great. A few non-blocking notes, since arrow is now public API:

  • Bump arrow to 59 — pinning 54 forces all downstreams onto 54 to pass a RecordBatch in.

  • pub use arrow; so users go through chdb_rust::arrow::… instead of pulling their own (mismatched) copy.

  • Make it optional, default-on (optional = true + default = ["arrow"]) so SQL-only users don't build all of arrow.

Pinning arrow 54 forced downstreams onto the same version to pass
RecordBatch into chdb-rust. Re-export arrow as chdb_rust::arrow so
callers share one version, and gate the dependency behind an optional
default-on feature so SQL-only users can skip building arrow.
@austin-barrington

Copy link
Copy Markdown
Contributor Author

@wudidapaopao Fixed, please take another look 😄

@wudidapaopao

Copy link
Copy Markdown
Contributor

LGTM! Just a minor cargo fmt CI failure to fix.

wudidapaopao
wudidapaopao previously approved these changes Jul 29, 2026
@wudidapaopao
wudidapaopao merged commit deef002 into chdb-io:main Jul 30, 2026
3 checks passed
@austin-barrington
austin-barrington deleted the feat/arrow-bulk-insert branch July 31, 2026 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants