Repository navigation
feat(arrow): add RecordBatch bulk insert via C Data Interface - #31
Merged
wudidapaopao merged 3 commits intoJul 30, 2026
Merged
Conversation
Add high-throughput Arrow ingest for RecordBatch and RecordBatchReader,
with automatic fallback between direct libchdb FFI (chdb_insert_arrow_*)
and a register → INSERT … SELECT FROM ArrowStream('name') → unregister
SQL path when the symbols are absent at link time.
Breaking change: ArrowStream, ArrowSchema, and ArrowArray now wrap Arrow
C Data Interface pointers (FFI_ArrowArrayStream/Schema/Array). Registered
names must be queried via ArrowStream('name'), not as bare table identifiers.
Add arrow_stream_table_sql() to build the correct table expression.
Also includes named column lists on the SQL insert path (maps by column
name, not position), InsertOptions tuning, session helpers, example 08,
integration tests, and docs. Bumps crate version to 1.4.0.
Contributor
|
Thanks, this is great. A few non-blocking notes, since
|
Pinning arrow 54 forced downstreams onto the same version to pass RecordBatch into chdb-rust. Re-export arrow as chdb_rust::arrow so callers share one version, and gate the dependency behind an optional default-on feature so SQL-only users can skip building arrow.
Contributor
Author
|
@wudidapaopao Fixed, please take another look 😄 |
Contributor
|
LGTM! Just a minor cargo fmt CI failure to fix. |
wudidapaopao
previously approved these changes
Jul 29, 2026
wudidapaopao
approved these changes
Jul 30, 2026
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add high-throughput Arrow ingest for RecordBatch and RecordBatchReader, with automatic fallback between direct libchdb FFI (chdb_insert_arrow_*) and a register → INSERT … SELECT FROM ArrowStream('name') → unregister SQL path when the symbols are absent at link time.
Breaking change: ArrowStream, ArrowSchema, and ArrowArray now wrap Arrow C Data Interface pointers (FFI_ArrowArrayStream/Schema/Array). Registered names must be queried via ArrowStream('name'), not as bare table identifiers. Add arrow_stream_table_sql() to build the correct table expression.
Also includes named column lists on the SQL insert path (maps by column name, not position), InsertOptions tuning, session helpers, example 08, integration tests, and docs. Bumps crate version to 1.4.0.
Note
Add Arrow RecordBatch bulk insert to
Sessionvia C Data Interfaceinsert_record_batch,insert_record_batches, andinsert_record_batch_readermethods toSessionandConnection, using Arrow's FFI (C Data Interface) to insert in-memory Arrow data into chDB tables without writing to disk.ArrowStream('name')table function (always available), and a direct FFI path viachdb_insert_arrow_array/chdb_insert_arrow_stream(enabled when the detectedlibchdbexports these symbols, gated bycfg(direct_arrow_insert)).InsertOptionsin src/arrow_options.rs to configure ClickHouse insert settings (e.g.max_threads,max_insert_block_size) passed as a SQLSETTINGSclause or direct FFI options.arrowfeature is enabled by default (pulling in Arrow 59), so downstream crates that setdefault-features = falsewill lose Arrow support.arrowcrate becomes a default dependency, increasing compile times and binary size for all users who do not opt out.Macroscope summarized ee60642.