Skip to content

feat(c): build a catalog-free Table from a resolved schema JSON - #595

Draft
JunRuiLee wants to merge 2 commits into
apache:mainfrom
JunRuiLee:feat/table-descriptor
Draft

feat(c): build a catalog-free Table from a resolved schema JSON#595
JunRuiLee wants to merge 2 commits into
apache:mainfrom
JunRuiLee:feat/table-descriptor

Conversation

@JunRuiLee

@JunRuiLee JunRuiLee commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Purpose

Add a catalog-free way to build a read-only Table from an already-resolved Paimon TableSchema, so a non-Java engine (e.g. Doris FE/BE via the C FFI) can rebuild a table for scan/read without a catalog lookup.

Today the only way to obtain a Table is Catalog::get_table, which re-resolves the table — risking schema/snapshot drift versus what the planner (the FE) already resolved — and ties the reader to a filesystem-catalog layout.

This is intentionally minimal and does not introduce a new cross-language DTO. An earlier iteration proposed a dedicated TableDescriptor abstraction on the Java side to bundle path/schema/branch, but the community felt it lacked sufficient motivation: it would re-define Paimon table metadata on the Java side and couple the paimon-rust interface to Doris integration details. That approach is dropped in favor of a generic, catalog-free entry point.

Design

The caller (Doris FE) passes everything a reader needs directly:

  • table_path — the table location
  • table_schema_json — the full TableSchema JSON, already resolved by the FE (parsed with the existing serde TableSchema shape; not re-loaded or overridden)
  • database / table_name — used to synthesize the table Identifier
  • branch — selects the branch-scoped managers (schema/snapshot/tag read paths); nullable, defaults to main
  • storage_options — used only to build FileIO; they are not merged into the supplied schema

The Rust side constructs the Table directly: no Catalog, no warehouse derivation, and no re-resolution of the FE-provided schema. branch only decides which branch's schema/snapshot/tag paths reads follow; reads can still use FE-generated splits, and Rust-side time travel remains available through the existing read-builder options.

Brief change log

  • crates/paimon/src/table/mod.rs: Table::from_resolved_schema(file_io, identifier, location, schema, branch) — preserves the supplied schema as-is, validates the branch name, wires a branch-scoped SchemaManager, sets rest_env = None and no time travel.
  • bindings/c/src/table.rs: C FFI paimon_table_from_schema_json(table_path, table_schema_json, database, table_name, branch, storage_options, storage_options_len), mirroring the existing paimon_catalog_get_table handle/free contract, with an ABI signature guard. branch may be null (defaults to main); a non-null pointer is still fully UTF-8 and branch-name validated. Freed via paimon_table_free.
  • Historical per-file schemas are still read from the immutable on-disk schema/schema-N by SchemaManager; the resolved schema pins the planned/current schema only.

Tests

  • paimon-c FFI tests: build a table from a resolved schema JSON; null-branch defaults to main; invalid branch name (e.g. ../dev) rejected; malformed JSON rejected; null-pointer / null storage_options with non-zero length rejected.

API and Format

  • New public API: paimon::table::Table::from_resolved_schema, and C FFI paimon_table_from_schema_json (additive; existing symbols unchanged).
  • No new on-disk or wire format, and no new cross-language DTO. Consumes the existing serde TableSchema JSON.

Documentation

None yet (new, evolving C FFI surface for the Doris integration); docs can follow once the integration stabilizes.

@JunRuiLee
JunRuiLee force-pushed the feat/table-descriptor branch from a8cb300 to 83f2db7 Compare July 23, 2026 04:19
Allow the branch pointer of paimon_table_from_schema_json to be null,
defaulting to DEFAULT_MAIN_BRANCH ("main"). A non-null pointer still goes
through the full UTF-8 and branch-name validation, so invalid names such as
"../dev" are still rejected. Update the doc and Safety comment accordingly
and add a test covering the null-branch default.
@JunRuiLee
JunRuiLee force-pushed the feat/table-descriptor branch from 83f2db7 to 7636fda Compare July 28, 2026 08:31
@JunRuiLee JunRuiLee changed the title feat(table): build a catalog-free Table from a portable TableDescriptor feat(c): build a catalog-free Table from a resolved schema JSON Jul 28, 2026
@JunRuiLee
JunRuiLee marked this pull request as ready for review July 28, 2026 08:33
@JunRuiLee
JunRuiLee marked this pull request as draft July 28, 2026 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant