Skip to content

feat(io): let FileIO wrap a caller-provided filesystem operator - #618

Open
jordepic wants to merge 2 commits into
apache:mainfrom
jordepic:fileio-operator-hook
Open

feat(io): let FileIO wrap a caller-provided filesystem operator#618
jordepic wants to merge 2 commits into
apache:mainfrom
jordepic:fileio-operator-hook

Conversation

@jordepic

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #616

Storage::build resolves the backing opendal operator through a closed scheme match, so an embedding engine cannot supply its own operator (custom service, metrics/caching layer, preconfigured client). Java Paimon exposes this axis through the FileIO interface, the FileIOLoader SPI, and CatalogContext's preferIO/fallbackIO; this is the minimal Rust analog.

Brief change log

  • FileIOBuilder::with_operator(op) stores a prebuilt opendal::Operator; when set, it takes precedence over scheme resolution in build.
  • Storage::Custom { op } variant carries it through; create() reuses the fs-relative path handling.

Tests

  • cargo test -p paimon --lib io:: (85 tests) passes; the builder path is exercised by the existing FileIO suites plus downstream use in StreamFusion, which drives all its state-table IO through an operator injected here.

API and Format

Additive API on FileIOBuilder; no storage-format change.

Documentation

Doc comments on the new builder method.

Embedders that manage their own storage backend today have no way to
hand it to FileIO: Storage::build is a closed match over registered
schemes. FileIOBuilder::with_operator accepts a prebuilt operator (for
example a customized local-filesystem service) and short-circuits the
scheme dispatch through a new Storage::Custom variant. Paths reach the
operator in scheme-relative form, exactly as the built-in
local-filesystem backend receives them, so the operator's root decides
what they resolve against.
@jordepic

Copy link
Copy Markdown
Contributor Author

cc @JingsongLi — this is the minimal Rust analog of Java Paimon's configurable FileIO (FileIOLoader/preferIO); would appreciate your review. #619 is a companion from the same downstream use.

@JingsongLi

Copy link
Copy Markdown
Contributor

Storage::Custom always uses fs_relative_path without exception; for example, s3://bucket/a is converted to 3://bucket/a. As a result, preconfigured operators such as S3 or OSS are effectively unusable, and directory listings will return incorrect paths. It is recommended to include a scheme/path resolver or to explicitly restrict the API and name it the “filesystem operator.”

The hook resolved every path with the local-filesystem rules, which
silently mangle scheme'd paths (s3://bucket/a -> 3://bucket/a). Rename it
with_fs_operator, document the filesystem contract, and reject scheme'd
paths in Storage::CustomFs instead of misresolving them; object-store
operators need the bucket/scheme resolution the built-in backends perform,
which this hook deliberately does not reimplement. Adds resolution and
rejection tests.
@jordepic

Copy link
Copy Markdown
Contributor Author

Thanks @JingsongLi — you're right, the hook resolved every path with the filesystem rules, so a scheme'd path like s3://bucket/a was silently mangled to 3://bucket/a. Took your second suggestion and scoped the API explicitly: renamed to with_fs_operator, documented the filesystem contract (absolute paths, file: URLs, Windows drive paths, resolved against the operator's root), and Storage::CustomFs now rejects scheme'd paths with a clear error instead of misresolving them — object-store operators need the bucket/scheme resolution the built-in backends perform, which this hook deliberately doesn't reimplement. Added resolution and rejection tests, and fixed the rustfmt CI failure.

@jordepic jordepic changed the title feat(io): let FileIO wrap a caller-provided opendal operator feat(io): let FileIO wrap a caller-provided filesystem operator Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FileIO cannot use a caller-provided opendal operator

2 participants