feat(io): let FileIO wrap a caller-provided filesystem operator - #618
feat(io): let FileIO wrap a caller-provided filesystem operator#618jordepic wants to merge 2 commits into
Conversation
Embedders that manage their own storage backend today have no way to hand it to FileIO: Storage::build is a closed match over registered schemes. FileIOBuilder::with_operator accepts a prebuilt operator (for example a customized local-filesystem service) and short-circuits the scheme dispatch through a new Storage::Custom variant. Paths reach the operator in scheme-relative form, exactly as the built-in local-filesystem backend receives them, so the operator's root decides what they resolve against.
|
cc @JingsongLi — this is the minimal Rust analog of Java Paimon's configurable FileIO ( |
|
|
The hook resolved every path with the local-filesystem rules, which silently mangle scheme'd paths (s3://bucket/a -> 3://bucket/a). Rename it with_fs_operator, document the filesystem contract, and reject scheme'd paths in Storage::CustomFs instead of misresolving them; object-store operators need the bucket/scheme resolution the built-in backends perform, which this hook deliberately does not reimplement. Adds resolution and rejection tests.
|
Thanks @JingsongLi — you're right, the hook resolved every path with the filesystem rules, so a scheme'd path like |
Purpose
Linked issue: close #616
Storage::buildresolves the backing opendal operator through a closed scheme match, so an embedding engine cannot supply its own operator (custom service, metrics/caching layer, preconfigured client). Java Paimon exposes this axis through theFileIOinterface, theFileIOLoaderSPI, andCatalogContext'spreferIO/fallbackIO; this is the minimal Rust analog.Brief change log
FileIOBuilder::with_operator(op)stores a prebuiltopendal::Operator; when set, it takes precedence over scheme resolution inbuild.Storage::Custom { op }variant carries it through;create()reuses the fs-relative path handling.Tests
cargo test -p paimon --lib io::(85 tests) passes; the builder path is exercised by the existing FileIO suites plus downstream use in StreamFusion, which drives all its state-table IO through an operator injected here.API and Format
Additive API on
FileIOBuilder; no storage-format change.Documentation
Doc comments on the new builder method.