[python] Cache file-format metadata across reads

[XiaoHongbo-Hope]

Purpose

Cache reusable PyArrow Dataset metadata across repeated reads in the same process.

The process-wide LRU is isolated by filesystem identity and bounded by estimated metadata size. It benefits long-lived readers such as persistent PyTorch workers; separate worker processes keep independent caches. The cache assumes immutable Paimon data files.

Option:

• file-format.metadata-cache.max-size (default: 50 mb; 0 b bypasses)

Tests

• pypaimon/tests/parquet_metadata_cache_test.py
• pypaimon/tests/reader_base_test.py

[JingsongLi]

Please document this in pytorch doc.

[JingsongLi]

Thanks for working on this. I checked how similar systems handle repeated Parquet reads, and this is a well-established optimization rather than a Paimon-specific workaround:

• PyArrow's ParquetFileFragment already caches metadata across scans when the fragment/dataset object is reused (cache_metadata=True by default): https://arrow.apache.org/docs/python/generated/pyarrow.dataset.ParquetFileFragment.html
• DataFusion 50 added a production-ready Parquet metadata cache for footers, statistics, and page indexes. It uses a memory-bounded LRU (50 MB by default), validates entries using file identity metadata, and reports a 12x improvement for point-read microbenchmarks: https://datafusion.apache.org/blog/output/2025/09/29/datafusion-50.0.0/#parquet-metadata-cache
• DuckDB has cached Parquet metadata since 2020 (Add object cache and Parquet metadata cache duckdb/duckdb#1103) and later changed its object cache to a memory-bounded LRU after unbounded metadata entries caused excessive memory usage: Apply LRU cache to parquet metadata duckdb/duckdb#20157

So the overall direction here looks sound, especially for immutable Paimon data files and long-lived PyTorch workers. Reusing the PyArrow Dataset is also a pragmatic way to extend the lifetime of PyArrow's built-in fragment metadata cache. The per-FileIO / filesystem isolation, single-flight loading, fork reset, disabled-by-default behavior, and the test that verifies reduced footer I/O are all useful safeguards.

A few suggestions:

1. Consider bounding by estimated bytes, not only entry count. Footer metadata size can vary substantially with the number of columns and row groups, so 256 entries does not correspond to a predictable memory budget. DataFusion and current DuckDB both use byte-based limits. This could be a follow-up if estimating the retained PyArrow Dataset size is difficult.
2. Please make the immutability/staleness contract explicit. A path-only key is safe for Paimon-managed immutable data files, but could return stale schema/statistics if this reader is ever used for files overwritten in place. More general caches include size, last-modified time, ETag, or object version in the identity.
3. A remote-storage benchmark would make the benefit easier to evaluate. For example: repeated filtered reads over OSS/S3 across multiple epochs, reporting latency, remote request count, metadata size, and cache hit rate. The current tests prove that footer I/O is reduced, but not how much it changes an expected training workload.
4. Some observability would help production tuning. Hit/miss/load/eviction counters and estimated retained metadata bytes would make it easier to choose a cache size and confirm that a workload actually benefits. This is particularly relevant because separately deserialized FileIO instances (for example, separate Ray tasks) intentionally do not share entries.

Overall, this has strong precedent in other Parquet engines. My main design concern is the predictability of memory usage; the cache scope and immutable-file assumption should also be documented clearly.

[JingsongLi]

One additional thought on naming: file-format.metadata-cache.enabled may be a better long-term public option than parquet.metadata-cache-enabled if we intend this to be a capability of FormatPyArrowReader rather than a permanently Parquet-specific feature.

I would suggest:

file-format.metadata-cache.enabled
file-format.metadata-cache.max-entries

max-entries is preferable to size here because the current limit counts cached objects rather than bytes. It also avoids confusion with options such as local-cache.max-size, which represent an actual memory size.

If we use the generic file-format.* name, I think the implementation should be generalized at the same time:

• Route all supported formats through a _get_or_load_file_dataset(...) helper instead of enabling it only under file_format == "parquet".
• Include file_format (and any format options that affect Dataset construction) in the cache key.
• Rename the implementation to something like _FileFormatDatasetCache, while keeping the public option named after the semantic effect rather than the cached Python object.
• Document that the exact cached metadata and benefit are format-dependent. For Parquet, Dataset reuse retains footer-derived fragment metadata; for other formats it may currently only avoid dataset/schema discovery work.

For example, the option description could say:

Cache reusable PyArrow Dataset and fragment metadata across reads in the current process. The cached metadata and performance benefit depend on the file format. For Parquet, this reuses footer-derived metadata such as schema and row-group statistics.

If this PR intentionally guarantees and tests only Parquet footer reuse, keeping a parquet.* public option is more precise. But if we want one stable API that can cover ORC/IPC improvements later, I prefer file-format.metadata-cache.*, provided the code path and cache key are made format-aware now.

[XiaoHongbo-Hope]

Please document this in pytorch doc.

Added

[XiaoHongbo-Hope]

Thanks for working on this. I checked how similar systems handle repeated Parquet reads, and this is a well-established optimization rather than a Paimon-specific workaround:

• PyArrow's ParquetFileFragment already caches metadata across scans when the fragment/dataset object is reused (cache_metadata=True by default): https://arrow.apache.org/docs/python/generated/pyarrow.dataset.ParquetFileFragment.html
• DataFusion 50 added a production-ready Parquet metadata cache for footers, statistics, and page indexes. It uses a memory-bounded LRU (50 MB by default), validates entries using file identity metadata, and reports a 12x improvement for point-read microbenchmarks: https://datafusion.apache.org/blog/output/2025/09/29/datafusion-50.0.0/#parquet-metadata-cache
• DuckDB has cached Parquet metadata since 2020 (Add object cache and Parquet metadata cache duckdb/duckdb#1103) and later changed its object cache to a memory-bounded LRU after unbounded metadata entries caused excessive memory usage: Apply LRU cache to parquet metadata duckdb/duckdb#20157

So the overall direction here looks sound, especially for immutable Paimon data files and long-lived PyTorch workers. Reusing the PyArrow Dataset is also a pragmatic way to extend the lifetime of PyArrow's built-in fragment metadata cache. The per-FileIO / filesystem isolation, single-flight loading, fork reset, disabled-by-default behavior, and the test that verifies reduced footer I/O are all useful safeguards.

A few suggestions:

1. Consider bounding by estimated bytes, not only entry count. Footer metadata size can vary substantially with the number of columns and row groups, so 256 entries does not correspond to a predictable memory budget. DataFusion and current DuckDB both use byte-based limits. This could be a follow-up if estimating the retained PyArrow Dataset size is difficult.
2. Please make the immutability/staleness contract explicit. A path-only key is safe for Paimon-managed immutable data files, but could return stale schema/statistics if this reader is ever used for files overwritten in place. More general caches include size, last-modified time, ETag, or object version in the identity.
3. A remote-storage benchmark would make the benefit easier to evaluate. For example: repeated filtered reads over OSS/S3 across multiple epochs, reporting latency, remote request count, metadata size, and cache hit rate. The current tests prove that footer I/O is reduced, but not how much it changes an expected training workload.
4. Some observability would help production tuning. Hit/miss/load/eviction counters and estimated retained metadata bytes would make it easier to choose a cache size and confirm that a workload actually benefits. This is particularly relevant because separately deserialized FileIO instances (for example, separate Ray tasks) intentionally do not share entries.

Overall, this has strong precedent in other Parquet engines. My main design concern is the predictability of memory usage; the cache scope and immutable-file assumption should also be documented clearly.

Thanks. I’ve documented the immutable-file and per-process/FileIO cache scope. Byte-based limits, observability, and remote-storage benchmarks will be follow-ups, since PyArrow does not expose the exact retained Dataset size. This PR remains opt-in and entry-bounded, with a test verifying reduced footer I/O.

[XiaoHongbo-Hope]

One additional thought on naming: file-format.metadata-cache.enabled may be a better long-term public option than parquet.metadata-cache-enabled if we intend this to be a capability of FormatPyArrowReader rather than a permanently Parquet-specific feature.

I would suggest:

file-format.metadata-cache.enabled
file-format.metadata-cache.max-entries

max-entries is preferable to size here because the current limit counts cached objects rather than bytes. It also avoids confusion with options such as local-cache.max-size, which represent an actual memory size.

If we use the generic file-format.* name, I think the implementation should be generalized at the same time:

• Route all supported formats through a _get_or_load_file_dataset(...) helper instead of enabling it only under file_format == "parquet".
• Include file_format (and any format options that affect Dataset construction) in the cache key.
• Rename the implementation to something like _FileFormatDatasetCache, while keeping the public option named after the semantic effect rather than the cached Python object.
• Document that the exact cached metadata and benefit are format-dependent. For Parquet, Dataset reuse retains footer-derived fragment metadata; for other formats it may currently only avoid dataset/schema discovery work.

For example, the option description could say:

Cache reusable PyArrow Dataset and fragment metadata across reads in the current process. The cached metadata and performance benefit depend on the file format. For Parquet, this reuses footer-derived metadata such as schema and row-group statistics.

If this PR intentionally guarantees and tests only Parquet footer reuse, keeping a parquet.* public option is more precise. But if we want one stable API that can cover ORC/IPC improvements later, I prefer file-format.metadata-cache.*, provided the code path and cache key are made format-aware now.

Thanks. This PR intentionally guarantees and tests only Parquet footer reuse, so I kept the more precise parquet.* namespace and renamed the limit to parquet.metadata-cache-max-entries. A format-aware generic cache can be introduced if ORC/IPC support is added later.

[JingsongLi]

Just one option: file-format.metadata-cache.max-size = 10 mb.

[JingsongLi]

This is a public API option, so we need to be cautious. Additionally, we’ve already implemented many standardized configurations on the Paimon side, such as file.compression, etc. Please specify the exact issue with these standardized configurations.

[JingsongLi]

Another potentially more useful cache might be the Blob file cache.

[XiaoHongbo-Hope]

Thanks. Updated as suggested:

• Replaced the two Parquet-specific options with file-format.metadata-cache.max-size (10 mb by default; 0 b disables).
• Generalized Dataset caching and cache keys across FormatPyArrowReader formats.
• Changed LRU eviction to use estimated metadata bytes.
• Updated the PyTorch documentation and tests.

The Blob file cache can be handled separately.

[JingsongLi]

Maybe 50MB by default? I remember DataFusion is 50 MB.

[XiaoHongbo-Hope]

Maybe 50MB by default? I remember DataFusion is 50 MB.

DataFusion uses 50 MB per RuntimeEnv, while this cache is scoped per FileIO, so multiple FileIO instances may each use up to the configured limit. Given the difference in scope, what default would you suggest?

[JingsongLi]

Maybe 50MB by default? I remember DataFusion is 50 MB.

DataFusion uses 50 MB per RuntimeEnv, while this cache is scoped per FileIO, so multiple FileIO instances may each use up to the configured limit. Given the difference in scope, what default would you suggest?

Why this cache is scoped per FileIO?

[XiaoHongbo-Hope]

Maybe 50MB by default? I remember DataFusion is 50 MB.

DataFusion uses 50 MB per RuntimeEnv, while this cache is scoped per FileIO, so multiple FileIO instances may each use up to the configured limit. Given the difference in scope, what default would you suggest?

Why this cache is scoped per FileIO?

Revised and updated 50MB as default.