Skip to content

[WIP]Core, Parquet: Carry avg value sizes for v4 content stats - #17451

Draft
huan233usc wants to merge 1 commit into
apache:mainfrom
huan233usc:geo-avg-value-size-manifest
Draft

[WIP]Core, Parquet: Carry avg value sizes for v4 content stats#17451
huan233usc wants to merge 1 commit into
apache:mainfrom
huan233usc:geo-avg-value-size-manifest

Conversation

@huan233usc

@huan233usc huan233usc commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #17333.

#17333 collects the average serialized WKB size in FieldMetrics, but ParquetMetrics previously discarded it while assembling Metrics. As a result, the value could not reach a DataFile or be exposed to v4 content_stats adapters.

This change keeps the metric in Iceberg's shared in-memory file model without changing the v1-v3 manifest schema.

  • Carry per-field average non-null value sizes through Metrics, ContentFile, file builders, copies, and filtering.
  • Populate the map from Parquet field metrics and verify DataWriter to DataFile propagation.
  • Expose v4 content stats through the legacy ContentFile view used by manifest readers.
  • Preserve compatibility with older Java-serialized Metrics objects.

Scope and dependencies

Test Plan

  • API serialization, including an old serialized Metrics fixture
  • Core content-stats map and tracked-file adapter tests
  • Parquet metrics and DataWriter Geo propagation tests
  • API, Core, and Parquet RevAPI checks
  • API, Core, and Parquet Spotless checks

Verification Commands

./gradlew :iceberg-api:test --tests org.apache.iceberg.TestMetricsSerialization \
  :iceberg-core:test --tests org.apache.iceberg.TestContentStatsBackedMap \
    --tests org.apache.iceberg.TestTrackedFileAdapters \
  :iceberg-data:test \
    --tests org.apache.iceberg.parquet.TestParquetMetrics.testMetricsForGeospatialTypes \
  :iceberg-parquet:test \
    --tests org.apache.iceberg.parquet.TestParquetDataWriter.testGeospatialRoundTrip

./gradlew :iceberg-api:revapi :iceberg-core:revapi :iceberg-parquet:revapi \
  :iceberg-api:spotlessCheck :iceberg-core:spotlessCheck \
  :iceberg-parquet:spotlessCheck

AI Disclosure

  • Model: GPT-5
  • Platform/Tool: Codex
  • Human Oversight: partially reviewed
  • Prompt Summary: Propagate Geo average WKB sizes through Metrics and ContentFile for v4 content stats in a separate follow-up PR.

@github-actions github-actions Bot added API spark parquet core Specification Issues that may introduce spec changes. labels Jul 31, 2026
@huan233usc
huan233usc marked this pull request as draft July 31, 2026 14:54
Propagate average non-null value sizes from Parquet metrics through Metrics and ContentFile so v4 content stats adapters can preserve them. Retain compatibility with older serialized Metrics instances.

Generated-by: Codex
@huan233usc
huan233usc force-pushed the geo-avg-value-size-manifest branch from 018c802 to 4bdbfaf Compare July 31, 2026 17:45
@huan233usc huan233usc changed the title Core, Parquet: Carry avg value sizes for v4 content stats [WIP]Core, Parquet: Carry avg value sizes for v4 content stats Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

API core parquet spark Specification Issues that may introduce spec changes.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant