Skip to content

[iceberg] Omit unknown null counts and bounds in IcebergDataFileMeta#8732

Merged
JingsongLi merged 1 commit into
apache:masterfrom
plusplusjiajia:iceberg-manifest-stats-fixes
Jul 20, 2026
Merged

[iceberg] Omit unknown null counts and bounds in IcebergDataFileMeta#8732
JingsongLi merged 1 commit into
apache:masterfrom
plusplusjiajia:iceberg-manifest-stats-fixes

Conversation

@plusplusjiajia

Copy link
Copy Markdown
Member

Purpose

IcebergDataFileMeta.create publishes wrong stats when a SimpleStats slot is
unknown (null), silently corrupting Iceberg's metrics-based pruning:

  • an unknown null count was published as 0, so IS NULL pruning skips files
    that actually contain nulls;
  • a required field's unknown min/max was read without a null check (the getter of
    a non-nullable type has none) and published as garbage bounds like [0, 0];
  • bounds-ineligible types (nested, variant, vector, blob) were rejected only after
    reading the stats slots, which can crash on a required field.

Unknown counts and bounds are now omitted (a missing entry means unknown per the
Iceberg spec), the getter is built from the nullable copy of the type, and the
eligibility check runs before any slot is read. No format or API change.

@plusplusjiajia
plusplusjiajia marked this pull request as ready for review July 20, 2026 07:02
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit 345526e into apache:master Jul 20, 2026
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants