Skip to content

Core: present v4 ContentStats as legacy stat maps via ContentStatsBackedMap #17322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stevenzwu merged 8 commits into from
Jul 24, 2026
Merged

Core: present v4 ContentStats as legacy stat maps via ContentStatsBackedMap #17322

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
dfcd6e9
Core: present v4 ContentStats as legacy stat maps via ContentStatsBac…
stevenzwu Jul 21, 2026
132377a
Core: address review feedback on ContentStatsBackedMap
stevenzwu Jul 23, 2026
49a135b
Core: keep FieldStatsStruct constructor counts as primitive long
stevenzwu Jul 23, 2026
13c0b61
Core: consolidate stats test helpers into StatsTestUtil
stevenzwu Jul 23, 2026
c2695c2
Core: address Ryan's comments on ContentStatsBackedMap and FieldStats…
stevenzwu Jul 23, 2026
68db8ef
Core: address Gabor's review feedback on ContentStatsBackedMap
stevenzwu Jul 24, 2026
ddc6d8a
Core: revert FieldStatsStruct test changes; mock FieldStats in map tests
stevenzwu Jul 24, 2026
a07d902
Core: drop map-behavior test from the adapter suite
stevenzwu Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
171 changes: 171 additions & 0 deletions core/src/main/java/org/apache/iceberg/ContentStatsBackedMap.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iceberg;

import java.nio.ByteBuffer;
import java.util.AbstractMap;
import java.util.Map;
import java.util.Set;
import org.apache.iceberg.relocated.com.google.common.collect.Sets;
import org.apache.iceberg.types.Conversions;
import org.apache.iceberg.types.Type;

/**
* A lazy, read-only {@link Map} view of one stat across the columns of a {@link ContentStats},
* keyed by field ID, mirroring the per-column stat maps on {@link ContentFile}.
*/
class ContentStatsBackedMap<V> extends AbstractMap<Integer, V> {
private enum Kind {
VALUE_COUNT,
NULL_VALUE_COUNT,
NAN_VALUE_COUNT,
LOWER_BOUND,
UPPER_BOUND
}

/** Per-column value counts, or null if no column tracks the value count. */
static <V> Map<Integer, V> valueCounts(ContentStats stats) {
return viewOrNull(stats, Kind.VALUE_COUNT);
}

/** Per-column null value counts, or null if no column tracks the null value count. */
static <V> Map<Integer, V> nullValueCounts(ContentStats stats) {
return viewOrNull(stats, Kind.NULL_VALUE_COUNT);
}

/** Per-column NaN value counts, or null if no column tracks the NaN value count. */
static <V> Map<Integer, V> nanValueCounts(ContentStats stats) {
return viewOrNull(stats, Kind.NAN_VALUE_COUNT);
}

/** Per-column lower bounds, or null if no column tracks a lower bound. */
static <V> Map<Integer, V> lowerBounds(ContentStats stats) {
return viewOrNull(stats, Kind.LOWER_BOUND);
}

/** Per-column upper bounds, or null if no column tracks an upper bound. */
static <V> Map<Integer, V> upperBounds(ContentStats stats) {
return viewOrNull(stats, Kind.UPPER_BOUND);
}

private final ContentStats stats;
private final Kind kind;
private Set<Entry<Integer, V>> materialized;

private ContentStatsBackedMap(ContentStats stats, Kind kind) {
this.stats = stats;
this.kind = kind;
}

@Override
public V get(Object key) {
if (!(key instanceof Integer)) {
return null;
}

FieldStats<?> fieldStats = stats.statsFor((Integer) key);
if (fieldStats == null) {
return null;
}

return statValue(fieldStats, kind);
}

@Override
public boolean containsKey(Object key) {
return get(key) != null;
}

@Override
public boolean isEmpty() {

@gaborkaszab gaborkaszab Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm not mistaken, in the constructing methods we return null if the map were empty. Following this, in case we have constructed a map object it's not empty. Can we simply return false here?

// a factory returns null for an empty view, so a constructed instance always has an entry;
// this also avoids AbstractMap's default, which materializes entrySet() to answer emptiness
return false;
}

@Override
public Set<Entry<Integer, V>> entrySet() {

@gaborkaszab gaborkaszab Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for my understanding: I'm trying to understand the reason for introducing the cached materialized result. Do we expect entrySet to be called multiple times on the same ContentStatsBackedMap object? Apart from that e.g. ContentStatsBackedMap.valueCounts(stats) seems identical to MetricsUtil.valueCounts(stats) to me, unless I miss something.
Asking the question from a different angle, can this entrySet function be "pass-through" without caching similarly to get?

@stevenzwu stevenzwu Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

entrySet() is the only method that has to materialize (it's a full projection), so its result is cached: AbstractMap routes size(), forEach(), toString(), and equals() through entrySet(), and without the cache any caller that does more than a single pass would rebuild the LinkedHashSet every time. get()/containsKey() stay pass-through and allocate nothing.

It isn't equivalent to the old MetricsUtil.valueCounts(stats), which eagerly built and returned a full map on every call. Here the map is a lazy view: a caller that only does get() or a null check never materializes a set, and an empty metric returns null from an allocation-free scan rather than an empty map. That laziness, plus returning null instead of an empty map, is the reason for the class.

if (materialized == null) {
Set<Entry<Integer, V>> entries = Sets.newLinkedHashSet();
for (FieldStats<?> fieldStats : stats.fieldStats()) {
if (fieldStats != null) {
V value = statValue(fieldStats, kind);
if (value != null) {
entries.add(new SimpleImmutableEntry<>(fieldStats.fieldId(), value));
}
}
}

this.materialized = entries;
}

return materialized;
}

private static <V> Map<Integer, V> viewOrNull(ContentStats stats, Kind kind) {
return isEmpty(stats, kind) ? null : new ContentStatsBackedMap<>(stats, kind);
}

/** Returns true if no field tracks the metric. */
private static boolean isEmpty(ContentStats stats, Kind kind) {
if (stats == null) {
return true;
}

for (FieldStats<?> fieldStats : stats.fieldStats()) {

@gaborkaszab gaborkaszab Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe with stream?

return stats.fieldStats().stream()
    .filter(Objects::nonNull)
    .noneMatch(fieldStats -> isKnown(fieldStats, kind));

@stevenzwu stevenzwu Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isEmpty() is on the scan-planning's hot code path. A stream adds a Stream + lambda allocation per call, and ContentStats.fieldStats() returns an Iterable, so it would also need a Streams.stream(...) wrapper. The imperative loop short-circuits on the first contributing field and allocates nothing, so I'd prefer to keep it here.

if (fieldStats != null && isKnown(fieldStats, kind)) {
return false;
}
}

return true;
}

private static boolean isKnown(FieldStats<?> fieldStats, Kind kind) {
return switch (kind) {
case VALUE_COUNT -> fieldStats.hasValueCount();
case NULL_VALUE_COUNT -> fieldStats.hasNullValueCount();
case NAN_VALUE_COUNT -> fieldStats.hasNanValueCount();
case LOWER_BOUND -> fieldStats.lowerBound() != null;
case UPPER_BOUND -> fieldStats.upperBound() != null;
};
}

@SuppressWarnings("unchecked")
private static <V> V statValue(FieldStats<?> fieldStats, Kind kind) {
return switch (kind) {
case VALUE_COUNT ->
fieldStats.hasValueCount() ? (V) Long.valueOf(fieldStats.valueCount()) : null;
case NULL_VALUE_COUNT ->
fieldStats.hasNullValueCount() ? (V) Long.valueOf(fieldStats.nullValueCount()) : null;
case NAN_VALUE_COUNT ->
fieldStats.hasNanValueCount() ? (V) Long.valueOf(fieldStats.nanValueCount()) : null;
case LOWER_BOUND ->
(V) bound(fieldStats, fieldStats.lowerBound(), StatsUtil.LOWER_BOUND_NAME);
case UPPER_BOUND ->
(V) bound(fieldStats, fieldStats.upperBound(), StatsUtil.UPPER_BOUND_NAME);
};
}

private static ByteBuffer bound(FieldStats<?> fieldStats, Object bound, String boundFieldName) {
Type boundType = fieldStats.type().fieldType(boundFieldName);
// toByteBuffer returns null for a null bound
return boundType == null ? null : Conversions.toByteBuffer(boundType, bound);
}
}
15 changes: 12 additions & 3 deletions core/src/main/java/org/apache/iceberg/FieldStats.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,22 @@ interface FieldStats<T> {
*/
boolean tightBounds();

/** The total value count, including null and NaN */
/** Whether a value count is tracked for this field. */
boolean hasValueCount();

/** The total value count, including null and NaN, defined only when {@link #hasValueCount()}. */
long valueCount();

/** The total null value count */

@rdblue rdblue Jul 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we also need a few more methods:

  • hasValueCount: the value count may not be projected or may not have been written
  • missingNullValueCount, missingNaNValueCount: I think it is good to have negations to make code easier to read, like Map.nonEmpty
  • hasAvgValueSizeInBytes: this field may be null for the same reason as the others, so we should treat it the same way (this was an oversight on my part)

@stevenzwu stevenzwu Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add hasValueCount. But let's add others when they are actually used/needed. Introducing them now is a bit weird since there are no callers.

  • missingNullValueCount
  • missingNaNValueCount
  • hasAvgValueSizeInBytes

/** Whether a null value count is tracked for this field. */
boolean hasNullValueCount();

/** The total null value count, defined only when {@link #hasNullValueCount()}. */
long nullValueCount();

/** The total NaN value count */
/** Whether a NaN value count is tracked for this field. */
boolean hasNanValueCount();

/** The total NaN value count, defined only when {@link #hasNanValueCount()}. */
long nanValueCount();

/**
Expand Down
15 changes: 15 additions & 0 deletions core/src/main/java/org/apache/iceberg/FieldStatsStruct.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,16 +143,31 @@ public boolean tightBounds() {
return tightBounds;
}

@Override
public boolean hasValueCount() {
return valueCount != null;
}

@Override
public long valueCount() {
return valueCount;
}

@Override
public boolean hasNullValueCount() {
return nullValueCount != null;
}

@Override
public long nullValueCount() {
return nullValueCount;
}

@Override
public boolean hasNanValueCount() {
return nanValueCount != null;
}

@Override
public long nanValueCount() {
return nanValueCount;
Expand Down
86 changes: 0 additions & 86 deletions core/src/main/java/org/apache/iceberg/MetricsUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,6 @@

import static org.apache.iceberg.types.Types.NestedField.optional;

import java.nio.ByteBuffer;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
Expand Down Expand Up @@ -478,88 +476,4 @@ public <T> void set(int pos, T value) {
throw new UnsupportedOperationException("StructWithReadableMetrics is read only");
}
}

static Map<Integer, Long> valueCounts(ContentStats stats) {
if (stats == null) {
return null;
}

Map<Integer, Long> result = Maps.newHashMap();
for (FieldStats<?> fs : stats.fieldStats()) {
if (fs != null) {
result.put(fs.fieldId(), fs.valueCount());
}
}

return result.isEmpty() ? null : Collections.unmodifiableMap(result);
}

static Map<Integer, Long> nullValueCounts(ContentStats stats) {
if (stats == null) {
return null;
}

Map<Integer, Long> result = Maps.newHashMap();
for (FieldStats<?> fs : stats.fieldStats()) {
if (fs != null) {
result.put(fs.fieldId(), fs.nullValueCount());
}
}

return result.isEmpty() ? null : Collections.unmodifiableMap(result);
}

static Map<Integer, Long> nanValueCounts(ContentStats stats) {
if (stats == null) {
return null;
}

Map<Integer, Long> result = Maps.newHashMap();
for (FieldStats<?> fs : stats.fieldStats()) {
if (fs != null) {
Type boundType = fs.type().fieldType("lower_bound");
if (boundType.typeId() == Type.TypeID.FLOAT || boundType.typeId() == Type.TypeID.DOUBLE) {
result.put(fs.fieldId(), fs.nanValueCount());
}
}
}

return result.isEmpty() ? null : Collections.unmodifiableMap(result);
}

static Map<Integer, ByteBuffer> lowerBounds(ContentStats stats) {
if (stats == null) {
return null;
}

Map<Integer, ByteBuffer> result = Maps.newHashMap();
for (FieldStats<?> fs : stats.fieldStats()) {
if (fs != null) {
Type boundType = fs.type().fieldType("lower_bound");
if (fs.lowerBound() != null && boundType != null) {
result.put(fs.fieldId(), Conversions.toByteBuffer(boundType, fs.lowerBound()));
}
}
}

return result.isEmpty() ? null : Collections.unmodifiableMap(result);
}

static Map<Integer, ByteBuffer> upperBounds(ContentStats stats) {
if (stats == null) {
return null;
}

Map<Integer, ByteBuffer> result = Maps.newHashMap();
for (FieldStats<?> fs : stats.fieldStats()) {
if (fs != null) {
Type boundType = fs.type().fieldType("upper_bound");
if (fs.upperBound() != null && boundType != null) {
result.put(fs.fieldId(), Conversions.toByteBuffer(boundType, fs.upperBound()));
}
}
}

return result.isEmpty() ? null : Collections.unmodifiableMap(result);
}
}
7 changes: 5 additions & 2 deletions core/src/main/java/org/apache/iceberg/StatsUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@ private StatsUtil() {}
static final int NAN_VALUE_COUNT_OFFSET = 6;
static final int AVG_VALUE_SIZE_OFFSET = 7;

static final String LOWER_BOUND_NAME = "lower_bound";
static final String UPPER_BOUND_NAME = "upper_bound";

// Offsets used within geo_lower struct
private static final int GEO_LOWER_X_OFFSET = 10;
private static final int GEO_LOWER_Y_OFFSET = 11;
Expand Down Expand Up @@ -249,12 +252,12 @@ private static Types.StructType geoUpperBound(int baseId) {

private static Types.NestedField lowerBoundField(Type type, int baseId) {
Type boundType = isGeoType(type) ? geoLowerBound(baseId) : type;
return optional(baseId + LOWER_BOUND_OFFSET, "lower_bound", boundType);
Comment on lines 254 to -252

@dramaticlly dramaticlly Jul 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not direct relate to your change, but I think geoLowerBound(baseId) actually returns StructType, but the Conversions.toByteBuffer called within ContentStatsBackedMap.bound() today might not handle the struct type for geometry properly.

iceberg/api/src/main/java/org/apache/iceberg/types/Conversions.java

Lines 95 to 145 in c8a4b98

switch (typeId) {
case BOOLEAN:
return ByteBuffer.allocate(1).put(0, (Boolean) value ? (byte) 0x01 : (byte) 0x00);
case INTEGER:
case DATE:
return ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(0, (int) value);
case LONG:
case TIME:
case TIMESTAMP:
case TIMESTAMP_NANO:
return ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, (long) value);
case FLOAT:
return ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putFloat(0, (float) value);
case DOUBLE:
return ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putDouble(0, (double) value);
case STRING:
CharBuffer buffer = CharBuffer.wrap((CharSequence) value);
try {
return ENCODER.get().encode(buffer);
} catch (CharacterCodingException e) {
throw new RuntimeIOException(e, "Failed to encode value as UTF-8: %s", value);
}
case UUID:
return UUIDUtil.convertToByteBuffer((UUID) value);
case FIXED:
case BINARY:
return (ByteBuffer) value;
case DECIMAL:
return ByteBuffer.wrap(((BigDecimal) value).unscaledValue().toByteArray());
case VARIANT:
// Produce a concatenated buffer of metadata and value
Variant variant = (Variant) value;
VariantMetadata variantMetadata = variant.metadata();
VariantValue variantValue = variant.value();
ByteBuffer variantBuffer =
ByteBuffer.allocate(variantMetadata.sizeInBytes() + variantValue.sizeInBytes())
.order(ByteOrder.LITTLE_ENDIAN);
variantMetadata.writeTo(variantBuffer, 0);
variantValue.writeTo(variantBuffer, variantMetadata.sizeInBytes());
return variantBuffer;
case GEOMETRY:
case GEOGRAPHY:
// Geometry and geography lower/upper bounds are single points encoded as an
// x:y:z:m concatenation of 8-byte little-endian IEEE 754 doubles. See the
// Bound Serialization section of the Iceberg spec.
return ((GeospatialBound) value).toByteBuffer();
case UNKNOWN:
// underlying type not known
return null;
default:
throw new UnsupportedOperationException("Cannot serialize type: " + typeId);
.

I am not sure the status of upper/lower bound of geometry/geography type in main, but it seems we can run into UnsupportedOperationException.

@vishnuprakaz vishnuprakaz Aug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed ,this does throw UnsupportedOperationException for geometry and geography bounds.
Repro and proposed fix in #17493.

return optional(baseId + LOWER_BOUND_OFFSET, LOWER_BOUND_NAME, boundType);
}

private static Types.NestedField upperBoundField(Type type, int baseId) {
Type boundType = isGeoType(type) ? geoUpperBound(baseId) : type;
return optional(baseId + UPPER_BOUND_OFFSET, "upper_bound", boundType);
return optional(baseId + UPPER_BOUND_OFFSET, UPPER_BOUND_NAME, boundType);
}

@VisibleForTesting
Expand Down
10 changes: 5 additions & 5 deletions core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,27 +174,27 @@ public Map<Integer, Long> columnSizes() {

@Override
public Map<Integer, Long> valueCounts() {
return MetricsUtil.valueCounts(file().contentStats());
return ContentStatsBackedMap.valueCounts(file().contentStats());
}

@Override
public Map<Integer, Long> nullValueCounts() {
return MetricsUtil.nullValueCounts(file().contentStats());
return ContentStatsBackedMap.nullValueCounts(file().contentStats());
}

@Override
public Map<Integer, Long> nanValueCounts() {
return MetricsUtil.nanValueCounts(file().contentStats());
return ContentStatsBackedMap.nanValueCounts(file().contentStats());
}

@Override
public Map<Integer, ByteBuffer> lowerBounds() {
return MetricsUtil.lowerBounds(file().contentStats());
return ContentStatsBackedMap.lowerBounds(file().contentStats());
}

@Override
public Map<Integer, ByteBuffer> upperBounds() {
return MetricsUtil.upperBounds(file().contentStats());
return ContentStatsBackedMap.upperBounds(file().contentStats());
}
}

Expand Down
Loading
Loading