Skip to content

API, Core: Make variant classes serializable - #17260

Merged
rdblue merged 1 commit into
apache:mainfrom
rdblue:variant-serializable
Jul 17, 2026
Merged

API, Core: Make variant classes serializable#17260
rdblue merged 1 commit into
apache:mainfrom
rdblue:variant-serializable

Conversation

@rdblue

@rdblue rdblue commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Testing FieldStats with all types for #17159 uncovered that variant instances are not Serializable. This makes all the variant classes serializable, so that stats can be serialized in tasks. Stats are not currently serialized in tasks because stats are discarded during planning (they're expensive) but I think it's a good idea to ensure that stats can be serialized.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
}

private Object writeReplace() {
return new SerializationProxy(this);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of the Serialized classes use a serialization proxy so that the serialized form of the variant is used.

private ByteBuffer buffer = null;
// binary values are held as a serializable byte array rather than a non-serializable ByteBuffer
private final byte[] binary;
private transient ByteBuffer buffer = null;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only change needed for primitives was to track binary values using a byte[] instead of the ByteBuffer. This is consistent with how we handle Serializable in other classes that have binary data.

private final Map<String, VariantValue> shreddedFields = Maps.newHashMap();
private final Set<String> removedFields = Sets.newHashSet();
private SerializationState serializationState = null;
private transient SerializationState serializationState = null;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is lazily created, so there is no problem making it transient.

@rdblue

rdblue commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for reviewing, @stevenzwu!

@rdblue
rdblue merged commit 47c1214 into apache:main Jul 17, 2026
38 checks passed
@nssalian nssalian added this to the Iceberg 1.12.0 milestone Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants