Skip to content

HIVE-29770: Iceberg: Return VARIANT values as JSON over the HS2 Thrift protocol #6644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
deniskuzZ wants to merge 1 commit into
base: master
Choose a base branch
from
Open

HIVE-29770: Iceberg: Return VARIANT values as JSON over the HS2 Thrift protocol #6644

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions iceberg/iceberg-handler/src/test/queries/positive/variant_type_projection.q
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
-- SORT_QUERY_RESULTS
set hive.explain.user=false;

drop table if exists variant_proj;

CREATE EXTERNAL TABLE variant_proj (
id INT,
data VARIANT
) STORED BY ICEBERG
TBLPROPERTIES (
'format-version'='3'
);

INSERT INTO variant_proj VALUES
(1, parse_json('{"name": "John", "age": 30}')),
(2, parse_json('[1, 2, 3]')),
(3, parse_json('"plain string"')),
(4, parse_json('42'));

@csringhofer csringhofer Jul 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can Hive create VARIANTs from other types?
An interesting case is BINARY. Impala and Trino base64 encodes it, but you can't just create it by parsing JSON

@deniskuzZ deniskuzZ Jul 28, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hive can only create variants from JSON text (parse_json), so binary variants can't be authored in Hive SQL, but Hive fully understands them on the read side, and renders them exactly the way Trino does (base64).

parse_json is the only variant constructor Hive registers. It's the same-named, same-semantics function as Spark's. There is no CAST(x AS VARIANT) in Hive yet


INSERT INTO variant_proj (id) SELECT 5;

-- raw projection: variants are returned as their logical JSON
EXPLAIN SELECT id, data FROM variant_proj;

SELECT id, data FROM variant_proj;

SELECT * FROM variant_proj;

-- raw projection across a shuffle
SELECT id, data FROM variant_proj ORDER BY id DESC;

-- variant nested inside a struct is decoded within the folded output
-- (array<variant>/map<..,variant> reads fail in Iceberg column projection; tracked separately)
drop table if exists variant_nested;

CREATE EXTERNAL TABLE variant_nested (
id INT,
s STRUCT<label:STRING, v:VARIANT>
) STORED BY ICEBERG
TBLPROPERTIES (
'format-version'='3'
);

INSERT INTO variant_nested SELECT
1,
named_struct('label', 'x', 'v', parse_json('{"k": 1}'));

SELECT * FROM variant_nested;

SELECT id, to_json(s.v) FROM variant_nested;

-- a plain struct with the variant shape keeps the generic struct rendering
drop table if exists bin_struct;

CREATE EXTERNAL TABLE bin_struct (
id INT,
s STRUCT<metadata:BINARY, value:BINARY>
) STORED BY ICEBERG;

INSERT INTO bin_struct SELECT 1, named_struct('metadata', cast('x' as binary), 'value', cast('y' as binary));

SELECT * FROM bin_struct;

drop table variant_proj;
drop table variant_nested;
drop table bin_struct;
235 changes: 235 additions & 0 deletions iceberg/iceberg-handler/src/test/results/positive/variant_type_projection.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,235 @@
PREHOOK: query: drop table if exists variant_proj
PREHOOK: type: DROPTABLE
PREHOOK: Output: database:default
POSTHOOK: query: drop table if exists variant_proj
POSTHOOK: type: DROPTABLE
POSTHOOK: Output: database:default
PREHOOK: query: CREATE EXTERNAL TABLE variant_proj (
id INT,
data VARIANT
) STORED BY ICEBERG
TBLPROPERTIES (
'format-version'='3'
)
PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
PREHOOK: Output: default@variant_proj
POSTHOOK: query: CREATE EXTERNAL TABLE variant_proj (
id INT,
data VARIANT
) STORED BY ICEBERG
TBLPROPERTIES (
'format-version'='3'
)
POSTHOOK: type: CREATETABLE
POSTHOOK: Output: database:default
POSTHOOK: Output: default@variant_proj
PREHOOK: query: INSERT INTO variant_proj VALUES
(1, parse_json('{"name": "John", "age": 30}')),
(2, parse_json('[1, 2, 3]')),
(3, parse_json('"plain string"')),
(4, parse_json('42'))
PREHOOK: type: QUERY
PREHOOK: Input: _dummy_database@_dummy_table
PREHOOK: Output: default@variant_proj
POSTHOOK: query: INSERT INTO variant_proj VALUES
(1, parse_json('{"name": "John", "age": 30}')),
(2, parse_json('[1, 2, 3]')),
(3, parse_json('"plain string"')),
(4, parse_json('42'))
POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Output: default@variant_proj
PREHOOK: query: INSERT INTO variant_proj (id) SELECT 5
PREHOOK: type: QUERY
PREHOOK: Input: _dummy_database@_dummy_table
PREHOOK: Output: default@variant_proj
POSTHOOK: query: INSERT INTO variant_proj (id) SELECT 5
POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Output: default@variant_proj
PREHOOK: query: EXPLAIN SELECT id, data FROM variant_proj
PREHOOK: type: QUERY
PREHOOK: Input: default@variant_proj
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: EXPLAIN SELECT id, data FROM variant_proj
POSTHOOK: type: QUERY
POSTHOOK: Input: default@variant_proj
POSTHOOK: Output: hdfs://### HDFS PATH ###
STAGE DEPENDENCIES:
Stage-0 is a root stage

STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
TableScan
alias: variant_proj
Select Operator
expressions: id (type: int), data (type: struct<metadata:binary,value:binary>)
outputColumnNames: _col0, _col1
ListSink

PREHOOK: query: SELECT id, data FROM variant_proj
PREHOOK: type: QUERY
PREHOOK: Input: default@variant_proj
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: SELECT id, data FROM variant_proj
POSTHOOK: type: QUERY
POSTHOOK: Input: default@variant_proj
POSTHOOK: Output: hdfs://### HDFS PATH ###
1 {"age":30,"name":"John"}
2 [1,2,3]
3 "plain string"
4 42
5 NULL
PREHOOK: query: SELECT * FROM variant_proj
PREHOOK: type: QUERY
PREHOOK: Input: default@variant_proj
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: SELECT * FROM variant_proj
POSTHOOK: type: QUERY
POSTHOOK: Input: default@variant_proj
POSTHOOK: Output: hdfs://### HDFS PATH ###
1 {"age":30,"name":"John"}
2 [1,2,3]
3 "plain string"
4 42
5 NULL
PREHOOK: query: SELECT id, data FROM variant_proj ORDER BY id DESC
PREHOOK: type: QUERY
PREHOOK: Input: default@variant_proj
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: SELECT id, data FROM variant_proj ORDER BY id DESC
POSTHOOK: type: QUERY
POSTHOOK: Input: default@variant_proj
POSTHOOK: Output: hdfs://### HDFS PATH ###
1 {"age":30,"name":"John"}
2 [1,2,3]
3 "plain string"
4 42
5 NULL
PREHOOK: query: drop table if exists variant_nested
PREHOOK: type: DROPTABLE
PREHOOK: Output: database:default
POSTHOOK: query: drop table if exists variant_nested
POSTHOOK: type: DROPTABLE
POSTHOOK: Output: database:default
PREHOOK: query: CREATE EXTERNAL TABLE variant_nested (
id INT,
s STRUCT<label:STRING, v:VARIANT>
) STORED BY ICEBERG
TBLPROPERTIES (
'format-version'='3'
)
PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
PREHOOK: Output: default@variant_nested
POSTHOOK: query: CREATE EXTERNAL TABLE variant_nested (
id INT,
s STRUCT<label:STRING, v:VARIANT>
) STORED BY ICEBERG
TBLPROPERTIES (
'format-version'='3'
)
POSTHOOK: type: CREATETABLE
POSTHOOK: Output: database:default
POSTHOOK: Output: default@variant_nested
PREHOOK: query: INSERT INTO variant_nested SELECT
1,
named_struct('label', 'x', 'v', parse_json('{"k": 1}'))
PREHOOK: type: QUERY
PREHOOK: Input: _dummy_database@_dummy_table
PREHOOK: Output: default@variant_nested
POSTHOOK: query: INSERT INTO variant_nested SELECT
1,
named_struct('label', 'x', 'v', parse_json('{"k": 1}'))
POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Output: default@variant_nested
PREHOOK: query: SELECT * FROM variant_nested
PREHOOK: type: QUERY
PREHOOK: Input: default@variant_nested
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: SELECT * FROM variant_nested
POSTHOOK: type: QUERY
POSTHOOK: Input: default@variant_nested
POSTHOOK: Output: hdfs://### HDFS PATH ###
1 {"label":"x","v":{"k":1}}
PREHOOK: query: SELECT id, to_json(s.v) FROM variant_nested
PREHOOK: type: QUERY
PREHOOK: Input: default@variant_nested
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: SELECT id, to_json(s.v) FROM variant_nested
POSTHOOK: type: QUERY
POSTHOOK: Input: default@variant_nested
POSTHOOK: Output: hdfs://### HDFS PATH ###
1 {"k":1}
PREHOOK: query: drop table if exists bin_struct
PREHOOK: type: DROPTABLE
PREHOOK: Output: database:default
POSTHOOK: query: drop table if exists bin_struct
POSTHOOK: type: DROPTABLE
POSTHOOK: Output: database:default
PREHOOK: query: CREATE EXTERNAL TABLE bin_struct (
id INT,
s STRUCT<metadata:BINARY, value:BINARY>
) STORED BY ICEBERG
PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
PREHOOK: Output: default@bin_struct
POSTHOOK: query: CREATE EXTERNAL TABLE bin_struct (
id INT,
s STRUCT<metadata:BINARY, value:BINARY>
) STORED BY ICEBERG
POSTHOOK: type: CREATETABLE
POSTHOOK: Output: database:default
POSTHOOK: Output: default@bin_struct
PREHOOK: query: INSERT INTO bin_struct SELECT 1, named_struct('metadata', cast('x' as binary), 'value', cast('y' as binary))
PREHOOK: type: QUERY
PREHOOK: Input: _dummy_database@_dummy_table
PREHOOK: Output: default@bin_struct
POSTHOOK: query: INSERT INTO bin_struct SELECT 1, named_struct('metadata', cast('x' as binary), 'value', cast('y' as binary))
POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Output: default@bin_struct
PREHOOK: query: SELECT * FROM bin_struct
PREHOOK: type: QUERY
PREHOOK: Input: default@bin_struct
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: SELECT * FROM bin_struct
POSTHOOK: type: QUERY
POSTHOOK: Input: default@bin_struct
POSTHOOK: Output: hdfs://### HDFS PATH ###
1 {"metadata":x,"value":y}
PREHOOK: query: drop table variant_proj
PREHOOK: type: DROPTABLE
PREHOOK: Input: default@variant_proj
PREHOOK: Output: database:default
PREHOOK: Output: default@variant_proj
POSTHOOK: query: drop table variant_proj
POSTHOOK: type: DROPTABLE
POSTHOOK: Input: default@variant_proj
POSTHOOK: Output: database:default
POSTHOOK: Output: default@variant_proj
PREHOOK: query: drop table variant_nested
PREHOOK: type: DROPTABLE
PREHOOK: Input: default@variant_nested
PREHOOK: Output: database:default
PREHOOK: Output: default@variant_nested
POSTHOOK: query: drop table variant_nested
POSTHOOK: type: DROPTABLE
POSTHOOK: Input: default@variant_nested
POSTHOOK: Output: database:default
POSTHOOK: Output: default@variant_nested
PREHOOK: query: drop table bin_struct
PREHOOK: type: DROPTABLE
PREHOOK: Input: default@bin_struct
PREHOOK: Output: database:default
PREHOOK: Output: default@bin_struct
POSTHOOK: query: drop table bin_struct
POSTHOOK: type: DROPTABLE
POSTHOOK: Input: default@bin_struct
POSTHOOK: Output: database:default
POSTHOOK: Output: default@bin_struct
4 changes: 4 additions & 0 deletions jdbc/src/java/org/apache/hive/jdbc/JdbcColumn.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,8 @@
return Type.NULL_TYPE;
case serdeConstants.UNKNOWN_TYPE_NAME:
return Type.UNKNOWN_TYPE;
case serdeConstants.VARIANT_TYPE_NAME:
return Type.VARIANT_TYPE;
default:
throw new SQLException("Unrecognized column type: " + type);
}
Expand Down Expand Up @@ -228,6 +230,8 @@
return serdeConstants.VOID_TYPE_NAME;
case serdeConstants.UNKNOWN_TYPE_NAME:
return serdeConstants.UNKNOWN_TYPE_NAME;
case serdeConstants.VARIANT_TYPE_NAME:

Check warning on line 233 in jdbc/src/java/org/apache/hive/jdbc/JdbcColumn.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

'case' child has incorrect indentation level 6, expected level should be 4. 


See more on https://sonarcloud.io/project/issues?id=apache_hive&issues=AZ-lF6QwFp8FN2qbFIw_&open=AZ-lF6QwFp8FN2qbFIw_&pullRequest=6644
return serdeConstants.VARIANT_TYPE_NAME;

Check warning on line 234 in jdbc/src/java/org/apache/hive/jdbc/JdbcColumn.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

'block' child has incorrect indentation level 8, expected level should be 6. 


See more on https://sonarcloud.io/project/issues?id=apache_hive&issues=AZ-lF6QwFp8FN2qbFIxA&open=AZ-lF6QwFp8FN2qbFIxA&pullRequest=6644
case "map":
return serdeConstants.MAP_TYPE_NAME;
case "array":
Expand Down
2 changes: 1 addition & 1 deletion serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ protected void serializeField(ByteStream.Output out, Object obj, ObjectInspector
if (!objInspector.getCategory().equals(Category.PRIMITIVE) || (objInspector.getTypeName().equalsIgnoreCase(serdeConstants.BINARY_TYPE_NAME))) {
//do this for all complex types and binary
try {
serialize(out, SerDeUtils.getJSONString(obj, objInspector, serdeParams.getNullSequence().toString()),
serialize(out, SerDeUtils.getJSONString(obj, objInspector, serdeParams.getNullSequence().toString(), true),
PrimitiveObjectInspectorFactory.javaStringObjectInspector, serdeParams.getSeparators(),
1, serdeParams.getNullSequence(), serdeParams.isEscaped(), serdeParams.getEscapeChar(),
serdeParams.getNeedsEscape());
Expand Down
Loading
Loading