Repository navigation
Conversation
Record the standard calendar field for one-shot and chunked writers, expose it through ORC metadata, and add matching C++ and Java/JNI options. Reject unsupported legacy output rather than mislabeling proleptic data. Encode DATE statistics using DateStatistics instead of IntegerStatistics. Cover historical and nested dates, nulls, empty input, calendar defaults, and Apache ORC interoperability through both JNI sink paths. Contributes to NVIDIA#23794. Signed-off-by: Allen Xu <allxu@nvidia.com>
Rebase DATE leaves before encoding and statistics, and add opt-in conversion when reading explicitly marked hybrid files. Keep raw reader behavior for existing callers, and reject lossy dates and legacy TIMESTAMP schemas. Cover nested slices, hidden nulls, mixed calendars, retries, wide dates, and Apache ORC interoperability. Signed-off-by: Allen Xu <allxu@nvidia.com>
Preserve unknown protobuf calendar values and make shared decode errors sticky across columns. Bound legacy DATE scratch memory to written slices, skip empty nested ranges, and retain compacted column ownership through statistics persistence. Compact top-level DATE slices before device-view construction to avoid duplicate setup in chunked writes. Cover raw enum values, decoder overflow state, bounded slice allocations, nested DATE statistics, and indexed reading. Clarify calendar constants and parameter documentation. Contributes to NVIDIA#23794. Signed-off-by: Allen Xu <allxu@nvidia.com>
Signed-off-by: Allen Xu <allxu@nvidia.com>
Signed-off-by: Allen Xu <allxu@nvidia.com>
Signed-off-by: Allen Xu <allxu@nvidia.com>
Signed-off-by: Allen Xu <allxu@nvidia.com>
Signed-off-by: Allen Xu <allxu@nvidia.com>
3 tasks done
wjxiz1992
added
cuIO
ChatGPT Codex Connector
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Depends on #24733. Closes #23794. Java-only review diff (7 files).
Expose the ORC calendar controls from #24733 through Java and JNI. Writers can select proleptic Gregorian or legacy DATE encoding, and readers can opt into converting legacy DATE values to proleptic dates. Reader conversion remains disabled by default for callers that already perform their own rebasing; writer defaults match the C++ parent.
Java API and nested DATE roundtrip tests remain in the standard test source root consumed by downstream JNI builds. Apache ORC interoperability tests run automatically from a separate source root in cuDF's Java build, keeping their additional test dependencies out of the independent JNI build. Both the C++ intermediate and the combined version passed cudf-spark ORC validation on Spark 3.5.4 and 4.0.0; detailed results are linked from #24733.
Prepared with OpenAI Codex assistance.
Draft status
This Draft targets
mainand depends on #24733. Until the parent merges, GitHub's full PR diff also includes the parent C++ changes; use the Java-only comparison above for this review. Before marking it ready:maininto this branch as needed and verify the Java-only PR diff.main.Checklist