Refact reader branch#64904
Draft
Gabriel39 wants to merge 57 commits into
Draft
Conversation
### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Refactor the file table reader stack around the format_v2 reader implementation. This includes the new file reader abstractions, parquet reader components, table reader adapters for Hive/Iceberg/Paimon/JDBC, ColumnMapper filter and projection handling, expression clone support used by file-local filter rewrites, Iceberg row lineage materialization, and related BE unit tests and design notes. This commit is the squashed result of rebasing the current branch onto master. ### Release note None ### Check List (For Author) - Test: No need to test (history rewrite only in this step; no code changes beyond the already rebased branch content) - Behavior changed: No - Does this need documentation: No
) ### What problem does this PR solve? This PR refactors the new parquet reader complex-column path around schema projection and reader creation. It clarifies `ParquetColumnReaderFactory` recursion, keeps schema projection as a single public helper, normalizes file-local `ColumnDefinition` children to Doris semantic children, and folds the Parquet MAP `key_value/entry` wrapper into the MAP schema node during parquet schema construction. With that shape, MAP and LIST both expose direct semantic children to the table/file reader boundary, and ParquetReader no longer needs a semantic-to-physical projection translation layer for MAP. ### Release note None ### Check List - Test: Manual test - Ran `git diff --check`. - Tried `./run-be-ut.sh -j 8 --run --filter="ParquetColumnReaderTest.ReadProjectedMapStructValueChildren:ColumnMapperScanRequestTest.MapValueStructProjectionPrunesValueChildren"`, but local toolchain failed before tests with `ld: library 'c++' not found`. - Behavior changed: No - Does this need documentation: No
…ns (apache#64480) ### What problem does this PR solve? Localize slot-rooted struct element predicates through ColumnMapping so renamed nested fields rewrite both selector names and projected file child return types. Keep computed complex parents and evolved MAP_KEYS-style filters at the table layer instead of generating unsafe file-local casts. Rebuild complex scan projections and rematerialize struct children in table type order, add debug-only block sanity checks with contextual errors, preserve function-call clone state, and handle nullable Iceberg delete columns. Move ColumnMapper coverage into column_mapper_test and add tests for complex child projection, nested predicate projection, map/array fallback, and file-local conjunct localization boundaries. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
### What problem does this PR solve? Issue Number: None Related PR: apache#63893 Problem Summary: Refresh regression expected outputs for the new Parquet INT96 timestamp semantics. The new reader decodes INT96 timestamps without applying the session timezone offset, so affected Asia/Shanghai-based expectations move 8 hours earlier. This commit updates only the expected result files for the INT96 offset cases in p0 and external suites, while leaving unrelated timestamp failures unchanged. ### Release note None ### Check List (For Author) - Test: Manual test - Verified staged diffs are limited to regression expected-output files. - Verified 634 changed lines are exactly timestamp minus 8 hours, with paimon c2 handled as a selective LTZ/INT96 column update. - Full regression test not run. - Behavior changed: No, test expectations only - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#63893 Problem Summary: The new Parquet reader rejected MAP schemas whose key field is optional with `Unsupported nullable parquet MAP key`, while the old reader only logged a warning and continued. Some external Parquet writers can emit optional MAP keys, and the v2 schema builder already preserves definition levels and exposes MAP key/value types as nullable. This change removes the schema-level hard rejection for optional MAP keys while keeping the existing structural MAP layout checks. ### Release note Allow the new Parquet reader to read external Parquet MAP columns with optional key fields. ### Check List (For Author) - Test: Manual test - `build-support/clang-format.sh be/src/format_v2/parquet/parquet_column_schema.cpp` - `git diff --check` - Behavior changed: Yes, the new Parquet reader no longer rejects optional MAP key schemas. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#63893 Problem Summary: RuntimeFilterExpr is a wrapper around the concrete runtime filter predicate, but its normal column execution path returned `Not implement RuntimeFilterExpr::execute_column_impl`. Partition pruning for external tables can evaluate runtime filter wrappers as ordinary expressions, so Hive/Iceberg/Paimon runtime filter partition pruning failed before evaluating the wrapped predicate. This change delegates RuntimeFilterExpr::execute_column_impl to the wrapped implementation. The scan filter path still uses execute_filter, preserving the existing selectivity counters and runtime-filter-specific row filtering behavior. ### Release note Fix runtime filter wrapper expression execution for external table partition pruning. ### Check List (For Author) - Test: Manual test - `build-support/clang-format.sh be/src/exprs/runtime_filter_expr.cpp` - `git diff --check` - Behavior changed: Yes, runtime filter wrappers can now be evaluated through the normal expression execution path. - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#63893 Problem Summary: The new parquet reader resolved decimal columns with precision greater than 38 to TYPE_DECIMAL256, but then marked those columns as unsupported for the scalar record reader. This made external parquet scans fail with errors such as "Current parquet scalar reader does not support column amount" even though the old parquet reader and the decoded decimal serde already support Decimal256 for the common parquet decimal physical carriers. Remove the extra precision-based block so Decimal256 columns can use the existing record reader and decoded serde path, while unsupported physical types remain rejected. ### Release note Support Decimal256 parquet columns in the new parquet reader. ### Check List (For Author) - Test: Manual test - Ran build-support/clang-format.sh be/src/format_v2/parquet/parquet_type.cpp - Ran git diff --check - Behavior changed: Yes (new parquet reader now accepts parquet decimal columns with precision greater than 38 when they fit Doris Decimal256) - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#63893 Problem Summary: The new parquet reader created the FilteredRowsByLazyRead profile counter but did not pass it to the scan scheduler or update it after predicate filtering. As a result lazy materialization profile tests saw zero filtered lazy-read rows even when non-predicate columns were read lazily. Pass the counter into ParquetScanScheduler and update it by the number of rows filtered by conjuncts when non-predicate columns are present. ### Release note None ### Check List (For Author) - Test: Manual test - Ran build-support/clang-format.sh be/src/format_v2/parquet/parquet_scan.h be/src/format_v2/parquet/parquet_scan.cpp be/src/format_v2/parquet/parquet_reader.cpp - Ran git diff --check - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#63893 Problem Summary: Clean up format_v2 code style after the new parquet reader changes. This change fixes local tidy-style issues in the parquet selection vector by using std::cmp_greater/std::cmp_greater_equal for mixed signed and unsigned comparisons and designated initializers for RowRange construction. It also removes unused include directives from format_v2 implementation files. The format_v2 directory was checked with the repository clang-format script, and the include cleanup was validated by compiling the Fedora DEBUG Format target. ### Release note None ### Check List (For Author) - Test: Manual test - Ran build-support/clang-format.sh be/src/format_v2 - Ran git diff --check - Ran targeted clang-tidy on Fedora with checks modernize-use-integer-sign-comparison and modernize-use-designated-initializers for be/src/format_v2/parquet/selection_vector.h using the DEBUG compile database without PCH - Ran clang-include-cleaner on selected format_v2 implementation files to collect remove candidates - Ran /home/socrates/ldb_toolchain/bin/ninja Format in /home/socrates/code/doris/be/build_DEBUG on Fedora - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#63893 Problem Summary: TeamCity external regression build 970191 still had several expected output files using old timestamp values. The new parquet timestamp semantics return the corrected values for the affected external table cases, including Hive, Iceberg, Paimon, and TVF parquet result files. This commit refreshes the corresponding regression expected outputs from the observed CI results and keeps unrelated non-timestamp failures untouched. ### Release note None ### Check List (For Author) - Test: Manual test - Compared TeamCity build 970191 failure details with the updated expected output files. Full regression test was not rerun locally. - Behavior changed: No - Does this need documentation: No
struct type should be use name mode with nested type
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: New parquet profile definitions and wiring were split across ParquetReader, ParquetScan, and column reader headers. This made ParquetReader own counter initialization, pruning counter updates, and scheduler sub-profile assembly directly even though parquet_profile.h already existed for profile-related types. This change centralizes the new parquet RuntimeProfile counter ownership in parquet_profile.h/.cpp and keeps ParquetReader responsible only for invoking the profile helper methods.
### Release note
None
### Check List (For Author)
- Test: Manual test
- Ran build-support/clang-format.sh for touched files.
- Ran git diff --check.
- Tried ./run-be-ut.sh --run '--filter=NewParquetReaderTest.*', but local CMake compiler detection failed before building Doris because /opt/homebrew/opt/llvm@16/bin/clang++ could not link a simple program: ld: library 'c++' not found.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Align format_v2 implementation namespaces with the format_v2 ownership boundary. Parquet, Hive, Paimon, Iceberg, and JDBC implementations now live under doris::format subnamespaces, while shared format_v2 expression helpers live under doris::format. Call sites and tests were updated to use the new namespace layout.
### Release note
None
### Check List (For Author)
- Test: Manual test
- Ran build-support/clang-format.sh on the modified BE files
- Ran git diff --check
- Ran namespace residue scans for old doris::parquet/hive/paimon/jdbc/iceberg namespaces and duplicate format::format references
- Attempted targeted BE UT with ./run-be-ut.sh --run '--filter=NewParquetReaderTest.*:ParquetColumnReaderTest.*:TableReaderTest.*:CastTest.*:DeletePredicateTest.*:EqualityDeletePredicateTest.*', but local CMake compiler detection failed before Doris code compiled because /opt/homebrew/opt/llvm@16/bin/clang++ could not link libc++: ld: library 'c++' not found
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#63893 Problem Summary: The new parquet reader reports timestamp values with the updated INT96 timestamp interpretation for existing external parquet coverage. This commit updates the affected regression expected outputs from the latest TeamCity P0 and external regression real outputs. Doris parquet export/write cases with suspicious timestamp offsets are intentionally excluded because those require separate writer-side analysis. ### Release note None ### Check List (For Author) - Test: Manual test - Validated modified expected rows against TeamCity builds 970619 and 970620 failure logs, and ran `git diff --check`. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#63893 Problem Summary: The new parquet reader did not map TIMESTAMP(NANOS) logical columns to a supported Doris timestamp type, and DATETIMEV2 decoded INT64 timestamp values only handled millis and micros. As a result Hive parquet timestamp nanos data was materialized as NULL instead of the expected timestamp values. This change maps parquet timestamp nanos to DATETIMEV2(6), decodes nanos by truncating to microseconds, and adds decoded-value coverage for DATETIMEV2 nanos. It also refreshes the external TVF group4 expected output for a parquet file containing BC timestamp values that Doris cannot represent, where the new reader correctly returns NULL for those rows. ### Release note None ### Check List (For Author) - Test: Manual test - Ran `git diff --check`. - Verified the relevant parquet files with DuckDB to confirm timestamp nanos and BC timestamp source values. - Attempted `./run-be-ut.sh --run '--filter=DataTypeSerDeDecodedValuesTest.*'`, but local CMake failed before compiling tests because the macOS toolchain cannot link a simple C++ program: `ld: library 'c++' not found`. - Behavior changed: Yes. The new parquet reader now reads TIMESTAMP(NANOS) values as DATETIMEV2(6) instead of producing NULL through unsupported conversion. - Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Refine the new Parquet reader row group pruning flow so scan range filtering is applied before more expensive statistics, dictionary, bloom filter, and page index pruning. Also document the Parquet reader, scan scheduler, statistics pruning, and nested column reader APIs, and update affected namespace references in BE tests.
### Release note
None
### Check List (For Author)
- Test: Manual test
- Ran build-support/clang-format.sh on touched BE C++ files and git diff --check locally.
- Started BE UT validation on Fedora with NewParquetReaderTest.* and ParquetBloomFilterPruningTest.*; fixed compile issues found during validation. Full rerun was interrupted before completion by follow-up history cleanup request.
- Behavior changed: No
- Does this need documentation: No
Rewrite comments for the entry-point and foundational modules: parquet_reader.h: - Class-level doc: role boundary, lifecycle (init→get_schema→open→get_block→close) - TableReader calling relationship explained - Each method and field annotated parquet_type.h: - ParquetExtraTypeInfo: each variant documented - ParquetTypeDescriptor: full field-by-field descriptions - Three-level resolution priority (logical→converted→physical) explained - resolve_parquet_type / supports_record_reader / decoded_value_kind docs parquet_column_schema.h: - Class-level doc: design decisions (wrapper folding, nullable, Dremel levels) - All fields grouped into sections (identifier / type / levels / children) - Each field annotated with its role and valid domain (PRIMITIVE vs complex) parquet_column_schema.cpp: - SchemaBuildContext fields annotated parquet_file_context.cpp: - DorisRandomAccessFile adapter class documented parquet_profile.h: - All Profile structs with section-based Chinese comments - Counter groups organized (RG pruning / page skip / batch read / column reader / file ops / decompress & cache / decode / others) Co-Authored-By: Claude <noreply@anthropic.com>
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The new Parquet scanner does not implement condition cache yet, so the parquet condition cache regression case can fail when the session uses FileScannerV2. Force this case to use the old file scanner path before enabling condition cache and profile checks.
### Release note
None
### Check List (For Author)
- Test: No need to test
- Regression-only session variable adjustment for an existing case.
- Behavior changed: No
- Does this need documentation: No
Problem Summary: The TimeStampTz protobuf unit test constructed the type with auto, so the static type was std::shared_ptr<DataTypeTimeStampTz>. Calling to_protobuf(PTypeDesc*) through that concrete type failed because the three-argument override hides the base-class one-argument wrapper. Use DataTypePtr so the test exercises the IDataType public interface that owns the one-argument wrapper.
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Bare repeated parquet fields can encode an empty repeated parent with a low definition-level shape slot. The nested LIST and MAP builders filtered out every slot whose definition level was below the repeated ancestor level before accounting for parent rows, so an empty repeated primitive field could report zero output rows for a row group that should produce one row. Preserve low-definition slots that start a parent row while still skipping unrelated nested slots, and keep MAP scalar value alignment consistent with the key stream.
### Release note
None
### Check List (For Author)
- Test: Regression test / Unit Test
- Regression test: on Fedora, ran external_table_p0/tvf/test_hdfs_parquet_group2 after rebuilding DEBUG BE; the original failing test_10 proto-struct-with-array.parquet returned the expected row with repeatedprimitive=[]. The full suite then failed later at unrelated timestamp-nanos expected output.
- Unit Test: added ParquetColumnReaderControlTest cases for empty bare repeated primitive LIST rows and empty MAP rows. Background Fedora UT compilation was started but not confirmed complete because SSH to Fedora timed out.
- Behavior changed: Yes. Empty repeated parquet LIST/MAP parent rows are preserved instead of being dropped.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The new Parquet reader passed small-width logical integer annotations such as INT(8, false) and INT(16, false) to numeric SerDe as plain INT32 values. Physical values that should be interpreted with Parquet logical bit-width cast semantics could then be preserved or nulled by Doris target-type range checks instead of matching Arrow and the old reader behavior. This change carries logical integer bit width and signedness through DecodedColumnView, applies the logical cast before Doris materialization, and uses the same metadata when converting Parquet statistics min/max fields.
### Release note
Fix Parquet logical integer decoding for small-width signed and unsigned annotations in the new reader.
### Check List (For Author)
- Test: Unit Test / Manual test
- Added decoded-value unit tests for signed and unsigned Parquet logical integer casts, including the field path used by statistics.
- Ran git diff --check.
- Attempted ./run-be-ut.sh --run --filter='DataTypeSerDeDecodedValuesTest.ReadUnsignedLogicalIntegersCastsPhysicalValues:DataTypeSerDeDecodedValuesTest.ReadSignedLogicalIntegersCastsPhysicalValues:DataTypeSerDeDecodedValuesTest.ReadFieldLogicalIntegerCastsPhysicalValue', but local macOS toolchain failed during CMake compiler detection with ld: library 'c++' not found before building Doris tests.
- Behavior changed: Yes. New Parquet reader now matches Parquet logical integer bit-width cast semantics for annotated integers.
- Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#63781, apache#64671 Problem Summary: File scanner v2 did not carry the same fixes as the existing file scanner path. Predicate rows filtered inside v2 file readers were still reported through scanner load counters unless the scanner was a real load source, and Hive TEXTFILE empty physical lines were still skipped unless read_csv_empty_line_as_null was enabled. This change gates v2 load counter reporting with the same FILE_STREAM exception used by FileScanner and adds a delimited text hook so Hive Text v2 treats empty physical lines as records while CSV keeps the old default behavior. ### Release note Fix file scanner v2 load counter reporting and Hive TEXTFILE empty-line handling. ### Check List (For Author) - Test: Unit Test / Manual test - Added TextV2ReaderTest coverage for Hive TEXTFILE empty line records, single-column empty string fields, and COUNT pushdown. - Ran git diff --check. - Ran clang-format v16 through build-support/run_clang_format.py for changed files. - Attempted ./run-be-ut.sh --run --filter='TextV2ReaderTest.*:FileScannerV2Test.*', but the local run was blocked because the script needed to update/download datasketches-cpp and network access was unavailable; no BE UT binary was already built. - Attempted clang-tidy with the available compile_commands.json, but it pointed at a stale /mnt/disk3/gabriel path; the project clang-tidy wrapper also requires bash 4+ while only system bash is available. - Behavior changed: Yes. File scanner v2 now matches v1 load counter gating and Hive TEXTFILE empty-line semantics. - Does this need documentation: No ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: File scanner v2 reads Parquet through Arrow, so the old
vparquet page cache path is not used. Repeated scans still go through
the Doris file reader for serialized Parquet column chunk data even when
the Parquet page cache option is enabled. This change registers the
selected Parquet column chunk byte ranges after row-group planning and
lets the Arrow RandomAccessFile adapter reuse StoragePageCache for reads
inside those ranges. Footer and metadata reads happen before range
registration and are intentionally excluded.
### Release note
None
### Check List (For Author)
- Test: Manual test
- Ran git diff --check.
- Ran build-support/run_clang_format.py with clang-format 16 on modified
BE files.
- Could not compile with existing be/cmake-build-debug-dev-perf because
CMakeCache.txt was generated for /mnt/disk3/gabriel/Workspace/dev1/doris
and the configured ninja path is not available in this worktree.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: FileScannerV2 did not support Doris Native files. This
change adds a native v2 FileReader implementation instead of wrapping
the legacy NativeReader. The reader validates the Native header, reads
serialized PBlock payloads, caches and replays the first block for
schema probing, exposes nullable file-local schema, projects requested
columns, and applies file-local filters. Shared materialized-column
filtering is also used by JSON and delimited text readers so predicate
accounting stays consistent. WAL is intentionally not implemented on the
v2 path because current group commit WAL scans are load scans and
FileScanOperator only selects FileScannerV2 when src_tuple_id does not
resolve to an input tuple.
### Release note
None
### Check List (For Author)
- Test:
- Style check: build-support/check-format.sh
- Unit Test: not run locally because sandbox execution cannot write
.git/modules for submodule setup and cannot download datasketches-cpp;
the attempted run-be-ut command failed before compiling tests.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: FileScannerV2 could not read Remote Doris Arrow Flight splits because FORMAT_ARROW was not routed to a v2 table reader and no v2-native Remote Doris reader existed. This change adds a Remote Doris TableReader/FileReader implementation for FileScannerV2 that opens Arrow Flight streams directly, builds the file-local schema from planned file slots, materializes Arrow RecordBatch data by column name into the v2 file-local block, applies localized filters through the v2 materialized-reader helper, validates protocol mismatches, and closes Flight resources. FORMAT_ARROW is enabled in FileScannerV2 only for table_format_type=remote_doris so ordinary Arrow stream files stay on the existing path.
### Release note
Support Remote Doris scans in FileScannerV2 when FileScannerV2 is enabled.
### Check List (For Author)
- Test: Manual test
- BE unit test: attempted PARALLEL=1 ./run-be-ut.sh --run --filter='FileScannerV2Test.*:RemoteDorisV2ReaderTest.*', but the sandbox could not update .git/modules/contrib/datasketches-cpp and network fallback to github.com was unavailable; escalated retries timed out in approval review.
- Manual test: python3 build-support/run_clang_format.py --clang-format-executable /usr/local/opt/llvm@16/bin/clang-format --style file --inplace false --extensions c,h,C,H,cpp,hpp,cc,hh,c++,h++,cxx,hxx --exclude none <modified files>
- Behavior changed: Yes. Remote Doris FORMAT_ARROW scan ranges can be routed to FileScannerV2.
- Does this need documentation: No
Gabriel39
requested review from
BiteTheDDDDt,
airborne12,
csun5285,
eldenmoon,
gavinchou,
liaoxin01,
morningman,
yiguolei and
zclllyybb
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 29346 ms |
Contributor
TPC-DS: Total hot run time: 171893 ms |
Contributor
ClickBench: Total hot run time: 25.98 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)