[AURON #2434] Support full-data-file Iceberg changelog deletes

[weimingdiit]

Which issue does this PR close?
Closes #2434

Rationale for this change
Auron native Iceberg changelog scan currently supports insert-only changelog tasks.

Full-data-file delete changelog tasks can be executed natively without row-level delete-file handling. Auron can scan the deleted data file directly and materialize changelog metadata from the Iceberg changelog task.

Supporting this case improves native Iceberg changelog scan coverage while keeping more complex delete and update semantics on Spark's reader.

What changes are included in this PR?
Allows native Iceberg changelog scan to accept DeletedDataFileScanTask when:

• the changelog operation is DELETE
• existingDeletes() is empty

Keeps existing native support for AddedRowsScanTask when:

• the changelog operation is INSERT
• deletes() is empty

Keeps fallback behavior for unsupported changelog tasks, including row-level deletes, position/equality deletes, update changelog tasks, mixed file formats, and delete tasks with existing delete files.

Adds coverage for:

• full-data-file delete changelog native scan
• a changelog range that contains both insert tasks and full-data-file delete tasks

Are there any user-facing changes?
No user-facing API changes. More Iceberg changelog scans can now be executed natively by Auron.

How was this patch tested?
UT.

[merrily01]

@weimingdiit Thanks for the contribution. The implementation looks correct to me, just a few comments inline.

[Copilot]

🟡 Not ready to approve

The native-acceptance guard for DeletedDataFileScanTask does not enforce the PR’s stated operation == DELETE constraint, risking incorrect native execution for non-DELETE changelog tasks.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR expands Auron’s native Iceberg changelog scan support to include full-data-file delete changelog tasks (while keeping row-level delete semantics on Spark’s reader), improving native coverage for simple DELETE changelog ranges.

Changes:

• Extend native changelog planning to accept DeletedDataFileScanTask when it can be handled by scanning the data file directly.
• Refactor changelog task handling to a unified “native changelog data-file task” representation used for native scan task construction.
• Add integration tests covering full-data-file delete changelog scans and mixed insert + full-data-file delete ranges.

File summaries

File	Description
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/auron/iceberg/IcebergScanSupport.scala	Adds native handling for full-data-file delete changelog tasks by mapping supported changelog tasks into native scan tasks.
thirdparty/auron-iceberg/src/test/scala/org/apache/auron/iceberg/AuronIcebergIntegrationSuite.scala	Adds integration coverage for native execution of delete-only and mixed insert+delete changelog ranges.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 2
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[weiqingy]

No change requested, just evidence on this one.

The changelog task type that would actually be unsafe to run natively is DeletedRowsScanTask, the merge-on-read sibling that carries addedDeletes(). In Iceberg 1.10.1 it declares (javadoc stripped):

public interface DeletedRowsScanTask extends ChangelogScanTask, ContentScanTask<DataFile> {
  List<DeleteFile> addedDeletes();

  List<DeleteFile> existingDeletes();

  @Override
  default ChangelogOperation operation() {
    return ChangelogOperation.DELETE;
  }
}

It extends ChangelogScanTask, not DeletedDataFileScanTask, so the type match on this line is what keeps it off the native path. Its operation() is hard-coded DELETE as well, so an operation() check would not have excluded it either. Only the type match does.

For the _change_type angle: the metadata case at IcebergScanSupport.scala:620 reads the value straight off the task (requiredChangelogTask(name).operation().name()) rather than off the Scala type, so a subtype that overrode operation() would stamp its own value instead of a mismatched one.

[weimingdiit]

Thanks, that matches my reading.

The native path is intentionally limited by the concrete task type here. DeletedRowsScanTask is a separate ChangelogScanTask implementation, not a DeletedDataFileScanTask, so it will not match this case and will continue to fall back. Keeping or removing the operation() == DELETE check does not affect that, since DeletedRowsScanTask also hard-codes operation() to DELETE.

For _change_type, I agree the value should come from the task operation itself. The metadata materialization path already uses requiredChangelogTask(name).operation().name(), so if Iceberg ever provides a supported changelog task with a different operation value, the metadata column will reflect the task value rather than the Scala match type.

No additional code change from my side.

[slfan1989]

Thanks for the contribution! The task-type guard correctly limits native execution to AddedRowsScanTask and DeletedDataFileScanTask, while the empty-delete checks preserve the fallback boundary for row-level delete semantics. The changelog metadata and mixed-format fallback behavior are also retained. The changes look good to me.