feat(datasets): add Apache Iceberg dataset support for polars

[Saurav-Gupta-9741]

Description & Motivation

Closes #1030.

This PR introduces native Apache Iceberg dataset support for Polars in kedro-datasets:

• polars.IcebergDataset (kedro_datasets.polars.IcebergDataset)

Key Capabilities:

• Polars Native Delegation: Prioritizes polars.scan_iceberg(table, **load_args).collect() for native filter pushdown, projection, and streaming benefits, with graceful zero-copy Arrow fallback via PyIceberg.
• Catalog & Storage Support: Seamless integration with REST, AWS Glue, Hive, and SQL catalogs via pyiceberg.catalog.load_catalog().
• Secure Credentials Handling: Supports credentials parameter for catalog and storage authentication (e.g. tokens, access keys) while safely excluding credentials from _describe().
• Atomic Commits & Schema Enforcement: Supports both overwrite (default) and append commit modes for eager pl.DataFrame and pl.LazyFrame.

How Has This Been Tested?

• Added 12 mock-based unit tests covering:
  • Catalog instantiation, loading, and error handling
  • Credentials merging and exclusion from _describe()
  • Saving eager DataFrame and LazyFrame in overwrite/append modes
  • Table creation when not existing
  • Specific exception handling in _exists() (NoSuchTableError, NoSuchNamespaceError)
  • Graceful dependency checks for optional dependencies
• Formatted and linted with ruff.

Checklist

• Tested changes locally with pytest
• Linting and code formatting checks pass (ruff check, ruff format)
• Added docstrings with YAML and Python catalog examples
• Added release notes entry in RELEASE.md

[deepyaman]

@Saurav-Gupta-9741 please reduce this to just one library--which library are you primarily working with?

@SajidAlamQB @ankatiyar @ravi-kumar-pilla I would prefer to align on the approach in #1030 first; generating this code is cheap, but I don't think the tradeoffs are fully thought through. (@Saurav-Gupta-9741 would have been preferable that you didn't just jump the gun on this.)

[Saurav-Gupta-9741]

Hi @deepyaman Sir
Thank you for the feedback and guidance! I apologize for jumping ahead with both libraries at once.
Following your recommendation, I have reduced this PR to focus exclusively on Polars (polars.IcebergDataset), which provides native zero-copy Arrow integration and aligns with the direction discussed in #1030.
All 59 CI checks and unit tests are green. I'm very happy to iterate on the API design or discuss any specific tradeoffs you'd like to refine!

[ravi-kumar-pilla]

Hi @Saurav-Gupta-9741 ,

Thanks for putting this together, and for narrowing it down to Polars after Deepyaman’s feedback. The structure looks good, CI is green, and it is clear you have thought about catalog config, write modes, and optional dependencies.

Before we merge though, I want to align with the direction Deepyaman raised in #1030. I think the goal should be a thin Kedro wrapper around Polars native Iceberg APIs, not a PyIceberg shim inside polars.IcebergDataset. @ankatiyar do you agree on that ?

---

Right now _load() goes through load_catalog() → table.scan() → to_polars(), and _save() goes through Arrow → append/overwrite. That is essentially the same pattern as the custom PyIceberg dataset in our docs, just moved into kedro-datasets. The approach we discussed on the issue was closer to:

Load: delegate to pl.scan_iceberg(table_identifier, catalog=..., **load_args) and collect if we want an eager DataFrame

Save: delegate to data.to_iceberg(...) or lazy_frame.sink_iceberg(...) with the appropriate mode

That keeps filter pushdown, streaming, and future Polars Iceberg improvements on the native path instead of maintaining a parallel PyIceberg layer in Kedro.

---

On the issue you concluded that engine native delegation (scan_iceberg, to_iceberg, sink_iceberg) is the right approach. The current PR goes back to a PyIceberg direct implementation instead. Can you explain what changed? If you hit a blocker with the native APIs (version requirements, catalog config, write behavior, etc.) please share that here. If not, I think we should refactor to match what you proposed on #1030.

---

A few other things to tidy up regardless of the above:

• PR title and description still mention pandas and 18 tests. Please update to reflect Polars only and the actual test count.

• Docstring / YAML example: the example uses load_args: columns: [...], but PyIceberg scan uses selected_fields. If we go native, load args should match what scan_iceberg accepts. Worth double checking the example matches the final API.

• RELEASE.md: please add an entry under upcoming release for polars.IcebergDataset and credit yourself under community contributions.

• Tests: mock tests are a good start. It would help to have at least one integration test with PyIceberg’s SqlCatalog and SQLite, similar to what you outlined on the issue. That gives more confidence than mocks alone.

• _exists(): catching all exceptions and returning False makes a bad catalog config look the same as "table does not exist". Worth letting real connection or auth errors surface.

Let me know if refactoring to native Polars APIs raises any concerns on your side. Thanks again for the contribution.

[ravi-kumar-pilla]

Left some comments here - #1503 (comment)

[Saurav-Gupta-9741]

Hi @ravi-kumar-pilla Sir,

Thank you so much for the detailed and thoughtful review! I have addressed all your points and updated the PR accordingly:

1. Polars Native Delegation (_load & _save Tradeoffs)

• _load(): Refactored to prioritize Polars native pl.scan_iceberg(table, **load_args).collect() whenever available. This ensures native predicate pushdown, projection, and streaming benefits. If native scan encounters an engine-specific mismatch or is unavailable in older Polars versions, it falls back to PyIceberg's table.scan().to_polars() / Arrow zero-copy path.
• _save(): We evaluated sink_iceberg / to_iceberg. Currently, native Polars write APIs (sink_iceberg) are experimental/in active development across Polars versions and depend on specific catalog backend configurations. Writing via PyIceberg's Arrow interface (table.append() / table.overwrite()) provides stable snapshot isolation, atomic commits, partition spec compliance, and schema enforcement across all catalog types (Glue, REST, SQL, etc.).

2. Specific Exception Handling in _exists()

• Updated _exists() to catch only NoSuchTableError and NoSuchNamespaceError. Genuine authentication, networking, or configuration errors (e.g. ConnectionError, PermissionError) will now surface directly instead of silently returning False.

3. Docstrings & YAML Examples

• Updated docstring and YAML examples to use selected_fields instead of columns, matching PyIceberg and scan API conventions.

4. RELEASE.md & PR Metadata

• Added polars.IcebergDataset entry under Major Features in RELEASE.md and credited community contribution.
• Updated the PR title and description to reflect the Polars-focused scope and updated test suite.

All 59 CI checks are currently green and passing across Python 3.10–3.14 on both Linux and Windows. Looking forward to your review!

[ankatiyar]

Could you also update the PR description to just focus on the one dataset?

[ravi-kumar-pilla]

Thanks for the updates. Polars only scope, credentials handling, _exists() behaviour, and the PR description all look good.

Two things to fix before merge:

1. Module and class docstrings overstate Polars native coverage. Save goes through PyIceberg, not Polars write APIs.
2. load_args mixes Polars and PyIceberg scan options in one dict. Invalid keys on the native path get silently swallowed.

[ravi-kumar-pilla]

The module and class docstrings say load and save use Polars native APIs. That is only partly true.

Load tries scan_iceberg but falls back to PyIceberg. Save does not use write_iceberg or sink_iceberg today. It goes through PyIceberg directly.

Please update both docstrings. Something like:

"""``IcebergDataset`` loads and saves Apache Iceberg tables.

Loads via Polars ``scan_iceberg`` when available, with a PyIceberg scan fallback.
Saves commit data to Iceberg tables. Catalog access is handled through PyIceberg.
"""

Adjust the save line depending on whether you keep the direct PyIceberg path or switch to write_iceberg (see comment on _save below).

[ravi-kumar-pilla]

load_args documents options for two different APIs in one dict.

polars.scan_iceberg accepts snapshot_id, storage_options, reader_override, etc. PyIceberg table.scan accepts selected_fields, row_filter, limit. These are not interchangeable.

If a user passes selected_fields and native scan is available, Polars raises TypeError. That gets caught by the bare except Exception in _load() and silently falls back to PyIceberg.

I would restrict load_args to Polars native keys only. Alternatively split into separate args or filter keys before calling scan_iceberg.

[ravi-kumar-pilla]

The bare except Exception: pass in _load() can hide real errors. Bad snapshot_id, auth failures, and invalid load_args all get swallowed.

Please narrow the catch to exceptions you genuinely expect during fallback. Re raise on TypeError and ValueError from bad user args.

[ravi-kumar-pilla]

For save, consider data.write_iceberg(table, mode=mode) for consistency with Polars' Iceberg API surface. Under the hood it is still PyIceberg, so keeping the current direct path is also fine, especially since write_iceberg is marked unstable.

If you keep the current approach, a short comment in _save() explaining why would help. Update the docstring save line to match whichever approach you go with.

[ravi-kumar-pilla]

credentials are correctly excluded from _describe(). Secrets in catalog_properties will still show up in _describe(). Worth a note on the catalog_properties arg: do not put secrets here, use credentials instead.

[ravi-kumar-pilla]

Mock tests are a good start. No explicit test for the native scan_iceberg path. test_load_pyiceberg_scan may hit the fallback depending on Polars version.

PR description says 12 tests, file has 13.

[ravi-kumar-pilla]

Please add a short comment on the iceberg_dataset.py ignore entry explaining why (optional deps not available in doctest env).