feat(datasets): add PaginatedAPIDataset

kedro-datasets/RELEASE.md

@@ -1,6 +1,7 @@
 # Upcoming release
 
 ## Major features and improvements
+- Added `api.PaginatedAPIDataset` for combining list results across JSON API pages. Pagination follows same-host links by default and supports explicitly configured `allowed_hosts` for trusted cross-host APIs.
 - Added support for configuring external Hive table locations in `spark.SparkHiveDataset` through `save_args.path`.
 - Added standard Kedro versioning support to the experimental `netcdf.NetCDFDataset`, including local and remote (S3) files.
 

kedro-datasets/docs/api/kedro_datasets/api.PaginatedAPIDataset.md

@@ -0,0 +1,8 @@
+# PaginatedAPIDataset
+
+`PaginatedAPIDataset` loads list results from a JSON API by following a configured absolute HTTP(S) next-page URL and concatenating the results in page order.
+
+::: kedro_datasets.api.PaginatedAPIDataset
+    options:
+        members: true
+        show_source: true

kedro-datasets/docs/index.md

@@ -7,6 +7,7 @@
 Name | Description
 ------|-------------
 [api.APIDataset](api/kedro_datasets/api.APIDataset.md) | ``APIDataset`` loads/saves data from/to HTTP(S) APIs. It uses the python requests library: <https://requests.readthedocs.io/en/latest/>
+[api.PaginatedAPIDataset](api/kedro_datasets/api.PaginatedAPIDataset.md) | ``PaginatedAPIDataset`` loads and combines list results from JSON APIs using next-page links.
 [biosequence.BioSequenceDataset](api/kedro_datasets/biosequence.BioSequenceDataset.md) | ``BioSequenceDataset`` loads and saves data to a sequence file.
 [dask.CSVDataset](api/kedro_datasets/dask.CSVDataset.md) | ``CSVDataset`` loads and saves data to comma-separated value file(s). It uses Dask remote data services to handle the corresponding load and save operations.
 [dask.ParquetDataset](api/kedro_datasets/dask.ParquetDataset.md) | ``ParquetDataset`` loads and saves data to parquet file(s). It uses Dask remote data services to handle the corresponding load and save operations.

kedro-datasets/kedro_datasets/api/__init__.py

@@ -1,4 +1,4 @@
-"""``APIDataset`` loads the data from HTTP(S) APIs
+"""Datasets for loading data from HTTP(S) APIs
 and returns them into either as string or json Dict.
 It uses the python requests library: https://requests.readthedocs.io/en/latest/
 """
@@ -8,12 +8,13 @@
 import lazy_loader as lazy
 
 try:
-    from .api_dataset import APIDataset
+    from .api_dataset import APIDataset, PaginatedAPIDataset
 except (ImportError, RuntimeError):
     # For documentation builds that might fail due to dependency issues
     # https://github.com/pylint-dev/pylint/issues/4300#issuecomment-1043601901
     APIDataset: Any
+    PaginatedAPIDataset: Any
 
 __getattr__, __dir__, __all__ = lazy.attach(
-    __name__, submod_attrs={"api_dataset": ["APIDataset"]}
+    __name__, submod_attrs={"api_dataset": ["APIDataset", "PaginatedAPIDataset"]}
 )