Improve agent documentation

.github/.licenserc.yaml

@@ -68,6 +68,7 @@ header:
     - '.github/Pull_Request_template.md'
     - '.github/renovate.json'
     - '.github/instructions/*.md'
+    - '.github/copilot-instructions.md'
     # Specific files
     - 'LICENSE'
     - 'AGENTS.md'

.github/copilot-instructions.md

@@ -0,0 +1,7 @@
+# Copilot instructions - Extension for scikit-learn
+
+`AGENTS.md` is the canonical repository guidance. Before reviewing or editing a file, read the root `AGENTS.md` and every applicable directory-level `AGENTS.md`.
+
+For example work, also read `examples/AGENTS.md`.
+
+Also follow the matching `.github/instructions/*.instructions.md` files. They provide additive, Copilot-specific guidance selected by `applyTo`; do not duplicate or override `AGENTS.md` policy here.

.github/instructions/build-config.instructions.md

@@ -1,3 +1,7 @@
+---
+applyTo: "setup.py,setup.cfg,pyproject.toml,dependencies-dev,requirements*.txt,conda-recipe/**,.ci/**,.github/workflows/**"
+---
+
 # Build Configuration Files
 
 ## Purpose
@@ -42,7 +46,7 @@ Build system configuration for Extension for scikit-learn using setup.py, conda,
 
 ## For GitHub Copilot
 
-See [.ci/AGENTS.md](../.ci/AGENTS.md) for comprehensive information including:
+See [.ci/AGENTS.md](../../.ci/AGENTS.md) for comprehensive information including:
 - Platform-specific build configurations
 - CI/CD pipeline details
 - Common build issues and solutions

.github/instructions/daal4py.instructions.md

@@ -1,3 +1,7 @@
+---
+applyTo: "daal4py/**,generator/**"
+---
+
 # daal4py/* - Direct oneDAL Python Bindings
 
 ## Purpose
@@ -18,7 +22,7 @@ pytest daal4py/sklearn/tests/
 
 ## For GitHub Copilot
 
-See [daal4py/AGENTS.md](../daal4py/AGENTS.md) for comprehensive information including:
+See [daal4py/AGENTS.md](../../daal4py/AGENTS.md) for comprehensive information including:
 - Detailed native oneDAL API patterns
 - Model builder conversion process
 - Monkeypatch system architecture

.github/instructions/general.instructions.md

@@ -1,39 +1,13 @@
-# General Repository Instructions - Intel Extension for scikit-learn
-
-## Repository Overview
-
-Extension for scikit-learn accelerates scikit-learn up to 100X using oneDAL (varies by algorithm and data). Zero code changes required for existing sklearn applications.
+---
+applyTo: "**"
+---
 
-**Architecture**: 4-layer system (sklearnex ⇒ {daal4py, onedal} → oneDAL C++)
-**Platforms**: Linux, Windows; CPU (x86_64, ARM), GPU (Intel via SYCL)
-
-**Version Requirements**:
-For current supported versions, always refer to:
-- `setup.py` - Python version classifiers
-- `requirements-test.txt` - scikit-learn and runtime dependencies
-- `dependencies-dev` - Build dependencies
-
-## Quick Build and Test
+# General Repository Instructions - Intel Extension for scikit-learn
 
-```bash
-export DALROOT=/path/to/onedal
-python setup.py develop
-pytest --verbose sklearnex
-```
+Read [../../AGENTS.md](../../AGENTS.md). It is the canonical guide for architecture, version floors, build and test commands, review rules, and directory guides.
 
-## For GitHub Copilot
+Use `setup.py`, `requirements-test.txt`, and `dependencies-dev` as the source of truth for supported versions and dependencies.
 
-See [AGENTS.md](../AGENTS.md) for comprehensive information including:
-- Detailed architecture and layer interactions
-- Algorithm support and GPU compatibility
-- Performance patterns and optimization strategies
-- Development setup and environment configuration
-- Testing strategy and validation approaches
+Read the nearest `AGENTS.md` for subsystem-specific guidance.
 
-## Related Instructions
-- `build-config.instructions.md` - Build system and environment setup
-- `sklearnex.instructions.md` - Primary sklearn interface
-- `daal4py.instructions.md` - Direct oneDAL bindings
-- `onedal.instructions.md` - Low-level C++ bindings
-- `src.instructions.md` - Core C++/Cython implementation
-- `tests.instructions.md` - Testing infrastructure
+More specific `.github/instructions/*.instructions.md` files are additive Copilot guidance selected by `applyTo`; they do not override `AGENTS.md`.

.github/instructions/onedal.instructions.md

@@ -1,3 +1,7 @@
+---
+applyTo: "onedal/**"
+---
+
 # onedal/* - Low-Level C++ Bindings
 
 ## Purpose
@@ -19,7 +23,7 @@ pytest onedal/common/tests/
 
 ## For GitHub Copilot
 
-See [onedal/AGENTS.md](../onedal/AGENTS.md) for comprehensive information including:
+See [onedal/AGENTS.md](../../onedal/AGENTS.md) for comprehensive information including:
 - Backend system architecture (DPC++/Host selection)
 - Data conversion methods and zero-copy patterns
 - Algorithm structure and implementation patterns

.github/instructions/sklearnex.instructions.md

@@ -1,3 +1,7 @@
+---
+applyTo: "sklearnex/**"
+---
+
 # sklearnex/* - Primary sklearn-compatible Interface
 
 ## Purpose
@@ -24,7 +28,7 @@ pytest sklearnex/tests/test_config.py
 
 ## For GitHub Copilot
 
-See [sklearnex/AGENTS.md](../sklearnex/AGENTS.md) for comprehensive information including:
+See [sklearnex/AGENTS.md](../../sklearnex/AGENTS.md) for comprehensive information including:
 - Detailed patching system architecture
 - Device offloading and fallback mechanisms
 - Algorithm support conditions and GPU compatibility

.github/instructions/src.instructions.md

@@ -1,3 +1,7 @@
+---
+applyTo: "src/**"
+---
+
 # src/* - Core C++/Cython Implementation
 
 ## Purpose
@@ -21,7 +25,7 @@ pytest tests/test_daal4py_serialization.py
 
 ## For GitHub Copilot
 
-See [src/AGENTS.md](../src/AGENTS.md) for comprehensive information including:
+See [src/AGENTS.md](../../src/AGENTS.md) for comprehensive information including:
 - C++/Cython architecture and memory management
 - GIL protection patterns and thread safety
 - MPI communication layer and distributed algorithms

.github/instructions/tests.instructions.md

@@ -1,3 +1,7 @@
+---
+applyTo: "tests/**,**/tests/**,deselected_tests.yaml,.circleci/**"
+---
+
 # tests/* - Testing Infrastructure
 
 ## Purpose
@@ -31,7 +35,7 @@ For detailed coverage information, see tests/AGENTS.md.
 
 ## For GitHub Copilot
 
-See [tests/AGENTS.md](../tests/AGENTS.md) for comprehensive information including:
+See [tests/AGENTS.md](../../tests/AGENTS.md) for comprehensive information including:
 - Validation patterns and numerical accuracy requirements
 - Performance testing and timeout configurations
 - Cross-platform testing strategies

AGENTS.md

@@ -1,162 +1,67 @@
 # AGENTS.md - Extension for scikit-learn
 
-## Quick Context
-- **Purpose**: Accelerate scikit-learn using Intel oneDAL optimizations
-- **License**: Apache 2.0
-- **Languages**: Python, C++, Cython
-- **Platforms**: CPU (x86_64, ARM), GPU (Intel via SYCL)
+Accelerates scikit-learn with oneDAL. Python, C++ (pybind11) and Cython; CPU (x86_64, ARM) and Intel GPU via SYCL.
 
-## Architecture (4 Layers)
+## Architecture
 ```text
-User Apps → sklearnex/ ⇒ {daal4py/, onedal/} → Intel oneDAL C++
-                             ↓          ↓
-                       (Cython ext)  (pybind11)
+sklearnex/  ->  daal4py/ (Cython)  -> oneDAL C++
+            ->  onedal/  (pybind11) -> oneDAL C++
 ```
+- `sklearnex/`: scikit-learn API, patching, dispatch between oneDAL and scikit-learn
+- `daal4py/`: Cython bindings generated from oneDAL headers, model builders
+- `onedal/`: pybind11 bindings, data conversion, CPU/GPU backend selection
+- `src/`: C++/Cython core shared by both bindings
+
+## Key Files
+- `sklearnex/dispatcher.py`: patching and algorithm dispatch
+- `sklearnex/_device_offload.py`: device selection and offloading
+- `sklearnex/_config.py`: `config_context` options
+- `onedal/__init__.py`: backend selection (DPC++/host)
+
+## Version Floors
+The floors are enforced in code; read them there rather than trusting prose:
+- oneDAL: `ONEDAL_VERSION` check in `setup.py` (currently >= 2025.0)
+- scikit-learn and Python: `install_requires` / `python_requires` in `setup.py`
+- `sklearn_check_version` gates below the scikit-learn floor are always true and should be deleted
+- Test and build dependencies: `requirements-test.txt`, `dependencies-dev`
 
-**Layer Functions:**
-- `sklearnex/`: sklearn API compatibility, patching, uses both daal4py and onedal
-- `daal4py/`: Direct oneDAL access via Cython, model builders, legacy compatibility
-- `onedal/`: Modern pybind11 bindings, memory management, GPU/CPU backend selection
-- `src/`: C++/Cython core implementation shared by both bindings
-
-## Entry Points by Use Case
-
-**sklearn acceleration** - Global patching or selective imports from sklearnex
-
-**Native oneDAL** - Direct algorithm access via daal4py for maximum performance
-
-**Model conversion** - Convert XGBoost/LightGBM/CatBoost models to oneDAL format for accelerated inference
-
-## Accelerated Algorithms
-- **Clustering**: DBSCAN, K-Means
-- **Classification**: SVM, RandomForest, LogisticRegression, NaiveBayes
-- **Regression**: LinearRegression, Ridge, Lasso, ElasticNet, SVR
-- **Decomposition**: PCA, IncrementalPCA
-- **Neighbors**: KNeighbors (classification/regression)
-- **Preprocessing**: Scalers, normalizers
-- **Statistics**: Basic statistics, covariance
-
-## Device Configuration
-GPU offloading and device control available through sklearnex config_context for supported algorithms.
-
-## Performance Patterns
-- **Memory**: Zero-copy NumPy↔oneDAL, SYCL USM for GPU
-- **Parallelism**: Intel TBB threading, MPI distributed (SPMD), SIMD vectorization
-- **Fallbacks**: oneDAL → sklearn cascading fallback on unsupported operations
-- **Speedups**: Up to 100X acceleration over sklearn (varies by algorithm and data characteristics)
-
-## Key Files for AI Agents
-- `sklearnex/dispatcher.py`: Patching system and algorithm dispatch
-- `sklearnex/_device_offload.py`: Device selection and offloading
-- `onedal/__init__.py`: Backend selection (DPC++/Host)
-- `daal4py/__init__.py`: Native API entry point
-- `src/`: C++/Cython core (distributed computing, memory management)
-
-## Installation
-
-### Quick Install (Users)
-```bash
-# Recommended: via conda
-conda install -c conda-forge scikit-learn-intelex
+## Code Generation
+`generator/` generates daal4py's Cython bindings from oneDAL C++ headers. Modify `generator/wrappers.py` to add new oneDAL algorithms; use direct Python implementation for sklearn compatibility layers.
 
-# Or via pip
-pip install scikit-learn-intelex
-```
+Never edit the generated `build/daal4py_cy.pyx`; change `generator/` and rebuild.
 
-### From Source (Contributors)
+## Build & Test
 ```bash
-# 1. Install oneDAL and set DALROOT
-export DALROOT=/path/to/onedal  # Required
-
-# 2. Install dependencies
+export DALROOT=/path/to/onedal   # required; setup.py fails with "Not set DALROOT variable"
 pip install -r dependencies-dev
-
-# 3. Build in development mode
 python setup.py develop
-```
-
-### Environment Variables
-- `DALROOT`: Path to oneDAL (required for source builds)
-- `MPIROOT`: Path to MPI for distributed support
-- `NO_DPC`: Disable GPU support
-- `NO_DIST`: Disable distributed computing
-- `NO_STREAM`: Disable streaming mode
-
-## Testing Strategy
-Core test suites cover legacy tests, native oneDAL (daal4py), sklearn compatibility (sklearnex), low-level backend (onedal), and global patching. MPI required for distributed (SPMD) testing.
-
-## Performance Expectations
 
-### Algorithm Support
-oneDAL acceleration requires:
-- Supported dtypes: float32, float64
-- Contiguous memory layout preferred
-- Algorithm-specific parameter compatibility
-
-### GPU Support Status
-- **Full GPU**: DBSCAN, K-Means, PCA, KNeighbors
-- **Limited GPU**: LogisticRegression, SVM
-- **CPU Only**:  Ridge, IncrementalPCA
-
-### Error Handling
-Fallback chain: oneDAL → sklearn → error. Configurable via allow_sklearn_after_onedal setting.
-
-### Memory Requirements
-oneDAL requires contiguous data for zero-copy operations. C-contiguous preferred over Fortran-contiguous.
-
-## GPU Hardware
-**Supported Intel GPUs**: Integrated (UHD Graphics, Iris Xe), Discrete (Arc series), Datacenter (Flex)
-**Requirements**: SYCL/DPC++ support, Intel oneAPI toolkit, Unified Shared Memory (USM)
-
-### GPU Setup & Troubleshooting
-**Verify GPU availability:**
-```bash
-python -c "import dpctl; print(dpctl.get_devices())"
+pytest sklearnex/linear_model/tests/     # one module; start here
+pytest --pyargs sklearnex                # one package, as conda-recipe/run_test.sh does
 ```
-
-**Common GPU issues:**
-- **"No GPU device found"** → Install Intel GPU drivers and `intel-opencl-icd` (Linux) or Intel Graphics drivers (Windows). For conda environments, also install `intel-gpu-ocl-icd-system` from Intel's conda channel.
-- **`ImportError: dpctl`** → Install GPU runtime: `pip install dpctl dpnp`
-- **Fallback to CPU** → Check `verbose=True` to see reason (unsupported param, sparse data, etc.)
-- **Out of memory** → Reduce data size or use `target_offload="cpu"`
-
-## Common Errors
-
-**Build/Setup:**
-- **"Not set DALROOT variable"** → Export DALROOT pointing to oneDAL installation
-- **"MPIROOT is not set"** → For distributed mode, set MPIROOT or use `NO_DIST=1`
-
-**Runtime:**
-- **"oneDAL backend not available"** → Algorithm/parameter not supported by oneDAL, fallback to sklearn
-- **"Unsupported parameter"** → Parameter value incompatible with oneDAL (check documentation)
-- **"Sparse data not supported"** → Convert to dense or use sklearn (except SVM, NaiveBayes support CSR)
-- **MPI errors in SPMD** → Call `daal4py.daalinit()` before distributed operations, `daalfini()` at end
-
-## Version Compatibility
-
-**For current supported versions, always check:**
-- `setup.py` - Python version classifiers
-- `requirements-test.txt` - scikit-learn and runtime dependencies
-- `dependencies-dev` - Build dependencies
-
-### Version Support Policy
-
-**Python**:
-Supports officially maintained Python versions. Support for newly released versions may be delayed; support for older versions may extend beyond EOL to accommodate user needs.
-
-**scikit-learn**:
-Aims to support the last 4 scikit-learn releases. sklearn 1.0 maintained as special case for production environments.
-
-**oneDAL**:
-Backwards compatible with oneDAL 2021.1+. Forward compatibility not guaranteed.
-
-## Code Generation
-The generator/ directory contains automated code generation from oneDAL C++ headers to Python bindings. Modify generator/wrappers.py to add new oneDAL algorithms; use direct Python implementation for sklearn compatibility layers.
-
-## Component Documentation
+- Build switches read by `setup.py`: `NO_DIST=1` (no MPI; otherwise `MPIROOT` must be set), `NO_DPC=1` (no GPU), `NO_STREAM=1`. More variants: `doc/sources/building-from-source.rst`.
+- `conda-recipe/run_test.sh` is the full suite (legacy `tests/`, `daal4py`, `sklearnex`, `onedal`, global patching, then MPI). It is slow; run a module first.
+- GPU cases come from the `get_queues()` / `get_dataframes_and_queues()` parametrizations, which only emit a GPU queue when one is available; there is no `gpu` marker.
+- MPI tests need `mpirun` and `--with-mpi`; see the MPI block in `conda-recipe/run_test.sh`.
+
+## Rules for Changes
+These come from recurring review comments; each one has been asked for on several PRs.
+- Comments describe the code as it will be once merged. Don't reference discarded approaches, narrate the change, or mention "this PR".
+- Keep comments short and plain: explain why in one line when one line is enough, and in two sentences rather than a paragraph. Agent-written comments have historically been bloated and hard to read; don't restate the code, hedge, or add emphasis.
+- One PR, one logical change. Drive-by fixes, renames, and mechanical changes (formatting, codegen, mass renames) go in their own PRs.
+- Search before adding a helper, fixture, constant table, or validation routine. Extend the existing one, and name it in the PR description.
+- Don't add a lock, guard, `try`/`except`, or redundant check unless you can name the failure it prevents.
+- A bug fix comes with a test that fails without the fix.
+- Don't hardcode versions, URLs, or paths that a source-of-truth file or Renovate already tracks.
+- New functions get full type hints and a numpydoc docstring.
+- New files use the header `Copyright contributors to the oneDAL project`; leave existing headers alone.
+
+## Directory Guides
+Read the `AGENTS.md` nearest the files you change:
 - `sklearnex/AGENTS.md`: API patterns, device offloading
 - `daal4py/AGENTS.md`: Native oneDAL bindings, model builders
 - `onedal/AGENTS.md`: Pybind11 implementation, memory management
+- `onedal/datatypes/AGENTS.md`: Data conversion, Python C-API reference ownership
 - `src/AGENTS.md`: C++/Cython core, distributed computing
 - `examples/AGENTS.md`: Usage patterns and example scripts
 - `tests/AGENTS.md`: Testing infrastructure, validation patterns

daal4py/AGENTS.md

@@ -147,7 +147,7 @@ daal4py and onedal are **separate** Python binding implementations to oneDAL C++
 5. Update monkeypatch dispatcher for sklearn compatibility
 
 ### Modifying Existing Algorithms
-- Native API: Modify generated sources or generator templates
+- Native API: Modify the generator (`generator/`), never the generated `build/daal4py_cy.pyx`
 - sklearn API: Direct edits in `daal4py/sklearn/`
 - Model builders: Edit `daal4py/mb/`
 

doc/AGENTS.md

@@ -30,3 +30,9 @@ Production deployment: `./build-doc.sh --gh-pages`
 - Include proper docstrings for autodoc generation
 - Test documentation builds locally before submitting
 - Maintain cross-references and intersphinx links
+
+## Rules for Changes
+- When a fact changes, update every place it appears, including `README.md` (the PyPI description via `setup.py`) and `doc/sources/tests.rst` for test commands.
+- Use Sphinx roles for API names (`:obj:`, `:class:`, `:func:`) and the `|sklearnex|` / `|onedal|` substitutions from `rst_prolog` in `sources/conf.py`, not raw double backticks or spelled-out product names.
+- Keep pages user-facing. Implementation detail belongs in code comments or `AGENTS.md`.
+- When revising a page, shorten or replace text rather than appending to it.