diff --git a/.ci/AGENTS.md b/.ci/AGENTS.md index 54d7e20908e..3339092c023 100644 --- a/.ci/AGENTS.md +++ b/.ci/AGENTS.md @@ -5,61 +5,10 @@ This document describes the CI infrastructure for oneDAL (Intel Data Analytics Library) ## Directory Structure -### `.ci` Folder - -The `.ci` folder contains the core CI infrastructure scripts and configurations organized into three main subdirectories: - -#### `.ci/env/` - Environment Setup Scripts -- **`apt.sh`** - Package installation script for Ubuntu/Debian systems with functions for: - - Intel OneAPI toolkit components (DPC++, TBB, DPL, MKL) - - Development tools (clang-format, editorconfig-checker) - - Base development packages and dependencies -- **`bazelisk.sh`** - Bazel build system setup and installation -- **`editorconfig-checker.sh`** - EditorConfig compliance checker installation -- **`environment.yml`** - Conda environment specification -- **`openblas.sh`** - OpenBLAS library installation and configuration -- **`openrng.sh`** - OpenRNG backend setup for random number generation -- **`tbb.sh`** / **`tbb.bat`** - Intel TBB (Threading Building Blocks) setup for Linux/Windows -- **`riscv64-clang-crosscompile-toolchain.cmake`** - RISC-V cross-compilation toolchain configuration - -#### `.ci/pipeline/` - CI Pipeline Definitions -- **`ci.yml`** - Main Azure DevOps pipeline configuration with: - - Multi-platform build matrices (Linux, Windows) - - Compiler configurations (GNU, Clang, Intel) - - Build targets (daal, onedal_c) - - Testing and validation jobs - - Artifact publishing -- **`docs.yml`** - Documentation build and deployment pipeline - -#### `.ci/scripts/` - Build and Test Scripts -- **`build.sh`** / **`build.bat`** - Cross-platform build orchestration with support for: - - Multiple compilers (gnu, clang, icx) - - Architecture optimizations (AVX2, etc.) - - Backend configurations (MKL, reference implementations) - - Cross-compilation capabilities -- **`test.sh`** / **`test.bat`** - Comprehensive testing framework execution -- **`clang-format.sh`** - Code formatting verification -- **`describe_system.sh`** - System information collection for debugging -- **`abi_check.sh`** - ABI compatibility verification -- **`install_basekit.bat`** - Intel OneAPI Base Toolkit installation for Windows -- **`collect_opencl_rt.ps1`** - OpenCL runtime collection script - -### `.github/workflows/` - GitHub Actions Workflows - -#### Core CI Workflows -- **`ci.yml`** - Main CI pipeline for x86 platforms with DPC++ builds -- **`ci-aarch64.yml`** - AArch64 (ARM64) specific CI pipeline -- **`nightly-build.yml`** / **`nightly-test.yml`** - Automated nightly builds and testing - -#### Specialized Workflows -- **`docker-validation-ci.yml`** / **`docker-validation-nightly.yml`** - Container-based validation -- **`docs-release.yml`** - Documentation deployment and release management -- **`label-enforcement.yml`** - PR labeling automation -- **`pr-checklist.yml`** - Pull request compliance verification -- **`renovate-validation.yml`** - Dependency update validation -- **`skywalking-eyes.yml`** - License header compliance checking -- **`slack-pr-notification.yml`** - Team notification system -- **`openssf-scorecard.yml`** - Security scorecard assessment +- `.ci/pipeline/ci.yml`: the Azure DevOps pipeline (build matrix, `FormatterChecks`); `docs.yml` builds the docs. +- `.ci/env/`: dependency installers. `apt.sh` takes a component name (`dev-base`, `mkl`, ...); `tbb`, `openblas` and `bazelisk` each have `.sh` and Windows variants. +- `.ci/scripts/`: `build.sh` / `build.bat` (compiler, optimization, backend and cross-compile options), `test.sh` / `test.bat`, `clang-format.sh`, `abi_check.sh`, and the Windows release checks (`compare_windows_release.ps1`, `test_bazel_release_cmake_example.ps1`). +- `.github/workflows/`: GitHub Actions. `ci.yml`, `ci-win.yml` and `ci-aarch64.yml` build and test; `nightly-build.yml` produces artifacts other repositories download (see `.github/AGENTS.md`). ## CI/CD Architecture @@ -74,7 +23,6 @@ The oneDAL CI infrastructure supports: ### CI Platform Integration - **GitHub Actions**: Primary CI/CD platform for public workflows - **Azure DevOps**: Extended validation and internal testing -- **Mergify**: Automated merge management - **Renovate**: Dependency update automation - **Codefactor**: Code quality analysis @@ -92,6 +40,10 @@ The CI system employs comprehensive build matrices covering: - **Security**: OpenSSF Scorecard integration - **Documentation**: Automated doc generation and validation +## Rules for Changes +- Pipelines call scripts in `.ci/scripts/` and `.ci/env/`. Change the script, not only the pipeline YAML that calls it. +- Style checks (clang-format, editorconfig-checker) run only in Azure `FormatterChecks`. See "Verification Before You Push" in the root `AGENTS.md`. + ## Usage Guidelines ### Local Development diff --git a/.gitattributes b/.gitattributes index 7af5a85c695..2adc80089b7 100644 --- a/.gitattributes +++ b/.gitattributes @@ -5,7 +5,7 @@ # to LF line endings on checkout. *.c text *.h text -*.i text +*.i text linguist-language=C++ *.hpp text *.cpp text *.def text diff --git a/.github/.licenserc.yaml b/.github/.licenserc.yaml index cf9e7d1a4fe..a53fba4dda8 100644 --- a/.github/.licenserc.yaml +++ b/.github/.licenserc.yaml @@ -63,7 +63,6 @@ header: - 'docs/source/substitutions_common.txt' - 'docs/source/substitutions_specific.txt' # Some files from .ci/.github - - '.github/instructions/*.md' - '.github/CODEOWNERS' - '.github/pull_request_template.md' - '.github/renovate.json' diff --git a/.github/instructions/build-systems.instructions.md b/.github/instructions/build-systems.instructions.md deleted file mode 100644 index ea25e7c399f..00000000000 --- a/.github/instructions/build-systems.instructions.md +++ /dev/null @@ -1,149 +0,0 @@ ---- -applyTo: ["**/makefile", "**/Makefile", "**/BUILD", "**/BUILD.bazel", "**/*.bazel", "**/*.mk", "**/CMakeLists.txt"] ---- - -# Build Systems Instructions for GitHub Copilot - -## Build System Priority - -1. **Make** - Primary production builds, core library -2. **CMake** - End-user integration, examples -3. **Bazel** - Development, testing, new features - -## πŸš€ Essential Commands - -### Make (Production) -```bash -# Build everything -`make` - -# Platform-specific builds -`make PLAT=lnx32e COMPILER=icx` -`make PLAT=win32e COMPILER=vc` - -# CPU targets -`make REQCPU="sse2 avx2 avx512"` - -# Backend selection -`make BACKEND_CONFIG=mkl` # Intel MKL (default) -`make BACKEND_CONFIG=ref` # Reference/OpenBLAS -``` - -### Bazel (Development) -```bash -# Build targets -`bazel build //cpp/oneapi/dal:core` -`bazel build //examples/daal/cpp:association_rules` - -# Test targets -`bazel test //cpp/oneapi/dal:tests` -`bazel test --config=dpc //cpp/oneapi/dal:tests` # GPU tests -``` - -### CMake (Integration) -```bash -# Configure and build -`cmake -B build -S . -DCMAKE_BUILD_TYPE=Release` -`cmake --build build --parallel` -``` - -## πŸ—οΈ CPU Architecture Support - -- **x86-64**: sse2, avx2, avx512 (Intel/AMD) -- **ARM64**: sve (ARM Scalable Vector Extension) -- **RISC-V**: rv64 (RISC-V 64-bit) - -### CPU Dispatch Pattern -```cpp -// Runtime CPU detection and optimal code selection -template -services::Status compute(/* parameters */) { - // CPU-specific optimized implementation -} -``` - -## πŸ”§ Platform Configuration - -### Linux (lnx32e) -```makefile -PLAT := lnx32e -COMPILER := icx # Intel oneAPI C++/DPC++ -COMPILER := gnu # GCC -COMPILER := clang # Clang -``` - -### Windows (win32e) -```makefile -PLAT := win32e -COMPILER := vc # Microsoft Visual C++ -COMPILER := icx # Intel oneAPI -``` - -### Required Environment -```bash -# Intel oneAPI (for DPC++) -export ONEAPI_ROOT=/path/to/oneapi -export PATH=$ONEAPI_ROOT/compiler/latest/linux/bin:$PATH - -# TBB (required) -export TBBROOT=/path/to/tbb - -# MKL (if using MKL backend) -export MKLROOT=/path/to/mkl -``` - -## 🎯 Bazel Build Rules - -### Algorithm Module Pattern -```python -# DAAL module -daal_module( - name = "kmeans", - features = [ "c++17" ], - cpu_defines = { - "sse2": [ "DAAL_CPU=sse2" ], - "avx2": [ "DAAL_CPU=avx2" ], - "avx512": [ "DAAL_CPU=avx512" ], - }, -) - -# oneAPI module -dal_module( - name = "kmeans", - compile_as = ["c++", "dpc++"], # CPU and GPU -) -``` - -### Test Configuration -```python -dal_test_module( - name = "core_test", - compile_as = ["c++", "dpc++"], - dal_deps = [":core"], -) -``` - -## πŸ” Common Issues - -### Build Failures -- **C++17 compliance**: Ensure GCC 7+, Clang 6+, MSVC 2017+ -- **Missing dependencies**: Check TBB, MKL installation -- **CPU targets**: Verify CPU flags match target architecture - -### Platform Differences -- **Windows**: Use `vc` compiler for native Windows builds -- **Linux**: Prefer `icx` for Intel optimizations, `gnu` for compatibility -- **ARM**: Use `clang` compiler, set `PLAT=lnxarm` - -## 🎯 Critical Rules - -- **Make**: Primary for production builds and CI/CD -- **Bazel**: Development workflow and new feature development -- **CMake**: User integration, not primary build system -- **CPU Dispatch**: Always implement multi-architecture support -- **Dependencies**: TBB required, MKL preferred for performance - -## πŸ”— References - -- **[general.instructions.md](general.instructions.md)** - Repository overview -- **[cpp-coding-guidelines.instructions.md](cpp-coding-guidelines.instructions.md)** - C++ standards diff --git a/.github/instructions/cpp-coding-guidelines.instructions.md b/.github/instructions/cpp-coding-guidelines.instructions.md deleted file mode 100644 index e3792e0821e..00000000000 --- a/.github/instructions/cpp-coding-guidelines.instructions.md +++ /dev/null @@ -1,293 +0,0 @@ ---- -applyTo: ["**/*.cpp", "**/*.hpp", "**/*.h", "**/cpp/**", "**/include/**"] ---- - -# C++ Development and Coding Guidelines for GitHub Copilot - -## πŸ“‹ **C++ Standards and Language Features** - -### Language Compliance -- **C++ Standard**: C++17 -- **Compiler Support**: GCC 7+, Clang 6+, MSVC 2017+ -- **Extensions**: Avoid compiler-specific extensions - -### Code Organization -- **Header Files**: Keep headers clean with minimal dependencies -- **Implementation**: Implement in .cpp files, not headers -- **DAAL .i Files**: Use `.i` files for template implementations requiring compile-time inclusion -- **Forward Declarations**: Use to minimize header dependencies -- **Template Specializations**: Place in appropriate headers - -## πŸ—οΈ **Interface-Specific Development Patterns** - -### oneAPI Interface - `cpp/oneapi/` -```cpp -#include "oneapi/dal/algo/kmeans.hpp" -#include "oneapi/dal/table/homogen.hpp" - -auto desc = kmeans::descriptor() - .set_cluster_count(10) - .set_max_iteration_count(100); - -auto train_result = train(desc, train_data); -auto infer_result = infer(desc, train_result.get_model(), test_data); -``` - -**oneAPI Patterns**: -- **Headers**: Use `.hpp` extension with `#pragma once` -- **Namespaces**: `oneapi::dal` structure -- **Memory Management**: `std::unique_ptr`, `std::shared_ptr`, `std::make_unique` -- **Error Handling**: C++ exceptions (`throw`, `try/catch`) - -### DAAL Interface - `cpp/daal/` -```cpp -#include "algorithms/kmeans/kmeans_batch.h" -#include "data_management/data/homogen_numeric_table.h" - -auto training = new kmeans_batch(); -auto parameter = training->getParameter(); -parameter->nClusters = 10; - -training->input.set(kmeans_batch_input::data, data); -services::Status status = training->compute(); -if (status != services::Status::OK) { - // Handle error -} -``` - -**DAAL Patterns**: -- **Headers**: Use `.h` extension with traditional include guards (`#ifndef __FILE_NAME_H__`) -- **Implementation Files**: Use `.i` extension for template implementations -- **Namespaces**: `daal::algorithms`, `daal::data_management`, `daal::services` -- **Memory Management**: `daal::services::SharedPtr` (validated in codebase) -- **Error Handling**: `services::Status` return codes (validated in codebase) - -### DAAL Template Implementation Files (.i files) -```cpp -// Example: kmeans_init_impl.i -#include "algorithms/algorithm.h" -#include "data_management/data/numeric_table.h" - -namespace daal::algorithms::kmeans::init::internal { - -template -Status init(size_t p, size_t n, size_t nRowsTotal, size_t nClusters, - algorithmFPType * clusters, NumericTable * ntData, unsigned int seed) { - // Template implementation here -} - -} // namespace -``` - -**DAAL .i File Patterns**: -- **Usage**: Include in `.cpp` files: `#include "algorithm_method_impl.i"` -- **Purpose**: Template implementations requiring compile-time instantiation -- **Location**: Exclusively in `cpp/daal/src/` directory structure -- **CPU Specialization**: Support different CPU architectures (SSE, AVX, AVX512) -- **Naming**: Follow pattern: `{algorithm}_{method}_impl.i` -- **Build System**: Listed as headers in Bazel BUILD files - -## πŸ’Ύ **Memory Management Patterns** - -### DAAL Interface -```cpp -// βœ… CORRECT - DAAL patterns found in codebase -daal::services::SharedPtr data_; -services::SharedPtr error_ptr; - -// Pattern found in error handling -typedef SharedPtr ErrorPtr; -``` - -### oneAPI Interface -```cpp -// βœ… CORRECT - oneAPI patterns found in codebase -std::unique_ptr> voting_; -std::shared_ptr store_ = std::make_shared(); -explicit array(const std::shared_ptr& data, std::int64_t count); - -// RAII Pattern -class DataProcessor { -private: - std::unique_ptr buffer_; - std::shared_ptr table_; - -public: - DataProcessor(size_t size) - : buffer_(std::make_unique(size)) - , table_(std::make_shared(buffer_.get(), rows, cols)) - { } -}; -``` - -## 🚨 **Error Handling** - -### DAAL Interface - Status Codes -```cpp -// Pattern validated in codebase -services::Status compute() { - // Implementation - return services::Status::OK; -} - -// Usage pattern -services::Status status = algorithm->compute(); -if (status != services::Status::OK) { - daal::services::throwIfPossible(status); - return nullptr; -} -``` - -### oneAPI Interface - Exceptions -```cpp -// Pattern validated in codebase -try { - auto result = train(desc, data); - return result.get_model(); -} catch (const std::exception& e) { - std::cerr << "Training failed: " << e.what() << std::endl; - throw; -} - -// Custom exceptions found in codebase -throw std::invalid_argument{ "Data types do not match" }; -throw std::runtime_error{ "We reached the end of input stream" }; -throw unimplemented{ dal::detail::error_messages::unsupported_data_layout() }; -``` - -## 🏷️ **Naming Conventions** - -Based on actual codebase analysis: - -- **File Names**: - - DAAL: Lowercase with underscores: `kmeans_batch.h`, `homogen_numeric_table.h` - - oneAPI: Lowercase with underscores: `train.hpp`, `compute_kernel.hpp` -- **Classes/Structs**: - - DAAL: `BatchContainer`, `HomogenNumericTable`, `KMeansBatch` - - oneAPI: `train_ops`, `compute_kernel`, `uniform_voting` -- **Functions**: Descriptive verb-noun: `compute()`, `get_data()`, `wait_and_throw()` -- **Variables**: Meaningful names: `cluster_count`, `error_count`, `host_copy` -- **Class Members**: With underscore suffix: `entries_`, `store_`, `comm_` -- **Constants**: UPPER_CASE: `MAX_ITERATIONS`, `DEFAULT_CLUSTER_COUNT` - -## πŸ—οΈ **Template Patterns** - -### Function Templates -```cpp -// Pattern from codebase -template -struct train_ops - : dal::kmeans::detail::train_ops {}; - -template -auto process_data(const std::vector& input) -> std::vector { - std::vector output; - output.reserve(input.size()); - - for (const auto& value : input) { - output.push_back(process_value(value)); - } - return output; -} -``` - -### Class Templates (DAAL Pattern) -```cpp -// Pattern validated in DAAL codebase -template -class BatchContainer : public daal::algorithms::AnalysisContainerIface -{ -public: - virtual services::Status compute() override; -}; - -template -class DAAL_EXPORT HomogenNumericTable : public NumericTable -{ - typedef DataType baseDataType; -}; -``` - -## πŸ“ **Coding Style Standards** - -### General Rules -- **Consistency**: Maintain consistent coding style across the codebase -- **Readability**: Code should be self-documenting and easy to understand -- **Indentation**: Use 4 spaces (not tabs) -- **File Endings**: Always put an empty line at the end of files -- **Include Guards**: - - oneAPI: Use `#pragma once` - - DAAL: Traditional guards (`#ifndef __FILENAME_H__`) - -### Class Organization (VALIDATED) -```cpp -template -class Algorithm { -public: - // Public interface first - services::Status compute(); - Result getResult() const; - -protected: - // Protected members - virtual void initialize(); - -private: - // Private implementation and data - DataTable data_; - Parameters params_; -}; -``` - -### Move Semantics -```cpp -// βœ… CORRECT - Modern C++ patterns -class DataManager { -public: - DataManager(std::vector data) : data_(std::move(data)) {} - - auto get_data() && -> std::vector { - return std::move(data_); - } - -private: - std::vector data_; -}; -``` - -## πŸ” **PR Review Checklist** - -### Code Quality (VALIDATED CRITERIA) -- [ ] **Interface consistency** - No mixing DAAL/oneAPI patterns - - DAAL: `services::Status`, `SharedPtr`, `.h` headers, traditional guards - - oneAPI: exceptions, `std::unique_ptr`/`std::shared_ptr`, `.hpp` headers, `#pragma once` -- [ ] **Memory management** follows interface patterns - - DAAL: `daal::services::SharedPtr` - - oneAPI: `std::unique_ptr`, `std::shared_ptr` -- [ ] **Error handling** appropriate for interface - - DAAL: `services::Status` return codes - - oneAPI: C++ exceptions -- [ ] **Naming conventions** followed consistently -- [ ] **Header guards** correct for interface (`#pragma once` vs traditional) - -### Style and Security -- [ ] **C++17 maximum standard** - no C++20/23 features -- [ ] **Indentation** uses 4 spaces (no tabs) -- [ ] **File headers** include copyright and description (validated format) -- [ ] **Type safety** with proper validation -- [ ] **Const correctness** applied appropriately -- [ ] **Bounds checking** implemented where needed - -## 🚨 **Critical Reminders for PR Review** - -1. **Interface Separation** - NEVER mix DAAL and oneAPI patterns in the same file -2. **Memory Management** - Use correct smart pointer types for each interface -3. **Error Handling** - Use status codes for DAAL, exceptions for oneAPI -4. **Header Extensions** - `.h` for DAAL, `.hpp` for oneAPI -5. **Include Guards** - Traditional guards for DAAL, `#pragma once` for oneAPI -6. **C++17 Compliance** - No C++20/23 features for maximum compatibility - -## πŸ”„ **Cross-Reference** -- **[general.md](/.github/instructions/general.md)** - General repository context -- **[build-systems.md](/.github/instructions/build-systems.md)** - Build system instructions -- **[examples.md](/.github/instructions/examples.md)** - Example code patterns \ No newline at end of file diff --git a/.github/instructions/documentation.instructions.md b/.github/instructions/documentation.instructions.md deleted file mode 100644 index ba13be1b7f9..00000000000 --- a/.github/instructions/documentation.instructions.md +++ /dev/null @@ -1,98 +0,0 @@ ---- -applyTo: ["**/docs/**", "**/*.rst", "**/*.md", "**/Doxyfile"] ---- - -# Documentation Instructions for GitHub Copilot - -## Documentation Architecture - -oneDAL uses a multi-format documentation system: -- **API Reference**: Doxygen-generated C++ API documentation -- **Developer Guide**: reStructuredText-based comprehensive guides -- **Examples**: Code examples and tutorials -- **User Guides**: Installation and usage instructions - -## C++ API Documentation (Doxygen) - -### Header File Documentation -```cpp -/** - * @brief K-means clustering algorithm implementation - * - * @tparam Float Floating-point type for computations - * @tparam Method Algorithm method (lloyd_dense, lloyd_csr) - * - * @par Example - * @code - * auto desc = kmeans::descriptor().set_cluster_count(10); - * auto result = train(desc, data); - * @endcode - * - * @par Thread Safety - * This class is not thread-safe. - */ -template -class kmeans { -public: - /** - * @brief Sets the number of clusters - * @param[in] count Number of clusters to generate - * @return Reference to this descriptor for method chaining - */ - auto& set_cluster_count(std::int64_t count); -}; -``` - -### Implementation Documentation -```cpp -/** - * @brief Computes K-means clustering - * @param[in] desc Algorithm descriptor with parameters - * @param[in] data Input data table - * @return Training result containing the trained model - * - * @par Exception Safety - * Strong exception guarantee - if an exception is thrown, the program state remains unchanged. - * - * @par Thread Safety - * This function is not thread-safe. - */ -template -auto train(const kmeans::descriptor& desc, const table& data); -``` - -## reStructuredText Documentation - -### Section Structure -```rst -K-Means Clustering -================== - -Overview --------- -K-means clustering groups similar data points into clusters. - -Usage Example ------------- -.. code-block:: cpp - - auto desc = kmeans::descriptor().set_cluster_count(10); - auto result = train(desc, data); -``` - -## Documentation Standards - -### Content Guidelines -- **Accuracy**: Ensure technical correctness -- **Completeness**: Cover all public APIs -- **Clarity**: Use clear, concise language -- **Examples**: Provide working code examples - -### Maintenance Guidelines -- **Updates**: Keep documentation synchronized with code -- **Testing**: Validate all code examples compile and run -- **Links**: Test internal and external links regularly - -## Cross-Reference -- **[AGENTS.md](/docs/AGENTS.md)** - Documentation context -- **[examples.md](/.github/instructions/examples.md)** - Example patterns \ No newline at end of file diff --git a/.github/instructions/examples.instructions.md b/.github/instructions/examples.instructions.md deleted file mode 100644 index a1f03ad2c99..00000000000 --- a/.github/instructions/examples.instructions.md +++ /dev/null @@ -1,123 +0,0 @@ ---- -applyTo: ["**/examples/**", "**/samples/**"] ---- - -# Examples Instructions for GitHub Copilot - -## 🎯 **PRIMARY GOAL: PR Review Assistance** - -**GitHub Copilot's main purpose is to assist with PR reviews. Examples should be concise and review-friendly.** - -## Quick Reference Patterns - -### πŸ”΄ **Critical Patterns (Must Include)** -- **Error Handling**: Always include proper error checking -- **Resource Management**: Use RAII and smart pointers -- **Interface Consistency**: Follow DAAL vs oneAPI patterns -- **Build Compatibility**: Ensure Make compatibility - -### 🟑 **Important Patterns (Should Include)** -- **Documentation**: Clear comments and examples -- **Testing**: Include test cases where appropriate -- **Performance**: Consider performance implications -- **Platform Support**: Cross-platform compatibility - -## πŸš€ Quick Start Examples - -### 1. Basic Algorithm Usage (Concise) -```cpp -// Quick K-means example -auto desc = kmeans::descriptor() - .set_cluster_count(10) - .set_max_iteration_count(100); -auto result = train(desc, data); -``` - -### 2. Error Handling (Essential) -```cpp -try { - auto result = train(desc, data); - return result; -} catch (const std::exception& e) { - std::cerr << "Training failed: " << e.what() << std::endl; - throw; -} -``` - -### 3. Resource Management (Critical) -```cpp -// Use smart pointers for ownership -auto data = std::make_unique(rows, cols); -auto result = std::make_unique(); -``` - -## πŸ“š Example Categories - -### Algorithm Examples -- **Classification**: Decision trees, SVM, Naive Bayes -- **Clustering**: K-means, DBSCAN, EM -- **Regression**: Linear regression, Ridge regression -- **Dimensionality Reduction**: PCA, SVD - -### Data Management Examples -- **Data Loading**: CSV, binary, streaming -- **Data Transformation**: Feature scaling, normalization -- **Memory Management**: Efficient data handling - -### Performance Examples -- **CPU Optimization**: SIMD, threading -- **GPU Acceleration**: SYCL, memory management -- **Distributed Computing**: MPI, load balancing - -## πŸ” PR Review Example Checklist - -### For New Examples -- [ ] **Concise and Clear**: Easy to understand quickly -- [ ] **Complete**: Self-contained and runnable -- [ ] **Error Handling**: Proper exception safety -- [ ] **Build Compatibility**: Works with Make builds -- [ ] **Interface Consistency**: Uses appropriate interface -- [ ] **Documentation**: Clear comments and usage - -### For Example Updates -- [ ] **Backward Compatibility**: Existing examples still work -- [ ] **Documentation**: Examples are updated -- [ ] **Validation**: Examples are executed and passing - -## 🎯 Context-Aware Examples - -### oneAPI Context (`cpp/oneapi/`, `examples/oneapi/`) -```cpp -// Modern oneAPI pattern (concise) -#include "oneapi/dal/algo/kmeans.hpp" -#include "oneapi/dal/table/homogen.hpp" - -auto desc = kmeans::descriptor().set_cluster_count(10); -auto data = read(csv_file); -auto result = train(desc, data); -``` - -### DAAL Context (`cpp/daal/`, `examples/daal/`) -```cpp -// Traditional DAAL pattern (concise) -#include "algorithms/kmeans/kmeans_batch.h" -#include "data_management/data/homogen_numeric_table.h" - -auto algorithm = new kmeans::Batch(); -algorithm->input.set(kmeans::data, data); -algorithm->compute(); -auto result = algorithm->getResult(); -``` - -### SYCL GPU Example -```cpp -#include - -sycl::queue q(sycl::gpu_selector_v); -auto desc = kmeans::descriptor().set_cluster_count(10); -auto result = train(q, desc, data); // GPU execution -``` - -## Cross-Reference -- **[AGENTS.md](/examples/AGENTS.md)** - Example patterns context -- **[coding-guidelines.md](/.github/instructions/cpp-coding-guidelines.md)** - Coding standards \ No newline at end of file diff --git a/.github/instructions/general.instructions.md b/.github/instructions/general.instructions.md deleted file mode 100644 index ade6f23ab50..00000000000 --- a/.github/instructions/general.instructions.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -applyTo: "**" ---- - -# General Repository Instructions for GitHub Copilot - -## Repository Context - -**oneDAL** (oneAPI Data Analytics Library) is a high-performance C++ library for machine learning algorithms with dual interfaces: - -- **Traditional DAAL**: Legacy CPU-focused interface for backward compatibility -- **Modern oneAPI**: GPU-accelerated interface with SYCL support for new development - -**Integration Note**: oneDAL works together with [scikit-learn-intelex](https://github.com/intel/scikit-learn-intelex). They share common validation aspects and work together to provide accelerated machine learning capabilities. - -## 🎯 **PRIMARY GOAL: PR Review Assistance** - -**GitHub Copilot's main purpose in this repository is to assist with PR reviews and validation.** - -### πŸ“‹ **PR Review Priority Checklist** -- [ ] **🟑 IMPORTANT**: Interface consistency preserved -- [ ] **🟑 IMPORTANT**: Coding standards followed -- [ ] **🟑 IMPORTANT**: Cross-repository impact assessed - -## Critical Rules - -### C++ Standards -- **Language**: Use C++17 -- **Headers**: Use `#pragma once` for oneAPI, traditional guards for DAAL -- **Smart Pointers**: Always use `std::unique_ptr` and `std::shared_ptr` -- **RAII**: Follow Resource Acquisition Is Initialization principles - -### Coding Standards -- **Comprehensive Guidelines**: Follow [coding-guidelines.md](/.github/instructions/cpp-coding-guidelines.md) for all code -- **Naming Conventions**: Use consistent naming patterns -- **Code Structure**: Follow proper declaration order and organization -- **Documentation**: Include proper comments and documentation - -## Context-Aware Behavior - -### When Working in `cpp/oneapi/` -- Suggest oneAPI patterns and SYCL integration -- Use modern C++17 features -- Include appropriate oneAPI headers -- Follow oneAPI naming conventions -- Suggest GPU-accelerated patterns when appropriate - -### When Working in `cpp/daal/` -- Suggest DAAL patterns and legacy compatibility -- Use modern C++17 features -- Include appropriate DAAL headers -- Follow DAAL naming conventions -- Maintain backward compatibility - -### When Working in `examples/` or `samples/` -- Ensure examples are complete and runnable -- Use appropriate interface based on subdirectory -- Include proper error handling -- Follow example patterns established in the directory - -## What NOT to Generate - -- Interface mixing between DAAL and oneAPI -- Raw pointers for ownership -- Outdated C++98 patterns -- C++20/23 features (for compatibility reasons) -- Platform-specific hardcoded code -- Incomplete error handling -- Examples that don't compile or run - -## What TO Generate - -- RAII-compliant resource management -- Exception-safe code -- Context-appropriate interface usage -- Proper dependency management -- Complete, runnable examples -- Proper error handling and validation - -## πŸ” **PR Review Assistance (PRIMARY FOCUS)** - -### Common PR Review Scenarios - -#### **1. New Algorithm Implementation** -- [ ] **Interface Consistency**: Uses appropriate interface (oneAPI for new, DAAL for legacy) -- [ ] **Make Compatibility**: Works with Make build system -- [ ] **Bazel Testing**: Includes proper test configuration -- [ ] **C++17 Compliance**: No C++20/23 features used -- [ ] **Coding Standards**: Follows comprehensive guidelines - -#### **2. Build System Changes** -- [ ] **Cross-Platform**: Changes work on Linux, Windows -- [ ] **Dependency Management**: Proper dependency handling - -#### **3. Interface Changes** -- [ ] **Backward Compatibility**: Changes are not breaking backward compatibility -- [ ] **scikit-learn-intelex Impact**: Consider impact on integration -- [ ] **API Consistency**: New APIs follow established patterns - -## Cross-Reference -- **[coding-guidelines.md](/.github/instructions/cpp-coding-guidelines.md)** - Comprehensive coding standards -- **[build-systems.md](/.github/instructions/build-systems.md)** - Build system guidance -- **[examples.md](/.github/instructions/examples.md)** - Example patterns \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 7e704ce4fd5..1f62ab2fb10 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,98 +1,96 @@ +# AGENTS.md - oneDAL -# oneDAL Repository - AI Agents Context Guide +oneDAL is a C++ machine learning library with two interfaces: DAAL (`cpp/daal`, CPU) and oneAPI (`cpp/oneapi`, CPU and SYCL GPU). Make builds releases, Bazel runs tests (`dev/`); CI lives in `.ci/` and `.github/`. It is the backend for [scikit-learn-intelex](https://github.com/uxlfoundation/scikit-learn-intelex). +Also at the top level: `examples/`, `samples/` (MPI/CCL), `docs/`, `data/`, `deploy/`, `cmake/` (release CMake configs), `conda-recipe/`. -> **Purpose**: Comprehensive context for AI agents working with the oneDAL repository structure, coding standards, and development guidelines. +## Rules for Changes -## 🎯 Repository Overview +These come from recurring maintainer review comments. Directory-specific rules are in the nearest `AGENTS.md`. -**oneDAL** (oneAPI Data Analytics Library) is a high-performance C++ library for machine learning algorithms, providing both traditional DAAL interfaces and modern oneAPI interfaces with SYCL support for GPU acceleration. +- Comments describe the code as it will be once merged. Don't reference discarded approaches, narrate the change, or mention "this PR". +- Keep comments short and plain: explain why in one line when one line is enough, and in two sentences rather than a paragraph. Agent-written comments have historically been bloated and hard to read; don't restate the code, hedge, or add emphasis. +- One PR, one logical change. Drive-by fixes, renames and mechanical changes (formatting, generated code, mass renames) go in their own PRs. +- Search before adding a helper, constant table or validation routine. Extend the existing one and name it in the PR description. +- Don't add a lock, critical section, guard or redundant check unless you can name the failure it prevents. +- A bug fix comes with a test that fails without the fix. +- Don't hardcode versions, URLs or paths that have a source of truth (`makefile.ver`, `MODULE.bazel`, `.github/renovate.json`). +- New source files use the header `Copyright contributors to the oneDAL project`. Leave existing headers alone. +- ASCII only in source, comments and docs. Keep each file's existing line endings. +- Scripts with a `#!/bin/sh` shebang use POSIX `sh` only. `.bat` files follow `cmd.exe` quoting; don't mix PowerShell and CMD syntax. +- New Bash scripts start with `set -euo pipefail`. Use `mkdir -p`, and don't silence failures with a bare `|| true`. +- Keep to the interface you are in: DAAL uses `services::Status` and `SharedPtr`, oneAPI uses exceptions and standard smart pointers. Never mix them in one file (see `cpp/AGENTS.md`). +- Flag any public API or ABI change; it needs a deprecation path (see `cpp/AGENTS.md`). -**Integration Note**: oneDAL works with [scikit-learn-intelex](https://github.com/intel/scikit-learn-intelex). They share common validation aspects and provide accelerated machine learning capabilities together. +## Reviewing -### Key Characteristics -- **Language**: Modern C++ (17+) -- **Architecture**: Dual interface system (DAAL + oneAPI) -- **Build Systems**: Make (production), CMake (integration), Bazel (development/testing) -- **Targets**: CPU (SIMD optimized), GPU (SYCL), Distributed (MPI) -- **License**: Apache License 2.0 +Report source-confirmed problems with correctness, compatibility, ownership, CPU dispatch, error handling and test coverage. Don't report what CI already enforces (formatting, license headers, editorconfig). -## πŸ—οΈ Repository Structure +## Verification Before You Push -``` -daal/ -β”œβ”€β”€ cpp/ # Core C++ implementation -β”‚ β”œβ”€β”€ daal/ # Traditional DAAL interface -β”‚ └── oneapi/ # Modern oneAPI interface -β”œβ”€β”€ dev/ # Development tools and build configs -β”œβ”€β”€ examples/ # Usage examples and tutorials -β”œβ”€β”€ docs/ # Documentation and API references -└── deploy/ # Deployment and packaging -``` - -## πŸ”— Context Files for AI Agents - -Specialized AGENTS.md files for detailed context: - -### Core Implementation -- **[cpp/AGENTS.md](cpp/AGENTS.md)** - C++ implementation details and patterns -- **[cpp/daal/AGENTS.md](cpp/daal/AGENTS.md)** - Traditional DAAL interface context -- **[cpp/oneapi/AGENTS.md](cpp/oneapi/AGENTS.md)** - Modern oneAPI interface context - -### Build Systems & Development -- **[dev/AGENTS.md](dev/AGENTS.md)** - Development tools and build system context -- **[dev/bazel/AGENTS.md](dev/bazel/AGENTS.md)** - Bazel build system specifics - -### Documentation, Examples & Infrastructure -- **[docs/AGENTS.md](docs/AGENTS.md)** - Documentation structure and guidelines -- **[examples/AGENTS.md](examples/AGENTS.md)** - Example code patterns and usage -- **[deploy/AGENTS.md](deploy/AGENTS.md)** - Deployment and distribution context -- **[ci/AGENTS.md](ci/AGENTS.md)** - CI/CD infrastructure context +### Format and style (blocking: Azure `FormatterChecks`) -## πŸ“‹ Critical Development Rules - -### Code Style and Standards -- **ClangFormat**: Use project's `.clang-format` configuration -- **EditorConfig**: Follow `.editorconfig` rules -- **Modern C++**: Use C++14/17 features appropriately -- **STL**: Leverage standard library containers and algorithms -- **RAII**: Follow Resource Acquisition Is Initialization principles - -### Architecture Patterns -- **Interface Design**: Follow existing DAAL/oneAPI patterns -- **Memory Management**: Use smart pointers and RAII -- **Threading**: Use oneDAL threading layer, not direct primitives -- **CPU Features**: Implement CPU feature dispatching for optimizations - -### Testing and Validation -- **Build Tests**: All changes must pass build system validation -- **Examples**: Ensure examples build and run correctly -- **Documentation**: Update relevant documentation - -## πŸš€ Quick Start for AI Agents +```bash +pip install pre-commit && pre-commit install # one-time +pre-commit run --all-files +editorconfig-checker +``` -1. **Understand Context**: Read relevant AGENTS.md file for your task -2. **Follow Patterns**: Study existing code in similar areas -3. **Respect Standards**: Apply coding guidelines consistently -4. **Test Thoroughly**: Ensure changes work with build system +CI runs `.ci/scripts/clang-format.sh` with clang-format 20.1.8 (other versions format differently). It reformats files in place, so commit first: `CLANG_FORMAT_EXE=clang-format-20 .ci/scripts/clang-format.sh`. -### πŸ”„ Cross-Repository Considerations -- **scikit-learn-intelex integration impact** -- **API compatibility preservation** -- **Performance consistency maintenance** +### Tests (Bazel) -## πŸ” Key Files -- **[CONTRIBUTING.md](CONTRIBUTING.md)** - Contribution guidelines -- **[INSTALL.md](INSTALL.md)** - Build and installation instructions -- **[MODULE.bazel](MODULE.bazel)** - Bazel module configuration -- **[.clang-format](.clang-format)** - Code formatting rules +```bash +bazel test --config=host //cpp/oneapi/dal/algo/:tests # one algorithm, CPU only +bazel test --config=host //cpp/oneapi/dal:tests # oneAPI interface, CPU only +bazel test --config=dpc --device=gpu //cpp/oneapi/dal:tests # DPC++ on GPU +``` -## πŸ“š Additional Resources -- **API Documentation**: [oneDAL Developer Guide](https://uxlfoundation.github.io/oneDAL/) -- **Coding Guidelines**: [Detailed coding guide](https://uxlfoundation.github.io/oneDAL/contribution/coding_guide.html) -- **CPU Features**: [CPU feature dispatching guide](https://uxlfoundation.github.io/oneDAL/contribution/cpu_features.html) -- **Threading**: [Threading layer guide](https://uxlfoundation.github.io/oneDAL/contribution/threading.html) +Without `--config`, Bazel builds and runs all tests, including DPC++ ones that need the Intel DPC++ compiler. See `dev/bazel/README.md`. ---- +### Full build (Make) -**Note**: This file serves as the main entry point. For specific implementation details, refer to the relevant sub-AGENTS.md file in the appropriate directory. +```bash +make -f makefile daal oneapi_c PLAT=lnx32e -j$(nproc) +``` +See `INSTALL.md` for other platforms and build variants. + +### Where the checks live + +| Check | System | Config | +| --- | --- | --- | +| clang-format, editorconfig-checker | Azure DevOps | `.ci/pipeline/ci.yml` (`FormatterChecks`) | +| Make (GNU/MKL, LLVM/OpenBLAS rv64, VC, Intel), Bazel, release compare, sklearnex | Azure DevOps | `.ci/pipeline/ci.yml` | +| Make + DPC++ (icx), ABI check, Make GNU/MKL conda | GitHub Actions | `.github/workflows/ci.yml` | +| Windows (incl. arm64) | GitHub Actions | `.github/workflows/ci-win.yml` | +| aarch64 | GitHub Actions | `.github/workflows/ci-aarch64.yml` | +| License headers | GitHub Actions | `.github/workflows/skywalking-eyes.yml` | +| Bazel Linux/Windows | GitHub Actions (nightly) | `.github/workflows/nightly-test.yml` | + +Style is not gated in GitHub Actions: a green Actions run does not mean formatting passes. + +## Conventions +- C++17; no C++20/23 features. +- clang-format configs are per source tree (`cpp/daal/`, `cpp/oneapi/`, `examples/*/`, `samples/*/`, `dev/l0_tools/`); there is no root `.clang-format`. +- Parallelize through the oneDAL threading layer, never TBB directly. +- Optimized kernels dispatch on CPU features; see `docs/source/contribution/cpu_features.rst`. + +## Directory Guides + +Read the `AGENTS.md` nearest the files you change. When editing these files, every command, path and snippet must match the repository; delete what can't be verified rather than soften it. + +- [cpp/AGENTS.md](cpp/AGENTS.md): C++ implementation details and patterns +- [cpp/daal/AGENTS.md](cpp/daal/AGENTS.md): Traditional DAAL interface context +- [cpp/oneapi/AGENTS.md](cpp/oneapi/AGENTS.md): Modern oneAPI interface context +- [dev/AGENTS.md](dev/AGENTS.md): Development tools and build system context +- [dev/bazel/AGENTS.md](dev/bazel/AGENTS.md): Bazel build system specifics +- [dev/make/AGENTS.md](dev/make/AGENTS.md): Make build fragments +- [docs/AGENTS.md](docs/AGENTS.md): Sphinx docs layout and build rules +- [examples/AGENTS.md](examples/AGENTS.md): Example layout, `BUILD` registration and docs coupling +- [deploy/AGENTS.md](deploy/AGENTS.md): Deployment and distribution context +- [.ci/AGENTS.md](.ci/AGENTS.md): CI/CD infrastructure context +- [.github/AGENTS.md](.github/AGENTS.md): Workflow constraints (`nightly-build.yml`) + +## Further Reading +- `CONTRIBUTING.md`, `INSTALL.md` +- `docs/source/contribution/coding_guide.rst`, `docs/source/contribution/threading.rst` diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 76263377596..de91f2f2196 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -57,10 +57,10 @@ Public and private CIs are enabled for the repository. Your PR should pass all o **Prerequisites:** ClangFormat `20.1.8`. -Our repository contains [clang-format configurations](https://github.com/uxlfoundation/oneDAL/blob/main/.clang-format) that you should use on your code. To do this, run: +Our repository contains clang-format configurations, one per source tree (`cpp/daal`, `cpp/oneapi`, `examples/daal`, `examples/oneapi`, `samples/daal`, `samples/oneapi`, `dev/l0_tools`), that you should use on your code. To do this, run from the repository root: ``` -clang-format style=file +clang-format -style=file -i ``` Refer to [ClangFormat documentation](https://clang.llvm.org/docs/ClangFormat.html) for more information. diff --git a/cpp/AGENTS.md b/cpp/AGENTS.md index 2ccc1cf7d41..3a085d34889 100644 --- a/cpp/AGENTS.md +++ b/cpp/AGENTS.md @@ -1,159 +1,67 @@ +# AGENTS.md - C++ (cpp/) -# oneDAL C++ Implementation - AI Agents Context +## Purpose +The two C++ interfaces: DAAL (`cpp/daal/`, CPU) and oneAPI (`cpp/oneapi/`, CPU and SYCL GPU, primary development focus). Interface-specific rules are in `cpp/daal/AGENTS.md` and `cpp/oneapi/AGENTS.md`. -> **Purpose**: Context for AI agents working with oneDAL's dual C++ interface architecture. +## Interface Conventions -## πŸ—οΈ C++ Architecture Overview +| | DAAL (`cpp/daal/`) | oneAPI (`cpp/oneapi/`) | +| --- | --- | --- | +| Headers | `.h` with `#ifndef __FILE_NAME_H__` guards | `.hpp` with `#pragma once` | +| Namespaces | `daal::algorithms`, `daal::data_management`, `daal::services` | `oneapi::dal`, `oneapi::dal::`; `detail`, `backend`, `preview` sub-namespaces | +| Ownership | `daal::services::SharedPtr` | `std::unique_ptr`, `std::shared_ptr` | +| Errors | Kernels return `services::Status`; the interface layer converts it with `services::throwIfPossible()` | Exceptions from `cpp/oneapi/dal/exceptions.hpp` (`invalid_argument`, `domain_error`, `unimplemented`, ...) with messages from `cpp/oneapi/dal/detail/error_messages.hpp` | +| Kernels | Template bodies in `.i` files under `cpp/daal/src/`, templated on `CpuType cpu` | `cpp/oneapi/dal/algo//backend/{cpu,gpu}` | +| Naming | Classes `CamelCase`, functions and variables `lowerCamelCase` | `snake_case` throughout; private members end in `_` | -oneDAL provides **two distinct C++ interfaces**: +## Rules for Changes +- Never mix the two interfaces in one file. C++17 only; no C++20/23 features. +- Parallelize through the threading layer (`cpp/oneapi/dal/detail/threading.hpp`, `cpp/daal/src/threading/threading.h`), never TBB directly. +- Keep CPU dispatch intact: optimized code is templated on `CpuType` and selected at runtime. +- Removing anything from a public header needs a deprecation period and an entry in `docs/source/deprecation.rst`. Expected symbol removals go in `.github/.abignore`, which the ABI check reads. +- Don't change copy semantics (shallow vs deep) as a side effect of another change. +- Export macros (`DAAL_EXPORT`, `ONEDAL_EXPORT`) and symbol visibility must stay the same between the Bazel and Make builds. -### 1. Traditional DAAL Interface (`cpp/daal/`) -- **Target**: CPU-focused with SIMD optimizations, backward compatible -- **Style**: Traditional C++ with `daal::services::SharedPtr`, `services::Status` return codes, highly-nested namespaces -- **Headers**: `.h` files with `#ifndef` guards, `daal::algorithms` namespaces +## Review Checklist +- The interface contract in the table above is preserved. +- Ownership, lifetime, error propagation, type safety and bounds handling, where the change touches them. +- CPU dispatch is preserved and no C++20/23 features are introduced. +- Public API or ABI changes account for compatibility. -### 2. Modern oneAPI Interface (`cpp/oneapi/`) -- **Target**: CPU + GPU (SYCL) + distributed computing (primary development focus) -- **Style**: Modern C++17 with STL smart pointers, exceptions, RAII -- **Headers**: `.hpp` files with `#pragma once`, `oneapi::dal` namespaces +## CPU Dispatch -## πŸ”§ Development Standards +DAAL containers and kernels are templated on `CpuType cpu` (values per architecture in `cpp/daal/src/services/cpu_type.h`). oneAPI operations go through a dispatcher templated on the context (from `cpp/oneapi/dal/algo/kmeans/detail/train_ops.hpp`): -- **C++ Standard**: C++17 (no C++20/23 features for compatibility) -- **Architecture**: x86_64, ARM64 (SVE), RISC-V 64-bit with CPU-specific optimizations -- **Build**: Bazel with `dal.bzl`/`daal.bzl` rules, MKL/OpenBLAS backend selection - -## 🎭 Key Template Patterns - -### Template Specialization & CPU Dispatch ```cpp -// DAAL - Multi-dimensional specialization for CPU optimization -template -class BatchContainer : public daal::algorithms::AnalysisContainerIface { - virtual services::Status compute() override; -}; - -// oneAPI - Type-safe dispatching with perfect forwarding -template +template struct train_ops_dispatcher { - train_result operator()(const Context&, const descriptor_base&, - const train_parameters&, const train_input&) const; -}; -``` - -## πŸ›οΈ Core Design Patterns - -### Memory Management -```cpp -// DAAL - Custom smart pointers -daal::services::SharedPtr data_; -// DAAL - Custom objects collections -daal::services::Collection collection(5); - -// oneAPI - STL smart pointers with RAII -std::unique_ptr ptr_ = std::make_unique(); -std::shared_ptr table_ = std::make_shared
(); -// oneAPI - STL containers -std::vector vec(5); -``` - -### Error Handling -- **DAAL**: `services::Status` return codes with `throwIfPossible()` conversion -- **oneAPI**: STL exceptions (`std::invalid_argument`, `std::domain_error`) - -## ⚑ Platform Optimizations - -### Multi-Architecture CPU Support -```cpp -// Compile-time CPU optimization selection -#if defined(TARGET_X86_64) - enum CpuType { sse2 = 0, avx2 = 4, avx512 = 6 }; -#elif defined(TARGET_ARM) - enum CpuType { sve = 0 }; // ARM SVE -#elif defined(TARGET_RISCV64) - enum CpuType { rv64 = 0 }; // RISC-V 64-bit -#endif - -// SIMD optimization -#define PRAGMA_FORCE_SIMD _Pragma("ivdep") // Intel compiler vectorization -``` - -### Runtime CPU Feature Detection -```cpp -enum class cpu_feature : uint64_t { - unknown = 0ULL, - sstep = 1ULL << 0, // Intel(R) SpeedStep - tb = 1ULL << 1, // Intel(R) Turbo Boost - avx512_bf16 = 1ULL << 2, // AVX512 bfloat16 - avx512_vnni = 1ULL << 3, // AVX512 VNNI - tb3 = 1ULL << 4 // Intel(R) Turbo Boost Max 3.0 + train_result operator()(const Context&, + const descriptor_base&, + const train_input&) const; }; ``` -## 🌐 Dependencies & Namespaces - -### Key Dependencies -- **Math**: Intel MKL (primary), OpenBLAS (reference) -- **Threading**: Intel TBB for task-based parallelism -- **GPU**: Intel SYCL for heterogeneous computing -- **Distributed**: MPI via `oneapi::dal::preview::spmd` - -### Namespace Structure - -- **oneAPI**: - - `oneapi::dal`: Top level oneDAL namespace. - - `oneapi::dal::{...}::backend`: APIs for internal oneDAL use, not visible to the users. - - `oneapi::dal::{...}::detail`: APIs that are visible to the users, but might be a subject to change. Those APIs do not follow ABI compatibility requirements. - - `oneapi::dal::{...}::preview`: Functionality added into a product for users to try it out. Also might be a subject to change or removal, and does not follow the ABI compatibility requirements. - - `oneapi::dal::`, for example `oneapi::dal::kmeans`: Namespace of a respective algorithm. -- **DAAL**: - - `daal`: Top level DAAL namespace. - - `daal::algorithms`: Algorithms and related classes like `Parameter`, `Input`, `Result`. - - `daal::data_management`: Numeric tables and data sources. - - `daal::services`: Error handling, `SharedPtr`, `Collection`. - - `daal::{}::internal`: APIs for internal DAAL use, not visible to the users. - -## πŸ“š Algorithm Interface Patterns - -### DAAL Pattern -```cpp -// Traditional algorithm lifecycle with explicit memory management -using rr_train = daal::ridge_regression::training; -rr_train::Batch training(2.0 /* ridge coefficient */); -training.input.set(rr_train::data, data_table); -training.input.set(rr_train::dependentVariables, dependents); -training.compute(); -auto result = training->getResult(); -auto model = result->get(rr_train::model); -``` - -### oneAPI Pattern -```cpp -// Modern fluent interface with automatic resource management -auto desc = dal::kmeans::descriptor() - .set_cluster_count(10) - .set_max_iteration_count(100); -auto train_result = dal::train(desc, data_table); -auto infer_result = infer(desc, train_result.get_model(), test_data); -``` - -## 🎯 Critical Rules +Runtime CPU feature flags are in `cpp/oneapi/dal/detail/cpu.hpp`; see also `docs/source/contribution/cpu_features.rst`. -### Interface Separation -- **NEVER mix DAAL and oneAPI patterns** in same file -- **DAAL**: `.h` headers, `#ifndef` guards, `daal::services::SharedPtr` -- **oneAPI**: `.hpp` headers, `#pragma once`, `std::unique_ptr/shared_ptr` +## Dependencies +- Math: Intel MKL (default), OpenBLAS (`BACKEND_CONFIG=ref`) +- Threading: oneTBB, only through the threading layer +- GPU: SYCL; distributed: `oneapi::dal::preview::spmd` -### Memory & Error Handling -- **DAAL**: Custom smart pointers, `services::Status` codes -- **oneAPI**: STL RAII, C++ exceptions +## Namespace Structure -### Performance -- **CPU Dispatch**: Templates specialized by `CpuType` for optimal SIMD -- **Threading**: TBB integration for task-based parallelism -- **GPU**: SYCL for heterogeneous computing +- **oneAPI** (`oneapi::dal`): + - `backend`: internal, not visible to users. + - `detail`: visible to users but may change; not ABI-stable. + - `preview`: experimental functionality; may change or be removed, not ABI-stable. + - ``, for example `oneapi::dal::kmeans`. +- **DAAL** (`daal`): + - `algorithms`: algorithms and their `Parameter`, `Input`, `Result` classes. + - `data_management`: numeric tables and data sources. + - `services`: error handling, `SharedPtr`, `Collection`. + - `{...}::internal`: internal, not visible to users. -## πŸ“– Further Reading +## Further Reading - **[AGENTS.md](/AGENTS.md)** - Repository overview and context - **[cpp/daal/AGENTS.md](/cpp/daal/AGENTS.md)** - DAAL interface specifics - **[cpp/oneapi/AGENTS.md](/cpp/oneapi/AGENTS.md)** - oneAPI interface specifics diff --git a/cpp/daal/AGENTS.md b/cpp/daal/AGENTS.md index b4b53d308d3..014352e0d17 100644 --- a/cpp/daal/AGENTS.md +++ b/cpp/daal/AGENTS.md @@ -7,56 +7,25 @@ - **Headers**: `.h` files with `#ifndef __FILE_H__` guards - **Memory**: `daal::services::SharedPtr` custom smart pointers - **Errors**: `services::Status` return codes with `throwIfPossible()` -- **Threading**: TBB-based with CPU-specific kernels +- **Threading**: the DAAL threading layer (`src/threading/threading.h`), with CPU-specific kernels - **Optimization**: Multi-architecture dispatch (SSE2, AVX2, AVX-512, ARM SVE, RISC-V) +- **Naming**: classes in `CamelCase` (`BatchContainer`, `HomogenNumericTable`); functions and variables in `lowerCamelCase` (`getResult()`, `nClusters`, `nRowsTotal`); the floating-point template parameter is `algorithmFPType` ## πŸš€ Essential Commands ```bash # Build DAAL interface -`make daal_c` +make daal_c # Platform-specific builds -`make daal_c PLAT=lnx32e COMPILER=icx` +make daal_c PLAT=lnx32e COMPILER=icx # CPU target selection -`make daal_c REQCPU="sse2 avx2 avx512"` +make daal_c REQCPU="sse2 avx2 avx512" ``` ## πŸ› οΈ Core Patterns -### Algorithm Usage Pattern -```cpp -#include "algorithms/kmeans/kmeans_batch.h" -#include "data_management/data/homogen_numeric_table.h" - -// Traditional algorithm lifecycle -kmeans::Batch training(clusters, iterations); - -training.input.set(kmeans::data, data_table); -services::Status status = training.compute(); -if (!status) { - throwIfPossible(status); -} - -auto result = training.getResult(); -auto model = result->get(kmeans::model); -``` - -### Memory Management -```cpp -// Custom smart pointer system -daal::services::SharedPtr data_; -services::SharedPtr errors_; - -// Usage pattern -using dm = daal::data_management -services::Status status; -services::SharedPtr> table(dm::HomogenNumericTable::create(data_array, rows, cols, &status)); -// Shorter variant -dm::NumericTablePtr table(dm::HomogenNumericTable::create(data_array, rows, cols, &status)); -``` - ### Error Handling DAAL has a mixed approach to error handling: @@ -66,77 +35,58 @@ DAAL has a mixed approach to error handling: If `DAAL_NOTHROW_EXCEPTIONS` macro is defined during DAAL build `throwIfPossible()` doesn't perform status code to exception conversion. ```cpp -// Hierarchical error status system services::Status compute() { services::Status status; - - // Implementation - if (/* Error condition */) { - // Add error code ID to the status - status.add(services::SomeErrorCodeID); + if (/* allocation failed */) { + status.add(services::ErrorMemoryAllocationFailed); return status; } - // Implementation - // Computations are successful if status holds no errors on exit return status; } ``` ### CPU-Specific Kernel Dispatch + +Containers and kernels are templated on `CpuType cpu`; the values per architecture are in `src/services/cpu_type.h`. + ```cpp -// Multi-architecture template specialization template class BatchContainer : public daal::algorithms::AnalysisContainerIface { public: virtual services::Status compute() override; }; - -// CPU type enumeration -enum CpuType { -#if defined(TARGET_X86_64) - sse2 = 0, avx2 = 4, avx512 = 6 -#elif defined(TARGET_ARM) - sve = 0 // ARM Scalable Vector Extension -#elif defined(TARGET_RISCV64) - rv64 = 0 // RISC-V 64-bit -#endif -}; ``` ### Data Management -DAAL numeric tables do not own the data they work with, they can be viewed as wrappers over the user-provided data. +DAAL numeric tables do not own the data they work with, they can be viewed as wrappers over the user-provided data. The user allocates and frees that data. ```cpp -// NumericTable as primary data structure -float * data_array = new float[rows * cols]; -services::Status status; -auto table = HomogenNumericTable::create(data_array, rows, cols, &status); -// User is responsible for allocation and deletion of the data assocoated with numeric tables -delete[] data_array; - // CSV data source allows to produce a numeric table from CSV file -FileDataSource fileDataSource(datafile, - DataSource::doAllocateNumericTable, - DataSource::doDictionaryFromContext); +FileDataSource dataSource(datafile, + DataSource::doAllocateNumericTable, + DataSource::doDictionaryFromContext); dataSource.loadDataBlock(); // dataSource.loadDataBlock(10); loads next 10 rows from the file auto table = dataSource.getNumericTable(); ``` -## 🎯 Critical DAAL Rules +## πŸ“ Rules for Changes -- **Memory**: Use `daal::services::SharedPtr`, never raw pointers for ownership -- **Error Handling**: Check `services::Status` return codes or catch the exceptions -- **Headers**: Traditional `#ifndef` guards -- **Templates**: CPU-specific specialization for performance optimization -- **Interface**: Never mix DAAL and oneAPI patterns in same file +- Kernel bodies live in `.i` files, which `rg --type cpp` skips; search them with `rg -g '*.i'`. `*_fpt_cpu.cpp`, `*_fpt_dispatcher.cpp` and `*_fpt.cpp` only instantiate templates. +- Kernel templates take `daal::internal::CpuType cpu` and `algorithmFPType` as template parameters. Code compiled without the `cpu` parameter is not dispatched and can execute instructions the running CPU lacks. +- Allocate scratch memory with `TArray`, `TArrayCalloc`, `TArrayScalable` or `TArrayScalableCalloc` (`src/services/service_arrays.h`), not raw `new`. Parallelize through the threading layer (`src/threading/threading.h`, e.g. `daal::threader_for`), never TBB directly. +- For accuracy, accumulate each row into a zero-initialized local and add it back into the target, rather than accumulating in place. Clip results that must be non-negative. +- Guard every division by a row, observation or rank count against zero; in distributed runs a rank can have no rows. +- Don't wrap a temporary in a non-owning array. +- Don't use `reduction(- : x)` with OpenMP SIMD; reduce with `+` over negated terms. +- No magic numbers. Use a named constant and say where its value comes from. ## πŸ”— References - **[AGENTS.md](../../AGENTS.md)** - Repository overview - **[cpp/oneapi/AGENTS.md](../oneapi/AGENTS.md)** - Modern oneAPI interface -- **[.github/instructions/cpp-coding-guidelines.instructions.md](../../.github/instructions/cpp-coding-guidelines.instructions.md)** - Detailed C++ coding guidelines +- **[cpp/AGENTS.md](../AGENTS.md)** - Interface conventions shared by both interfaces **Note**: This interface is maintained for backward compatibility. For new development, consider the modern oneAPI interface. diff --git a/cpp/oneapi/AGENTS.md b/cpp/oneapi/AGENTS.md index e23fe008442..ea10cc55f0c 100644 --- a/cpp/oneapi/AGENTS.md +++ b/cpp/oneapi/AGENTS.md @@ -6,22 +6,24 @@ - **Headers**: `.hpp` files with `#pragma once` - **Memory**: STL RAII (`std::unique_ptr`, `std::shared_ptr`) -- **Errors**: C++ exceptions (`std::invalid_argument`, `std::domain_error`) +- **Errors**: exceptions from `cpp/oneapi/dal/exceptions.hpp` (`dal::invalid_argument`, `dal::domain_error`, ...) with messages from `detail/error_messages.hpp`, e.g. `throw invalid_argument{ dal::detail::error_messages::queues_in_different_contexts() };` (`backend/common.hpp`) +- **Naming**: `snake_case` for types, functions, variables and constants (`train_ops`, `get_data()`, `row_count`); private members take a trailing underscore (`store_`, `comm_`) - **GPU**: Intel SYCL with USM for CPU/GPU operations - **Namespace**: `oneapi::dal::v1` (stable), `preview` (experimental) +- **Interface**: Never mix DAAL and oneAPI patterns in same file ## πŸš€ Essential Commands ### Bazel build and test ```bash # Build oneAPI interface -`bazel build //cpp/oneapi/dal:core` +bazel build //cpp/oneapi/dal:core -# Run CPU tests -`bazel test //cpp/oneapi/dal:tests` +# Run CPU (host) tests +bazel test --config=host //cpp/oneapi/dal:tests -# Run GPU tests -`bazel test --config=dpc //cpp/oneapi/dal:tests` +# Run DPC++ tests on GPU +bazel test --config=dpc --device=gpu //cpp/oneapi/dal:tests ``` ### Make and CMake build @@ -33,11 +35,11 @@ make onedal_c # Build oneAPI interface with CPU and GPU support make onedal_dpc -# Build dynamic link version of examples -export CC=icx -export CXX=icpx -cmake -G "Unix Makefiles" -DONEDAL_LINK=dynamic -make +# Build the examples against the release tree (there is no root CMakeLists.txt) +source __release_lnx/daal/latest/env/vars.sh +cd examples/oneapi/cpp +cmake -B build -S . -DONEDAL_LINK=dynamic +cmake --build build --parallel ``` ## πŸ› οΈ Core Patterns @@ -58,9 +60,9 @@ auto result = train(desc, data); sycl::queue gpu_q(sycl::gpu_selector_v); auto gpu_result = train(gpu_q, desc, data); -// Distributed execution -auto comm = spmd::make_communicator(); -auto dist_result = train(comm, desc, data); +// Distributed execution (samples/oneapi/cpp/ccl) +auto comm = preview::spmd::make_communicator(); +auto dist_result = preview::train(comm, desc, data); ``` ### Data Tables @@ -78,12 +80,12 @@ auto table = homogen_table::wrap(data, rows, cols); // Access data auto accessor = row_accessor(table); auto subset = accessor.pull({0, 10}); // Rows 0-9 -const float * data_block subset.get_data(); +const float * data_block = subset.get_data(); // Pull memory with device access -auto subset_gpu = accessor.pull({0, 10}, sycl::usm::alloc::device); +auto subset_gpu = accessor.pull(gpu_q, {0, 10}, sycl::usm::alloc::device); // SYCL USM pointer -const float * gpu_data_block subset_gpu.get_data(); +const float * gpu_data_block = subset_gpu.get_data(); ``` ### Exception Handling @@ -97,20 +99,6 @@ try { } ``` -### Memory Management (RAII) -```cpp -class DataProcessor { -private: - std::unique_ptr buffer_; - std::shared_ptr table_; - -public: - DataProcessor(size_t size) - : buffer_(std::make_unique(size)) - , table_(std::make_shared(buffer_.get(), rows, cols)) {} -}; -``` - ### SYCL GPU Kernels ```cpp template @@ -119,9 +107,8 @@ sycl::event gpu_compute(sycl::queue& q, std::int64_t n, const std::vector& deps) { return q.submit([&](sycl::handler& cgh) { + cgh.depends_on(deps); cgh.parallel_for(sycl::nd_range<1>(n, 256), [=](sycl::nd_item<1> item) { - // Dependencies handling - cgh.depends_on(deps); const auto idx = item.get_global_id(0); // GPU computation }); @@ -129,16 +116,13 @@ sycl::event gpu_compute(sycl::queue& q, } ``` -## 🎯 Critical Rules +## πŸ“ Rules for Changes -- **Memory**: Always use STL smart pointers, never raw pointers for ownership -- **Headers**: Use `.hpp` with `#pragma once`, `oneapi::dal` namespace -- **GPU**: SYCL integration with USM for zero-copy operations -- **Type Safety**: Template metaprogramming with compile-time dispatch -- **Interface**: Never mix DAAL and oneAPI patterns in same file +- Public types are declared in a versioned namespace and re-exported: `namespace v1 { struct x {}; }` followed by `using v1::x;`. The namespace is not `inline`, so a type without the `using` line is unreachable as `oneapi::dal::...::x`. Reference: `cpp/oneapi/dal/algo/pca/common.hpp`. +- Use `nullptr`, never `0` or `NULL`. Write `override` directly, not through a macro. No `using namespace` in headers. ## πŸ”— References - **[AGENTS.md](../../AGENTS.md)** - Repository overview - **[cpp/daal/AGENTS.md](../daal/AGENTS.md)** - Traditional DAAL interface -- **[.github/instructions/cpp-coding-guidelines.instructions.md](../../.github/instructions/cpp-coding-guidelines.instructions.md)** - Detailed C++ standards +- **[cpp/AGENTS.md](../AGENTS.md)** - Interface conventions shared by both interfaces diff --git a/deploy/AGENTS.md b/deploy/AGENTS.md index 746aba57690..dc1373b95ca 100644 --- a/deploy/AGENTS.md +++ b/deploy/AGENTS.md @@ -121,4 +121,4 @@ deploy/local/vars_win.bat ## πŸ“– Further Reading - **[AGENTS.md](../AGENTS.md)** - Main repository context - **[dev/AGENTS.md](../dev/AGENTS.md)** - Development tools context -- **[ci/AGENTS.md](../ci/AGENTS.md)** - CI infrastructure context +- **[ci/AGENTS.md](../.ci/AGENTS.md)** - CI infrastructure context diff --git a/dev/AGENTS.md b/dev/AGENTS.md index c03f8b31c82..c29833a485b 100644 --- a/dev/AGENTS.md +++ b/dev/AGENTS.md @@ -1,127 +1,49 @@ +# AGENTS.md - Development Tools and Build Systems (dev/) -# Development Tools and Build Systems - AI Agents Context +## Purpose +Build systems and developer tooling. Make (root `makefile`) produces releases; Bazel builds and runs tests; CMake only builds the examples against an installed release. -> **Purpose**: Context for AI agents working with oneDAL's sophisticated build systems, Bazel rules, and development tools. +## Layout +- `bazel/`: Bazel macros and dependency rules; see `dev/bazel/AGENTS.md` +- `make/`: fragments included by the root `makefile`; see `dev/make/AGENTS.md` +- `release_tests/`: checks run against a built release tree (`compare_release_trees.py`, package metadata) +- `l0_tools/`: Level Zero GPU utilities +- `docker/`: development container (`onedal-dev.Dockerfile`) -## πŸ—οΈ Build System Architecture +## Rules for Changes +- A build change usually has to land in both Make and Bazel. Library versions, exports and symbol visibility must match between them. +- BUILD files use the `dal_module`, `dal_test_suite` and `daal_module` macros, never bare `cc_library` / `cc_test`. +- The Bazel version is pinned in `.bazelversion`; don't hardcode it elsewhere. +- Toolchain and dependency setup (oneAPI compilers, oneMKL, oneTBB) is in `INSTALL.md`; don't restate it here. -oneDAL uses **dual build systems** optimized for different workflows: +## Make -### Build Systems -- **Bazel**: Modern development build system with sophisticated C++ template instantiation -- **Make**: Production build system with platform-specific optimizations - -### Development Tools -- **Level Zero (L0) Tools**: GPU development utilities (`dev/l0_tools/`) -- **Docker**: Containerized development environment (`dev/docker/`) -- **Dependency Management**: Automated TBB, MKL, SYCL integration - -## πŸ“ Structure -``` -dev/ -β”œβ”€β”€ bazel/ # Bazel build system with custom rules -β”‚ β”œβ”€β”€ dal.bzl # oneAPI interface build rules -β”‚ β”œβ”€β”€ daal.bzl # DAAL interface build rules -β”‚ β”œβ”€β”€ cc/ # C++ compilation rules -β”‚ β”œβ”€β”€ deps/ # External dependency management -β”‚ └── config/ # Build configuration -β”œβ”€β”€ make/ # Make-based build system -β”‚ β”œβ”€β”€ common.mk # Common make patterns -β”‚ β”œβ”€β”€ deps.mk # Dependency resolution -β”‚ └── compiler_definitions/ # Compiler-specific settings -β”œβ”€β”€ l0_tools/ # Level Zero GPU development tools -└── docker/ # Development containers +```bash +make -f makefile daal oneapi_c PLAT=lnx32e -j$(nproc) ``` -## πŸ”§ Bazel Build System - -### Key Characteristics -- **Development Build System**: Used for development and CI/CD -- **Dependency Management**: Automatic dependency resolution -- **Multi-platform**: Supports Linux -- **Incremental Builds**: Fast incremental compilation +- Targets: `daal`, `daal_c`, `oneapi` (`oneapi_c` + `oneapi_dpc`), `onedal`, `onedal_c`, `onedal_dpc`. `make -f makefile help` lists all targets and variables. +- `PLAT`: `lnx32e`, `win32e`, `mac32e`, `lnxarm`, `winarm`, `lnxriscv64`. +- `COMPILER`: `icx` (x86-64 default), `gnu`, `clang`, `vc`; the allowed set per platform is in `make/function_definitions/`. +- `REQCPU`: subset of `sse2 avx2 avx512` on x86-64, `sve` on ARM, `rv64` on RISC-V. +- `BACKEND_CONFIG`: `mkl` (default on x86-64) or `ref` (OpenBLAS; default on ARM and RISC-V). -### Configuration Files -- **[MODULE.bazel](MODULE.bazel)** - Root module configuration -- **[.bazelrc](.bazelrc)** - Bazel configuration options -- **[dev/bazel/BUILD](bazel/BUILD)** - Root build configuration +## Bazel -### Common Commands ```bash -# Build entire project -bazel build //... - -# Run tests -bazel test //... - -# Clean build -bazel clean --expunge +bazel test --config=host //cpp/oneapi/dal/algo/pca:tests # one algorithm, CPU only +bazel test --config=dpc --device=gpu //cpp/oneapi/dal:tests # DPC++ on GPU ``` +See `dev/bazel/README.md` and `dev/bazel/AGENTS.md`. -## πŸ”§ Make Build System - -### Key Characteristics -- **Production Build System**: Main build system for production builds -- **Platform Specific**: Different configurations per platform -- **Dependency Management**: Manual dependency specification - -### Configuration Files -- **[makefile](makefile)** - Root makefile -- **[dev/make/common.mk](make/common.mk)** - Common make rules -- **[dev/make/deps.mk](make/deps.mk)** - Dependency management - - +## CMake -## πŸ” Build System Patterns +There is no root `CMakeLists.txt`. Build the examples against a release tree: -### Bazel Pattern -```python -cc_library( - name = "library_name", - srcs = glob(["src/**/*.cpp"]), - hdrs = glob(["include/**/*.h"]), - deps = ["//path/to:dependency"], - visibility = ["//visibility:public"], -) - -cc_test( - name = "library_test", - srcs = glob(["test/**/*.cpp"]), - deps = [":library_name", "//dev/bazel/deps:gtest"], -) -``` - -### Make Pattern -```makefile -LIBRARY_OBJS = $(patsubst %.cpp,%.o,$(wildcard src/*.cpp)) - -library_name: $(LIBRARY_OBJS) - $(CXX) $(LDFLAGS) -o $@ $^ - -%.o: %.cpp - $(CXX) $(CXXFLAGS) -c $< -o $@ +```bash +source __release_lnx/daal/latest/env/vars.sh +cd examples/oneapi/cpp +cmake -B build -S . -DONEDAL_LINK=dynamic +cmake --build build --parallel ``` - -## 🚫 Common Pitfalls -- **Build System Mixing**: Don't mix build systems, use consistent approach per project -- **Dependency Management**: Don't hardcode paths, use proper dependency tools -- **Configuration**: Don't assume defaults, test on target platforms - -## πŸ§ͺ Testing and Validation -- **Build Validation**: Ensure all build systems work -- **Dependencies**: Validate dependency resolution -- **Platforms**: Test on supported platforms - -## πŸ”§ Required Tools -- **Bazel**: 5.0+ for Bazel builds -- **Make**: GNU Make 3.81+ for Make builds -- **Compilers**: GCC 7+, Clang 6+, MSVC 2017+ -- **Intel oneAPI**: For SYCL development -- **Intel MKL**: For optimized math operations -- **Intel TBB**: For threading support - -## πŸ“– Further Reading -- **[dev/bazel/AGENTS.md](bazel/AGENTS.md)** - Bazel build system details -- **[cpp/AGENTS.md](../cpp/AGENTS.md)** - C++ implementation context -- **[docs/AGENTS.md](../docs/AGENTS.md)** - Documentation guidelines diff --git a/dev/bazel/AGENTS.md b/dev/bazel/AGENTS.md index 98b504f2252..eaaf374d41c 100644 --- a/dev/bazel/AGENTS.md +++ b/dev/bazel/AGENTS.md @@ -10,7 +10,7 @@ Bazel is the **development and testing build system** for oneDAL, providing fast ### Key Characteristics - **Development Build System**: Used for development and CI/CD - **Dependency Management**: Automatic dependency resolution -- **Multi-platform**: Linux, Windows, macOS support +- **Multi-platform**: Linux and Windows toolchains - **Incremental Builds**: Fast incremental compilation - **Hermetic Builds**: Reproducible build environments @@ -25,134 +25,88 @@ dev/bazel/ ``` ## 🎯 Configuration Files -- **[MODULE.bazel](MODULE.bazel)** - Root module configuration -- **[.bazelrc](.bazelrc)** - Bazel configuration options +- **[MODULE.bazel](../../MODULE.bazel)** - Root module configuration +- **[.bazelrc](../../.bazelrc)** - Bazel configuration options - **[dev/bazel/BUILD](BUILD)** - Root build configuration - **[dev/bazel/cc/BUILD](cc/BUILD)** - C++ build configuration - **[dev/bazel/deps/BUILD](deps/BUILD)** - Dependency management ### Module Configuration -```python -# MODULE.bazel -module( - name = "onedal", - version = "1.0.0", -) - -bazel_dep(name = "rules_cc", version = "0.2.18") -bazel_dep(name = "catch2", version = "3.9.1") -``` +`MODULE.bazel` declares a handful of `bazel_dep`s (`platforms`, `bazel_skylib`, `rules_cc`, `rules_shell`, `fmt`). Everything else is a repository rule: catch2 is an `http_archive` with `build_file = "//dev/bazel/deps:catch2.BUILD"`, and MKL, TBB, OpenBLAS, MPI, CCL, DPL and OpenCL come from the `dev/bazel/deps/*.bzl` rules. Read `MODULE.bazel` for current versions rather than copying them from here. ## πŸ”§ Build Rules and Patterns -### C++ Library Target +oneDAL does not use bare `cc_library` / `cc_test`. All modules go through the macros in `@onedal//dev/bazel:dal.bzl` (`dal_module`, `dal_test_suite`) and `@onedal//dev/bazel:daal.bzl` (`daal_module`). Compiler flags, CPU dispatch and threading are toolchain-owned (`dev/bazel/flags.bzl`); never hand-write `copts`. + ```python -cc_library( - name = "library_name", - srcs = glob(["src/**/*.cpp"]), - hdrs = glob(["include/**/*.h"]), - deps = [ - "//path/to:dependency", - "//dev/bazel/deps:external_lib", - ], - visibility = ["//visibility:public"], - copts = ["-std=c++17", "-O3"], +load("@onedal//dev/bazel:dal.bzl", "dal_module", "dal_test_suite") + +package(default_visibility = ["//visibility:public"]) + +dal_module( + name = "core", + auto = True, # globs sources by convention, excludes test/ + dal_deps = ["@onedal//cpp/oneapi/dal:core"], + extra_deps = ["@onedal//cpp/daal/src/algorithms/pca:kernel"], ) -``` -### C++ Test Target -```python -cc_test( - name = "library_test", - srcs = glob(["test/**/*.cpp"]), - deps = [ - ":library_name", - "//dev/bazel/deps:catch2", - ], - copts = ["-std=c++17", "-g"], +dal_module( + name = "pca", + dal_deps = [":core"], +) + +dal_test_suite( + name = "interface_tests", + srcs = glob(["test/*.cpp"]), + hdrs = glob(["test/*.hpp"]), + dal_deps = [":pca"], + framework = "catch2", # injects the catch2 main; don't add a catch2 dep by hand +) + +dal_test_suite( + name = "tests", # aggregate target CI invokes + tests = [":interface_tests"], ) ``` +Reference implementation: `cpp/oneapi/dal/algo/pca/BUILD`. + ## πŸ”§ Common Commands ```bash -# Build entire project -bazel build //... - -# Build specific target -bazel build //cpp/daal:daal +# Build the oneAPI core +bazel build //cpp/oneapi/dal:core -# Run tests -bazel test //... +# Test one algorithm, CPU only +bazel test --config=host //cpp/oneapi/dal/algo/pca:tests -# Clean build -bazel clean --expunge +# Build the release tree +bazel build //:release ``` -## πŸ”§ Dependency Management +Always scope targets, and pass `--config` to `bazel test` and `bazel run`; with it unset, tests include DPC++ targets that need the Intel DPC++ compiler. `dev/bazel/README.md` lists every config. -### External Dependencies -```python -# dev/bazel/deps/BUILD -cc_library( - name = "tbb", - srcs = glob(["tbb/src/**/*.cpp"]), - hdrs = glob(["tbb/include/**/*.h"]), - visibility = ["//visibility:public"], -) +## πŸ”§ Dependency Management -cc_library( - name = "mkl", - srcs = glob(["mkl/lib/**/*.so"]), - hdrs = glob(["mkl/include/**/*.h"]), - visibility = ["//visibility:public"], -) -``` +External libraries are referenced by their repository labels, e.g. `@mkl//:mkl_core`, `@tbb//:tbb`, `@openblas//:openblas`, `@mpi//:mpi`. `dev/bazel/deps/BUILD` declares no targets; the `*.tpl.BUILD` files next to it are the BUILD templates for those repositories. ## 🎯 Development Guidelines -### Build Target Naming -- **Libraries**: Use descriptive names (e.g., `daal_core`, `oneapi_dal`) -- **Tests**: Append `_test` suffix (e.g., `daal_core_test`) -- **Examples**: Use descriptive names (e.g., `kmeans_example`) - ### Dependencies -- **Internal**: Use relative paths (e.g., `//cpp/daal:daal`) -- **External**: Use dependency rules (e.g., `//dev/bazel/deps:tbb`) +- **Internal**: Full `@onedal//` labels in `dal_deps` (e.g., `@onedal//cpp/oneapi/dal:core`) +- **DAAL kernels**: `extra_deps` (e.g., `@onedal//cpp/daal/src/algorithms/pca:kernel`) - **Visibility**: Set appropriate visibility levels -## πŸ” Common Patterns - -### Conditional Compilation -```python -cc_library( - name = "platform_specific", - srcs = select({ - "//dev/bazel/config:linux": ["src/linux.cpp"], - "//dev/bazel/config:windows": ["src/windows.cpp"], - "//conditions:default": ["src/default.cpp"], - }), - deps = [":common"], -) -``` - -### Feature Detection -```python -cc_library( - name = "feature_detection", - srcs = ["src/feature_detection.cpp"], - copts = select({ - "//dev/bazel/config:avx512": ["-mavx512f"], - "//dev/bazel/config:avx2": ["-mavx2"], - "//conditions:default": [], - }), -) -``` +## πŸ“ Rules for Changes -## 🚫 Common Pitfalls -- **Build Configuration**: Don't hardcode platform-specific paths -- **Dependencies**: Don't mix different dependency management approaches -- **Toolchains**: Don't assume toolchain availability, test on target platforms +- `//conditions:default` in a `select()` means every platform not listed, macOS included, not just Linux. Put Linux-only flags under `@platforms//os:linux`. +- Don't add `allow_empty = True` to a glob that must match; it hides packaging mistakes. The optional globs in `dal.bzl`, `daal.bzl` and `deps/*.tpl.BUILD` need it. +- Don't hardcode the workspace name in test paths; use `${TEST_WORKSPACE}`. +- No `use_default_shell_env = True`; it breaks hermeticity. +- Don't write globs that match several `.so` variants of one library; they produce duplicate link inputs. +- Remove unused `load()` symbols. +- Don't hardcode platform-specific paths; reference external libraries by their repository labels. +- Library binary versions are `MAJORBINARY` / `MINORBINARY` in `makefile.ver`, mirrored by `_BINARY_MAJOR` / `_BINARY_MINOR` in `dev/bazel/repos.bzl`. Change both together; don't hardcode them anywhere else. ## πŸ§ͺ Testing and Validation - **Build Validation**: Ensure builds work on all supported platforms @@ -160,9 +114,7 @@ cc_library( - **CI/CD Integration**: Primary build system for CI/CD ## πŸ”§ Required Tools -- **Bazel**: 5.0+ for modern Bazel features -- **Python**: 3.7+ for build rule development -- **Compilers**: GCC 7+, Clang 6+, MSVC 2017+ +- **Bazel**: version pinned in `.bazelversion`; `.ci/env/bazelisk.sh` installs a matching launcher ## πŸ“– Further Reading - **[dev/AGENTS.md](../AGENTS.md)** - Development tools context diff --git a/dev/make/AGENTS.md b/dev/make/AGENTS.md new file mode 100644 index 00000000000..9bcaaf5f4ad --- /dev/null +++ b/dev/make/AGENTS.md @@ -0,0 +1,17 @@ +# AGENTS.md - Make Build Fragments (dev/make/) + +## Purpose +Fragments included by the root `makefile`, which drives the production build (see `INSTALL.md`). + +## Layout +- `common.mk`: compile, link and packaging commands shared by all platforms +- `deps.mk`, `deps.mkl.mk`, `deps.ref.mk`: third-party dependencies per backend (`BACKEND_CONFIG=mkl|ref`) +- `compiler_definitions/[.]..mk`: per-compiler flags +- `function_definitions/.mk`: per-platform helpers (`lnx32e`, `win32e`, `mac32e`, `lnxarm`, `winarm`, `lnxriscv64`) +- `identify_os.sh`: host OS detection + +## Rules for Changes +- Compile flags go in `COPT`, link flags in `LOPT`. +- Library binary versions come from `MAJORBINARY` / `MINORBINARY` in `makefile.ver`. The Bazel build mirrors them in `dev/bazel/repos.bzl`; change both together. +- Export and symbol-visibility changes here need the matching change in the Bazel build (`dev/bazel/`), and vice versa. +- Shell and batch script rules are in the root `AGENTS.md`. diff --git a/docs/AGENTS.md b/docs/AGENTS.md index 3c8b5a356d6..0118be58350 100644 --- a/docs/AGENTS.md +++ b/docs/AGENTS.md @@ -1,198 +1,24 @@ +# AGENTS.md - Documentation (docs/) -# Documentation Generation - AI Agents Context +## Purpose +Sphinx sources for the oneDAL documentation. Authoring conventions are in `docs/README.md`. -> **Purpose**: Context for AI agents working with oneDAL's sophisticated documentation generation system and customizations. +## Layout +- `source/`: reStructuredText pages; `source/conf.py` holds extensions, substitutions (`|short_name|`) and `extlinks` (`:cpp_example:`) +- `dalapi/`: custom Sphinx extension that renders the C++ API from Doxygen XML +- `doxygen/oneapi/Doxyfile`: Doxygen input is `cpp/oneapi/dal` only; DAAL API pages are written by hand +- `rst_examples.py`: generates `source/examples/{cpp,dpc}/*.rst` from `examples/oneapi/`; those files are gitignored, so don't edit or commit them -## πŸ—οΈ Documentation Architecture +## Rules for Changes +- Keep public docs in sync with the implementation: a change to a public API, command or example updates the pages that describe it in the same PR. +- The build runs Sphinx with `-W --keep-going -n`: warnings, including broken references, fail it. +- Cross-reference with explicit targets (`.. _my-page:` and `:ref:`), not headings. +- Record removals and deprecations of public functionality in `source/deprecation.rst`. -oneDAL implements a **sophisticated multi-stage documentation system** combining multiple technologies: - -### Core Technology Stack -- **Sphinx**: Primary documentation engine with reStructuredText -- **Doxygen**: C++ API documentation extraction -- **Custom dalapi Extension**: Bridge between Doxygen and Sphinx -- **Custom Python Tools**: RST generation and processing - -### Build Pipeline -1. **RST Generation**: `create_rst_examples` β†’ Dynamic example RST files -2. **Doxygen Processing**: `doxygen` β†’ XML API extraction from C++ headers -3. **Doxygen Parsing**: `parse-doxygen` β†’ Custom Python parser creates structured YAML -4. **Sphinx Build**: Multiple targets with different configurations - -## πŸ“ Structure -``` -docs/ -β”œβ”€β”€ Makefile # Build orchestration with parallel processing -β”œβ”€β”€ source/conf.py # Sphinx configuration with custom extensions -β”œβ”€β”€ rst_examples.py # Dynamic example RST generation -β”œβ”€β”€ dalapi/ # Custom Sphinx extension for C++ API -β”‚ β”œβ”€β”€ extension.py # Main extension with path resolution -β”‚ β”œβ”€β”€ directives.py # Custom RST directives -β”‚ β”œβ”€β”€ doxypy/ # Doxygen XML parser -β”‚ └── generator.py # RST content generation -β”œβ”€β”€ doxygen/oneapi/ # Doxygen configuration -β”‚ └── Doxyfile # Points to cpp/oneapi/dal sources -└── source/ # RST source files organized by API -``` - -## πŸ”§ Build System Integration - -### Makefile Orchestration -```makefile -# Parallel processing optimization -SPHINXOPTS = "-j`nproc`" # Uses all available CPU cores - -# Multi-stage build targets -create_rst_examples: # Dynamic RST generation from examples - python3 rst_examples.py - -doxygen: # C++ API extraction - cd doxygen/oneapi && doxygen - -parse-doxygen: doxygen # Custom XMLβ†’YAML conversion - python -m dalapi.doxypy.cli doxygen/oneapi/doxygen/xml --compact > build/tree.yaml - -html: create_rst_examples # Production build with Intel branding - sphinx-build -t use_intelname -b html source build - -html-github: create_rst_examples # GitHub-optimized build - sphinx-build -b html source build -``` - -### Sphinx Configuration Customizations -```python -# Custom extension stack -extensions = [ - 'sphinx_prompt', # Code prompt styling - 'sphinx_substitution_extensions', # Variable substitutions - 'sphinx.ext.extlinks', # External link shortcuts - 'sphinx_tabs.tabs', # Tabbed content - 'dalapi', # CUSTOM: oneDAL API integration - 'sphinx.ext.githubpages', # GitHub Pages optimization - 'notfound.extension' # 404 page handling -] - -# Variable substitution system -substitutions = [ - ('|short_name|', 'oneDAL'), - ('|daal_in_code|', 'daal') -] - -# External link management -extlinks = { - 'cpp_example': ('https://github.com/uxlfoundation/oneDAL/tree/main/examples/daal/cpp/source/%s', None), - 'daal4py_example': ('https://github.com/uxlfoundation/scikit-learn-intelex/tree/main/examples/daal4py/%s', None), -} -``` - -## 🎭 Advanced Customizations - -### Custom dalapi Extension -```python -# Path resolution for complex build relationships -class PathResolver(object): - def __init__(self, app, relative_doxyfile_dir, relative_sources_dir): - self.base_dir = app.confdir - self.doxyfile_dir = self.absjoin(self.base_dir, relative_doxyfile_dir) - self.doxygen_xml = self.absjoin(self.doxyfile_dir, 'doxygen', 'xml') - -# Processing pipeline: C++ Headers β†’ Doxygen XML β†’ dalapi Parser β†’ RST Content β†’ Sphinx HTML -``` - -### Dynamic Example Integration -```python -# rst_examples.py: Auto-generates RST from example source code -def create_rst(filedir, filename, lang): - rst_content = '.. _{}_{}:\n\n{}\n{}\n\n'.format(lang, filename, filename, '#' * len(filename)) - rst_content += '.. literalinclude:: ../../../../examples/oneapi/{}/source/{}/{}\n'.format(lang, filedir, filename) - rst_content += ' :language: cpp\n' - return rst_content -``` - -### Interface-Specific Processing +## Verification ```bash -# Doxygen configuration targets oneAPI interface ONLY -INPUT = ../../../cpp/oneapi/dal - -# But examples include both DAAL and oneAPI patterns -examples/daal/cpp/source/ # Traditional DAAL examples -examples/oneapi/cpp/source/ # Modern oneAPI examples -examples/oneapi/dpc/source/ # GPU-accelerated examples +cd docs +pip install -r requirements.txt +make html # what CI runs (.ci/pipeline/docs.yml); make html-github builds without Intel branding ``` - -## 🎯 Documentation Generation Options - -### Build Targets Available -```makefile -# Primary targets -make html # Production build with Intel branding (-t use_intelname) -make html-github # GitHub-optimized build (no Intel branding) -make html-test # Test/validation build - -# Component targets -make create_rst_examples # Generate example RST files -make doxygen # Generate Doxygen XML from C++ headers -make parse-doxygen # Convert Doxygen XML to structured YAML - -# Parallel processing (automatic) -SPHINXOPTS = "-j`nproc`" # Uses all available CPU cores -``` - -### Configuration Variants -```python -# Intel branding control via Sphinx tags -html: -t use_intelname # Production: Intel-specific branding enabled -html-github: # GitHub: Open-source branding - -# Output customization -exclude_patterns = [ # Selectively exclude content - 'daal/data-management/numeric-tables/*.rst', - 'daal/algorithms/dbscan/distributed-steps/*', - 'onedal/algorithms/.*/includes/*' -] -``` - -### Interface Targeting -- **API Documentation**: oneAPI interface ONLY via Doxygen (`INPUT = ../../../cpp/oneapi/dal`) -- **Example Integration**: Both DAAL and oneAPI examples via `rst_examples.py` -- **Cross-references**: Automatic linking between examples and API docs - -## πŸ›οΈ Customization Architecture - -### Extension System -- **dalapi**: Custom Sphinx extension bridges Doxygenβ†’Sphinx -- **Path Resolution**: Complex build relationship management -- **Custom Directives**: oneDAL-specific RST directives -- **XML Processing**: Advanced Doxygen XML parsing and transformation - -### Content Generation -- **Dynamic RST**: Examples auto-converted from source code -- **API Integration**: C++ headers β†’ XML β†’ YAML β†’ RST β†’ HTML pipeline -- **Variable Substitutions**: Global replacements (`|short_name|` β†’ `oneDAL`) -- **External Links**: Automated GitHub link generation - -### Multi-Format Support -- **Tabbed Content**: `sphinx_tabs.tabs` for interface comparisons -- **Code Prompts**: `sphinx_prompt` for terminal examples -- **Modern Theme**: `sphinx_book_theme` with responsive design -- **GitHub Integration**: Optimized for GitHub Pages deployment - -## 🎯 Critical Generation Rules - -### Build Dependencies -- **Sequential Processing**: RST generation β†’ Doxygen β†’ XML parsing β†’ Sphinx build -- **Parallel Optimization**: Use all CPU cores for Sphinx processing -- **Interface Separation**: oneAPI gets full API docs, DAAL examples included -- **Example Synchronization**: Auto-generated RST stays in sync with source code - -### Customization Guidelines -- **Extension Configuration**: Modify `dalapi` extension for API processing changes -- **Build Targets**: Use appropriate target for deployment context (Intel vs GitHub) -- **Content Exclusion**: Update `exclude_patterns` for content filtering -- **Link Management**: Maintain `extlinks` for external references - -## πŸ“– Further Reading -- **[cpp/AGENTS.md](../cpp/AGENTS.md)** - C++ implementation context -- **[cpp/oneapi/AGENTS.md](../cpp/oneapi/AGENTS.md)** - oneAPI interface patterns -- **[examples/AGENTS.md](../examples/AGENTS.md)** - Example integration patterns -- **[dev/AGENTS.md](../dev/AGENTS.md)** - Build system architecture +Warnings are also written to `docbuild-log.txt`. diff --git a/examples/AGENTS.md b/examples/AGENTS.md index 641df2b7089..bf37575aae1 100644 --- a/examples/AGENTS.md +++ b/examples/AGENTS.md @@ -1,175 +1,23 @@ +# AGENTS.md - Examples (examples/) -# Examples - AI Agents Context +## Purpose +Standalone programs shipped in the release and shown in the docs, one directory per algorithm under `source/`. -> **Purpose**: Context for AI agents working with oneDAL example patterns demonstrating dual C++ interface usage. +## Layout +- `daal/cpp/`: DAAL interface examples; shared helpers in `source/utils/` +- `oneapi/cpp/`: oneAPI CPU examples; shared helpers in `source/example_util/` +- `oneapi/dpc/`: oneAPI SYCL examples, where `main` runs `run(sycl::queue &q)` once per device from `list_devices()` +- `cmake/setup_examples.cmake`: CMake globs `source/*/*.cpp`; `target_excludes.cmake` lists examples to skip per platform +- Input data comes from the top-level `data/` directory via `get_data_path()` -## πŸ—οΈ Examples Architecture +## Rules for Changes +- Each example is self-contained and runnable, and uses one interface; don't mix DAAL and oneAPI code in one file. +- For a new algorithm directory, add it to the `algos` list of `dal_algo_example_suite` (`daal_algo_example_suite` for DAAL) in that tree's `BUILD`. Examples that don't match one algorithm library get their own `dal_example_suite`, like `graph`. +- The docs pull in every file under `oneapi/{cpp,dpc}/source/` via `docs/rst_examples.py`, and link DAAL examples by path with `:cpp_example:`. Renaming or removing an example breaks those references, so update `docs/source/` in the same PR. +- Reuse `example_util` / `utils` helpers instead of adding local printing or data-loading code. -oneDAL examples demonstrate **three distinct interface patterns** corresponding to the dual C++ architecture: - -### Interface Categories -- **DAAL Interface** (`examples/daal/cpp/source/`) - Traditional CPU-focused patterns -- **oneAPI CPU** (`examples/oneapi/cpp/source/`) - Modern C++ with fluent interfaces -- **oneAPI GPU** (`examples/oneapi/dpc/source/`) - Heterogeneous computing with SYCL - -## πŸ“ Structure -``` -examples/ -β”œβ”€β”€ daal/cpp/source/ # Traditional DAAL interface examples -β”‚ β”œβ”€β”€ covariance/ # Algorithm-specific examples -β”‚ β”œβ”€β”€ kmeans/ # K-means clustering examples -β”‚ └── [algorithms]/ # Other algorithm examples -β”œβ”€β”€ oneapi/cpp/source/ # Modern oneAPI CPU examples -β”‚ β”œβ”€β”€ covariance/ # Same algorithms, modern interface -β”‚ β”œβ”€β”€ kmeans/ # Modern K-means patterns -β”‚ └── example_util/ # Shared utilities -└── oneapi/dpc/source/ # GPU-accelerated examples - β”œβ”€β”€ kmeans/ # SYCL-enabled K-means - └── [algorithms]/ # GPU algorithm examples -``` - -## 🎭 Interface Pattern Comparison - -### 1. DAAL Traditional Pattern -```cpp -// Explicit lifecycle management with status codes -#include "daal.h" -using namespace daal::algorithms; - -covariance::Batch<> algorithm; -algorithm.input.set(covariance::data, dataSource.getNumericTable()); -algorithm.compute(); -covariance::ResultPtr res = algorithm.getResult(); -printNumericTable(res->get(covariance::covariance), "Covariance matrix:"); -``` - -**Characteristics:** -- **Headers**: `#include "daal.h"` with `using namespace daal::algorithms` -- **Memory**: Custom `SharedPtr` and `NumericTable` management -- **Workflow**: Instantiate β†’ set input β†’ compute() β†’ getResult() -- **Error Handling**: Implicit status checking - -### 2. oneAPI CPU Pattern -```cpp -// Modern fluent interface with RAII -#include "oneapi/dal/algo/kmeans.hpp" -namespace dal = oneapi::dal; - -const auto kmeans_desc = dal::kmeans::descriptor<>() - .set_cluster_count(20) - .set_max_iteration_count(5) - .set_accuracy_threshold(0.001); -const auto result_train = dal::train(kmeans_desc, x_train, initial_centroids); -std::cout << "Centroids:\n" << result_train.get_model().get_centroids() << std::endl; -``` - -**Characteristics:** -- **Headers**: `#include "oneapi/dal/algo/[algorithm].hpp"` -- **Memory**: STL RAII with automatic resource management -- **Workflow**: Descriptor configuration β†’ train/compute β†’ result access -- **Data**: Modern `dal::table` with CSV data sources - -### 3. oneAPI GPU Pattern -```cpp -// SYCL queue integration for heterogeneous computing -#include -#include "oneapi/dal/algo/kmeans.hpp" - -void run(sycl::queue &q) { - const auto x_train = dal::read(q, dal::csv::data_source{...}); - const auto result_train = dal::train(q, kmeans_desc, x_train, initial_centroids); -} -``` - -**Characteristics:** -- **Headers**: SYCL integration with `#include ` -- **Queue Parameter**: All operations accept `sycl::queue& q` as first parameter -- **Data Loading**: Queue-aware `dal::read(q, data_source)` -- **Execution**: GPU-accelerated with same API as CPU version - -## πŸ”§ Build System Integration - -### Bazel Configuration -```python -# examples/oneapi/cpp/BUILD -dal_example_suite( - name = "kmeans", - compile_as = ["c++"], - srcs = glob(["source/kmeans/*.cpp"]), - dal_deps = ["@onedal//cpp/oneapi/dal/algo:kmeans"], - data = ["@onedal//examples/oneapi:data"], - extra_deps = [":example_util"], -) -``` - -### Common Patterns -- **Algorithm Suites**: Each algorithm gets `dal_example_suite` target -- **Shared Utilities**: `example_util` module for common helpers -- **Data Dependencies**: Centralized test data management -- **OpenCL Integration**: GPU examples require OpenCL binary dependencies - -## 🎯 Example Usage Patterns - -### Data Loading Evolution -```cpp -// DAAL: Explicit data source management -FileDataSource dataSource(fileName, - DataSource::doAllocateNumericTable, - DataSource::doDictionaryFromContext); -dataSource.loadDataBlock(); - -// oneAPI CPU: Modern data loading -const auto input = dal::read(dal::csv::data_source{fileName}); - -// oneAPI GPU: Queue-aware data loading -const auto input = dal::read(queue, dal::csv::data_source{fileName}); -``` - -### Algorithm Configuration Evolution -```cpp -// DAAL: Parameter-based configuration -const size_t nClusters = 20; -const size_t nIterations = 5; -// Configuration through algorithm parameters - -// oneAPI: Fluent descriptor pattern -const auto desc = dal::kmeans::descriptor<>() - .set_cluster_count(20) - .set_max_iteration_count(5) - .set_accuracy_threshold(0.001); +## Verification +```bash +bazel test //examples/oneapi/cpp:kmeans # one algorithm +bazel test //examples/oneapi/cpp:all # CI runs this and //examples/daal/cpp:all, with --test_link_mode variants ``` - -## πŸ›οΈ Design Philosophy - -### Progressive Modernization -1. **DAAL Examples**: Demonstrate traditional patterns for backward compatibility -2. **oneAPI CPU**: Show modern C++ best practices with same algorithms -3. **oneAPI GPU**: Extend CPU patterns to heterogeneous computing - -### Interface Consistency -- **Same Algorithm Logic**: Core computation remains identical across interfaces -- **Consistent Results**: All three patterns produce equivalent outputs -- **Performance Scaling**: GPU examples demonstrate acceleration without API complexity - -## 🎯 Critical Example Rules - -### Interface Separation -- **NEVER mix interfaces** within single example -- **DAAL**: Traditional headers, explicit lifecycle, custom smart pointers -- **oneAPI**: Modern headers, fluent API, STL RAII - -### GPU Programming -- **Queue Management**: Always pass `sycl::queue` as first parameter -- **Data Locality**: Use queue-aware data loading for optimal GPU performance -- **Memory Management**: Leverage USM for CPU/GPU data sharing - -### Build Dependencies -- **Algorithm Dependencies**: Match example to correct `dal_deps` in BUILD files -- **Utility Sharing**: Use `example_util` for common patterns across examples -- **Data Management**: Reference centralized data dependencies - -## πŸ“– Further Reading -- **[cpp/AGENTS.md](../cpp/AGENTS.md)** - C++ implementation overview -- **[cpp/daal/AGENTS.md](../cpp/daal/AGENTS.md)** - DAAL interface patterns -- **[cpp/oneapi/AGENTS.md](../cpp/oneapi/AGENTS.md)** - oneAPI interface patterns -- **[dev/AGENTS.md](../dev/AGENTS.md)** - Build system and development tools