Skip to content
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 8 additions & 56 deletions .ci/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,61 +5,10 @@
This document describes the CI infrastructure for oneDAL (Intel Data Analytics Library)
## Directory Structure

### `.ci` Folder

The `.ci` folder contains the core CI infrastructure scripts and configurations organized into three main subdirectories:

#### `.ci/env/` - Environment Setup Scripts
- **`apt.sh`** - Package installation script for Ubuntu/Debian systems with functions for:
- Intel OneAPI toolkit components (DPC++, TBB, DPL, MKL)
- Development tools (clang-format, editorconfig-checker)
- Base development packages and dependencies
- **`bazelisk.sh`** - Bazel build system setup and installation
- **`editorconfig-checker.sh`** - EditorConfig compliance checker installation
- **`environment.yml`** - Conda environment specification
- **`openblas.sh`** - OpenBLAS library installation and configuration
- **`openrng.sh`** - OpenRNG backend setup for random number generation
- **`tbb.sh`** / **`tbb.bat`** - Intel TBB (Threading Building Blocks) setup for Linux/Windows
- **`riscv64-clang-crosscompile-toolchain.cmake`** - RISC-V cross-compilation toolchain configuration

#### `.ci/pipeline/` - CI Pipeline Definitions
- **`ci.yml`** - Main Azure DevOps pipeline configuration with:
- Multi-platform build matrices (Linux, Windows)
- Compiler configurations (GNU, Clang, Intel)
- Build targets (daal, onedal_c)
- Testing and validation jobs
- Artifact publishing
- **`docs.yml`** - Documentation build and deployment pipeline

#### `.ci/scripts/` - Build and Test Scripts
- **`build.sh`** / **`build.bat`** - Cross-platform build orchestration with support for:
- Multiple compilers (gnu, clang, icx)
- Architecture optimizations (AVX2, etc.)
- Backend configurations (MKL, reference implementations)
- Cross-compilation capabilities
- **`test.sh`** / **`test.bat`** - Comprehensive testing framework execution
- **`clang-format.sh`** - Code formatting verification
- **`describe_system.sh`** - System information collection for debugging
- **`abi_check.sh`** - ABI compatibility verification
- **`install_basekit.bat`** - Intel OneAPI Base Toolkit installation for Windows
- **`collect_opencl_rt.ps1`** - OpenCL runtime collection script

### `.github/workflows/` - GitHub Actions Workflows

#### Core CI Workflows
- **`ci.yml`** - Main CI pipeline for x86 platforms with DPC++ builds
- **`ci-aarch64.yml`** - AArch64 (ARM64) specific CI pipeline
- **`nightly-build.yml`** / **`nightly-test.yml`** - Automated nightly builds and testing

#### Specialized Workflows
- **`docker-validation-ci.yml`** / **`docker-validation-nightly.yml`** - Container-based validation
- **`docs-release.yml`** - Documentation deployment and release management
- **`label-enforcement.yml`** - PR labeling automation
- **`pr-checklist.yml`** - Pull request compliance verification
- **`renovate-validation.yml`** - Dependency update validation
- **`skywalking-eyes.yml`** - License header compliance checking
- **`slack-pr-notification.yml`** - Team notification system
- **`openssf-scorecard.yml`** - Security scorecard assessment
- `.ci/pipeline/ci.yml`: the Azure DevOps pipeline (build matrix, `FormatterChecks`); `docs.yml` builds the docs.
- `.ci/env/`: dependency installers. `apt.sh` takes a component name (`dev-base`, `mkl`, ...); `tbb`, `openblas` and `bazelisk` each have `.sh` and Windows variants.
- `.ci/scripts/`: `build.sh` / `build.bat` (compiler, optimization, backend and cross-compile options), `test.sh` / `test.bat`, `clang-format.sh`, `abi_check.sh`, and the Windows release checks (`compare_windows_release.ps1`, `test_bazel_release_cmake_example.ps1`).
- `.github/workflows/`: GitHub Actions. `ci.yml`, `ci-win.yml` and `ci-aarch64.yml` build and test; `nightly-build.yml` produces artifacts other repositories download (see `.github/AGENTS.md`).

## CI/CD Architecture

Expand All @@ -74,7 +23,6 @@ The oneDAL CI infrastructure supports:
### CI Platform Integration
- **GitHub Actions**: Primary CI/CD platform for public workflows
- **Azure DevOps**: Extended validation and internal testing
- **Mergify**: Automated merge management
- **Renovate**: Dependency update automation
- **Codefactor**: Code quality analysis

Expand All @@ -92,6 +40,10 @@ The CI system employs comprehensive build matrices covering:
- **Security**: OpenSSF Scorecard integration
- **Documentation**: Automated doc generation and validation

## Rules for Changes
- Pipelines call scripts in `.ci/scripts/` and `.ci/env/`. Change the script, not only the pipeline YAML that calls it.
- Style checks (clang-format, editorconfig-checker) run only in Azure `FormatterChecks`. See "Verification Before You Push" in the root `AGENTS.md`.

## Usage Guidelines

### Local Development
Expand Down
2 changes: 1 addition & 1 deletion .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# to LF line endings on checkout.
*.c text
*.h text
*.i text
*.i text linguist-language=C++
*.hpp text
*.cpp text
*.def text
Expand Down
1 change: 1 addition & 0 deletions .github/.licenserc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ header:
- 'docs/source/substitutions_specific.txt'
# Some files from .ci/.github
- '.github/instructions/*.md'
- '.github/copilot-instructions.md'
- '.github/CODEOWNERS'
- '.github/pull_request_template.md'
- '.github/renovate.json'
Expand Down
5 changes: 5 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Copilot instructions - oneDAL

`AGENTS.md` is the canonical repository guidance. Before reviewing or editing a file, read the root `AGENTS.md` and every applicable directory-level `AGENTS.md`.

Also follow the matching `.github/instructions/*.instructions.md` files. They provide additive, Copilot-specific guidance selected by `applyTo`; do not duplicate or override `AGENTS.md` policy here.
11 changes: 11 additions & 0 deletions .github/instructions/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Agent Instructions for `.github/instructions`

These files provide Copilot guidance for paths selected by their YAML `applyTo` front matter.

## Rules for Changes

- Keep `applyTo` patterns aligned with every file type the instruction governs, including templates and configuration files.
- Treat overlapping instruction files as additive. Keep broad guidance brief and let scoped files add detail without contradicting it.
- Keep PR review rules limited to source-confirmed correctness, compatibility, ownership, dispatch, error handling, and test coverage. Do not duplicate checks enforced by formatting, license, or other CI jobs.
- Link only to existing files, using paths relative to the instruction file.
- Put durable subsystem context in the nearest `AGENTS.md`; use these files only for Copilot-specific and path-scoped guidance.
52 changes: 28 additions & 24 deletions .github/instructions/build-systems.instructions.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
applyTo: ["**/makefile", "**/Makefile", "**/BUILD", "**/BUILD.bazel", "**/*.bazel", "**/*.mk", "**/CMakeLists.txt"]
applyTo: ["**/makefile", "**/Makefile", "**/BUILD", "**/BUILD.bazel", "**/*.tpl.BUILD", "**/*.bazel", "**/*.bzl", ".bazelrc", "**/.bazelrc", "**/*.mk", "**/CMakeLists.txt"]
---

# Build Systems Instructions for GitHub Copilot
Expand All @@ -15,36 +15,38 @@ applyTo: ["**/makefile", "**/Makefile", "**/BUILD", "**/BUILD.bazel", "**/*.baze
### Make (Production)
```bash
# Build everything
`make`
make -f makefile daal oneapi PLAT=lnx32e

# Platform-specific builds
`make PLAT=lnx32e COMPILER=icx`
`make PLAT=win32e COMPILER=vc`
make -f makefile daal oneapi_c PLAT=lnx32e COMPILER=gnu
make -f makefile oneapi_c PLAT=win32e COMPILER=vc

# CPU targets
`make REQCPU="sse2 avx2 avx512"`
make -f makefile daal PLAT=lnx32e REQCPU="avx2 avx512"

# Backend selection
`make BACKEND_CONFIG=mkl` # Intel MKL (default)
`make BACKEND_CONFIG=ref` # Reference/OpenBLAS
make -f makefile daal PLAT=lnx32e BACKEND_CONFIG=mkl # Intel MKL (default)
make -f makefile daal PLAT=lnx32e BACKEND_CONFIG=ref # Reference/OpenBLAS
```

### Bazel (Development)
```bash
# Build targets
`bazel build //cpp/oneapi/dal:core`
`bazel build //examples/daal/cpp:association_rules`
bazel build //cpp/oneapi/dal:core
bazel build //examples/daal/cpp:cholesky

# Test targets
`bazel test //cpp/oneapi/dal:tests`
`bazel test --config=dpc //cpp/oneapi/dal:tests` # GPU tests
bazel test --config=host //cpp/oneapi/dal:tests # CPU only
bazel test --config=dpc --device=gpu //cpp/oneapi/dal:tests # DPC++ on GPU
```

### CMake (Integration)
There is no root `CMakeLists.txt`. CMake builds the examples against an installed release, found with `find_package(oneDAL)`:
```bash
# Configure and build
`cmake -B build -S . -DCMAKE_BUILD_TYPE=Release`
`cmake --build build --parallel`
source __release_lnx/daal/latest/env/vars.sh # Make release tree
cd examples/oneapi/cpp
cmake -B build -S . -DONEDAL_LINK=dynamic
cmake --build build --parallel
```

## 🏗️ CPU Architecture Support
Expand Down Expand Up @@ -96,21 +98,23 @@ export MKLROOT=/path/to/mkl

### Algorithm Module Pattern
```python
# DAAL module
# DAAL kernel (cpp/daal/src/algorithms/kmeans/BUILD); the macro adds C++17 and CPU dispatch defines
daal_module(
name = "kmeans",
features = [ "c++17" ],
cpu_defines = {
"sse2": [ "DAAL_CPU=sse2" ],
"avx2": [ "DAAL_CPU=avx2" ],
"avx512": [ "DAAL_CPU=avx512" ],
},
name = "kernel",
auto = True,
deps = [
"@onedal//cpp/daal:core",
"@onedal//cpp/daal:engines",
"@onedal//cpp/daal/src/algorithms/distributions:kernel",
],
)

# oneAPI module
# oneAPI module (cpp/oneapi/dal/algo/kmeans/BUILD)
dal_module(
name = "kmeans",
compile_as = ["c++", "dpc++"], # CPU and GPU
auto = True,
dal_deps = ["@onedal//cpp/oneapi/dal:core"],
extra_deps = ["@onedal//cpp/daal/src/algorithms/kmeans:kernel"],
)
```

Expand Down
Loading
Loading