V2.7.1 分支提供依赖镜像 - #138
Conversation
Add .ci/docker/ directory with separated builder (manylinux) and test (ubuntu) images for both x86_64 and aarch64 architectures. - Builder images: PyTorch + build toolchain (gcc, cmake, ninja, mold, ccache) - Test images: PyTorch CPU + CANN runtime + triton-ascend + test frameworks - CANN chip selection via CANN_CHIP build arg (A1/A2/A3) - docker_build.sh follows upstream pytorch/pytorch tag parsing pattern - GitHub Actions workflow for building and pushing to quay.io Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Docker tags must be lowercase. Tag format changed from cannA1 to cann-a1, while CANN_CHIP build arg stays uppercase (A1/A2/A3) for install_cann.sh. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
requirements-*.txt files are at the docker context root, not in common/. Add explicit COPY instructions in all four Dockerfiles. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Builder images no longer install CANN/OBS, so COPY common/ is not needed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PyTorch is arch-specific (torch==2.7.1+cpu for x86_64, torch==2.7.1 for aarch64). Installing directly in Dockerfile avoids pip env-var expansion issues with requirements files. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…mmands - workflow_dispatch now accepts optional 'tag' to build a single image - summary job collects results via artifacts and outputs docker pull commands Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
GitHub Actions outputs must be single-line. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…n test Dockerfiles
…input from build workflow
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
…N_VERSION in top-level workflow - Move docker_image_build and docker_image_test to top-level workflow inputs with defaults - Remove pytorch_version from all workflow inputs, hardcode as env.PYTORCH_VERSION=2.7.1 - Remove python_version from trigger/top-level inputs, hardcode as env.PYTHON_VERSION=3.10 - Pass docker_image to build workflow instead of hardcoding in container definition - Update all sub-workflows to remove pytorch_version input parameter
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
…ith: section env context is not available in with: section of reusable workflow calls
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
- Replace hardcoded Ascend/pytorch action ref with ./.github/actions/setup-npu-test-env - Add pytorch_version and python_version as workflow inputs in top-level test workflow - Remove unused env.PYTORCH_VERSION from build workflow Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
…kflows - Remove unused pyyaml from builder, beartype/attrs/decorator from test deps - Change local action ref to fork repo ref to fix PR checkout issue: `./.github/actions/setup-npu-test-env` -> `kerer-ai/pytorch/.github/actions/setup-npu-test-env@v2.7.1_image` Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
…ll_cases.py Log LD_LIBRARY_PATH, libhccl.so location, CANN version at two levels: - Workflow: before running collect script - Python: appended to error output on pytest collection failure Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
…ioned paths A3 CANN 9.0.0-beta.2 installs to versioned directory, not /usr/local/Ascend/cann/ Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Custom workflow now auto-detects nodeid format (contains '::') vs file path - Nodeid mode: runs each case as independent pytest process, strictly serial - File path mode: falls back to run_npu_test_shard.py (backward compatible) - Trigger narrowed to 5 specific failed test_grad_* cases for HBM isolation verification
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
Root cause: pytest could not find test files (rc=4) because: 1. Working directory was not pytorch-test-src/test/ 2. nodeids still had 'test/' prefix (should be 'functorch/test_ops.py::...') Fix: - cd pytorch-test-src/test before running pytest - Strip 'test/' prefix from nodeids (matching run_npu_test_shard.py behavior) - Resolve REPORT_DIR to absolute path before cd
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
Three execution modes now: 1. SAME_PROCESS: prefix → all nodeids in ONE pytest process (control: eliminates both HBM concurrency AND per-case subprocess) 2. Contains '::' → per-case independent pytest processes (serial) 3. Otherwise → file path mode (run_npu_test_shard.py) Control experiment logic: SAME_PROCESS PASS → HBM concurrency IS root cause SAME_PROCESS FAIL → pytest.main() batch IS root cause
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
8 warmup cases (preceding test_grad cases that passed in original run) run before the 5 target cases in the same pytest process, simulating the 'not-first-in-process' condition from the original parallel run. Full experiment matrix: A: per-case process + serial (1 worker, 1 case/batch) → 4/5 PASS B: same-process + warmup + serial (1 worker, 13 cases in 1 pytest) C: original parallel (16 workers, 100 cases/batch) → 5/5 FAIL If B passes 4/5 target cases → HBM concurrency confirmed If B fails 4/5 target cases → pytest.main() batch state is the cause
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
90 warmup cases (case_idx 26588-26677, all from test_ops.py) run before
the 5 target cases in a single pytest process. These are the exact same
cases that preceded test_grad_to_sparse in the original parallel run.
Full experiment:
95 cases total (90 warmup + 5 target)
Single pytest process
Single worker (no cross-process HBM contention)
Control: if target cases pass → HBM concurrency confirmed
if target cases fail → pytest.main() batch is the cause
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
Both SAME_PROCESS and per-case nodeid modes now explicitly bind to a single NPU device, matching the original run_npu_test_shard.py behavior where each worker subprocess was pinned to one device via round-robin. Without this binding, the pytest process might span multiple NPU devices, adding an uncontrolled variable to the experiment.
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
File path mode now uses --max-cases-per-batch 10000 so all cases run in a single pytest.main() invocation — simulating the same behavior as 'pytest test/functorch/test_ops.py'. This tests whether the NPU memory corruption occurs simply from running multiple test cases in the same process, without any CI-specific batching infrastructure.
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
SAME_PROCESS:test/functorch/test_ops.py → runs the ENTIRE file via 'python -m pytest functorch/test_ops.py' in a single process. This is the purest form of the experiment — no run_npu_test_shard.py, no batching, no subprocesses. Just pytest running all test functions in one interpreter.
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
…tion File path mode with --max-cases-per-batch 10000 uses the EXACT same code path as the original parallel run: worker subprocess → pytest.main() with all cases. The only variable changed is --max-workers 1 (serial). This eliminates python -m pytest vs pytest.main() as a confounding factor.
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
SAME_PROCESS:test/functorch/test_ops.py results in: cd pytorch-test-src/test export ASCEND_RT_VISIBLE_DEVICES=0 python -m pytest functorch/test_ops.py Simplest form: one pytest command, one process, all test cases. No run_npu_test_shard.py, no batching, no subprocesses.
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
…ntext
When a test case hits EZ9999/EE9999/EZ1009/vector core exception,
the NPU device context gets poisoned and all subsequent cases in
the same worker subprocess will fail with garbage/corrupted data.
This change:
1. Adds _is_fatal_npu_error() to detect device-fatal error patterns
(EZ9999, EE9999, EZ1009, ERR00100, vector core exception,
rtDeviceSynchronizeWithTimeout, device error, Kernel Run failed)
2. In the worker stdout reader thread, checks each case result for
fatal error patterns and sets a flag when detected
3. In the main polling loop, immediately kills the poisoned worker,
marks the first unreported case as device-fatal-error, and
spawns a new worker for the remaining cases with a clean device
This prevents one hard NPU error from poisoning the entire batch
of test cases in the same subprocess.
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
- File path mode: --max-workers 16 (matching regular workflow) - Removed --max-cases-per-batch override (uses default 100) - Restored dynamic test_files from detect job in trigger - Kept SAME_PROCESS and per-case nodeid modes for debugging The fatal NPU error detection (EZ9999/EE9999 → worker restart) added in the previous commit will now protect 16 concurrent workers from device context poisoning.
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
… restart Replace parent-side proc.kill() approach with worker-side os._exit(70): Worker-side (_worker_main): After each case, check combined output for fatal signatures: - "The process exits for this inner error" (NPUQueue CAN_EXIT) - EZ9999, EE9999, EZ1009 (hardware device errors) - vector core exception, rtDeviceSynchronizeWithTimeout If detected → os._exit(NPU_QUEUE_FATAL_EXIT_CODE=70) Exit happens BETWEEN cases → no case is sacrificed Parent-side (_execute_worker_batch): Handle returncode 70: retry ALL not_reported cases in new worker (unlike coredump handler which sacrifices the first unreported case) Removed: parent-side stdout reader fatal detection, main loop kill logic, post-loop fatal error handler — all replaced by cleaner worker-side exit. Kept: coredump detection (SIGSEGV/SIGABRT), idle timeout, both as fallbacks for truly crashed/hung workers. Net: -25 lines, +78 new, -103 old. Merges NPUQueue CAN_EXIT and hardware error detection into a single unified mechanism.
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
Layer 1: Extended signature matching (zero overhead)
- NPUQueue ERROR_EXIT: "The process exits for this inner error"
- deviceErrorMap labels: UCE ERROR, HBM MULTI BIT ECC ERROR,
SUSPECT MEM ERROR, HCCS LINK ERROR, HCCL OP RETRY FAILED,
SUSPECT REMOTE ERROR (NPUQueue.cpp:175-183 ThrowDeviceError)
- CANN runtime codes: EZ9999, EE9999, EZ1009 (redundant, kept)
Layer 2: Probe computation (~1ms, catches unknown patterns)
- torch.ones(4, device='npu').sum().item() != 4.0
- If CAN_EXIT: ops become silent no-ops → garbage result ≠ 4.0
- If device error: sync throws → caught as exception
- Only runs when Layer 1 doesn't match, for failed/error cases
- Zero overhead for passing tests
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
CLA Signature Guide@kerer-ai , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
2 participants
No description provided.