[Do not merge] Testing bot for auto-enabled gpu tests by smuzaffar · Pull Request #51691 · cms-sw/cmssw

smuzaffar · 2026-08-12T19:10:50Z

No description provided.

cmsbuild · 2026-08-12T19:11:14Z

cms-bot internal usage

cmsbuild · 2026-08-12T19:12:32Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51691/50570

smuzaffar · 2026-08-12T19:13:53Z

please test

smuzaffar · 2026-08-12T19:41:47Z

please test

cmsbuild · 2026-08-12T21:03:41Z

-1

Failed Tests: UnitTests
Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cd9a80/55292/summary.html
COMMIT: 347376c
CMSSW: CMSSW_20_1_X_2026-08-12-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51691/55292/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test testLLVMAnalyzers had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 44
DQMHistoTests: Total histograms compared: 3435544
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3435526
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
Checked 191 log files, 159 edm output root files, 44 DQM output files
TriggerResults: no differences found

AMD_MI300X Comparison Summary

There are some workflows for which there are errors in the baseline:
37634.404 step 2
37634.7503 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially removed 149 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 47 differences found in the comparisons
DQMHistoTests: Total files compared: 6
DQMHistoTests: Total histograms compared: 134901
DQMHistoTests: Total failures: 5551
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 129350
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 5 files compared)
Checked 23 log files, 20 edm output root files, 6 DQM output files
TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 106 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 11113
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 162472
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 70 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 7879
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 165706
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: found differences in 1 / 6 workflows

NVIDIA_L4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 58 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 8503
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 165082
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: found differences in 1 / 6 workflows

NVIDIA_L40S Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 108 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 10993
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 162592
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_T4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 55 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 8439
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 165146
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 37634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid step2 max memory diff -226.1 exceeds +/- 30.0 MiB

smuzaffar · 2026-08-12T21:13:36Z

please test

cmsbuild · 2026-08-12T21:15:32Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51691/50572

cmsbuild · 2026-08-12T21:16:01Z

Pull request #51691 was updated.

cmsbuild · 2026-08-12T22:46:05Z

-1

Failed Tests: UnitTests
Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cd9a80/55294/summary.html
COMMIT: 55d1bbb
CMSSW: CMSSW_20_1_X_2026-08-12-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51691/55294/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test testLLVMAnalyzers had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 44
DQMHistoTests: Total histograms compared: 3435544
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3435526
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
Checked 191 log files, 159 edm output root files, 44 DQM output files
TriggerResults: no differences found

cmsbuild · 2026-08-13T05:54:04Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51691/50574

smuzaffar · 2026-08-13T05:54:16Z

please test

cmsbuild · 2026-08-13T05:54:33Z

Pull request #51691 was updated.

smuzaffar · 2026-08-13T09:15:50Z

please test

smuzaffar · 2026-08-13T09:32:20Z

please test

smuzaffar · 2026-08-13T09:33:32Z

please test with cms-sw/cms-bot#2825

cmsbuild · 2026-08-13T09:36:45Z

-1

Failed Tests: UnitTests RelVals-AMD_W7900
Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cd9a80/55298/summary.html
COMMIT: d52f44a
CMSSW: CMSSW_20_1_X_2026-08-12-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51691/55298/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test testLLVMAnalyzers had ERRORS

Failed RelVals-AMD_W7900

37634.40437634.404_TTbar_14TeV+Run4D127_Patatrack_PixelOnlyAlpaka_Profiling/step2_TTbar_14TeV+Run4D127_Patatrack_PixelOnlyAlpaka_Profiling.log

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 1 differences found in the comparisons
DQMHistoTests: Total files compared: 46
DQMHistoTests: Total histograms compared: 3679791
DQMHistoTests: Total failures: 76
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3679697
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
Checked 199 log files, 169 edm output root files, 46 DQM output files
TriggerResults: found differences in 1 / 45 workflows

NVIDIA_H100 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 117 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 12867
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 160718
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_L4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 109 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 11205
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 162380
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: found differences in 1 / 6 workflows

NVIDIA_L40S Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 105 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 11537
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 162048
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: found differences in 1 / 6 workflows

NVIDIA_T4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 153 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 14743
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 158842
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

smuzaffar · 2026-08-13T12:14:32Z

please test with cms-sw/cms-bot#2825

smuzaffar · 2026-08-13T13:06:53Z

please test with cms-sw/cmsdist#10784, cms-sw/cms-bot#2825

cmsbuild · 2026-08-13T20:15:05Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cd9a80/55309/summary.html
COMMIT: d52f44a
CMSSW: CMSSW_20_1_X_2026-08-13-0800/el9_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51691/55309/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cd9a80/55309/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cd9a80/55309/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 46
DQMHistoTests: Total histograms compared: 3679791
DQMHistoTests: Total failures: 11
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3679762
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
Checked 199 log files, 169 edm output root files, 46 DQM output files
TriggerResults: no differences found

smuzaffar · 2026-08-14T09:30:47Z

please test with cms-sw/cms-bot#2825

cmsbuild · 2026-08-14T12:28:31Z

-1

Failed Tests: RelVals-AMD_MI300X
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cd9a80/55340/summary.html
COMMIT: d52f44a
CMSSW: CMSSW_20_1_X_2026-08-13-2300/el9_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51691/55340/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.

Failed RelVals-AMD_MI300X

37634.75137634.751_TTbar_14TeV+Run4D127_HLT75e33TimingAlpaka/step2_TTbar_14TeV+Run4D127_HLT75e33TimingAlpaka.log
37634.750337634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid/step2_TTbar_14TeV+Run4D127_HLTHeterogeneousValid.log

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 47
DQMHistoTests: Total histograms compared: 3836054
DQMHistoTests: Total failures: 13
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3836023
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 46 files compared)
Checked 203 log files, 173 edm output root files, 47 DQM output files
TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 120 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 10016
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 163569
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_L4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 75 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 7843
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 165742
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_L40S Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 103 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 9581
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 164004
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_T4 Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 118 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 9356
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 164229
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

Max Memory Comparisons exceeding threshold NVIDIA_L4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 37634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid step2 max memory diff -49.1 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 37634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid step2 max memory diff -48.3 exceeds +/- 30.0 MiB

[Do not merge] Testing bot for auto-enabled gpu tests

347376c

cmsbuild added this to the CMSSW_20_1_X milestone Aug 12, 2026

cmsbuild added core-pending pending-signatures tests-pending orp-pending code-checks-pending labels Aug 12, 2026

cmsbuild added the requires-external label Aug 12, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Aug 12, 2026

cmsbuild added tests-started and removed tests-pending labels Aug 12, 2026

cmsbuild added tests-rejected and removed tests-started labels Aug 12, 2026

Change CUDA test dependency to 'cuda-gcc-support'

55d1bbb

cmsbuild added tests-started code-checks-pending and removed tests-rejected code-checks-approved labels Aug 12, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Aug 12, 2026

cmsbuild added tests-started code-checks-approved and removed tests-pending code-checks-pending labels Aug 13, 2026

cmsbuild removed the requires-external label Aug 13, 2026

cmsbuild added the requires-external label Aug 13, 2026

cmsbuild added tests-rejected and removed tests-started labels Aug 13, 2026

cmsbuild added tests-started and removed tests-rejected labels Aug 13, 2026

This was referenced Aug 13, 2026

support for auto enabling gpu tests cms-sw/cms-bot#2825

Open

[do not merge] testing cms-bot cms-sw/cmsdist#10784

Draft

cmsbuild added tests-approved and removed tests-started labels Aug 13, 2026

cmsbuild added tests-started and removed tests-approved labels Aug 14, 2026

cmsbuild added tests-rejected and removed tests-started labels Aug 14, 2026

Uh oh!

Conversation

smuzaffar commented Aug 12, 2026

Uh oh!

cmsbuild commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Aug 12, 2026

Uh oh!

smuzaffar commented Aug 12, 2026

Uh oh!

smuzaffar commented Aug 12, 2026

Uh oh!

cmsbuild commented Aug 12, 2026

Failed Unit Tests

Comparison Summary

AMD_MI300X Comparison Summary

AMD_W7900 Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L4 Comparison Summary

NVIDIA_L40S Comparison Summary

NVIDIA_T4 Comparison Summary

Max Memory Comparisons exceeding threshold NVIDIA_T4

Uh oh!

smuzaffar commented Aug 12, 2026

Uh oh!

cmsbuild commented Aug 12, 2026

Uh oh!

cmsbuild commented Aug 12, 2026

Uh oh!

cmsbuild commented Aug 12, 2026

Failed Unit Tests

Comparison Summary

Uh oh!

cmsbuild commented Aug 13, 2026

Uh oh!

smuzaffar commented Aug 13, 2026

Uh oh!

cmsbuild commented Aug 13, 2026

Uh oh!

smuzaffar commented Aug 13, 2026

Uh oh!

smuzaffar commented Aug 13, 2026

Uh oh!

smuzaffar commented Aug 13, 2026

Uh oh!

cmsbuild commented Aug 13, 2026

Failed Unit Tests

Failed RelVals-AMD_W7900

Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L4 Comparison Summary

NVIDIA_L40S Comparison Summary

NVIDIA_T4 Comparison Summary

Uh oh!

smuzaffar commented Aug 13, 2026

Uh oh!

smuzaffar commented Aug 13, 2026

Uh oh!

cmsbuild commented Aug 13, 2026

Comparison Summary

Uh oh!

smuzaffar commented Aug 14, 2026

Uh oh!

cmsbuild commented Aug 14, 2026

Failed RelVals-AMD_MI300X

Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L4 Comparison Summary

NVIDIA_L40S Comparison Summary

NVIDIA_T4 Comparison Summary

Max Memory Comparisons exceeding threshold NVIDIA_L4

Max Memory Comparisons exceeding threshold NVIDIA_T4

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

cmsbuild commented Aug 12, 2026 •

edited

Loading