support for auto enabling gpu tests

[smuzaffar]

No description provided.

[smuzaffar]

please test

[cmsbuild]

A new Pull Request was created by @smuzaffar for branch master.

@akritkbehera, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[smuzaffar]

please test with cms-sw/cmssw#51689

[cmsbuild]

Pull request #2825 was updated.

[cmsbuild]

Pull request #2825 was updated.

[cmsbuild]

Pull request #2825 was updated.

[smuzaffar]

please test

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b45c27/55288/summary.html
COMMIT: 7b3b892
CMSSW: CMSSW_20_0_X_2026-08-12-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2825/55288/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test testLLVMAnalyzers had ERRORS

Comparison Summary

Summary:

• No significant changes to the logs found
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 44
• DQMHistoTests: Total histograms compared: 2641501
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2641483
• DQMHistoTests: Total skipped: 18
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
• Checked 168 log files, 141 edm output root files, 44 DQM output files
• TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 0
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 0
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
• Checked 0 log files, 0 edm output root files, 7 DQM output files
• TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

AMD_W7900 Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 0
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 0
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
• Checked 0 log files, 0 edm output root files, 7 DQM output files
• TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

NVIDIA_H100 Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 0
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 0
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
• Checked 0 log files, 0 edm output root files, 7 DQM output files
• TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

NVIDIA_L4 Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 0
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 0
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
• Checked 0 log files, 0 edm output root files, 7 DQM output files
• TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

NVIDIA_L40S Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 0
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 0
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
• Checked 0 log files, 0 edm output root files, 7 DQM output files
• TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

NVIDIA_T4 Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 0
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 0
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
• Checked 0 log files, 0 edm output root files, 7 DQM output files
• TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

[cmsbuild]

Pull request #2825 was updated.

[cmsbuild]

Pull request #2825 was updated.

[smuzaffar]

please test with cms-sw/cmssw#51691

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b45c27/55312/summary.html
COMMIT: 677f2e1
CMSSW: CMSSW_20_1_X_2026-08-13-0800/el9_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2825/55312/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 44
• DQMHistoTests: Total histograms compared: 3435544
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3435526
• DQMHistoTests: Total skipped: 18
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
• Checked 191 log files, 159 edm output root files, 44 DQM output files
• TriggerResults: no differences found

[smuzaffar]

please test with cms-sw/cmssw#51691

[cmsbuild]

Pull request #2825 was updated.

[cmsbuild]

-1

Failed Tests: RelVals-AMD_W7900
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b45c27/55321/summary.html
COMMIT: 2eca285
CMSSW: CMSSW_20_1_X_2026-08-13-0800/el9_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2825/55321/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-AMD_W7900

• 37634.40437634.404_TTbar_14TeV+Run4D127_Patatrack_PixelOnlyAlpaka_Profiling/step2_TTbar_14TeV+Run4D127_Patatrack_PixelOnlyAlpaka_Profiling.log

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 44
• DQMHistoTests: Total histograms compared: 3435544
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3435526
• DQMHistoTests: Total skipped: 18
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
• Checked 191 log files, 159 edm output root files, 44 DQM output files
• TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 121 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 173585
• DQMHistoTests: Total failures: 9835
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 163750
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 25 log files, 20 edm output root files, 7 DQM output files
• TriggerResults: no differences found

NVIDIA_L4 Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 90 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 173585
• DQMHistoTests: Total failures: 8992
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 164593
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 25 log files, 20 edm output root files, 7 DQM output files
• TriggerResults: no differences found

NVIDIA_L40S Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 82 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 173585
• DQMHistoTests: Total failures: 8596
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 164989
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 25 log files, 20 edm output root files, 7 DQM output files
• TriggerResults: no differences found

NVIDIA_T4 Comparison Summary

Summary:

• You potentially removed 2 lines from the logs
• Reco comparison results: 97 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 173585
• DQMHistoTests: Total failures: 8098
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 165487
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 25 log files, 20 edm output root files, 7 DQM output files
• TriggerResults: no differences found

Max Memory Comparisons exceeding threshold NVIDIA_H100

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

• Error: Workflow 37634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid step2 max memory diff 235.0 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_L4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

• Error: Workflow 37634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid step2 max memory diff -39.0 exceeds +/- 30.0 MiB

[smuzaffar]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b45c27/55339/summary.html
COMMIT: 2eca285
CMSSW: CMSSW_20_1_X_2026-08-13-2300/el9_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2825/55339/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 2 lines from the logs
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 45
• DQMHistoTests: Total histograms compared: 3591807
• DQMHistoTests: Total failures: 9
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3591780
• DQMHistoTests: Total skipped: 18
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
• Checked 195 log files, 163 edm output root files, 45 DQM output files
• TriggerResults: no differences found