Skip to content

Add Torch FP16 inference support - #50790

Merged
cmsbuild merged 1 commit into
cms-sw:masterfrom
EmanueleCoradin:torch-FP16-support
May 22, 2026
Merged

Add Torch FP16 inference support#50790
cmsbuild merged 1 commit into
cms-sw:masterfrom
EmanueleCoradin:torch-FP16-support

Conversation

@EmanueleCoradin

@EmanueleCoradin EmanueleCoradin commented Apr 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

This PR results from a work in collaboration with @valsdav.

PR description:

This PR introduces FP16 (half precision) inference support in the PyTorch Alpaka pipeline, alongside the existing FP32 execution path.
FP16 results are expected to be close to FP32, but not identical due to reduced precision.

Runtime FP16 conversion

Input data is stored in SoA format with full FP32 precision.
Conversion to FP16 happens at inference time via the PyTorch API, enabled by passing torch::kHalf to model.forward(...).

PR Validation

A small inspection module has been extended to compare FP32 and runtime FP16 outputs.
The module checks that the relative difference between FP32 and FP16 output is less then 1%.

They can be run by:

cd PhysicsTools/PyTorchAlpakaTest/test
cmsRun runPyTorchAlpakaTest.py -e 3

@EmanueleCoradin

EmanueleCoradin commented Apr 22, 2026

Copy link
Copy Markdown
Contributor Author

type ngt

@cmsbuild cmsbuild added this to the CMSSW_17_0_X milestone Apr 22, 2026
@cmsbuild

cmsbuild commented Apr 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

cms-bot internal usage

@cmsbuild

cmsbuild commented Apr 22, 2026

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50790/49102

@cmsbuild cmsbuild mentioned this pull request Apr 23, 2026
Merged
@cmsbuild

cmsbuild commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50790/49111

@cmsbuild

cmsbuild commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

Pull request #50790 was updated.

@cmsbuild

cmsbuild commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50790/49113

@cmsbuild

cmsbuild commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

Pull request #50790 was updated.

@mmusich

mmusich commented May 9, 2026

Copy link
Copy Markdown
Contributor

allow @EmanueleCoradin test rights

@mmusich

mmusich commented May 9, 2026

Copy link
Copy Markdown
Contributor

@cmsbuild, please test

@cmsbuild

cmsbuild commented May 9, 2026

Copy link
Copy Markdown
Contributor

-1

Failed Tests: RelVals-AMD_W7900
Size: This PR adds an extra 32KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f0926e/53162/summary.html
COMMIT: 194b4c8
CMSSW: CMSSW_17_0_X_2026-05-09-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50790/53162/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-AMD_W7900

The relvals timed out after 4 hours.

Comparison Summary

Summary:

AMD_MI300X Comparison Summary

There are some workflows for which there are errors in the baseline:
34634.403 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

  • You potentially removed 81 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 341 differences found in the comparisons
  • DQMHistoTests: Total files compared: 12
  • DQMHistoTests: Total histograms compared: 200180
  • DQMHistoTests: Total failures: 25200
  • DQMHistoTests: Total nulls: 35
  • DQMHistoTests: Total successes: 174945
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 11 files compared)
  • Checked 47 log files, 48 edm output root files, 12 DQM output files
  • TriggerResults: found differences in 2 / 11 workflows

NVIDIA_H100 Comparison Summary

Summary:

NVIDIA_L40S Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 348 differences found in the comparisons
  • DQMHistoTests: Total files compared: 13
  • DQMHistoTests: Total histograms compared: 216259
  • DQMHistoTests: Total failures: 27938
  • DQMHistoTests: Total nulls: 33
  • DQMHistoTests: Total successes: 188288
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
  • Checked 49 log files, 50 edm output root files, 13 DQM output files
  • TriggerResults: found differences in 2 / 12 workflows

@EmanueleCoradin

EmanueleCoradin commented May 9, 2026

Copy link
Copy Markdown
Contributor Author

This time the failure looks unrelated

@EmanueleCoradin

EmanueleCoradin commented May 9, 2026

Copy link
Copy Markdown
Contributor Author

@cmsbuild, please test

@cmsbuild

cmsbuild commented May 10, 2026

Copy link
Copy Markdown
Contributor

-1

Failed Tests: RelVals-AMD_MI300X
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f0926e/53164/summary.html
COMMIT: 194b4c8
CMSSW: CMSSW_17_0_X_2026-05-09-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50790/53164/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-AMD_MI300X

The relvals timed out after 4 hours.

Comparison Summary

Summary:

  • You potentially removed 1 lines from the logs
  • Reco comparison results: 4 differences found in the comparisons
  • DQMHistoTests: Total files compared: 53
  • DQMHistoTests: Total histograms compared: 4187168
  • DQMHistoTests: Total failures: 9
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4187139
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
  • Checked 227 log files, 197 edm output root files, 53 DQM output files
  • TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

NVIDIA_H100 Comparison Summary

Summary:

  • You potentially added 5 lines to the logs
  • Reco comparison results: 345 differences found in the comparisons
  • DQMHistoTests: Total files compared: 13
  • DQMHistoTests: Total histograms compared: 216259
  • DQMHistoTests: Total failures: 36583
  • DQMHistoTests: Total nulls: 32
  • DQMHistoTests: Total successes: 179644
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
  • Checked 49 log files, 50 edm output root files, 13 DQM output files
  • TriggerResults: found differences in 3 / 12 workflows

NVIDIA_L40S Comparison Summary

Summary:

@EmanueleCoradin

EmanueleCoradin commented May 11, 2026

Copy link
Copy Markdown
Contributor Author

@cmsbuild, please test

@cmsbuild

cmsbuild commented May 11, 2026

Copy link
Copy Markdown
Contributor

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f0926e/53183/summary.html
COMMIT: 194b4c8
CMSSW: CMSSW_17_0_X_2026-05-11-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/50790/53183/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 3 lines to the logs
  • Reco comparison results: 5 differences found in the comparisons
  • DQMHistoTests: Total files compared: 53
  • DQMHistoTests: Total histograms compared: 4187168
  • DQMHistoTests: Total failures: 6
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4187142
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
  • Checked 227 log files, 197 edm output root files, 53 DQM output files
  • TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 381 differences found in the comparisons
  • DQMHistoTests: Total files compared: 13
  • DQMHistoTests: Total histograms compared: 216259
  • DQMHistoTests: Total failures: 32196
  • DQMHistoTests: Total nulls: 35
  • DQMHistoTests: Total successes: 184028
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
  • Checked 49 log files, 50 edm output root files, 13 DQM output files
  • TriggerResults: found differences in 1 / 12 workflows

AMD_W7900 Comparison Summary

There are some workflows for which there are errors in the baseline:
34634.751 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

  • You potentially added 39 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 328 differences found in the comparisons
  • DQMHistoTests: Total files compared: 12
  • DQMHistoTests: Total histograms compared: 184046
  • DQMHistoTests: Total failures: 31853
  • DQMHistoTests: Total nulls: 15
  • DQMHistoTests: Total successes: 152178
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 11 files compared)
  • Checked 48 log files, 50 edm output root files, 12 DQM output files
  • TriggerResults: found differences in 1 / 11 workflows

NVIDIA_H100 Comparison Summary

Summary:

NVIDIA_L40S Comparison Summary

Summary:

@EmanueleCoradin

EmanueleCoradin commented May 12, 2026

Copy link
Copy Markdown
Contributor Author

Everything should be in order now.
@cms-sw/ml-l2 @cms-sw/heterogeneous-l2 do you have any further comments or reviews?

@cmsbuild cmsbuild mentioned this pull request May 13, 2026
Merged
@smuzaffar smuzaffar mentioned this pull request May 13, 2026
Open
@smuzaffar

smuzaffar commented May 13, 2026

Copy link
Copy Markdown
Contributor

@EmanueleCoradin , following the issue #50923 , i created #50927 to fix the failign unit test. I see that this PR also fixes the failing unit test.

About the change in #50783 (auto_freeze), I have proposed/asked (in #50927)

For cms::torch::Model constructor , we should call the freeze() if auto_freeze is true. Currently model become frozen only after calling first Model.to(dev) . If the intend is that if a model is created with auto_freeze then it should not be allowed to move to different device then may be we should call freeze() in the constructor

@EmanueleCoradin

EmanueleCoradin commented May 13, 2026

Copy link
Copy Markdown
Contributor Author

@smuzaffar , thanks for the information. I reply to the proposal in #50927 👍

@mmusich

mmusich commented May 19, 2026

Copy link
Copy Markdown
Contributor

@EmanueleCoradin this branch has conflicts that must be resolved.

@cmsbuild

cmsbuild commented May 19, 2026

Copy link
Copy Markdown
Contributor

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50790/49390

ERROR: Build errors found during clang-tidy run.

src/PhysicsTools/PyTorch/interface/Model.h:31:7: error: expected expression [clang-diagnostic-error]
   31 |       TORCH_CHECK(!is_frozen_ && "Model is frozen, cannot be moved to another device!");
      |       ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02942/el8_amd64_gcc13/external/py3-torch/2.11.0-49cae687339be021473bf2d2521fdda0/include/c10/util/Exception.h:593:3: note: expanded from macro 'TORCH_CHECK'
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

@cmsbuild

cmsbuild commented May 19, 2026

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50790/49391

@cmsbuild

cmsbuild commented May 19, 2026

Copy link
Copy Markdown
Contributor

Pull request #50790 was updated. @cmsbuild, @fwyzard, @hjkwon260, @makortel, @valsdav, @y19y19 can you please check and sign again.

@EmanueleCoradin

EmanueleCoradin commented May 19, 2026

Copy link
Copy Markdown
Contributor Author

@cmsbuild, please test

@cmsbuild

cmsbuild commented May 19, 2026

Copy link
Copy Markdown
Contributor

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f0926e/53332/summary.html
COMMIT: 0875d0c
CMSSW: CMSSW_17_0_X_2026-05-19-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50790/53332/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 4 lines to the logs
  • Reco comparison results: 6 differences found in the comparisons
  • DQMHistoTests: Total files compared: 53
  • DQMHistoTests: Total histograms compared: 4198477
  • DQMHistoTests: Total failures: 26
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4198431
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
  • Checked 227 log files, 197 edm output root files, 53 DQM output files
  • TriggerResults: no differences found

AMD_MI300X Comparison Summary

There are some workflows for which there are errors in the baseline:
34634.403 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

  • You potentially removed 119 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 319 differences found in the comparisons
  • DQMHistoTests: Total files compared: 12
  • DQMHistoTests: Total histograms compared: 202640
  • DQMHistoTests: Total failures: 20246
  • DQMHistoTests: Total nulls: 34
  • DQMHistoTests: Total successes: 182360
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 11 files compared)
  • Checked 47 log files, 48 edm output root files, 12 DQM output files
  • TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

NVIDIA_H100 Comparison Summary

Summary:

  • You potentially added 7 lines to the logs
  • Reco comparison results: 334 differences found in the comparisons
  • DQMHistoTests: Total files compared: 13
  • DQMHistoTests: Total histograms compared: 218719
  • DQMHistoTests: Total failures: 29224
  • DQMHistoTests: Total nulls: 32
  • DQMHistoTests: Total successes: 189463
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
  • Checked 49 log files, 50 edm output root files, 13 DQM output files
  • TriggerResults: found differences in 1 / 12 workflows

NVIDIA_L40S Comparison Summary

Summary:

fwyzard
fwyzard reviewed May 20, 2026
View reviewed changes
Comment thread PhysicsTools/PyTorchAlpakaTest/plugins/alpaka/SimpleNet.cc Outdated
@EmanueleCoradin

EmanueleCoradin commented May 20, 2026

Copy link
Copy Markdown
Contributor Author

@cmsbuild, please test

@cmsbuild

cmsbuild commented May 20, 2026

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50790/49425

@cmsbuild

cmsbuild commented May 20, 2026

Copy link
Copy Markdown
Contributor

Pull request #50790 was updated. @fwyzard, @hjkwon260, @makortel, @valsdav, @y19y19 can you please check and sign again.

@cmsbuild

cmsbuild commented May 20, 2026

Copy link
Copy Markdown
Contributor

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f0926e/53398/summary.html
COMMIT: 74b02ab
CMSSW: CMSSW_17_0_X_2026-05-20-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50790/53398/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 4 lines to the logs
  • Reco comparison results: 6 differences found in the comparisons
  • DQMHistoTests: Total files compared: 53
  • DQMHistoTests: Total histograms compared: 4198477
  • DQMHistoTests: Total failures: 65
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4198392
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
  • Checked 227 log files, 197 edm output root files, 53 DQM output files
  • TriggerResults: no differences found

AMD_MI300X Comparison Summary

There are some workflows for which there are errors in the baseline:
34634.402 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

  • You potentially removed 79 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 340 differences found in the comparisons
  • DQMHistoTests: Total files compared: 12
  • DQMHistoTests: Total histograms compared: 203010
  • DQMHistoTests: Total failures: 22532
  • DQMHistoTests: Total nulls: 38
  • DQMHistoTests: Total successes: 180440
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 11 files compared)
  • Checked 47 log files, 48 edm output root files, 12 DQM output files
  • TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

NVIDIA_H100 Comparison Summary

Summary:

NVIDIA_L40S Comparison Summary

Summary:

@fwyzard

fwyzard commented May 21, 2026

Copy link
Copy Markdown
Contributor

+1

@mmusich

mmusich commented May 21, 2026

Copy link
Copy Markdown
Contributor

@cms-sw/ml-l2 kind ping.

@valsdav

valsdav commented May 21, 2026

Copy link
Copy Markdown
Contributor

+ml

@cmsbuild

cmsbuild commented May 21, 2026

Copy link
Copy Markdown
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @sextonkennedy, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

@mandrenguyen

mandrenguyen commented May 22, 2026

Copy link
Copy Markdown
Contributor

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers
@makortel makortel makortel left review comments
@fwyzard fwyzard fwyzard left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

allow-EmanueleCoradin code-checks-approved fully-signed heterogeneous-approved ml-approved ngt orp-approved tests-approved

Projects

None yet

Milestone

CMSSW_17_0_X

Development

Successfully merging this pull request may close these issues.

8 participants

@EmanueleCoradin @cmsbuild @mmusich @smuzaffar @fwyzard @valsdav @mandrenguyen @makortel