[GPU-Nvidia] nvidia_t4,nvidia_h100,nvidia_l4,nvidia_ls40s failing

[raoatifshad]

GPU NVIDIA tests for PhysicsTools/PyTorch are failing in testModelJit logs

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @raoatifshad.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign ml, heterogeneous

[cmsbuild]

New categories assigned: ml,heterogeneous

@fwyzard,@hjkwon260,@makortel,@valsdav,@y19y19 you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

Error is

===== Test "testModelJit" ====
Running ..Running test on device cuda:0
..Running test on device cuda:0
testModelJit: src/PhysicsTools/PyTorch/interface/Model.h:31: void cms::torch::Model::to(c10::Device, bool): Assertion `!is_frozen_ && "Model is frozen, cannot be moved to another device!"' failed.
timeout: the monitored command dumped core
/bin/sh: line 1: 507501 Aborted                 timeout 3600 sh -c 'testModelJit '

---> test testModelJit had ERRORS

[smuzaffar]

#50790 should fix the unit test. I also have created a PR with simple fix at #50927