SoA Schema evolution by Electricks94 · Pull Request #50691 · cms-sw/cmssw

Electricks94 · 2026-04-08T14:14:34Z

PR description:

Extension of the custom streamer of the SoA Backend to encounter for the following cases:

Added columns/eigen columns/scalars are initialised to 0
Removed or rearranged columns are handle correctly by ROOT
columns with changed type (e.g. double to float) are handle by ROOTm this holds also for Eigen columns and scalars
Eigen columns with changed dimensions for example Eigen::Matrix<float, 4, 2> -> Eigen::Matrix<float, 3, 2>; result in a meaningfull error at reading
Columns with enum types are stored as integers in the SoA backend and only exposed as enums. Hence, the schema evolution behaves exactly like in the case of integers and also 8 byte large enums are supported which is not case with ROOT normally (see Add support for enums with non-default size root-project/root#17009)
Examples using ioread rules are provided to showcase how complex cases (e.g. using custom types for SoA columns) can be handled

PR validation:

A set of SoA Layouts is provided which evolve from a base SoA Layout. The base SoA Layout is stored multiple times using a different type alias which is then changed in the code to simulate an evolving class. The created ROOT files are then used to validate the reading of the previously stored Collections.

This PR is an extension of #50487

@felicepantaleo fyi

Electricks94 · 2026-04-08T14:15:01Z

type ngt

cmsbuild · 2026-04-08T14:15:10Z

cms-bot internal usage

cmsbuild · 2026-04-08T14:17:16Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50691/48929

Found files with invalid states:
- DataFormats/PortableTestObjects/test/SoAReader_cfg.py:
  - Added: 0f58329
  - Deleted: c11fc86
- DataFormats/PortableTestObjects/test/SoAWriter_cfg.py:
  - Added: 0f58329
  - Deleted: c11fc86
- DataFormats/PortableTestObjects/plugins/SchemaEvolutionAnalyzer.cc:
  - Added: 0f58329
  - Modified: dc9ab48, 94880ed
  - Deleted: c11fc86
- DataFormats/PortableTestObjects/plugins/SchemaEvolutionProducer.cc:
  - Added: 0f58329
  - Modified: dc9ab48, 94880ed
  - Deleted: c11fc86
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/SoATemplate/interface/SoACommon.h modified in PR(s): Various fixes for heterogeneous utilities #47605, Added AoSWrapper and transposition function #49072
- File DataFormats/SoATemplate/interface/SoALayout.h modified in PR(s): Various fixes for heterogeneous utilities #47605, Added AoSWrapper and transposition function #49072, [Draft] Allow users to set a valid data range for the SoA View #49204, Extend SoA read streamer to support schema evolution #50487
- File DataFormats/SoATemplate/test/SoAUnitTests.cc modified in PR(s): [Draft] Allow users to set a valid data range for the SoA View #49204

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50691/48929/code-checks.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50691/48929/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly
code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50691/48929/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50691/48929/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2026-04-09T11:05:46Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50691/48940

Found files with invalid states:
- DataFormats/PortableTestObjects/test/SoAReader_cfg.py:
  - Added: 0f58329
  - Deleted: c11fc86
- DataFormats/PortableTestObjects/plugins/SchemaEvolutionAnalyzer.cc:
  - Added: 0f58329
  - Modified: dc9ab48, 94880ed
  - Deleted: c11fc86
- DataFormats/PortableTestObjects/plugins/SchemaEvolutionProducer.cc:
  - Added: 0f58329
  - Modified: dc9ab48, 94880ed
  - Deleted: c11fc86
- DataFormats/PortableTestObjects/test/SoAWriter_cfg.py:
  - Added: 0f58329
  - Deleted: c11fc86
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Portable/README.md modified in PR(s): Added AoSWrapper and transposition function #49072
- File DataFormats/SoATemplate/interface/SoACommon.h modified in PR(s): Various fixes for heterogeneous utilities #47605, Added AoSWrapper and transposition function #49072
- File DataFormats/SoATemplate/interface/SoALayout.h modified in PR(s): Various fixes for heterogeneous utilities #47605, Added AoSWrapper and transposition function #49072, [Draft] Allow users to set a valid data range for the SoA View #49204, Extend SoA read streamer to support schema evolution #50487
- File DataFormats/SoATemplate/test/SoAUnitTests.cc modified in PR(s): [Draft] Allow users to set a valid data range for the SoA View #49204

cmsbuild · 2026-04-09T11:06:09Z

Pull request #50691 was updated.

cmsbuild · 2026-05-04T13:46:25Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50691/49225

Found files with invalid states:
- DataFormats/PortableTestObjects/test/SoAReader_cfg.py:
  - Added: 9da2892
  - Deleted: d0d32d2
- DataFormats/PortableTestObjects/plugins/SchemaEvolutionAnalyzer.cc:
  - Added: 9da2892
  - Modified: 554364a, f4b6c19
  - Deleted: d0d32d2
- DataFormats/PortableTestObjects/plugins/SchemaEvolutionProducer.cc:
  - Added: 9da2892
  - Modified: 554364a, f4b6c19
  - Deleted: d0d32d2
- DataFormats/PortableTestObjects/test/SoAWriter_cfg.py:
  - Added: 9da2892
  - Deleted: d0d32d2
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Portable/README.md modified in PR(s): Added AoSWrapper and transposition function #49072
- File DataFormats/PortableTestObjects/src/classes_def.xml modified in PR(s): Add Torch FP16 inference support #50790
- File DataFormats/SoATemplate/interface/SoACommon.h modified in PR(s): Various fixes for heterogeneous utilities #47605, Added AoSWrapper and transposition function #49072
- File DataFormats/SoATemplate/interface/SoALayout.h modified in PR(s): Various fixes for heterogeneous utilities #47605, Added AoSWrapper and transposition function #49072, [Draft] Allow users to set a valid data range for the SoA View #49204, Extend SoA read streamer to support schema evolution #50487
- File DataFormats/SoATemplate/test/SoAUnitTests.cc modified in PR(s): [Draft] Allow users to set a valid data range for the SoA View #49204

cmsbuild · 2026-05-04T13:46:48Z

Pull request #50691 was updated.

cmsbuild · 2026-06-24T11:29:12Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50691/49918

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Portable/README.md modified in PR(s): Added AoSWrapper and transposition function #49072
- File DataFormats/SoATemplate/interface/SoACommon.h modified in PR(s): Various fixes for heterogeneous utilities #47605, Added AoSWrapper and transposition function #49072, Add source location to SoA out of range error messages #50888
- File DataFormats/SoATemplate/interface/SoALayout.h modified in PR(s): Various fixes for heterogeneous utilities #47605, Added AoSWrapper and transposition function #49072, [Draft] Allow users to set a valid data range for the SoA View #49204, Add source location to SoA out of range error messages #50888
- File DataFormats/SoATemplate/test/SoAUnitTests.cc modified in PR(s): [Draft] Allow users to set a valid data range for the SoA View #49204

fwyzard · 2026-06-24T11:29:35Z

please test

cmsbuild · 2026-06-24T11:29:55Z

Pull request #50691 was updated. @fwyzard, @makortel can you please check and sign again.

Electricks94 · 2026-06-24T14:39:55Z

I see now PrologueTest and PrologueTest_multiblock failing in the unit tests but these tests also seem to fail in the master branch without the changes proposed here. Is this a known problem?

cmsbuild · 2026-06-24T14:46:50Z

-1

Failed Tests: RelVals-AMD_W7900 amd_mi300xUnitTests amd_w7900UnitTests
Size: This PR adds an extra 128KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0cee1e/54249/summary.html
COMMIT: 49d05a0
CMSSW: CMSSW_20_1_X_2026-06-23-2300/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/50691/54249/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-AMD_W7900

34434.40434434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling/step2_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling.log

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3414477
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3414459
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
Checked 195 log files, 163 edm output root files, 45 DQM output files
TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 83 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 11825
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 155182
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: found differences in 3 / 6 workflows

NVIDIA_L40S Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 64 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 10378
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 156629
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: found differences in 4 / 6 workflows

NVIDIA_T4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 95 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 17493
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 149514
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: found differences in 3 / 6 workflows

fwyzard · 2026-06-24T14:59:17Z

I see now PrologueTest and PrologueTest_multiblock failing in the unit tests but these tests also seem to fail in the master branch without the changes proposed here. Is this a known problem?

Yes, they should be fixed by #51201.

fwyzard · 2026-06-24T14:59:32Z

please test with #51201

cmsbuild · 2026-06-24T18:24:43Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0cee1e/54252/summary.html
COMMIT: 49d05a0
CMSSW: CMSSW_20_1_X_2026-06-24-1100/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/50691/54252/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3414477
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3414459
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
Checked 195 log files, 163 edm output root files, 45 DQM output files
TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 58 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 12867
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 154140
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: found differences in 3 / 6 workflows

AMD_W7900 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 67 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 12830
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 154177
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: found differences in 3 / 6 workflows

NVIDIA_H100 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 76 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 13801
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 153206
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: found differences in 3 / 6 workflows

NVIDIA_L40S Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 52 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 12079
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 154928
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: found differences in 3 / 6 workflows

NVIDIA_T4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 56 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 12868
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 154139
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: found differences in 2 / 6 workflows

fwyzard · 2026-06-24T21:56:47Z

+heterogeneous

fwyzard · 2026-06-25T13:26:58Z

The new package is added by cms-sw/cms-bot#2798 .

fwyzard · 2026-06-26T10:35:03Z

@smuzaffar can you merge cms-sw/cms-bot#2798 to make the new package known ?

Electricks94 · 2026-08-13T08:46:41Z

Friendly reminder: I think this PR got forgotten.

cmsbuild · 2026-08-13T08:47:07Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @sextonkennedy, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2026-08-14T06:54:55Z

+1

cmsbuild added this to the CMSSW_17_0_X milestone Apr 8, 2026

cmsbuild added pending-signatures tests-pending orp-pending code-checks-pending heterogeneous-pending changes-dataformats ngt labels Apr 8, 2026

Electricks94 marked this pull request as draft April 8, 2026 14:15

cmsbuild added code-checks-rejected code-checks-pending and removed code-checks-pending code-checks-rejected labels Apr 8, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 9, 2026

This was referenced Apr 20, 2026

PyTorchAlpaka: Batched inference with TensorCollections, eval and frozen model, FP16 convsersion support #50498

Closed

Add Torch FP16 inference support #50790

Merged

Electricks94 force-pushed the SoASerialisation branch from a61a4a7 to aeef38a Compare May 4, 2026 13:44

cmsbuild added code-checks-pending and removed code-checks-approved labels May 4, 2026

Electricks94 changed the title ~~[Draft] SoA Schema evolution~~ SoA Schema evolution May 4, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels May 4, 2026

cmsbuild added tests-pending code-checks-pending and removed code-checks-approved labels Jun 24, 2026

cmsbuild added tests-started code-checks-approved and removed tests-pending code-checks-pending labels Jun 24, 2026

cmsbuild added tests-rejected and removed tests-started labels Jun 24, 2026

This was referenced Jun 25, 2026

[Backport to 20_0_X (#50691)] SoA Schema evolution #51315

Merged

[Backport to 17_0_X (#50691)] SoA Schema evolution #51316

Merged

fwyzard mentioned this pull request Jul 13, 2026

[RelVal] Failures at FEVTDEBUGHLT Merge Step #49968

Open

This was referenced Jul 20, 2026

Remove explicit implementation of SoAConstParametersImpl #51528

Merged

Nested SoABlocks #51554

Merged

cmsbuild mentioned this pull request Aug 3, 2026

Improve SoATemplate README #51615

Merged

cmsbuild mentioned this pull request Aug 11, 2026

Introspection for SoABlocks #51670

Open

Uh oh!

Conversation

Electricks94 commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Electricks94 commented Apr 8, 2026

Uh oh!

cmsbuild commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Apr 8, 2026

Uh oh!

cmsbuild commented Apr 9, 2026

Uh oh!

cmsbuild commented Apr 9, 2026

Uh oh!

cmsbuild commented May 4, 2026

Uh oh!

cmsbuild commented May 4, 2026

Uh oh!

cmsbuild commented Jun 24, 2026

Uh oh!

fwyzard commented Jun 24, 2026

Uh oh!

cmsbuild commented Jun 24, 2026

Uh oh!

Electricks94 commented Jun 24, 2026

Uh oh!

cmsbuild commented Jun 24, 2026

Failed RelVals-AMD_W7900

Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L40S Comparison Summary

NVIDIA_T4 Comparison Summary

Uh oh!

fwyzard commented Jun 24, 2026

Uh oh!

fwyzard commented Jun 24, 2026

Uh oh!

cmsbuild commented Jun 24, 2026

Comparison Summary

AMD_MI300X Comparison Summary

AMD_W7900 Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L40S Comparison Summary

NVIDIA_T4 Comparison Summary

Uh oh!

fwyzard commented Jun 24, 2026

Uh oh!

fwyzard commented Jun 25, 2026

Uh oh!

fwyzard commented Jun 26, 2026

Uh oh!

Electricks94 commented Aug 13, 2026

Uh oh!

cmsbuild commented Aug 13, 2026

Uh oh!

mandrenguyen commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Electricks94 commented Apr 8, 2026 •

edited

Loading

cmsbuild commented Apr 8, 2026 •

edited

Loading