Skip to content

SoA Schema evolution #50691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cmsbuild merged 10 commits into from
Aug 14, 2026
Merged

SoA Schema evolution #50691

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
b3c0b61
Extend SoA read streamer to support schema evolution
fwyzard Mar 20, 2026
7b461ab
Start working on test
Electricks94 Mar 27, 2026
52f70df
Added check for Eigen type evolution
Electricks94 Mar 27, 2026
dcb5321
Store enum as integer in SoA backend
Electricks94 Apr 2, 2026
6ab4cdc
Added test case
Electricks94 Apr 8, 2026
2b7efa3
Added test cases and documentation to SoA schema evolution
Electricks94 Apr 9, 2026
833334f
Extended Schema Evolution with custom io rules
Electricks94 May 4, 2026
f04bc90
Extended Documentation
Electricks94 Jun 5, 2026
3b58f93
Move Schemaevolution tests to HeterogeniousCore
Electricks94 Jun 24, 2026
49d05a0
made README in DataFormats Portable clearer
Electricks94 Jun 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 69 additions & 1 deletion DataFormats/Portable/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,4 +178,72 @@ For the host xml, all SoA layouts have to be listed. The scripts are called as f
./DataFormats/Portable/scripts/portableDeviceCollectionHints portabletest::TestHostMultiCollection3
```
The layouts should not be added as parameters for the device collection. Those script can be used equally with the
single layout collections or multi layout collections.
single layout collections or multi layout collections.


## Schema Evolution of SoA Layouts

`ROOT` files written using a `PortableCollection` with a specific `SoA layout` are read through a custom streamer.
This streamer copies data into the SoA buffer and relies heavily on ROOT’s built-in schema evolution mechanisms when the layout definition changes in the CMSSW codebase.
The schema evolution behavior is summarized below:

### General Behavior
- **Removed columns**: Columns that exist in the file but are no longer present in the current layout are skipped during reading. Their data is not loaded into memory.
- **Added columns**: Columns that are present in the current layout but missing from the file are default-initialized to **zero**.
### Fundamental datatype changes
When the datatype of a column changes between versions, `ROOT` performs an element-wise conversion during reading, for example:
```
double → float
```
In this case, values are cast to the new type as they are read.
**Important notes:**
- Conversions are performed silently.
- Precision loss (e.g. truncation) is **not reported** by default.
- Overflow and underflow may occur if the new type cannot represent the original values.
### Eigen Columns
- **Supported evolution**: Eigen-based columns can only evolve with respect to their **underlying scalar type**, for example:
```
Eigen::Matrix<float, ...> -> Eigen::Matrix<double, ...>
```
- **Unsupported changes**: Changes to the matrix shape (number of rows or columns) are **not supported** and will result in a runtime error during reading.
### Enum Types
- Columns defined with `enum` types are fully supported.
- They behave identically to columns defined with the enum’s **underlying integer type**:
- Schema evolution follows the same rules as for fundamental integer types.
- Changing the underlying type (e.g. `uint16_t` → `uint32_t`) is supported via implicit conversion.

### User-defined Types

In the following user-defined types refers to non-fundamental types for example user-defined structs or classes.
Columns containing user-defined structs or classes require explicit I/O schema evolution rules.

To support schema evolution for a column of a user-defined type:

1. A class version and checksum must be defined in the SoA layout dictionary.
2. An I/O read rule must be provided that describes how data should be converted between schema versions.

Examples can be found in `HeterogeneousCore/TestModules/src/classes_def.xml`, where schema evolution rules are implemented for `edm::StdArray`.

When defining an I/O read rule:

* Both the source and target columns must be specified.
* The source-side leaf counter called `elements_` must be accessed and is used to determine the size of the temporary conversion buffer.

#### Known Limitations

##### Removing User-defined type Columns

Removing a column that contains a non-fundamental type consistently triggers a ROOT error during reading. See ROOT issue [#22097](https://github.com/root-project/root/issues/22097).

##### I/O Read Rules for SCALAR Columns

I/O read rules can also be defined for `SCALAR` columns, but they require special handling:

* The leaf counter `scalar_` must be included in the rule, even though its value is always `1`.
* Omitting this leaf counter will trigger and error from ROOT.

For details, see ROOT issue [#22329](https://github.com/root-project/root/issues/22329).

In addition, if an I/O read rule is defined for one `SCALAR` column, corresponding read rules must also be provided for all other `SCALAR` columns, even if their schema has not changed.

For details, see ROOT issue [#22330](https://github.com/root-project/root/issues/22330).
14 changes: 14 additions & 0 deletions DataFormats/SoATemplate/interface/SoACommon.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -916,6 +916,20 @@ namespace cms::soa::detail {
}
};

// Helper type trait for obtaining the underlying type of an enum, or the type itself if it's not an enum
template <typename T>
struct EnumTraits {
using type = T;

@fwyzard fwyzard Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is type used ?

@Electricks94 Electricks94 Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes type is used to define ParametersTypeOf_. This has the effect that for the user the integer pointer is represented as an enum. This means that the access of the view also returns an enum like before

@fwyzard fwyzard Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for the user the integer pointer is represented as an enum

ehm, what ?

@Electricks94 Electricks94 Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a column of an enum type is created, the internal ptr to the column is of type EnumTraits::value_type aka the underlying integer type of the enum, but when a view is created then the ptr to the column is of type EnumTraits::type aka the actual enum. In this way ROOT sees the enum as an integer and treats it like so while when accessing the column data users will get the enum type

@fwyzard fwyzard Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻

using value_type = T;
};

template <typename T>
requires std::is_enum_v<T>
struct EnumTraits<T> {
using type = T;
using value_type = std::underlying_type_t<T>;
};

Comment on lines +919 to +932

@fwyzard fwyzard Jun 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it work to use a concept instead of SFINAE ?

// Helper type trait for obtaining a span type for a column
template <typename ColumnType>
struct GetSpanType;
Expand Down
73 changes: 56 additions & 17 deletions DataFormats/SoATemplate/interface/SoALayout.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,15 +81,17 @@ namespace cms::soa {
cms::soa::SoAParameters_ColumnType<cms::soa::SoAColumnType::scalar>::DataType<CPP_TYPE>; \
SOA_HOST_DEVICE SOA_INLINE \
BOOST_PP_CAT(ParametersTypeOf_, NAME) BOOST_PP_CAT(parametersOf_, NAME)() const { \
return BOOST_PP_CAT(ParametersTypeOf_, NAME) (parent_.BOOST_PP_CAT(NAME, _)); \
return BOOST_PP_CAT(ParametersTypeOf_, NAME) \
(reinterpret_cast<cms::soa::detail::EnumTraits<CPP_TYPE>::type*>(parent_.BOOST_PP_CAT(NAME, _))); \
}, \
/* Column */ \
constexpr static cms::soa::SoAColumnType BOOST_PP_CAT(ColumnTypeOf_, NAME) = cms::soa::SoAColumnType::column; \
using BOOST_PP_CAT(ParametersTypeOf_, NAME) = \
cms::soa::SoAParameters_ColumnType<cms::soa::SoAColumnType::column>::DataType<CPP_TYPE>; \
SOA_HOST_DEVICE SOA_INLINE \
BOOST_PP_CAT(ParametersTypeOf_, NAME) BOOST_PP_CAT(parametersOf_, NAME)() const { \
return BOOST_PP_CAT(ParametersTypeOf_, NAME) (parent_.BOOST_PP_CAT(NAME, _)); \
return BOOST_PP_CAT(ParametersTypeOf_, NAME) \
(reinterpret_cast<cms::soa::detail::EnumTraits<CPP_TYPE>::type*>(parent_.BOOST_PP_CAT(NAME, _))); \
}, \
/* Eigen column */ \
constexpr static cms::soa::SoAColumnType BOOST_PP_CAT(ColumnTypeOf_, NAME) = cms::soa::SoAColumnType::eigen; \
Expand Down Expand Up @@ -244,6 +246,8 @@ namespace cms::soa {
(BOOST_PP_CAT(NAME, ElementsWithPadding_){_soa_impl_other.BOOST_PP_CAT(NAME, ElementsWithPadding_)}) \
(BOOST_PP_CAT(NAME, _){_soa_impl_other.BOOST_PP_CAT(NAME, _)}) \
(BOOST_PP_CAT(NAME, Stride_){_soa_impl_other.BOOST_PP_CAT(NAME, Stride_)}) \
(BOOST_PP_CAT(NAME, Rows_){_soa_impl_other.BOOST_PP_CAT(NAME, Rows_)}) \
(BOOST_PP_CAT(NAME, Cols_){_soa_impl_other.BOOST_PP_CAT(NAME, Cols_)}) \
)
// clang-format on

Expand All @@ -263,6 +267,8 @@ namespace cms::soa {
BOOST_PP_CAT(NAME, ElementsWithPadding_) = _soa_impl_other.BOOST_PP_CAT(NAME, ElementsWithPadding_); \
BOOST_PP_CAT(NAME, _) = _soa_impl_other.BOOST_PP_CAT(NAME, _); \
BOOST_PP_CAT(NAME, Stride_) = _soa_impl_other.BOOST_PP_CAT(NAME, Stride_); \
BOOST_PP_CAT(NAME, Rows_) = _soa_impl_other.BOOST_PP_CAT(NAME, Rows_); \
BOOST_PP_CAT(NAME, Cols_) = _soa_impl_other.BOOST_PP_CAT(NAME, Cols_); \
)
// clang-format on

Expand Down Expand Up @@ -365,18 +371,20 @@ namespace cms::soa {
#define _ASSIGN_SOA_COLUMN_OR_SCALAR_IMPL(VALUE_TYPE, CPP_TYPE, NAME, ARGS) \
_SWITCH_ON_TYPE(VALUE_TYPE, \
/* Scalar */ \
BOOST_PP_CAT(NAME, _) = reinterpret_cast<CPP_TYPE*>(_soa_impl_curMem); \
BOOST_PP_CAT(NAME, _) = reinterpret_cast<cms::soa::detail::EnumTraits<CPP_TYPE>::value_type*>(_soa_impl_curMem); \
_soa_impl_curMem += cms::soa::alignSize(sizeof(CPP_TYPE), alignment); \
, \
/* Column */ \
BOOST_PP_CAT(NAME, _) = reinterpret_cast<CPP_TYPE*>(_soa_impl_curMem); \
BOOST_PP_CAT(NAME, _) = reinterpret_cast<cms::soa::detail::EnumTraits<CPP_TYPE>::value_type*>(_soa_impl_curMem); \
_soa_impl_curMem += cms::soa::alignSize(elements_ * sizeof(CPP_TYPE), alignment); \
, \
/* Eigen column */ \
BOOST_PP_CAT(NAME, Stride_) = cms::soa::alignSize(elements_ * sizeof(CPP_TYPE::Scalar), alignment) \
/ sizeof(CPP_TYPE::Scalar); \
BOOST_PP_CAT(NAME, ElementsWithPadding_) = BOOST_PP_CAT(NAME, Stride_) \
* CPP_TYPE::RowsAtCompileTime * CPP_TYPE::ColsAtCompileTime; \
BOOST_PP_CAT(NAME, Rows_) = CPP_TYPE::RowsAtCompileTime; \
BOOST_PP_CAT(NAME, Cols_) = CPP_TYPE::ColsAtCompileTime; \
BOOST_PP_CAT(NAME, _) = reinterpret_cast<CPP_TYPE::Scalar*>(_soa_impl_curMem); \
_soa_impl_curMem += cms::soa::alignSize(elements_ * sizeof(CPP_TYPE::Scalar), alignment) \
* CPP_TYPE::RowsAtCompileTime * CPP_TYPE::ColsAtCompileTime; \
Expand Down Expand Up @@ -461,17 +469,46 @@ namespace cms::soa {
*/
// clang-format off
#define _STREAMER_READ_SOA_DATA_MEMBER_IMPL(VALUE_TYPE, CPP_TYPE, NAME, ARGS) \
_SWITCH_ON_TYPE(VALUE_TYPE, \
/* Scalar */ \
memcpy(BOOST_PP_CAT(NAME, _), onfile.BOOST_PP_CAT(NAME, _), sizeof(CPP_TYPE)); \
, \
/* Column */ \
memcpy(BOOST_PP_CAT(NAME, _), onfile.BOOST_PP_CAT(NAME, _), sizeof(CPP_TYPE) * onfile.elements_); \
, \
/* Eigen column */ \
memcpy(BOOST_PP_CAT(NAME, _), onfile.BOOST_PP_CAT(NAME, _), \
sizeof(CPP_TYPE::Scalar) * BOOST_PP_CAT(NAME, ElementsWithPadding_)); \
)
if (onfile.BOOST_PP_CAT(NAME, _) != nullptr) { \
_SWITCH_ON_TYPE(VALUE_TYPE, \
/* Scalar */ \
memcpy(BOOST_PP_CAT(NAME, _), onfile.BOOST_PP_CAT(NAME, _), sizeof(CPP_TYPE)); \
, \
/* Column */ \
memcpy(BOOST_PP_CAT(NAME, _), onfile.BOOST_PP_CAT(NAME, _), sizeof(CPP_TYPE) * onfile.elements_); \
, \
/* Eigen column */ \
const int rows = onfile.BOOST_PP_CAT(NAME, Rows_); \
const int cols = onfile.BOOST_PP_CAT(NAME, Cols_); \
if((rows * cols) > 0 && (rows != CPP_TYPE::RowsAtCompileTime || cols != CPP_TYPE::ColsAtCompileTime)){ \
cms::soa::detail::throwRuntimeError(("Incompatible eigen column dimensions. On file: " \
+ std::to_string(rows) + "x" + std::to_string(cols) + ", expected: " \
+ std::to_string(CPP_TYPE::RowsAtCompileTime) + "x" \
+ std::to_string(CPP_TYPE::ColsAtCompileTime)).c_str()); \
} \
if(BOOST_PP_CAT(NAME, Stride_) == onfile.BOOST_PP_CAT(NAME, Stride_)) { \
memcpy(BOOST_PP_CAT(NAME, _), onfile.BOOST_PP_CAT(NAME, _), \
sizeof(CPP_TYPE::Scalar) * BOOST_PP_CAT(NAME, ElementsWithPadding_)); \
} else { \
for (int i = 0; i < CPP_TYPE::RowsAtCompileTime * CPP_TYPE::ColsAtCompileTime; ++i) { \
memcpy(BOOST_PP_CAT(NAME, _) + i * BOOST_PP_CAT(NAME, Stride_), \
onfile.BOOST_PP_CAT(NAME, _) + i * onfile.BOOST_PP_CAT(NAME, Stride_), \
sizeof(CPP_TYPE::Scalar) * onfile.elements_); \
} \
} \
) \
} else { \
_SWITCH_ON_TYPE(VALUE_TYPE, \
/* Scalar */ \
memset(BOOST_PP_CAT(NAME, _), 0x00, sizeof(CPP_TYPE)); \
, \
/* Column */ \
memset(BOOST_PP_CAT(NAME, _), 0x00, sizeof(CPP_TYPE) * onfile.elements_); \
, \
/* Eigen column */ \
memset(BOOST_PP_CAT(NAME, _), 0x00, sizeof(CPP_TYPE::Scalar) * BOOST_PP_CAT(NAME, ElementsWithPadding_)); \
) \
}
// clang-format on

#define _STREAMER_READ_SOA_DATA_MEMBER(R, DATA, TYPE_NAME) \
Expand All @@ -483,15 +520,17 @@ namespace cms::soa {
#define _DECLARE_SOA_DATA_MEMBER_IMPL(VALUE_TYPE, CPP_TYPE, NAME, ARGS) \
_SWITCH_ON_TYPE(VALUE_TYPE, \
/* Scalar */ \
CPP_TYPE* BOOST_PP_CAT(NAME, _) EDM_REFLEX_SIZE(scalar_) = nullptr; \
cms::soa::detail::EnumTraits<CPP_TYPE>::value_type* BOOST_PP_CAT(NAME, _) EDM_REFLEX_SIZE(scalar_) = nullptr; \
, \
/* Column */ \
CPP_TYPE * BOOST_PP_CAT(NAME, _) EDM_REFLEX_SIZE(elements_) = nullptr; \
cms::soa::detail::EnumTraits<CPP_TYPE>::value_type* BOOST_PP_CAT(NAME, _) EDM_REFLEX_SIZE(elements_) = nullptr; \
, \
/* Eigen column */ \
size_type BOOST_PP_CAT(NAME, ElementsWithPadding_) = 0; /* For ROOT serialization */ \
CPP_TYPE::Scalar * BOOST_PP_CAT(NAME, _) EDM_REFLEX_SIZE(BOOST_PP_CAT(NAME, ElementsWithPadding_)) = nullptr; \
byte_size_type BOOST_PP_CAT(NAME, Stride_) = 0; \
int BOOST_PP_CAT(NAME, Rows_) = 0; \
int BOOST_PP_CAT(NAME, Cols_) = 0; \
)
// clang-format on

Expand Down
2 changes: 1 addition & 1 deletion DataFormats/SoATemplate/test/SoAStreamInternal_t.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ TEST_CASE("Stream SoA") {

std::ostringstream expected;
expected << "SoATemplate(32 elements, byte alignement= 64, @" << static_cast<void *>(slBuffer.get()) << "): \n"
<< " sizeof(SoATemplate): 176\n"
<< " sizeof(SoATemplate): 200\n"
<< " Column x at offset 0 has size 256 and padding 0\n"
<< " Column y at offset 256 has size 256 and padding 0\n"
<< " Column z at offset 512 has size 256 and padding 0\n"
Expand Down
Loading