diff --git a/DataFormats/Portable/README.md b/DataFormats/Portable/README.md index b59973e46c92b..0a38e1bad9c9b 100644 --- a/DataFormats/Portable/README.md +++ b/DataFormats/Portable/README.md @@ -178,4 +178,72 @@ For the host xml, all SoA layouts have to be listed. The scripts are called as f ./DataFormats/Portable/scripts/portableDeviceCollectionHints portabletest::TestHostMultiCollection3 ``` The layouts should not be added as parameters for the device collection. Those script can be used equally with the -single layout collections or multi layout collections. \ No newline at end of file +single layout collections or multi layout collections. + + +## Schema Evolution of SoA Layouts + +`ROOT` files written using a `PortableCollection` with a specific `SoA layout` are read through a custom streamer. +This streamer copies data into the SoA buffer and relies heavily on ROOT’s built-in schema evolution mechanisms when the layout definition changes in the CMSSW codebase. +The schema evolution behavior is summarized below: + +### General Behavior + - **Removed columns**: Columns that exist in the file but are no longer present in the current layout are skipped during reading. Their data is not loaded into memory. + - **Added columns**: Columns that are present in the current layout but missing from the file are default-initialized to **zero**. +### Fundamental datatype changes +When the datatype of a column changes between versions, `ROOT` performs an element-wise conversion during reading, for example: +``` +double → float +``` +In this case, values are cast to the new type as they are read. +**Important notes:** + - Conversions are performed silently. + - Precision loss (e.g. truncation) is **not reported** by default. + - Overflow and underflow may occur if the new type cannot represent the original values. +### Eigen Columns + - **Supported evolution**: Eigen-based columns can only evolve with respect to their **underlying scalar type**, for example: +``` +Eigen::Matrix -> Eigen::Matrix +``` + - **Unsupported changes**: Changes to the matrix shape (number of rows or columns) are **not supported** and will result in a runtime error during reading. +### Enum Types + - Columns defined with `enum` types are fully supported. + - They behave identically to columns defined with the enum’s **underlying integer type**: + - Schema evolution follows the same rules as for fundamental integer types. + - Changing the underlying type (e.g. `uint16_t` → `uint32_t`) is supported via implicit conversion. + +### User-defined Types + +In the following user-defined types refers to non-fundamental types for example user-defined structs or classes. +Columns containing user-defined structs or classes require explicit I/O schema evolution rules. + +To support schema evolution for a column of a user-defined type: + +1. A class version and checksum must be defined in the SoA layout dictionary. +2. An I/O read rule must be provided that describes how data should be converted between schema versions. + +Examples can be found in `HeterogeneousCore/TestModules/src/classes_def.xml`, where schema evolution rules are implemented for `edm::StdArray`. + +When defining an I/O read rule: + +* Both the source and target columns must be specified. +* The source-side leaf counter called `elements_` must be accessed and is used to determine the size of the temporary conversion buffer. + +#### Known Limitations + +##### Removing User-defined type Columns + +Removing a column that contains a non-fundamental type consistently triggers a ROOT error during reading. See ROOT issue [#22097](https://github.com/root-project/root/issues/22097). + +##### I/O Read Rules for SCALAR Columns + +I/O read rules can also be defined for `SCALAR` columns, but they require special handling: + +* The leaf counter `scalar_` must be included in the rule, even though its value is always `1`. +* Omitting this leaf counter will trigger and error from ROOT. + +For details, see ROOT issue [#22329](https://github.com/root-project/root/issues/22329). + +In addition, if an I/O read rule is defined for one `SCALAR` column, corresponding read rules must also be provided for all other `SCALAR` columns, even if their schema has not changed. + +For details, see ROOT issue [#22330](https://github.com/root-project/root/issues/22330). diff --git a/DataFormats/SoATemplate/interface/SoACommon.h b/DataFormats/SoATemplate/interface/SoACommon.h index 2f2aca5d2a7c3..fb168cc598151 100644 --- a/DataFormats/SoATemplate/interface/SoACommon.h +++ b/DataFormats/SoATemplate/interface/SoACommon.h @@ -916,6 +916,20 @@ namespace cms::soa::detail { } }; + // Helper type trait for obtaining the underlying type of an enum, or the type itself if it's not an enum + template + struct EnumTraits { + using type = T; + using value_type = T; + }; + + template + requires std::is_enum_v + struct EnumTraits { + using type = T; + using value_type = std::underlying_type_t; + }; + // Helper type trait for obtaining a span type for a column template struct GetSpanType; diff --git a/DataFormats/SoATemplate/interface/SoALayout.h b/DataFormats/SoATemplate/interface/SoALayout.h index 2db151935123f..c269574b0ee67 100644 --- a/DataFormats/SoATemplate/interface/SoALayout.h +++ b/DataFormats/SoATemplate/interface/SoALayout.h @@ -81,7 +81,8 @@ namespace cms::soa { cms::soa::SoAParameters_ColumnType::DataType; \ SOA_HOST_DEVICE SOA_INLINE \ BOOST_PP_CAT(ParametersTypeOf_, NAME) BOOST_PP_CAT(parametersOf_, NAME)() const { \ - return BOOST_PP_CAT(ParametersTypeOf_, NAME) (parent_.BOOST_PP_CAT(NAME, _)); \ + return BOOST_PP_CAT(ParametersTypeOf_, NAME) \ + (reinterpret_cast::type*>(parent_.BOOST_PP_CAT(NAME, _))); \ }, \ /* Column */ \ constexpr static cms::soa::SoAColumnType BOOST_PP_CAT(ColumnTypeOf_, NAME) = cms::soa::SoAColumnType::column; \ @@ -89,7 +90,8 @@ namespace cms::soa { cms::soa::SoAParameters_ColumnType::DataType; \ SOA_HOST_DEVICE SOA_INLINE \ BOOST_PP_CAT(ParametersTypeOf_, NAME) BOOST_PP_CAT(parametersOf_, NAME)() const { \ - return BOOST_PP_CAT(ParametersTypeOf_, NAME) (parent_.BOOST_PP_CAT(NAME, _)); \ + return BOOST_PP_CAT(ParametersTypeOf_, NAME) \ + (reinterpret_cast::type*>(parent_.BOOST_PP_CAT(NAME, _))); \ }, \ /* Eigen column */ \ constexpr static cms::soa::SoAColumnType BOOST_PP_CAT(ColumnTypeOf_, NAME) = cms::soa::SoAColumnType::eigen; \ @@ -244,6 +246,8 @@ namespace cms::soa { (BOOST_PP_CAT(NAME, ElementsWithPadding_){_soa_impl_other.BOOST_PP_CAT(NAME, ElementsWithPadding_)}) \ (BOOST_PP_CAT(NAME, _){_soa_impl_other.BOOST_PP_CAT(NAME, _)}) \ (BOOST_PP_CAT(NAME, Stride_){_soa_impl_other.BOOST_PP_CAT(NAME, Stride_)}) \ + (BOOST_PP_CAT(NAME, Rows_){_soa_impl_other.BOOST_PP_CAT(NAME, Rows_)}) \ + (BOOST_PP_CAT(NAME, Cols_){_soa_impl_other.BOOST_PP_CAT(NAME, Cols_)}) \ ) // clang-format on @@ -263,6 +267,8 @@ namespace cms::soa { BOOST_PP_CAT(NAME, ElementsWithPadding_) = _soa_impl_other.BOOST_PP_CAT(NAME, ElementsWithPadding_); \ BOOST_PP_CAT(NAME, _) = _soa_impl_other.BOOST_PP_CAT(NAME, _); \ BOOST_PP_CAT(NAME, Stride_) = _soa_impl_other.BOOST_PP_CAT(NAME, Stride_); \ + BOOST_PP_CAT(NAME, Rows_) = _soa_impl_other.BOOST_PP_CAT(NAME, Rows_); \ + BOOST_PP_CAT(NAME, Cols_) = _soa_impl_other.BOOST_PP_CAT(NAME, Cols_); \ ) // clang-format on @@ -365,11 +371,11 @@ namespace cms::soa { #define _ASSIGN_SOA_COLUMN_OR_SCALAR_IMPL(VALUE_TYPE, CPP_TYPE, NAME, ARGS) \ _SWITCH_ON_TYPE(VALUE_TYPE, \ /* Scalar */ \ - BOOST_PP_CAT(NAME, _) = reinterpret_cast(_soa_impl_curMem); \ + BOOST_PP_CAT(NAME, _) = reinterpret_cast::value_type*>(_soa_impl_curMem); \ _soa_impl_curMem += cms::soa::alignSize(sizeof(CPP_TYPE), alignment); \ , \ /* Column */ \ - BOOST_PP_CAT(NAME, _) = reinterpret_cast(_soa_impl_curMem); \ + BOOST_PP_CAT(NAME, _) = reinterpret_cast::value_type*>(_soa_impl_curMem); \ _soa_impl_curMem += cms::soa::alignSize(elements_ * sizeof(CPP_TYPE), alignment); \ , \ /* Eigen column */ \ @@ -377,6 +383,8 @@ namespace cms::soa { / sizeof(CPP_TYPE::Scalar); \ BOOST_PP_CAT(NAME, ElementsWithPadding_) = BOOST_PP_CAT(NAME, Stride_) \ * CPP_TYPE::RowsAtCompileTime * CPP_TYPE::ColsAtCompileTime; \ + BOOST_PP_CAT(NAME, Rows_) = CPP_TYPE::RowsAtCompileTime; \ + BOOST_PP_CAT(NAME, Cols_) = CPP_TYPE::ColsAtCompileTime; \ BOOST_PP_CAT(NAME, _) = reinterpret_cast(_soa_impl_curMem); \ _soa_impl_curMem += cms::soa::alignSize(elements_ * sizeof(CPP_TYPE::Scalar), alignment) \ * CPP_TYPE::RowsAtCompileTime * CPP_TYPE::ColsAtCompileTime; \ @@ -461,17 +469,46 @@ namespace cms::soa { */ // clang-format off #define _STREAMER_READ_SOA_DATA_MEMBER_IMPL(VALUE_TYPE, CPP_TYPE, NAME, ARGS) \ - _SWITCH_ON_TYPE(VALUE_TYPE, \ - /* Scalar */ \ - memcpy(BOOST_PP_CAT(NAME, _), onfile.BOOST_PP_CAT(NAME, _), sizeof(CPP_TYPE)); \ - , \ - /* Column */ \ - memcpy(BOOST_PP_CAT(NAME, _), onfile.BOOST_PP_CAT(NAME, _), sizeof(CPP_TYPE) * onfile.elements_); \ - , \ - /* Eigen column */ \ - memcpy(BOOST_PP_CAT(NAME, _), onfile.BOOST_PP_CAT(NAME, _), \ - sizeof(CPP_TYPE::Scalar) * BOOST_PP_CAT(NAME, ElementsWithPadding_)); \ - ) + if (onfile.BOOST_PP_CAT(NAME, _) != nullptr) { \ + _SWITCH_ON_TYPE(VALUE_TYPE, \ + /* Scalar */ \ + memcpy(BOOST_PP_CAT(NAME, _), onfile.BOOST_PP_CAT(NAME, _), sizeof(CPP_TYPE)); \ + , \ + /* Column */ \ + memcpy(BOOST_PP_CAT(NAME, _), onfile.BOOST_PP_CAT(NAME, _), sizeof(CPP_TYPE) * onfile.elements_); \ + , \ + /* Eigen column */ \ + const int rows = onfile.BOOST_PP_CAT(NAME, Rows_); \ + const int cols = onfile.BOOST_PP_CAT(NAME, Cols_); \ + if((rows * cols) > 0 && (rows != CPP_TYPE::RowsAtCompileTime || cols != CPP_TYPE::ColsAtCompileTime)){ \ + cms::soa::detail::throwRuntimeError(("Incompatible eigen column dimensions. On file: " \ + + std::to_string(rows) + "x" + std::to_string(cols) + ", expected: " \ + + std::to_string(CPP_TYPE::RowsAtCompileTime) + "x" \ + + std::to_string(CPP_TYPE::ColsAtCompileTime)).c_str()); \ + } \ + if(BOOST_PP_CAT(NAME, Stride_) == onfile.BOOST_PP_CAT(NAME, Stride_)) { \ + memcpy(BOOST_PP_CAT(NAME, _), onfile.BOOST_PP_CAT(NAME, _), \ + sizeof(CPP_TYPE::Scalar) * BOOST_PP_CAT(NAME, ElementsWithPadding_)); \ + } else { \ + for (int i = 0; i < CPP_TYPE::RowsAtCompileTime * CPP_TYPE::ColsAtCompileTime; ++i) { \ + memcpy(BOOST_PP_CAT(NAME, _) + i * BOOST_PP_CAT(NAME, Stride_), \ + onfile.BOOST_PP_CAT(NAME, _) + i * onfile.BOOST_PP_CAT(NAME, Stride_), \ + sizeof(CPP_TYPE::Scalar) * onfile.elements_); \ + } \ + } \ + ) \ + } else { \ + _SWITCH_ON_TYPE(VALUE_TYPE, \ + /* Scalar */ \ + memset(BOOST_PP_CAT(NAME, _), 0x00, sizeof(CPP_TYPE)); \ + , \ + /* Column */ \ + memset(BOOST_PP_CAT(NAME, _), 0x00, sizeof(CPP_TYPE) * onfile.elements_); \ + , \ + /* Eigen column */ \ + memset(BOOST_PP_CAT(NAME, _), 0x00, sizeof(CPP_TYPE::Scalar) * BOOST_PP_CAT(NAME, ElementsWithPadding_)); \ + ) \ + } // clang-format on #define _STREAMER_READ_SOA_DATA_MEMBER(R, DATA, TYPE_NAME) \ @@ -483,15 +520,17 @@ namespace cms::soa { #define _DECLARE_SOA_DATA_MEMBER_IMPL(VALUE_TYPE, CPP_TYPE, NAME, ARGS) \ _SWITCH_ON_TYPE(VALUE_TYPE, \ /* Scalar */ \ - CPP_TYPE* BOOST_PP_CAT(NAME, _) EDM_REFLEX_SIZE(scalar_) = nullptr; \ + cms::soa::detail::EnumTraits::value_type* BOOST_PP_CAT(NAME, _) EDM_REFLEX_SIZE(scalar_) = nullptr; \ , \ /* Column */ \ - CPP_TYPE * BOOST_PP_CAT(NAME, _) EDM_REFLEX_SIZE(elements_) = nullptr; \ + cms::soa::detail::EnumTraits::value_type* BOOST_PP_CAT(NAME, _) EDM_REFLEX_SIZE(elements_) = nullptr; \ , \ /* Eigen column */ \ size_type BOOST_PP_CAT(NAME, ElementsWithPadding_) = 0; /* For ROOT serialization */ \ CPP_TYPE::Scalar * BOOST_PP_CAT(NAME, _) EDM_REFLEX_SIZE(BOOST_PP_CAT(NAME, ElementsWithPadding_)) = nullptr; \ byte_size_type BOOST_PP_CAT(NAME, Stride_) = 0; \ + int BOOST_PP_CAT(NAME, Rows_) = 0; \ + int BOOST_PP_CAT(NAME, Cols_) = 0; \ ) // clang-format on diff --git a/DataFormats/SoATemplate/test/SoAStreamInternal_t.cc b/DataFormats/SoATemplate/test/SoAStreamInternal_t.cc index 06083fd4edf2b..6f0e0a71ae2f6 100644 --- a/DataFormats/SoATemplate/test/SoAStreamInternal_t.cc +++ b/DataFormats/SoATemplate/test/SoAStreamInternal_t.cc @@ -40,7 +40,7 @@ TEST_CASE("Stream SoA") { std::ostringstream expected; expected << "SoATemplate(32 elements, byte alignement= 64, @" << static_cast(slBuffer.get()) << "): \n" - << " sizeof(SoATemplate): 176\n" + << " sizeof(SoATemplate): 200\n" << " Column x at offset 0 has size 256 and padding 0\n" << " Column y at offset 256 has size 256 and padding 0\n" << " Column z at offset 512 has size 256 and padding 0\n" diff --git a/DataFormats/SoATemplate/test/SoAUnitTests.cc b/DataFormats/SoATemplate/test/SoAUnitTests.cc index 0c51ae64c0517..37f820530c1bc 100644 --- a/DataFormats/SoATemplate/test/SoAUnitTests.cc +++ b/DataFormats/SoATemplate/test/SoAUnitTests.cc @@ -6,12 +6,15 @@ #include "DataFormats/SoATemplate/interface/SoALayout.h" +enum class TestEnum : int16_t { s0 = -2, s1 = -1, s2 = 0, s3 = 1, s4 = 2 }; + // clang-format off GENERATE_SOA_LAYOUT(SimpleLayoutTemplate, SOA_COLUMN(float, x), SOA_COLUMN(float, y), SOA_COLUMN(float, z), - SOA_COLUMN(float, t)) + SOA_COLUMN(float, t), + SOA_COLUMN(TestEnum, e)) // clang-format on using SimpleLayout = SimpleLayoutTemplate<>; @@ -35,6 +38,7 @@ TEST_CASE("SoATemplate") { slv0.y() = 2; slv0.z() = 3; slv0.t() = 5; + slv0.e() = TestEnum::s3; // Fill up for (SimpleLayout::View::size_type i = 1; i < slv.metadata().size(); ++i) { auto slvi = slv[i]; @@ -58,6 +62,7 @@ TEST_CASE("SoATemplate") { REQUIRE(slcvi.y() == y); REQUIRE(slcvi.z() == z); REQUIRE(slcvi.t() == t); + REQUIRE(slcvi.e() == TestEnum::s3); auto tx = x; x += y; y += z; @@ -74,7 +79,7 @@ TEST_CASE("SoATemplate") { View slv{sl}; ConstView slcv{sl}; auto slv0 = slv[0]; - slv0 = {7, 11, 13, 17}; + slv0 = {7, 11, 13, 17, TestEnum::s3}; // Fill up for (SimpleLayout::View::size_type i = 1; i < slv.metadata().size(); ++i) { auto slvi = slv[i]; diff --git a/HeterogeneousCore/TestModules/BuildFile.xml b/HeterogeneousCore/TestModules/BuildFile.xml new file mode 100644 index 0000000000000..aee2c36faee00 --- /dev/null +++ b/HeterogeneousCore/TestModules/BuildFile.xml @@ -0,0 +1,5 @@ + + + + + diff --git a/HeterogeneousCore/TestModules/interface/SchemaEvolutionHostCollection.h b/HeterogeneousCore/TestModules/interface/SchemaEvolutionHostCollection.h new file mode 100644 index 0000000000000..88a5ac5b11bd4 --- /dev/null +++ b/HeterogeneousCore/TestModules/interface/SchemaEvolutionHostCollection.h @@ -0,0 +1,17 @@ +#ifndef HeterogeneousCore_TestModules_interface_SchemaEvolutionHostCollection_h +#define HeterogeneousCore_TestModules_interface_SchemaEvolutionHostCollection_h + +#include "DataFormats/Portable/interface/PortableHostCollection.h" + +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionSoA.h" + +namespace testmodules { + using HostCollectionEvolutionZero = PortableHostCollection; + using HostCollectionEvolutionOne = PortableHostCollection; + using HostCollectionEvolutionTwo = PortableHostCollection; + using HostCollectionEvolutionThree = PortableHostCollection; + using HostCollectionEvolutionFour = PortableHostCollection; + using HostCollectionEvolutionFive = PortableHostCollection; +} // namespace testmodules + +#endif // HeterogeneousCore_TestModules_interface_SchemaEvolutionHostCollection_h diff --git a/HeterogeneousCore/TestModules/interface/SchemaEvolutionSoA.h b/HeterogeneousCore/TestModules/interface/SchemaEvolutionSoA.h new file mode 100644 index 0000000000000..2ed77b2e6b9f6 --- /dev/null +++ b/HeterogeneousCore/TestModules/interface/SchemaEvolutionSoA.h @@ -0,0 +1,97 @@ +#ifndef HeterogeneousCore_TestModules_interface_SchemaEvolutionSoA_h +#define HeterogeneousCore_TestModules_interface_SchemaEvolutionSoA_h + +#include +#include + +#include "DataFormats/SoATemplate/interface/SoACommon.h" +#include "DataFormats/SoATemplate/interface/SoALayout.h" +#include "DataFormats/Common/interface/StdArray.h" + +// These layouts form the basis of the schema evolution test. +// ROOT files have been written with SoAEvolutionZeroLayout and will be read with SoAEvolutionOneLayout to SoAEvolutionFiveLayout +namespace testmodules { + + using SEArray = edm::StdArray; + using SEArrayTwo = edm::StdArray; + using SEEigenObject = Eigen::Matrix; + using SEEigenObjectTwo = Eigen::Matrix; + enum class SEEnumType : uint16_t { s1 = 0, s2 = 1, s3 = 2 }; + enum class SEEnumTypeTwo : uint32_t { s1 = 0, s2 = 1, s3 = 2 }; + + GENERATE_SOA_LAYOUT(SoAEvolutionZeroLayout, + SOA_COLUMN(float, cFloat), + SOA_COLUMN(int, cInt), + SOA_COLUMN(double, cDouble), + SOA_COLUMN(SEEnumType, cEnum), + SOA_COLUMN(SEArray, cArray), + SOA_EIGEN_COLUMN(SEEigenObject, eEigenObject), + SOA_SCALAR(int, sInt), + SOA_SCALAR(float, sFloat), + SOA_SCALAR(double, sDouble), + SOA_SCALAR(SEEnumType, sEnum)); + + GENERATE_SOA_LAYOUT(SoAEvolutionOneLayout, + SOA_COLUMN(double, cFloat), + SOA_COLUMN(float, cInt), + SOA_COLUMN(int8_t, cDouble), + SOA_COLUMN(SEEnumTypeTwo, cEnum), + SOA_COLUMN(SEArray, cArray), + SOA_EIGEN_COLUMN(SEEigenObjectTwo, eEigenObject), + SOA_SCALAR(int64_t, sInt), + SOA_SCALAR(int8_t, sFloat), + SOA_SCALAR(float, sDouble), + SOA_SCALAR(SEEnumTypeTwo, sEnum)); + + GENERATE_SOA_LAYOUT(SoAEvolutionTwoLayout, + SOA_COLUMN(SEArray, cArray), + SOA_COLUMN(float, cFloat), + SOA_COLUMN(int, newColumn), + SOA_COLUMN(SEEnumType, cEnum), + SOA_EIGEN_COLUMN(SEEigenObject, eEigenObject), + SOA_EIGEN_COLUMN(SEEigenObjectTwo, newEigenObject), + SOA_SCALAR(float, sFloatNewName), + SOA_SCALAR(int8_t, newScalar), + SOA_SCALAR(SEEnumType, sEnum)); + + GENERATE_SOA_LAYOUT(SoAEvolutionThreeLayout, + SOA_COLUMN(int, cInt), + SOA_COLUMN(SEEnumType, cEnum), + SOA_COLUMN(SEArray, cArray), + SOA_EIGEN_COLUMN(SEEigenObject, eEigenObject), + SOA_SCALAR(int, sInt), + SOA_SCALAR(SEEnumType, sEnum)); + + GENERATE_SOA_LAYOUT(SoAEvolutionFourLayout, + SOA_EIGEN_COLUMN(SEEigenObject, eEigenObject), + SOA_SCALAR(SEEnumType, sEnum), + SOA_SCALAR(double, sDouble), + SOA_SCALAR(float, sFloat), + SOA_SCALAR(int, sInt), + SOA_COLUMN(SEEnumType, cEnum), + SOA_COLUMN(SEArray, cArray), + SOA_COLUMN(double, cDouble), + SOA_COLUMN(int, cInt), + SOA_COLUMN(float, cFloat)); + + GENERATE_SOA_LAYOUT(SoAEvolutionFiveLayout, + SOA_COLUMN(float, cFloat), + SOA_COLUMN(int, cInt), + SOA_COLUMN(double, cDouble), + SOA_COLUMN(SEEnumType, cEnum), + SOA_COLUMN(SEArrayTwo, cArray), + SOA_EIGEN_COLUMN(SEEigenObject, eEigenObject), + SOA_SCALAR(int, sInt), + SOA_SCALAR(float, sFloat), + SOA_SCALAR(double, sDouble), + SOA_SCALAR(SEEnumType, sEnum)); + + using SoAEvolutionZero = SoAEvolutionZeroLayout<>; + using SoAEvolutionOne = SoAEvolutionOneLayout<>; + using SoAEvolutionTwo = SoAEvolutionTwoLayout<>; + using SoAEvolutionThree = SoAEvolutionThreeLayout<>; + using SoAEvolutionFour = SoAEvolutionFourLayout<>; + using SoAEvolutionFive = SoAEvolutionFiveLayout<>; +} // namespace testmodules + +#endif // HeterogeneousCore_TestModules_interface_SchemaEvolutionSoA_h diff --git a/HeterogeneousCore/TestModules/plugins/BuildFile.xml b/HeterogeneousCore/TestModules/plugins/BuildFile.xml new file mode 100644 index 0000000000000..2ce45a10e4bb8 --- /dev/null +++ b/HeterogeneousCore/TestModules/plugins/BuildFile.xml @@ -0,0 +1,9 @@ + + + + + + + + + diff --git a/HeterogeneousCore/TestModules/plugins/EvolutionFiveAnalyzer.cc b/HeterogeneousCore/TestModules/plugins/EvolutionFiveAnalyzer.cc new file mode 100644 index 0000000000000..71bc91d16166e --- /dev/null +++ b/HeterogeneousCore/TestModules/plugins/EvolutionFiveAnalyzer.cc @@ -0,0 +1,79 @@ +#include "FWCore/Framework/interface/Event.h" +#include "FWCore/Framework/interface/EventSetup.h" +#include "FWCore/Framework/interface/Frameworkfwd.h" +#include "FWCore/Framework/interface/global/EDAnalyzer.h" +#include "FWCore/MessageLogger/interface/MessageLogger.h" +#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h" +#include "FWCore/ParameterSet/interface/ParameterSet.h" +#include "FWCore/ParameterSet/interface/ParameterSetDescription.h" +#include "FWCore/Utilities/interface/EDGetToken.h" +#include "FWCore/Utilities/interface/InputTag.h" + +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionSoA.h" +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionHostCollection.h" + +using CollectionVersion = testmodules::HostCollectionEvolutionFive; + +class EvolutionFiveAnalyzer : public edm::global::EDAnalyzer<> { +public: + EvolutionFiveAnalyzer(edm::ParameterSet const& config) + : source_{config.getParameter("source")}, soaToken_{consumes(source_)} {} + + void analyze(edm::StreamID sid, edm::Event const& event, edm::EventSetup const&) const override { + CollectionVersion const& product = event.get(soaToken_); + auto const& view = product.const_view(); + + assert(view.metadata().size() == 23); + + constexpr float EPS_F = 1e-5f; + constexpr double EPS_D = 1e-10; + + for (int i = 0; i < view.metadata().size(); i++) { + auto element = view[i]; + + // Check floats + float expectedFloat = static_cast(i) + 0.1f; + assert(std::abs(element.cFloat() - expectedFloat) < EPS_F); + + // Check ints + int expectedInt = i; + assert(element.cInt() == expectedInt); + + // Check doubles + double expectedDouble = std::sin(static_cast(i)) * 1e3; + assert(std::abs(element.cDouble() - expectedDouble) < EPS_D); + + // Check enum + assert(element.cEnum() == testmodules::SEEnumType::s2); + + // Check Eigen matrix + testmodules::SEEigenObject expectedEigen; + expectedEigen << 10 * i + 1.1f, -10 * i - 1.2f, 10 * i + 2.3f, -10 * i - 2.4f, 10 * i + 3.5f, -10 * i - 3.6f, + 10 * i + 4.7f, -10 * i - 4.8f; + assert((element.eEigenObject() - expectedEigen).cwiseAbs().maxCoeff() < EPS_F); + + for (std::size_t j = 0; j < element.cArray().size(); j++) { + double expectedArrayValue = i * static_cast(6) + static_cast(j); + assert(std::abs(element.cArray()[j] - expectedArrayValue) < EPS_D); + } + } + + // Check Scalars + assert(view.sInt() == std::numeric_limits::max() - 7); + assert(std::abs(view.sFloat() - (1.0f / 3.0f)) < EPS_F); + assert(std::abs(view.sDouble() - (1.0 / 10.0)) < EPS_D); + assert(view.sEnum() == testmodules::SEEnumType::s1); + } + + static void fillDescriptions(edm::ConfigurationDescriptions& descriptions) { + edm::ParameterSetDescription desc; + desc.add("source"); + } + +private: + const edm::InputTag source_; + const edm::EDGetTokenT soaToken_; +}; + +#include "FWCore/Framework/interface/MakerMacros.h" +DEFINE_FWK_MODULE(EvolutionFiveAnalyzer); diff --git a/HeterogeneousCore/TestModules/plugins/EvolutionFourAnalyzer.cc b/HeterogeneousCore/TestModules/plugins/EvolutionFourAnalyzer.cc new file mode 100644 index 0000000000000..80829ea2c9d10 --- /dev/null +++ b/HeterogeneousCore/TestModules/plugins/EvolutionFourAnalyzer.cc @@ -0,0 +1,79 @@ +#include "FWCore/Framework/interface/Event.h" +#include "FWCore/Framework/interface/EventSetup.h" +#include "FWCore/Framework/interface/Frameworkfwd.h" +#include "FWCore/Framework/interface/global/EDAnalyzer.h" +#include "FWCore/MessageLogger/interface/MessageLogger.h" +#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h" +#include "FWCore/ParameterSet/interface/ParameterSet.h" +#include "FWCore/ParameterSet/interface/ParameterSetDescription.h" +#include "FWCore/Utilities/interface/EDGetToken.h" +#include "FWCore/Utilities/interface/InputTag.h" + +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionSoA.h" +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionHostCollection.h" + +using CollectionVersion = testmodules::HostCollectionEvolutionFour; + +class EvolutionFourAnalyzer : public edm::global::EDAnalyzer<> { +public: + EvolutionFourAnalyzer(edm::ParameterSet const& config) + : source_{config.getParameter("source")}, soaToken_{consumes(source_)} {} + + void analyze(edm::StreamID sid, edm::Event const& event, edm::EventSetup const&) const override { + CollectionVersion const& product = event.get(soaToken_); + auto const& view = product.const_view(); + + assert(view.metadata().size() == 23); + + constexpr float EPS_F = 1e-5f; + constexpr double EPS_D = 1e-10; + + for (int i = 0; i < view.metadata().size(); i++) { + auto element = view[i]; + + // Check floats + float expectedFloat = static_cast(i) + 0.1f; + assert(std::abs(element.cFloat() - expectedFloat) < EPS_F); + + // Check ints + int expectedInt = i; + assert(element.cInt() == expectedInt); + + // Check doubles + double expectedDouble = std::sin(static_cast(i)) * 1e3; + assert(std::abs(element.cDouble() - expectedDouble) < EPS_D); + + // Check enum + assert(element.cEnum() == testmodules::SEEnumType::s2); + + // Check Eigen matrix + testmodules::SEEigenObject expectedEigen; + expectedEigen << 10 * i + 1.1f, -10 * i - 1.2f, 10 * i + 2.3f, -10 * i - 2.4f, 10 * i + 3.5f, -10 * i - 3.6f, + 10 * i + 4.7f, -10 * i - 4.8f; + assert((element.eEigenObject() - expectedEigen).cwiseAbs().maxCoeff() < EPS_F); + + for (std::size_t j = 0; j < element.cArray().size(); j++) { + int expectedArrayValue = i * static_cast(element.cArray().size()) + static_cast(j); + assert(element.cArray()[j] == expectedArrayValue); + } + } + + // Check Scalars + assert(view.sInt() == std::numeric_limits::max() - 7); + assert(std::abs(view.sFloat() - (1.0f / 3.0f)) < EPS_F); + assert(std::abs(view.sDouble() - (1.0 / 10.0)) < EPS_D); + assert(view.sEnum() == testmodules::SEEnumType::s1); + } + + static void fillDescriptions(edm::ConfigurationDescriptions& descriptions) { + edm::ParameterSetDescription desc; + desc.add("source"); + } + +private: + const edm::InputTag source_; + const edm::EDGetTokenT soaToken_; +}; + +#include "FWCore/Framework/interface/MakerMacros.h" +DEFINE_FWK_MODULE(EvolutionFourAnalyzer); diff --git a/HeterogeneousCore/TestModules/plugins/EvolutionOneAnalyzer.cc b/HeterogeneousCore/TestModules/plugins/EvolutionOneAnalyzer.cc new file mode 100644 index 0000000000000..5ed638a03530b --- /dev/null +++ b/HeterogeneousCore/TestModules/plugins/EvolutionOneAnalyzer.cc @@ -0,0 +1,86 @@ +#include "FWCore/Framework/interface/Event.h" +#include "FWCore/Framework/interface/EventSetup.h" +#include "FWCore/Framework/interface/Frameworkfwd.h" +#include "FWCore/Framework/interface/global/EDAnalyzer.h" +#include "FWCore/MessageLogger/interface/MessageLogger.h" +#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h" +#include "FWCore/ParameterSet/interface/ParameterSet.h" +#include "FWCore/ParameterSet/interface/ParameterSetDescription.h" +#include "FWCore/Utilities/interface/EDGetToken.h" +#include "FWCore/Utilities/interface/InputTag.h" + +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionSoA.h" +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionHostCollection.h" + +using CollectionVersion = testmodules::HostCollectionEvolutionOne; + +class EvolutionOneAnalyzer : public edm::global::EDAnalyzer<> { +public: + EvolutionOneAnalyzer(edm::ParameterSet const& config) + : source_{config.getParameter("source")}, soaToken_{consumes(source_)} {} + + void analyze(edm::StreamID sid, edm::Event const& event, edm::EventSetup const&) const override { + CollectionVersion const& product = event.get(soaToken_); + auto const& view = product.const_view(); + + assert(view.metadata().size() == 23); + + constexpr float EPS_F = 1e-5f; + + for (int i = 0; i < view.metadata().size(); i++) { + auto element = view[i]; + + // Check that float has been casted to double + double expectedFloat = static_cast(i) + 0.1f; + assert(std::abs(element.cFloat() - expectedFloat) < EPS_F); + + // Check that int has been casted to float + float expectedInt = static_cast(i); + assert(std::abs(element.cInt() - expectedInt) < EPS_F); + + // Check that double has been casted to int8_t and iorule has been applied + const double originalDoubleValue = std::sin(static_cast(i)) * 1e3; + if (originalDoubleValue >= INT8_MIN && originalDoubleValue <= INT8_MAX) { + assert(element.cDouble() == static_cast(originalDoubleValue)); + } else { + assert(element.cDouble() == static_cast(0)); + } + + // Check that enum has been casted to the new enum type + assert(element.cEnum() == testmodules::SEEnumTypeTwo::s2); + + // Check that Eigen matrix has been casted to Eigen matrix with type double + testmodules::SEEigenObjectTwo expectedEigen; + expectedEigen << 10 * i + 1.1, -10 * i - 1.2, 10 * i + 2.3, -10 * i - 2.4, 10 * i + 3.5, -10 * i - 3.6, + 10 * i + 4.7, -10 * i - 4.8; + for (int r = 0; r < element.eEigenObject().rows(); ++r) { + for (int c = 0; c < element.eEigenObject().cols(); ++c) { + assert(std::abs(element.eEigenObject()(r, c) - expectedEigen(r, c)) < EPS_F); + } + } + + for (std::size_t j = 0; j < element.cArray().size(); j++) { + int expectedArrayValue = i * static_cast(element.cArray().size()) + static_cast(j); + assert(element.cArray()[j] == expectedArrayValue); + } + } + + // Check Scalars + assert(view.sInt() == static_cast(std::numeric_limits::max() - 7)); + assert(view.sFloat() == static_cast(1.0f / 3.0f)); + assert(std::abs(view.sDouble() - (1.0f / 10.0f)) < EPS_F); + assert(view.sEnum() == testmodules::SEEnumTypeTwo::s1); + } + + static void fillDescriptions(edm::ConfigurationDescriptions& descriptions) { + edm::ParameterSetDescription desc; + desc.add("source"); + } + +private: + const edm::InputTag source_; + const edm::EDGetTokenT soaToken_; +}; + +#include "FWCore/Framework/interface/MakerMacros.h" +DEFINE_FWK_MODULE(EvolutionOneAnalyzer); diff --git a/HeterogeneousCore/TestModules/plugins/EvolutionThreeAnalyzer.cc b/HeterogeneousCore/TestModules/plugins/EvolutionThreeAnalyzer.cc new file mode 100644 index 0000000000000..9523e8fb00636 --- /dev/null +++ b/HeterogeneousCore/TestModules/plugins/EvolutionThreeAnalyzer.cc @@ -0,0 +1,67 @@ +#include "FWCore/Framework/interface/Event.h" +#include "FWCore/Framework/interface/EventSetup.h" +#include "FWCore/Framework/interface/Frameworkfwd.h" +#include "FWCore/Framework/interface/global/EDAnalyzer.h" +#include "FWCore/MessageLogger/interface/MessageLogger.h" +#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h" +#include "FWCore/ParameterSet/interface/ParameterSet.h" +#include "FWCore/ParameterSet/interface/ParameterSetDescription.h" +#include "FWCore/Utilities/interface/EDGetToken.h" +#include "FWCore/Utilities/interface/InputTag.h" + +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionSoA.h" +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionHostCollection.h" + +using CollectionVersion = testmodules::HostCollectionEvolutionThree; + +class EvolutionThreeAnalyzer : public edm::global::EDAnalyzer<> { +public: + EvolutionThreeAnalyzer(edm::ParameterSet const& config) + : source_{config.getParameter("source")}, soaToken_{consumes(source_)} {} + + void analyze(edm::StreamID sid, edm::Event const& event, edm::EventSetup const&) const override { + CollectionVersion const& product = event.get(soaToken_); + auto const& view = product.const_view(); + + assert(view.metadata().size() == 23); + + constexpr float EPS_F = 1e-5f; + for (int i = 0; i < view.metadata().size(); i++) { + auto element = view[i]; + + // Check ints + int expectedInt = i; + assert(element.cInt() == expectedInt); + + // Check enum + assert(element.cEnum() == testmodules::SEEnumType::s2); + + // Check Eigen matrix + testmodules::SEEigenObject expectedEigen; + expectedEigen << 10 * i + 1.1f, -10 * i - 1.2f, 10 * i + 2.3f, -10 * i - 2.4f, 10 * i + 3.5f, -10 * i - 3.6f, + 10 * i + 4.7f, -10 * i - 4.8f; + assert((element.eEigenObject() - expectedEigen).cwiseAbs().maxCoeff() < EPS_F); + + for (std::size_t j = 0; j < element.cArray().size(); j++) { + int expectedArrayValue = i * static_cast(element.cArray().size()) + static_cast(j); + assert(element.cArray()[j] == expectedArrayValue); + } + } + + // Check Scalars + assert(view.sInt() == std::numeric_limits::max() - 7); + assert(view.sEnum() == testmodules::SEEnumType::s1); + } + + static void fillDescriptions(edm::ConfigurationDescriptions& descriptions) { + edm::ParameterSetDescription desc; + desc.add("source"); + } + +private: + const edm::InputTag source_; + const edm::EDGetTokenT soaToken_; +}; + +#include "FWCore/Framework/interface/MakerMacros.h" +DEFINE_FWK_MODULE(EvolutionThreeAnalyzer); diff --git a/HeterogeneousCore/TestModules/plugins/EvolutionTwoAnalyzer.cc b/HeterogeneousCore/TestModules/plugins/EvolutionTwoAnalyzer.cc new file mode 100644 index 0000000000000..05eb225f9ce68 --- /dev/null +++ b/HeterogeneousCore/TestModules/plugins/EvolutionTwoAnalyzer.cc @@ -0,0 +1,84 @@ +#include "FWCore/Framework/interface/Event.h" +#include "FWCore/Framework/interface/EventSetup.h" +#include "FWCore/Framework/interface/Frameworkfwd.h" +#include "FWCore/Framework/interface/global/EDAnalyzer.h" +#include "FWCore/MessageLogger/interface/MessageLogger.h" +#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h" +#include "FWCore/ParameterSet/interface/ParameterSet.h" +#include "FWCore/ParameterSet/interface/ParameterSetDescription.h" +#include "FWCore/Utilities/interface/EDGetToken.h" +#include "FWCore/Utilities/interface/InputTag.h" + +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionSoA.h" +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionHostCollection.h" + +using CollectionVersion = testmodules::HostCollectionEvolutionTwo; + +class EvolutionTwoAnalyzer : public edm::global::EDAnalyzer<> { +public: + EvolutionTwoAnalyzer(edm::ParameterSet const& config) + : source_{config.getParameter("source")}, soaToken_{consumes(source_)} {} + + void analyze(edm::StreamID sid, edm::Event const& event, edm::EventSetup const&) const override { + CollectionVersion const& product = event.get(soaToken_); + auto const& view = product.const_view(); + + assert(view.metadata().size() == 23); + + constexpr float EPS_F = 1e-5f; + constexpr double EPS_D = 1e-10; + + for (int i = 0; i < view.metadata().size(); i++) { + auto element = view[i]; + + // Check that float is still correctly read + double expectedFloat = static_cast(i) + 0.1f; + assert(std::abs(element.cFloat() - expectedFloat) < EPS_F); + + // Check that new column is zero initialized when reading old data + assert(element.newColumn() == static_cast(0)); + + // Check that enum is still correct + assert(element.cEnum() == testmodules::SEEnumType::s2); + + // Check that Eigen matrix is still correct + testmodules::SEEigenObject expectedEigen; + expectedEigen << 10 * i + 1.1, -10 * i - 1.2, 10 * i + 2.3, -10 * i - 2.4, 10 * i + 3.5, -10 * i - 3.6, + 10 * i + 4.7, -10 * i - 4.8; + for (int r = 0; r < element.eEigenObject().rows(); ++r) { + for (int c = 0; c < element.eEigenObject().cols(); ++c) { + assert(std::abs(element.eEigenObject()(r, c) - expectedEigen(r, c)) < EPS_F); + } + } + + // Check that new Eigen matrix is zero initialized when reading old data + for (int r = 0; r < element.newEigenObject().rows(); ++r) { + for (int c = 0; c < element.newEigenObject().cols(); ++c) { + assert(std::abs(element.newEigenObject()(r, c) - static_cast(0)) < EPS_D); + } + } + + for (std::size_t j = 0; j < element.cArray().size(); j++) { + int expectedArrayValue = i * static_cast(element.cArray().size()) + static_cast(j); + assert(element.cArray()[j] == expectedArrayValue); + } + } + + // Check Scalars + assert(std::abs(view.sFloatNewName() - static_cast(0.0f)) < EPS_F); + assert(view.newScalar() == static_cast(0)); + assert(view.sEnum() == testmodules::SEEnumType::s1); + } + + static void fillDescriptions(edm::ConfigurationDescriptions& descriptions) { + edm::ParameterSetDescription desc; + desc.add("source"); + } + +private: + const edm::InputTag source_; + const edm::EDGetTokenT soaToken_; +}; + +#include "FWCore/Framework/interface/MakerMacros.h" +DEFINE_FWK_MODULE(EvolutionTwoAnalyzer); diff --git a/HeterogeneousCore/TestModules/plugins/EvolutionZeroAnalyzer.cc b/HeterogeneousCore/TestModules/plugins/EvolutionZeroAnalyzer.cc new file mode 100644 index 0000000000000..b50224a0254cf --- /dev/null +++ b/HeterogeneousCore/TestModules/plugins/EvolutionZeroAnalyzer.cc @@ -0,0 +1,79 @@ +#include "FWCore/Framework/interface/Event.h" +#include "FWCore/Framework/interface/EventSetup.h" +#include "FWCore/Framework/interface/Frameworkfwd.h" +#include "FWCore/Framework/interface/global/EDAnalyzer.h" +#include "FWCore/MessageLogger/interface/MessageLogger.h" +#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h" +#include "FWCore/ParameterSet/interface/ParameterSet.h" +#include "FWCore/ParameterSet/interface/ParameterSetDescription.h" +#include "FWCore/Utilities/interface/EDGetToken.h" +#include "FWCore/Utilities/interface/InputTag.h" + +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionSoA.h" +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionHostCollection.h" + +using CollectionVersion = testmodules::HostCollectionEvolutionZero; + +class EvolutionZeroAnalyzer : public edm::global::EDAnalyzer<> { +public: + EvolutionZeroAnalyzer(edm::ParameterSet const& config) + : source_{config.getParameter("source")}, soaToken_{consumes(source_)} {} + + void analyze(edm::StreamID sid, edm::Event const& event, edm::EventSetup const&) const override { + CollectionVersion const& product = event.get(soaToken_); + auto const& view = product.const_view(); + + assert(view.metadata().size() == 23); + + constexpr float EPS_F = 1e-5f; + constexpr double EPS_D = 1e-10; + + for (int i = 0; i < view.metadata().size(); i++) { + auto element = view[i]; + + // Check floats + float expectedFloat = static_cast(i) + 0.1f; + assert(std::abs(element.cFloat() - expectedFloat) < EPS_F); + + // Check ints + int expectedInt = i; + assert(element.cInt() == expectedInt); + + // Check doubles + double expectedDouble = std::sin(static_cast(i)) * 1e3; + assert(std::abs(element.cDouble() - expectedDouble) < EPS_D); + + // Check enum + assert(element.cEnum() == testmodules::SEEnumType::s2); + + // Check Eigen matrix + testmodules::SEEigenObject expectedEigen; + expectedEigen << 10 * i + 1.1f, -10 * i - 1.2f, 10 * i + 2.3f, -10 * i - 2.4f, 10 * i + 3.5f, -10 * i - 3.6f, + 10 * i + 4.7f, -10 * i - 4.8f; + assert((element.eEigenObject() - expectedEigen).cwiseAbs().maxCoeff() < EPS_F); + + for (std::size_t j = 0; j < element.cArray().size(); j++) { + int expectedArrayValue = i * static_cast(element.cArray().size()) + static_cast(j); + assert(element.cArray()[j] == expectedArrayValue); + } + } + + // Check Scalars + assert(view.sInt() == std::numeric_limits::max() - 7); + assert(std::abs(view.sFloat() - (1.0f / 3.0f)) < EPS_F); + assert(std::abs(view.sDouble() - (1.0 / 10.0)) < EPS_D); + assert(view.sEnum() == testmodules::SEEnumType::s1); + } + + static void fillDescriptions(edm::ConfigurationDescriptions& descriptions) { + edm::ParameterSetDescription desc; + desc.add("source"); + } + +private: + const edm::InputTag source_; + const edm::EDGetTokenT soaToken_; +}; + +#include "FWCore/Framework/interface/MakerMacros.h" +DEFINE_FWK_MODULE(EvolutionZeroAnalyzer); diff --git a/HeterogeneousCore/TestModules/plugins/SchemaEvolutionSoAProducer.cc b/HeterogeneousCore/TestModules/plugins/SchemaEvolutionSoAProducer.cc new file mode 100644 index 0000000000000..f2344ae8b58eb --- /dev/null +++ b/HeterogeneousCore/TestModules/plugins/SchemaEvolutionSoAProducer.cc @@ -0,0 +1,63 @@ +#include "FWCore/Framework/interface/Event.h" +#include "FWCore/Framework/interface/MakerMacros.h" +#include "FWCore/Framework/interface/global/EDProducer.h" + +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionHostCollection.h" + +// This producer was used to create two different ROOT files +// using SoALayouts that resemble the SoAEvolutionZeroLayout but under different class names (SoAEvolutionOneLayout to SoAEvolutionTwoLayout). +// Then SoAEvolutionOneLayout to SoAEvolutionTwoLayout were changes to test if the schema evolution works correctly (see EvolutionOneAnalyzer.cc to EvolutionTwoAnalyzer.cc). + +using CollectionVersion = testmodules::HostCollectionEvolutionZero; +constexpr auto productName = "EvolutionZeroProduct"; + +class SchemaEvolutionSoAProducer : public edm::global::EDProducer<> { +public: + explicit SchemaEvolutionSoAProducer(const edm::ParameterSet&); + ~SchemaEvolutionSoAProducer() override; + + void produce(edm::StreamID, edm::Event&, const edm::EventSetup&) const override; + +private: +}; + +SchemaEvolutionSoAProducer::SchemaEvolutionSoAProducer(const edm::ParameterSet& iConfig) { + produces(productName); +} + +SchemaEvolutionSoAProducer::~SchemaEvolutionSoAProducer() {} + +DEFINE_FWK_MODULE(SchemaEvolutionSoAProducer); + +void SchemaEvolutionSoAProducer::produce(edm::StreamID iID, edm::Event& event, const edm::EventSetup& iSetup) const { + std::size_t elems = 23; + auto product = std::make_unique(cms::alpakatools::host(), elems); + auto& view = product->view(); + + for (int i = 0; i < view.metadata().size(); i++) { + auto element = view[i]; + element.cFloat() = static_cast(i) + 0.1f; + element.cInt() = static_cast(i); + element.cDouble() = std::sin(static_cast(i)) * 1e3; + + element.cEnum() = testmodules::SEEnumType::s2; + + element.eEigenObject() = testmodules::SEEigenObject({{10 * i + 1.1f, -10 * i - 1.2f}, + {10 * i + 2.3f, -10 * i - 2.4f}, + {10 * i + 3.5f, -10 * i - 3.6f}, + {10 * i + 4.7f, -10 * i - 4.8f}}); + + for (std::size_t j = 0; j < element.cArray().size(); j++) { + element.cArray()[j] = i * static_cast(element.cArray().size()) + static_cast(j); + } + } + + view.sInt() = std::numeric_limits::max() - 7; + view.sFloat() = 1.0f / 3.0f; + view.sDouble() = 1.0 / 10.0; + view.sEnum() = testmodules::SEEnumType::s1; + + std::cout << "Running " << __func__ << " with " << view.metadata().size() << " elements" << std::endl; + + event.put(std::move(product), productName); +} diff --git a/HeterogeneousCore/TestModules/src/classes.cc b/HeterogeneousCore/TestModules/src/classes.cc new file mode 100644 index 0000000000000..941341a236de7 --- /dev/null +++ b/HeterogeneousCore/TestModules/src/classes.cc @@ -0,0 +1,11 @@ +#include "DataFormats/Portable/interface/PortableHostCollectionReadRules.h" +#include "DataFormats/Portable/interface/PortableHostObjectReadRules.h" + +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionHostCollection.h" + +SET_PORTABLEHOSTCOLLECTION_READ_RULES(testmodules::HostCollectionEvolutionZero); +SET_PORTABLEHOSTCOLLECTION_READ_RULES(testmodules::HostCollectionEvolutionOne); +SET_PORTABLEHOSTCOLLECTION_READ_RULES(testmodules::HostCollectionEvolutionTwo); +SET_PORTABLEHOSTCOLLECTION_READ_RULES(testmodules::HostCollectionEvolutionThree); +SET_PORTABLEHOSTCOLLECTION_READ_RULES(testmodules::HostCollectionEvolutionFour); +SET_PORTABLEHOSTCOLLECTION_READ_RULES(testmodules::HostCollectionEvolutionFive); diff --git a/HeterogeneousCore/TestModules/src/classes.h b/HeterogeneousCore/TestModules/src/classes.h new file mode 100644 index 0000000000000..d2ee55077f63d --- /dev/null +++ b/HeterogeneousCore/TestModules/src/classes.h @@ -0,0 +1,8 @@ +// these first to make sure they get included before any SoA header +#include +#include + +#include "DataFormats/Common/interface/Wrapper.h" + +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionHostCollection.h" +#include "HeterogeneousCore/TestModules/interface/SchemaEvolutionSoA.h" diff --git a/HeterogeneousCore/TestModules/src/classes_def.xml b/HeterogeneousCore/TestModules/src/classes_def.xml new file mode 100644 index 0000000000000..0f4ce2bccc649 --- /dev/null +++ b/HeterogeneousCore/TestModules/src/classes_def.xml @@ -0,0 +1,80 @@ + + + + + + + + + + + + + + + + + + + = INT8_MIN && onFileValue <= INT8_MAX) { + cDouble_[i] = static_cast(onFileValue); + } else { + // Warning could be inserted like this: + // printf("Warning: Value %f at index %i is out of int8_t range. Setting to 0.\n", onFileValue, i); + cDouble_[i] = static_cast(0); + } + }]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + [onfile.elements_]; + for (cms::soa::size_type i = 0; i < onfile.elements_; i++) { + cArray_[i][0] = (double)onfile.cArray_[i][0]; + cArray_[i][1] = (double)onfile.cArray_[i][1]; + cArray_[i][2] = (double)onfile.cArray_[i][2]; + cArray_[i][3] = (double)onfile.cArray_[i][3]; + }]]> + + + + + diff --git a/HeterogeneousCore/TestModules/test/SchemaEvolutionFiveReader.py b/HeterogeneousCore/TestModules/test/SchemaEvolutionFiveReader.py new file mode 100644 index 0000000000000..1bf4168c40403 --- /dev/null +++ b/HeterogeneousCore/TestModules/test/SchemaEvolutionFiveReader.py @@ -0,0 +1,18 @@ +import FWCore.ParameterSet.Config as cms + +process = cms.Process('Reader') +process.maxEvents = cms.untracked.PSet(input = cms.untracked.int32(1)) + +# read the products from a 'test.root' file +process.source = cms.Source('PoolSource', + fileNames = cms.untracked.vstring('file:/afs/cern.ch/user/m/maholzer/SchemaEvolutionTestData/SEFive.root') +) + +# enable logging for the analyser +process.MessageLogger.soAAnalyzer = cms.untracked.PSet() + +process.evolutionFiveAnalyzer = cms.EDAnalyzer('EvolutionFiveAnalyzer', + source = cms.InputTag("soaproducer", "EvolutionFiveProduct"), +) + +process.p = cms.Path(process.evolutionFiveAnalyzer) diff --git a/HeterogeneousCore/TestModules/test/SchemaEvolutionFourReader.py b/HeterogeneousCore/TestModules/test/SchemaEvolutionFourReader.py new file mode 100644 index 0000000000000..11f120f4478e8 --- /dev/null +++ b/HeterogeneousCore/TestModules/test/SchemaEvolutionFourReader.py @@ -0,0 +1,18 @@ +import FWCore.ParameterSet.Config as cms + +process = cms.Process('Reader') +process.maxEvents = cms.untracked.PSet(input = cms.untracked.int32(1)) + +# read the products from a 'test.root' file +process.source = cms.Source('PoolSource', + fileNames = cms.untracked.vstring('file:/afs/cern.ch/user/m/maholzer/SchemaEvolutionTestData/SEFour.root') +) + +# enable logging for the analyser +process.MessageLogger.soAAnalyzer = cms.untracked.PSet() + +process.evolutionFourAnalyzer = cms.EDAnalyzer('EvolutionFourAnalyzer', + source = cms.InputTag("soaproducer", "EvolutionFourProduct"), +) + +process.p = cms.Path(process.evolutionFourAnalyzer) diff --git a/HeterogeneousCore/TestModules/test/SchemaEvolutionOneReader.py b/HeterogeneousCore/TestModules/test/SchemaEvolutionOneReader.py new file mode 100644 index 0000000000000..5216a536bed7c --- /dev/null +++ b/HeterogeneousCore/TestModules/test/SchemaEvolutionOneReader.py @@ -0,0 +1,18 @@ +import FWCore.ParameterSet.Config as cms + +process = cms.Process('Reader') +process.maxEvents = cms.untracked.PSet(input = cms.untracked.int32(1)) + +# read the products from a 'testOne.root' file +process.source = cms.Source('PoolSource', + fileNames = cms.untracked.vstring('file:/afs/cern.ch/user/m/maholzer/SchemaEvolutionTestData/SEOne.root') +) + +# enable logging for the analyser +process.MessageLogger.soAAnalyzer = cms.untracked.PSet() + +process.evolutionOneAnalyzer = cms.EDAnalyzer('EvolutionOneAnalyzer', + source = cms.InputTag("soaproducer", "EvolutionOneProduct"), +) + +process.p = cms.Path(process.evolutionOneAnalyzer) diff --git a/HeterogeneousCore/TestModules/test/SchemaEvolutionThreeReader.py b/HeterogeneousCore/TestModules/test/SchemaEvolutionThreeReader.py new file mode 100644 index 0000000000000..d9f753294f9b5 --- /dev/null +++ b/HeterogeneousCore/TestModules/test/SchemaEvolutionThreeReader.py @@ -0,0 +1,18 @@ +import FWCore.ParameterSet.Config as cms + +process = cms.Process('Reader') +process.maxEvents = cms.untracked.PSet(input = cms.untracked.int32(1)) + +# read the products from a 'testThree.root' file +process.source = cms.Source('PoolSource', + fileNames = cms.untracked.vstring('file:/afs/cern.ch/user/m/maholzer/SchemaEvolutionTestData/SEThree.root') +) + +# enable logging for the analyser +process.MessageLogger.soAAnalyzer = cms.untracked.PSet() + +process.evolutionThreeAnalyzer = cms.EDAnalyzer('EvolutionThreeAnalyzer', + source = cms.InputTag("soaproducer", "EvolutionThreeProduct"), +) + +process.p = cms.Path(process.evolutionThreeAnalyzer) diff --git a/HeterogeneousCore/TestModules/test/SchemaEvolutionTwoReader.py b/HeterogeneousCore/TestModules/test/SchemaEvolutionTwoReader.py new file mode 100644 index 0000000000000..864ddb4efb1a9 --- /dev/null +++ b/HeterogeneousCore/TestModules/test/SchemaEvolutionTwoReader.py @@ -0,0 +1,18 @@ +import FWCore.ParameterSet.Config as cms + +process = cms.Process('Reader') +process.maxEvents = cms.untracked.PSet(input = cms.untracked.int32(1)) + +# read the products from a 'testTwo.root' file +process.source = cms.Source('PoolSource', + fileNames = cms.untracked.vstring('file:/afs/cern.ch/user/m/maholzer/SchemaEvolutionTestData/SETwo.root') +) + +# enable logging for the analyser +process.MessageLogger.soAAnalyzer = cms.untracked.PSet() + +process.evolutionTwoAnalyzer = cms.EDAnalyzer('EvolutionTwoAnalyzer', + source = cms.InputTag("soaproducer", "EvolutionTwoProduct"), +) + +process.p = cms.Path(process.evolutionTwoAnalyzer) diff --git a/HeterogeneousCore/TestModules/test/SchemaEvolutionWriter.py b/HeterogeneousCore/TestModules/test/SchemaEvolutionWriter.py new file mode 100644 index 0000000000000..2602cf1915406 --- /dev/null +++ b/HeterogeneousCore/TestModules/test/SchemaEvolutionWriter.py @@ -0,0 +1,24 @@ +import FWCore.ParameterSet.Config as cms + +process = cms.Process("Writer") +process.source = cms.Source('EmptySource') +process.maxEvents = cms.untracked.PSet(input = cms.untracked.int32(1)) + +process.load("FWCore.MessageService.MessageLogger_cfi") +process.MessageLogger.cerr.FwkReport.reportEvery = 1 + +# enable logging for the analysers +process.MessageLogger.soAAnalyzer = cms.untracked.PSet() + +# Produce zeroth evolution of the SoA product +process.soaproducer = cms.EDProducer("SchemaEvolutionSoAProducer") + +# write all products to a 'test.root' file +process.output = cms.OutputModule('PoolOutputModule', + fileName = cms.untracked.string('/afs/cern.ch/user/m/maholzer/SchemaEvolutionTestData/SEZero.root'), + outputCommands = cms.untracked.vstring('keep *') +) + +# Add to process path +process.p = cms.Path(process.soaproducer) +process.output_path = cms.EndPath(process.output) diff --git a/HeterogeneousCore/TestModules/test/SchemaEvolutionZeroReader.py b/HeterogeneousCore/TestModules/test/SchemaEvolutionZeroReader.py new file mode 100644 index 0000000000000..04b9aa3907ea1 --- /dev/null +++ b/HeterogeneousCore/TestModules/test/SchemaEvolutionZeroReader.py @@ -0,0 +1,18 @@ +import FWCore.ParameterSet.Config as cms + +process = cms.Process('Reader') +process.maxEvents = cms.untracked.PSet(input = cms.untracked.int32(1)) + +# read the products from a 'test.root' file +process.source = cms.Source('PoolSource', + fileNames = cms.untracked.vstring('file:/afs/cern.ch/user/m/maholzer/SchemaEvolutionTestData/SEZero.root') +) + +# enable logging for the analyser +process.MessageLogger.soAAnalyzer = cms.untracked.PSet() + +process.evolutionZeroAnalyzer = cms.EDAnalyzer('EvolutionZeroAnalyzer', + source = cms.InputTag("soaproducer", "EvolutionZeroProduct"), +) + +process.p = cms.Path(process.evolutionZeroAnalyzer) diff --git a/HeterogeneousCore/TestModules/test/TestSchemaEvolution.sh b/HeterogeneousCore/TestModules/test/TestSchemaEvolution.sh new file mode 100755 index 0000000000000..a2e9e3994d38f --- /dev/null +++ b/HeterogeneousCore/TestModules/test/TestSchemaEvolution.sh @@ -0,0 +1,38 @@ + #! /bin/bash -e + +if ! [ "${LOCALTOP}" ]; then + export LOCALTOP=${CMSSW_BASE} + cd ${CMSSW_BASE} +fi + +echo "--------------------------------------------------------------------------------" +echo "$ cmsRun ${LOCALTOP}/src/HeterogeneousCore/TestModules/test/SchemaEvolutionZeroReader.py" +echo +cmsRun ${LOCALTOP}/src/HeterogeneousCore/TestModules/test/SchemaEvolutionZeroReader.py || exit $? +echo +echo "--------------------------------------------------------------------------------" +echo "$ cmsRun ${LOCALTOP}/src/HeterogeneousCore/TestModules/test/SchemaEvolutionOneReader.py" +echo +cmsRun ${LOCALTOP}/src/HeterogeneousCore/TestModules/test/SchemaEvolutionOneReader.py || exit $? +echo +echo "--------------------------------------------------------------------------------" +echo "$ cmsRun ${LOCALTOP}/src/HeterogeneousCore/TestModules/test/SchemaEvolutionTwoReader.py" +echo +cmsRun ${LOCALTOP}/src/HeterogeneousCore/TestModules/test/SchemaEvolutionTwoReader.py || exit $? +echo +echo "--------------------------------------------------------------------------------" +echo "$ cmsRun ${LOCALTOP}/src/HeterogeneousCore/TestModules/test/SchemaEvolutionThreeReader.py" +echo +cmsRun ${LOCALTOP}/src/HeterogeneousCore/TestModules/test/SchemaEvolutionThreeReader.py || exit $? +echo +echo "--------------------------------------------------------------------------------" +echo "$ cmsRun ${LOCALTOP}/src/HeterogeneousCore/TestModules/test/SchemaEvolutionFourReader.py" +echo +cmsRun ${LOCALTOP}/src/HeterogeneousCore/TestModules/test/SchemaEvolutionFourReader.py || exit $? +echo +echo "--------------------------------------------------------------------------------" +echo "$ cmsRun ${LOCALTOP}/src/HeterogeneousCore/TestModules/test/SchemaEvolutionFiveReader.py" +echo +cmsRun ${LOCALTOP}/src/HeterogeneousCore/TestModules/test/SchemaEvolutionFiveReader.py || exit $? +echo +echo "--------------------------------------------------------------------------------"