Add Gaudi Functional C++ Class Generator

[ianna]

BEGINRELEASENOTES

• Gaudi Functional C++ Class Generator (generateFunctional)

ENDRELEASENOTES

Gaudi Functional C++ Class Generator

k4FWCore/helpers/generateFunctional generates Gaudi Functional C++ algorithm boilerplate based on user-defined specifications. It supports both Gaudi::Functional and k4FWCore frameworks.

Features

• Framework support: Both Gaudi and k4FWCore (default: k4FWCore)
• 5 functional types: Consumer, Producer, Transformer, MultiTransformer, FilterPredicate
• Type inference: functional type is auto-detected from the number of inputs and outputs; only filter needs to be specified explicitly
• Smart type parsing: handles complex types like podio::UserDataCollection<float>
• Constructor scaffolding: generates KeyValue/KeyValues initialization with correct brace wrapping for all functional types
• Type-safe operator(): creates the correct signature and a stub implementation
• Gaudi properties: add configurable Gaudi::Property<T> members with optional private: placement (--private-properties)
• EDM4hep/podio support: automatic #include directives for edm4hep and podio types
• Runtime (variable-length) collections: --runtime-inputs and --runtime-outputs for std::vector<const T*> / std::vector<T> patterns
• Type aliases: --type-aliases emits using XxxColl = …; aliases and uses them in the signature
• Event context: --event-context adds const EventContext& as the first operator() argument and scaffolds finalize() with mutex bookkeeping
• CMake skeleton: --cmake also writes a CMakeLists.txt next to the source
• Safe writes: refuses to overwrite existing files unless --force is passed
• Command tracking: records the generation command as a comment in the output file

Input format

For inputs/outputs: TypeName:LocationKey

• TypeName: required (supports templates like podio::UserDataCollection<float>)
• LocationKey: optional — defaults to the type name stripped of namespace and Collection suffix

For properties: Type:Name:DefaultValue[:Description]

Examples

k4FWCore Producer (type inferred)

generateFunctional MyProducer \
  -o 'edm4hep::MCParticleCollection:OutputCollection' \
  -p 'int:ExampleInt:3:Example integer property'

k4FWCore Multi-output Producer (MultiTransformer inferred from multiple outputs)

generateFunctional MyProducer \
  -o 'podio::UserDataCollection<float>:VectorFloat' \
     'edm4hep::MCParticleCollection:MCParticles1' \
     'edm4hep::SimTrackerHitCollection:SimTrackerHits' \
  -p 'int:ExampleInt:3:Example int that can be used in the algorithm' \
     'int:magicNumberOffset:0:Integer to add to the dummy values'

k4FWCore Consumer (type inferred)

generateFunctional MyConsumer \
  -i 'edm4hep::MCParticleCollection:InputCollection'

k4FWCore Transformer (type inferred, properties private)

generateFunctional MyTransformer \
  -i 'edm4hep::MCParticleCollection:InputCollection' \
  -o 'edm4hep::MCParticleCollection:OutputCollection' \
  --private-properties \
  -p 'int:Offset:10:Integer to add to values'

Runtime (variable-length) collections

generateFunctional MyConsumer \
  -i 'edm4hep::MCParticleCollection:Inputs' \
  --runtime-inputs 'edm4hep::MCParticleCollection:Inputs:MCParticles0,MCParticles1'

With EventContext and finalize() scaffold

generateFunctional MyTransformer \
  -i 'edm4hep::MCParticleCollection:InputCollection' \
  -o 'edm4hep::MCParticleCollection:OutputCollection' \
  --event-context --private-properties

FilterPredicate (type must be explicit)

generateFunctional MyFilter filter \
  -i 'edm4hep::MCParticleCollection:InputCollection'

Gaudi framework, namespace, CMake output

generateFunctional MySum \
  -i 'Input1:Loc1' 'Input2:Loc2' \
  -o 'Output:OutLoc' \
  --framework gaudi -n MyNamespace --cmake

Command-line options

positional arguments:
  class_name                       Name of the C++ class to generate
  {consumer,producer,transformer,filter}
                                   Functional type (optional — inferred when omitted)
options:
  -i, --inputs TYPE:KEY [...]      Input collection specs
  -o, --outputs TYPE:KEY [...]     Output collection specs (mutually exclusive with --runtime-outputs)
  --runtime-outputs TYPE           Dynamic output: returns std::vector<TYPE> (k4FWCore only)
  --runtime-inputs TYPE:KEY:DEF[,DEF2,...] [...]
                                   Variable-length inputs with default location names (k4FWCore only)
  -p, --properties TYPE:NAME:DEFAULT[:DESC] [...]
                                   Gaudi::Property<T> members
  -n, --namespace NAME             Wrap class in a C++ namespace
  --framework {gaudi,k4fwcore}     Target framework (default: k4fwcore)
  --use-class                      Emit 'class … { public: }' instead of 'struct'
  --private-properties             Place properties under 'private:'
  --type-aliases                   Emit 'using XxxColl = …;' aliases for input types
  --all-keyvalues                  Treat all inputs as KeyValues vectors
  --keyvalues-inputs KEY [...]     Promote specific inputs to KeyValues vectors
  --event-context                  Add EventContext arg and finalize() scaffold
  --cmake                          Also emit a CMakeLists.txt skeleton
  -f, --output-file PATH           Output file (default: <ClassName>.cpp)
  --force                          Overwrite existing files

How to run

# Recommended — uv resolves Python and Jinja2 automatically (PEP 723)
uv run generateFunctional MyProducer -o 'edm4hep::MCParticleCollection:MCParticles'

# Direct execution via shebang (requires uv on PATH)
./generateFunctional MyProducer -o 'edm4hep::MCParticleCollection:MCParticles'

# Plain Python (requires jinja2 in the active environment)
python3 generateFunctional MyProducer -o 'edm4hep::MCParticleCollection:MCParticles'

After cmake --install, generateFunctional is available on PATH in the Key4hep environment.

[ianna]

@BrieucF - please, check. Thanks!

[jmcarcell]

Without any tests, inevitably this will not work in the future if something changes and we won't be able to know.

[ianna]

Without any tests, inevitably this will not work in the future if something changes and we won't be able to know.

Thanks for looking into it. A very good point! Shall I add the generation and compilation for it to CI tests? Thanks

[Zehvogel]

@tmadlener weren't you also working on something like this? :)

[tmadlener]

Yes, sort of. I have a semi-working thing for a static website. But that also doesn't have the tests it should have. In the end, I don't care too much which version lands as long as one does.

[tmadlener]

Thanks a lot for this. I think the general idea of the script goes into the exact right direction, i.e. we want something that generates boilerplate for something that the user wants. I didn't look in extreme detail, but I see a few general improvements (also partially highlighted in the inline comments):

• Currently the user needs to know which functional type they want. But this is entirely specified by the number of inputs and outputs, so I would just determine that from there.
• Generally the implementation could probably be improved by sprinkling a few classes into the whole thing for holding intermediate information instead of passing around tuples of strings of various lengths. This would probably also make it easier to simply do some of the parsing / processing up-front and then pass the information around instead of re-parsing it several times.
• The current implementation does not handle the possibility of variable length inputs / outputs (I think). This is only possible in k4FWCore Functional algorithms though.

I think this is borderline complex enough to warrant the use of a template engine to handle all the string formatting. It introduces some overhead, and would require writing some templates, but it would simplify the python script parts by quite potentially.

As already mentioned we definitely need tests that ensure that the outputs compile. That would probably be the first thing I do, because once that is in place refactoring and extending the implementation can be done with some guard rails.

[tmadlener]

As mentioned in the meeting today already it would be nice to have tests for this. Given that you also generate a minimal cmake script this should be fairly straight forward, essentially for various options (I would say each type of algorithm at least once) one would

• Run the script (probably inside a "sandbox" directory) to generate source code and cmake
• Run cmake
• Run build

The main problem with that is that we would have to hook that into the existing test harness in the CMakeLists.txt if we just want to run it as part of the "standard" CI. An alternative option would be a custom github actions workflow where we could run this without that harness. (Opinions @jmcarcell?).

In any case I think a first step would to just write some bash scripts for each option with the three steps above and then we can decide later where / how we hook them up to CI).

---

Some other general comments:

• gaudi_gen.py is probably not descriptive enough. In the end people would like to use this from within a Key4hep environment and I think something like generatFunctional.py or generateAlgorithm.py (or even without the .py suffix) would make that a bit more explicit
• This needs to be installed for it to be truly useful, otherwise people will have to clone k4FWCore and get the script from there.

For that something similar to what we do for k4run:

k4FWCore/k4FWCore/CMakeLists.txt

Line 20 in dd8d891

gaudi_install(SCRIPTS)

In this case it would probably have to be something like

gaudi_install(SCRIPTS helpers)

or alternatively via an explicit call to cmake install.

[jmcarcell]

I tried this and there are several options that are not explained, then I tried them and they seem not to do anything. Like --all-keyvalues, or --type-aliases.

The main problem with that is that we would have to hook that into the existing test harness in the CMakeLists.txt if we just want to run it as part of the "standard" CI. An alternative option would be a custom github actions workflow where we could run this without that harness. (Opinions @jmcarcell?).
Hmm if it doesn't run in ctest then it's not tested regularly, but things don't change that much here so that it would stop working. If it's going to use the cmake it creates then I think it has to be outside because it calls find_package(k4FWCore) that won't be there until k4FWCore is installed, and probably is more similar to how people will use it in the future. In that case we can have a different workflow in .github/workflows that is unrelated to the build one, that may also build k4FWCore.

I wonder at which point, in a repo that already has algorithms, it's just easier to copy and paste an existing algorithm than get the command right to run the script. If there are many different times one has to copy and paste all of them with all the edm4hep::. Maybe edm4hep:: could be removed and assume it's an EDM4hep collection (true for now, but probably not in the future). Also one has to remember the colon separators:

python test.py MyProducer -o 'edm4hep::MCParticleCollection:MCParticles' 'edm4hep::TrackCollection:Tracks' -i "edm4hep::MCParticleCollection" -p 'int:ExampleInt:3:An example integer property'

[ianna]

Here's what was created:

k4FWCore/helpers/tests/ — 8 individual test scripts + shared infrastructure:

Script	Covers
test_producer.sh	Single output, property
test_consumer.sh	Single input, property
test_transformer.sh	Single in/out, --private-properties
test_multitransformer.sh	Multiple in/out, --type-aliases, podio::UserDataCollection
test_filter.sh	FilterPredicate
test_runtime_consumer.sh	--runtime-inputs / KeyValues
test_runtime_transformer.sh	--runtime-outputs / std::vector return
test_event_context.sh	--event-context + finalize() scaffold
test_gaudi_framework.sh	--framework gaudi + --namespace
_test_common.sh	Shared: locates gaudiGen.py (installed or source), creates sandbox, run_cmake_build()
run_all_tests.sh	Runs all test_*.sh, reports pass/fail summary

k4FWCore/CMakeLists.txt — added:

install(PROGRAMS helpers/gaudiGen.py
  DESTINATION ${CMAKE_INSTALL_BINDIR})

This puts gaudiGen.py on PATH in a Key4hep environment, so the test scripts (and users) can call it without knowing where k4FWCore was cloned. The _test_common.sh helper checks the installed location first and falls back to the source tree.

[ianna]

@jmcarcell and @andresailer - please see instructions on how to run the tests here -- the README.md is a part of this PR. Please let me know if there is anything else needed. Thanks.

[ianna]

@jmcarcell and @andresailer - please review. Thanks!

[ianna]

@jmcarcell and @andresailer - please review. Thanks!

[ianna]

@jmcarcell and @andresailer - please review. Thanks!

[ianna]

All tests pass -- including the ones with generated by the script classes. The failing workflow is due to a needed change that has already been fixed in EDM4hep: key4hep/EDM4hep#497 but that change is not in the latest tag/release. I'm confident the PR can me merged.

@jmcarcell and @andresailer - please review. Thanks!

[jmcarcell]

The combination of these two does not let you to have both normal and vector inputs at the same time

[jmcarcell]

Does this force to have uv available? Why?

[jmcarcell]

Why not make it an executable?

[jmcarcell]

Given that there is a lot of AI-generated code here, have you tried running it? Do you think it is something that people will use instead of going to an LLM? Would an LLM use it instead of reading one of the existing ones? Would you use it over telling a LLM to write an algorithm? I have been playing with it, and you still have to type the types and names of the collections at a minimum. In addition, this is not very easy to maintain, the template is very complex and one (or a LLM) would have to look for where everything is defined.

[ianna]

Given that there is a lot of AI-generated code here, have you tried running it? Do you think it is something that people will use instead of going to an LLM? Would an LLM use it instead of reading one of the existing ones? Would you use it over telling a LLM to write an algorithm? I have been playing with it, and you still have to type the types and names of the collections at a minimum. In addition, this is not very easy to maintain, the template is very complex and one (or a LLM) would have to look for where everything is defined.

I’ve run it, and I see it mainly as a convenience tool. It gives you a clean, framework‑correct scaffold when you already know the types and collections, but it won’t replace asking an LLM to write an algorithm. And yes, the template is complex — that’s the cost of covering all the framework edge cases.

Where it does help is as input to an LLM: the generated scaffold gives the model a stable structure to refine instead of inventing everything from scratch.

Thanks for reviewing it.