Update OverlayTiming to support BIB random mixing by madbaron · Pull Request #413 · key4hep/k4FWCore

madbaron · 2026-07-06T13:11:13Z

BEGINRELEASENOTES

Extend OverlayTiming with random background-file mixing: the new RandomMixBackgroundFiles option treats each file in a background group as an independent event source and picks a random set of files for every overlaid event. BackgroundFileNames entries may now be directories (their .root files are used).
Add the MergeMCParticles option to OverlayTiming (default true); when false, background MCParticles are not stored, tracker hits keep the momentum of their originating particle and calorimeter contributions get an empty particle.
Serialize all background ROOT I/O on a dedicated worker thread so OverlayTiming is safe to run with intra-event multithreading.

ENDRELEASENOTES

This PR updates OverlayTiming with the logic used by the muon collider software to overlay the BIB pseudo-events (from https://github.com/MuonColliderSoft/k4Reco/blob/main/k4Reco/Overlay/components/OverlayTimingRandomMix.cpp).
I opted for porting the changes over rather than asking to include a second algorithm, since the code was 95% the same.

The updated algorithm uses TBB for intra-event multithreading in processing the thousands of inputs for the overlay.

madbaron · 2026-07-07T06:42:43Z

The downstream build failure doesn't seem related to the changes - but I can't retrigger it.

andread3vita · 2026-07-24T11:49:42Z

Hi! I tried this implementation, and if I simply set overlay.RandomMixBackgroundFiles = True, the overlay works using just a single background file instead of 40:

from Gaudi.Configuration import INFO

from k4FWCore import ApplicationMgr
from k4FWCore import IOSvc
from Configurables import EventDataSvc
from Configurables import OverlayTiming
from Configurables import UniqueIDGenSvc

from pathlib import Path


background_base = Path("/eos/experiment/fcc/ee/simulation/key4hep_2026_04_20/91GeV/IDEA_o1_v03/IPC_Z_background")
background_file_list = []
for d in sorted(background_base.iterdir()):
    background_file_list.append(str(d))
            
id_service = UniqueIDGenSvc("UniqueIDGenSvc")
eds = EventDataSvc("EventDataSvc")
iosvc = IOSvc()
# iosvc.Input = "/afs/cern.ch/user/a/aloeschc/fcc_fullsim_testing_grounds/data/IDEA/IDEA_o1_v03/physics_events/p8_ee_Z_qqbar_ud/1000_91.188GeV_ISR_FSR/000/IDEA_o1_v03_1000_p8_ee_Z_qqbar_ud_91.188GeV_ISR_FSR.root"
iosvc.Input = "/afs/cern.ch/work/a/adevita/public/testBIB/IDEA_test/muonGunIDEAv3o1.root" 

iosvc.Output = "IDEA_o1_v03_OverlayIPC_test.root"

overlay = OverlayTiming()
overlay.MCParticles = "MCParticles"
overlay.BackgroundMCParticleCollectionName = "MCParticles"
overlay.SimTrackerHits = ["DCHCollection", "MuonSystemCollection", "SiWrDCollection", "SiWrBCollection", "VertexBarrelCollection", "VertexEndcapCollection", "PreshowerSystemCollection"]
overlay.SimCalorimeterHits = []
overlay.OutputSimTrackerHits = ["OverlayDCHCollection", "OverlayMuonSystemCollection", "OverlaySiWrDCollection", "OverlaySiWrBCollection", "OverlayVertexBarrelCollection", "OverlayVertexEndcapCollection", "OverlayPreshowerSystemCollection"]
overlay.OutputSimCalorimeterHits = []
overlay.OutputMCParticles = "OverlayMCParticles"
overlay.OutputCaloHitContributions = []
overlay.AllowReusingBackgroundFiles = True
overlay.CopyCellIDMetadata = True
overlay.NBunchtrain = 41          # total BX in train
overlay.NumberBackground = [1]    # one background event per BX
overlay.Delta_t = 20              # ns between BX
overlay.PhysicsBX = 21            # puts physics at 21 with 20 before & 30 after (allow for hits 200ns after event time)
overlay.Poisson_random_NOverlay = [False]
overlay.StartBackgroundEventIndex = -1
# overlay.BackgroundFileNames = [
#       background_file_list
# ]

overlay.RandomMixBackgroundFiles = True
overlay.BackgroundFileNames = [["/eos/experiment/fcc/ee/simulation/key4hep_2026_04_20/91GeV/IDEA_o1_v03/IPC_Z_background"]]

overlay.TimeWindows = {"MCParticles": [-400, 400], "DCHCollection": [-400, 400], "MuonSystemCollection": [-20, 0], "SiWrDCollection": [-20, 0],"SiWrBCollection": [-20, 0], "VertexBarrelCollection": [-20, 0],"VertexEndcapCollection": [-20, 0], "PreshowerSystemCollection": [-20, 0]}

iosvc.outputCommands = ["drop *", "keep OverlayDCHCollection*", "keep OverlaySiWrDCollection*", "keep OverlaySiWrBCollection*", "keep OverlayVertexBarrelCollection*", "keep OverlayVertexEndcapCollection*", "keep OverlayMC*", "keep *EventHeader*"]


ApplicationMgr(TopAlg=[overlay],
               EvtSel="NONE",
               EvtMax=1,
               ExtSvc=[eds],
               OutputLevel=INFO,
               )

ArinaPon · 2026-07-28T11:17:45Z

Hello! while testing the implementation, I think I might have an idea why only one background file was being used across the bunch train.

fileIndices is shuffled before the BX loop, but k starts again from 0 for every BX. With NumberBackground = [1], every BX therefore selects:

fileIndices[0]

And I guess this means that the same randomly selected background file is reused for all BXs of the signal event.

I tried adding a counter outside the BX loop:

size_t fileCursor = 0;

and changing the file selection to:

const int fileIndex =
    m_randomMix ? fileIndices[fileCursor++ % fileIndices.size()] : 0;

With this change, the code continues through the shuffled list instead of starting again from the first file for every BX.

I rebuilt and tested this with 120 IPC background files and the overlay completed successfully, the different events showed different background patterns.

madbaron · 2026-07-31T21:33:06Z

Thanks for the checks @andread3vita and the suggestion for a fix @ArinaPon.
I ended up implementing it ~1:1, with the main difference that on wrap of the fileCursor I reshuffle so that there is no recurring pattern in overlaid background events.

…Core into add_overlay_BIB_random_mix

tmadlener · 2026-08-11T19:14:37Z

+With many large background files the algorithm is dominated by reading and
+decompressing them. Set `OverlayThreads` to a value greater than 1 to read and
+decompress the background files of a single event on several threads:


Conceptually this might interfere with Gaudis internal scheduling (even if we also use tbb to do our multithreading). It's unclear to me whether the Gaudi internal tbb bits communicate with the tbb bits here.

There is precedent for doing this though as the CKF in k4ActsTracking also does some internal multithreading. This might need some policy discussion as it could imply different usage patterns for different community (e.g. run the general chain on a single thread but branch out to multi-threading in dedicated algorithms vs. running the full chain on multiple threads with Gaudi scheduling but no algorithm-internal multi-threading).

Indeed. In the ideal world you might want to allow users to do a combination of both, if possible.
For now, especially in colliders that are computationally challenging per event, being able to use MT inside the same event is much more important than multi-threading over events, which can be done trivially in batch jobs anyway.

Maybe @jmcarcell knows if functional algorithms can already propagate that to the Gaudi scheduler somehow. Otherwise the potential interplay will for now just be another thing to document.

tmadlener · 2026-08-12T11:48:41Z

+  // Advance the cursor for (group, file) and return the raw entry to read.
+  // Cheap and I/O-free, so calling it serially (during the work-list build)
+  // does not limit read parallelism.
+  size_t reserve(int group, int file) {
+    std::lock_guard<std::mutex> lock(m_ioMutex);
+    size_t& entry = m_nextEntry[group][file];
+    const size_t e = entry;
+    const size_t total = m_totalNumberOfEvents[group][file];
+    entry = (total > 0) ? (entry + 1) % total : entry + 1; // wrap once the total is known
+    return e;
+  }


Do we need to lock this? The comment seems to imply it happens in a sequential piece of code(?).

Within a single event this is only called from the sequential work-list build.
However, my understanding is that operator() is const and the scheduler may run several events concurrently that all share this EventHolder, so the lock is still needed.
(The code comment is misleading and I'll fix it)

Ok, I think now I understand. The lock is there for "Gaudi does the multithreading"-mode.

tmadlener · 2026-08-12T11:50:53Z

+  // the shared per-group reader is used under the mutex.
+  podio::Frame readAt(int group, int file, size_t rawEntry) {
+    if (m_randomMix) {
+      podio::Reader reader = podio::makeReader(m_fileNames[group][file]);


Intuitively this has a lot of overhead hat we are repeating for every event, because opening the file initializes a bunch of state in the reader which is quite expensive since the assumption is that this amortizes over many events and so we do a bunch of work upfront to save later in the event.

I can't say how much overhead this is in practice, but if there is a way to avoid it that might be a good idea.

I had measured it on some synthetic files. It's real, but small.

Timing breakdown for the one-pseudo-event-per-file pattern:

per-file size open (makeReader+getEntries) readEvent decompress/materialise crop+merge

~40 MB (BIB-like) 1.8% (≈5 ms/file) 52% 35% 12%

55 KB 62% (≈1.2 ms/file) 33% 3% 1%

So the repeated open costs ~5 ms/file, ≈2% of the overlay time for the ~40 MB files we use.

Thanks a lot for testing and the numbers. IIUC, the driving factor here is the size of the event / collection that you read and not the file-size per se? Does the 55 KB file have a realistic number of elements? I am trying to understand whether we should document that this random mixing as it is done at the moment might incur some (quite significant) overhead if it is used on small overlay events.

madbaron added 8 commits July 6, 2026 12:57

merging randomMix functionalities into overlaytiming

5c76efa

docs

54afad5

more docs

4c17050

undo over-eager release notes

9ef1063

phrasing of pseudo-events

fdfde27

use TBB

a315f27

clang format

24b2107

update MT docs

b517240

Merge branch 'main' into add_overlay_BIB_random_mix

2fb0c4f

madbaron added 5 commits July 31, 2026 15:38

Merge branch 'main' into add_overlay_BIB_random_mix

d3758ff

implement suggestion from ArinaPon

a983a90

Fix clang-format in OverlayTiming.cpp

9aab53b

updated comments and added shuffle on wrap

349e589

make pre-commit happy again

b8b4f57

madbaron mentioned this pull request Aug 9, 2026

Tracking-related preparation for release 3.1 MuonColliderSoft/MAIAConfig#6

Merged

madbaron added 4 commits August 10, 2026 13:41

merge with upstream

0fad14e

Merge branch 'add_overlay_BIB_random_mix' of github.com:madbaron/k4FW…

33f9829

…Core into add_overlay_BIB_random_mix

preempt clang tidy

6e6b20c

fix clang-tidy

3b80ce7

tmadlener reviewed Aug 12, 2026

View reviewed changes

Merge branch 'main' into add_overlay_BIB_random_mix

a25b300

per-file size	open (`makeReader`+`getEntries`)	`readEvent`	decompress/materialise	crop+merge
~40 MB (BIB-like)	1.8% (≈5 ms/file)	52%	35%	12%
55 KB	62% (≈1.2 ms/file)	33%	3%	1%

Uh oh!

Conversation

madbaron commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

madbaron commented Jul 7, 2026

Uh oh!

andread3vita commented Jul 24, 2026

Uh oh!

ArinaPon commented Jul 28, 2026

Uh oh!

madbaron commented Jul 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

madbaron Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

madbaron commented Jul 6, 2026 •

edited

Loading

madbaron Aug 14, 2026 •

edited

Loading