[DRAFT POC] Add Arrow IPC stream reader for EDM4hep input

[wdconinc]

Briefly, what does this PR introduce? Please link to any relevant presentations or discussions.

This proof-of-concept implements an Apache Arrow IPC stream reader for EDM4hep input in EICrecon. This enables real-time streaming from simulation (npsim/ddsim) to reconstruction (eicrecon) using named pipes or files, without intermediate file I/O.

Related Work:

• DD4hep Arrow writer PR: [DRAFT POC] Add Arrow IPC stream writer for EDM4hep output AIDASoft/DD4hep#1658
• podio Arrow backend: Refactor backend library loading and rename Arrow suffix to PodioArrow AIDASoft/podio#999

Architecture:

The implementation adds a new JANA2 event source (JEventSourcePODIOArrowStream) that:

1. Opens Apache Arrow IPC streams (files or named pipes)
2. Reads RecordBatches incrementally (one per event)
3. Converts to podio::Frame using podio::convertTableToFrame() (pending podio upgrade)
4. Provides frames to JANA2 for reconstruction

Current Status:

• ✅ Arrow dependency integrated into build system
• ✅ Event source implemented and compiles successfully
• ✅ Registered with JANA2 framework
• ✅ CI test added
• ✅ Documentation provided
• ⚠️ Arrow-to-Frame conversion pending podio >= 1.8 with Arrow support

Pending Dependencies:

1. DD4hep PR Merge CalorimeterClusterRecoCoG and ImagingClusterReco #1658 (Arrow writer) - Currently under review
2. podio >= 1.8 with ENABLE_ARROW in eic-shell

What is the urgency of this PR?

• High (please describe reason below)
• Medium
• Low

Low urgency - This is a proof-of-concept that establishes the integration framework. Full functionality depends on upstream dependencies.

What kind of change does this PR introduce?

• Bug fix (issue #__)
• New feature (issue #__)
• Optimization (issue #__)
• Updated parameters, constants (issue #__)
• Updated documentation
• other: __

New feature: Arrow IPC stream reader for real-time simulation-to-reconstruction streaming

Please check if any of the following apply

• This PR requires changes to geometry (epic PR: __)
• This PR requires changes to EDM4eic (EDM PR: __)
• This PR introduces breaking changes. Please describe changes users need to make below.
• This PR changes default behavior. Please describe changes below.
• AI was used in preparing this PR. Please describe usage below.

AI Usage: GitHub Copilot assisted with:

• Implementation of Arrow stream reader following existing JEventSourcePODIO patterns
• CMake integration for Arrow dependency
• Documentation and test script development
• PR description preparation

---

Implementation Details

Files Added/Modified

New Files:

• src/services/io/podio/JEventSourcePODIOArrowStream.h - Event source header
• src/services/io/podio/JEventSourcePODIOArrowStream.cc - Event source implementation
• src/tests/arrow_stream_test/ - CI test for verification
• docs/arrow_stream_reader.md - Comprehensive documentation

Modified Files:

• CMakeLists.txt - Add Apache Arrow dependency (>= 10.0.0)
• src/services/io/podio/CMakeLists.txt - Link arrow_shared library
• src/services/io/podio/podio.cc - Register Arrow event source
• src/tests/CMakeLists.txt - Add Arrow stream test

Usage Examples (Future)

Once dependencies are available:

File-based input:

npsim --outputFile simulation.arrow --outputType Geant4Output2EDM4hepArrowStream --numberOfEvents 100
eicrecon simulation.arrow -Ppodio:output_file=reconstructed.root

Named pipe streaming:

mkfifo simulation.arrow
npsim --outputFile simulation.arrow --outputType Geant4Output2EDM4hepArrowStream &
eicrecon simulation.arrow -Ppodio:output_file=reconstructed.root

Known Limitations

Current POC:

• Arrow-to-Frame conversion not yet functional (requires podio >= 1.8)
• Logs Arrow schema but stops after first event
• No error recovery or performance optimization

Next Steps:

1. Wait for DD4hep PR Merge CalorimeterClusterRecoCoG and ImagingClusterReco #1658 to merge
2. Update to podio >= 1.8 with Arrow support
3. Enable full Arrow-to-Frame conversion
4. Test end-to-end streaming
5. Performance optimization (zero-copy, parallel processing)

CI/Testing

A proof-of-concept test is included that verifies:

• Code compiles successfully
• Arrow dependencies are properly linked
• Event source is registered with JANA2
• Integration framework is in place

Full end-to-end testing will be enabled once dependencies are available.

---

Why This Matters

Real-time streaming eliminates the need for intermediate file I/O, enabling:

• Lower latency: Reconstruction begins as soon as first event is simulated
• Reduced disk usage: No large intermediate simulation files
• Better scalability: Parallel simulation and reconstruction pipelines
• Debugging: Inspect reconstructed events in real-time during simulation

This infrastructure will be valuable for high-throughput workflows at the EIC.

[github-actions]

Capybara summary for PR 2857

• rec_dis_10x100_minQ2=0_craterlake_10x100_MT (Δ: 354)
• rec_dis_10x100_minQ2=100_craterlake_tracking_2DStrip (Δ: 0)
• rec_dis_10x100_minQ2=100_craterlake_tracking_only (Δ: 0)
• rec_dis_10x130_minQ2=1_craterlake_10x130 (Δ: 0)
• rec_dis_10x250_minQ2=1_craterlake_10x250 (Δ: 0)
• rec_dis_18x275_minQ2=0_craterlake_18x275 (Δ: 0)
• rec_dis_18x275_minQ2=0_craterlake_18x275_MT (Δ: 343)
• rec_dis_18x275_minQ2=0_craterlake_18x275_ST_MT (Δ: 332)
• rec_dis_18x275_minQ2=100_craterlake_18x275 (Δ: 0)
• rec_dis_18x275_minQ2=100_craterlake_18x275_MT (Δ: 447)
• rec_dis_18x275_minQ2=100_craterlake_18x275_ST_MT (Δ: 395)
• rec_dis_5x41_minQ2=0_craterlake_5x41 (Δ: 0)
• rec_e_1GeV_20GeV_craterlake (Δ: 0)
• rec_pi_1GeV_20GeV_craterlake (Δ: 3)
• two_stage_raw_e_1GeV_20GeV_craterlake (Δ: 0)
• two_stage_rec_e_1GeV_20GeV_craterlake (Δ: 0)

Last updated 2026-08-11T21:22-04:00 cf925f8