PerfTools/Perfetto: in-process Perfetto tracing service - #51271
PerfTools/Perfetto: in-process Perfetto tracing service#51271felicepantaleo wants to merge 4 commits into
Conversation
|
cms-bot internal usage |
|
-code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51271/49862 ERROR: Build errors found during clang-tidy run. |
|
Demonstration video |
|
test parameters:
|
|
@cmsbuild please test |
|
type ngt |
|
@cmsbuild code-checks |
|
-code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51271/49863 ERROR: Build errors found during clang-tidy run. |
|
@cmsbuild please test with cms-sw/cmsdist#10668 |
|
@cmsbuild code-checks with cms-sw/cmsdist#10668 |
|
@cmsbuild code-checks with cms.week0.PR_596340/56.1 |
|
@cmsbuild code-checks with cms.week0_PR_596340/56.1 |
|
code-checks with cms.week0.PR_3f29859a/100.0-cced86a6d5071160d38b54fd5b3ba33d |
| @@ -0,0 +1,101 @@ | |||
| // Original author: Felice Pantaleo, felice.pantaleo@cern.ch, 02/2026 | |||
| #pragma once | |||
There was a problem hiding this comment.
AFAIK we do not use #pragma once in CMSSW.
|
+heterogenous |
for some reason this did not work, so I've removed pragma onces |
|
@cmsbuild please test |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51271/49968
|
|
+1 Size: This PR adds an extra 32KB to repository Comparison SummarySummary:
|
Ah... that's because my phone has decided to write "heterogenous" instead of "heterogeneous", and it doesn't seem possible to make it re-learn the correct spelling :-( |
|
+heterogeneous |
|
@cms-sw/orp-l2 this is now ready for merging. |
|
@felicepantaleo, there is still @makortel's question about adding the new package, and in particular of who should be responsible for reviewing and signing for it. |
cms-sw/cms-bot#2802 |
|
Pull request #51271 was updated. @Dr15Jones, @makortel, @smuzaffar can you please check and sign again. |
|
The Tracer service already can create a perfecto trace. |
fair point, and you're right that the Tracer can already produce a Perfetto-viewable trace. The difference is how: the Tracer converts its text log offline (edmTracerCompactLogViewer.py --web), whereas this service writes a native .pftrace in-process, so you can profile the application live as it runs rather than post-processing a log. |
|
any other comment? @smuzaffar @Dr15Jones @makortel |
|
@makortel is on vacation for the next week or so. The full review will happen once he's back. |
|
I don't see anything here that would require this package to be under The dependence on CUDA makes me wonder if I can take a deeper look in the coming days. |
|
I think this PR can go as it is now.
The value of making this PR available now for memory management optimization outweighs the tiny details I mentioned, that will cost extra delays. PS: there is also a glitch on the event rate for the very first events: (It's high, should be low): |
| // A small fixed-size, allocation-free per-thread stack. Module re-entrancy | ||
| // from work-stealing is shallow in practice; beyond the cap we keep counting | ||
| // depth (so push/pop stay balanced) but stop recording, and report no module. | ||
| constexpr int kMaxDepth = 64; |
There was a problem hiding this comment.
We don't need a stack here. Tasks are not preempted by TBB, nor suspended by us (in which future case we would need to stop/restart the range using new FW signals).
Pre/post signals guaranteed as thread local here are also guaranteed to not intersect with each other within a thread (@makortel ?).
A successful run with this patch (suggested) confirmed it:
// Original author: Felice Pantaleo, felice.pantaleo@cern.ch, 02/2026
#include "PerfTools/Perfetto/interface/CMSSWPerfettoModuleContext.h"
+#include <cassert>
+
namespace cms::perfetto {
namespace {
// A small fixed-size, allocation-free per-thread stack. Module re-entrancy
// from work-stealing is shallow in practice; beyond the cap we keep counting
// depth (so push/pop stay balanced) but stop recording, and report no module.
- constexpr int kMaxDepth = 64;
- thread_local ModuleContext g_stack[kMaxDepth];
- thread_local int g_depth = 0;
+ thread_local ModuleContext g_context;
+ thread_local bool g_present = false;
const ModuleContext g_none{};
} // namespace
void pushModuleContext(ModuleContext const& ctx) noexcept {
- if (g_depth >= 0 && g_depth < kMaxDepth)
- g_stack[g_depth] = ctx;
- ++g_depth;
+ assert(not g_present);
+ g_present = true;
+ g_context = ctx;
}
void popModuleContext() noexcept {
- if (g_depth > 0)
- --g_depth;
+ assert(g_present);
+ g_present = false;
}
- void resetModuleContext() noexcept { g_depth = 0; }
+ void resetModuleContext() noexcept { g_present = false; }
ModuleContext const& currentModuleContext() noexcept {
- if (g_depth > 0 && g_depth <= kMaxDepth)
- return g_stack[g_depth - 1];
+ if (g_present) {
+ return g_context;
+ }
return g_none;
}
} // namespace cms::perfetto
There was a problem hiding this comment.
We don't need a stack here. Tasks are not preempted by TBB, nor suspended by us (in which future case we would need to stop/restart the range using new FW signals).
If a module uses TBB's parallel constructs, e.g. tbb::parallel_for() without enclosing the call in tbb::this_task_arena::isolate(), that parallel_for() can steal tasks from other activities in other threads.
While our recommendation is to isolate all TBB parallel calls, it is not enforced.
Seed the sliding rate window with the start of the first event: at startup all streams complete their first (slow) event almost simultaneously, so a window of completion times alone spans only that tight burst and reports a spuriously high rate exactly when the job is at its slowest.
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51271/50430
|
|
Pull request #51271 was updated. @Dr15Jones, @cmsbuild, @makortel, @smuzaffar can you please check and sign again. |
Right, the throughput counter's sliding window was built from event-completion times only, at startup all N streams complete their first (slow) event nearly simultaneously, so the window spanned just that burst and read spuriously high (up to 4.9 ev/s vs 1.4 ev/s steady in an 8-stream NGT-scouting test). The window is now seeded with the first event start, so the counter starts at ~1/T(first event) and ramps up honestly: first samples 0.74 → 0.94 → 1.30 → … → 1.40 ev/s steady, no spike. |
|
@cmsbuild please test |
|
+1 Size: This PR adds an extra 28KB to repository Comparison SummarySummary:
|
7 participants
Adds PerfTools/Perfetto, an EDM service (PerfettoTraceService) that records an in-process Perfetto (https://perfetto.dev) trace (.pftrace) of a cmsRun job, openable by drag-and-drop at https://perfetto.web.cern.ch , entirely client-side, together with a small dependency-free monitor hook in HeterogeneousCore/AlpakaInterface that the Alpaka caching allocator uses to report device-memory traffic.
What it records:
edm::stream, so independent modules running concurrently within a stream, and an ExternalWork module'sacquire()/produce()running on different threads — nest correctly without overlapping or mis-paired slices;CMS_PERFETTO_FUNC()/CMS_PERFETTO_SCOPE()macros for optional intra-module instrumentation, and a traceModules filter for focused, low-overhead runs.Everything beyond the per-stream slices and counters is opt-in and off by default: with the optional features disabled the per-allocation cost is a single relaxed atomic load, and disabled trace categories cost only a predicated load.
Usage:
cmsDriver.py … --customise PerfTools/Perfetto/customisePerfetto.customise, or add the service directly; seePerfTools/Perfetto/README.md@rovere @makortel @fwyzard