Skip to content

ci: run the stages in order, and remove the cluster stage - #166

Merged
Thecave3 merged 2 commits into
mainfrom
ci/e2e-diagnostics
Nov 11, 2026
Merged

Thecave3 merged 2 commits into
mainfrom
ci/e2e-diagnostics

Conversation

@Thecave3

@Thecave3 Thecave3 commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

What changes for contributors

CI runs as stages, in the order drawn below, and a stage starts only when every stage before it passed. A PR that fails the commit policy no longer spends the end-to-end tests.

Commit policy
  -> Unit Tests
    -> Wheels, latrec portability, MPMC benchmark       (in parallel)
      -> E2E dApp Integration, E2E Topologies, Full-loop benchmark   (in parallel)
  • Wheels, latrec portability and the MPMC benchmark run only when the PR changes what they test: the wheels' inputs, latrec.h / latrec.c, and the queue (mpmc_queue.hpp) or its benchmark (bench_mpmc_queue.cpp, which the old filter missed). A stage that did not run does not hold up the next; a failed or cancelled one stops it.
  • The build and test stages are skipped when no changed file is code, as before. A change to .github/workflows/ or .github/scripts/ now starts every stage, so a PR that edits CI is tested by it.
  • On the PR page the checks are named CI / <stage> / <job>. CI report is still the one required check and still posts the one comment; its table now has one row for CI (the stages' detail is inside it) and one for Docs.
  • The cluster stage is removed. The end-to-end tests and the benchmarks run in their hosted workflows, as before it was added.

Change

  • ci.yml calls the stages and orders them with needs. commit-trailers.yml keeps its push trigger, build-wheels.yml its use by tag-and-release.yml. The other stage workflows lose pull_request and their paths filters and gain workflow_call (workflow_dispatch stays where they had it).
  • .github/scripts/detect_changes.sh says which stages a diff starts, on top of detect_code_changes.sh.
  • Removes the cluster stage as one change: cluster-ci.yml, cluster_stage.py with its tests, scripts/ci/, the report's rule about a skipped cluster stage, and the split of docker/Dockerfile.build into a packages image and a build image (reverts 4ea046a, 47e5b59, 3c3dd2c, 6499c9f).
  • CONTRIBUTING.md describes the order.

Checked

  • test_ci_gates.py checks the order (each stage waits for the ones before it, nothing runs before the commit policy, a skippable stage does not block the next, no stage starts on its own) and, against throwaway git repositories, which stages each kind of change starts. Breaking the order three ways, one at a time, fails it each time. 34 tests pass in .github/scripts; actionlint finds nothing in the structure of the workflows.
  • This PR's own run: every stage ran, and in order. The commit policy ended at 00:20:48, Unit Tests started at 00:20:51, the three parallel stages at 00:26:35, and the end-to-end stage at 01:02:59, three seconds after the wheels finished. CI report passed.

Not shown

  • The path where stages are skipped (a code change that touches neither latrec nor the queue, so the second stage is partly skipped). The !failure() && !cancelled() condition is written for it and the gate test checks the text, but this PR changes CI files, so its own run took the path where everything runs.
  • The wheels build ran 36 minutes on x86_64 in this run (6 and 8 minutes in earlier runs), and the gate puts it ahead of the end-to-end stage, so this run took 47 minutes. When wheels is slow, everything after it waits.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

CI report — a05c83f — ✅ all checks passed

Workflow Result Time Run
CI ✅ success 46m42s #1
Docs ✅ success 0m26s #75
CI

✅ posix/ipc

  • dApp exit: 0
  • Indications received: 7

✅ posix/tcp

  • dApp exit: 0
  • Indications received: 7

✅ zmq/ipc

  • dApp exit: 0
  • Indications received: 7

✅ zmq/tcp

  • dApp exit: 0
  • Indications received: 7

zmq/ipc

  • ✅ 1 RAN - 1 dApp: indications=5
    • dapp peer=t11 ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.8 max=1 @seq=1) hist[<=1:5 2-5:0 6-10:0 >10:0]
  • ✅ 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
    • dapp1 peer=t12 ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.8 max=1 @seq=1) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp2 peer=t12 ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=0.833333 max=1 @seq=1) hist[<=1:6 2-5:0 6-10:0 >10:0]
  • ✅ 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
    • dapp peer=t2a ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp peer=t2b ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]

zmq/tcp

  • ✅ 1 RAN - 1 dApp: indications=5
    • dapp peer=default ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
  • ✅ 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
    • dapp1 peer=default ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp2 peer=default ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:6 2-5:0 6-10:0 >10:0]
  • ✅ 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
    • dapp peer=default ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp peer=off100 ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]

Full-loop latency

Full-loop latency benchmark (N=1104 after 50 warmup)

All values in microseconds (μs). Link: zmq, transport: ipc, encoding: ASN.1 APER.

# Description Tags mean p50 p99 max
1 Collect indication data RECORD_BEGIN to ENCODE_E3SM_BEGIN 0.12 0.11 0.28 1.84
2 Create & encode indication ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE 0.84 0.81 1.33 13.82
3 Encode E3AP (indication) EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE 2.89 2.66 4.49 13.35
4 Queuing (indication) ENQUEUE to DEQUEUE 6.14 4.88 25.59 68.16
5 Delivery (indication) ENCODE_E3AP_DONE to SEND_DONE 0.53 0.44 2.57 13.07
6 E3 wire (RAN -> dApp) SEND_DONE to RECV 30.92 18.39 76.21 90.55
7 Decode E3AP (indication) RECV to DECODE_E3AP_DONE 1.57 1.44 8.10 16.99
8 libe3 dispatch (indication) DECODE_E3AP_DONE to DELIVER_BEGIN 0.12 0.10 0.29 0.41
9 Decode indication DELIVER_BEGIN to DECODE_E3SM_DONE 0.46 0.46 0.77 10.01
10 Process data DECODE_E3SM_DONE to ENCODE_E3SM_BEGIN 0.04 0.04 0.14 0.27
11 Create & encode control ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE 0.31 0.27 0.61 13.12
12 Encode E3AP (control) EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE 3.18 2.92 10.76 17.42
13 Queuing (control) ENQUEUE to DEQUEUE 12.22 11.96 27.20 39.85
14 Delivery (control) ENCODE_E3AP_DONE to SEND_DONE 6.65 5.37 17.93 26.00
15 E3 wire (dApp -> RAN) SEND_DONE to RECV 35.35 30.73 64.82 90.50
16 Decode E3AP (control) RECV to DECODE_E3AP_DONE 2.85 1.82 15.45 23.60
17 libe3 dispatch (control) DECODE_E3AP_DONE to DECODE_E3SM_BEGIN 0.42 0.34 0.83 7.75
18 Decode & handle control DECODE_E3SM_BEGIN to DECODE_E3SM_DONE 0.21 0.16 0.57 9.24
Total Total round-trip 105.40 93.25 173.05 194.43

ubuntu-latest, Release build, ZMQ + IPC, ASN.1 APER.

These numbers are measured inside a GitHub Actions container and should be treated as an upper bound on E3AP's and the library's own latency, not a representative deployment measurement.

MPMC queue benchmark

MpmcQueue End-to-End Latency (SPSC, 30000 items per queue size)

Queue Capacity P50 (ns) P95 (ns) P99 (ns) P99.9 (ns)
16 906 966 1047 14677
64 2529 3401 3420 22819
256 176 6094 16469 16800
1024 205 16891 17781 17842
4096 32043 43169 43900 44111

LockFreeQueue End-to-End Latency (SPSC, 30000 items per queue size)

Queue Capacity P50 (ns) P95 (ns) P99 (ns)
16 166 1857 1958
64 1948 6836 7546
256 186 1707 15408
1024 196 5343 14797
4096 196 7696 13295

Throughput (400000 items per configuration)

Configuration Queue Cap Throughput (Mops/s)
SPSC (1P×1C) 256 22.41
MPSC (4P×1C) 1024 13.81
MPMC (4P×4C) 4096 13.13
MPMC (8P×4C) 4096 11.22

Stress / Correctness (100000 items each)

Test Items Result
SPSC (1P×1C) 100000 ✅ PASS
MPSC (4P×1C) 100000 ✅ PASS
MPMC (4P×4C) 100000 ✅ PASS
MPMC (8P×8C) 100000 ✅ PASS

ubuntu-latest, Release build.

Docs

Doxygen warnings

0 warning(s) in the files this pull request changes, which fail the check; 280 more elsewhere, which do not. The full list is in the job log and the doxygen-warnings artifact of the Docs workflow.

Gaps in files this pull request does not change, by file:

Warnings File
37 include/libe3/sm_interface.hpp
35 include/libe3/e3_connector.hpp
34 include/libe3/e3_agent.hpp
29 include/libe3/c_api.h
19 src/c_api.cpp
17 include/libe3/logger.hpp
17 swig/libe3_swig.hpp
17 swig/e3_dapp_session.hpp
15 include/libe3/subscription_manager.hpp
11 src/core/e3_agent.cpp

Ready to merge (fast-forward only)

A maintainer can land the reviewed commits with:

git fetch origin
git checkout main && git merge --ff-only a05c83f955b5ef73cd768104f2328381c0850a2e && git push origin main

Head: a05c83f955b5ef73cd768104f2328381c0850a2e (branch ci/e2e-diagnostics). If --ff-only fails as non-fast-forward, the branch must be rebased on the latest main.

One comment per PR, rewritten in place once every workflow for a05c83f finished.

The cluster stage ran the end-to-end tests and the benchmarks on a cluster after the hosted
workflows passed. It needed a polling gate between the two, a pipeline kept in another
repository, a registry for the packages image and a report format of its own, and its runs
failed for reasons that had nothing to do with the code (a moved base image, a mirror too slow
for apt, tests that failed on one node and passed on the next). CI runs on GitHub again, and
the end-to-end tests and the benchmarks stay in their hosted workflows.

This reverts, as one change, 4ea046a, 47e5b59, 3c3dd2c and 6499c9f: cluster-ci.yml,
cluster_stage.py with its tests, scripts/ci/, the report's rule that a skipped cluster stage is
a problem, and the split of docker/Dockerfile.build into a packages image and a build image.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Thecave3 <thecave003@gmail.com>
Every workflow started at once on a pull request, so a PR that failed the commit policy still
spent the end-to-end tests, and the order of the checks was nowhere written down. ci.yml now
calls the stages as workflows of their own and orders them with `needs`:

    Commit policy -> Unit Tests -> Wheels, latrec portability, MPMC benchmark (parallel)
                  -> E2E dApp Integration, E2E Topologies, Full-loop benchmark (parallel)

Wheels, latrec portability and the MPMC benchmark run only when the PR changes what they test
(the paths their own filters named: the wheels' inputs, latrec.h and latrec.c, the queue). A
stage that did not run does not hold up the next, because the stages after them are written
`!failure() && !cancelled()`; a failed or cancelled one does stop them. The build and test
stages are skipped when no changed file is code, as before, and a change to .github/workflows/
or .github/scripts/ now starts every stage, which the per-workflow filters did one file at a
time.

The stage workflows lose their pull_request trigger and path filters and gain workflow_call
(workflow_dispatch stays where they had it). commit-trailers.yml keeps its push trigger and
build-wheels.yml its use by tag-and-release.yml. .github/scripts/detect_changes.sh says which
stages a diff starts, on top of detect_code_changes.sh. CI report still waits for every workflow
that starts on a pull request, which are now CI, Docs and itself.

test_ci_gates.py checks the order (each stage waits for the ones before it, nothing runs before
the commit policy, a skippable stage does not block the next, no stage starts on its own) and,
against throwaway git repositories, which stages each kind of change starts. Breaking the order
three ways, one at a time, fails it each time. test_workflow_filters.py now holds what is left
of the old filter checks. CONTRIBUTING.md describes the order.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Thecave3 <thecave003@gmail.com>
@Thecave3 Thecave3 changed the title ci: say why a cluster e2e cell failed, and wait for the agent instead of one second ci: run the stages in order, and remove the cluster stage Nov 11, 2026
@Thecave3
Thecave3 marked this pull request as ready for review November 11, 2026 01:39
@Thecave3
Thecave3 merged commit a05c83f into main Nov 11, 2026
30 checks passed
@Thecave3
Thecave3 deleted the ci/e2e-diagnostics branch November 11, 2026 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant