Skip to content

ci: run the end-to-end tests and the benchmark on the cluster - #152

Merged
Thecave3 merged 4 commits into
mainfrom
ci/81-cluster-stage
Nov 10, 2026
Merged

Thecave3 merged 4 commits into
mainfrom
ci/81-cluster-stage

Conversation

@Thecave3

@Thecave3 Thecave3 commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add a second CI stage, the Cluster stage workflow. It waits for the other workflows for the commit to pass, builds the commit on the cluster with the images of the pull request below, runs the end-to-end tests for every link, transport and encoding (SCTP included) and the integration tests, then runs the full-loop latency benchmark for the same cells. The verdicts and the latency numbers (wire time each way and the round trip, p50 and p99) go to CI report as a fragment, so numbers of different pull requests come from the same machines.
  • CI report fails when the stage was skipped, which is what a pull request from a fork gets: a maintainer reads the commit, pushes it to a branch of this repository and runs the workflow on it. A missing secret, an unreachable cluster, a failed build, a failed end-to-end cell, a failed benchmark and a timeout each fail the stage with the reason in the fragment, so a stage that did not run never reads as passed.
  • scripts/ci/e2e.sh and scripts/ci/bench.sh run inside the build image, .github/scripts/cluster_stage.py talks to the cluster over its REST API with the Python standard library, and cluster-ci.yml joins the shared code filter and CODE_WORKFLOWS.

The existing end-to-end, topologies and benchmark workflows are kept. Moving them (the topologies one needs a script first) is the rest of #81 and is not done here.

Type of change

  • Bug fix
  • New feature / enhancement
  • New Service Model
  • Refactor (no behavior change)
  • Documentation
  • Test / CI / packaging
  • Other (explain):

Linked issue

Closes #79.
Part of #81.

Testing

  • The workflow-script tests pass (48, with the existing ones). Breaking the end-to-end verdict, the missing-secret check, the skipped-stage check and the node pinning each makes the matching test fail.
  • e2e.sh in the build image on aarch64 Linux (Docker Desktop): all 15 cells and the 11 integration tests pass. bench.sh parsing was run for one cell.
  • The workflow ran once from a throwaway branch with a push trigger and without the gate step: it passed in about 9 minutes. On the cluster, the end-to-end tests passed for all 15 cells (SCTP included) and the benchmark printed all 15 tables. The report fragment was as expected.

Not run live: the pull_request path and the gate step (this pull request is their first run), the fork path, and the failure paths other than in unit tests. Hosted runners can now open SCTP sockets (observed on ubuntu-latest, kernel 7.0.0-1019-azure), but the stage runs on the cluster on purpose, so every configuration is measured on the same nodes.

Operation

The stage needs repository secrets for the cluster token, address and namespace, and optionally a node to pin the end-to-end and benchmark pods to. The token can only create and read pipeline runs in one namespace. Anyone with write access to this repository can read those secrets by editing a workflow, so write access is the trust boundary; pull requests from forks never receive them. The cluster side is a pipeline and a task kept with the cluster's other pipelines, not in this repository.

Twin-repo coordination

  • This PR does not change the E3 wire protocol or public ABI, OR a paired PR exists in each affected twin repo (link below).
  • This PR does not change the wire format or a behavior a peer (OAI, Aerial, OCUDU) can observe, OR I added a row to the change ledger in docs/interop.md.

Paired PR(s): none needed.

@Thecave3 Thecave3 added the enhancement New feature or request label Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

CI report — 6499c9f — ✅ all checks passed

Workflow Result Time Run
Cluster stage ✅ success 8m12s #4
Commit policy ✅ success 4m46s #264
Docs ✅ success 0m09s #61
E2E dApp Integration ✅ success 2m08s #227
E2E Topologies (multi-dApp / multi-RAN) ✅ success 2m57s #225
Full-loop Latency Benchmark ✅ success 0m57s #226
Unit Tests ✅ success 7m05s #253
Build libe3py wheels ⏭️ not started (no changed file matches its paths filter) — —
latrec portability ⏭️ not started (no changed file matches its paths filter) — —
MPMC Queue Benchmark ⏭️ not started (no changed file matches its paths filter) — —
Cluster stage

Cluster stage (libe3-ci-2z95n)

End-to-end, every link, transport and encoding:

PASS zmq/ipc/asn1: 3 indications
PASS zmq/ipc/json: 3 indications
PASS zmq/ipc/protobuf: 3 indications
PASS zmq/tcp/asn1: 3 indications
PASS zmq/tcp/json: 3 indications
PASS zmq/tcp/protobuf: 3 indications
PASS posix/ipc/asn1: 3 indications
PASS posix/ipc/json: 3 indications
PASS posix/ipc/protobuf: 3 indications
PASS posix/tcp/asn1: 3 indications
PASS posix/tcp/json: 3 indications
PASS posix/tcp/protobuf: 3 indications
PASS posix/sctp/asn1: 3 indications
PASS posix/sctp/json: 3 indications
PASS posix/sctp/protobuf: 3 indications
100% tests passed, 0 tests failed out of 11

Full-loop latency in microseconds:

cell N wire RAN to dApp p50 / p99 wire dApp to RAN p50 / p99 round trip p50 / p99
zmq/ipc/asn1 1108 32.86 / 79.54 36.98 / 77.74 102.10 / 169.54
zmq/ipc/json 1073 25.36 / 36.37 24.92 / 50.43 73.98 / 151.64
zmq/ipc/protobuf 1180 29.13 / 53.39 24.86 / 36.49 77.60 / 144.40
zmq/tcp/asn1 1140 26.48 / 43.36 26.25 / 39.74 71.31 / 102.94
zmq/tcp/json 1218 27.09 / 47.17 27.48 / 52.82 79.53 / 155.46
zmq/tcp/protobuf 1143 27.83 / 46.27 27.50 / 39.74 69.63 / 103.23
posix/ipc/asn1 1052 3.37 / 10.76 6.36 / 11.78 22.56 / 52.91
posix/ipc/json 1265 6.34 / 14.38 6.52 / 10.95 42.08 / 60.54
posix/ipc/protobuf 1210 6.30 / 18.05 6.61 / 12.25 31.01 / 54.91
posix/tcp/asn1 1365 6.06 / 11.10 8.76 / 14.68 31.72 / 51.93
posix/tcp/json 1280 7.22 / 32.50 7.94 / 16.65 45.89 / 107.06
posix/tcp/protobuf 1069 7.94 / 14.15 8.75 / 13.99 39.03 / 55.41
posix/sctp/asn1 1050 5.12 / 26.21 4.77 / 18.21 39.72 / 86.90
posix/sctp/json 1076 5.04 / 10.23 5.06 / 8.15 44.47 / 57.82
posix/sctp/protobuf 1273 5.04 / 13.37 5.58 / 10.70 50.79 / 78.94

End-to-end and benchmark pods pinned to one node.

Cluster stage (libe3-ci-2z95n)

End-to-end, every link, transport and encoding:

PASS zmq/ipc/asn1: 3 indications
PASS zmq/ipc/json: 3 indications
PASS zmq/ipc/protobuf: 3 indications
PASS zmq/tcp/asn1: 3 indications
PASS zmq/tcp/json: 3 indications
PASS zmq/tcp/protobuf: 3 indications
PASS posix/ipc/asn1: 3 indications
PASS posix/ipc/json: 3 indications
PASS posix/ipc/protobuf: 3 indications
PASS posix/tcp/asn1: 3 indications
PASS posix/tcp/json: 3 indications
PASS posix/tcp/protobuf: 3 indications
PASS posix/sctp/asn1: 3 indications
PASS posix/sctp/json: 3 indications
PASS posix/sctp/protobuf: 3 indications
100% tests passed, 0 tests failed out of 11

Full-loop latency in microseconds:

cell N wire RAN to dApp p50 / p99 wire dApp to RAN p50 / p99 round trip p50 / p99
zmq/ipc/asn1 1108 32.86 / 79.54 36.98 / 77.74 102.10 / 169.54
zmq/ipc/json 1073 25.36 / 36.37 24.92 / 50.43 73.98 / 151.64
zmq/ipc/protobuf 1180 29.13 / 53.39 24.86 / 36.49 77.60 / 144.40
zmq/tcp/asn1 1140 26.48 / 43.36 26.25 / 39.74 71.31 / 102.94
zmq/tcp/json 1218 27.09 / 47.17 27.48 / 52.82 79.53 / 155.46
zmq/tcp/protobuf 1143 27.83 / 46.27 27.50 / 39.74 69.63 / 103.23
posix/ipc/asn1 1052 3.37 / 10.76 6.36 / 11.78 22.56 / 52.91
posix/ipc/json 1265 6.34 / 14.38 6.52 / 10.95 42.08 / 60.54
posix/ipc/protobuf 1210 6.30 / 18.05 6.61 / 12.25 31.01 / 54.91
posix/tcp/asn1 1365 6.06 / 11.10 8.76 / 14.68 31.72 / 51.93
posix/tcp/json 1280 7.22 / 32.50 7.94 / 16.65 45.89 / 107.06
posix/tcp/protobuf 1069 7.94 / 14.15 8.75 / 13.99 39.03 / 55.41
posix/sctp/asn1 1050 5.12 / 26.21 4.77 / 18.21 39.72 / 86.90
posix/sctp/json 1076 5.04 / 10.23 5.06 / 8.15 44.47 / 57.82
posix/sctp/protobuf 1273 5.04 / 13.37 5.58 / 10.70 50.79 / 78.94

End-to-end and benchmark pods pinned to one node.

E2E Topologies (multi-dApp / multi-RAN)

zmq/ipc

  • ✅ 1 RAN - 1 dApp: indications=5
    • dapp peer=t11 ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
  • ✅ 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
    • dapp1 peer=t12 ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=1 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp2 peer=t12 ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=1 max=1 @seq=0) hist[<=1:6 2-5:0 6-10:0 >10:0]
  • ✅ 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
    • dapp peer=t2a ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.2 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp peer=t2b ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]

zmq/tcp

  • ✅ 1 RAN - 1 dApp: indications=5
    • dapp peer=default ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.2 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
  • ✅ 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
    • dapp1 peer=default ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=1 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp2 peer=default ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=1 max=1 @seq=0) hist[<=1:6 2-5:0 6-10:0 >10:0]
  • ✅ 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
    • dapp peer=default ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=1 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp peer=off100 ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
E2E dApp Integration

✅ posix/ipc

  • dApp exit: 0
  • Indications received: 7

✅ posix/tcp

  • dApp exit: 0
  • Indications received: 7

✅ zmq/ipc

  • dApp exit: 0
  • Indications received: 7

✅ zmq/tcp

  • dApp exit: 0
  • Indications received: 7
Full-loop Latency Benchmark

Full-loop latency

Full-loop latency benchmark (N=1128 after 50 warmup)

All values in microseconds (μs). Link: zmq, transport: ipc, encoding: ASN.1 APER.

# Description Tags mean p50 p99 max
1 Collect indication data RECORD_BEGIN to ENCODE_E3SM_BEGIN 0.10 0.09 0.17 0.82
2 Create & encode indication ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE 0.95 0.91 1.43 14.47
3 Encode E3AP (indication) EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE 2.36 2.29 4.82 20.00
4 Queuing (indication) ENQUEUE to DEQUEUE 5.68 7.04 11.75 36.34
5 Delivery (indication) ENCODE_E3AP_DONE to SEND_DONE 0.27 0.22 1.59 5.47
6 E3 wire (RAN -> dApp) SEND_DONE to RECV 41.42 40.15 60.59 85.13
7 Decode E3AP (indication) RECV to DECODE_E3AP_DONE 1.51 1.51 5.66 21.58
8 libe3 dispatch (indication) DECODE_E3AP_DONE to DELIVER_BEGIN 0.08 0.08 0.14 0.30
9 Decode indication DELIVER_BEGIN to DECODE_E3SM_DONE 0.50 0.53 0.65 16.69
10 Process data DECODE_E3SM_DONE to ENCODE_E3SM_BEGIN 0.04 0.04 0.05 0.17
11 Create & encode control ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE 0.35 0.34 0.70 1.42
12 Encode E3AP (control) EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE 3.10 3.10 4.34 21.67
13 Queuing (control) ENQUEUE to DEQUEUE 11.63 11.06 24.93 27.92
14 Delivery (control) ENCODE_E3AP_DONE to SEND_DONE 5.97 6.16 8.84 21.43
15 E3 wire (dApp -> RAN) SEND_DONE to RECV 40.79 40.07 57.40 82.98
16 Decode E3AP (control) RECV to DECODE_E3AP_DONE 2.64 2.60 3.58 10.43
17 libe3 dispatch (control) DECODE_E3AP_DONE to DECODE_E3SM_BEGIN 0.52 0.49 0.68 15.55
18 Decode & handle control DECODE_E3SM_BEGIN to DECODE_E3SM_DONE 0.31 0.33 0.43 0.56
Total Total round-trip 118.74 116.74 146.82 179.21

ubuntu-latest, Release build, ZMQ + IPC, ASN.1 APER.

These numbers are measured inside a GitHub Actions container and should be treated as an upper bound on E3AP's and the library's own latency, not a representative deployment measurement.

Ready to merge (fast-forward only)

A maintainer can land the reviewed commits with:

git fetch origin
git checkout main && git merge --ff-only 6499c9f940bb1d18a71db4d214a2389c00ba70a3 && git push origin main

Head: 6499c9f940bb1d18a71db4d214a2389c00ba70a3 (branch ci/81-cluster-stage). If --ff-only fails as non-fast-forward, the branch must be rebased on the latest main.

One comment per PR, rewritten in place once every workflow for 6499c9f finished.

Base automatically changed from docker/79-images to main November 9, 2026 22:57
@Thecave3
Thecave3 marked this pull request as ready for review November 9, 2026 22:58
Thecave3 added a commit that referenced this pull request Nov 10, 2026
The test failed now and then on pull requests that did not touch it (#146, #152 and the
branch of #96). With the original test, 100 runs under load on main failed 3 and then 13
times; on the one-endpoint POSIX branch 7 of 320 failed. Two assertions failed together:
no_drops_were_recorded (one LATREC_DROP_SEND) and every_enqueued_pdu_is_either_sent_or_dropped
(one PDU more enqueued than sent or dropped, in two runs one or two more with no send drop).

The test ran the dApp and the RAN's service model until the end and then stopped the dApp
first. The service model emits every few milliseconds, so a PDU could be sent to the
connection that had just closed (a send failure, which the test counts as a defect), or be
enqueued near the shutdown of the outbound queue with neither a send nor a drop recorded.
Each combination now unsubscribes first, waits until the RAN has no subscriber left, and
pauses for what is queued to reach the dApp, before it stops anything. It also waits for the
dApp to hold its subscription after it asks (under load none was held 500 ms after the
request), and pauses before asking, as other POSIX tests do.

The test also explains a failure now: for every non-shutdown drop it prints the combination,
how long before the combination ended it happened, and each stage the PDU passed with its
aux values, and it notes an unsubscribe that did not succeed or a dApp that never held its
subscription. That is how the cause above was found.

Measured in a Linux container (aarch64), 8 runs at a time: 0 failures in 320 runs on main
with this change, against 3 to 13 in 100 without it; 0 in 320 on the one-endpoint branch
with the same test against 7 in 320 without. One thing was not found: on the three-connection
layout a dApp was sometimes seen not to learn of a subscription that the RAN held (its
unsubscribe returned SUBSCRIPTION_NOT_FOUND); the test no longer depends on it, but its cause
is open. The ZeroMQ combinations were not seen to fail.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
scripts/ci/e2e.sh runs one RAN and one dApp process for every (link,
transport) cell with every encoding, SCTP included, then the integration
tests; it prints a PASS or FAIL line per cell. scripts/ci/bench.sh runs the
full-loop latency benchmark for the same cells and encodings and prints one
table per cell. Both run inside the libe3-build image and make an SCTP
failure a failure (LIBE3_REQUIRE_SCTP=1). The image's build directory is
made writable so the scripts can run as an unprivileged user.

Run e2e.sh in the libe3-build image on aarch64 Linux (Docker Desktop), as
root: all 15 cells pass and the 11 integration tests pass. bench.sh and the
unprivileged run have not been run yet.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
When REPORT names a file, scripts/ci/e2e.sh writes the PASS and FAIL lines
and the ctest summary to it, and scripts/ci/bench.sh writes a CSV of the
18 rows and the total (mean, p50, p99, max) for each cell. bench.sh takes an
optional link and transport, so each cell group fits the 4096 bytes a Tekton
result can hold and the cluster stage can read them without access to pod
logs.

Run in the libe3-build image on aarch64 Linux (Docker Desktop): bench.sh for
posix/tcp writes 1455 bytes, 57 rows. e2e.sh was not re-run after this change.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
Add the Cluster stage workflow. It waits for the other workflows for the
commit to pass, then starts the libe3-ci pipeline on the cluster for the
branch, waits for it, and publishes the end-to-end verdicts and the latency
numbers of every link, transport and encoding as a report fragment for CI
report. The cluster is reached with a bearer token over its REST API.

CI report now fails when the cluster stage was skipped, which is what a pull
request from a fork gets: a maintainer pushes the reviewed commit to a branch
of this repository and runs the workflow on it. A missing secret, an
unreachable cluster, a failed build, a failed end-to-end cell, a failed
benchmark and a timeout each fail the stage with their reason in the fragment.

The new workflow joins the shared code filter and CODE_WORKFLOWS. The unit
tests of cluster_stage.py and of the CI report problem detection pass (47
tests with the existing ones); breaking the e2e verdict, the missing-secret
check and the skipped check each makes one of them fail. The workflow has not
run on GitHub yet, and the existing e2e and benchmark workflows are kept.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
@Thecave3
Thecave3 force-pushed the ci/81-cluster-stage branch from eb5fd9a to 3c3dd2c Compare November 10, 2026 01:40
The cluster builds took their packages from archive.ubuntu.com on every run, and two runs
failed when that mirror was unreachable. docker/Dockerfile.deps now holds the packages
(build_libe3 -I) and docker/Dockerfile.build starts from it. The cluster stage computes a
tag from a hash of build_libe3 and Dockerfile.deps and passes it as DEPS_TAG; the pipeline
builds the packages image only when that tag is not in the registry, so a run that changes
only libe3 sources downloads no packages.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Thecave3 <thecave003@gmail.com>
@Thecave3
Thecave3 merged commit 6499c9f into main Nov 10, 2026
23 of 25 checks passed
@Thecave3
Thecave3 deleted the ci/81-cluster-stage branch November 10, 2026 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Add dockerfile folder and let the workflows use it

1 participant