Repository navigation
ci: run the end-to-end tests and the benchmark on the cluster - #152
Conversation
CI report —
|
| Workflow | Result | Time | Run |
|---|---|---|---|
| Cluster stage | ✅ success | 8m12s | #4 |
| Commit policy | ✅ success | 4m46s | #264 |
| Docs | ✅ success | 0m09s | #61 |
| E2E dApp Integration | ✅ success | 2m08s | #227 |
| E2E Topologies (multi-dApp / multi-RAN) | ✅ success | 2m57s | #225 |
| Full-loop Latency Benchmark | ✅ success | 0m57s | #226 |
| Unit Tests | ✅ success | 7m05s | #253 |
| Build libe3py wheels | ⏭️ not started (no changed file matches its paths filter) |
— | — |
| latrec portability | ⏭️ not started (no changed file matches its paths filter) |
— | — |
| MPMC Queue Benchmark | ⏭️ not started (no changed file matches its paths filter) |
— | — |
Cluster stage
Cluster stage (libe3-ci-2z95n)
End-to-end, every link, transport and encoding:
PASS zmq/ipc/asn1: 3 indications
PASS zmq/ipc/json: 3 indications
PASS zmq/ipc/protobuf: 3 indications
PASS zmq/tcp/asn1: 3 indications
PASS zmq/tcp/json: 3 indications
PASS zmq/tcp/protobuf: 3 indications
PASS posix/ipc/asn1: 3 indications
PASS posix/ipc/json: 3 indications
PASS posix/ipc/protobuf: 3 indications
PASS posix/tcp/asn1: 3 indications
PASS posix/tcp/json: 3 indications
PASS posix/tcp/protobuf: 3 indications
PASS posix/sctp/asn1: 3 indications
PASS posix/sctp/json: 3 indications
PASS posix/sctp/protobuf: 3 indications
100% tests passed, 0 tests failed out of 11
Full-loop latency in microseconds:
| cell | N | wire RAN to dApp p50 / p99 | wire dApp to RAN p50 / p99 | round trip p50 / p99 |
|---|---|---|---|---|
| zmq/ipc/asn1 | 1108 | 32.86 / 79.54 | 36.98 / 77.74 | 102.10 / 169.54 |
| zmq/ipc/json | 1073 | 25.36 / 36.37 | 24.92 / 50.43 | 73.98 / 151.64 |
| zmq/ipc/protobuf | 1180 | 29.13 / 53.39 | 24.86 / 36.49 | 77.60 / 144.40 |
| zmq/tcp/asn1 | 1140 | 26.48 / 43.36 | 26.25 / 39.74 | 71.31 / 102.94 |
| zmq/tcp/json | 1218 | 27.09 / 47.17 | 27.48 / 52.82 | 79.53 / 155.46 |
| zmq/tcp/protobuf | 1143 | 27.83 / 46.27 | 27.50 / 39.74 | 69.63 / 103.23 |
| posix/ipc/asn1 | 1052 | 3.37 / 10.76 | 6.36 / 11.78 | 22.56 / 52.91 |
| posix/ipc/json | 1265 | 6.34 / 14.38 | 6.52 / 10.95 | 42.08 / 60.54 |
| posix/ipc/protobuf | 1210 | 6.30 / 18.05 | 6.61 / 12.25 | 31.01 / 54.91 |
| posix/tcp/asn1 | 1365 | 6.06 / 11.10 | 8.76 / 14.68 | 31.72 / 51.93 |
| posix/tcp/json | 1280 | 7.22 / 32.50 | 7.94 / 16.65 | 45.89 / 107.06 |
| posix/tcp/protobuf | 1069 | 7.94 / 14.15 | 8.75 / 13.99 | 39.03 / 55.41 |
| posix/sctp/asn1 | 1050 | 5.12 / 26.21 | 4.77 / 18.21 | 39.72 / 86.90 |
| posix/sctp/json | 1076 | 5.04 / 10.23 | 5.06 / 8.15 | 44.47 / 57.82 |
| posix/sctp/protobuf | 1273 | 5.04 / 13.37 | 5.58 / 10.70 | 50.79 / 78.94 |
End-to-end and benchmark pods pinned to one node.
Cluster stage (libe3-ci-2z95n)
End-to-end, every link, transport and encoding:
PASS zmq/ipc/asn1: 3 indications
PASS zmq/ipc/json: 3 indications
PASS zmq/ipc/protobuf: 3 indications
PASS zmq/tcp/asn1: 3 indications
PASS zmq/tcp/json: 3 indications
PASS zmq/tcp/protobuf: 3 indications
PASS posix/ipc/asn1: 3 indications
PASS posix/ipc/json: 3 indications
PASS posix/ipc/protobuf: 3 indications
PASS posix/tcp/asn1: 3 indications
PASS posix/tcp/json: 3 indications
PASS posix/tcp/protobuf: 3 indications
PASS posix/sctp/asn1: 3 indications
PASS posix/sctp/json: 3 indications
PASS posix/sctp/protobuf: 3 indications
100% tests passed, 0 tests failed out of 11
Full-loop latency in microseconds:
| cell | N | wire RAN to dApp p50 / p99 | wire dApp to RAN p50 / p99 | round trip p50 / p99 |
|---|---|---|---|---|
| zmq/ipc/asn1 | 1108 | 32.86 / 79.54 | 36.98 / 77.74 | 102.10 / 169.54 |
| zmq/ipc/json | 1073 | 25.36 / 36.37 | 24.92 / 50.43 | 73.98 / 151.64 |
| zmq/ipc/protobuf | 1180 | 29.13 / 53.39 | 24.86 / 36.49 | 77.60 / 144.40 |
| zmq/tcp/asn1 | 1140 | 26.48 / 43.36 | 26.25 / 39.74 | 71.31 / 102.94 |
| zmq/tcp/json | 1218 | 27.09 / 47.17 | 27.48 / 52.82 | 79.53 / 155.46 |
| zmq/tcp/protobuf | 1143 | 27.83 / 46.27 | 27.50 / 39.74 | 69.63 / 103.23 |
| posix/ipc/asn1 | 1052 | 3.37 / 10.76 | 6.36 / 11.78 | 22.56 / 52.91 |
| posix/ipc/json | 1265 | 6.34 / 14.38 | 6.52 / 10.95 | 42.08 / 60.54 |
| posix/ipc/protobuf | 1210 | 6.30 / 18.05 | 6.61 / 12.25 | 31.01 / 54.91 |
| posix/tcp/asn1 | 1365 | 6.06 / 11.10 | 8.76 / 14.68 | 31.72 / 51.93 |
| posix/tcp/json | 1280 | 7.22 / 32.50 | 7.94 / 16.65 | 45.89 / 107.06 |
| posix/tcp/protobuf | 1069 | 7.94 / 14.15 | 8.75 / 13.99 | 39.03 / 55.41 |
| posix/sctp/asn1 | 1050 | 5.12 / 26.21 | 4.77 / 18.21 | 39.72 / 86.90 |
| posix/sctp/json | 1076 | 5.04 / 10.23 | 5.06 / 8.15 | 44.47 / 57.82 |
| posix/sctp/protobuf | 1273 | 5.04 / 13.37 | 5.58 / 10.70 | 50.79 / 78.94 |
End-to-end and benchmark pods pinned to one node.
E2E Topologies (multi-dApp / multi-RAN)
zmq/ipc
- ✅ 1 RAN - 1 dApp: indications=
5dapppeer=t11 ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
- ✅ 1 RAN - 2 dApps: dApp#1 ind=
5sub=1, dApp#2 ind=6sub=2, RAN saw2dApps - ✅ 2 RANs - 1 dApp: from ran-a ind=
5, from ran-b ind=5
zmq/tcp
- ✅ 1 RAN - 1 dApp: indications=
5dapppeer=default ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.2 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
- ✅ 1 RAN - 2 dApps: dApp#1 ind=
5sub=1, dApp#2 ind=6sub=2, RAN saw2dApps - ✅ 2 RANs - 1 dApp: from ran-a ind=
5, from ran-b ind=5
E2E dApp Integration
✅ posix/ipc
- dApp exit:
0 - Indications received:
7
✅ posix/tcp
- dApp exit:
0 - Indications received:
7
✅ zmq/ipc
- dApp exit:
0 - Indications received:
7
✅ zmq/tcp
- dApp exit:
0 - Indications received:
7
Full-loop Latency Benchmark
Full-loop latency
Full-loop latency benchmark (N=1128 after 50 warmup)
All values in microseconds (μs). Link: zmq, transport: ipc, encoding: ASN.1 APER.
| # | Description | Tags | mean | p50 | p99 | max |
|---|---|---|---|---|---|---|
| 1 | Collect indication data | RECORD_BEGIN to ENCODE_E3SM_BEGIN |
0.10 | 0.09 | 0.17 | 0.82 |
| 2 | Create & encode indication | ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE |
0.95 | 0.91 | 1.43 | 14.47 |
| 3 | Encode E3AP (indication) | EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE |
2.36 | 2.29 | 4.82 | 20.00 |
| 4 | Queuing (indication) | ENQUEUE to DEQUEUE |
5.68 | 7.04 | 11.75 | 36.34 |
| 5 | Delivery (indication) | ENCODE_E3AP_DONE to SEND_DONE |
0.27 | 0.22 | 1.59 | 5.47 |
| 6 | E3 wire (RAN -> dApp) | SEND_DONE to RECV |
41.42 | 40.15 | 60.59 | 85.13 |
| 7 | Decode E3AP (indication) | RECV to DECODE_E3AP_DONE |
1.51 | 1.51 | 5.66 | 21.58 |
| 8 | libe3 dispatch (indication) | DECODE_E3AP_DONE to DELIVER_BEGIN |
0.08 | 0.08 | 0.14 | 0.30 |
| 9 | Decode indication | DELIVER_BEGIN to DECODE_E3SM_DONE |
0.50 | 0.53 | 0.65 | 16.69 |
| 10 | Process data | DECODE_E3SM_DONE to ENCODE_E3SM_BEGIN |
0.04 | 0.04 | 0.05 | 0.17 |
| 11 | Create & encode control | ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE |
0.35 | 0.34 | 0.70 | 1.42 |
| 12 | Encode E3AP (control) | EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE |
3.10 | 3.10 | 4.34 | 21.67 |
| 13 | Queuing (control) | ENQUEUE to DEQUEUE |
11.63 | 11.06 | 24.93 | 27.92 |
| 14 | Delivery (control) | ENCODE_E3AP_DONE to SEND_DONE |
5.97 | 6.16 | 8.84 | 21.43 |
| 15 | E3 wire (dApp -> RAN) | SEND_DONE to RECV |
40.79 | 40.07 | 57.40 | 82.98 |
| 16 | Decode E3AP (control) | RECV to DECODE_E3AP_DONE |
2.64 | 2.60 | 3.58 | 10.43 |
| 17 | libe3 dispatch (control) | DECODE_E3AP_DONE to DECODE_E3SM_BEGIN |
0.52 | 0.49 | 0.68 | 15.55 |
| 18 | Decode & handle control | DECODE_E3SM_BEGIN to DECODE_E3SM_DONE |
0.31 | 0.33 | 0.43 | 0.56 |
| Total | Total round-trip | 118.74 | 116.74 | 146.82 | 179.21 |
ubuntu-latest, Release build, ZMQ + IPC, ASN.1 APER.
These numbers are measured inside a GitHub Actions container and should be treated as an upper bound on E3AP's and the library's own latency, not a representative deployment measurement.
Ready to merge (fast-forward only)
A maintainer can land the reviewed commits with:
git fetch origin
git checkout main && git merge --ff-only 6499c9f940bb1d18a71db4d214a2389c00ba70a3 && git push origin mainHead: 6499c9f940bb1d18a71db4d214a2389c00ba70a3 (branch ci/81-cluster-stage). If --ff-only fails as non-fast-forward, the branch must be rebased on the latest main.
One comment per PR, rewritten in place once every workflow for
6499c9ffinished.
The test failed now and then on pull requests that did not touch it (#146, #152 and the branch of #96). With the original test, 100 runs under load on main failed 3 and then 13 times; on the one-endpoint POSIX branch 7 of 320 failed. Two assertions failed together: no_drops_were_recorded (one LATREC_DROP_SEND) and every_enqueued_pdu_is_either_sent_or_dropped (one PDU more enqueued than sent or dropped, in two runs one or two more with no send drop). The test ran the dApp and the RAN's service model until the end and then stopped the dApp first. The service model emits every few milliseconds, so a PDU could be sent to the connection that had just closed (a send failure, which the test counts as a defect), or be enqueued near the shutdown of the outbound queue with neither a send nor a drop recorded. Each combination now unsubscribes first, waits until the RAN has no subscriber left, and pauses for what is queued to reach the dApp, before it stops anything. It also waits for the dApp to hold its subscription after it asks (under load none was held 500 ms after the request), and pauses before asking, as other POSIX tests do. The test also explains a failure now: for every non-shutdown drop it prints the combination, how long before the combination ended it happened, and each stage the PDU passed with its aux values, and it notes an unsubscribe that did not succeed or a dApp that never held its subscription. That is how the cause above was found. Measured in a Linux container (aarch64), 8 runs at a time: 0 failures in 320 runs on main with this change, against 3 to 13 in 100 without it; 0 in 320 on the one-endpoint branch with the same test against 7 in 320 without. One thing was not found: on the three-connection layout a dApp was sometimes seen not to learn of a subscription that the RAN held (its unsubscribe returned SUBSCRIPTION_NOT_FOUND); the test no longer depends on it, but its cause is open. The ZeroMQ combinations were not seen to fail. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
scripts/ci/e2e.sh runs one RAN and one dApp process for every (link, transport) cell with every encoding, SCTP included, then the integration tests; it prints a PASS or FAIL line per cell. scripts/ci/bench.sh runs the full-loop latency benchmark for the same cells and encodings and prints one table per cell. Both run inside the libe3-build image and make an SCTP failure a failure (LIBE3_REQUIRE_SCTP=1). The image's build directory is made writable so the scripts can run as an unprivileged user. Run e2e.sh in the libe3-build image on aarch64 Linux (Docker Desktop), as root: all 15 cells pass and the 11 integration tests pass. bench.sh and the unprivileged run have not been run yet. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
When REPORT names a file, scripts/ci/e2e.sh writes the PASS and FAIL lines and the ctest summary to it, and scripts/ci/bench.sh writes a CSV of the 18 rows and the total (mean, p50, p99, max) for each cell. bench.sh takes an optional link and transport, so each cell group fits the 4096 bytes a Tekton result can hold and the cluster stage can read them without access to pod logs. Run in the libe3-build image on aarch64 Linux (Docker Desktop): bench.sh for posix/tcp writes 1455 bytes, 57 rows. e2e.sh was not re-run after this change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
Add the Cluster stage workflow. It waits for the other workflows for the commit to pass, then starts the libe3-ci pipeline on the cluster for the branch, waits for it, and publishes the end-to-end verdicts and the latency numbers of every link, transport and encoding as a report fragment for CI report. The cluster is reached with a bearer token over its REST API. CI report now fails when the cluster stage was skipped, which is what a pull request from a fork gets: a maintainer pushes the reviewed commit to a branch of this repository and runs the workflow on it. A missing secret, an unreachable cluster, a failed build, a failed end-to-end cell, a failed benchmark and a timeout each fail the stage with their reason in the fragment. The new workflow joins the shared code filter and CODE_WORKFLOWS. The unit tests of cluster_stage.py and of the CI report problem detection pass (47 tests with the existing ones); breaking the e2e verdict, the missing-secret check and the skipped check each makes one of them fail. The workflow has not run on GitHub yet, and the existing e2e and benchmark workflows are kept. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
eb5fd9a to
3c3dd2c
Compare
The cluster builds took their packages from archive.ubuntu.com on every run, and two runs failed when that mirror was unreachable. docker/Dockerfile.deps now holds the packages (build_libe3 -I) and docker/Dockerfile.build starts from it. The cluster stage computes a tag from a hash of build_libe3 and Dockerfile.deps and passes it as DEPS_TAG; the pipeline builds the packages image only when that tag is not in the registry, so a run that changes only libe3 sources downloads no packages. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Thecave3 <thecave003@gmail.com>
Summary
CI reportas a fragment, so numbers of different pull requests come from the same machines.CI reportfails when the stage was skipped, which is what a pull request from a fork gets: a maintainer reads the commit, pushes it to a branch of this repository and runs the workflow on it. A missing secret, an unreachable cluster, a failed build, a failed end-to-end cell, a failed benchmark and a timeout each fail the stage with the reason in the fragment, so a stage that did not run never reads as passed.scripts/ci/e2e.shandscripts/ci/bench.shrun inside the build image,.github/scripts/cluster_stage.pytalks to the cluster over its REST API with the Python standard library, andcluster-ci.ymljoins the shared code filter andCODE_WORKFLOWS.The existing end-to-end, topologies and benchmark workflows are kept. Moving them (the topologies one needs a script first) is the rest of #81 and is not done here.
Type of change
Linked issue
Closes #79.
Part of #81.
Testing
e2e.shin the build image on aarch64 Linux (Docker Desktop): all 15 cells and the 11 integration tests pass.bench.shparsing was run for one cell.pushtrigger and without the gate step: it passed in about 9 minutes. On the cluster, the end-to-end tests passed for all 15 cells (SCTP included) and the benchmark printed all 15 tables. The report fragment was as expected.Not run live: the
pull_requestpath and the gate step (this pull request is their first run), the fork path, and the failure paths other than in unit tests. Hosted runners can now open SCTP sockets (observed onubuntu-latest, kernel 7.0.0-1019-azure), but the stage runs on the cluster on purpose, so every configuration is measured on the same nodes.Operation
The stage needs repository secrets for the cluster token, address and namespace, and optionally a node to pin the end-to-end and benchmark pods to. The token can only create and read pipeline runs in one namespace. Anyone with write access to this repository can read those secrets by editing a workflow, so write access is the trust boundary; pull requests from forks never receive them. The cluster side is a pipeline and a task kept with the cluster's other pipelines, not in this repository.
Twin-repo coordination
docs/interop.md.Paired PR(s): none needed.