Repository navigation
Conversation
… posix_connector
The endpoint parsing, the creation and tuning of a socket, listen, connect and admit,
the TCP length prefix, the SCTP send info and the test for an SCTP notification now live
in socket_link.{hpp,cpp}, so a second link can put the same bytes on the same sockets.
posix_connector.cpp keeps what is its own: the threads, the queues, the per-peer state.
Nothing a peer can observe changes and no POSIX test changed. test_socket_link pins the
framing the two links must share: a whole PDU, a partial one, one of the limit and one a
byte over it, the prefix written, and the Invalid-PDU-length message the POSIX link logs.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
Linux gives an SCTP association a minimum retransmission timeout of one second. On loopback a burst of large PDUs can lose one packet to a full receive window, the association then waits that second, and the send stalls past the 500 ms send timeout: the RAN drops a dApp that is alive. A burst of 2000 PDUs of up to 64 KiB from the RAN, repeated 100 times, dropped the dApp 8 times with the POSIX link and 7 times with the io_uring link of the next commits, and never over TCP; with SCTP_RTOINFO set on the socket (initial 120 ms, minimum 20 ms, maximum 250 ms, the maximum kept under the send timeout) it happened 0 times in 100 with each link. This changes the timers of #153's POSIX SCTP sockets too, for the better as far as the runs show, but it is a choice about a transport's behaviour, which is why it is a commit of its own: dropping it removes the bound and nothing else. test_socket_link reads the values back from the socket and fails without it. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
A third link beside ZeroMQ and POSIX: the POSIX link's endpoint, sockets and bytes (IPC,
TCP and SCTP, one endpoint per dApp, the framing of the one-endpoint wire), driven
through Linux io_uring. A RAN on one of the two socket links serves a dApp on the other.
For a user the facade gains one enumerator: URING = 2, the string "uring", C API range
0..2, E3LinkLayer_URING in Python. No E3Config field and no E3Connector method is added
and PosixOptions apply as they are. The ring is tuned by LIBE3_URING_* environment
variables read when the connector is created (depth, registered files, registered buffers,
a kernel submission thread, busy polling of the completion queue): deployment tuning, not
ABI.
The calling threads drive two rings, one for receives and one for sends, with no I/O thread
of the connector's own, so send() returns when the kernel has the bytes, as it does with
POSIX, and the SEND_DONE stamp compares row by row. A broadcast to several dApps is one
batch of sends, so a dApp that stopped reading no longer delays the others for the send
timeout. uring_ring.{hpp,cpp} is the only file that includes <liburing.h>.
The link is built when it can be: LIBE3_ENABLE_URING (default ON) needs Linux and liburing,
and LIBE3_HAS_URING is 0 or 1 for what links libe3. Without liburing, or where the kernel
refuses io_uring (Docker's default seccomp profile, kernel.io_uring_disabled, a kernel
before 5.11), create_connector returns null and says why, as it does for ZeroMQ without
libzmq: there is no silent fallback. build_libe3 learns --enable-uring and --disable-uring
and installs liburing-dev with its other dependencies.
The decisions, what was left out and why (per-thread rings with SINGLE_ISSUER and
DEFER_TASKRUN, a thread of its own for I/O, a direct-send bypass, multishot receive) are in
docs/connectors/uring.md.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
…_uring link, compare the links The tests of the one-endpoint wire take --ran-link and --dapp-link (tests/link_under_test.hpp) and run once per pair in CTest: both POSIX, both io_uring and each mixed pair. Where the kernel refuses io_uring they exit 77, which CTest reports as skipped, unless LIBE3_REQUIRE_URING is set, so that a machine meant to run io_uring cannot pass by skipping. test_uring_connector covers the ring options, a broadcast to more peers than the ring has entries, a stalled peer, a reset peer, 2000 PDUs of every size on every transport, the PDU limit, the receive timeout and the wake of a waiting receive by shutdown. The ports of these tests now come from free_ports(): below the range of source ports, free when asked, and claimed with a lock on a file in /tmp/libe3-test-ports that the process holds until it exits, so that two tests never take the same port however they order asking and listening. The tests chose their ports from the range the kernel also uses for the source ports of outgoing connections, and each from a formula on its pid with a base and a width of its own, so the bands overlapped. Once enough connections had come and gone the listen of a later test met a source port still in TIME_WAIT (2 runs in 30 of test_uring_connector), and two tests with overlapping bands met each other (8 runs in 12 of the connector subset at -j3, always test_role_pair_posix or a variant). 0 in 80 and 0 in 12 after. The link-matrix entries run with LATREC_ENTRIES_LOG2=12: every agent thread leaves an 8 MiB pre-faulted latrec ring in <build>/latrec, and repeating the matrix filled a disk. test_bench_full_loop_latency takes --link uring, --iterations, --warmup and CPU pinning, and reports the CPU and context switches per round trip. tools/bench_matrix.py runs it over links, transports and encodings in interleaved rounds and names the fastest link per transport only when its slowest round beats every other link's best; its parsing and verdict are tested on canned output (test_bench_matrix.py). The benchmark pairs the dApp's inbound records with the RAN's outbound ones by position and checked that each pair was in order; a receiver that polls can have a PDU before the sender has stamped its send as done (SCTP over loopback delivers inside the send call), so about one run in ten of SCTP with busy polling was refused as misaligned although the counts of PDUs out and in agreed. The check now allows 5 us of skew (a pair that is off by one round trip is off by 12 us or more), the phase arithmetic is signed, and a refusal says which pair was first and how many groups there were. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
…he others The E2E dApp job runs the links [zmq, posix, uring] over [ipc, tcp, sctp] (ZeroMQ has no SCTP) with SCTP built in; the unit and integration jobs set LIBE3_REQUIRE_URING=1 and print the kernel's io_uring settings first, so a runner that refuses io_uring fails instead of skipping; one job builds without liburing and checks that the link is reported as not built. The full-loop job runs tools/bench_matrix.py over ZeroMQ, POSIX, io_uring, and io_uring with busy polling with and without the kernel submission thread, on IPC, TCP and SCTP, three rounds, and appends the table to the CI report. The images carry liburing2, and run-pair.sh starts a container with seccomp unconfined when the link is uring (Docker's default profile refuses io_uring_setup). example_simple_agent and example_simple_dapp accept --link uring. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
docs/connectors/uring.md says what a user can do with the link, which link to pick, the architecture and the eleven decisions with what the others would have made of it, where it runs, the ring options, notes for a real-time deployment, the tests, the measurements on an x86-64 server and on an Arm virtual machine (every number is in the page and in the tools' CSV output, nothing is rounded to a story) and what was deferred with its cost. The README, CLAUDE.md, the path-A control-loop page (the wire hop can read below zero), the integrator guide (the link and its framing code pointers, the SCTP timer bound), the encoding chapter of the specification (one sentence: how a link moves the bytes is not part of E3AP; the Word edition is rebuilt), interop.md (rows of the pairs table, a mixed-links note, a ledger row for the new link and for the SCTP timer bound), and the SWIG README are updated. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
CI report —
|
| Workflow | Result | Time | Run |
|---|---|---|---|
| CI | ✅ success | 32m55s | #20 |
| Docs | ✅ success | 1m03s | #94 |
CI
✅ posix/ipc
- dApp exit:
0 - Indications received:
7
✅ posix/sctp
- dApp exit:
0 - Indications received:
7
✅ posix/tcp
- dApp exit:
0 - Indications received:
7
✅ uring/ipc
- dApp exit:
0 - Indications received:
7
✅ uring/sctp
- dApp exit:
0 - Indications received:
7
✅ uring/tcp
- dApp exit:
0 - Indications received:
7
✅ zmq/ipc
- dApp exit:
0 - Indications received:
7
✅ zmq/tcp
- dApp exit:
0 - Indications received:
7
zmq/ipc
- ✅ 1 RAN - 1 dApp: indications=
5dapppeer=t11 ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
- ✅ 1 RAN - 2 dApps: dApp#1 ind=
5sub=1, dApp#2 ind=6sub=2, RAN saw2dApps - ✅ 2 RANs - 1 dApp: from ran-a ind=
5, from ran-b ind=5
zmq/tcp
- ✅ 1 RAN - 1 dApp: indications=
5dapppeer=default ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.8 max=1 @seq=1) hist[<=1:5 2-5:0 6-10:0 >10:0]
- ✅ 1 RAN - 2 dApps: dApp#1 ind=
5sub=1, dApp#2 ind=6sub=2, RAN saw2dApps - ✅ 2 RANs - 1 dApp: from ran-a ind=
5, from ran-b ind=5
Full-loop latency
Full-loop latency benchmark (N=1040 after 50 warmup)
All values in microseconds (μs). Link: zmq, transport: ipc, encoding: ASN.1 APER.
| # | Description | Tags | mean | p50 | p99 | max |
|---|---|---|---|---|---|---|
| 1 | Collect indication data | RECORD_BEGIN to ENCODE_E3SM_BEGIN |
0.10 | 0.09 | 0.24 | 0.52 |
| 2 | Create & encode indication | ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE |
0.86 | 0.83 | 1.43 | 9.94 |
| 3 | Encode E3AP (indication) | EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE |
3.24 | 3.04 | 5.68 | 28.25 |
| 4 | Queuing (indication) | ENQUEUE to DEQUEUE |
12.33 | 11.99 | 25.08 | 34.26 |
| 5 | Delivery (indication) | ENCODE_E3AP_DONE to SEND_DONE |
6.55 | 7.14 | 10.17 | 24.80 |
| 6 | E3 wire (RAN -> dApp) | SEND_DONE to RECV |
41.00 | 38.01 | 60.77 | 833.49 |
| 7 | Decode E3AP (indication) | RECV to DECODE_E3AP_DONE |
2.69 | 2.45 | 15.41 | 26.94 |
| 8 | libe3 dispatch (indication) | DECODE_E3AP_DONE to DELIVER_BEGIN |
0.16 | 0.14 | 0.24 | 16.62 |
| 9 | Decode indication | DELIVER_BEGIN to DECODE_E3SM_DONE |
0.52 | 0.55 | 0.80 | 1.22 |
| 10 | Process data | DECODE_E3SM_DONE to ENCODE_E3SM_BEGIN |
0.04 | 0.04 | 0.06 | 0.31 |
| 11 | Create & encode control | ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE |
0.32 | 0.32 | 0.55 | 0.71 |
| 12 | Encode E3AP (control) | EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE |
3.42 | 3.29 | 5.55 | 23.16 |
| 13 | Queuing (control) | ENQUEUE to DEQUEUE |
11.96 | 11.10 | 25.31 | 36.16 |
| 14 | Delivery (control) | ENCODE_E3AP_DONE to SEND_DONE |
6.06 | 4.84 | 22.29 | 36.22 |
| 15 | E3 wire (dApp -> RAN) | SEND_DONE to RECV |
40.48 | 40.27 | 61.27 | 214.43 |
| 16 | Decode E3AP (control) | RECV to DECODE_E3AP_DONE |
2.37 | 2.22 | 4.67 | 19.18 |
| 17 | libe3 dispatch (control) | DECODE_E3AP_DONE to DECODE_E3SM_BEGIN |
0.59 | 0.58 | 1.02 | 1.26 |
| 18 | Decode & handle control | DECODE_E3SM_BEGIN to DECODE_E3SM_DONE |
0.24 | 0.20 | 0.59 | 0.76 |
| Total | Total round-trip | 133.35 | 127.31 | 179.87 | 947.89 |
CPU per round trip, both ends: 81.21 us user, 225.87 us system; context switches per round trip: 9.03 voluntary, 0.08 involuntary.
ubuntu-latest, Release build, ZMQ + IPC, ASN.1 APER.
Connector comparison: full-loop round trip
Same machine, 3 interleaved round(s) per cell, 1000 round trips per run after 50 of warm-up. Each value is the median over the rounds of that round's statistic, with the range of the rounds in brackets. Microseconds. CPU is user plus system time of the process (both ends) per round trip.
| Transport | Encoding | Link | Round trip p50 | Round trip p99 | Wire RAN to dApp p50 | Wire dApp to RAN p50 | Send indication p50 | Send control p50 | CPU per round trip | Runs |
|---|---|---|---|---|---|---|---|---|---|---|
| ipc | asn1 | zmq | 127.7 [125.6 to 128.4] | 160.5 [156.7 to 162.4] | 39.4 [38.9 to 39.6] | 39.2 [39.0 to 39.7] | 4.8 [4.8 to 5.0] | 4.9 [4.8 to 4.9] | 303.4 [301.9 to 303.8] | 3 |
| ipc | asn1 | posix | 48.3 [44.8 to 49.1] | 69.6 [68.1 to 72.5] | 10.9 [10.3 to 11.3] | 11.0 [9.7 to 11.3] | 7.5 [6.7 to 7.6] | 5.2 [4.1 to 7.1] | 162.0 [155.4 to 163.3] | 3 |
| ipc | asn1 | uring | 47.0 [39.5 to 47.2] | 66.8 [60.5 to 68.6] | 8.7 [7.7 to 9.2] | 7.6 [7.5 to 8.5] | 7.7 [5.8 to 9.0] | 8.5 [5.7 to 9.4] | 156.6 [146.1 to 158.2] | 3 |
| ipc | asn1 | uring-busy | 29.3 [27.6 to 30.7] | 42.9 [41.4 to 45.4] | 3.0 [2.8 to 3.2] | 3.2 [3.0 to 3.5] | 5.2 [5.1 to 5.8] | 5.0 [5.0 to 5.5] | 148.2 [142.6 to 151.7] | 3 |
| ipc | asn1 | uring-sqpoll-busy | 942.3 [892.3 to 1008.2] | 1980.9 [1970.6 to 2988.1] | 6.2 [6.2 to 7.4] | 6.8 [6.6 to 7.4] | 3.4 [3.2 to 3.5] | 3.2 [3.1 to 3.3] | 4230.4 [4191.1 to 4364.3] | 3 |
| tcp | asn1 | zmq | 159.0 [158.4 to 159.4] | 197.3 [196.8 to 197.9] | 52.2 [51.9 to 53.6] | 52.6 [52.0 to 53.0] | 5.2 [5.1 to 5.7] | 5.2 [5.1 to 5.4] | 332.9 [332.9 to 333.3] | 3 |
| tcp | asn1 | posix | 45.0 [44.6 to 64.0] | 68.6 [67.1 to 88.6] | 6.5 [6.4 to 6.9] | 6.5 [6.4 to 8.3] | 10.7 [10.7 to 19.5] | 10.6 [10.5 to 13.9] | 153.1 [152.9 to 185.3] | 3 |
| tcp | asn1 | uring | 64.1 [60.4 to 64.6] | 90.2 [87.7 to 91.4] | 8.6 [7.5 to 8.8] | 8.1 [7.0 to 8.9] | 16.4 [14.5 to 17.6] | 17.2 [12.0 to 19.3] | 198.9 [197.1 to 202.5] | 3 |
| tcp | asn1 | uring-busy | 39.8 [37.4 to 40.6] | 56.5 [52.1 to 58.4] | 1.5 [1.5 to 1.6] | 1.8 [1.7 to 1.8] | 12.1 [11.5 to 12.9] | 11.0 [11.0 to 11.2] | 192.1 [182.5 to 202.9] | 3 |
| tcp | asn1 | uring-sqpoll-busy | 1027.2 [1007.7 to 1033.8] | 1992.4 [1465.0 to 2002.5] | 5.8 [5.6 to 5.9] | 5.8 [5.6 to 6.0] | 12.6 [9.2 to 13.3] | 12.6 [9.3 to 13.1] | 4245.5 [4244.0 to 4418.8] | 3 |
| sctp | asn1 | posix | 69.1 [68.6 to 69.8] | 93.6 [92.0 to 97.4] | 3.1 [-0.6 to 3.2] | -0.8 [-1.0 to 3.5] | 21.9 [21.7 to 27.6] | 29.6 [22.8 to 29.8] | 213.1 [210.4 to 217.3] | 3 |
| sctp | asn1 | uring | 72.7 [69.1 to 73.8] | 97.9 [96.7 to 99.4] | -0.1 [-0.7 to 3.1] | -0.3 [-1.3 to 3.4] | 28.9 [23.5 to 29.4] | 31.1 [21.1 to 31.7] | 225.0 [222.7 to 231.1] | 3 |
| sctp | asn1 | uring-busy | 56.4 [56.0 to 58.4] | 78.2 [75.1 to 78.3] | -4.0 [-4.1 to -3.9] | -3.5 [-3.7 to -3.0] | 23.7 [23.4 to 25.3] | 25.9 [22.9 to 26.5] | 268.3 [266.3 to 276.8] | 3 |
| sctp | asn1 | uring-sqpoll-busy | 1044.5 [995.9 to 1044.7] | 1929.8 [1745.7 to 3017.1] | 4.9 [4.9 to 5.0] | 5.3 [5.2 to 5.3] | 23.5 [23.0 to 23.6] | 22.7 [22.3 to 22.9] | 4325.7 [4322.6 to 4445.4] | 3 |
Fastest link per transport
- ipc/asn1: fastest is uring-busy (29.3 us), then uring (47.0 us, 38% slower). Against posix (48.3 us): 39% less.
- tcp/asn1: fastest is uring-busy (39.8 us), then posix (45.0 us, 12% slower). Against posix (45.0 us): 12% less.
- sctp/asn1: fastest is uring-busy (56.4 us), then posix (69.1 us, 18% slower). Against posix (69.1 us): 18% less.
These numbers are measured inside a GitHub Actions container and should be treated as an upper bound on E3AP's and the library's own latency, not a representative deployment measurement.
MPMC queue benchmark
MpmcQueue End-to-End Latency (SPSC, 30000 items per queue size)
| Queue Capacity | P50 (ns) | P95 (ns) | P99 (ns) | P99.9 (ns) |
|---|---|---|---|---|
| 16 | 562 | 593 | 614 | 13314 |
| 64 | 2456 | 3812 | 5847 | 27485 |
| 256 | 7592 | 12109 | 16889 | 19217 |
| 1024 | 5782 | 17407 | 25581 | 31174 |
| 4096 | 7816 | 15998 | 18086 | 19446 |
LockFreeQueue End-to-End Latency (SPSC, 30000 items per queue size)
| Queue Capacity | P50 (ns) | P95 (ns) | P99 (ns) |
|---|---|---|---|
| 16 | 383 | 631 | 765 |
| 64 | 381 | 611 | 720 |
| 256 | 547 | 16707 | 19870 |
| 1024 | 518 | 7892 | 34899 |
| 4096 | 512 | 770 | 6104 |
Throughput (400000 items per configuration)
| Configuration | Queue Cap | Throughput (Mops/s) |
|---|---|---|
| SPSC (1P×1C) | 256 | 9.29 |
| MPSC (4P×1C) | 1024 | 5.87 |
| MPMC (4P×4C) | 4096 | 6.23 |
| MPMC (8P×4C) | 4096 | 4.29 |
Stress / Correctness (100000 items each)
| Test | Items | Result |
|---|---|---|
| SPSC (1P×1C) | 100000 | ✅ PASS |
| MPSC (4P×1C) | 100000 | ✅ PASS |
| MPMC (4P×4C) | 100000 | ✅ PASS |
| MPMC (8P×8C) | 100000 | ✅ PASS |
ubuntu-latest, Release build.
Docs
Doxygen warnings
0 warning(s) in the files this pull request changes, which fail the check; 48 more elsewhere, which do not. The full list is in the job log and the doxygen-warnings artifact of the Docs workflow.
Gaps in files this pull request does not change, by file:
| Warnings | File |
|---|---|
| 17 | include/libe3/logger.hpp |
| 9 | include/libe3/dapp_subscription_state.hpp |
| 7 | include/libe3/lockfree_queue.hpp |
| 6 | include/libe3/mpmc_queue.hpp |
| 1 | include/libe3/error_codes.h |
| 1 | include/libe3/libe3.hpp |
| 1 | src/connector/zmq_connector.hpp |
Ready to merge (fast-forward only)
A maintainer can land the reviewed commits with:
git fetch origin
git checkout main && git merge --ff-only d429af715262fcf0a725079e175bcbf755d1cfad && git push origin mainHead: d429af715262fcf0a725079e175bcbf755d1cfad (branch feat/140-uring-connector). If --ff-only fails as non-fast-forward, the branch must be rebased on the latest main.
One comment per PR, rewritten in place once every workflow for
d429af7finished.
Red Hat Enterprise Linux 9 and OpenShift's nodes ship kernel.io_uring_disabled=2, which refuses the io_uring link even to root. docs/connectors/uring.md gains a section with the least that has to be opened: set kernel.io_uring_group to a group of its own, set io_uring_disabled to 1, run the RAN or the dApp as a member of that group with no capability, and put both values back afterwards (from a trap, so that a failed run does not leave the node open), with the check that it took and that another user is still refused, and what changes in a pod (a privileged container is needed to write the sysctl, which is the host's and not namespaced, so it cannot be set from securityContext.sysctls). The README gets the same commands in a short section of its own, with a pointer to the page. This is what was done on the cluster for the pod measurements: the window was open for four minutes, the node read 2 and -1 again afterwards, a user outside the group was still refused while it was open, and one in the group created rings with no capability. Persisting the setting across reboots is described but was not tried. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
Summary
E3LinkLayer::URING(--link uring): the POSIX link's endpoint, sockets and bytes (IPC, TCP and SCTP, as framed since posix: one endpoint per dApp, one PDU per message, and each PDU to its dApp's own connection #153), driven through Linux io_uring. A RAN of one link serves a dApp of the other; tested in both directions on all three transports, and against a real gNB (below).URING = 2, stringuring, C API range0..2, PythonE3LinkLayer_URING). No newE3Configfield orE3Connectormethod:PosixOptionsapply as they are. Ring options are deployment tuning read fromLIBE3_URING_*when the connector is created, not ABI.LIBE3_ENABLE_URING, default ON = Linux with liburing found;LIBE3_HAS_URING0/1). Without it, or where the kernel refuses io_uring (Docker's default seccomp,kernel.io_uring_disabled, a kernel before 5.11),create_connectorreturns null with the reason, like ZMQ without libzmq. No silent fallback.send()still returns when the kernel has the bytes andSEND_DONEcompares row by row with POSIX. A broadcast is one batch: a stalled dApp no longer delays the others.LIBE3_URING_BUSY_POLL_US) and kernel submission thread (LIBE3_URING_SQPOLL); registered files on, registered buffers off by default, each with a fallback.connector: bound the retransmission timeout of every SCTP socket): each SCTP association bounds its retransmission timeout (120 ms first, 20 to 250 ms after) instead of Linux's 1 s minimum. A burst of 2000 PDUs of up to 64 KiB dropped the dApp (a send stalled past the 500 ms timeout) in 8 of 100 runs on POSIX and 7 of 100 on io_uring; 0 of 100 on both with the bound.test_socket_linkreads the values back. Drop that commit if the POSIX timers should stay as posix: one endpoint per dApp, one PDU per message, and each PDU to its dApp's own connection #153 left them.posix_connector.cpptosocket_link.{hpp,cpp}with no change of behaviour (the POSIX tests are unchanged and pass).free_ports(), which reserves them with a lock file (a port from the range the kernel uses for source ports collided with a TIME_WAIT one, 2 runs in 30 oftest_uring_connector; and three test files derived their ports from the pid with bands of their own that overlapped, sotest_role_pair_posixfailed in 8 runs of 12 beside the others; 0 in 80 and 0 in 12 after); the link-matrix CTest entries run withLATREC_ENTRIES_LOG2=12(the 8 MiB pre-faulted latrec ring of every agent thread filled a disk when the suite was repeated); the full-loop benchmark's pairing check allowed no skew between a send stamp and the receive stamp of a receiver that polls, and refused about one run in ten of SCTP with busy polling although the counts of PDUs out and in agreed (it now allows 5 us, says which pair was first, and its arithmetic is signed).docs/connectors/uring.md.Which link is fastest
tools/bench_matrix.py, nine interleaved rounds of 3000 round trips per cell, ASN.1, a Linux 6.8 virtual machine of 12 vCPUs on an Intel Xeon Gold 6240R server (nothing pinned or isolated; the cluster's host kernels disable io_uring, so it had to be a VM); round trip p50 in µs, CPU per round trip in brackets:LIBE3_URING_SQPOLLwithout busy polling is not a general choice (29% below POSIX on IPC, the same on TCP, 16% above on SCTP).docs/connectors/uring.md.Type of change
Linked issue
Closes #140.
Decisions taken without the owner (the issue put three to him)
send()returns once the kernel has the bytes, as POSIX; latrec rows compare one to onesend()returns on enqueue, a thread hop on every PDU (a wakeup, 15 to 20 µs on the Arm test machine)LIBE3_ENABLE_URINGAlso: per-thread rings with
SINGLE_ISSUER/DEFER_TASKRUNnot done (the facade lets several threads read and send). Consequence written in the doc: a socket call with a timeout on the RAN's inbound thread (where SM control handlers run) can returnEINTRwhile other dApps' receives are armed.docs/interop.mdhas a ledger row for the new link and for the SCTP timer bound. No VERSION bump (0.6.0 is unreleased).Testing
Linux 6.8 on x86-64 (a KubeVirt VM), on the final sources:
LIBE3_REQUIRE_URING=1 LIBE3_REQUIRE_SCTP=1: 81/81../build_libe3 -c -r --all-encodings --enable-sctp -tfrom scratch: 62/62. Each of the first six commits was built on its own; the connector tests passed on the first four. The six were then run again as one tree (the seventh changes documents only): 81/81 as above, and the 28 port-using tests ten times over at-j2and at-j6(--repeat until-fail:10): 0 failures in each.test_uring_connectoralone 80 times, no failure (2 in 30 before the port fix). The connector subset (21 tests at-j3) 12 times, no failure; before the second port fix it failed in 8 of 12, alwaystest_role_pair_posixor a variant (see below). The same on an Arm VM: the io_uring and link-matrix tests 15 times each, clean.example_simple_agent+example_simple_dappas two processes: ZMQ, POSIX and io_uring on each transport and both mixed directions on IPC, TCP and SCTP: 14/14.--E3Configuration.link uring) and one nrUE on the RF simulator, and the dApp library's spectrum dApp, on IPC, TCP and SCTP with the gNB and the dApp each on POSIX or io_uring: 12/12 pairs run, 16,000 to 18,600 sensing slots handled in 15 s, 99.7 to 99.9% of them with their sensing ranges, no assertion in the gNB. The dApp library's own tests: 141 passed, 4 skipped, with libe3py built from this branch. Two small patches for the twins (below) came out of it.E3LinkLayer_URING == 2.URINGexisted; the stalled-peer test caught theEINTRhazard of decision 6;test_socket_linkfails with the SCTP timer block removed;test_bench_matrixfails without the signed parser.One test failure I could not explain, not in this PR's code: in a stress run (
ctest -j2 --repeat until-fail:15over 29 tests)test_latrec_stagesfailed once in itsposix-ipc-asn1combination: the RAN's outbound queue (capacity 128) was full for about a second and 200 indications were dropped (Buffer too small), with nothing logged by the connector. It did not reproduce in 100 runs of the same test at this branch and 100 at #153's head under the same load. The POSIX path and the queue are what #153 left; I have not touched them beyond the shared-socket move. Worth a look if it shows up in CI.Not run: GitHub's runners (the CI changes are untested until a PR runs them; the first run rebuilds the asn1c cache because
build_libe3changed); bare metal with isolated cores; two hosts; OCUDU's agent; a gNB with a core network.Found on the way, not changed here
srwxr-xr-x root rootfor a gNB undersudo), so a dApp of another user cannot connect to it. The same on the POSIX link. A one-linechmodinsocket_link::listen_onwould change it; it is a decision about who may talk to a RAN, so I left it.unsupportedIdentifier. A dApp-library fix.Twin-repo coordination
docs/interop.md.The ABI change is additive (one enumerator, a wider C API range). A twin only needs a change to offer
uringin its own configuration parsing; nothing breaks without it. Both changes are on branches, built and run as above: the dApp library (E3LinkLayer.URINGand a link map that offersuringonly when the binding has it) and OAI (uringamong the E3 configuration's accepted strings and the README). No pull request is open for either.Base
Stacked on #153 (
feat/104-107-posix-one-endpoint); seven commits (the last one is the section on switching io_uring on for one group, in the README anddocs/connectors/uring.md).