Repository navigation
Conversation
CI report —
|
| Workflow | Result | Time | Run |
|---|---|---|---|
| CI | ✅ success | 25m50s | #9 |
| Docs | ✅ success | 0m31s | #83 |
CI
✅ posix/ipc
- dApp exit:
0 - Indications received:
7
✅ posix/tcp
- dApp exit:
0 - Indications received:
7
✅ zmq/ipc
- dApp exit:
0 - Indications received:
7
✅ zmq/tcp
- dApp exit:
0 - Indications received:
7
zmq/ipc
- ✅ 1 RAN - 1 dApp: indications=
5dapppeer=t11 ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.4 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
- ✅ 1 RAN - 2 dApps: dApp#1 ind=
5sub=1, dApp#2 ind=6sub=2, RAN saw2dApps - ✅ 2 RANs - 1 dApp: from ran-a ind=
5, from ran-b ind=5
zmq/tcp
- ✅ 1 RAN - 1 dApp: indications=
5dapppeer=default ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
- ✅ 1 RAN - 2 dApps: dApp#1 ind=
5sub=1, dApp#2 ind=6sub=2, RAN saw2dApps - ✅ 2 RANs - 1 dApp: from ran-a ind=
5, from ran-b ind=5
Full-loop latency
Full-loop latency benchmark (N=1161 after 50 warmup)
All values in microseconds (μs). Link: zmq, transport: ipc, encoding: ASN.1 APER.
| # | Description | Tags | mean | p50 | p99 | max |
|---|---|---|---|---|---|---|
| 1 | Collect indication data | RECORD_BEGIN to ENCODE_E3SM_BEGIN |
0.06 | 0.06 | 0.13 | 0.25 |
| 2 | Create & encode indication | ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE |
0.40 | 0.38 | 0.61 | 9.80 |
| 3 | Encode E3AP (indication) | EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE |
1.30 | 1.24 | 3.12 | 11.55 |
| 4 | Queuing (indication) | ENQUEUE to DEQUEUE |
0.79 | 0.62 | 5.24 | 15.62 |
| 5 | Delivery (indication) | ENCODE_E3AP_DONE to SEND_DONE |
0.18 | 0.16 | 0.35 | 2.51 |
| 6 | E3 wire (RAN -> dApp) | SEND_DONE to RECV |
28.02 | 27.40 | 39.66 | 49.70 |
| 7 | Decode E3AP (indication) | RECV to DECODE_E3AP_DONE |
0.96 | 0.86 | 1.37 | 19.12 |
| 8 | libe3 dispatch (indication) | DECODE_E3AP_DONE to DELIVER_BEGIN |
0.06 | 0.05 | 0.11 | 0.27 |
| 9 | Decode indication | DELIVER_BEGIN to DECODE_E3SM_DONE |
0.30 | 0.29 | 0.44 | 0.83 |
| 10 | Process data | DECODE_E3SM_DONE to ENCODE_E3SM_BEGIN |
0.03 | 0.03 | 0.04 | 0.14 |
| 11 | Create & encode control | ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE |
0.19 | 0.17 | 0.29 | 8.81 |
| 12 | Encode E3AP (control) | EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE |
1.80 | 1.78 | 2.44 | 12.02 |
| 13 | Queuing (control) | ENQUEUE to DEQUEUE |
1.29 | 0.64 | 9.58 | 16.78 |
| 14 | Delivery (control) | ENCODE_E3AP_DONE to SEND_DONE |
4.25 | 4.11 | 6.31 | 14.71 |
| 15 | E3 wire (dApp -> RAN) | SEND_DONE to RECV |
29.99 | 29.45 | 40.41 | 329.61 |
| 16 | Decode E3AP (control) | RECV to DECODE_E3AP_DONE |
1.16 | 1.04 | 3.27 | 14.70 |
| 17 | libe3 dispatch (control) | DECODE_E3AP_DONE to DECODE_E3SM_BEGIN |
0.23 | 0.22 | 0.52 | 0.91 |
| 18 | Decode & handle control | DECODE_E3SM_BEGIN to DECODE_E3SM_DONE |
0.18 | 0.18 | 0.27 | 0.63 |
| Total | Total round-trip | 71.42 | 69.10 | 95.75 | 385.54 |
ubuntu-latest, Release build, ZMQ + IPC, ASN.1 APER.
These numbers are measured inside a GitHub Actions container and should be treated as an upper bound on E3AP's and the library's own latency, not a representative deployment measurement.
Docs
Doxygen warnings
0 warning(s) in the files this pull request changes, which fail the check; 119 more elsewhere, which do not. The full list is in the job log and the doxygen-warnings artifact of the Docs workflow.
Gaps in files this pull request does not change, by file:
| Warnings | File |
|---|---|
| 37 | include/libe3/sm_interface.hpp |
| 17 | include/libe3/logger.hpp |
| 17 | swig/libe3_swig.hpp |
| 17 | swig/e3_dapp_session.hpp |
| 9 | include/libe3/dapp_subscription_state.hpp |
| 7 | include/libe3/lockfree_queue.hpp |
| 6 | include/libe3/mpmc_queue.hpp |
| 1 | include/libe3/error_codes.h |
| 1 | include/libe3/libe3.hpp |
| 1 | src/connector/zmq_connector.hpp |
Ready to merge (fast-forward only)
A maintainer can land the reviewed commits with:
git fetch origin
git checkout main && git merge --ff-only 0fbf154ea32a14672f9259ffc9789084b2ffc639 && git push origin mainHead: 0fbf154ea32a14672f9259ffc9789084b2ffc639 (branch fix/93-121-vanished-dapp-and-reconnect). If --ff-only fails as non-fast-forward, the branch must be rebased on the latest main.
One comment per PR, rewritten in place once every workflow for
0fbf154finished.
451ee31 to
0118c8f
Compare
0118c8f to
8530e90
Compare
The test failed now and then on pull requests that did not touch it (#146, #152 and the branch of #96). With the original test, 100 runs under load on main failed 3 and then 13 times; on the one-endpoint POSIX branch 7 of 320 failed. Two assertions failed together: no_drops_were_recorded (one LATREC_DROP_SEND) and every_enqueued_pdu_is_either_sent_or_dropped (one PDU more enqueued than sent or dropped, in two runs one or two more with no send drop). The test ran the dApp and the RAN's service model until the end and then stopped the dApp first. The service model emits every few milliseconds, so a PDU could be sent to the connection that had just closed (a send failure, which the test counts as a defect), or be enqueued near the shutdown of the outbound queue with neither a send nor a drop recorded. Each combination now unsubscribes first, waits until the RAN has no subscriber left, and pauses for what is queued to reach the dApp, before it stops anything. It also waits for the dApp to hold its subscription after it asks (under load none was held 500 ms after the request), and pauses before asking, as other POSIX tests do. The test also explains a failure now: for every non-shutdown drop it prints the combination, how long before the combination ended it happened, and each stage the PDU passed with its aux values, and it notes an unsubscribe that did not succeed or a dApp that never held its subscription. That is how the cause above was found. Measured in a Linux container (aarch64), 8 runs at a time: 0 failures in 320 runs on main with this change, against 3 to 13 in 100 without it; 0 in 320 on the one-endpoint branch with the same test against 7 in 320 without. One thing was not found: on the three-connection layout a dApp was sometimes seen not to learn of a subscription that the RAN held (its unsubscribe returned SUBSCRIPTION_NOT_FOUND); the test no longer depends on it, but its cause is open. The ZeroMQ combinations were not seen to fail. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
b67557c to
bfcd006
Compare
bfcd006 to
87c6b47
Compare
87c6b47 to
d79a331
Compare
A libe3 RAN removed a dApp only when the dApp sent a release. A dApp that crashed or lost its network kept its registration, its id and its subscriptions for good, and the RAN function kept running for it (#93). Aerial's agent drops a dApp after 1800 s and tells it; OCUDU closes the association after 30 s. Each registered dApp now carries the time of its last message. The RAN's inbound loop moves it for every PDU that names a sender, before the dispatch, so a report that the queue then drops still counts. The expiry thread asks the subscription manager which dApps have been quiet for E3Config::dapp_idle_timeout_s, sends each a ReleaseMessage, then removes it the way a release from the dApp does: the registration and the subscriptions go, a RAN function whose last subscriber it was is stopped, and the status handler is told. The release is queued first so it still names the old owner when the freed id is handed out again. * The default is 1800 s, as Aerial. 0 never removes a quiet dApp. OCUDU's 30 s is a setting, not the default. * Only messages from the dApp reset the time. The RAN broadcasts, so an indication sent to a dApp that is gone looks the same as one sent to a dApp that is there; counting those would keep a crashed dApp that still has a subscription for good. * A dApp that only listens sends nothing, so it is released too. * A RAN built through the C API sets the time with set_dapp_idle_timeout and dapp_idle_timeout_s in e3_config_t. The time is applied only when the flag is set, as pin_io_threads gates io_thread_affinity: 0 is a legitimate time, so it cannot also mean "keep the default", and a zeroed struct keeps the default. The two fields are appended, so e3_config_t grows and a C program must be rebuilt. Specification and guide: 03 and 05 say how an E3 Agent detects the end of a dApp and that only what the dApp sends shows it is there; the integrator guide gets the row and its pointers; the Word edition is rebuilt. README lists dapp_idle_timeout_s. The interop page gets the libe3 row of "When the RAN goes away", the rows of the table of differences that this changes, the Aerial table row, and a ledger row. Tests: the subscription manager's idle bookkeeping on a fake clock; a quiet dApp over ZeroMQ is released and removed no sooner than the idle time, its RAN function stops, and a second dApp that keeps sending reports is untouched; 0 never removes one; a RAN made from C with a time releases a quiet dApp, with the flag and 0 keeps it, and with a time and no flag ignores the time. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Thecave3 <thecave003@gmail.com>
…connection A dApp that crashes closes its sockets. A POSIX RAN saw that and dropped the socket, but never told the rest of the library: the dApp kept its id, its subscriptions and a running RAN function until the idle time (#93). And the E3AP messages carry no authentication, so any connected dApp could end or control another by naming its id. The POSIX connector gives each accepted inbound peer a token that is never reused, since a file descriptor is, and reports the token of the peer that supplied the last bytes and the tokens of peers that closed. E3Connector gets two virtual functions for that, with defaults that report no peers, so other connectors are unchanged. The RAN's inbound loop binds a dApp id to the first connection that sends a message naming it, for as long as that registration lasts. * When a connection closes, the dApps bound to it are removed the way a release would remove them. Nothing is sent: the connection is gone. It is acted on after the bytes the peer sent before closing were delivered, so a release or a last report that arrives with the hang-up is not lost. * A later message that names a bound id on another connection (a release, a subscription delete, a control, a report) is dropped with a warning, and does not count as a message from the dApp, so a forged message cannot keep an idle dApp alive either. A message for an id that is not registered is left to its handler, as before. * A registration has a generation. An id is handed out again after its dApp leaves, so a record made for an earlier generation is replaced, and a connection that closes later cannot take the next holder of the id with it. * ZeroMQ cannot tell its peers apart, so nothing changes there. Nor does a dApp that crashes before its first message get removed at once: nothing on the wire carries identity at connect time, so that waits for the idle time. The first message wins: a connection that names an id before its dApp speaks takes it. Specification and guide: 03 says an E3 Agent detects the end of a dApp from a closed connection too, and that one that can tell its connections apart should bind each dApp identifier to the connection that first named it; the integrator guide gets the rows, its checklist item and the pointers; the Word edition is rebuilt. The interop page gets the closed-connection and binding paragraphs, the rows of the table of differences for the release and the POSIX link, and a ledger row. Tests: a stopped dApp is removed within a second with the idle time off and a second dApp keeps its subscription; one that never spoke is removed by the idle time; one that released is not removed twice, and a dApp that was handed its id survives the old connection closing; a release and a subscription delete that name a dApp but arrive on a second connection are dropped, the dApp keeps its registration, subscription and RAN function and hears nothing, can still subscribe, and can still release itself. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Thecave3 <thecave003@gmail.com>
A libe3 dApp that was released by the RAN, or lost its connection to it, reported the disconnect and then stayed down until the program called stop() and start(). Aerial's manager runs setup again every second until it works (#121). A RAN that releases a quiet dApp (#93) would otherwise leave every such dApp down. A dApp started with start() now has a thread that, after E3Config::reconnect_interval_ms (default 1000, 0 is off), stops and starts the interface again, and again every interval until the RAN answers. The cases are: * the RAN released the dApp, or the connection was lost: the disconnect handler runs first, as before; * setup did not finish or the RAN refused it, including a dApp that was started before any RAN was up; * a restart that fails because the RAN cannot be reached asks again; * a dApp whose inbound or outbound connection cannot be made because the RAN left first: those threads ended with a log line and the dApp never noticed that the RAN was gone, so nothing would have brought it back. It reports CONNECTION_LOST now. The same gap made test_dapp_ran_loss fail now and then on the POSIX link, whenever the RAN stopped before the dApp had connected. The new session has a new dApp id and none of the old subscriptions. The application subscribes again; it learns the dApp is back from the setup response handler, which runs on every setup. A dApp that called release(), and one that called stop(), are not restarted, and stop() does not wait for a RAN. start() and stop() keep their meaning: they are now the locked halves start_locked() and stop_locked() behind one lock, which the reconnect thread takes to restart, so a restart cannot overlap a call of the application. stop() joins the reconnect thread before it takes that lock. test_dapp_ran_loss and test_dapp_idle_e2e set the interval to 0: the first restarts the dApp by hand, the second counts dApps on the RAN. Specification and guide: 03 and 05 say a dApp may run Setup again after a release or a lost link, with a new identifier and no subscriptions; the integrator guide gets the row and its pointers; the Word edition is rebuilt. README and swig/README.md name reconnect_interval_ms and set_disconnect_handler, and the handler's comment and the example say the library registers again. The interop page replaces "the dApp does not reconnect", says what a dApp sees against OCUDU's 30 s idle close, and gets a ledger row. Tests: a dApp registers with a new RAN on ZeroMQ and POSIX without stop() and start(), hears of it, and can subscribe; one released for being idle is told RELEASED_BY_RAN and comes back; one started before any RAN registers when one comes up; one that released itself stays released; stop() ends the retrying quickly; 0 keeps the dApp down. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Thecave3 <thecave003@gmail.com>
…gen flags The Docs check fails a pull request on any Doxygen warning in a file it changes, old ones included (#132). This change touches e3_agent.hpp, e3_connector.hpp, subscription_manager.hpp, c_api.h and posix_connector.hpp, so their gaps are filled here instead of being left for the check to report. * @return and @PARAM for the accessors, setters and callbacks of E3Agent, E3Connector and SubscriptionManager; a one-line description for the handler type aliases, the move operations and the data members. * e3_config_t and e3_c_service_model_desc_t get a `///<` description for every member, in place of the `//` and `/* */` comments, which Doxygen does not read. The struct and the SM handle in c_api.cpp, E3Agent::Impl and DAppSession::Impl are described, as is each of their members. * A log path pattern written `<role>` is written without the angle brackets, which Doxygen read as an HTML tag. The integrator guide's line pointers follow the added comment lines. Comments only: no code changes. VERSION 0.4.1: this change adds fields to E3Config and e3_config_t and two virtual functions to E3Connector, so OAI's E3 agent and the dApp library are rebuilt with it. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Thecave3 <thecave003@gmail.com>
…and did not test Three claims of the dApp lifecycle changes had no test. An indication the RAN sends to a quiet dApp does not count as activity from it. The idle test's Service Model sent nothing, so a RAN that counted what it sends would have passed. A new test streams an indication every 100 ms to a subscribed dApp that sends nothing and checks that it is still released after the idle time. On POSIX a control and a report that name a dApp but come on another connection are dropped, as a release and a subscription delete already were. The test sends both from a second connection, then from the dApp's own, and checks that only the second is taken. The defaults of the idle time (1800 s) and of the reconnect interval (1000 ms) are asserted with the other E3Config defaults. Each of the two new tests fails when the behaviour is broken: admitting a message from another connection fails the binding test, and counting an indication sent to a dApp as activity fails the idle test. No code, no grammar and no VERSION change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Thecave3 <thecave003@gmail.com>
d79a331 to
0fbf154
Compare
Summary
A libe3 RAN keeps a dApp that crashed or lost its network for good: its id, its subscriptions and the RAN function it was the last subscriber of (#93). A libe3 dApp whose RAN went away, or released it, stays down until the program calls
stop()thenstart()(#121). This PR fixes both.What a RAN does now
E3Config::dapp_idle_timeout_s(default 1800 s, as Aerial's agent;0never) gets areleaseMessagenaming it. Its registration and subscriptions are then dropped, a RAN function it was the last subscriber of is stopped, and the dApp status handler is called. It is the removal a release from the dApp already gets.set_dapp_idle_timeoutanddapp_idle_timeout_sine3_config_t.What a dApp does now
E3Config::reconnect_interval_ms(default 1000, as Aerial's manager;0never), until the RAN answers.release()orstop()is not restarted, andstop()does not wait for a RAN.CONNECTION_LOST. Those threads used to end with a log line and the dApp never noticed; this was also behind the occasionaltest_dapp_ran_lossfailure on the POSIX link.What a user sees
CONNECTION_LOSTand stays downWhat is not done
Commits
One per feature, each with its tests, specification and guide updates, README rows, interop page rows and ledger row, so each builds and passes the tests on its own:
core: release and remove a dApp that has sent nothing for too long([Interop]: A libe3 RAN agent keeps a dApp that vanished without sending a release #93), with the C API fieldposix: remove a dApp whose connection closed, and bind a dApp to its connection([Interop]: A libe3 RAN agent keeps a dApp that vanished without sending a release #93)core: register a dApp with the RAN again when it lost the RAN([Interop]: A libe3 dApp does not reconnect after the RAN goes away #121)docs: document the members of the files this change touches that Doxygen flags, comments onlyNotes for review
VERSIONis not bumped. This is the first of a stack of interop changes and the stack's bump comes later.E3Configgets two fields,e3_config_ttwo, andE3Connectortwo virtual functions, so OAI's E3 agent and the dApp library have to be rebuilt with it.E3Interface::start()andstop()keep their meaning. They now take one lock and callstart_locked()andstop_locked(); the reconnect thread takes the same lock to restart, so a restart cannot overlap a call of the application.stop()joins that thread first.test_dapp_ran_lossandtest_dapp_idle_e2esetreconnect_interval_msto 0: the first restarts the dApp by hand, the second counts dApps on the RAN.test_e2e_report_path(latrec build only) andtest_bench_latrecfail onmaintoo. The new tests use generous waits for the same reason.Type of change
Linked issue
Closes #93
Closes #121
Twin-repo coordination
docs/interop.md.The wire protocol is unchanged, but
E3ConfigandE3Connectorchange, so the first box is not ticked: the two twin repos have to be rebuilt with it, and the paired PRs are not open yet.Paired PR(s): none yet