Hard-exit ACQ on terminate to avoid device-finalizer SIGABRT (#710) - #826
Draft
lwhite1 wants to merge 1 commit into
Draft
Hard-exit ACQ on terminate to avoid device-finalizer SIGABRT (#710)#826lwhite1 wants to merge 1 commit into
lwhite1 wants to merge 1 commit into
Conversation
run_acq handled TerminateServerRequest by tearing down devices via close_streams() and then returning, which let Python GC-finalize the device objects as the frame unwound. close_streams() has already freed their native handles, so the finalizers (LSL outlets, MetaWear/warble, camera SDKs) re-ran native teardown on freed handles and aborted the process with SIGABRT before it could exit. This is the dominant ACQ_0 silent death: 18 of 22 faulthandler dumps in the 2026-05-28..06-11 window, each with the main thread parked at server_acq.py:46 (run_acq just returned) and the last DB log line 'Microphone: Exiting LSL Thread' (device teardown complete). Flush logging and os._exit() directly from the handler instead of returning, so no device finalizer runs. The terminate path keeps its clean log trail (Stopping ACQ -> Closing app log db connection). Covers the TerminateServerRequest path. The exception path (run_acq raises -> frame unwind -> same finalizer abort) is a separate follow-up. Confirm the exact aborting module via WER LocalDumps before merge.
This was referenced Jun 16, 2026
Contributor
Author
|
Holding this draft — the merge gate isn't cleared, and #818 may be the actual root cause.
Keeping it open as hardening (it skips device finalizers on terminate regardless), but it should not merge until (a) a ProcDump native stack confirms the finalizer, and (b) the #818 env rebuild has been tested. |
lwhite1
added a commit
that referenced
this pull request
Jul 1, 2026
…ool (#835) extras/perf had accumulated single-use investigation scripts tied to now-resolved issues: the inter-task gap analysis, the 2026-04-29 LSL-startup-timeout forensics (sessions 3220/3223), and the #819 / break_video empty-stream checks. Each is a leaf node -- nothing imports it and no test covers it -- so removing them leaves the reusable Win11-readiness harnesses, the retrospective perf tools, and the shared libraries (_db, _baseline_common, _sdk_probe, intertask_report) untouched. _flir_queue_history (#775) is removed too and stays recoverable from history if still needed. Also commits _investigate_silent_exit.py -- the #818/#826 silent-exit audit tool that docs/booth_venv_rebuild.md depends on. It had only ever been staged, never committed, so it was one git gc away from being lost.
This was referenced Jul 10, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — do not merge until the WER LocalDump confirms the line‑46 finalizer (see Merge gate).
What
On
TerminateServerRequest,run_acqtore down devices viaclose_streams()and then returned, letting Python GC-finalize the device objects as the frame unwound.close_streams()has already freed their native handles, so the finalizers (LSL outlets, MetaWear/warble, camera SDKs) re-ran native teardown on freed handles and aborted the process with SIGABRT before it could exit.This flushes logging and calls
os._exit()directly from the terminate handler instead of returning, so no device finalizer runs. The terminate path keeps its clean log trail (Stopping ACQ→Closing app log db connection).Why this is the fix
Production
log_applicationcorrelation (extras/perf/_investigate_silent_exit.py --audit, Wang, 2026‑05‑28…06‑11):Fatal Python error: Aborted, main thread parked atserver_acq.py:46(run_acq just returned), only the app‑log SSH‑tunnel threads alive (bystanders), no device threads.close_streams()— i.e. full device teardown completed, then abort. Not idle, not mid‑session.logging.shutdown(). The surviving tunnel threads belong to the app‑log handler (torn down later) — bystanders, not the crash site.Mechanism detail and correction: #710. Full census + falsified alternatives: #824.
Merge gate (per "falsify before shipping")
DumpType=2,python.exeon ACQ_0) naming the aborting native module at the line‑46 finalization. The fix skips all finalizers so it is robust regardless, but the dump proves we are eliminating the fault, not masking a different one.Scope / follow‑ups
TerminateServerRequestpath (the 18 Family‑A crashes).run_acqraises → frame unwind → same finalizer abort,server_acq.py:213‑217) — separate follow‑up.logger.debug("Stopping ACQ")atmain()line 47 is left in place to keep the diff minimal; can be removed.Relates to #710, #824, #759.