Skip to content

Hard-exit ACQ on terminate to avoid device-finalizer SIGABRT (#710) - #826

Draft
lwhite1 wants to merge 1 commit into
masterfrom
fix/710-acq-shutdown-finalizer-abort
Draft

Hard-exit ACQ on terminate to avoid device-finalizer SIGABRT (#710)#826
lwhite1 wants to merge 1 commit into
masterfrom
fix/710-acq-shutdown-finalizer-abort

Conversation

@lwhite1

@lwhite1 lwhite1 commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Draft — do not merge until the WER LocalDump confirms the line‑46 finalizer (see Merge gate).

What

On TerminateServerRequest, run_acq tore down devices via close_streams() and then returned, letting Python GC-finalize the device objects as the frame unwound. close_streams() has already freed their native handles, so the finalizers (LSL outlets, MetaWear/warble, camera SDKs) re-ran native teardown on freed handles and aborted the process with SIGABRT before it could exit.

This flushes logging and calls os._exit() directly from the terminate handler instead of returning, so no device finalizer runs. The terminate path keeps its clean log trail (Stopping ACQClosing app log db connection).

Why this is the fix

Production log_application correlation (extras/perf/_investigate_silent_exit.py --audit, Wang, 2026‑05‑28…06‑11):

  • 18 of 22 ACQ_0 faulthandler dumps are this crash: Fatal Python error: Aborted, main thread parked at server_acq.py:46 (run_acq just returned), only the app‑log SSH‑tunnel threads alive (bystanders), no device threads.
  • Each one's last DB log line is "Microphone: Exiting LSL Thread" — the final line of close_streams() — i.e. full device teardown completed, then abort. Not idle, not mid‑session.
  • Clean exits log "Closing app log db connection" (line 55); these never reach it → the abort is at the line‑46 return/finalization boundary, before logging.shutdown(). The surviving tunnel threads belong to the app‑log handler (torn down later) — bystanders, not the crash site.

Mechanism detail and correction: #710. Full census + falsified alternatives: #824.

Merge gate (per "falsify before shipping")

  • WER LocalDump (full minidump, DumpType=2, python.exe on ACQ_0) naming the aborting native module at the line‑46 finalization. The fix skips all finalizers so it is robust regardless, but the dump proves we are eliminating the fault, not masking a different one.
  • Post‑deploy field check: ACQ_0 unclean‑exit rate falls toward the STM box's, and "Closing app log db connection" becomes the consistent last line in the shutdown audit.

Scope / follow‑ups

  • Covers the TerminateServerRequest path (the 18 Family‑A crashes).
  • Not covered: the exception path (run_acq raises → frame unwind → same finalizer abort, server_acq.py:213‑217) — separate follow‑up.
  • The now‑unreachable logger.debug("Stopping ACQ") at main() line 47 is left in place to keep the diff minimal; can be removed.

Relates to #710, #824, #759.

run_acq handled TerminateServerRequest by tearing down devices via close_streams() and then returning, which let Python GC-finalize the device objects as the frame unwound. close_streams() has already freed their native handles, so the finalizers (LSL outlets, MetaWear/warble, camera SDKs) re-ran native teardown on freed handles and aborted the process with SIGABRT before it could exit.

This is the dominant ACQ_0 silent death: 18 of 22 faulthandler dumps in the 2026-05-28..06-11 window, each with the main thread parked at server_acq.py:46 (run_acq just returned) and the last DB log line 'Microphone: Exiting LSL Thread' (device teardown complete).

Flush logging and os._exit() directly from the handler instead of returning, so no device finalizer runs. The terminate path keeps its clean log trail (Stopping ACQ -> Closing app log db connection).

Covers the TerminateServerRequest path. The exception path (run_acq raises -> frame unwind -> same finalizer abort) is a separate follow-up. Confirm the exact aborting module via WER LocalDumps before merge.
@lwhite1

lwhite1 commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Holding this draft — the merge gate isn't cleared, and #818 may be the actual root cause.

Keeping it open as hardening (it skips device finalizers on terminate regardless), but it should not merge until (a) a ProcDump native stack confirms the finalizer, and (b) the #818 env rebuild has been tested.

lwhite1 added a commit that referenced this pull request Jul 1, 2026
…ool (#835)

extras/perf had accumulated single-use investigation scripts tied to
now-resolved issues: the inter-task gap analysis, the 2026-04-29
LSL-startup-timeout forensics (sessions 3220/3223), and the #819 /
break_video empty-stream checks. Each is a leaf node -- nothing imports it
and no test covers it -- so removing them leaves the reusable Win11-readiness
harnesses, the retrospective perf tools, and the shared libraries (_db,
_baseline_common, _sdk_probe, intertask_report) untouched. _flir_queue_history
(#775) is removed too and stays recoverable from history if still needed.

Also commits _investigate_silent_exit.py -- the #818/#826 silent-exit audit
tool that docs/booth_venv_rebuild.md depends on. It had only ever been staged,
never committed, so it was one git gc away from being lost.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant