fix(scheduler): survive stop/start races and self-heal a dead scheduler thread

[federicodeponte]

Scope, so nobody misreads it

The PRIMARY cause of the 2026-08-02 incident was not in this repo. It was restartPolicyType = "on_failure" in the cloud repo's railway.toml: a clean SIGTERM meant Railway never restarted the service, and it stayed down 5h33m. That is fixed in floomhq/workeros-cloud#1287.

This PR fixes a separate latent bug in the engine with the same symptom: a scheduler that stops firing and never comes back. It was reproduced deterministically, it is real, and it would have produced the same customer-visible outcome on its own. It is not the incident's root cause.

The reproduced race

apps/api/scheduler.py controlled the scheduler thread with one MODULE-GLOBAL _stop_event shared across thread generations:

• The cloud wrapper calls stop_scheduler() and, milliseconds later, start_scheduler() again when its Postgres advisory-lock connection blips.
• The old thread is still sleeping in _stop_event.wait(60), so it is is_alive().
• start_scheduler() early-returned WITHOUT clearing _stop_event.
• The old thread then woke, saw the flag set, and exited.

Net result: no scheduler thread, _stop_event stuck set, and nothing ever restarted it. Reproduced as alive=False stop_set=True.

Changes

1. Per-generation stop event. Every start_scheduler() binds a FRESH threading.Event and hands it to the thread it launches by closure, so a stop aimed at generation N can never terminate generation N+1. stop_scheduler() still stops the current generation. The module-level names _stop_event, _scheduler_thread and _scheduler_lock are preserved (existing tests read them); _scheduler_lock became an RLock so the watchdog can check and restart under one lock.

2. start_scheduler() no longer early-returns while a stop is pending. If the existing thread is alive but stopping, it is joined (bounded, 5s) and replaced. If it is alive and healthy, the call stays the no-op it always was.

3. The poll loop is unkillable by incidental exceptions. _record_scheduler_heartbeat(), _tick() and the sleep are all inside guards, and logging goes through a _safe_log helper, so a broken stdout pipe (seen after a container restart) cannot end the scheduler for the lifetime of the process. Only the generation's own stop event ends the loop.

4. ensure_scheduler_running(), an idempotent watchdog. Three states, only one of which it recovers:

   • ok: live generation, fresh heartbeat. No-op.
   • dead: no thread, thread not alive, or stopping and gone within the bounded join. Recovered by launching a fresh generation.
   • wedged: alive, not stopping, heartbeat stale. Logged loudly and left alone. Python cannot kill a thread, so a replacement would leave two schedulers firing the same triggers and duplicating side effects (emails, CRM writes), which is worse than the delay. A stale heartbeat is also not proof of death: a recovery tick working through many overdue triggers legitimately runs long. This matches the three-state model in the cloud PR.

   It holds _scheduler_lock for the whole check-and-restart, so it never spawns a duplicate thread, and it only ever RE-starts: a process that never started a scheduler (WORKEROS_ROLE=web) cannot grow one by calling it. It is wired as a coarse backstop into the hourly sweep loop in main.py, off the event loop via asyncio.to_thread.

5. Health surfacing (additive only). _health_check_scheduler() now adds alive, heartbeat_age_seconds, stale and stale_after_seconds to the existing scheduler_status() payload, so an alive-but-wedged scheduler is visible. ok deliberately keeps its old meaning and no HTTP status-code behaviour changes.

   scheduler_heartbeat_status() also gained an explicit alive. running is alive and not stopping, so a live thread winding down used to look identical to no thread at all, and a supervisor must tell those apart: restarting or releasing a leader lock while a live thread may still be firing triggers double-fires runs. scheduler_status() is deliberately untouched, since its payload is asserted by exact dict equality in existing tests.

   Known gap, deliberately out of scope: the cloud runs leader election, so a follower legitimately has no scheduler thread and is perfectly healthy. Making "no scheduler thread" unhealthy would mark every follower unhealthy. Leader-aware health semantics are a follow-up.

6. Honest user-facing failure copy in apps/api/services/public_view.py. scheduler_missed, executor_lost_mid_run and run_claimed_without_dispatch had no headline entry, so a customer's platform-abandoned run fell back to unknown_error copy: "This worker failed to run. Check the run logs for details, then edit or re-run the worker." That blames the customer for our restart. They now read:

   • scheduler_missed: Floom's scheduler was delayed, so this scheduled run started later than its scheduled time. The worker itself is fine and no action is needed.
   • executor_lost_mid_run: Floom's execution service stopped while this run was in progress. This is a platform fault, not a problem with the worker. Re-run it if the work did not complete.
   • run_claimed_without_dispatch: Floom picked this run up but the platform stopped before the worker started. Nothing in the worker ran. This is a platform fault; re-run it.

Tests

New: apps/api/tests/test_scheduler_stop_start_race.py, apps/api/tests/test_platform_fault_operator_copy.py. Deterministic: POLL_INTERVAL_SECONDS is monkeypatched and ticks are awaited on an Event, no fixed sleeps, no network.

BEFORE (the new tests run against origin/main in a scratch worktree)

$ cd apps/api && python3 -m pytest tests/test_scheduler_stop_start_race.py::test_start_after_stop_leaves_a_live_ticking_scheduler -q

>       assert ticks.wait_for_first(), "a fresh generation must tick after a stop/start race"
E       AssertionError: a fresh generation must tick after a stop/start race
E       assert False
E        +  where False = wait_for_first()

tests/test_scheduler_stop_start_race.py:86: AssertionError
=========================== short test summary info ============================
FAILED tests/test_scheduler_stop_start_race.py::test_start_after_stop_leaves_a_live_ticking_scheduler
1 failed in 5.74s

Both new files against origin/main: 22 failed, 2 passed.

AFTER (this branch)

$ cd apps/api && python3 -m pytest tests/test_scheduler_stop_start_race.py tests/test_platform_fault_operator_copy.py -q
24 passed

Existing suites, unbroken

$ cd apps/api && python3 -m pytest tests/test_scheduler_missed_fire.py tests/test_scheduler_source_tag.py \
    tests/test_run_metrics.py tests/test_1067_worker_limits_and_cron.py \
    tests/db/test_worker_triggers_normalize.py tests/test_backend_p2_reliability.py \
    tests/test_health_info_disclosure.py tests/test_scheduler_stop_start_race.py \
    tests/test_platform_fault_operator_copy.py -q
56 passed

and from the repo root:

$ python3 -m pytest tests/test_operator_hygiene.py tests/test_ops_alerting.py -q
40 passed

ruff check is clean on every touched file. The full apps/api/tests suite is left to CI.

Related: floomhq/workeros-cloud#1287 (the primary fix: Railway restart policy).

🤖 Generated with Claude Code

[federicodeponte]

CI on ad11b1430: Runtime tests pass on ubuntu-latest and windows-latest, Python lint, MCP tests, Secret scan and Dependency review all pass.

Web lint fails on the Audit production dependencies step (npm advisories in hono / @hono/node-server / brace-expansion / DOMPurify / fast-uri). That step is already red on main at the base commit 1d7b841d9, and this PR touches Python only. Unrelated, not introduced here.

One regression this PR did introduce was caught by CI and fixed in ad11b1430: reading the new heartbeat fields with a from-import broke _health_check_scheduler() against the minimal SimpleNamespace scheduler stub several tests inject, which flipped /health/details to degraded (test_s35_observability.py::test_health_runs_dependency_checks_and_uses_cache). The heartbeat is now read through getattr and never fails the check.