Skip to content

fix: reconcile orphaned in_progress sync_history rows on startup - #89

Open
kilo-WATT wants to merge 1 commit into
Woahai321:mainfrom
kilo-WATT:fix/reconcile-orphaned-sync-status
Open

fix: reconcile orphaned in_progress sync_history rows on startup#89
kilo-WATT wants to merge 1 commit into
Woahai321:mainfrom
kilo-WATT:fix/reconcile-orphaned-sync-status

Conversation

@kilo-WATT

Copy link
Copy Markdown

Fixes #88

Bug

If the core sync process is interrupted mid-sync (container restart, OOM kill, host reboot, Watchtower auto-update, etc.), its sync_history row is never closed out — it stays in_progress=1, end_time=NULL forever, since nothing else updates that row after the process dies.

/api/sync/status/live derives "is a sync currently running" from the most recent sync_history row's in_progress flag rather than checking whether that session's process is actually still alive. So once a row is orphaned this way, the dashboard's Sync Management page shows "Sync in Progress" permanently, with duration_seconds eventually going negative as wall-clock time outpaces the stale start_time. Restarting the container again does not fix it, since the stale row is read from the DB, not held in memory — see #88 for the full repro and a real example payload.

Fix

Adds a reconciliation step to init_database(), which already runs at startup in both api_server.py and list_sync/main.py: any sync_history row still marked in_progress=1 with no end_time gets closed out as 'interrupted' with end_time set to now.

Since a freshly-starting process can't have any sync from a previous process instance genuinely still running, this is safe to do unconditionally rather than trying to detect whether the recorded pid is still alive (which would also be unreliable across container restarts, where PIDs get reused).

Testing

  • python3 -m py_compile clean
  • Reproduced the original bug in a real deployment (container restarted mid-sync a few times while iterating on an unrelated config change), confirmed the dashboard got stuck on "Sync in Progress" with negative duration_seconds
  • Applied the equivalent UPDATE manually against the live DB, confirmed /api/sync/status/live correctly returned to {"is_running": false, "status": "idle", ...} after a restart
  • This PR moves that same fix into init_database() so it happens automatically rather than requiring manual DB surgery

If the core sync process is interrupted mid-sync (container restart,
OOM kill, host reboot, Watchtower auto-update, etc.), its sync_history
row is never closed out — it stays in_progress=1, end_time=NULL
forever, since nothing else updates that row after the process dies.

/api/sync/status/live derives "is a sync currently running" from the
most recent sync_history row's in_progress flag rather than checking
whether the session's process is actually still alive. So once a row
is orphaned this way, the dashboard's Sync Management page shows
"Sync in Progress" permanently, with duration_seconds eventually going
negative as wall-clock time outpaces the stale start_time. Restarting
the container again does not fix it, since the stale row is read from
the DB, not held in memory.

This adds a reconciliation step to init_database(), which already runs
at startup in both api_server.py and list_sync/main.py: any
sync_history row still marked in_progress=1 with no end_time gets
closed out as 'interrupted' with end_time set to now. Since a fresh
process starting up can't have any sync from a previous process
instance genuinely still running, this is safe to do unconditionally
rather than trying to detect whether the recorded pid is still alive.

Fixes Woahai321#88
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sync status stuck at "running" forever if process is interrupted mid-sync

1 participant