Skip to content

[codex] Add advisor watchdog for stale task-output waits - #3461

Draft
morganmcg1 wants to merge 1 commit into
mainfrom
codex/generic-advisor-watchdog
Draft

[codex] Add advisor watchdog for stale task-output waits#3461
morganmcg1 wants to merge 1 commit into
mainfrom
codex/generic-advisor-watchdog

Conversation

@morganmcg1

@morganmcg1 morganmcg1 commented May 24, 2026

Copy link
Copy Markdown
Member

Summary

Adds a narrowly scoped advisor watchdog path for stale Claude task-output waiters. This covers both observed DL24 stall variants without matching one literal sentinel string and without adding new launch/config surface.

What changed

  • Detects Claude task-output wait loops scoped to /tmp/claude-.../tasks/*.output that use until, grep -q, sleep, and done.
  • Intervenes only when the referenced output file has stopped changing for the existing script-level threshold and the waiter has no real worker descendants left.
  • Kills just the stale waiter subtree so the outer advisor loop can recover and re-poll.
  • Adds shell regression coverage for the original ERROR|=== waiter, the later custom grep waiter, and the safety case where a live Python worker is still running.

Why

The DL24 advisor pod was kube-healthy but the advisor loop wedged behind stale Bash waiters on Claude task output files. The first stall waited for the ERROR|=== sentinel; the later recurrence waited for a custom grep sentinel (no val yet|... state|... error). In both cases the task output file was stale and no real worker process remained below the waiter.

The fix keeps the boundary narrow: it still requires a Claude task-output path, a polling grep/sleep waiter, stale file mtime, and no real work below the waiter. It just stops depending on one exact sentinel string.

Validation

  • bash -n k8s/advisor-claude-watchdog.sh tests/test_advisor_claude_watchdog.sh
  • bash tests/test_advisor_claude_watchdog.sh

@github-actions

Copy link
Copy Markdown


Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

@morganmcg1
morganmcg1 force-pushed the codex/generic-advisor-watchdog branch from 50cec12 to 221f91e Compare May 24, 2026 08:34
@morganmcg1
morganmcg1 force-pushed the codex/generic-advisor-watchdog branch from 221f91e to 1a05ace Compare May 25, 2026 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants