The Ateles T3 daemons (Apis, Formica, neotoma-agent) must run from a
clean, stable checkout — never from the development checkout (~/repos/ateles),
whose branch and working tree churn during day-to-day work. Otherwise the
running swarm silently executes whatever code happens to be checked out for
dev, including half-finished feature branches.
This mirrors Neotoma's ~/neotoma-rc-src + com.neotoma.rc-autodeploy pattern.
| Piece | Location | Role |
|---|---|---|
| Deploy checkout | ~/ateles-rc-src |
Clean clone pinned to origin/main; the only tree the daemons run from. |
| Daemon venv | ~/ateles-rc-src/.venv |
Provisioned from execution/scripts/daemon-requirements.txt. |
| Redeploy script | execution/scripts/redeploy_daemons_from_main.sh |
Fast-forwards the deploy checkout to origin/main, refreshes the venv when deps change, hard-restarts the three daemons. |
| Autodeploy agent | com.ateles.rc-autodeploy |
Runs the redeploy script every 120s. |
| Installer | execution/scripts/install_rc_autodeploy.sh |
Provisions the checkout, venv, and autodeploy agent. |
merge to main ──> (≤120s) com.ateles.rc-autodeploy fires
──> redeploy_daemons_from_main.sh
├─ git fetch origin main; exit if already current
├─ defer (exit 0, no change) while a Fly job is in flight
├─ abort if deploy HEAD diverged (non-ff)
├─ git merge --ff-only origin/main
├─ refresh .venv iff daemon-requirements.txt changed
└─ launchctl kickstart -k each daemon (fresh module import)
The redeploy script is idempotent (no-ops when already at origin/main),
single-flighted via an atomic lockdir, and fail-soft (daemons stay on the prior
build on any error).
Deferral while a Fly job runs. Apis awaits every Fly job it starts, so a hard
restart would orphan the job and leave its task executing. Before fast-forwarding,
the script starts execution/scripts/fly_jobs_in_flight.py --hold, which takes the
Fly runner's own admission lock (<ledger>.admission.lock, the cross-process
flock every reservation is made under), then counts Fly runner ledger
reservations (APIS_FLY_LEDGER_FILE, default
~/.config/ateles/fly-runner-ledger.json) that are still live. A reservation is
live until 300 seconds after its deadline, because a run past its deadline may
still be collecting its last answer and settling its result; unconfirmed
reservations count on the same rule. If any are live, the script logs
DEFERRED: <n> Fly job(s) in flight and exits 0 with nothing changed; the checkout
stays behind, so the next 120s run retries. If none are, the script keeps holding
the lock through the fast-forward, any venv refresh and the restarts, so no new job
can be reserved between the check and the restart. A launch that tries meanwhile
waits for the lock. One waiting in another process resumes once the lock is
released; one waiting inside an Apis that is being restarted dies with it before it
reserves or starts a Fly machine, so no machine is orphaned, but its task and any
GitHub token it already minted (the launch mints before it waits for admission) are
left as any restart leaves an in-progress dispatch. The lock dies with the holder, which
also exits if the script dies. After 15 minutes at most the holder ends the deploy
before it lets go: it sends SIGTERM to the script (its own parent, and only while it
is still its parent), waits up to 30 seconds for it to exit (then SIGKILL), waits a
further 5 seconds (a fixed allowance for a launchctl kickstart already started,
not proof that it has finished), and only then releases the lock. Its signalling
re-checks that the pid is still its parent immediately before each signal; the
check and the signal are separate operations, so this narrows, but does not
eliminate, the window in which a reused pid could be signalled. A deploy that has
been ended cannot restart anything, so no restart follows the release when the
holder reaches its own limit, and a hung deploy still cannot block Fly admission
for ever. That ordering guarantee covers the holder's own limit only. Before each
launchctl kickstart, in both the gui and the user domain attempt, the script
also checks the holder is still alive; if it is gone the script logs
Fly admission holder is gone, restarts nothing more and exits non-zero. This is
best effort, for a holder killed from outside (at its own limit the holder ends the
deploy first): the check and the kickstart are two operations, so a restart can
still race a holder that dies between them. If the lock cannot be
taken within 60 seconds (admission or the reaper is at work), the run logs
DEFERRED: Fly admission lock busy and exits 0. The check fails open: if the
holder cannot start or answer, the script logs one fixed warning and deploys, and a
missing or unreadable ledger, or an entry with no readable deadline (including one
too large to convert), never holds a deploy; a corrupt entry is skipped without
hiding a live one.
Accepted limits of a deploy that is ended early:
- A deploy ended after the fast-forward (the holder's 15-minute limit during a long
dependency refresh, a failed
pip install, or a holder that is gone partway through the restarts) leaves the checkout updated but the restarts incomplete, so daemons may be on mixed builds: the next run sees the checkout at origin and does nothing. The script logsDEPLOY DID NOT FINISHsaying so; the next merge to main, or a manual restart, picks up the new build. This is how a failedpip installalready behaved. - If the holder ends a deploy while
pipis running, thatpipchild may outlive the script and finish its install. It restarts nothing.
What remains: consecutive Fly jobs can still defer a deploy for as long as they keep overlapping. This change stops admission only from the check to the restart; it does not pause admission ahead of a pending deploy so that running jobs can drain. That limit is known and left for a later change.
The per-daemon plists (com.ateles.{apis,formica,neotoma-agent,cotinga,cyphorhinus,piculet,sylvia,phoenicurus-prepare}) are
machine-local (gitignored) and must:
- Point
ProgramArgumentsat~/ateles-rc-src/.venv/bin/python3and the daemon script under~/ateles-rc-src/execution/daemons/.... - Set
ATELES_PRIVATE_KEYS_DIRto the operator's real~/repos/ateles-private/keys. The deploy checkout has no siblingateles-privateoverlay, andaauth_signer.pyotherwise resolves the keys dir relative to the checkout — so without this override the daemons fall back to stub signers and attribute observations to the operator token. - Carry the daemon's
NEOTOMA_BASE_URL, SSE subscription id, and (Apis) claude binary path, as before.
Deploy-bound daemons (cotinga, cyphorhinus, piculet, sylvia,
phoenicurus-prepare) refuse to start unless their git checkout root is the
documented deploy tree. This is checkout identity ("right tree?") — distinct
from checkout drift ("current?") in lib/daemon_runtime/checkout_drift.py.
Incidents: ateles#339, #361, #412, #515.
Startup enforcement is fail-closed for those five daemons. There is no
production bypass env. Tests may set ATELES_CHECKOUT_IDENTITY_ROOT (actual
root override) or ATELES_DEPLOY_CHECKOUT (expected root; default
~/ateles-rc-src).
| Code | Condition | FATAL line 1 prefix |
|---|---|---|
78 |
Wrong tree (actual ≠ expected) |
FATAL: wrong checkout — refusing to start. |
79 |
Cannot verify (no .git, unreadable HEAD, …) |
FATAL: cannot determine checkout identity |
80 |
Expected deploy root path does not exist | FATAL: deploy checkout not provisioned |
81 |
Unexpected I/O/permission during check | FATAL: checkout identity check failed |
FATAL: wrong checkout — refusing to start.
daemon: phoenicurus-prepare
running from: /Users/markmhendrickson/repos/ateles (branch: feature/example)
expected: /Users/markmhendrickson/ateles-rc-src (must track origin/main)
plist: com.ateles.phoenicurus-prepare
This daemon drives releases. Running from a session clone risks silent
wrong-code execution (see ateles#339, #361, #412, #515).
Fix:
bash ~/ateles-rc-src/execution/scripts/isolate_daemons_to_rc_src.sh --apply
Docs: docs/daemon_rc_autodeploy.md
execution/scripts/isolate_daemons_to_rc_src.sh --apply inventories daemon-local
state in both trees, snapshots hashed rollback copies under
~/.config/ateles/daemon-state-backups/, merges losslessly into the RC tree
(or refuses), rewrites/reloads only the five labels above, then proves a
new PID and executable/script paths under ~/ateles-rc-src. Any failure or
unexpected exception after the state snapshot restores state (including files
that were absent before reconciliation),
restores all five prior plist configurations, reloads the prior fleet, and does
not claim cutover complete. Both rollback sets are retained. The shared session
clone is left untouched until all five pass. XDG relocation is out of scope.
An installed plist symlink into the shared checkout is snapshotted with its
exact link target, repointed to the corresponding RC plist without becoming a
regular file, and restored to its original target during automatic rollback.
The RC target must exist as a readable, regular plist and pass plutil -lint
before state reconciliation or link replacement. Broken links, links outside
the shared checkout, unreadable targets, and other non-regular inputs abort
before state reconciliation; only a genuinely missing plist is recorded as
absent. This preserves path identity as well as bytes.
Credential boundary: the deploy checkout holds code only. Secrets stay in
~/.config/neotoma/.env, ateles-private/keys, and plist EnvironmentVariables
— never copied into ~/ateles-rc-src.
~/repos/ateles/execution/scripts/install_rc_autodeploy.sh
# then repoint the per-daemon plists per "Per-daemon plist requirements" above
# for the five wrong-tree daemons (or any still on ~/repos/ateles):
bash ~/ateles-rc-src/execution/scripts/isolate_daemons_to_rc_src.sh --apply