Skip to content

fix(framework): Restart AgentApps after dependency installation - #8427

Open
smoroso wants to merge 9 commits into
mainfrom
codex/fix-agentapp-dependency-imports
Open

smoroso wants to merge 9 commits into
mainfrom
codex/fix-agentapp-dependency-imports

Conversation

@smoroso

@smoroso smoroso commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

What

Restart cold AgentApp processes after runtime dependency installation, with the app environment on the import path from Python startup. Preserve the already-claimed task input in a private file, restore it once after restart, and consume the chat prompt only afterward. For stdin-authenticated launches, hand the resolved token to the existing token-file CLI path using a private file. The restart does not put the token in argv or repeat the startup acknowledgement. A host-only supervisor now starts the app interpreter, forwards termination, and reports a failed task if incompatible app dependencies prevent startup. The restart handoff is authenticated and restricted to the configured runtime root and claimed run before cleanup ownership is accepted. Preloaded workers keep their existing path.

Why

New Chat can fail with cannot import name 'Converter' from 'attr'. The runner has already cached an older attr module when the app environment supplies a newer attrs package. Changing sys.path cannot replace those cached imports. A fresh interpreter avoids mixing the two versions without pinning the app's dependencies or unloading live modules.

How to manually test

With the framework development environment installed, run this exact block from the repository root:

set -eu
cd framework
export PYTHONPATH="$PWD/py"
export RAY_ENABLE_UV_RUN_RUNTIME_ENV=0
uv run --no-sync --python=3.11.14 python -m pytest --override-ini addopts= -q py/flwr/supercore/task_process/agent py/flwr/supercore/superexec/dependency_installer_test.py py/flwr/supercore/superexec/executor/subprocess_executor_test.py py/flwr/supercore/cli/flwr_agentapp_test.py py/flwr/common/args_test.py

Expected: all tests pass. The subprocess regression first reproduces a cached dependency's missing Converter import, then checks fresh imports after restart, restoration without a second task-input claim, private input-file permissions, and exactly-once finalization and cleanup. A separate test checks that restart happens before consuming the initial prompt. The stdin-authenticated CLI subprocess regression checks token recovery through a private file, no token in argv or captured output, exactly one startup acknowledgement, one-time input restoration, and cleanup. New regressions cover real HTTP failure reporting when an app shadows httpx, rejection of invalid restart paths and handoffs without deleting unrelated files, and forwarding of SIGINT, SIGTERM, and SIGQUIT where available. The signal test covers both PID- and group-directed delivery, with the child in a separate POSIX session so it can finalize without duplicate delivery.

Validation

  • Synced with latest main. The exact manual block passed: 102 runtime, CLI, and token-parsing tests under Python 3.11.14.
  • Type checking, Ruff, Pylint, Black, and isort passed for all three changed files, along with the copyright check.
  • Re-ran the disposable Python 3.13 reproduction with real attrs/jsonschema packages. It reproduced the original import error, then passed imports, schema validation, task-input restoration, and cleanup after restart.
  • The title is valid. Its failed CI attempt stopped during the external uv setup download; rerunning that job passed. CI is running on the updated head.
  • A full New Chat conversation has not been tested against a rebuilt Labs executor. Labs must consume this Flower change and rebuild its executor before that check. Existing local services were not modified.

Copilot AI balanced review requested due to automatic review settings October 28, 2026 09:31
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-28T18:14:49.167913Z 2a79921 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Cold AgentApp restarts with --token-stdin fail because stdin has already been consumed and closed, causing the CLI to exit before restoring the claimed task or registering cleanup.

1 open issue
What changed in this PR

This PR restarts cold AgentApps after runtime dependency installation so app dependencies load in a fresh interpreter, avoiding conflicts with cached modules. Preloaded workers retain their existing path.

Changes:

  • Preserve claimed task input in a private file and restore it once after restart.
  • Prepare dependencies and restart before consuming the initial chat prompt.
  • Add regression tests for fresh imports, input restoration, cleanup, and prompt ordering.

The changes address one cohesive runtime issue; no PR split is needed.

File Description
framework/​py/​flwr/​supercore/​task_process/​agent/​run_agentapp.py Adds interpreter restart and task-input restoration before prompt consumption.
framework/​py/​flwr/​supercore/​task_process/​agent/​run_agentapp_test.py Tests dependency isolation, private input handoff, cleanup, and restart ordering.

Review effort: Balanced

Comment thread framework/py/flwr/supercore/task_process/agent/run_agentapp.py Outdated
@github-actions github-actions Bot added the Maintainer Used to determine what PRs (mainly) come from Flower maintainers. label Oct 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The restart via os.execve discards the running log uploader and undrained queue, losing pending Runtime logs; the queue must be flushed before restarting without finalizing the claimed task.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread framework/py/flwr/supercore/task_process/agent/run_agentapp.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a71590a4b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread framework/py/flwr/supercore/task_process/agent/run_agentapp.py Outdated
Comment thread framework/py/flwr/supercore/task_process/agent/run_agentapp.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 58ff633861

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread framework/py/flwr/supercore/task_process/agent/restart_agentapp.py

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The child shares the supervisor’s POSIX process group, allowing duplicate signal delivery to terminate AgentApp before task finalization and runtime cleanup.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread framework/py/flwr/supercore/task_process/agent/restart_agentapp.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because a critical POSIX cancellation race can kill the supervisor while leaving AgentApp running without supervisor recovery.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Maintainer Used to determine what PRs (mainly) come from Flower maintainers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants