[kuanlinhuang]

Problem

Rolling out to prod, large AD-Workbench workspaces freeze agent bootstrap — the agent never starts ("Agent could not start" / "could not reach server"), with no useful logs. It works with a few files; it's fatal at scale.

Root cause

on_chat_start builds a file inventory for the sidebar + agent system prompt. On a large workspace on a slow network mount (Azure Files / blobfuse — every stat is a round-trip) this was catastrophic:

1. Unbounded traversal. The tree was walked with recursive os.walk; the max_items "preview" caps only limited what was displayed, so the walk still stat-ed every file to compute totals.
2. ~4 walks per chat start. Inventory + sidebar overview + tree + system-prompt build each walked the same tree independently, with no caching.
3. On the event loop. _build_user_data_sidebar_elements() ran synchronously in on_chat_start, blocking the asyncio event loop for the whole walk → /healthz could not be served → liveness (period 20s × failureThreshold 3) failed → Kubernetes crash-looped the pod.

Fix

New biomni/fs_scan.py: one bounded, cached, thread-safe directory scanner; every workspace walk routes through it.

• Bounded by both a file-count cap (BIOMNI_WORKSPACE_MAX_FILES, default 10000) and a wall-clock deadline (BIOMNI_WORKSPACE_SCAN_TIMEOUT_S, default 15s) — a scan can never hang.
• TTL-cached (BIOMNI_WORKSPACE_SCAN_TTL_S, default 60s) behind a per-key lock → the ~4 builders of one chat start traverse the tree once.
• Off the event loop: the sidebar build now runs in run_in_executor and is guarded (a scan hiccup degrades to no panel, never a dead session). cl.Text construction in the worker is safe — run_in_executor copies the Chainlit ws context and replays it via ctx.run.
• Chainlit inventory/sidebar/overview/welcome and agent _get_data_root_items / _get_user_data_resources / _get_data_lake_items all go through the scanner — no raw unbounded os.walk on a user/network path remains on the boot path.
• Honest UX: bounded scans show counts as a floor (10000+, "listing bounded"); a timed-out scan reads "listing unavailable" (not "no files found"); the agent is told to enumerate on demand.

Ops can tune the three env knobs without a redeploy (e.g. raise caps for a large-but-fast mount). The anti-hang guarantee holds regardless.

Verification

Docker (real container, 50k-file workspace mounted read-only):

Check	Result
Boot + /healthz	200 in ~6s; Restarts=0, Health=healthy (crash-loop gone)
File-cap bound	50k → capped at 10,000, truncated, 28 ms
Wall-clock deadline	10 ms budget → aborted at 1,805 files, timed_out
Cache	2nd scan of same root = 0.0 ms
Previously-blocking on_chat_start fns	all return ≤0.3 ms
runs/ / hidden / excluded / depth pruning	correct

Tests: tests/test_fs_scan.py — cap & deadline bounds, cache identity, depth + exclusion pruning, env fallbacks (16 cases). Full suite collects clean (224).

Code review: high-effort multi-agent review; findings addressed in this PR — bounded _get_data_lake_items (a parallel hang vector on the boot path), timed-out-vs-empty messaging, runs/ exclusion, non-positive env fallback, and consistent floor rendering across all UI surfaces.

Note on base branch

Targets feat/adworkbench (the repo default), not biomni-ad: this change imports biomni.observability / biomni.health and edits post-divergence code that exist only on feat/adworkbench (biomni-ad is 58 commits behind and lacks them). Happy to retarget if the intended flow differs.

Follow-ups (out of scope)

• biomni/agent/ad1.py still has raw os.walk/os.listdir in the legacy Gradio path (not the Chainlit prod path) — route through fs_scan in a follow-up.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.