feat: PAUSED+PAGED app tier — freeze idle apps, page to swap, wake in ms (#81) by ClaydeCode · Pull Request #145 · FreeshardBase/freeshard

ClaydeCode · 2026-07-07T16:01:50Z

Closes #81 (tracker). Design spec: freeshard-controller docs/superpowers/specs/2026-05-01-app-pause-page-tier-design.md (PR FreeshardBase/freeshard-controller#244).

What

The PAUSED+PAGED app tier: idle apps are frozen (docker compose pause) with their memory proactively paged to swap via cgroup v2 memory.reclaim, instead of cold-stopped. Wake from PAUSED is a compose unpause — measured <2s (integration-pinned) vs 30s+ cold start. Two-tier progression RUNNING → (idle ≥ T1) PAUSED → (idle ≥ T2) STOPPED, plus PSI-driven LRU demotion under memory pressure. Everything behind apps.lifecycle.pause_enabled (default off).

Per the tracker, one bundled PR:

Status.PAUSED + Lifecycle v1.3 (always_on, skip_pause, idle_for_pause, idle_for_stop) with the full validator; AppMeta 1.2→1.3 migrates on read (idle_time_for_shutdown → idle_for_stop), app-repository files untouched. No DB migration (status is a text column).
memory_pressure.py (new): PSI some avg10 from /host/pressure/memory (0.0 when the mount is absent → older core versions degrade gracefully), memory.reclaim writes per container (systemd + cgroupfs layouts), reclaim runs via asyncio.to_thread because the write blocks while the kernel pages out.
app_tools.py: docker_pause_app / docker_unpause_app; frozen-stack handling everywhere a paused app can be stopped/removed — docker_stop_app (unpause→stop), docker_shutdown_app force path (process shutdown), uninstall worker (unpauses while status still says PAUSED), docker_start_app short-circuits PAUSED→unpause.
app_lifecycle.py: two-tier _control_app_time, _demote_lru (one demotion per cycle; excludes always_on, <5s-accessed), wake dispatch in ensure_app_is_running routes PAUSED to unpause directly — also dodges docker_start_app's global 5s throttle, which would silently drop wakes.
Settings: default_idle_for_pause=60, default_idle_for_stop=1800, psi_threshold=10.0, pause_enabled=false under [apps.lifecycle]; env override verified (FREESHARD_APPS__LIFECYCLE__PAUSE_ENABLED=true).
Telemetry: new pause_metrics accumulator module (transitions per app, pause/unpause latency samples, PSI snapshots); send_telemetry packages them into the payload's optional pause_tier field (p50/p95 at send time, swap from /proc/meminfo) — controller-side ingestion is PR FreeshardBase/freeshard-controller#358; backend/telemetry_model.py synced from that branch.

Status-audit (architecture contract: "adding a status? audit all six")

Site	Decision
`docker_start_app` allow-list	PAUSED short-circuits to unpause
`docker_stop_app` allow-list	PAUSED accepted, unfreezes first
`docker_shutdown_app`	unchanged allow-list; unfreezes on force path
worker asserts	unchanged (pause never touches `*_QUEUED`)
`write_traefik_dyn_config` exclusions	unchanged — PAUSED apps stay routed (wake-on-request needs the route)
`control_apps` exclusions	unchanged — PAUSED is lifecycle-managed

Every new status write fires on_apps_update.send_async() (forwardAuth cache invalidation).

Deliberate behavior changes / decisions to review

always_on apps are no longer stopped on low disk — the spec's _control_app_time puts the always_on branch before the disk check; the old loop stopped everything. Flagging since it's a subtle semantic change.
Backout semantics: with pause_enabled=false, already-PAUSED apps are not force-woken by the loop; they wake on next request (the wake dispatch stays status-aware). Matches the spec's backout section.
Legacy stop-only branch now requires status == RUNNING before stopping (previously implicit via docker_stop_app's allow-list). Net behavior identical.
No defensive try/except around pause/reclaim failures, per the spec's risk register — failures propagate and surface.

Tests (all passed locally)

Unit: Lifecycle validator combos; 1.2→1.3 migration incl. every mock_app_store fixture; PSI parsing (malformed/missing); memory.reclaim writes against a fake cgroup tree (both driver layouts); _control_app_time tier decisions (flag on/off, skip_pause, per-app overrides, always_on); _demote_lru selection/exclusion/one-per-cycle; telemetry pause_tier packaging + reset + omission when inactive.
Integration (real Docker, new pause_cycle mock app): full cycle RUNNING→PAUSED→wake→PAUSED→STOPPED with docker-reported container states; recorded unpause latency asserted <2000ms; uninstall-while-PAUSED removes the frozen stack.
Full suite locally: all pass except test_app_starts_and_stops, which cannot run on this dev VM — the quick_stop mock app publishes host port 80, which is occupied by an unrelated local Traefik. Environmental collision (Bind for :::80 failed: port is already allocated), pre-existing on this machine; CI runners are clean and will exercise it.
Not covered by CI (needs root cgroup access / real memory load): actual RSS drop after memory.reclaim, and the PSI memory-hog pressure test — per the spec these are the manual pre-rollout benchmark (Immich/Paperless wake latency, WebSocket long-pause behavior).

Cross-repo / rollout

Blocked-by F-SWAP shipped as controller PR FreeshardBase/freeshard-controller#356 (v23 swap+zswap reconciler); compose mounts stub #246 as PR FreeshardBase/freeshard-controller#357 (v24); telemetry ingestion as PR FreeshardBase/freeshard-controller#358. This PR is safe to merge in any order with those — without the v24 mounts, PSI reads 0.0 and reclaim no-ops with a WARN; without #358 the controller drops the extra telemetry field.
Rollout per spec: merge + release shard_core with flag off → fleet on v24 → flip pause_enabled on internal shards → observe telemetry → staged rollout. Merged ≠ shipped: needs version bump, release, and a controller core-version image bump.

Recommended reading order

shard_core/data_model/app_meta.py + app_meta_migration.py (model)
shard_core/settings.py, config.toml, tests/config.toml
shard_core/service/memory_pressure.py (new leaf)
shard_core/service/app_tools.py (primitives)
shard_core/service/app_lifecycle.py (control loop)
shard_core/service/app_installation/worker.py (uninstall edge)
shard_core/service/pause_metrics.py + telemetry.py + data_model/backend/telemetry_model.py
Tests: test_model.py, test_app_meta_migration.py, test_memory_pressure.py, test_app_lifecycle_pause.py, test_telemetry.py, test_app_pause_lifecycle.py
agents.md

🤖 Generated with Claude Code

The PAUSED+PAGED tier (#81) needs a status between RUNNING and STOPPED, and per-app control over the two idle thresholds. Lifecycle is reworked per the design spec: always_on (unchanged), skip_pause (opt out of the pause tier, e.g. apps that misbehave when frozen), idle_for_pause (T1) and idle_for_stop (T2), both optional with fallback to new global defaults. AppMeta v1.2 -> v1.3 migrates on read: idle_time_for_shutdown carries over as idle_for_stop, so existing app-repository files keep their tuned stop timing without regeneration. No DB migration: installed_apps.status is a text column, only the accepted value set grows. Migration is exercised against every mock_app_store fixture (the app-repository stand-ins), including the always_on and missing-lifecycle shapes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…switch Global defaults for the two idle tiers (T1 pause 60s, T2 stop 1800s), the PSI demotion threshold, and pause_enabled=false as the rollout kill-switch — flipping it back must revert the control loop to legacy stop-only behavior. Prod values are stated explicitly in config.toml for operator visibility; tests get fast tiers (5s/12s) while keeping the flag off so the existing suite exercises the legacy path unchanged. Env override verified: FREESHARD_APPS__LIFECYCLE__PAUSE_ENABLED=true lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

read_memory_pressure() parses `some avg10` from the PSI file the controller mounts at /host/pressure/memory (freeshard-controller#246), returning 0.0 when the mount is absent so shards on older core versions and dev setups degrade to no pressure signal instead of failing. reclaim_compose_stack() writes each container's memory.current into its cgroup v2 memory.reclaim, covering both the systemd and cgroupfs driver layouts. The write blocks while the kernel pages out — potentially seconds for a GB-RSS app — so it runs via asyncio.to_thread. A partial reclaim (EAGAIN) is logged at WARN and not treated as failure, per the design's no-defensive-fallbacks rule: pause still succeeded, only less was paged out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docker_pause_app freezes a RUNNING app via compose pause, pages its memory out through memory_pressure.reclaim_compose_stack, and lands on Status.PAUSED. docker_unpause_app reverses it in milliseconds — the fast wake this tier exists for. The frozen state has one sharp edge, handled at every affected site: a frozen container can be neither stopped nor removed. docker_stop_app accepts PAUSED and unfreezes before compose stop (the PAUSED -> STOPPED tier transition and docker_stop_all_apps); docker_shutdown_app unfreezes on the force path (process shutdown while apps are paused); the uninstall worker unfreezes while the DB status still says PAUSED, before flipping to UNINSTALLING. docker_start_app short-circuits PAUSED into an unpause so every legacy start path wakes a paused app correctly. Status audit per the architecture contract: worker assertions unchanged (pause never touches *_QUEUED), write_traefik_dyn_config keeps routing PAUSED apps (wake-on-request needs the route), control_apps handles PAUSED in the follow-up lifecycle commit. Every new status write fires on_apps_update, keeping the forwardAuth app cache honest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

control_apps gains the second tier: RUNNING pauses after idle >= T1, PAUSED stops after idle >= T2, per-app Lifecycle overrides winning over the global defaults. With pause_enabled=false or a skip_pause app, the branch collapses to the exact legacy stop-only behavior — the backout path costs nothing new. When PSI `some avg10` exceeds psi_threshold, _demote_lru pushes exactly one app (the least recently used; never always_on, never accessed within the 5s grace window) down one tier per cycle. One per cycle is deliberate: the next cycle re-reads PSI, so pressure relief is gradual instead of a stop-flood. ensure_app_is_running dispatches PAUSED wakes straight to docker_unpause_app: keeps the wake off docker_start_app's global 5s throttle (which silently drops calls) and stays active when the flag is off, so already-paused apps still wake after a backout. Behavior change vs the old loop, per the approved design: always_on apps are no longer stopped when disk space runs low — the always_on branch now runs first. Flagged in the PR for review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

pause_metrics is a new dependency-light accumulator module: per-app transition counters, pause/unpause latency samples (recorded by app_tools around the compose calls), and a PSI snapshot per control cycle (recorded by control_apps when the tier is on). It lives outside service.telemetry because app_tools importing telemetry trips the pre-existing signed_call/portal_controller import cycle. send_telemetry packages the accumulators into the Telemetry payload's new optional pause_tier field (p50/p95 computed at send time, swap total/free read from /proc/meminfo) and resets them on success. Without pause activity the field stays None — the wire shape for old controllers is unchanged, and a controller without the matching change simply drops the extra field. telemetry_model.py in data_model/backend/ is synced from the freeshard-controller change (PR freeshard-controller#358) that adds the field and its jsonb storage; equivalent to a `just get-types` run once that PR is merged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ainers New pause_cycle mock app (nginx:alpine, empty lifecycle) rides the global test tiers (T1=5s, T2=12s, 2s refresh). Two api_client tests: - Full cycle: install -> wake -> PAUSED at T1 (docker reports 'paused') -> wake-on-request back to RUNNING -> re-pause -> STOPPED at T2. Pins the compose unpause latency under the 2s design target via the recorded pause_metrics sample (wall time only gets a loose CI-safe ceiling), and checks the transition counters. - Uninstall while PAUSED: the frozen stack must unfreeze on the way out and the containers must actually be gone afterwards. Both passed locally against real Docker (72s). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

max-tet

Always-on apps should still stop on low disk. We need to prevent apps from completely filling the disk.

max-tet · 2026-07-08T19:47:09Z

+    idle_for_pause: Optional[int] = None  # None = use global default
+    idle_for_stop: Optional[int] = None  # None = use global default


Use int | None if possible.

Fixed in 41ecc27 — both fields now int | None.

max-tet · 2026-07-08T19:51:00Z

 class AppLifecycleSettings(BaseModel):
    refresh_interval: int = 30
+    default_idle_for_pause: int = 60  # T1: RUNNING -> PAUSED, seconds
+    default_idle_for_stop: int = 1800  # T2: PAUSED -> STOPPED, seconds


Should be longer, few hours. Will stop with memory pressure anyway.

Set to 10800 (3h) in settings and config.toml (41ecc27). Pressure demotion covers the RAM-needed case earlier.

max-tet · 2026-07-08T20:12:25Z

+        await docker_stop_app(app.name)
+
+
+async def _demote_lru(apps: List[InstalledApp]):


paused apps will mostly have a less recent last access, so this favours stopping over pausing. Since pausing is the thing with less user impact, it should be preferred. Consider running apps first and pause one, only if no candidates exist, go to paused apps and stop one.

Reworked in 41ecc27: candidates split by status — LRU among RUNNING is paused first (or stopped for skip_pause apps); a PAUSED app is only stopped when nothing is left to pause. Tests pin both preferences, including the case where the paused app is older.

max-tet · 2026-07-09T05:23:43Z

    no_of_requests += 1


+def _percentile(samples: List[float], fraction: float) -> float:


guard the allowed values for fraction. Easy to pass 95 instead of 0.95.

Guarded in 41ecc27 — fraction outside [0, 1] raises ValueError; test covers 95 and -0.1.

… demotion, longer T2, misc Review feedback from max-tet on PR #145, all four inline items plus the top-level change request: - Low disk stops every app again, always_on included — the disk check moves back in front of the always_on branch. Preventing a full disk beats keeping an always_on app up; the app restarts once disk recovers. (Reverts the spec-sketch ordering this PR had introduced.) - _demote_lru prefers pausing a RUNNING app over stopping a PAUSED one: paused apps mostly have older last-access times, so a single LRU sort across both tiers systematically stopped paused apps — the higher-impact action. Now: LRU among RUNNING first, PAUSED only when nothing is left to pause. - default_idle_for_stop 1800 -> 10800 (3h) in settings and config.toml. A paused app costs little RAM, and pressure demotion stops it earlier when the RAM is actually needed. - Lifecycle idle fields use int | None instead of Optional[int]. - _percentile rejects fractions outside [0, 1] — passing 95 instead of 0.95 now raises instead of silently indexing nonsense. New tests pin the disk-stop-for-always_on behavior, both demotion preferences, and the fraction guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ClaydeCode · 2026-07-09T05:35:56Z

All five review items addressed in 41ecc27:

always_on + low disk: disk check restored in front of the always_on branch — low disk stops every app again; the app comes back once disk recovers. New test pins it.
_demote_lru: pause-first — LRU among RUNNING demoted before any PAUSED app is stopped (details in the thread).
default_idle_for_stop: 1800 → 10800 (3h), settings + config.toml.
int | None for the Lifecycle idle fields.
_percentile fraction guarded to [0, 1].

Ran locally: pause unit + integration suites and telemetry tests, 24 passed (integration on real Docker). CI running on the push.

ClaydeCode and others added 9 commits July 7, 2026 15:23

docs: describe the pause tier in agents.md

0028bc5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

style: black formatting

8a4cb16

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ClaydeCode requested a review from max-tet July 7, 2026 16:01

ClaydeCode mentioned this pull request Jul 7, 2026

PAUSED+PAGED app tier (tracker) #81

Open

max-tet requested changes Jul 9, 2026

View reviewed changes

ClaydeCode requested a review from max-tet July 9, 2026 05:35

max-tet approved these changes Jul 9, 2026

View reviewed changes

ClaydeCode mentioned this pull request Jul 9, 2026

Show PAUSED app state in the app detail modal FreeshardBase/web-terminal#33

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: PAUSED+PAGED app tier — freeze idle apps, page to swap, wake in ms (#81)#145

feat: PAUSED+PAGED app tier — freeze idle apps, page to swap, wake in ms (#81)#145
ClaydeCode wants to merge 10 commits into
mainfrom
feature/clayde/paused-paged-tier

ClaydeCode commented Jul 7, 2026 •

edited

Loading

Uh oh!

max-tet left a comment

Uh oh!

max-tet Jul 8, 2026

Uh oh!

ClaydeCode Jul 9, 2026

Uh oh!

max-tet Jul 8, 2026

Uh oh!

ClaydeCode Jul 9, 2026

Uh oh!

max-tet Jul 8, 2026

Uh oh!

ClaydeCode Jul 9, 2026

Uh oh!

max-tet Jul 9, 2026

Uh oh!

ClaydeCode Jul 9, 2026

Uh oh!

ClaydeCode commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		idle_for_pause: Optional[int] = None # None = use global default
		idle_for_stop: Optional[int] = None # None = use global default

		await docker_stop_app(app.name)


		async def _demote_lru(apps: List[InstalledApp]):

		no_of_requests += 1


		def _percentile(samples: List[float], fraction: float) -> float:

Uh oh!

Conversation

ClaydeCode commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Status-audit (architecture contract: "adding a status? audit all six")

Deliberate behavior changes / decisions to review

Tests (all passed locally)

Cross-repo / rollout

Recommended reading order

Uh oh!

max-tet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ClaydeCode commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ClaydeCode commented Jul 7, 2026 •

edited

Loading