Skip to content

Let systemd-oomd kill a runaway app instead of losing the session - #6445

Open
dhh wants to merge 1 commit into
quattrofrom
oomd-protect-session
Open

Let systemd-oomd kill a runaway app instead of losing the session#6445
dhh wants to merge 1 commit into
quattrofrom
oomd-protect-session

Conversation

@dhh

@dhh dhh commented Jul 30, 2026

Copy link
Copy Markdown
Member

Why

Post-mortem of a real crash: a 62 GB box ran out of RAM, no OOM daemon intervened, and the kernel OOM killer never fired (it acts only after an allocation fails outright, and the machine was thrashing in reclaim instead). A Chromium renderer died on a fatal allocation CHECK, its wl_shm pool was torn down mid-frame, and Hyprland took SIGBUS copying from the unbacked page — the whole session gone because one app misbehaved.

What

  • etc/systemd/oomd.conf.d/10-omarchy.conf — thresholds: act when half of a 10s window is spent stalled on reclaim, sustained for 20s (Fedora's shipped desktop defaults). systemd-oomd keys on PSI stall time, so it fires while the machine thrashes, not after an allocation fails.
  • default/systemd/user/app.slice.d/10-oomd.conf — kill candidacy on app.slice and only app.slice. Hyprland runs in session.slice, so the compositor is structurally ineligible as a victim: oomd takes the browser/terminal that caused the pressure and the session survives to show the notification. Ships as a vendor drop-in under /usr/lib/systemd/user/ so existing users get it on package upgrade with no per-user seeding, and ~/.config/systemd/user/app.slice.d/ stays free as an override layer.
  • install/config/enable-services.sh — new installs enable the daemon (socket comes along via Also=).
  • migrations/1785424256.sh — existing installs enable it; if a user had oomd enabled already, it's restarted so it doesn't keep pre-package thresholds until reboot. Then a user-manager daemon-reload so app.slice candidacy is reported without a relogin.
  • test/shell.d/systemd-test.sh — the load-bearing assertion is the negative one: the app.slice drop-in must be the only ManagedOOM*=kill in the tree. Setting candidacy on user@.service instead (what Fedora does) puts the compositor back in the victim pool — the exact crash this prevents.

Deliberately not earlyoom: it triggers only when free RAM and free swap are both under threshold. The crashed box had 73 GB of untouched swap, so earlyoom would have slept through the entire failure — Omarchy's large hibernation swapfiles make that its steady state.

Kill granularity

oomd kills a whole cgroup, so the victim is an entire app-*.scope — the whole browser (session-restorable), or the whole terminal scope including a tmux server inside it. "Lose one app" rather than "lose one tab," but categorically better than losing the compositor. ManagedOOMPreference=avoid exists per-unit if this ever needs tuning; no preferences shipped so the candidate pool can't be whittled down to nothing.

Verified

  • Installed both files to their real target paths on a live install and restarted oomd: systemctl --user show app.slice resolves the /usr/lib drop-in, and oomctl lists app.slice under both monitored sections with 50%/20s.
  • ./test/shell (three new assertions) and ./test/cli pass. One pre-existing failure (bar icons share slot and baseline geometry) fails identically on a clean tree.
  • Migration exits 0 on rerun in both branches (0644, no shebang, idempotent).

Needs a companion change

omarchy-settings PKGBUILD in omarchy-pkgs needs to install the new vendor file — the existing user-unit glob won't pick up a subdirectory:

install -Dm644 default/systemd/user/app.slice.d/10-oomd.conf \
  "$pkgdir/usr/lib/systemd/user/app.slice.d/10-oomd.conf"

(And a check that etc/systemd/oomd.conf.d/ rides along with however etc/** is installed.)

🤖 Generated with Claude Code

Nothing stood between "memory is tight" and "processes die at random":
the kernel OOM killer only fires after an allocation has already failed,
so a machine under pressure thrashes in reclaim until something breaks
somewhere fatal — seen in practice as Hyprland taking SIGBUS mid-memcpy
from a client shm pool that memory pressure was tearing down.

Enable systemd-oomd, which keys on PSI stall time rather than free
pages, and mark app.slice — and only app.slice — as its kill candidate.
Hyprland runs in session.slice, so the compositor is structurally
ineligible as a victim: oomd takes the app that caused the pressure and
the session survives. Thresholds (50% stall over 20s, Fedora's desktop
defaults) live in an oomd.conf.d drop-in; candidacy ships as a vendor
drop-in under /usr/lib/systemd/user so existing users get it on package
upgrade with no per-user seeding.

New installs enable the daemon from enable-services.sh; a migration
covers existing ones, restarting an already-running oomd so it doesn't
keep stale thresholds until reboot.

Deliberately not earlyoom: it triggers only when free RAM and free swap
are both under threshold, so Omarchy's large, mostly-idle swapfiles
keep it asleep through exactly this failure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 30, 2026 17:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants