bug: Podman drops CAP_KILL and cannot gracefully stop cross-UID workloads

[krishicks]

User Story

As an OpenShell user running Podman-backed sandboxes, I want sandbox stop and delete operations to terminate the managed workload gracefully, so that lifecycle operations complete promptly without waiting for Podman's SIGKILL timeout.

Problem Statement

The Podman driver explicitly drops CAP_KILL from the container running openshell-sandbox as PID 1. The supervisor runs as UID 0, while the managed workload runs as the sandbox image or policy UID (for example UID 10001).

On SIGTERM, the supervisor attempts to forward SIGTERM to the differently-owned workload and then waits for it to exit. Linux rejects that signal with EPERM because PID 1 lacks CAP_KILL. The supervisor waits indefinitely, so Podman consumes the complete configured stop grace period (45 seconds by default) and then kills the container.

The contradictory source paths are:

• crates/openshell-driver-podman/src/container.rs: explicitly drops KILL with the comment that the supervisor does not send signals to arbitrary processes.
• crates/openshell-supervisor-process/src/run.rs: handles supervisor SIGTERM by signaling the managed entrypoint and awaiting its exit.

Impact / Why This Matters

• Podman sandbox stop and delete operations routinely take the full 45-second timeout for images whose workload UID differs from PID 1.
• Graceful workload cleanup does not run; Podman must use SIGKILL.
• Slow deletes can amplify lifecycle races and block unrelated operations. Related issue fix(server): prevent unrelated sandbox deletes from blocking deletion events #2326 explicitly left PID 1 signal forwarding out of scope for a separate product issue.
• The current workaround is reducing OPENSHELL_STOP_TIMEOUT or tolerating SIGKILL. A shorter timeout only reduces the delay; it does not restore graceful termination and can cause data loss or incomplete cleanup.

Acceptance Criteria

• A Podman sandbox whose supervisor runs as UID 0 and whose managed entrypoint runs as a different non-root UID exits promptly after SIGTERM.
• The supervisor can signal its managed entrypoint without logging failed to signal entrypoint process / EPERM.
• Podman does not need to wait for the stop timeout and send SIGKILL during a normal graceful stop.
• Regression coverage exercises a cross-UID workload and verifies prompt SIGTERM forwarding.
• The fix preserves least privilege and does not grant signal authority beyond the container/process namespace required for managed child lifecycle.

Reproduction Steps

The following standalone Podman reproduction models the OpenShell PID 1 behavior without requiring a gateway. PID 1 runs as root, starts a UID 10001 child, catches SIGTERM, forwards it to that child, and waits.

podman run --detach --name cap-kill-repro \
  --user 0:0 \
  --cap-drop KILL \
  ubuntu:24.04 \
  bash -c 'setpriv --reuid=10001 --regid=10001 --clear-groups sleep infinity & child=$!; trap "kill -TERM $child; wait $child" TERM; wait $child'

time podman stop --time 2 cap-kill-repro
podman logs cap-kill-repro
podman rm cap-kill-repro

Observed:

real 2.10
/bin/bash: line 1: kill: (2) - Operation not permitted

Control: repeat without --cap-drop KILL.

real 0.17

The same permission failure is observable in an OpenShell Podman sandbox:

cid=$(podman ps -q --filter label=openshell.ai/sandbox-name=openshell-dev)
podman top "$cid" pid,ppid,user,args
podman exec --user 0 "$cid" sh -c 'kill -0 <workload-pid>'

Observed process layout and result:

PID  PPID  USER     COMMAND
1    0     root     /opt/openshell/bin/openshell-sandbox --workdir /sandbox
41   1     sandbox  sleep infinity

kill: Operation not permitted

Stopping a prior affected sandbox emitted:

WARN openshell_supervisor_process::run: failed to signal entrypoint process

Environment

• OpenShell CLI: 0.0.83
• OpenShell checkout: 3c98683f
• Host: macOS, Podman machine using libkrun
• Podman: 6.0.2, rootless remote socket
• OCI runtime: crun 1.28-dirty
• Container stop signal: SIGTERM
• Container stop timeout: 45 seconds
• Supervisor UID: 0
• Managed workload UID: 10001

Agent Investigation

History

1. PR Openshell driver podman #904 introduced the Podman driver with cap_drop: ALL and a selective cap_add list that did not include CAP_KILL. At that point the capability was already unavailable.
2. PR Two podman driver fixes #1077 replaced cap_drop: ALL with an explicit drop list. It continued to drop KILL, documenting the assumption that the supervisor did not send signals to arbitrary processes.
3. PR fix(podman): tolerate shutdown transport closes #2498 added SIGTERM handling and forwarding in the process supervisor and increased the Podman stop timeout from 10 to 45 seconds, but did not revisit the Podman capability list. This made the stale capability assumption directly conflict with the shutdown path.

Relevant commits:

• d44d8a1e2716de855bd299079d1eb5a5b476874b / PR Openshell driver podman #904: initial Podman driver
• b77b60b30434e2e6cb0229949482ac18a9414af3 / PR Two podman driver fixes #1077: explicit capability drop list
• f00ad23a2603d7935a5ac11bee3389fd57a2eb41 / PR fix(podman): tolerate shutdown transport closes #2498: supervisor shutdown signal forwarding

Driver comparison

This appears specific to the Podman driver configuration:

• Podman: runs the combined supervisor as UID 0, the workload as a different UID, and explicitly drops KILL. Reproduced live.
• Docker: runs the combined supervisor as UID 0 and adds required capabilities without setting cap_drop, so Docker's default capability set retains KILL.
• Kubernetes combined topology: adds required capabilities without dropping the default set, retaining KILL for cross-UID supervision.
• Kubernetes sidecar topology: the process supervisor and workload run under the same sandbox UID; the process container drops all capabilities but does not need CAP_KILL to signal a same-UID child.

Issue #2844 also involves Podman SIGTERM falling back to SIGKILL, but it is a distinct CI runner AppArmor denial involving pasta, not supervisor-to-workload signal permissions inside the sandbox container.