f2903ce still enters 100% CPU busy loop during ReleaseCheck

[swesty]

Summary

pop-upgrade daemon enters a persistent 100%-of-one-core userspace busy loop during a release check on Pop!_OS 22.04, despite running commit f2903ce, which was intended to fix the metadata-fetch infinite retry / 100% CPU issue.

The daemon remained in this state for multiple days. A one-minute watcher recorded 3,791 consecutive samples at approximately 98–101% CPU. APT/DPKG were idle and held no locks.

Environment

• OS: Pop!_OS 22.04
• pop-upgrade: 1.0.0~1778861967~22.04~f2903ce (amd64)
• Installed binary build ID: f549629fb753e75a2c415efd24b915ad2fba5bf4
• Upstream commit: f2903ce2631c18d721530ee263aff6e31c5951b1
• Daemon PID in capture: 1233442

Trigger and last daemon messages

The busy loop began after a routine source refresh completed and the daemon entered its release check:

pop-upgrade[1233442]: [INFO ] daemon/mod.rs:1144: updating apt sources
pop-upgrade[...]: Fetched 2798 kB in 2s
pop-upgrade[...]: Reading package lists...
pop-upgrade[1233442]: [INFO ] daemon/mod.rs:1055: performing a release check
pop-upgrade[1233442]: [INFO ] release_api.rs:63: checking for build 24.04 in channel generic

No subsequent completion or error was logged.

The endpoint used by this path was tested separately and returned HTTP 200 with valid release JSON:

https://api.pop-os.org/builds/24.04/generic

Observations

• CPU stayed at 98–101% of one core for at least 3,791 consecutive one-minute samples.
• No APT or DPKG lock holder was present.
• No apt, apt-get, or dpkg child existed beneath pop-upgrade.
• Unlike pop-upgrade 100% CPU usage #297, there was no defunct apt-get child.
• strace -f showed the Tokio worker threads waiting in futex/epoll calls, while the CPU-burning main thread made no meaningful system calls.
• GDB confirmed that thread 1 (the daemon main thread) was running continuously in stripped Rust code. The Tokio worker threads were asleep.
• The sampled main-thread instruction was in an atomic reference-count cleanup path (lock decq / return), consistent with an async future being polled/dropped repeatedly in userspace.
• The daemon process mapped the current /usr/bin/pop-upgrade inode; it was not an older, deleted executable left running after package replacement.

Expected behavior

The release API request should either complete successfully or return an error after the configured timeout. The daemon should not spin indefinitely or consume a full CPU core.

Additional evidence available

I have retained:

• per-minute CPU/thermal/process samples;
• process tree and /proc state;
• strace summary and raw trace;
• six-hour journal capture around detection;
• complete GDB thread apply all bt output;
• exact package version, binary build ID, and upstream source revision.

I can attach sanitized captures if useful. The full journal is not attached initially because it may contain unrelated host information.

Possible regression/incomplete fix

The installed revision is the commit from #396, “Remove Infinite Retry on Metadata Fetch To Prevent 100% CPU Usage.” This capture suggests either another busy-loop path remains in the release API request/future handling or the fix does not cover this trigger.

[Jachimo]

I took the liberty of throwing an AI debugger at this for a while, because it keeps happening often enough to be annoying, but I didn't want to just disable it completely.

Based on its RCA, PR #396 didn't fix the issue because the RELEASE_CHECK handler (hanging, in my case, on the builds/22.04/nvidia request) doesn't cause either connect_timeout() or read_timeout() to fire.

This seems plausible, and would be consistent with the messages I keep seeing:

Jul 27 12:25:02  daemon/mod.rs:1055: performing a release check
Jul 27 12:25:03  release_api.rs:63: checking for build 24.04 in channel generic   ← OK
Jul 27 12:25:03  daemon/mod.rs:1062: Release { current: "22.04", next: "24.04", … }
Jul 27 12:25:03  release_api.rs:63: checking for build 22.04 in channel nvidia    ← HANGS
Jul 28 12:26:04  daemon/mod.rs:1144: updating apt sources  ← next day's timer fires

It appears it hangs on the 22.04 nvidia build, for some reason. (I'm unclear if that's not a valid build anymore, or if it's an uncommon configuration, but it doesn't look wrong…?)

Anyway, the RELEASE_CHECK handler is in daemon/src/daemon/mod.rs:626-693, and it calls futures::executor::block_on on the D-Bus event-loop thread:

b.method(
    methods::RELEASE_CHECK,
    ("development",),
    ("current", "next", "build", "urgent", "is_lts"),
    |_ctx: &mut Context, daemon: &mut Daemon, (development,): (bool,)| {
        // ... guard checks ...
        futures::executor::block_on(async {
            // Request 1: build_exists("24.04", "intel") → "generic"
            //   https://api.pop-os.org/builds/24.04/generic  → THIS SUCCEEDS
            let status = daemon.release_check(development).await?;

            // Request 2: get_release(status.current, "nvidia")
            //   https://api.pop-os.org/builds/22.04/nvidia  → THIS HANGS
            let release = Release::get_release(status.current, "nvidia").await;
            // ...
        })
    },
);

The D-Bus event loop runs at mod.rs:799-800:

loop {
    let _ = connection.process(std::time::Duration::from_millis(500));
}

Theory: When the RELEASE_CHECK handler enters block_on, connection.process() never returns, freezing the D-Bus event
loop. This explains the GDB output (see issue #401), where the main thread was in a refcount loop with the Tokio workers asleep.

[mmstick]

Every dbus method is handled synchronously from a background thread that the dbus connection maintains so the use of futures::executor::block_on within the dbus method handler makes no difference to the tokio runtime.

DBus message handling being blocked is probably just a symptom rather than the cause. If there is an existing task that acquires a lock on the daemon's mutex then the dbus method handlers will be blocked on waiting for the mutex to be unlocked.

Since you're able to replicate the issue you could try spawning this part on another thread, either with tokio or a physical thread with its own tokio executor.

[fredfluharty]

Confirming on Pop!_OS 22.04 (2026-09-28): same sequence (updating apt sources → checking for build 24.04 in channel generic → 100% CPU on one core, state RNsl, no further logs) recurring every few days; restarted the service each time. Difference from #401's capture: in my case the apt-get update child (PID 905607) is left as a zombie [apt-get] <defunct> under the daemon — the daemon spins and fails to reap the child. Daily recurrence suggests the spin is tied to the scheduled apt update cycle completing. Happy to provide strace/journal captures if useful.