0.15.2 infinite loop during deduping

[axet]

add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip, off: 133308416, dest: 1
(dedupe) btrfs same info: ioctl_file: "/media/axet/2TB/Games/Skyrim/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip"
(dedupe) logical_offset: 133308416, length: 0, dest_count: 1
(dedupe) info[0]: name: "/media/axet/2TB/Games/Skyrim/Skyrim/MO2/downloads/Lordbound Main File-160493-0-8-1-1760641847.zip", fd: 7, logical_offset: 133308416, bytes_deduped: 0, status: 0
add ioctl request /media/a^C

[Squishems]

This is happening regularly for me too. I've had to set a timeout (RuntimeMaxSec=86400) because otherwise it will just sit there forever.

It seems to happen consistently with the same files, but I can't figure out what exactly is in common between the files it gets stuck on?

Edit: In case it matters, I'm running on the latest commit, not the latest release.

[paluszak]

Exactly the same things happening here, I'm on on the latest release (0.15.2).

[Root-Core]

I have the same or a similar issue with the current release.
However, I tested #402 and it seems to fix it for me.
Could somebody verify this?

[martinus]

The mechanism here is in process_dedupes() in dedupe.c: it advances by info->bytes_deduped and requeues the request whenever the ioctl succeeded and req_total has not reached orig_len. If a successful FIDEDUPERANGE reports zero bytes deduped, the request goes back on the queue with identical parameters and dedupe_extents() reissues the same call forever — which matches the debug output in the first post exactly (length: 0, bytes_deduped: 0, status: 0, repeating).

I opened #408 for this: it stops the loop when a whole round makes no progress, and accounts the stuck requests as completed rather than spinning at 100% CPU. Productive dedupe always moves more than zero bytes per round, so it doesn't cut real work short. #415 has the kernel-side detail on how a successful call can report zero.

@Root-Core — good to know #402 helps, and I'm not doubting the result; it changes how the dedupe phase fiemaps, so it may well avoid the requests that trigger this. #408 is the narrower fix aimed at the loop itself, so the two are complementary rather than alternatives.

Both are in my fork and have been running there for a while.