Skip to content

gh-151378: Fix unbounded memory growth when recording binary and the thread is sleeping#152089

Merged
pablogsal merged 8 commits into
python:mainfrom
maurycy:rle-overlow-fixed
Jul 4, 2026
Merged

gh-151378: Fix unbounded memory growth when recording binary and the thread is sleeping#152089
pablogsal merged 8 commits into
python:mainfrom
maurycy:rle-overlow-fixed

Conversation

@maurycy

@maurycy maurycy commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Fixes #151378

I've described the root cause in #151378.

Instead of adding some LIMIT and checking it, I'm doing two tricks here that spare us one struct and some complexity: using a fixed buffer (lazy alloc, so threads do not pay any price if there's no repeats), instead of a dynamic array, and eager encoding, instead of having PendingRLESample.

Truth be told, it's not easy to write a test. We currently have no test for any OverflowError

Also, I'm wondering if we couldn't simplify alloc-dealloc in the reader, too:

if (i == 0 || status != batch_status) {
if (timestamps_list) {
int rc = emit_batch(state, collector, thread_id, interpreter_id,
batch_status, ts->current_stack, ts->current_stack_depth,
reader, timestamps_list, batch_idx);
Py_DECREF(timestamps_list);
if (rc < 0) {
return -1;
}
}
timestamps_list = PyList_New(count - i);
if (!timestamps_list) {
return -1;
}
batch_status = status;
batch_idx = 0;
}

if (PyList_SetSlice(timestamps_list, actual_size, PyList_GET_SIZE(timestamps_list), NULL) < 0) {

but it's out of the current scope (and maybe PyList_New(run_len) would be sufficient there; just call it a day.)

(No) reproduction

The same as in #151378

2026-06-24T15:42:07.243116000+0200 maurycy@gimel /Users/maurycy/work/cpython (rle-overlow-fixed ee78d43+) % ./python.exe -c "
import time
time.sleep(3600)
" & TARGET=$!

sudo ./python.exe -m profiling.sampling attach --binary -r 1000khz -d 300 -o /tmp/rle.bin $TARGET &
sleep 2; PROF=$(pgrep -fn "profiling.sampling attach")
for i in $(seq 5); do printf "t=%2dmin  RSS=%d MB\n" $i $(($(ps -o rss= -p $PROF|tr -d ' ')/1024)); sleep 60; done
[1] 65625
[2] 65626
t= 1min  RSS=32 MB
t= 2min  RSS=33 MB
t= 3min  RSS=33 MB
t= 4min  RSS=33 MB
t= 5min  RSS=33 MB
Captured 99,316,092 samples in 300.00 seconds
Sample rate: 331,053.64 samples/sec
Error rate: 0.00
  Binary Encoding:
    Records:          12,131
      RLE repeat:     12,130 (100.0%) [99,308,159 samples]
      Full stack:     1 (0.0%)
      Suffix match:   0 (0.0%)
      Pop-push:       0 (0.0%)
  Frame Efficiency:
    Frames written:   1
    Frames saved:     99,308,159 (100.0%)
    Bytes (pre-zstd): 189.6 MB
Warning: missed 200683909 samples from the expected total of 300000001 (66.89%)
Binary profile written to /tmp/rle.bin (99316092 samples)

[2]  + done       sudo ./python.exe -m profiling.sampling attach --binary -r 1000khz -d 300 -o 
2026-06-24T15:47:52.516465000+0200 maurycy@gimel /Users/maurycy/work/cpython (rle-overlow-fixed ee78d43+) % ./python.exe -VV
Python 3.16.0a0 (heads/main-dirty:ee78d4323c1, Jun 24 2026, 12:32:26) [Clang 21.0.0 (clang-2100.1.1.101)]

@maurycy maurycy changed the title gh-151378: Stop unbounded memory growth when recording binary and the thread is sleeping gh-151378: Fix unbounded memory growth when recording binary and the thread is sleeping Jun 24, 2026

@pablogsal pablogsal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice fix. I checked the RLE boundary case locally and it held up.

@pablogsal pablogsal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments

Comment thread Modules/_remote_debugging/binary_io_writer.c
Comment thread Modules/_remote_debugging/binary_io_writer.c
@maurycy maurycy requested a review from pablogsal July 4, 2026 15:09
@pablogsal pablogsal merged commit 639a552 into python:main Jul 4, 2026
54 checks passed
@pablogsal

Copy link
Copy Markdown
Member

Amazing! great job!

@pablogsal pablogsal added the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Jul 4, 2026
@miss-islington-app

Copy link
Copy Markdown

Thanks @maurycy for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

@bedevere-app

bedevere-app Bot commented Jul 4, 2026

Copy link
Copy Markdown

GH-153047 is a backport of this pull request to the 3.15 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Jul 4, 2026
@maurycy maurycy deleted the rle-overlow-fixed branch July 4, 2026 19:25
pablogsal pushed a commit that referenced this pull request Jul 4, 2026
…nd the thread is sleeping (GH-152089) (#153047)

gh-151378: Fix unbounded memory growth when recording binary and the thread is sleeping (GH-152089)
(cherry picked from commit 639a552)

Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

_remote_debugging: unbounded memory growth when recording binary and the thread is sleeping

2 participants