[rwgk]

Description

Builds on #6211. Reduce the time spent in tests/test_gil_scoped.py by batching subprocess checks while preserving all 17 basic cases, fresh worker threads, and six isolated cross-module initialization checks.

I expanded these tests during the 2022 investigation of PyTorch #83101, when we suspected a problem in gil_scoped_acquire. That work led to #4216, which added PYBIND11_SIMPLE_GIL_MANAGEMENT as a diagnostic alternative and a much larger matrix of GIL tests. We never obtained a controlled pybind11-only reproducer for the original problem. The expanded tests also exposed separate subprocess/threading problems, leading to the Linux forkserver and macOS spawn choices in #4306 and #4577.

I have considered the runtime excessive for a long time: this one file was taking about 18 of the 42 seconds needed for a serial run of the pytest suite. Threading code has never been something I approached comfortably, so the cleanup kept getting postponed. Coding-agent assistance has now made it practical to revisit the history, compare alternatives, and work through the correctness and coverage details carefully.

The initial analysis showed that process startup and repeated imports dominated the time. Most individual subprocess cases took about 0.23–0.27 seconds, while almost all direct basic cases rounded to 0.00 seconds. Removing only the separate one-worker parametrization helped, but still paid the startup cost for most of the original matrix. Batching offered a much larger reduction.

The implementation keeps the basic tests directly callable by pytest and also runs all 17 in three subprocess batches:

• On the child's main thread.
• In eight fresh worker threads per case, started and joined sequentially.
• In eight fresh worker threads per case, all started before any are joined.

The first sequential worker covers the former separate one-worker parametrization. Every case still creates and joins its own workers; there is no persistent thread pool. The factory-construction regression remains separate.

Six additional subprocesses preserve isolated first use of the helper module's private internals: entry with the GIL released or held, crossed with the three execution modes above. This retains fresh-interpreter coverage for those initialization paths.

The reasonable tradeoff is that later cases within each batch share an interpreter that earlier cases have already initialized. They no longer each receive a fresh process. Fresh worker creation and cleanup, all three execution modes, and the six selected first-use paths remain covered. This reduces the subprocess count from 71 to 14, including the detector, startup/shutdown controls, and the pipe-allocation regressions.

The timing comparisons below are single-run development measurements of this file on the same Linux host with CPython 3.14.4, the GIL enabled, the default pybind11 GIL implementation, and serial pytest execution:

Configuration	Runtime
Original subprocess matrix	18.74 s
Remove only the separate one-worker cases; retain the other per-case subprocesses	14.75 s
Two worker batches; retain six isolated first-use cases, but drop the other direct subprocess cases	3.77 s
Three-batch prototype, restoring all basic cases on the child's main thread	4.11 s

The third batch cost about 0.34 seconds in the prototype comparison and retained a distinct execution mode. That seemed a reasonable price for keeping main-thread coverage. The implemented consolidation subsequently measured 3.97 seconds. The progress-pipe version and its additional subprocess regression measured 4.36 seconds. With #6211's semaphore-free startup/finish signaling and its extra regression included, the current combined implementation measured 4.62 seconds: about a 75% reduction from the 18.74-second baseline.

Batch progress is reported to the parent over a one-way pipe. The parent uses it to identify the active case in failure messages and to renew the execution allowance when the batch advances, giving each case at least ten seconds. Detecting a deadlock can take one additional wait interval. Separate startup and shutdown allowances and worker-exception propagation come from #6211.

The first CI run exposed a Windows/PyPy failure when the original progress counter allocated shared ctypes memory. Replacing that counter with a pipe avoids the mmap allocation problem documented in PyPy #3488. A regression makes shared-ctypes allocation unavailable and verifies that a batch still completes.

Validation: full local regular and free-threaded CPython 3.14.4 suites, explicit spawn checks, and pre-commit passed. The final combined implementation passed all 35 GIL tests and full local suites: 1,396 tests on regular CPython, 1,394 on free-threaded CPython, and 1,007 on Linux/PyPy. Each CPython build also passed 31 C++ cases with 1,709 assertions.

After #6211 passed independent CI, the temporary hold commit was removed. Fresh CI and CIBW runs passed, along with the external checks. All five Linux/Windows PyPy jobs passed all 35 GIL tests and the later refcount test, with no GIL skips or xfails and no resource-tracker or unraisable errors.

Suggested changelog entry:

• Reduce test_gil_scoped.py runtime by batching subprocess checks while retaining isolated cross-module initialization coverage and fresh worker threads.