Fix deadlock in rayon pool

[seanyoung]

Description

Block-STM workers ran on a rayon thread pool (RAYON_EXEC_POOL), which makes a
deadlock reachable from any user transaction that calls a rayon-using Move native.

A rayon worker that blocks is not idle. Rayon's wait_until loop keeps it busy by
stealing other jobs from its own pool onto that thread. So a worker that blocks
inside a Move native's nested par_iter() — ark_ec MSM, pairing, multi-pairing,
ark_ff::batch_inversion under hash_to_structure — can pick up a sibling worker
task, and that stolen task can then block on state the original task still holds:
the writer-preferring RwLock over per-txn status in scheduler.rs, or a v1
dependency condvar for the very transaction this thread had been executing. Neither
side progresses, and the scheduler still believes the transaction is executing.

Every precondition is present in this tree today:

• workers on rayon (executor_thread_pool.scope(...), two sites in executor.rs)
• parking_lot::RwLock on txn_status and DependencyCondvar in scheduler.rs
• ark-std 0.5.0 carries rayon in Cargo.lock, so the natives reach it
• CRYPTOGRAPHY_ALGEBRA_NATIVES, BLS12_381_STRUCTURES and BN254_STRUCTURES are
  all default-enabled, so the path is reachable from an ordinary transaction

This makes Block-STM workers plain OS threads via std::thread::scope. Threads not
registered with rayon cannot have rayon jobs stolen onto them; a native's nested
par_iter() runs on rayon's global pool while the worker parks on OS primitives.
That closes the whole class structurally — for every native, including ones nobody
remembered to isolate — rather than requiring each rayon-using native to opt in.

With the pool no longer driving workers, RAYON_EXEC_POOL had no remaining purpose
and is removed, along with the plumbing that existed only to carry it:

Removed	Note
RAYON_EXEC_POOL	the num_cpus-thread par_exec-* pool
BlockExecutor::new(config, pool, hook)	now new(config, hook)
execute_block_on_thread_pool(pool, …)	merged into execute_block(…); the wrapper only injected the pool
create_executor_thread_pool() + 12 per-test pools	test plumbing
pool params on execute_block_parallel / run_tests_with_groups	test plumbing

Two call sites needed a real decision rather than a mechanical edit:

• FakeExecutor held a pool but only ever read current_num_threads() off it.
  Replaced with concurrency_level: usize, which is what the surrounding comment
  already said it wanted. from_genesis_with_existing_thread_pool becomes
  from_genesis_with_concurrency_level; the four fuzz targets that used it each
  built their pool with num_threads(FUZZER_CONCURRENCY_LEVEL), so they now pass
  that constant directly — no behavior change.
• NATIVE_EXECUTOR_POOL (executor-benchmark) is kept. Unlike RAYON_EXEC_POOL
  it has independent .install() users in aptos_vm_uncoordinated.rs and
  parallel_uncoordinated_block_executor.rs. Only the argument was dropped.
  The sharded executor's pools are likewise kept — they drive the cross-shard commit
  receiver and aggregator, not Block-STM workers.

Relationship to upstream

Motivated by aptos-labs/aptos-core@ad7011f771, which fixes the same deadlock class.
This implementation is independent, written from the failure mode rather than ported,
and diverges deliberately:

• upstream removes rayon from Block-STM via a hand-rolled persistent worker pool
  (~423 lines, using mem::transmute to forge a 'static lifetime on the work
  closure, plus catch_unwind, a Barrier, and a panic slot). This uses
  std::thread::scope, which gives lifetime safety and panic propagation from the
  compiler instead.
• upstream's pool exists to amortise thread spawn across blocks. That matters most on
  their tree because their with_native_rayon keeps its per-caller rayon pool in a
  thread_local!, which scoped workers would rebuild every block. We do not carry
  with_native_rayon, so natives use rayon's process-wide global pool and nothing is
  rebuilt. Measured cost of the simpler approach here: ~185us per parallel block for
  16 workers, scaling down with block size (num_workers is
  concurrency_level.min(num_txns / 2).max(2), so a small block spawns 2 threads).

If we later adopt with_native_rayon for defence in depth, that trade-off flips and
a persistent pool becomes worth revisiting.

How Has This Been Tested?

• cargo check --workspace --all-targets — clean. This is the load-bearing check:
  per-crate checks miss fuzzer-fuzz, which is a separate package from fuzzer.
• aptos-block-executor lib suite, 237/237: 98 combinatorial proptests (the
  randomised parallel-execution stress: contention, aborts, gas limits, delayed
  fields, resource groups), 113 scheduler/module tests, 26 unit + invariant tests.
• New regression test executor::worker_thread_tests::workers_are_not_rayon_registered
  pins the actual invariant: every worker sees rayon::current_thread_index() == None,
  driven from inside a rayon pool — the worst case, and the configuration the sharded
  executor uses. It also asserts the driving thread is rayon-registered, so it
  cannot pass vacuously.
• rustfmt clean on all 17 changed files; no new clippy findings.

Note for reviewers: unit_tests::test_resource_group_deletion is flaky and it is
not caused by this change. resource_group_bcs_fallback arms the process-global
failpoint fail-point-resource-group-serialization while libtest runs other tests in
parallel. Measured 6 failures in 8 runs both on this branch and with all files reverted
to the base commit — identical exposure — and 26/26 passes with --test-threads=1.
Worth a serial_test guard in a separate PR (serial_test is already a workspace dep).

Key Areas to Review

• spawn_block_stm_worker in executor.rs — the core of the change. The doc
  comment carries the deadlock rationale; that reasoning is the thing to check.
• Spawn-failure behaviour. .expect() on spawn_scoped means that if worker k
  of N fails to spawn, thread::scope joins workers 0..k-1 first — they execute
  the whole block with fewer workers, then the panic propagates. Not a hang and not
  incorrect output (Block-STM has no barrier and no worker-count termination
  condition; num_workers is used only for worker_id bounds-checking and a
  scheduling heuristic), but it is a new failure mode in kind: with the pre-spawned
  pool, thread exhaustion could not fail block execution. Upstream avoids this by
  spawning all threads before dispatching any task.
• Thread naming. Workers are now blockstm-{id}; the rayon pool used
  par_exec-{i} and upstream kept that name. Nothing in-repo references par_exec
  any more, but external runbooks or thread-dump tooling might — this bug was
  originally diagnosed from a production thread dump. Happy to switch back.
• AptosBlockExecutorWrapper::execute_block widened from pub(crate) to pub,
  since it absorbed the public execute_block_on_thread_pool.
• Sharded executor still calls execute_block from a rayon worker. Harmless now
  (the VM runs on std threads beneath it) and it is not wired into aptos-node, but
  it is the one place the old shape survives.

Type of Change

• Bug fix
• Refactoring

BlockExecutor::new loses its Arc<rayon::ThreadPool> parameter,
execute_block_on_thread_pool is removed in favour of execute_block, and
FakeExecutor::from_genesis_with_existing_thread_pool becomes
from_genesis_with_concurrency_level. All in-tree callers are updated.

Which Components or Systems Does This Change Impact?

• Validator Node
• Move/Aptos Virtual Machine
• Developer Infrastructure

Checklist

• I have read and followed the CONTRIBUTING doc
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I identified and added all stakeholders and component owners affected by this change as reviewers
• I tested both happy and unhappy path of the functionality
• I have made corresponding changes to the documentation

Follow-ups (not in this PR)

• aptos_rayon_execution_seconds still names and describes a rayon pool that no
  longer exists. Left as-is for dashboard continuity — upstream main carries the
  same misnomer — but the description is now false and should be corrected.
• serial_test guard for the failpoint-using tests described above.
• Optionally port with_native_rayon for defence in depth, so the isolation does not
  depend solely on workers staying off rayon.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}