wslc: idle-terminate inactive per-user session VMs#41077
Draft
benhillis wants to merge 34 commits into
Draft
Conversation
Per-user WSLC container session VMs now idle-terminate when no container is in a non-terminal (Created/Running) state, freeing host memory, and lazily restart on the next operation that needs the VM. - Centralize VM lifecycle in WSLCSession via TearDownVmLockHeld / StartVmLockHeld and an atomic VmExitDisposition (Active / StopRequested / ExitClaimed) to arbitrate expected stops vs. spontaneous VM exits without a polling thread. - Gate VM-requiring entrypoints behind AcquireVmLease(), which brings the VM up on demand and keeps it alive for the operation's duration. - Add IWSLCSession::BeginContainerOperation so a CLI command can hold the VM alive across resolve + operate + streamed output. - Preserve the session WarningCallback for the lifetime of the session so warnings emitted by the lazy VM start (e.g. resource recovery) are still delivered to the CLI invocation. - Remove the dtor lock in HcsVirtualMachine; OnExit/OnCrash are lock-free. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Only Running containers now hold an activity reference that keeps the
per-user session VM alive. Previously a container in either Created or
Running state held the reference, so a `create`d-but-never-started
container pinned the VM indefinitely and defeated idle termination.
A created container's metadata persists on the containerd VHD across VM
teardown and is rebuilt by RecoverExistingContainers on the next
VM-requiring operation, so create -> idle-terminate -> start later works;
the 30s grace period covers the common create-then-start gap.
Also fix m_stateChangedAt recovery for created containers: docker inspect
reports FinishedAt as the zero date ("0001-01-01T00:00:00Z") for a
never-started container, which parsed to year 1 and rendered as "created
2026 years ago". Use the container's Created time for the Created state.
This recovery path was previously unreachable, since created containers
never got torn down.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses review feedback: WSLCContainer.h still described the activity hold as held while Created/Running, but it now only pins the VM while Running. Update the two header comments to match the implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Make the VM idle grace period configurable via settings.yaml (session.idleTimeout, default 30s) instead of a hardcoded constant. - Assert UserSid is non-null in PersistSettings rather than tolerating a null SID. - Drop the session warning-callback GIT fallback; warnings emitted outside a callback-bearing operation are logged and event-logged only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Recovery warnings emitted during lazy VM start run outside the user's current command, so they are now logged (and written to the event log) instead of being routed back to the session-creation warning callback. Update the three WarningCallback*Recovery unit tests and the e2e test to assert the warning is no longer delivered to the session callback / printed on stderr. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- BeginContainerOperation: reject new operations once the session is terminating/terminated, mirroring EnsureVmRunning's gate, so a started operation cannot pin a VM that is being torn down. - TearDownVmLockHeld: reset m_storageMounted after unmounting so the flag does not stay stale across an idle teardown. - CLI Session model: stop retaining the IWarningCallback for the session lifetime. Recovery warnings from lazy VM start are logged rather than delivered to the session callback, so the stashed callback (and its now-misleading comments) is dead state. The callback is still passed to CreateSession, where it is consumed during initialization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Probe the storage VHD once with the std::error_code overload so an access-denied/transient I/O error surfaces as a clear Win32 error via WIL instead of a generic filesystem_error-to-HRESULT conversion, and avoid the duplicate exists() check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move the VM instance and its lifecycle state (VM, IO relay, docker client, event tracker, volumes, containerd/dockerd processes, exit-disposition and vm-state atomics, session lock, idle state, published-port bookkeeping, and ephemeral swap/storage state) out of WSLCSession into a new WSLCSessionRuntime. WSLCSession retains identity and orchestration and drives the runtime through BringUp/RecoverState/TearDownSessionState/OnSpontaneousExit/OnCrashDump hooks plus a SessionContext carrying the lifecycle primitives the runtime observes, so the runtime no longer reaches into WSLCSession private members. VM state is reached through the runtime, with Acquire()/LockedRuntime as the ergonomic lease accessor. Concurrency invariants are preserved: the exit-disposition claim protocol, the relay-thread teardown guard, lock ordering, GIT re-fetch per VM creation, and release-lock-before-Disarm in shutdown. The VM-exit monitor is armed between bring-up and state recovery to match the pre-extraction ordering. This is an isolated refactor commit on top of the idle-terminate work so it can be reverted independently. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy the termination reason/details into the session-visible fields before SetEvent() so a waiter woken by the terminated event cannot observe the default Unknown reason. Restores the pre-refactor ordering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…factories Add a test-only TriggerIdleTermination COM method to force VM idle teardown on demand, with three TAEF tests that hammer the VM idle-termination race paths. Refactor WSLCContainer construction to take WSLCSessionRuntime& instead of five individual runtime members. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The TriggerIdleTerminationRecoversRunningContainer test left the persisted container in the shared default session on early-exit or if the recovered handle's best-effort delete failed, making later ListContainers-based tests order-dependent. Add a scope_exit that force-deletes the container by name regardless of exit path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…dle-terminate-vm # Conflicts: # src/windows/wslcsession/WSLCContainer.cpp # src/windows/wslcsession/WSLCSession.h
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Session-level WSLC plugin hooks (OnSessionCreated/OnSessionStopping) assumed session == running VM. With idle termination (PR #40781) the VM is created lazily, torn down when idle, and transparently recreated while the session persists, so plugins had no notification of actual VM lifecycle. Add VM-level hooks OnWslcVmStarted/OnWslcVmStopping that fire on every VM (re)start and teardown, in addition to the once-per-session hooks. Both are best-effort (errors logged and ignored). OnWslcVmStarted fires after releasing the runtime exclusive lock so a plugin may reentrantly call back into the session (e.g. WSLCCreateProcess) without deadlocking; OnWslcVmStopping fires under the lock during teardown, gated on a fire-once flag so a failed bring-up emits no spurious stopping. Adds a PluginTests::WslcVmRestart case that drives first start, forced idle teardown (TriggerIdleTermination), and lazy restart, asserting the hook sequence and proving reentrancy is deadlock-free. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f1c36ca0-19d0-46a7-82b3-10f1e1ee3e4c
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Session-scope DockerEventTracker; rebind per VM start, preserve subscriptions - Fire OnVmStopping with m_lock dropped to avoid plugin-reentrancy deadlock - Keep containers alive on idle teardown; clear only on permanent shutdown - Redesign OnIdleTimer to commit teardown unconditionally after notifying, matching the test-trigger path; removes phantom OnVmStopping->OnVmStarted and the concurrent-Terminate is_signaled crash - Dedup containerd storage mount point into WSLCSessionDefaults.h - Mark WslcVmStarted/WslcVmStopping hooks as introduced in 2.9.5
NotifyVmStarted/NotifyVmStopping invoked the plugin hook while holding m_notifyLock. During idle teardown a plugin may reentrantly restart the VM (WSLCCreateProcess), re-entering these notifications and self-deadlocking on the non-recursive mutex. Flip the pairing state under the lock, copy the hook, then invoke it after releasing the lock. Also clarify Shutdown's lock parameter (runtimeLock is the exclusive hold on m_lock) per review feedback.
The VM-restart plugin test already validates a reentrant process launch and mount+unmount from OnVmStopping. Extend OnVmStarted to also mount+unmount so both lifecycle notifications validate reentrant mount management does not deadlock, and assert the new log lines.
UploadArchive and DownloadArchive streamed data without holding a VmLease, so idle teardown could race and tear down the docker client mid-transfer. Acquire a VmLease and wrap in try/CATCH_RETURN to match Export/Logs and the other VM-dependent container operations.
…tion TriggerIdleTerminationConcurrentWithOperations bounded the worker join with a std::async future, but on timeout the failed VERIFY would unwind and block in the future destructor until the (deadlocked) task completed -- hanging the test host. Fail fast with a dump on timeout so a real deadlock fails cleanly.
OnIdleTimer dropped the runtime lock to fire OnVmStopping, then unconditionally tore the VM down after reacquiring. A lease that started in that window found the VM still Running, so it did not restart and could leave a long-lived activity token (e.g. a process keep-alive), and the teardown would then kill it. Re-check the activity count after reacquiring the lock and, if non-zero, abandon the stop and re-pair the notification with a fresh OnVmStarted. Also set m_vmStartNotified whenever the VM starts rather than only when an OnVmStarted hook is installed, so a hooks user that sets only OnVmStopping still receives paired stop notifications.
ReleaseRuntimeResources unconditionally called UnmountWindowsFolder via Vm() during container teardown. After an unexpected VM exit the guest is already gone, so each call only blocks on the RPC timeout and emits a spurious unmount-failed warning. A dead VM has already dropped every guest mount, so mark the mounts inactive locally instead. Add WSLCSessionRuntime::VmExited() (mirrors TearDownVmLockHeld's VM-dead check) so the container can detect this via the runtime it already references.
VmExited() read m_vmExitedEvent without the runtime lock, but StartVmLockHeld and TearDownVmLockHeld reset/replace that event under the lock, so a container teardown running on another thread could observe the handle mid-reset and fail-fast. Mirror the state in a std::atomic<bool>: cleared when a VM instance starts and latched when the guest is observed dead at the top of teardown (before session-state cleanup, so ReleaseRuntimeResources sees it). VmExited() now returns the atomic.
added 2 commits
July 13, 2026 20:40
OnIdleTimer's abort path re-paired OnVmStarted without checking whether the VM died while m_lock was dropped for OnVmStopping. OnVmExited() declines that exit (the expected-stop claim is held) and the exit handle is one-shot, so the session was left with a dead VM marked Running until a later idle cycle. Tear the VM down in that case so a waiting lease restarts a fresh instance, matching the commit path. EnsureVmRunning's running fast path returned before the terminating gate, so a reentrant plugin lease during Shutdown's OnVmStopping (lock dropped) could run work against a VM being permanently torn down. Enforce the gate before the fast path, as its comment already documents.
The runtime extraction exposed mutable internals directly, letting WSLCSession drive them from outside. Replace those handouts with intent-revealing methods so the runtime owns its invariants: - Remove dead accessors StateAtomic() and VmExitedEvent() (no callers). - Replace ExitDispositionAtomic().load() call sites with the existing ExitDisposition() value getter and drop the atomic-ref accessor. - Replace the raw srwlock handout Lock() with TryLockExclusive(), used only by the Terminate/Shutdown handoff. - Replace DockerdReadyEvent() with Reset/Is/SignalDockerdReady(); replace the ContainerdProcess()/DockerdProcess() ref assignments with setters; replace SwapVhdPath() with SetSwapVhdPath(). No behavior change. Idle, VM restart, VM-kill, and plugin tests pass.
benhillis
changed the base branch from
user/benhill/wslc-idle-terminate-vm
to
master
July 14, 2026 17:05
Contributor
There was a problem hiding this comment.
Pull request overview
Refactors WSLC’s per-user session implementation to tighten the boundary between WSLCSession and VM/runtime state by introducing WSLCSessionRuntime and shared IdleState, reducing direct access to runtime internals and enabling lazy VM start + idle teardown/restart workflows.
Changes:
- Introduces
WSLCSessionRuntime(VM lifecycle, docking readiness, VM lease/locking, idle teardown orchestration) andIdleState(shared activity refcount + threadpool timer) to narrow the session/runtime seam. - Adds VM-lifecycle plugin hooks (
OnVmStarted/OnVmStopping) and a client-side operation token (BeginContainerOperation) to prevent VM idle-teardown mid multi-RPC container operations. - Updates container/VM port reservation handling and expands tests/e2e coverage for lazy start, forced idle termination, recovery, and plugin VM restart sequencing.
Reviewed changes
Copilot reviewed 38 out of 38 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| test/windows/WSLCTests.cpp | Adds/updates tests for lazy VM start and forced idle-termination lifecycle/recovery behaviors. |
| test/windows/wslc/e2e/WSLCE2EWarningTests.cpp | Adjusts expectations for when recovery warnings are surfaced on stderr vs logged. |
| test/windows/wslc/e2e/WSLCE2ESessionEnterTests.cpp | Aligns error output with localized “storage not found” messaging for session enter. |
| test/windows/testplugin/Plugin.cpp | Implements VM started/stopping plugin hooks and reentrant-call exercises. |
| test/windows/PluginTests.h | Extends plugin test modes with WslcVmRestart. |
| test/windows/PluginTests.cpp | Adds plugin test validating VM restart hook sequence across idle teardown and session termination. |
| src/windows/wslcsession/WSLCVirtualMachine.h | Introduces per-VM port reservation table and updates VmPortAllocation ownership model. |
| src/windows/wslcsession/WSLCVirtualMachine.cpp | Refactors port allocation/release to use shared reservation table (weak refs). |
| src/windows/wslcsession/WSLCSessionRuntime.h | New runtime abstraction defining VM lifecycle, leases, hooks, and test-only idle termination. |
| src/windows/wslcsession/WSLCSessionRuntime.cpp | Implements runtime VM start/stop/teardown, idle timer, notifications, and lease semantics. |
| src/windows/wslcsession/WSLCSession.h | Routes VM/runtime responsibilities through WSLCSessionRuntime; adds activity-token APIs. |
| src/windows/wslcsession/WSLCSession.cpp | Moves VM bring-up/teardown logic behind runtime hooks; adds BeginContainerOperation + test-only idle trigger. |
| src/windows/wslcsession/WSLCProcessControl.cpp | Preserves real container exit code when releasing without signaling; only synthesizes SIGKILL if none. |
| src/windows/wslcsession/WSLCProcess.h | Adds process keep-alive token support to pin VM activity for client-held process proxies. |
| src/windows/wslcsession/WSLCIdleState.h | New shared activity refcount + threadpool-timer idle teardown state with safe disarm. |
| src/windows/wslcsession/WSLCExecutionContext.h | Makes warning callback usage more robust by snapshotting pointer before use. |
| src/windows/wslcsession/WSLCContainer.h | Refactors container to depend on runtime (not raw VM/docker/relay refs) and adds VM teardown/recovery helpers. |
| src/windows/wslcsession/WSLCContainer.cpp | Implements survivor reconciliation/recovery (ports, --rm cleanup) and activity holds for running containers + client-held execs. |
| src/windows/wslcsession/IORelay.h | Adds IsRelayThread() helper for safe teardown decisions. |
| src/windows/wslcsession/IORelay.cpp | Joins MTA in relay thread (safe COM proxy release during callbacks) + implements IsRelayThread(). |
| src/windows/wslcsession/DockerEventTracker.h | Splits construction from per-VM Connect() to preserve registrations across VM restarts. |
| src/windows/wslcsession/DockerEventTracker.cpp | Implements reconnectable event tracking model. |
| src/windows/wslcsession/CMakeLists.txt | Adds new runtime/idle headers and runtime source to build. |
| src/windows/wslc/services/SessionService.cpp | Clarifies warning-callback lifetime semantics during session creation. |
| src/windows/wslc/services/SessionModel.h | Adds BeginContainerOperation() helper on client model to hold VM-alive token. |
| src/windows/wslc/services/ContainerService.cpp | Wraps multi-RPC container ops with begin-operation tokens to prevent idle teardown between calls. |
| src/windows/service/inc/wslc.idl | Extends COM interfaces with VM lifecycle notifications, idle timeout, TriggerIdleTermination (test), and BeginContainerOperation. |
| src/windows/service/exe/WSLCSessionManager.h | Avoids nested plugin notifications during session pruning when already in plugin context. |
| src/windows/service/exe/WSLCSessionManager.cpp | Plumbs idle timeout from user settings and adds plugin-notification-context detection. |
| src/windows/service/exe/WSLCPluginNotifier.h | Extends notifier interface with OnVmStarted / OnVmStopping. |
| src/windows/service/exe/WSLCPluginNotifier.cpp | Forwards VM lifecycle notifications to plugin manager under COM execution context. |
| src/windows/service/exe/PluginManager.h | Adds VM lifecycle hook dispatch entrypoints. |
| src/windows/service/exe/PluginManager.cpp | Dispatches OnWslcVmStarted / OnWslcVmStopping across plugins with logging. |
| src/windows/service/exe/HcsVirtualMachine.cpp | Avoids locking in destructor to prevent callback/teardown deadlocks. |
| src/windows/inc/WslPluginApi.h | Adds public plugin hook typedefs/fields for WSLC VM started/stopping lifecycle. |
| src/windows/common/WSLCUserSettings.h | Adds session.idleTimeout setting key mapping. |
| src/windows/common/WSLCUserSettings.cpp | Documents and validates session.idleTimeout (must be > 0 to take effect). |
| src/windows/common/WSLCSessionDefaults.h | Centralizes containerd storage mount point constant. |
benhillis
changed the title
The BeginContainerOperation() lease is held only for its scope lifetime to keep the session VM alive during the operation. Annotate it so its intent is explicit and a future edit doesn't drop it to a discarded temporary.
- OnIdleTimer: release the expected-stop disposition claim before the second lock drop for NotifyVmStarted(), so a VM crash in that window is claimable by OnVmExited() as a normal spontaneous exit instead of being silently dropped while m_vmState stays Running. - NotifyVmStarted/NotifyVmStopping: hold m_notifyLock (now recursive) across both the pairing-state flip and the hook invocation, so a start and a stop racing on different threads cannot interleave and deliver a stale OnVmStarted after its OnVmStopping.
Member
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
dkbennett
reviewed
Jul 17, 2026
dkbennett
left a comment
dkbennett
left a comment
Member
There was a problem hiding this comment.
This looks like a lot of goodness! Would like to get it in soon so we have a lot of usage time to find any corner cases in the lifetime management that might come about.
This seems especially useful for third party developers utilizing the WSLC API to use containers in their apps, dont want those sessions piling up long after the app is done running. Though that could still happen if they add in a long-running container that could persist after the app is done. That might be intentional but perhaps we can enhance this later with some additional session termination behavior to try to prevent long-running third party sessions that aren't actually necessary but might be held open by a container they created.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Idle-terminates per-user WSLC session VMs after a period of inactivity, so an idle container session releases its utility VM instead of holding memory indefinitely. Includes the supporting refactor that makes the VM lifecycle safe to reason about.
What this does
Session::BeginContainerOperation) so the VM cannot be idle-terminated mid-operation.OnVmStarted/OnVmStoppingplugin hooks, paired correctly across start/stop and VM-restart.VM lifecycle hardening
WSLCSessionRuntimestate machine (start / stop / teardown / spontaneous-exit), replacing ad-hoc state spread acrossWSLCSession.EnsureVmRunning).Encapsulation cleanup
WSLCSession/WSLCSessionRuntimeseam: replaces ~9 accessors that handed out mutable internals with intent-revealing methods (TryLockExclusive,SetContainerdProcess,Set/Reset/Signal/IsDockerdReady,SetSwapVhdPath), and drops dead accessors.[[maybe_unused]]to document that they are held for their scope lifetime.Testing
Full build (x64 + arm64) + package, then locally: idle-termination (5/5), WslcVmRestart, VmKillTerminatesSession, StuckVmTermination, reentrant-mount plugin hook, and PluginTests::WslcSuccess all pass.