This is a living record of concrete gaps encountered while using MacVM Testbed. Keep observed behavior, effect, workaround, and a likely improvement direction together so later work can reproduce the problem.
Status: System Settings password entry fixed and live-verified 2026-09-27. Resume reporting was corrected separately below.
The target was initially suspended, so doctor correctly reported unavailable
administration and resident control. After target up through the common
machine-control client, guest administration and the resident worked. A
direct bin/macvm exec without the private target binding failed exact
identity resolution; machine-control --target macos os -- ... supplied that
binding and succeeded. The deployed resident was older than the controller's
lock-state contract. maintenance repair --profile development refreshed it,
after which target doctor reported a ready unlocked desktop with semantic,
capture, and input control.
Later, inventory credentials macvm reported the declared guest password file
as ready, but the normal System Settings authorization sheet rejected two
one-shot automated submissions. Guest authd/opendirectoryd logs reported
invalid credentials. The user then entered that same prepared-image default
password directly in the Tart window, and macOS accepted it. The saved
credential was valid; the secure authorization-sheet input route was not.
Stop automated retries after an authentication failure and offer direct user
entry. A ready inventory result verifies the credential file's presence and
permissions, not successful submission through a particular UI route.
A later controlled reproduction used the same strict System Settings sheet while adding an app to Accessibility. Process-targeted key events again left the sheet open. The resident now checks that System Settings and its secure field have foreground focus, then posts physical keys to the guest session event stream for this profile. One submission through the same private credential handoff dismissed the sheet and opened the app chooser, the next step of the requested settings change. The SecurityAgent fixture still uses its proven process-targeted route. Other macOS authorization sheet variants were not exercised in this reproduction.
After a later suspend and resume, target up reported that it could not
resolve the VM IP, although target status and guest administration showed
the VM running. target doctor reported desktopState: unknown and disabled
resident semantic/input operations. The outer screenshot showed an unlocked
desktop, and testbed -- ui health and testbed -- ui tree still worked with
Accessibility trust. This is a separate post-resume resident state-detection
failure; the app test used the direct testbed UI. maintenance audit --profile development confirmed the Aqua session was unlocked while semantic and
target-native readiness were unavailable. maintenance repair --profile development restarted only the resident LaunchAgent and restored unlocked
desktop, semantic, capture, and input readiness without a reboot.
The start command now reports observed VM power without requiring an address.
ip remains the separate strict address probe. Static tests cover a running
VM whose guest address is unavailable; a live resume still needs to verify
the startup result independently of the resident's desktop-state issue.
Status: mitigated 2026-09-26. Every restore attempted from an agent while
the controller user's screen was locked failed within a second:
VZErrorDomain Code=12 ... failed to restore with error "permission denied".
It failed for a snapshot taken seconds earlier with identical run flags, both
through the launchd runner and from a direct tart run. A cold boot of the
same disk succeeded in about 20 seconds. Apple's Virtualization engineers
state that saved state is protected by the keychain and needs the Mac
unlocked; an earlier snapshot also predated a host major-version upgrade.
Restoring from an unlocked session has not yet been re-tested here.
Additional adapter defects made the failure look intermittent:
suspendreturned while Tart was still writing the snapshot and reportedrunning, so a promptupcould replace the runner mid-write;uppolled for the full boot timeout after Tart had already exited and reported only a generic adapter failure; and- nothing reported the host lock state or the keychain requirement.
The adapter now reports extensions.hostSession and a lifecycle projection,
refuses suspend while the host is not unlocked or when
MACVM_SUSPENDABLE=false, waits for the saved state, fails a restore
immediately with the diagnosis, and offers discard-suspended-state.
Controllers that may be locked when a VM is next started should disable
suspend and park with shutdown.
Status: guarded 2026-09-26; interactive lock/unlock test pending. With the
controller locked, loginwindow was frontmost and no Tart window existed for
a VM started by an agent outside the host's Aqua session, even though Screen
Recording and event posting were granted. Host input is posted to the global
HID stream, so while locked it would reach the host lock screen rather than
the VM. Host input now refuses unless the host session is unlocked, and the
controller's hostAttendance gates outer UI and suspend.
Still to verify interactively, by locking and unlocking the controller:
screenshot and input on an unlocked unattended host with a Tart window,
refusal when attended, refusal right after a lock, and whether a restore
from an unlocked session succeeds.
Observed: macvm up bootstraps tart run into gui/$UID with
ProcessType Interactive and LimitLoadToSessionType Aqua, and the VM boots
and serves tart exec normally. But no Tart window is ever created, so every
outer command fails with No visible Tart window named 'NAME' was found; run without --no-graphics even though --no-graphics was never passed. The window
is not merely off-screen: a full CGWindowListCopyWindowInfo(.optionAll) sweep
finds zero Tart windows.
The cause is the controller session, not Tart and not the plist. When the
invoking agent or terminal is not itself in an Aqua GUI session, processes it
spawns get no window-server connection, and launchctl bootstrap gui/$UID
from that context does not repair it. This reproduces cleanly with any GUI
binary: launching one through LaunchServices (open -a Calculator) yields a
window, while executing the same binary directly
(Calculator.app/Contents/MacOS/Calculator) yields none. Screen Recording is
irrelevant to this failure; window titles were fully readable throughout.
Effect: on such a controller the documented bootstrap consent flow is
unreachable, because granting guest Accessibility and Screen Recording needs
the outer path and the outer path needs a window. Guest administration,
deploy-ui, and authorize-ui still work, so the target looks healthy while
semantic, capture, and input stay unavailable with no obvious cause.
Workaround: start the VM through LaunchServices so it inherits the Aqua session, then activate it before each outer command, because the window is created unmapped and only appears on screen once the application is activated:
open -a "$(dirname "$(dirname "$MACVM_TART")")/libexec/tart.app" --args \
run --suspendable --capture-system-keys --dir=macvm-testbed:REPO:ro NAME
open -a .../tart.app # activate; otherwise the window stays off-screen
bin/macvm screenshot /tmp/guest.pngA VM started this way is outside the launchd runner, so macvm up will not
manage it and a later up reproduces the windowless state.
Possible direction: have the provider detect the absent window and either
launch through LaunchServices itself or fail with a diagnostic that names the
non-GUI controller session as the cause, rather than suggesting
--no-graphics. A one-shot macvm activate-window would also make the
unmapped-window step explicit instead of incidental.
Observed: a VM suspended while MACVM_FORBID_OUTER_UI=true (so tart run
omitted --capture-system-keys) refused to resume once that flag was flipped,
failing with VZErrorDomain Code=12 ... failed to restore with error "invalid argument". Virtualization.framework requires an identical machine
configuration to restore saved state, and the outer-UI policy changes it.
A stale state.vzvmsave also blocks a cold boot until removed, and killing a
mid-suspend tart leaves Failed to lock auxiliary storage.
Effect: a policy change unrelated to lifecycle can strand a suspended VM, and the error names neither the flag nor the snapshot.
Workaround: shut the VM down rather than suspending it before changing
outer-UI policy; to recover, remove ~/.tart/vms/NAME/state.vzvmsave and cold
boot, accepting the loss of suspended RAM state.
Possible direction: record the run flags beside the snapshot and refuse to suspend, or warn on resume, when the effective configuration has changed.
Status: mitigated 2026-08-10. The testbed now defaults
MACVM_CAPTURE_SYSTEM_KEYS to false, and outer-UI-forbidden launches suppress
the Tart flag even if local configuration requests it. System-key capture is
an attended outer-recovery option only.
A long-running Tart process started with --capture-system-keys retained the
host keyboard grab after focus moved to another host application. The host
menu bar continued to identify Tart and other applications could not receive
typing. Suspending the VM removed the Tart process and released the grab;
restarting host SystemUIServer and Control Center restored the menu bar.
Effect: a VM doing target-resident tests could still interfere with its controller host even though no host-side screenshot or input operation ran. The outer-UI command guard alone was insufficient while the Tart launch itself requested a global system-key capture.
Status: resolved 2026-08-04. macvm up now submits tart run as a
transient job in the current user's launchd GUI domain. A separate invocation
verified the VM remained running after the launcher command and its process
group exited. Suspend is allowed to finish and exit the runner naturally so
the saved state remains available; the next up replaces the inactive job.
providers/tart-macos/provider.sh starts tart run with nohup ... & and
returns once Tart reports running. Under an agent command runner that reaps
the completed command's process group, Tart exits as soon as the macvm up
invocation ends even though nohup was used. Ordinary interactive shells may
not reproduce this.
Effect: a successful up is followed immediately by a stopped VM and later
commands fail. Keeping the launching shell alive in a persistent execution
session made the VM stable for the full smoke.
The launchd runner retains the graphical Aqua session, suspendability, system-key capture, and read-only repository share used by the prior direct launch.
Status: mitigated in the resident facade 2026-08-10. Legacy text output is unchanged, but resident full and compact records carry label and value separately. A composed Cua result exposed both a System Settings checkbox label and its value during the live resident test.
macui records title, description, and AX value, but formatted(_:) prints
only the first non-empty member. A labeled checkbox or switch therefore shows
its label but not its 0/1 value. Queries can still match the value, but a
human or agent cannot inspect checked state from normal tree output.
Effect: the 200 OK app-settings switches were discoverable by name but their state had to be verified through persisted settings or side effects.
Possible direction: print non-duplicate value= data separately, especially
for switches, checkboxes, radio buttons, text fields, and sliders.
Setting the AX value of a React number input changed the accessible value but did not dispatch the DOM input/change event used by React, so the new port was not persisted.
Effect: macvm ui set-value can report success without changing application
state for WebView inputs.
Workaround: use physical focus/select/type input and verify the application side effect. Possible direction: document this boundary explicitly or add a WebView-aware input path that generates real keyboard/DOM events.
The macOS folder chooser sometimes exposed row/cell elements with empty title, description, and value fields. In that state semantic queries could not name the target directory even though it was visibly present.
Workaround: use Command-Shift-G and enter an exact path, or use a fresh screenshot and coordinates. Possible direction: document the path-navigation recipe and consider deriving row text from named child cells when available.
Status: mitigated in the resident facade 2026-08-10. In Chrome, macvm ui tree exposed meaningful controls,
including the address field, extension toolbar controls, and extension page
content. Fresh ui find, ui press, and ui set-value invocations could not
reliably rediscover some of those same controls, even with the same explicit
application selector and an increased traversal depth.
Effect: a control can appear automatable during inspection but fail when the test attempts to act on it. Because each command starts a fresh helper and rebuilds the Accessibility snapshot, dynamic browser UI and focus changes may contribute to the disagreement.
Possible direction: make tree output and state-changing actions share one query/traversal implementation, add diagnostic output explaining why visible records were excluded, and add regressions for Chrome's address field, extension toolbar controls, and extension page buttons. A longer-lived guest-side helper may be worth evaluating if fresh AX references are the root cause.
The resident holds generation-scoped references and its Cua adapter preserves the provider snapshot token through the subsequent action. Stale-reference conformance lives in the machine-control corpus; legacy one-shot commands remain diagnostic compatibility commands.
Status: corrected for Shift and Command; other modifiers remain guarded.
The original sender turned AbC!@Z into abc12z and Command shortcuts into
literal keys even though host event monitoring saw the general flags. The
controlled comparison found that
supplying matching left-side device flags restored guest text and real menu
actions. Explicit flagsChanged events alone did not fix the effect.
The implementation now sends both flag forms and explicit modifier transitions. Control, Option, and Fn remain behind the non-secret diagnostic override. Unsupported text still refuses before any prefix; the owner-only secret route retains its exact-focused-window check and uses the verified Shift sequence. This is a sufficient sender correction, not proof of the exact private Tart/Virtualization.framework mechanism. Ordinary post-bootstrap input remains target-resident.
Status: unresolved design gap. macvm click, macvm type, and macvm key
foreground the Tart window and send input from the host. This is appropriate
for bootstrap, consent, and recovery, but it steals focus and can interfere
with someone using the host when semantic automation falls back to it during a
routine test.
Effect: unattended MacVM validation is not currently safe to run alongside interactive host work if it may use outer input.
Possible direction: make routine automation guest-native and non-disruptive by default. Add an explicit no-host-input mode that fails rather than falling back, reserve outer input for an opt-in exclusive/recovery mode, and provide guest-native primitives for common operations such as opening a URL in a chosen browser and sending a keyboard chord to a chosen application. Document which commands can activate the Tart window.