Skip to content

Add native Editor window screenshots - #1417

Open
DelPariah wants to merge 4 commits into
CoplayDev:betafrom
DelPariah:codex/editor-window-screenshots
Open

DelPariah wants to merge 4 commits into
CoplayDev:betafrom
DelPariah:codex/editor-window-screenshots

Conversation

@DelPariah

@DelPariah DelPariah commented Oct 1, 2026 •

Copy link
Copy Markdown

Description

Agents can capture cameras and Scene View viewports, but cannot inspect arbitrary
open Editor tabs. This adds manage_editor_windows to list windows and capture
one Inspector, Console, custom tool window or inactive docked tab from Unity's
own buffer.

The default result is one PNG image for model inspection plus metadata, with no
saved file. Its assistant-audience annotation is a client hint; it cannot enforce
hiding, attachment display or privacy.

Type of Change

  • New feature
  • Documentation update
  • Test update

Changes Made

  • Add matching C#/Python tools using existing registration and instance routing,
    plus CLI editor windows and editor screenshot.
  • Capture physical content pixels using native backing scale and host margins,
    excluding dock tabs/borders. Share graphics-API orientation correction between
    Editor windows and Scene View, with the gamma-correct downscaler. No desktop
    fallback or EditorApplication.Step loop.
  • Use ToolParams for snake_case and camelCase batch parameters; reject invalid
    explicit selectors instead of falling back to the focused window.
  • Find the previous docked tab by host identity before selection, including after
    resize. Restore tab/focus, preserve later user focus changes, bound image size,
    reject concurrent captures, and handle closure, reload and shutdown.
  • Save unique PNGs under Library/McpEditorScreenshots only when requested;
    reuse encoded bytes for unscaled file/image output. The CLI requests file-only
    output. Declare destructiveHint=false and readOnlyHint=false.
  • Add Python/CLI/protocol tests, Unity tests, generated references and a guide.

Compatibility / Package Source

  • Unity tested on Windows: 2021.3.38f1, 2022.3.44f1, 6000.0.51f1, 6000.3.7f1.

  • Package source: local file: reference to this checkout's MCPForUnity folder.
    Package lock source: local; resolved Git lock commit: not applicable.

  • Validated base: 8be7d96d95aa3e262894c64412f0df3b432efa05 on beta.

  • Runtime/Editor semantic compilation passed for Windows/macOS/Linux symbols on
    all four installed editor versions. macOS/Linux graphical runtime and the
    exact pinned CI patches remain untested locally.

  • Prepare metadata/inline pixels before optional PNG persistence. Remove
    incomplete output on write failure; if cleanup fails, return its path and
    cleanup status. Server image-conversion errors also retain saved-path metadata
    without exposing encoded pixels. A lost connection after success does not
    roll back a requested file save.

  • Document capture ownership, focus/lifecycle invariants and fixture contracts.

Testing/Screenshots/Recordings

Latest local follow-up validation: 2026-10-02, source committed as
51f11ee0a59f26486d469180c280d3d1c533b5b7. The persistence/error-handling follow-up has fresh headless
validation; the graphical checks below are explicitly earlier results.

  • Python 3.10 full server suite: 1,415 passed / 3 skipped. Four
    injected image-conversion regression cases failed before the fix and pass
    afterward; errors retain saved-path metadata without returning encoded pixels.
  • Full headless Unity EditMode: 1,188 passed / 99 skipped / zero failures
    on each of 2021.3.38f1, 2022.3.44f1, 6000.0.51f1 and 6000.3.7f1
    (1,287 cases per Editor). All five new persistence cases pass on each: late
    inline-processing failure leaves no PNG, file-only/combined success preserves
    bytes, write failure removes incomplete output, and Windows cleanup failure
    reports its retained path. The partial-file cleanup case injects an argument
    failure with an existing owned partial file; it does not simulate a disk-full
    short write. The locked-file case uses a real Windows sharing violation.
  • Runtime/Editor semantic compilation: 24 assemblies passed, four
    installed Editor reference versions and Windows/macOS/Linux symbols.
  • Tooling 123 passed (two existing unawaited-mock warnings), generated
    reference drift check and website production build passed.
  • Fresh wheel/sdist build with cached dependencies; extracted-wheel imports,
    CLI help, image annotations and saved-path error checks passed. No persistent
    install, artifact upload or user Editor interaction was used.
  • Meaningful XML/docstring contracts cover the new tool and fixtures.
    CodeRabbit's automatic review of this head completed: 92.31% docstring
    coverage
    , passing its 80% threshold (91 functions / 10 files, one
    unsupported function skipped). No actionable or inline review comments were
    generated. This result is separate from the local scoped inventory.

Earlier graphical validation (2026-10-01, source 6e005a10)

These checks were not rerun for this follow-up, to preserve the active user's
mouse/keyboard and graphical Editor session. Buffer geometry, orientation,
selection/focus code, dependencies and packaging configuration are unchanged.

Unity editor Windows graphics APIs tested at 150% backing scale
2021.3.38f1 OpenGLCore
2022.3.44f1 Direct3D11
6000.0.51f1 Direct3D12
6000.3.7f1 OpenGLCore, Direct3D11, Direct3D12
  • Graphical screenshot/gamma: 40 passed / 1 batch-only skip / zero failures
    per combination. Includes 2px colored edges / 6px black interior, four-corner
    orientation, dark downscaling, direct camelCase/real batch selectors, invalid
    selectors, resized docks, focus restoration, file bytes, cancellation, closure,
    assembly reload, actual Game View PlayMode and minimized completion/recovery.
  • Final docked/floating Scene View: 2/2 per API on Unity 6000.3 OpenGLCore,
    Direct3D11 and Direct3D12 after the fixture compatibility adjustment.
  • Live Unity 2022.3 MCP: one assistant-audience PNG, no default file or duplicated
    image bytes, target/camelCase/batch routing, invalid-selector rejection,
    cancelled request then retry, and focus restoration.
  • Separate CLI/strict-harness PlayMode 5/5 and bridge smoke 7/7 are earlier
    2026-09-30 records; no new PlayMode or live-client rerun is claimed here.

Runtime limits remain macOS/Linux graphics, Metal/Vulkan, other DPI scales,
mixed-monitor transitions and exact pinned CI patch versions
. Compile symbols
do not establish platform runtime behavior. Minimized tests establish bounded
completion/recovery, not fresh pixels. Headless skips are reported as skips.
Upstream fork workflows currently require repository approval (action_required);
local checks are not a claim that those workflows ran or passed.

Documentation Updates

  • Added a tool and updated its generated reference/manifest entries
  • Updated narrative and CLI guides; reviewed generated changes

Related Issues

The window-buffer path avoids the stepping used in the separate composited Game
View path discussed in #1289; this change does not repair that existing path.

Additional Notes

CodeRabbit's remaining moderate security-architecture note describes the intended
opt-in capability: admitted clients can read unredacted Editor tabs, including
private displayed data. Audience hints do not restrict disclosure. Existing
instance authentication and tool enablement remain in place; no authentication
bypass was identified. The guide documents trusted-client use and manual
retention of explicitly saved files. Per-window allowlists/consent and automatic
retention would be additional product controls, outside this contribution.

The combined smoke-then-PlayMode harness times out after smoke marks an unsaved
scene dirty and Unity cancels its save dialog in batch mode. A clean upstream-base
control reproduced the same failure; separate smoke and clean PlayMode legs pass.

Batch pixel capture and native OS dialogs are unsupported. Minimized windows can
return stale buffers; tests verify completion/recovery, not fresh minimized pixels.
macOS/Linux graphics, Metal/Vulkan, DPI scales other than 150%, mixed-monitor
transitions and exact pinned CI patches remain untested. Claude/Codex-specific
image display remains untested; image audience annotations are client hints.
Captures may expose private Editor data to the configured MCP client.

No project-specific adapter/defaults, credentials, screenshots, reference DLLs or
local reports are included in the contribution.

Summary by CodeRabbit

  • New Features
    • List open Unity Editor windows and capture a selected or focused window as an image or PNG file.
    • Choose windows by ID, title, or type. The previous tab and focus are restored by default.
  • Documentation
    • Added guides and CLI references for listing windows and capturing screenshots, including usage options, limitations, and privacy considerations.

@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 9dc7bc65-08b4-4659-8673-2dd7241a8e9c

📥 Commits

Reviewing files that changed from the base of the PR and between 6e005a1 and 51f11ee.

📒 Files selected for processing (7)
  • MCPForUnity/Editor/Helpers/EditorWindowScreenshotUtility.cs
  • MCPForUnity/Editor/Tools/ManageEditorWindows.cs
  • Server/src/services/tools/manage_editor_windows.py
  • Server/tests/test_cli_editor_windows.py
  • Server/tests/test_manage_editor_windows.py
  • TestProjects/UnityMCPTests/Assets/Tests/EditMode/Tools/ManageEditorWindowsTests.cs
  • website/docs/guides/editor-window-screenshots.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • Server/tests/test_cli_editor_windows.py
  • MCPForUnity/Editor/Helpers/EditorWindowScreenshotUtility.cs

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

The pull request adds a Unity Editor window listing and screenshot tool, with MCP server and CLI commands. It adds capture scaling and readback logic, selection and focus handling, optional PNG file and inline image output, tests, and usage documentation.

Changes

Editor window screenshots

Layer / File(s) Summary
Window pixel capture
MCPForUnity/Editor/Helpers/EditorWindowScreenshotUtility.cs
Capture uses the window’s backing scale when available, offsets for scaled host borders, and flips pixels based on graphics UV orientation. Cleanup destroys textures that are not returned.
Unity window selection and capture
MCPForUnity/Editor/Tools/ManageEditorWindows.cs, MCPForUnity/Editor/Tools/ManageEditorWindows.cs.meta, TestProjects/UnityMCPTests/Assets/Tests/EditMode/Tools/ManageEditorWindowsTests.cs, TestProjects/UnityMCPTests/Assets/Tests/EditMode/Tools/ManageEditorWindowsTests.cs.meta
The Unity tool lists windows and captures a selected or focused window. It validates selectors and capture limits, coordinates selection and focus restoration, and supports PNG file or inline image output. Edit Mode tests cover selection, capture lifecycle, failures, and output.
MCP, CLI, and documented access
Server/src/services/tools/manage_editor_windows.py, Server/src/services/tools/utils.py, Server/src/cli/commands/editor.py, Server/tests/test_manage_editor_windows.py, Server/tests/test_cli_editor_windows.py, Server/tests/test_tool_annotations.py, Server/src/cli/CLI_USAGE_GUIDE.md, manifest.json, website/docs/guides/editor-window-screenshots.md, website/docs/reference/cli.md, website/docs/reference/tools/core/*, website/docs/reference/tools/index.md, website/sidebars.js
The server registers the tool and returns assistant-audience image blocks when requested. The CLI adds editor windows and editor screenshot. Tests cover validation, transport, and image responses; references document the tool and commands.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~50 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant MCPClient
  participant manage_editor_windows
  participant UnityTransport
  participant ManageEditorWindows
  participant EditorWindowScreenshotUtility
  MCPClient->>manage_editor_windows: Submit list or screenshot request
  manage_editor_windows->>UnityTransport: Dispatch validated command
  UnityTransport->>ManageEditorWindows: Invoke Unity tool
  ManageEditorWindows->>EditorWindowScreenshotUtility: Capture selected window pixels
  EditorWindowScreenshotUtility-->>ManageEditorWindows: Return captured texture
  ManageEditorWindows-->>UnityTransport: Return metadata and optional PNG
  UnityTransport-->>manage_editor_windows: Return Unity response
  manage_editor_windows-->>MCPClient: Return metadata and optional image
Loading

Merge Risk: ⚪ Minimal · up to 51f11

The window screenshot feature has no identified merge-blocking issue after normal checks.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to 51f11

Screenshot access now includes private Editor tabs, potentially exposing displayed source code, paths, logs or credentials to trusted clients. Existing authentication and instance selection remain in place, and capture has bounded dimensions and lifecycle cleanup. Saved screenshots require separate retention management.

Retained concerns

  • Medium · security · observed: Enabling window capture extends admitted clients' readable scope to unredacted docked or focused Editor tabs, including inactive tabs selected for capture. Display-audience hints do not restrict disclosure. This is a documented capability expansion requiring trust in the receiving client, not evidence of an authentication bypass.
Security review details

Security Blast Radius

  • inferred — A compromised admitted client can enumerate and capture eligible tabs within an accessible Unity instance, including displayed secrets. Remote instance access remains user-scoped; the pixel source is the selected Unity host buffer, not arbitrary desktop windows. This trace does not establish cross-tenant access or broader operating-system capture.

Security Findings and Attack Paths

  • inferred — The material disclosure path is an enabled tool invoked by an admitted client, followed by window selection, unredacted buffer capture and image return or file persistence. The implementation intentionally permits this path. The examined routing controls counter an authentication-bypass interpretation, but do not provide per-window privacy filtering.

Trust Boundaries and Controls

  • observed — Controls include authenticated remote instance routing, Unity-side tool enablement, exact selector validation, bounded capture area and generated project-local output paths. The CLI route is omitted in remote-hosted mode. Selector and dimension checks constrain targeting and resources; they do not redact sensitive content.

Resilience and Maintainability Implications

  • observed — Failed file writes attempt deletion and disclose the retained path if cleanup also fails. Image-conversion failures preserve saved-path metadata without repeating encoded pixels. These mechanisms contain partial failures, but successful saved files remain after interruption or subsequent tool disablement.

Hardening Proposals

  • proposed — For deployments that should not grant clients access to every eligible Editor tab, consider explicit window-type allowlists or capture consent. Where file output is used, consider a configurable retention policy. These are additional controls, not assertions that an existing requirement was violated.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 92.31% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 91 functions across 10 files. (1 skipped: 1…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely identifies the main change: native Unity Editor window screenshot support.
Description check ✅ Passed The description is comprehensive and follows the repository template. It covers the feature, change type, implementation details, compatibility, testing, documentation, related context, and known limi…
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@DelPariah
DelPariah marked this pull request as ready for review October 2, 2026 00:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant