-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
feat: Add USB iOS capture and iPhone frames #906
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Merkost
wants to merge
12
commits into
webadderallorg:main
Choose a base branch
from
Merkost:feat/ios-usb-capture
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
c58eae4
docs: establish ios capture baseline and hardware test matrix
Merkost cdc4233
feat: define ios capture contracts and source capabilities
Merkost 08163a7
feat: add native iOS screen capture helper
Merkost 43bff87
feat: integrate experimental iOS capture and recovery
Merkost 3dd5bd2
docs: record iOS capture workflow and pending release gates
Merkost ed4b50b
Fix discovery of muxed-only iPhone screen sources
Merkost ce09db0
Initialize iPhone discovery on the main run loop
Merkost 717d5ea
Match iPhone picker rows to screen source styling
Merkost 31323ac
Use a concise Devices heading in the picker
Merkost 134c336
Improve device discovery and simplify the picker
Merkost 06591da
Support iPhone sRGB capture without changing colors
Merkost 62dbed0
Add realistic iPhone frames to the editor and exports
Merkost File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| name: iOS capture software checks | ||
| on: | ||
| pull_request: | ||
| workflow_dispatch: | ||
| permissions: | ||
| contents: read | ||
| concurrency: | ||
| group: ios-capture-${{ github.event.pull_request.number || github.ref }} | ||
| cancel-in-progress: true | ||
| jobs: | ||
| macos: | ||
| runs-on: macos-15 | ||
| timeout-minutes: 30 | ||
| steps: | ||
| - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e | ||
| with: | ||
| node-version: '22' | ||
| cache: npm | ||
| - name: Assert macOS and install locked test dependencies | ||
| run: | | ||
| test "$(uname -s)" = Darwin | ||
| npm ci --ignore-scripts | ||
| node node_modules/ffmpeg-static/install.js | ||
| - run: npm run build:ios-helper | ||
| - run: npm run test:ios-native | ||
| - run: npx tsc --noEmit | ||
| - run: npm test | ||
| - run: npm run lint | ||
| - run: npm run i18n:check | ||
| - name: Helper self-test without capture or TCC | ||
| run: | | ||
| if [ "$(uname -m)" = arm64 ]; then | ||
| electron/native/bin/darwin-arm64/recordly-ios-device-helper --self-test | ||
| else | ||
| electron/native/bin/darwin-x64/recordly-ios-device-helper --self-test | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| # iPhone and iPad USB capture | ||
|
|
||
| Recordly contains a development-preview capture path for recording the screen stream that macOS receives from a connected iPhone or iPad. The feature remains disabled by default in release packaged builds while physical-device and release gates are incomplete. An isolated local test build enables it for development validation. | ||
|
|
||
| ## Development preview | ||
|
|
||
| The preview requires macOS 14 or later, Xcode/Swift build tools and Recordly's normal development dependencies. Build the helper before starting the app: | ||
|
|
||
| ```bash | ||
| npm run build:ios-helper | ||
| RECORDLY_ENABLE_IOS_CAPTURE=1 npm run dev | ||
| ``` | ||
|
|
||
| `RECORDLY_ENABLE_IOS_CAPTURE` is read by the Electron main process and is honored only in an unpackaged development run. Setting it does not enable the feature in a packaged application. | ||
|
|
||
| Connect one iPhone or iPad using a data-capable USB cable, unlock it and approve the system Trust prompt if one appears. Open the source picker and choose **iPhone / iPad**, then select the device. Discovery stays active while the launcher is open, including when the picker is closed. **Refresh** rebuilds idle discovery and shows progress while waiting for devices. Close QuickTime or another application if it already owns the device. An empty list does not prove that the device is locked or untrusted; use Refresh after checking the cable and system prompts. | ||
|
|
||
| Recordly prepares one device at a time and waits for real video samples before showing Ready. The preview is an aspect-fit, low-bandwidth framing preview. Its longest edge is limited to 480 pixels and it updates at no more than five frames per second; recording uses the native device stream rather than the preview image. Hiding the picker stops preview work without stopping a prepared or active recording. | ||
|
|
||
| If Camera or Microphone permission was denied, review Recordly's access in macOS **System Settings → Privacy & Security**, grant the permission needed for the selected inputs, and relaunch before preparing again. Device-screen capture does not require granting desktop Screen Recording permission. Installed-app prompt identity still requires the physical release checks below. | ||
|
|
||
| Keep the phone in one orientation during a take. A detected format or transform change stops before incompatible samples are appended and preserves the preceding valid media as an interrupted take. Some physical rotations may change only the displayed content and cannot be detected reliably. | ||
|
|
||
| ### Status bar for showcases | ||
|
|
||
| The familiar showcase status bar uses **9:41** with full signal and battery indicators. On a physical iPhone, this is controlled by iOS during USB screen capture; Recordly has no documented public API for forcing those values. The user confirmed this appearance on the connected iPhone during the 8 September 2026 Recordly test. Check the phone's status bar after preparation; this single-device result is not a guarantee for other devices or OS versions. | ||
|
|
||
| QuickTime's **File → New Movie Recording** preview is a known way to trigger this appearance: choose your iPhone in the capture-device selection menu, without pressing Record. The override lasts while that preview is open; close it before selecting the phone in Recordly if the device is busy. See [Apple's USB capture instructions](https://support.apple.com/en-gb/guide/quicktime-player/qtp356b55534/mac) and the [physical-device status bar walkthrough](https://shareshot.app/blog/Clean-9-41-Status-Bar-In-Screenshots.html). Simulator status-bar overrides apply only to simulators. | ||
|
|
||
| ## Video and audio behavior | ||
|
|
||
| The helper accepts a compatible baseline H.264 stream for passthrough, or supported 8-bit 4:2:0 SDR input for H.264 encoding. Rec.709 primaries and matrix are required; the transfer function can be Rec.709, or sRGB on macOS 15 and later. The source's transfer metadata is preserved. Unsupported codecs, color formats and clock mappings fail before or during capture with a specific status. Recordly does not promise lossless framebuffer capture, a particular frame rate, HDR, Display P3, 4K, or 60/120 fps. The displayed geometry and format come from samples actually received; observed frame rate is omitted when it has not been measured. | ||
|
|
||
| **Device audio** and **Mac narration** are separate controls. Device audio defaults on when available; narration is optional and uses a microphone identity supplied by the native helper. These settings do not reuse desktop system-audio or browser microphone IDs. If requested audio cannot be opened before recording, Recordly asks for a settings change instead of silently substituting a source. If audio disappears after video begins, the valid video is retained and the result reports the interruption. | ||
|
|
||
| Device audio and narration are stored as native sidecars and aligned to the video using common-clock timing. A completed take has one mixed editor soundtrack: one source uses unity gain; two sources use 0.5 linear gain each. The video stream is copied during normal audio assembly. Original source media remains available for recovery under the normal session retention policy. | ||
|
|
||
| The mobile source does not support webcam capture, pause/resume, touch control, wireless capture or simultaneous desktop capture. Keyboard, menu and HUD actions reject unsupported operations rather than presenting a false state. Desktop capture preferences are preserved when switching sources. | ||
|
|
||
| ## Recording and editor workflow | ||
|
|
||
| The existing countdown runs only after the source is Ready. Cancelling the countdown keeps the prepared preview and creates no media. The recording timer starts only after the helper accepts the first eligible video sample. Stop enters **Saving recording…** until native finalization, validation, optional audio assembly, session commit and editor handoff finish. | ||
|
|
||
| A fresh mobile take opens with its transformed native aspect, no crop, zero corner radius, no cursor overlay or cursor telemetry, and no automatic mouse-driven zooms. These are initial defaults only. Saved crop, aspect, annotations, manual zooms and cursor choices remain authoritative when a project is reopened. Mobile provenance survives raw-session reopen, project save/load and Save As even when no webcam exists. | ||
|
|
||
| Interrupted recordings show the recorded outcome in the capture UI and editor. Requested audio is not described as recorded unless committed metadata confirms it. | ||
|
|
||
| ### iPhone mockup frames | ||
|
|
||
| In the editor, open **Scene → Frame → Device frame** and choose **Black** or | ||
| **White**. The iPhone 16 Pro artwork includes the bezel, metallic edges, side | ||
| buttons and Dynamic Island. **None** restores the unframed recording. Use | ||
| **Padding** for space around the phone and **Shadow** to adjust its depth on the | ||
| background. | ||
|
|
||
| The frame follows the original recording orientation and keeps the phone and | ||
| video proportions intact. A cropped recording with a different aspect fits inside | ||
| the phone with black letterboxing. Frame selection is stored in projects, | ||
| preferences and presets, and is included in thumbnails, MP4 and GIF output. | ||
| The artwork is bundled locally with its [MIT license](../public/third-party/Maya-LICENSE.txt). | ||
|
|
||
| ## Recovery and privacy | ||
|
|
||
| The main process owns session directories and validates every media path. Native media, timing checkpoints and an atomic journal are retained when finalization fails or the helper, renderer or parent process exits unexpectedly. The launcher no longer displays a device recovery section. Existing saved or interrupted media is retained; a damaged final fragment, power loss or filesystem failure may still be unrecoverable. | ||
|
|
||
| Capture, preview, audio and diagnostics remain local. Diagnostics are opt-in and omit device names, raw device identifiers, home-directory paths and media content. Notifications and sensitive screen content can appear in a recording. Recordly does not automatically suppress or redact them, and it does not bypass protected content. | ||
|
|
||
| ## Current verification status | ||
|
|
||
| Software tests cover protocol validation, native writers and inspection, preview bounds, controller lifecycle, IPC authorization, storage/finalization, persistence, recovery and renderer routing. Synthetic media verifies geometry, timing failures, audio offsets, final video packet preservation and manifest reopen. These checks do not prove physical iPhone/iPad behavior. | ||
|
|
||
| G1 and G2 have **Partial** evidence from one USB iPhone on the Apple Silicon development host. Its 1206 × 2622 H.264 High-profile stream negotiates to `420v` with Rec.709 primaries/matrix and sRGB transfer. After the transfer-policy fix, Ready and a live preview were directly observed; the user confirmed that recording completed and opened in the editor. Direct editor inspection then showed a saved recording, recorded device-audio metadata and approximately 20.2 seconds of clip duration. A synthetic native round trip separately verifies the same geometry, color tags and decoded flat patches. | ||
|
|
||
| G3 and G4 remain **Not tested**. Audible device-audio playback, narration, audio sync, export fidelity, five- and thirty-minute runs, physical unplug/rotation behavior, Intel runtime, signed/notarized clean-account permission and installation checks, manual keyboard/screen-reader/reduced-motion review and the full platform matrix remain unverified. No release gate has passed. See [implementation evidence](testing/ios-usb-capture-implementation.md), [feasibility evidence](testing/ios-usb-capture-feasibility.md), and the [acceptance matrix](testing/ios-usb-capture-matrix.md). |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
Repository: webadderallorg/Recordly
Length of output: 17125
🏁 Script executed:
Repository: webadderallorg/Recordly
Length of output: 3951
Make the format check required.
.github/workflows/quality.ymlrunsnpm run format:checkwithcontinue-on-error: true, so formatting is not enforced. Removecontinue-on-erroror add a required format check to this workflow.🤖 Prompt for AI Agents