diff --git a/.github/workflows/mentra-app-ios-build.yml b/.github/workflows/mentra-app-ios-build.yml index d6fee5d864..b9bcf50e83 100644 --- a/.github/workflows/mentra-app-ios-build.yml +++ b/.github/workflows/mentra-app-ios-build.yml @@ -106,7 +106,18 @@ jobs: - name: Install CocoaPods dependencies working-directory: ./mobile/ios - run: pod install + run: | + # Self-heal a corrupted prebuilt-framework cache: the cached Pods can be + # valid-by-lockfile-key yet missing the React-Core-prebuilt React.xcframework + # Info.plist (a known RN-0.83 prebuilt + CocoaPods cache flake on the + # self-hosted runners). On failure, clear the project Pods + the global pod + # cache and reinstall fresh. Happy path is unchanged. + pod install || { + echo "::warning::pod install failed (likely a stale/corrupt prebuilt cache) — clearing caches and retrying" + rm -rf Pods Podfile.lock + pod cache clean --all || true + pod install --repo-update + } - name: List available simulators run: xcrun simctl list devices available diff --git a/agents/island-facade-buildout.md b/agents/island-facade-buildout.md new file mode 100644 index 0000000000..2dc0e70c4a --- /dev/null +++ b/agents/island-facade-buildout.md @@ -0,0 +1,106 @@ +# Island facade buildout — tracking spec + +Goal: build the OEM-facing `toolkit.*` typed facades by moving the backing logic +into `@mentra/island`, domain by domain, on branch `aisraelov/island-namespace-wifi` +(PR #3167). One branch, one commit per domain, green at every commit. + +## Two move-patterns +1. **Self-contained logic** — move the file into island, fix relative imports + (btsdk types via `../../../bluetooth-sdk/build/_internal`), wrap in a facade. + Shim the old host path if the app still imports it. (Stores, speech, logs, + permissions, incidents.) +2. **Host-service-coupled** — move the logic in. Where it needs a host capability, + prefer **owning it in island** (the keystone moved storage + the status store + + the client itself in; only `auth` + endpoints come from the host, via the + permanent `configure()` front door, NOT a `configureRuntime` adapter). Per + OS-1622, every `configureRuntime` adapter is transitional scaffolding that + **deletes itself** as its domain lands — "zero permanent adapters remain." So a + `configureRuntime` bridge is a temporary means, not the destination; aim + adapter-free. The one permanent seam is `auth.getSubjectToken`. + +Rule: stores are the Mentra-app escape hatch (`toolkit.stores.*`), NOT the OEM +contract. OEMs use the typed facade functions. + +## cloud-v2 mobile-CI integration (was fully broken on dev) +The cloud-v2 merge left the mobile CI red on dev (install died on a 404, so the +typecheck never even ran). Three fixes, all on this branch (they un-red dev too): +1. **Spurious dep** — `mobile/modules/island/package.json` declared + `"@mentra/cloud-client": "*"`; cloud-client is resolved via metro+tsconfig path + aliases, not npm, so the `*` 404'd. Removed it. +2. **island standalone build** — `postinstall` builds island via `expo-module` + (`build:module`), whose isolated tsconfig lacks the cloud-v2 aliases → fails on + cloud-v2 imports. But island's `build/` is unused (metro + tsconfig resolve + `@mentra/island` → src). Made it non-fatal in `mobile/scripts/postinstall.mjs`. +3. **cloud-v2 deps** — the mobile typecheck follows the aliases into cloud-v2 + SOURCE (`../cloud-v2/packages/*`), which import `zod`/`tweetnacl`; resolution is + file-relative so they must be in `cloud-v2/node_modules`, never installed (cloud-v2 + is a separate bun workspace). Added a `bun install` in `../cloud-v2` to the mobile + postinstall. +Don't re-introduce island's `@mentra/cloud-client` package.json dep, and keep +`island/tsconfig.json`'s cloud-v2 `paths` (a local `build:module` regenerates and +strips them — don't commit that). + +## Host-coupling reality (corrects the earlier "mechanical" optimism) +Only facades whose logic is ALREADY in island are quick wraps (done: glasses, wifi, +display.mirror, speech). The rest are HOST-SERVICE moves whose services are coupled +to host utils (i18n, theme, AlertUtils, storage, RestComms), so they need the +adapter-injection pattern (#2), not a trivial move: +- `permissions` — PermissionsUtils.tsx (1008 LOC, 24 consumers): imports i18n, theme, + AlertUtils, NotificationServiceUtils, storage. Adapter-coupled. +- `settings` — the keystone (RestComms + storage + react-native-localize + expo-device). +- `incidents`, `dev`, `gallery`, `phoneNotifications` — similar host coupling. +These are real per-domain efforts, each its own careful commit. + +## Docs (keep current as we go) +Each shipped domain is documented in the OEM docs at `docs/glasses-oems/toolkit.mdx` +(Mintlify). When a facade lands, add its surface there in the same commit. The page is +written but **not yet nav-linked in `docs/docs.json`** — it's an unreleased Phase-1 +API, so publishing to the live OEM site is gated until release (one-line nav add when +greenlit). + +## Verification per commit +`npx tsc --noEmit -p .` (resolves `@mentra/island`→src, validates the real code) + +`bun run test`. The island standalone build can't run locally (cloud-v2 `zod` not +installed in this checkout) — CI confirms it; use the proven relative-`_internal` +pattern for btsdk types. + +## Domains +| Domain | Backing logic lives | Pattern | Status | +|---|---|---|---| +| glasses.wifi | btsdk passthrough + glasses store | 1 | DONE (#3167) | +| display.mirror | island display store | 1 | DONE (#3167) | +| glasses (core) | glasses store + btsdk + ConnectionCoordinator | 1 | in progress | +| speech | STT/TTSModelManager (already island) | 1 | DONE (#3167) | +| ~~logs~~ | MentraJSLogPipeline (already island) | — | **NOT a facade** — island's pipeline is internal *miniapp*-log plumbing for MentraJSRouter; the app UI never reads it. The logging the UI uses (bug-report "send logs") is the HOST-side `logBuffer` (`mobile/src/utils/dev/logging.ts`) + `RestComms.uploadIncidentLogs` → belongs to the `incidents` domain, not a `logs` facade. Skip. | +| permissions | `utils/PermissionsUtils.tsx` (host) | 1 | todo | +| incidents | `services/bugReport/*` (host) | 1 | todo | +| dev | `utils/cloudClient/devHost.ts` + core store | 1 | todo | +| miniapps | apps store + LocalMiniappRuntime (island) + MiniappCatalog | 1/hard (WebView) | todo | +| pairing | pairing screens state machine (readiness primitive already island) | 1 (extract) | todo | +| **settings** | `stores/settings.ts` (964 LOC) + `RestComms` + `storage` | keystone | **DONE (#3167)** — settings store moved into island, `toolkit.stores.settings`. Moved **together with RestComms** (mutually coupled: settings→RestComms cloud-sync, RestComms→settings backend URL). Storage uses island's MMKV (ported `loadSubKeys`). Unblocks glasses.settings + phoneNotifications. The typed `toolkit.settings` keyed facade (get/set/onChanged) is still TODO on top of the moved store. | +| **RestComms** (v1 REST) | `services/RestComms.ts` (731 LOC) | move-with-settings | **DONE (#3167)** — moved into island with settings (the coupled pair). v1-transitional: deleted in place when v1 retires. Reads backend URL from the now-island settings store directly (no early-auth timing hack). Host `@/services/RestComms` is a shim. GlobalEventEmitter also moved in (one shared instance). | +| glasses.settings | settings store + btsdk | 2 (after settings) | blocked on settings | +| phoneNotifications | settings store + crust + permissions | 2 (after settings) | blocked on settings | +| gallery | `services/asg/gallerySyncService.ts` (~1000 LOC, hotspot) | hard | todo | +| notifications | scattered detectors → new event bus | hard (new) | todo | +| session | `cloud-client` (cloud-v2) | keystone | **DONE (#3167)** — `CloudClientService` owns the CloudClient in island (built from island UDP + MMKV secure store + `getAuth()` + endpoints via `getConfigValues()`); self-wires the `cloud`/`cloudConnection` runtime hooks; `toolkit.session` exposes status. Account ops (delete/export) deferred (still host RestComms). Host `@/services/cloudClient` is a thin wrapper keeping dev/settings endpoint resolution. | +| cloudClientStatus (store) | cloud-client types | — | **DONE (#3167)** — moved into island, `toolkit.stores.cloudClientStatus`. | +| cloud secure store (MMKV) | cloud-client KeyValueStore | — | **DONE (#3167)** — moved into island (react-native-mmkv already an island dep; adapter-free). | + +## Sequence +**Cheap (logic already in island) — DONE this PR:** glasses-core, glasses.wifi, +speech, display.mirror + 5 device-store moves. `logs` was investigated and is NOT a +facade (see table). So the cheap tier is exhausted; everything below is a +host-service move needing the `configureRuntime` adapter seam. + +**Decision point (host-coupled tier):** these move 1000+ LOC host services into +island behind adapters. The mobile engineer already pushed back on moving too much +in (routing/UI) — so align on the adapter contract BEFORE moving permissions/ +settings/bugReport, rather than doing it blind. Recommended order once greenlit: +permissions → incidents → dev → **settings keystone** (own commit; unblocks +glasses.settings + phoneNotifications) → pairing → gallery → miniapps WebView → +notifications. Last: `git merge dev`, then session + cloudClientStatus. + +This PR (#3167) is a clean, landable foundation at the cheap-tier boundary: the +core `toolkit.*` facade surface + store escape hatches, green. Land it, then +sequence the host-coupled tier deliberately. diff --git a/mintlify-docs/glasses-oems/toolkit.mdx b/mintlify-docs/glasses-oems/toolkit.mdx new file mode 100644 index 0000000000..91c447ec4f --- /dev/null +++ b/mintlify-docs/glasses-oems/toolkit.mdx @@ -0,0 +1,367 @@ +--- +title: "Mentra OEM Integration Toolkit" +description: "The toolkit.* API an OEM host calls to drive MentraOS on its glasses — connection, pairing, device + user settings, wifi, speech, display, cloud session, miniapps, permissions, notifications, and incidents, over a device-agnostic runtime." +--- + + +**Status: preview (Phase 1).** The Mentra OEM Integration Toolkit covers the full Phase-1 surface +documented below. Firmware **OTA** and the media **gallery** are intentionally +deferred to a later phase. The runtime is not yet packaged for external install — +reach out to [help@mentra.glass](mailto:help@mentra.glass) to integrate today. + + +## What it is + +The **Mentra OEM Integration Toolkit** is the API an OEM host app (the phone-side app +that talks to your glasses) calls to run MentraOS. It ships as the `@mentra/island` module and +exposes a single namespaced object, **`toolkit`**: + +```ts +import {toolkit} from "@mentra/island" +``` + +The dividing line: + +- **The host owns UI, navigation, and login.** Your screens, your router, your auth. +- **The toolkit owns the runtime and the device.** Connection, device state, wifi, + on-device speech models, the display pipeline, the miniapp runtime. + +So an OEM builds its own UI and calls `toolkit..()` for everything +device- and runtime-related. The toolkit is device-agnostic: the same calls work +across glasses models. + +## Lifecycle + +The host hands the toolkit its auth + config once, then starts it. + +```ts +import {toolkit} from "@mentra/island" + +toolkit.configure({ + // REQUIRED — the host owns login; the toolkit owns the rest. + auth: { + // Return your current (auto-refreshed) backend token. + getSubjectToken: async () => ({token: await getToken(), type: "supabase"}), + }, + // Optional cloud endpoints + OEM identity. + config: {coreUrl, runtimeUrl, oemId}, + // Optional analytics sink. + analytics: (event, props) => track(event, props), +}) + +await toolkit.start() // idempotent +// … +await toolkit.stop() // idempotent +``` + +| Call | Purpose | +|---|---| +| `toolkit.configure(opts)` | Hand the toolkit `auth` (required), `config`, `analytics`. Call once, before `start()`. | +| `toolkit.start()` | Mark the runtime started. Idempotent. | +| `toolkit.stop()` | Tear down. Idempotent. | + +`auth.getSubjectToken()` is the only must-have seam — the host owns login and returns +a fresh token on demand; the toolkit owns everything downstream. + +## `toolkit.glasses` + +Connection actions, a curated status/info read-model, capabilities, version, and the +discrete input events. + +### Connection + +```ts +await toolkit.glasses.connectDefault() // connect the last-paired glasses +await toolkit.glasses.connect(device) // connect a specific (discovered) device +await toolkit.glasses.connectSimulated() // built-in simulated glasses (dev) +await toolkit.glasses.setDefault(device) // make it the connectDefault() target +await toolkit.glasses.disconnect() +await toolkit.glasses.forget() + +// optional ring controller: +await toolkit.glasses.controller.connectDefault() +await toolkit.glasses.controller.disconnect() +await toolkit.glasses.controller.forget() +``` + +### Status & info (read-models) + +`status()` returns a snapshot projected from the runtime's glasses store, in a stable +shape that doesn't leak the internal store layout: + +```ts +const s = toolkit.glasses.status() +// { state, fullyBooted, battery, charging, +// case: {battery, charging, open, removed}, +// signal, micEnabled, vadEnabled, btClassic } + +const unsubscribe = toolkit.glasses.onStatus((s) => render(s)) + +const info = toolkit.glasses.info() +// { model, style, color, firmwareVersion, mtkFirmware, besFirmware, +// serialNumber, buildNumber, btMac } + +const caps = toolkit.glasses.capabilities() // model capability table +await toolkit.glasses.requestVersionInfo() // ask glasses to report fresh version info +``` + +### Input events + +```ts +const offButton = toolkit.glasses.onButtonPress((e) => { /* ButtonPressEvent */ }) +const offTouch = toolkit.glasses.onTouchGesture((e) => { /* TouchEvent */ }) +``` + +Every `on*()` returns an unsubscribe function. + +### `toolkit.glasses.wifi` + +```ts +const networks = await toolkit.glasses.wifi.scan() // WifiSearchResult[] +await toolkit.glasses.wifi.connect(ssid, password) // rejects with a coded error on failure +await toolkit.glasses.wifi.forget(ssid) +const status = toolkit.glasses.wifi.status() // WifiStatus snapshot +const off = toolkit.glasses.wifi.onStatus((status) => { … }) // subscribe +``` + +`connect()` propagates the bluetooth layer's coded errors (`bluetooth_powered_off`, +`request_timeout`, …) unchanged, so your UI keeps its own error mapping. + +### `toolkit.glasses.settings` + +Keyed **device** settings (brightness, head-up angle, dashboard, camera/button, +sensing, …). `set()` persists *and* auto-syncs to the connected glasses. + +```ts +toolkit.glasses.settings.get(key) +toolkit.glasses.settings.set(key, value) // also pushes to the device +toolkit.glasses.settings.onChanged(key, (v) => { … }) +toolkit.glasses.settings.descriptor(key) +toolkit.glasses.settings.available() // the device-setting keys (live list) +``` + +`available()` is the authoritative list at runtime (it varies by model/capabilities). +The current keys, their types, and defaults: + +| Key | Type | Default | Notes / values | +|---|---|---|---| +| `brightness` | number | `50` | display brightness, `0`–`100` | +| `auto_brightness` | boolean | `true` | auto-adjust brightness | +| `head_up_angle` | number | `45` | head-up activation angle in degrees, `0`–`60` | +| `screen_disabled` | boolean | `false` | turn the display off | +| `contextual_dashboard` | boolean | `true` | show the dashboard on head-up | +| `dashboard_height` | number | `4` | dashboard vertical position | +| `dashboard_depth` | number | `2` | dashboard distance | +| `use_native_dashboard` | boolean | `true` | native dashboard vs JS | +| `menu_apps` | `string[] \| null` | `null` | button-menu app package names | +| `button_photo_size` | enum | `"max"` | `low` (960×720) \| `medium` (1440×1088) \| `high` (3264×2448) \| `max` (camera maximum) | +| `button_video_settings` | object | `{ width: 1920, height: 1080, fps: 30 }` | button video recording | +| `button_camera_led` | boolean | `true` | capture LED indicator | +| `button_max_recording_time` | number | `10` | max button-triggered video length, in **minutes** (e.g. `3`, `5`, `10`, `15`, `20`) | +| `camera_fov` | object | `{ fov: 118, roi_position: 0 }` | camera field of view | +| `preferred_mic` | enum | `"auto"` | `auto` \| `glasses` \| `phone` | +| `lc3_frame_size` | enum (bytes) | `60` | `20` \| `40` \| `60` | +| `sensing_enabled` | boolean | `true` | onboard sensors | +| `voice_activity_detection_enabled` | boolean | `true` | voice-activity detection | +| `power_saving_mode` | boolean | `false` | low-power mode | +| `offline_mode` | boolean | `false` | offline operation | +| `gallery_mode` | boolean | `true` | capture-to-gallery enabled — when `true`, a camera-button press takes a photo to the on-device gallery. The Mentra app auto-manages this (enabled when the camera app is running or no other app is claiming button events); set it directly only if your host has no equivalent button-routing logic | +| `calendar_events` | array | `[]` | synced calendar events | +| `twelve_hour_time` | boolean | `true` | 12-hour vs 24-hour clock | +| `metric_system` | boolean | `false` | metric vs imperial | +| `nex_chinese_captions` | boolean | `false` | Mentra Nex: Chinese captions | +| `nex_audio_playback` | boolean | `false` | Mentra Nex: audio playback | + + +Device **identity** (paired-device id/name/address, controller address) and internal +**runtime flags** (e.g. STT-fallback state) are intentionally *not* in this surface — +read the connected device via `toolkit.glasses.info()`. + + +## `toolkit.pairing` + +First-time glasses discovery + pairing. (Reconnecting the already-paired default is +`toolkit.glasses.connectDefault()`.) + +```ts +toolkit.pairing.scan() // start scanning +toolkit.pairing.scanning() // is a scan in progress? +toolkit.pairing.searchResults() // discovered devices (snapshot) +const off = toolkit.pairing.onFound((r) => { … }) // subscribe; returns unsubscribe +await toolkit.pairing.pair(device) // connect to a discovered device (from searchResults) +await toolkit.pairing.setDefault(device) // make it the connectDefault() target +toolkit.pairing.onPairFailure((e) => { … }) +toolkit.pairing.onGlassesNotReady((e) => { … }) +``` + +## `toolkit.speech` + +On-device STT/TTS model management. + +```ts +toolkit.speech.stt.currentLanguage() +toolkit.speech.stt.languages() +toolkit.speech.stt.languageInfo() +await toolkit.speech.stt.download(/* model args */) +toolkit.speech.stt.activate(code) +toolkit.speech.stt.cancelDownload() +await toolkit.speech.stt.deleteModel(/* model args */) +toolkit.speech.stt.status() // offline-model auto-download status +toolkit.speech.stt.onStatusChanged((s) => { … }) // subscribe; returns unsubscribe + +// toolkit.speech.tts mirrors stt (no auto-download status stream). +``` + +## `toolkit.display.mirror` + +A typed read facade for a phone-side preview of the glasses screen. + +```ts +const event = toolkit.display.mirror.current() // current display event snapshot +const off = toolkit.display.mirror.onMirror((e) => { … }) // subscribe; returns unsubscribe +``` + +## `toolkit.session` + +The cloud (cloud-v2) live-session surface. island owns the cloud client — it +constructs it from island-owned transports + the `auth` you passed to +`configure()` — so this reads the session it manages. + +```ts +const s = toolkit.session.status() // { status, audioTransport } +const off = toolkit.session.onStatus((s) => { … }) // subscribe; returns unsubscribe +const live = toolkit.session.isConnected() // handshake completed? +``` + +`status` is one of `connected | connecting | reconnecting | disconnected`; +`audioTransport` is `udp | ws | offline | none`. + +```ts +await toolkit.session.account.delete() // backend emails a confirmation code +await toolkit.session.account.confirmDelete(requestId, code) // confirm the deletion +``` + +## `toolkit.settings` + +Typed keyed user settings over the island-owned settings store. + +```ts +toolkit.settings.get(key) // current value (or default) +toolkit.settings.set(key, value) // also syncs to backend (pass false to skip) +toolkit.settings.onChanged(key, (v) => { … }) // subscribe; returns unsubscribe +toolkit.settings.descriptor(key) // schema (type/default/options) +toolkit.settings.keys() // all known keys +``` + +## `toolkit.incidents` + +Bug-report / feedback submission. The OEM writes its own report screen + gathers +diagnostics (phone-state, logs, screenshots — native-coupled); island owns the +submission orchestration. + +```ts +// one-call: create -> upload logs -> notify glasses -> upload screenshots +const { incidentId } = await toolkit.incidents.file({ feedbackData, phoneState, logs, screenshots }) + +// or drive the steps yourself: +const r = await toolkit.incidents.create(feedbackData, phoneState) +await toolkit.incidents.uploadLogs(incidentId, logs) +await toolkit.incidents.uploadAttachments(incidentId, images) +await toolkit.incidents.sendFeedback(feedback) +``` + +## `toolkit.dev` + +Developer/debug surface — backend + cloud-v2 URL overrides, reconnect, version gate. + +```ts +toolkit.dev.minimumClientVersion() +toolkit.dev.backendUrl(); toolkit.dev.setBackendUrl(url) +toolkit.dev.cloudUrls(); toolkit.dev.setCloudUrls({core, runtime}) // sets + reconnects +toolkit.dev.savedUrls() +toolkit.dev.reconnectCloud() +``` + +## `toolkit.permissions` + +OS permissions — the raw check/request ops (your UI adds the rationale dialogs). + +```ts +await toolkit.permissions.check("microphone") // granted? +await toolkit.permissions.request("location") // -> granted? +await toolkit.permissions.openSettings() // open OS settings +await toolkit.permissions.requirementsForMiniapp(pkg) // -> AppletPermission[] +``` + +Feature keys: `microphone · camera · calendar · location · background_location · bluetooth · phone_state · post_notifications`. + +## `toolkit.phoneNotifications` + +Forward phone notifications to the glasses, with a per-app blocklist. **Android-only** +(getters return safe defaults on iOS). + +```ts +toolkit.phoneNotifications.enabled(); toolkit.phoneNotifications.setEnabled(true) +await toolkit.phoneNotifications.installedApps() // [{packageName, appName, icon}] +toolkit.phoneNotifications.blocklist(); toolkit.phoneNotifications.setBlocklist([...]) +await toolkit.phoneNotifications.hasListenerPermission() +await toolkit.phoneNotifications.requestListenerPermission() +``` + +## `toolkit.notifications` + +Inbound alerts island→host — conditions island detects, your UI renders. + +```ts +const off = toolkit.notifications.onNotification((n) => { + // n.kind: "miniapp_crashloop" | "version_incompatible" | "connection_failed_persistent" + // n.packageName, n.reason, n.timestamp, n.metadata +}) +``` + +## `toolkit.miniapps` + +Miniapp lifecycle. The miniapp **WebView is a host component** — island ships the +bridge primitives (`buildMiniappGlobalsScript`, `buildMentraUiShim`, the MentraJS +router), exported from `@mentra/island` directly; your app mounts the `` and +wires it to those. This facade is the lifecycle half. + +```ts +toolkit.miniapps.list() // installed miniapps (snapshot) +const off = toolkit.miniapps.onChanged((apps) => { … }) // subscribe; returns unsubscribe +await toolkit.miniapps.refresh() // re-fetch the installed list +await toolkit.miniapps.start(app, opts?) // start + foreground a miniapp +await toolkit.miniapps.stop(packageName) +await toolkit.miniapps.setForeground(packageName) +toolkit.miniapps.clearForeground() +await toolkit.miniapps.stopAll() +await toolkit.miniapps.install(url, opts?) +await toolkit.miniapps.uninstall(packageName, version?) +``` + +## `toolkit.stores.*` — escape hatch (not the OEM contract) + +The raw device-state stores are also exposed under `toolkit.stores` +(`glasses`, `display`, `core`, `connection`, `gallerySync`, `cloudClientStatus`, +`settings`) so the first-party Mentra app can keep using them directly during migration. + + +`toolkit.stores.*` is a **Mentra-app convenience, not the OEM contract.** It exposes +the internal store shape, which can change. OEMs should use the typed facades above; +prefer a facade wherever one exists. + + +## What the host provides + +The boundary is small and explicit: the host provides its **UI**, its **login**, and +optional **config** — island owns the entire runtime. The only required seam is +`auth.getSubjectToken` (passed to `configure()`); island owns token exchange, refresh, +storage, the cloud client, glasses/BLE, settings, speech, display, and the miniapp +runtime from there. + + +You may see internal `configureRuntime(...)` wiring in the first-party Mentra app — +that's **transitional** scaffolding for migrating Mentra's own screens, not part of +the OEM contract. It deletes itself as each domain lands in island. An OEM only ever +calls `configure({auth, config?})` + `start()`. + diff --git a/mobile/app.config.ts b/mobile/app.config.ts index ab8b4b037b..d3bec93e06 100644 --- a/mobile/app.config.ts +++ b/mobile/app.config.ts @@ -119,25 +119,6 @@ module.exports = ({config}: ConfigContext): Partial => { // while-in-use permission only. Blocking it avoids the Play Store // background-location declaration/video review. blockedPermissions: ["android.permission.ACCESS_BACKGROUND_LOCATION"], - intentFilters: [ - { - action: "VIEW", - autoVerify: true, - data: [ - { - scheme: "https", - host: "apps.mentra.glass", - pathPrefix: "/package/", - }, - { - scheme: "https", - host: "apps.mentraglass.com", - pathPrefix: "/package/", - }, - ], - category: ["DEFAULT", "BROWSABLE"], - }, - ], }, ios: { icon: variant.icon, @@ -147,7 +128,6 @@ module.exports = ({config}: ConfigContext): Partial => { bundleIdentifier: iosBundleId, appleTeamId: "T5XXXL6N36", ...(variant.googleServicesPlist ? {googleServicesFile: variant.googleServicesPlist} : {}), - associatedDomains: ["applinks:apps.mentra.glass", "applinks:apps.mentraglass.com"], infoPlist: { NSCameraUsageDescription: "This app needs access to your camera to capture images.", NSMicrophoneUsageDescription: diff --git a/mobile/bun.lock b/mobile/bun.lock index eb38cebfef..c842d3f903 100644 --- a/mobile/bun.lock +++ b/mobile/bun.lock @@ -257,25 +257,35 @@ "react-native-nitro-modules": "^0.31.10", }, "peerDependencies": { + "@craftzdog/react-native-buffer": "*", "@dr.pogodin/react-native-fs": "*", "@mentra/bluetooth-sdk": "*", "@mentra/cloud-client": "*", "@mentra/cloud-runtime": "*", + "@mentra/crust": "*", "@mentra/miniapp": "*", + "@react-native-community/netinfo": "*", "@types/react": "*", + "axios": "*", "expo": "*", + "expo-audio": "*", "expo-battery": "*", "expo-clipboard": "*", "expo-file-system": "*", "expo-location": "*", "expo-modules-core": "*", + "expo-notifications": "*", + "expo-task-manager": "*", "react": "*", "react-native": "*", + "react-native-localize": "*", "react-native-mmkv": "*", "react-native-nitro-bg-timer": "*", "react-native-nitro-modules": "*", + "react-native-permissions": "*", "react-native-share": "*", "react-native-udp": "*", + "react-native-wifi-reborn": "*", "react-native-zip-archive": "*", "semver": "*", "zustand": "*", diff --git a/mobile/jest.config.js b/mobile/jest.config.js index 91269795d1..abe38a1485 100644 --- a/mobile/jest.config.js +++ b/mobile/jest.config.js @@ -7,6 +7,11 @@ module.exports = { "^@/(.*)$": "/src/$1", "^@assets/(.*)$": "/assets/$1", "^@mentra/bluetooth-sdk-internal$": "/modules/bluetooth-sdk/src/_internal.ts", + // island-internal code (e.g. RestComms) reaches the full btsdk surface via the + // relative build/_internal path (the @mentra/bluetooth-sdk-internal alias + // doesn't resolve in island's standalone build). Map it to the same source so + // the jest.setup mock applies — otherwise requireActual loads the real native module. + "bluetooth-sdk/build/_internal$": "/modules/bluetooth-sdk/src/_internal.ts", "^expo/virtual/env$": "/src/test-utils/expoVirtualEnvMock.ts", "^react-native$": "/node_modules/react-native", "^crust$": "/modules/crust/src", diff --git a/mobile/jest.setup.js b/mobile/jest.setup.js index 4f82692104..4577ff372b 100644 --- a/mobile/jest.setup.js +++ b/mobile/jest.setup.js @@ -229,10 +229,52 @@ jest.mock("@dr.pogodin/react-native-fs", () => ({ writeFile: jest.fn(() => Promise.resolve()), })) +// LocalMiniappRuntime pulls heavy native modules (react-native-share, expo-*). +// requireActual'd island services that import it (e.g. GlassesStatusProjection) +// only need its forwardEvent side-effect, so stub it light here. +jest.mock("./modules/island/src/services/LocalMiniappRuntime", () => ({ + __esModule: true, + default: {forwardEvent: jest.fn()}, +})) + // Mock @mentra/island — its barrel pulls in many native modules // (react-native-share, expo-battery/clipboard/location, etc.). Tests that // only need a handful of exports get stubs here; specific tests can override. jest.mock("@mentra/island", () => { + // The glasses store moved into island; tests + the @/stores/glasses shim need its + // REAL behavior (setState/getState/subscribe), so pull the actual store in. It's + // pure (zustand + type-only btsdk imports), so it loads cleanly under the mock. + const realGlasses = jest.requireActual("./modules/island/src/stores/glasses") + const realDisplay = jest.requireActual("./modules/island/src/stores/display") + const realCore = jest.requireActual("./modules/island/src/stores/core") + const realConnection = jest.requireActual("./modules/island/src/stores/connection") + const realGallerySync = jest.requireActual("./modules/island/src/stores/gallerySync") + const realCloudStatus = jest.requireActual("./modules/island/src/stores/cloudClientStatus") + // Settings store + RestComms moved into island; tests used the real host store + // before the move, so requireActual preserves that exact behavior. + const realSettings = jest.requireActual("./modules/island/src/stores/settings") + const realRestComms = jest.requireActual("./modules/island/src/services/RestComms") + // toolkit.start() starts the island-owned device-settings -> glasses BLE sync; use + // the real one so its behavior is exercised where it now lives (not MantleManager). + const realGlassesSettingsSync = jest.requireActual("./modules/island/src/services/GlassesSettingsSync") + const realGlassesStatusProjection = jest.requireActual("./modules/island/src/services/GlassesStatusProjection") + const realOtaService = jest.requireActual("./modules/island/src/services/OtaService") + const realAudioCloudUplink = jest.requireActual("./modules/island/src/services/AudioCloudUplink") + const realDeviceEventRouter = jest.requireActual("./modules/island/src/services/DeviceEventRouter") + // Clock-skew utils moved into island; the host gallery sync + OTA checker import them + // from @mentra/island, so expose the real (pure) implementations through the mock. + const realGlassesClockSync = jest.requireActual("./modules/island/src/services/glassesClockSync") + const realGallerySyncClock = jest.requireActual("./modules/island/src/services/gallerySyncClock") + const realAsgOtaVersionUrl = jest.requireActual("./modules/island/src/services/asgOtaVersionUrl") + const realPhoneNotificationsSync = jest.requireActual("./modules/island/src/services/PhoneNotificationsSync") + // The on* event facades (button/touch/pair_failure/glasses_not_ready) are thin + // addListener wrappers in the real toolkit, so the mock delegates to the shared + // bluetoothSdkMock — emitBluetoothSdkEvent() + listener-leak counts keep working. + const {bluetoothSdkMock} = require("./src/test-utils/mockBluetoothSdk") + const subscribeVia = (eventName) => jest.fn((cb) => { + const sub = bluetoothSdkMock.addListener(eventName, cb) + return () => sub.remove() + }) const appStatusState = { apps: [], refresh: jest.fn(), @@ -249,6 +291,290 @@ jest.mock("@mentra/island", () => { return { __esModule: true, + // Real glasses store + its selectors/helpers (useGlassesStore, selectors, + // waitForGlassesState, getGlasesInfoPartial, getGlassesSystemTimeMs, predicates). + ...realGlasses, + // Real display/mirror store (useDisplayStore) — consumers need its real behavior. + ...realDisplay, + // Real core / connection / gallerySync stores (+ WebSocketStatus, selectors). + ...realCore, + ...realConnection, + ...realGallerySync, + // Real cloud-client runtime status store (useCloudClientStatusStore). + ...realCloudStatus, + // Real settings store (SETTINGS, useSettingsStore, useSetting, OFFLINE_APPLETS) + // + RestComms singleton — both moved into island. + ...realSettings, + restComms: realRestComms.default, + // Clock-skew utils (real, pure) — consumed by the host gallery sync + OTA checker. + fixGlassesClockIfSkewed: realGlassesClockSync.fixGlassesClockIfSkewed, + maybeFixGlassesClockFromVersionInfo: realGlassesClockSync.maybeFixGlassesClockFromVersionInfo, + // OTA manifest-URL resolution (real, pure) — consumed by the host OTA screens via + // the @/services/asg/asgOtaVersionUrl shim. + getAsgOtaVersionUrl: realAsgOtaVersionUrl.getAsgOtaVersionUrl, + detectClockSkew: realGallerySyncClock.detectClockSkew, + isSyncManifestEmpty: realGallerySyncClock.isSyncManifestEmpty, + CLOCK_SKEW_TOLERANCE_MS: realGallerySyncClock.CLOCK_SKEW_TOLERANCE_MS, + // The namespaced (A) host API. Mirrors the real `toolkit` object; members are + // jest.fn()s so host/screen tests can assert delegation without native btsdk. + toolkit: { + configure: jest.fn(), + start: jest.fn(() => { + realGlassesStatusProjection.startGlassesStatusProjection() + realOtaService.startOtaService() + realAudioCloudUplink.startAudioCloudUplink() + realDeviceEventRouter.startDeviceEventRouter() + realGlassesSettingsSync.startGlassesSettingsSync() + realPhoneNotificationsSync.startPhoneNotificationsSync() + return Promise.resolve() + }), + stop: jest.fn(() => { + realGlassesStatusProjection.stopGlassesStatusProjection() + realOtaService.stopOtaService() + realAudioCloudUplink.stopAudioCloudUplink() + realDeviceEventRouter.stopDeviceEventRouter() + realGlassesSettingsSync.stopGlassesSettingsSync() + realPhoneNotificationsSync.stopPhoneNotificationsSync() + return Promise.resolve() + }), + glasses: { + connectDefault: jest.fn(() => Promise.resolve()), + disconnect: jest.fn(() => Promise.resolve()), + forget: jest.fn(() => Promise.resolve()), + connect: jest.fn(() => Promise.resolve()), + connectSimulated: jest.fn(() => Promise.resolve()), + setDefault: jest.fn(() => Promise.resolve()), + reconnect: jest.fn(() => Promise.resolve(true)), + isFirstPairing: jest.fn(() => false), + controller: { + connectDefault: jest.fn(() => Promise.resolve()), + disconnect: jest.fn(() => Promise.resolve()), + forget: jest.fn(() => Promise.resolve()), + }, + // Thin passthroughs in the real facade — delegate to the shared + // bluetoothSdkMock so volume-return mocks + btsdk-call assertions work. + audio: { + getMediaVolume: jest.fn((...a) => bluetoothSdkMock.getGlassesMediaVolume(...a)), + setMediaVolume: jest.fn((...a) => bluetoothSdkMock.setGlassesMediaVolume(...a)), + setOwnAppPlaying: jest.fn((...a) => bluetoothSdkMock.setOwnAppAudioPlaying(...a)), + }, + status: jest.fn(() => ({state: "disconnected"})), + onStatus: jest.fn(() => () => {}), + info: jest.fn(() => ({})), + capabilities: jest.fn(() => ({})), + requestVersionInfo: jest.fn(() => Promise.resolve()), + onButtonPress: subscribeVia("button_press"), + onTouchGesture: subscribeVia("touch_event"), + wifi: { + scan: jest.fn(() => Promise.resolve([])), + connect: jest.fn(() => Promise.resolve()), + forget: jest.fn(() => Promise.resolve()), + status: jest.fn(() => ({state: "disconnected"})), + onStatus: jest.fn(() => () => {}), + }, + settings: { + get: jest.fn(() => undefined), + set: jest.fn(() => Promise.resolve({is_ok: () => true, is_error: () => false})), + onChanged: jest.fn(() => () => {}), + descriptor: jest.fn(() => undefined), + available: jest.fn(() => []), + }, + }, + speech: { + restartTranscriber: jest.fn(() => Promise.resolve()), + stt: { + currentLanguage: jest.fn(() => "en"), + languages: jest.fn(() => []), + languageInfo: jest.fn(() => Promise.resolve([])), + download: jest.fn(() => Promise.resolve()), + activate: jest.fn(), + cancelDownload: jest.fn(() => Promise.resolve()), + deleteModel: jest.fn(() => Promise.resolve()), + status: jest.fn(() => null), + onStatusChanged: jest.fn(() => () => {}), + }, + tts: { + currentLanguage: jest.fn(() => "en"), + languages: jest.fn(() => []), + languageInfo: jest.fn(() => Promise.resolve([])), + download: jest.fn(() => Promise.resolve()), + activate: jest.fn(), + cancelDownload: jest.fn(() => Promise.resolve()), + deleteModel: jest.fn(() => Promise.resolve()), + }, + }, + display: { + mirror: { + current: jest.fn(() => null), + onMirror: jest.fn(() => () => {}), + view: jest.fn(() => "main"), + setView: jest.fn(), + }, + text: jest.fn(() => Promise.resolve()), + clear: jest.fn(() => Promise.resolve()), + }, + notifications: { + onNotification: jest.fn(() => () => {}), + }, + permissions: { + check: jest.fn(() => Promise.resolve(false)), + request: jest.fn(() => Promise.resolve(false)), + openSettings: jest.fn(() => Promise.resolve()), + requirementsForMiniapp: jest.fn(() => Promise.resolve([])), + }, + phoneNotifications: { + enabled: jest.fn(() => false), + setEnabled: jest.fn(() => Promise.resolve({is_ok: () => true})), + installedApps: jest.fn(() => Promise.resolve([])), + blocklist: jest.fn(() => []), + setBlocklist: jest.fn(() => Promise.resolve({is_ok: () => true})), + hasListenerPermission: jest.fn(() => Promise.resolve(false)), + requestListenerPermission: jest.fn(() => Promise.resolve()), + }, + pairing: { + scan: jest.fn(), + scanning: jest.fn(() => false), + searchResults: jest.fn(() => []), + onFound: jest.fn(() => () => {}), + pair: jest.fn(() => Promise.resolve()), + setDefault: jest.fn(() => Promise.resolve()), + onPairFailure: subscribeVia("pair_failure"), + onGlassesNotReady: subscribeVia("glasses_not_ready"), + }, + miniapps: { + list: jest.fn(() => []), + onChanged: jest.fn(() => () => {}), + refresh: jest.fn(() => Promise.resolve()), + start: jest.fn(() => Promise.resolve(true)), + stop: jest.fn(() => Promise.resolve()), + setForeground: jest.fn(() => Promise.resolve()), + clearForeground: jest.fn(), + stopAll: jest.fn(() => Promise.resolve({is_ok: () => true})), + install: jest.fn(() => Promise.resolve({is_ok: () => true})), + uninstall: jest.fn(() => Promise.resolve({is_ok: () => true})), + }, + session: { + status: jest.fn(() => ({status: "disconnected", audioTransport: "none"})), + onStatus: jest.fn(() => () => {}), + isConnected: jest.fn(() => false), + account: { + delete: jest.fn(() => Promise.resolve({is_ok: () => true, is_error: () => false})), + confirmDelete: jest.fn(() => Promise.resolve({is_ok: () => true, is_error: () => false})), + }, + }, + settings: { + get: jest.fn(() => undefined), + set: jest.fn(() => Promise.resolve({is_ok: () => true, is_error: () => false})), + onChanged: jest.fn(() => () => {}), + descriptor: jest.fn(() => undefined), + keys: jest.fn(() => []), + }, + dev: { + minimumClientVersion: jest.fn(() => Promise.resolve({is_ok: () => true, value: {required: "0", recommended: "0"}})), + backendUrl: jest.fn(() => undefined), + setBackendUrl: jest.fn(() => Promise.resolve({is_ok: () => true, is_error: () => false})), + cloudUrls: jest.fn(() => ({})), + setCloudUrls: jest.fn(), + savedUrls: jest.fn(() => []), + reconnectCloud: jest.fn(), + getMemoryMB: jest.fn(() => 0), + }, + incidents: { + file: jest.fn(() => Promise.resolve({incidentId: "test"})), + notifyGlasses: jest.fn(), + create: jest.fn(() => Promise.resolve({is_ok: () => true, is_error: () => false, value: {incidentId: "test"}})), + uploadLogs: jest.fn(() => Promise.resolve({is_ok: () => true, is_error: () => false})), + uploadAttachments: jest.fn(() => Promise.resolve({is_ok: () => true, is_error: () => false})), + sendFeedback: jest.fn(() => Promise.resolve({is_ok: () => true, is_error: () => false})), + }, + ota: { + updateAvailable: jest.fn(() => null), + status: jest.fn(() => null), + onUpdateAvailable: jest.fn(() => () => {}), + onStatus: jest.fn(() => () => {}), + // Thin passthroughs — delegate to the shared bluetoothSdkMock so btsdk-call + // assertions (e.g. ota/progress.test) keep working. + install: jest.fn((...a) => bluetoothSdkMock.startOtaUpdate(...a)), + retry: jest.fn((...a) => bluetoothSdkMock.retryOtaVersionCheck(...a)), + }, + gallery: { + status: jest.fn(() => ({})), + onStatus: jest.fn(() => () => {}), + onNotice: jest.fn(() => () => {}), + sync: jest.fn(() => Promise.resolve()), + cancel: jest.fn(() => Promise.resolve()), + }, + stores: { + glasses: realGlasses.useGlassesStore, + display: realDisplay.useDisplayStore, + core: realCore.useCoreStore, + connection: realConnection.useConnectionStore, + gallerySync: realGallerySync.useGallerySyncStore, + cloudClientStatus: realCloudStatus.useCloudClientStatusStore, + settings: realSettings.useSettingsStore, + }, + }, + // Shared process-wide event bus (moved into island) — the REAL island + // instance (not a fresh one) so the instance RestComms emits on is the same + // one tests listen on across the boundary. + GlobalEventEmitter: jest.requireActual("./modules/island/src/utils/GlobalEventEmitter").default, + // Gallery cluster moved into island; host consumers (GalleryScreen, gallery-settings, + // NetworkMonitoring, MantleManager) import these from @mentra/island. Stub them here + // so those screens/services load under the mock without native deps. The gallery + // service's own jest test imports the REAL implementations by relative path instead. + gallerySyncService: { + initialize: jest.fn(), + startSync: jest.fn(() => Promise.resolve()), + cancelSync: jest.fn(() => Promise.resolve()), + isSyncing: jest.fn(() => false), + isSyncStarting: jest.fn(() => false), + queryGlassesGalleryStatus: jest.fn(() => Promise.resolve()), + }, + localStorageService: { + getDownloadedFiles: jest.fn(() => Promise.resolve([])), + convertToPhotoInfo: jest.fn((file) => file), + convertToDownloadedFile: jest.fn((file) => file), + saveDownloadedFile: jest.fn(() => Promise.resolve()), + deleteDownloadedFile: jest.fn(() => Promise.resolve()), + clearAllFiles: jest.fn(() => Promise.resolve()), + getSyncState: jest.fn(() => Promise.resolve({total_downloaded: 0, total_size: 0})), + updateSyncState: jest.fn(() => Promise.resolve()), + }, + asgCameraApi: { + setServer: jest.fn(), + syncWithServer: jest.fn(() => Promise.resolve()), + downloadCapture: jest.fn(() => Promise.resolve()), + deleteFilesFromServer: jest.fn(() => Promise.resolve()), + }, + gallerySettingsService: { + getSettings: jest.fn(() => Promise.resolve({})), + getAutoSaveToCameraRoll: jest.fn(() => Promise.resolve(false)), + setAutoSaveToCameraRoll: jest.fn(() => Promise.resolve()), + }, + MediaLibraryPermissions: { + checkPermission: jest.fn(() => Promise.resolve(true)), + requestPermission: jest.fn(() => Promise.resolve(true)), + saveToLibrary: jest.fn(() => Promise.resolve()), + }, + emitGalleryNotice: jest.fn(), + onGalleryNotice: jest.fn(() => () => {}), + // island now owns the cloud client (keystone #5); the host wrapper delegates + // to this. Mocked so host/service tests don't construct a real CloudClient. + audioPlaybackService: { + play: jest.fn(() => Promise.resolve()), + stopForApp: jest.fn(), + }, + cloudClientService: { + init: jest.fn(), + reconnect: jest.fn(), + startManagedPhoto: jest.fn(() => Promise.resolve({})), + awaitManagedPhotoReady: jest.fn(() => Promise.resolve({})), + startManagedStream: jest.fn(() => Promise.resolve({})), + getManagedStreamStatus: jest.fn(() => Promise.resolve({})), + stopManagedStream: jest.fn(() => Promise.resolve()), + isConnected: jest.fn(() => false), + onConnectionChange: jest.fn(() => () => {}), + }, BgTimer: { setInterval: jest.fn((callback, delay) => setInterval(callback, delay)), clearInterval: jest.fn((id) => clearInterval(id)), @@ -349,6 +675,20 @@ jest.mock("@mentra/island", () => { WIFI: "wifi", }, localDisplayManager: {}, + phonePhotoCoordinator: { + owns: jest.fn(() => false), + handlePhotoError: jest.fn(), + takePhoto: jest.fn(() => Promise.resolve({photoUrl: "", mimeType: "image/jpeg", size: 0, requestId: "x"})), + }, + phoneStreamCoordinator: { + owns: jest.fn(() => false), + handleGlassesStatus: jest.fn(), + handleKeepAliveAck: jest.fn(), + startUnmanaged: jest.fn(() => Promise.resolve({streamId: "x", status: "streaming"})), + startManaged: jest.fn(() => Promise.resolve({streamId: "x", status: "streaming"})), + stop: jest.fn(() => Promise.resolve()), + setStatusSubscriber: jest.fn(), + }, localMiniappRuntime: { cleanup: jest.fn(), forwardEvent: jest.fn(), @@ -371,6 +711,22 @@ jest.mock("@mentra/island", () => { }, throttle: jest.fn((callback) => callback), configureRuntime: jest.fn(), + configureLauncher: jest.fn(), + miniappLauncher: { + ensureConnected: jest.fn(() => Promise.resolve(true)), + ensureRunning: jest.fn(() => Promise.resolve(true)), + resolveBundle: jest.fn(() => Promise.resolve(null)), + stop: jest.fn(() => Promise.resolve()), + }, + ensureMiniappEngine: jest.fn(() => ({ + router: {logRing: {snapshot: jest.fn(() => [])}}, + uiRouter: {bindWebView: jest.fn(), unbindWebView: jest.fn(), notifyReopen: jest.fn()}, + crashController: {}, + })), + getMiniappEngine: jest.fn(() => null), + sttModelManager: { + isModelAvailable: jest.fn(() => Promise.resolve(false)), + }, getRuntimeHooks: jest.fn(() => ({})), ISLAND_SETTINGS_KEYS: {}, normalizeManifestPermissions: jest.fn(), diff --git a/mobile/modules/island/package.json b/mobile/modules/island/package.json index e11be96591..49dd83d39d 100644 --- a/mobile/modules/island/package.json +++ b/mobile/modules/island/package.json @@ -55,11 +55,21 @@ "semver": "*", "zustand": "*", "expo-battery": "*", + "expo-audio": "*", "expo-clipboard": "*", "expo-location": "*", + "expo-task-manager": "*", + "expo-notifications": "*", "react-native-share": "*", + "react-native-localize": "*", + "react-native-permissions": "*", + "react-native-wifi-reborn": "*", + "@react-native-community/netinfo": "*", "@dr.pogodin/react-native-fs": "*", + "@craftzdog/react-native-buffer": "*", + "axios": "*", "@mentra/bluetooth-sdk": "*", + "@mentra/crust": "*", "@mentra/cloud-client": "*", "@mentra/cloud-runtime": "*", "@mentra/miniapp": "*" diff --git a/mobile/modules/island/src/facades/dev.ts b/mobile/modules/island/src/facades/dev.ts new file mode 100644 index 0000000000..60c54e7052 --- /dev/null +++ b/mobile/modules/island/src/facades/dev.ts @@ -0,0 +1,79 @@ +/** + * dev facade — `toolkit.dev`: developer/debug surface over island-owned state. + * Backend/cloud URL overrides read+write the island settings store; the cloud + * URL setters reconnect the island-owned cloud client onto the new endpoints; + * `minimumClientVersion` hits the island RestComms. + * + * NOTE: the dev-time endpoint *resolution* (Metro-host auto-detect, env defaults) + * still lives host-side in `@/services/cloudClient` during the migration; this + * facade manages the explicit overrides + reconnect. + */ +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import {useSettingsStore, SETTINGS} from "../stores/settings" +import {cloudClientService} from "../services/CloudClientService" +import restComms from "../services/RestComms" +import {getConfigValues} from "../runtime/bootstrap" + +/** + * Reconnect the cloud client onto the current cloud-URL overrides. Explicit + * partial overrides merge over the boot-resolved config (which carries the + * host's Metro/env URL resolution); both empty clears the live override. + */ +function applyCloudUrlReconnect(): void { + const s = useSettingsStore.getState() + const core = trimSetting(s.getSetting(SETTINGS.cloud_core_url.key)) + const runtime = trimSetting(s.getSetting(SETTINGS.cloud_runtime_url.key)) + + if (!core && !runtime) { + cloudClientService.reconnect(null) + return + } + + const config = getConfigValues() + const resolvedCore = core || config.coreUrl?.trim() + const resolvedRuntime = runtime || config.runtimeUrl?.trim() + if (resolvedCore && resolvedRuntime) { + cloudClientService.reconnect({core: resolvedCore, runtime: resolvedRuntime}) + } else { + // Keep the previous client if the host did not provide enough boot config to + // complete a partial override; reconnecting with null would silently discard + // the explicit side the user just set. + console.warn("toolkit.dev.setCloudUrls: partial cloud override missing boot-resolved sibling endpoint") + } +} + +function trimSetting(value: unknown): string | undefined { + return typeof value === "string" && value.trim().length > 0 ? value.trim() : undefined +} + +export const dev = { + /** The backend's required/recommended client version. */ + minimumClientVersion: () => restComms.getMinimumClientVersion(), + + /** The configured backend (core REST) URL override, if any. */ + backendUrl: (): string | undefined => useSettingsStore.getState().getSetting(SETTINGS.backend_url.key), + /** Override the backend (core REST) URL. */ + setBackendUrl: (url: string) => useSettingsStore.getState().setSetting(SETTINGS.backend_url.key, url), + + /** The cloud-v2 core/runtime URL overrides. */ + cloudUrls: (): {core?: string; runtime?: string} => { + const s = useSettingsStore.getState() + return {core: s.getSetting(SETTINGS.cloud_core_url.key), runtime: s.getSetting(SETTINGS.cloud_runtime_url.key)} + }, + /** Override the cloud-v2 URLs and reconnect the live client onto them. */ + setCloudUrls: (urls: {core?: string; runtime?: string}) => { + const s = useSettingsStore.getState() + if (urls.core !== undefined) s.setSetting(SETTINGS.cloud_core_url.key, urls.core) + if (urls.runtime !== undefined) s.setSetting(SETTINGS.cloud_runtime_url.key, urls.runtime) + applyCloudUrlReconnect() + }, + + /** The saved backend URLs (the dev URL-switcher list). */ + savedUrls: (): string[] => useSettingsStore.getState().getSetting(SETTINGS.saved_backend_urls.key) ?? [], + + /** Tear down + rebuild the live cloud client (the dev "reconnect" button). */ + reconnectCloud: (): void => applyCloudUrlReconnect(), + + /** Current native (BLE-process) memory usage in MB — a dev/diagnostics gauge. */ + getMemoryMB: (): number => BluetoothSdk.getMemoryMB(), +} diff --git a/mobile/modules/island/src/facades/display.ts b/mobile/modules/island/src/facades/display.ts new file mode 100644 index 0000000000..f2d5183a5a --- /dev/null +++ b/mobile/modules/island/src/facades/display.ts @@ -0,0 +1,17 @@ +/** + * display facade — `toolkit.display`: the host/OEM display surface. `mirror` + * exposes the mirrored-view read-model; `text`/`clear` are raw passthroughs to the + * glasses display for dev tools and escape-hatch use (normal rendering goes through + * the island DisplayProcessor / layout system, not these). + */ +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import {displayMirror} from "./displayMirror" + +export const display = { + /** Mirrored-view read-model (current/onMirror/view/setView). */ + mirror: displayMirror, + /** Write raw text to the glasses display at (x, y) with a font size. */ + text: (...args: Parameters) => BluetoothSdk.displayText(...args), + /** Clear the glasses display. */ + clear: (): Promise => BluetoothSdk.clearDisplay(), +} diff --git a/mobile/modules/island/src/facades/displayMirror.ts b/mobile/modules/island/src/facades/displayMirror.ts new file mode 100644 index 0000000000..6b226b7912 --- /dev/null +++ b/mobile/modules/island/src/facades/displayMirror.ts @@ -0,0 +1,32 @@ +/** + * glasses display mirror — the typed read facade over the (now island-owned) + * display store, for a phone-side preview of the glasses screen + * (`toolkit.display.mirror`). The store is fed by the display paths + * (LocalDisplayManager + the cloud path) via `setDisplayEvent`; this facade is the + * read side: `current()` (snapshot) + `onMirror(cb)` (subscribe). The raw store is + * also exposed as `toolkit.displayStore` (the Mentra-app escape hatch). + */ +import {useDisplayStore} from "../stores/display" + +export type DisplayMirrorEvent = Record & {view?: string} + +export const displayMirror = { + /** The display event currently shown on the previewed view (snapshot). */ + current(): DisplayMirrorEvent { + return useDisplayStore.getState().currentEvent + }, + + /** Subscribe to changes of the previewed display event; returns an unsubscribe. */ + onMirror(cb: (event: DisplayMirrorEvent) => void): () => void { + return useDisplayStore.subscribe((s) => s.currentEvent, cb) + }, + + /** Which view is previewed — the main screen or the dashboard. */ + view(): string { + return useDisplayStore.getState().view + }, + /** Switch the previewed view between `"main"` and `"dashboard"`. */ + setView(view: "main" | "dashboard"): void { + useDisplayStore.getState().setView(view) + }, +} diff --git a/mobile/modules/island/src/facades/gallery.ts b/mobile/modules/island/src/facades/gallery.ts new file mode 100644 index 0000000000..088dd38acd --- /dev/null +++ b/mobile/modules/island/src/facades/gallery.ts @@ -0,0 +1,54 @@ +/** + * gallery facade — `toolkit.gallery`: the OEM-facing gallery-sync surface. Island's + * gallerySyncService orchestrates the sync (hotspot → wifi → HTTP fetch → process → + * store); this facade exposes the projected status + the structured notices the host + * renders its own UI from, plus the sync/cancel actions. No host-injected UI — the host + * owns all alerts/navigation/i18n (it renders them off `onNotice`). + */ +import { + useGallerySyncStore, + selectSyncProgress, + selectIssyncing, + selectGlassesGalleryStatus, +} from "../stores/gallerySync" +import {gallerySyncService} from "../services/asg/gallerySyncService" +import {onGalleryNotice, type GalleryNotice} from "../services/asg/galleryNotices" + +function projectStatus() { + const s = useGallerySyncStore.getState() + return { + ...selectSyncProgress(s), + isSyncing: selectIssyncing(s), + glassesGallery: selectGlassesGalleryStatus(s), + queueLength: s.queue.length, + queueIndex: s.queueIndex, + } +} + +export type GallerySyncStatus = ReturnType + +export const gallery = { + /** Snapshot of the current sync state + progress. */ + status: (): GallerySyncStatus => projectStatus(), + /** Subscribe to sync-state changes; returns an unsubscribe. Deduped on the projected + * status (the gallerySync store mutates many unrelated fields during a sync), so the + * host only re-renders when the projection actually changes — matches the other + * read-model facades. */ + onStatus: (cb: (status: GallerySyncStatus) => void): (() => void) => { + let last = JSON.stringify(projectStatus()) + return useGallerySyncStore.subscribe(() => { + const status = projectStatus() + const serialized = JSON.stringify(status) + if (serialized === last) return + last = serialized + cb(status) + }) + }, + /** Subscribe to user-actionable notices (host renders alerts / settings deep-links). */ + onNotice: (cb: (notice: GalleryNotice) => void): (() => void) => onGalleryNotice(cb), + + /** Start a gallery sync (the host gates connectivity/permissions first). */ + sync: (): Promise => gallerySyncService.startSync(), + /** Cancel an in-progress sync. */ + cancel: (): Promise => gallerySyncService.cancelSync(), +} diff --git a/mobile/modules/island/src/facades/glasses.ts b/mobile/modules/island/src/facades/glasses.ts new file mode 100644 index 0000000000..1d4a05ec94 --- /dev/null +++ b/mobile/modules/island/src/facades/glasses.ts @@ -0,0 +1,144 @@ +/** + * glasses facade — the core `toolkit.glasses.*` surface: connection actions (over + * the bluetooth-sdk passthrough), a curated status/info read-model (projected from + * the island-owned glasses store), capabilities (from the model table), and the + * discrete input events. The `wifi` sub-facade is nested here. + * + * Read-models project the raw glasses store into the stable shape OEM UIs consume, + * so the device store's field layout doesn't leak into every screen. + */ +// Internal btsdk surface — connectSimulated/reconnect/controller live on the full +// surface, not the public entry. Relative path (the alias doesn't resolve in island's +// standalone build); jest moduleNameMapper + tsconfig both resolve it. +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import type {ButtonPressEvent, TouchEvent} from "../../../bluetooth-sdk/build/_internal" +import {useGlassesStore} from "../stores/glasses" +import {useSettingsStore, SETTINGS} from "../stores/settings" +import {isGlassesConnected, isGlassesReady} from "../services/GlassesReadiness" +import {pushAllBluetoothSettings} from "../services/GlassesSettingsSync" +import {getModelCapabilities, type DeviceTypes} from "../types" +import {glassesWifi} from "./glassesWifi" +import {glassesSettings} from "./glassesSettings" + +function projectStatus() { + const s = useGlassesStore.getState() + return { + state: s.connection.state, + fullyBooted: isGlassesReady(s.connection), + battery: s.batteryLevel, + charging: s.charging, + case: {battery: s.caseBatteryLevel, charging: s.caseCharging, open: s.caseOpen, removed: s.caseRemoved}, + signal: s.signalStrength, + micEnabled: s.micEnabled, + vadEnabled: s.voiceActivityDetectionEnabled, + btClassic: s.bluetoothClassicConnected, + } +} + +function projectInfo() { + const s = useGlassesStore.getState() + return { + model: s.deviceModel, + style: s.style, + color: s.color, + firmwareVersion: s.firmwareVersion, + mtkFirmware: s.mtkFirmwareVersion, + besFirmware: s.besFirmwareVersion, + serialNumber: s.serialNumber, + buildNumber: s.buildNumber, + btMac: s.bluetoothMacAddress, + } +} + +export type GlassesStatusSnapshot = ReturnType +export type GlassesInfoSnapshot = ReturnType + +export const glasses = { + // --- connection (bluetooth-sdk passthrough) --- + /** + * Reconnect the default wearable. Seeds the phone's current device settings to + * native first, so they're primed before the connect handshake replays them to + * the glasses (this used to be a host-side step before `connectDefault`). + */ + connectDefault: async (): Promise => { + await pushAllBluetoothSettings() + return BluetoothSdk.connectDefault() + }, + disconnect: (): Promise => BluetoothSdk.disconnect(), + forget: (): Promise => BluetoothSdk.forget(), + /** Connect to a specific (discovered) device. */ + connect: async (...args: Parameters): Promise => { + await pushAllBluetoothSettings() + return BluetoothSdk.connect(...args) + }, + /** Connect the built-in simulated glasses (dev/testing). */ + connectSimulated: (): Promise => BluetoothSdk.connectSimulated(), + /** Set a device as the `connectDefault()` target. */ + setDefault: (...args: Parameters) => BluetoothSdk.setDefaultDevice(...args), + /** + * Reconnect to the saved default glasses. Resolves `false` when there's nothing to + * reconnect to (no default paired), `true` when already connected or after kicking + * off the connect. The host gates connectivity/permissions (its UI) before calling. + */ + reconnect: async (): Promise => { + const defaultWearable = useSettingsStore.getState().getSetting(SETTINGS.default_wearable.key) + if (!defaultWearable) return false + if (isGlassesConnected(useGlassesStore.getState().connection)) return true + await pushAllBluetoothSettings() + await BluetoothSdk.connectDefault() + return true + }, + /** True when no glasses has ever been paired (no saved default wearable) — e.g. to + * route a first-run host into the pairing/onboarding flow. */ + isFirstPairing: (): boolean => !useSettingsStore.getState().getSetting(SETTINGS.default_wearable.key), + + // --- read-model (projected from the island-owned glasses store) --- + status: (): GlassesStatusSnapshot => projectStatus(), + onStatus: (cb: (status: GlassesStatusSnapshot) => void): (() => void) => { + // Dedupe: the glasses store updates on many fields; only fire when the + // projected status snapshot actually changes. + let last = JSON.stringify(projectStatus()) + return useGlassesStore.subscribe(() => { + const snap = projectStatus() + const key = JSON.stringify(snap) + if (key === last) return + last = key + cb(snap) + }) + }, + info: (): GlassesInfoSnapshot => projectInfo(), + capabilities: () => getModelCapabilities(useGlassesStore.getState().deviceModel as DeviceTypes), + /** Ask the glasses to report fresh firmware/version info (updates the store). */ + requestVersionInfo: () => BluetoothSdk.requestVersionInfo(), + + // --- discrete input events (passthrough listeners) --- + onButtonPress: (cb: (event: ButtonPressEvent) => void): (() => void) => { + const sub = BluetoothSdk.addListener("button_press", cb) + return () => sub.remove() + }, + onTouchGesture: (cb: (event: TouchEvent) => void): (() => void) => { + const sub = BluetoothSdk.addListener("touch_event", cb) + return () => sub.remove() + }, + + // --- ring controller (optional input device) --- + controller: { + connectDefault: (): Promise => BluetoothSdk.connectDefaultController(), + disconnect: (): Promise => BluetoothSdk.disconnectController(), + forget: (): Promise => BluetoothSdk.forgetController(), + }, + + // --- audio (glasses media-volume + own-app playback hint) --- + audio: { + /** Read the glasses' current media volume. */ + getMediaVolume: () => BluetoothSdk.getGlassesMediaVolume(), + /** Set the glasses' media volume. */ + setMediaVolume: (level: number) => BluetoothSdk.setGlassesMediaVolume(level), + /** Tell native whether THIS app is currently playing audio (for ducking). */ + setOwnAppPlaying: (playing: boolean): Promise => BluetoothSdk.setOwnAppAudioPlaying(playing), + }, + + // --- sub-facades --- + wifi: glassesWifi, + settings: glassesSettings, +} diff --git a/mobile/modules/island/src/facades/glassesSettings.ts b/mobile/modules/island/src/facades/glassesSettings.ts new file mode 100644 index 0000000000..c541ddf08a --- /dev/null +++ b/mobile/modules/island/src/facades/glassesSettings.ts @@ -0,0 +1,47 @@ +/** + * glasses.settings facade — `toolkit.glasses.settings`: the keyed DEVICE-settings + * surface (brightness, head-up angle, dashboard, camera/button, sensing, …). These + * are the settings the island store auto-syncs to the glasses over the bluetooth-sdk + * (`BLUETOOTH_SETTING_KEYS`), so `set()` both persists and pushes to the device. + * + * Distinct from `toolkit.settings` (user/app prefs): this is scoped to the + * hardware-facing keys, minus the internal sync keys (auth/token) that aren't + * user-tunable device settings. + */ +import {useSettingsStore, SETTINGS, BLUETOOTH_SETTING_KEYS} from "../stores/settings" + +// Keys carried on BLUETOOTH_SETTING_KEYS for the device handshake that are NOT +// user-tunable settings — auth, device/controller IDENTITY (set by pairing), and +// internal RUNTIME flags. Kept out of the OEM-facing `available()` surface. +const INTERNAL_KEYS = new Set([ + // auth + SETTINGS.core_token.key, + SETTINGS.auth_email.key, + // glasses + controller identity (managed by pairing, not the user) + SETTINGS.pending_wearable.key, + SETTINGS.default_wearable.key, + SETTINGS.device_name.key, + SETTINGS.device_address.key, + SETTINGS.pending_controller.key, + SETTINGS.default_controller.key, + SETTINGS.controller_device_name.key, + SETTINGS.controller_address.key, + // internal runtime flags + SETTINGS.local_stt_fallback_active.key, + SETTINGS.offline_captions_running.key, +]) +const DEVICE_KEYS = BLUETOOTH_SETTING_KEYS.filter((k) => !INTERNAL_KEYS.has(k)) + +export const glassesSettings = { + /** Read a device setting by key. */ + get: (key: string): T | undefined => useSettingsStore.getState().getSetting(key) as T | undefined, + /** Write a device setting — persists and auto-syncs to the connected glasses. */ + set: (key: string, value: T) => useSettingsStore.getState().setSetting(key, value), + /** Subscribe to changes for one device-setting key; returns an unsubscribe. */ + onChanged: (key: string, cb: (value: T | undefined) => void): (() => void) => + useSettingsStore.subscribe((s) => s.getSetting(key) as T | undefined, cb), + /** The schema descriptor for a key (type, default, options…), or undefined. */ + descriptor: (key: string) => SETTINGS[key], + /** The device-setting keys synced to the glasses (excludes internal sync keys). */ + available: (): string[] => DEVICE_KEYS, +} diff --git a/mobile/modules/island/src/facades/glassesWifi.ts b/mobile/modules/island/src/facades/glassesWifi.ts new file mode 100644 index 0000000000..e8cd722798 --- /dev/null +++ b/mobile/modules/island/src/facades/glassesWifi.ts @@ -0,0 +1,51 @@ +/** + * glasses.wifi facade — the first typed island facade over the bluetooth-sdk + * passthrough, and the reference pattern the rest of the (A) host API copies. + * + * A facade wraps the raw `@mentra/bluetooth-sdk` surface into a small, typed, + * device-agnostic API (the `doX()` action shape here; `getX()`/`onX()` read-models + * come with domains that own their state). The host UI calls `toolkit.glasses.wifi.*` + * instead of importing bluetooth-sdk directly — that's the native-import boundary + * the toolkit is built around. + * + * `connect()` propagates bluetooth-sdk's coded errors unchanged so callers keep + * their existing error mapping. The `status()`/`onStatus()` read-model reads the + * glasses-state store island now owns (wifi status is derived there from the + * device's connection info). + */ +import BluetoothSdk from "@mentra/bluetooth-sdk" +import type {WifiSearchResult, WifiStatus} from "@mentra/bluetooth-sdk" +import {useGlassesStore} from "../stores/glasses" + +export type {WifiSearchResult, WifiStatus} + +export const glassesWifi = { + /** Scan for nearby wifi networks. Request/response — resolves with the results. */ + scan(): Promise { + return BluetoothSdk.requestWifiScan() + }, + + /** + * Send wifi credentials to the glasses. Resolves on success; rejects with the + * bluetooth-sdk coded error (`bluetooth_powered_off`, `request_timeout`, …) on + * failure, propagated unchanged for the caller to map. + */ + async connect(ssid: string, password: string): Promise { + await BluetoothSdk.sendWifiCredentials(ssid, password) + }, + + /** Forget a saved network on the glasses. */ + async forget(ssid: string): Promise { + await BluetoothSdk.forgetWifiNetwork(ssid) + }, + + /** The glasses' current wifi connection status (snapshot). */ + status(): WifiStatus { + return useGlassesStore.getState().wifi + }, + + /** Subscribe to wifi-status changes; returns an unsubscribe. */ + onStatus(cb: (status: WifiStatus) => void): () => void { + return useGlassesStore.subscribe((s) => s.wifi, cb) + }, +} diff --git a/mobile/modules/island/src/facades/incidents.ts b/mobile/modules/island/src/facades/incidents.ts new file mode 100644 index 0000000000..17b1cbf7ca --- /dev/null +++ b/mobile/modules/island/src/facades/incidents.ts @@ -0,0 +1,73 @@ +/** + * incidents facade — `toolkit.incidents`: bug-report / feedback submission over the + * now-island RestComms (the incident REST calls moved in with the settings+RestComms + * keystone). + * + * `file()` is the one-call submission: island orchestrates createIncident → upload + * logs → notify the glasses → upload screenshots. The OEM writes its own report + * SCREEN and gathers the diagnostics (phone-state snapshot, recent logs, screenshots) + * — that gathering is genuinely native-coupled (NetInfo/Constants/Location/ImagePicker + * + console interception), so it stays host-side and is passed in. The lower-level + * primitives are also exposed for callers that want to drive the steps themselves. + */ +import restComms from "../services/RestComms" +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import {useGlassesStore} from "../stores/glasses" +import {isGlassesConnected} from "../services/GlassesReadiness" +import {useSettingsStore} from "../stores/settings" + +export interface IncidentFileInput { + /** The user's report fields (description/expected/actual/severity/contactEmail…). */ + feedbackData: Record + /** The gathered phone-state diagnostics snapshot (host-built; native-coupled). */ + phoneState: Record + /** Recent phone logs to attach. */ + logs?: Parameters[1] + /** Optional screenshot attachments. */ + screenshots?: Parameters[1] +} + +export const incidents = { + /** + * File an incident end-to-end: create it, upload logs, notify the connected + * glasses, and upload screenshots. Returns the new incident id (or an error). + */ + async file(input: IncidentFileInput): Promise<{incidentId?: string; error?: string}> { + const backendUrl = useSettingsStore.getState().getRestUrl() + const res = await restComms.createIncident(input.feedbackData, input.phoneState) + if (res.is_error()) return {error: res.error.message} + + const {incidentId} = res.value + if (input.logs && input.logs.length > 0) { + const r = await restComms.uploadIncidentLogs(incidentId, input.logs) + if (r.is_error()) console.warn("incidents.file: upload logs failed:", r.error?.message) + } + // Reference the object directly (not `this`) so a detached call — + // `const {file} = toolkit.incidents; file(input)` — still notifies the glasses. + incidents.notifyGlasses(incidentId, backendUrl) + if (input.screenshots && input.screenshots.length > 0) { + const r = await restComms.uploadIncidentAttachments(incidentId, input.screenshots) + if (r.is_error()) console.warn("incidents.file: upload attachments failed:", r.error?.message) + } + return {incidentId} + }, + + // --- lower-level primitives (drive the steps yourself) --- + /** + * Notify the connected glasses of an incident id (no-op if disconnected). + * Defaults the API base URL to the island's current REST URL. + */ + notifyGlasses(incidentId: string, apiBaseUrl?: string | null): void { + if (!isGlassesConnected(useGlassesStore.getState().connection)) return + BluetoothSdk.sendIncidentId(incidentId, apiBaseUrl ?? useSettingsStore.getState().getRestUrl()) + }, + /** Create an incident (returns its id); pass the gathered phone-state snapshot. */ + create: (...args: Parameters) => restComms.createIncident(...args), + /** Upload the captured phone logs against an incident id. */ + uploadLogs: (...args: Parameters) => restComms.uploadIncidentLogs(...args), + /** Upload screenshot/image attachments against an incident id. */ + uploadAttachments: (...args: Parameters) => + restComms.uploadIncidentAttachments(...args), + /** Send freeform feedback (non-incident). */ + sendFeedback: (...args: Parameters) => restComms.sendFeedback(...args), +} diff --git a/mobile/modules/island/src/facades/miniapps.ts b/mobile/modules/island/src/facades/miniapps.ts new file mode 100644 index 0000000000..67e9749997 --- /dev/null +++ b/mobile/modules/island/src/facades/miniapps.ts @@ -0,0 +1,48 @@ +/** + * miniapps facade — `toolkit.miniapps`: miniapp lifecycle over the island-owned + * apps store. Thin forwarding (args via Parameters<>), so the facade never drifts + * from the store's actions. + * + * The miniapp WebView is a HOST component (island ships the bridge, not the view) — + * the bridge primitives (`buildMiniappGlobalsScript`, `buildMentraUiShim`, the + * MentraJS router) are exported from `@mentra/island` directly; the host mounts a + * `` wired to them. This facade is the lifecycle half. + */ +import {useAppStatusStore} from "../stores/apps" +import type {ClientApp} from "../types" + +type AppsState = ReturnType + +export const miniapps = { + /** All installed miniapps (snapshot). */ + list: (): ClientApp[] => useAppStatusStore.getState().apps, + /** Subscribe to the installed-app set; fires only when it changes. Returns an unsubscribe. */ + onChanged: (cb: (apps: ClientApp[]) => void): (() => void) => { + let last = JSON.stringify(useAppStatusStore.getState().apps) + return useAppStatusStore.subscribe(() => { + const apps = useAppStatusStore.getState().apps + const key = JSON.stringify(apps) + if (key === last) return + last = key + cb(apps) + }) + }, + + /** Re-fetch the installed-app list from the backend. */ + refresh: () => useAppStatusStore.getState().refresh(), + /** Start (foreground) a miniapp. */ + start: (...args: Parameters) => useAppStatusStore.getState().start(...args), + /** Stop a miniapp. */ + stop: (...args: Parameters) => useAppStatusStore.getState().stop(...args), + /** Bring a running miniapp to the foreground. */ + setForeground: (...args: Parameters) => + useAppStatusStore.getState().setForeground(...args), + /** Clear the current foreground miniapp. */ + clearForeground: () => useAppStatusStore.getState().clearForeground(), + /** Stop all running miniapps. */ + stopAll: () => useAppStatusStore.getState().stopAll(), + /** Install a miniapp from a bundle URL. */ + install: (...args: Parameters) => useAppStatusStore.getState().install(...args), + /** Uninstall a miniapp. */ + uninstall: (...args: Parameters) => useAppStatusStore.getState().uninstall(...args), +} diff --git a/mobile/modules/island/src/facades/notifications.ts b/mobile/modules/island/src/facades/notifications.ts new file mode 100644 index 0000000000..bfc887a8ef --- /dev/null +++ b/mobile/modules/island/src/facades/notifications.ts @@ -0,0 +1,15 @@ +/** + * notifications facade — `toolkit.notifications`: inbound alerts island→host. The + * OEM subscribes and renders them however it likes (modal, toast, banner). Today: + * miniapp crashloop; version-incompatible miniapp and persistent connection loss + * land as their detectors are wired into the emitter. + */ +import {islandNotifications, type IslandNotification} from "../services/NotificationsEmitter" + +export type {IslandNotification, IslandNotificationKind} from "../services/NotificationsEmitter" + +export const notifications = { + /** Subscribe to island-raised notifications; returns an unsubscribe. */ + onNotification: (cb: (notification: IslandNotification) => void): (() => void) => + islandNotifications.subscribe(cb), +} diff --git a/mobile/modules/island/src/facades/ota.ts b/mobile/modules/island/src/facades/ota.ts new file mode 100644 index 0000000000..a6414bbfb8 --- /dev/null +++ b/mobile/modules/island/src/facades/ota.ts @@ -0,0 +1,52 @@ +/** + * ota facade — `toolkit.ota`: the OEM-facing OTA read/observe surface. Island's + * OtaService projects the glasses' OTA BLE events into the island store; this facade + * exposes the snapshot + change subscriptions the host renders its update prompt and + * progress UI from. No host-injected UI — the host owns all alerts/navigation/i18n. + * + * Install ACTIONS (check/install/retry) land in Phase 2 (the install orchestration + * moves out of the host progress screen; bootloop-risk, needs on-device verification). + */ +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import type {OtaStatus, OtaUpdateInfo} from "../../../bluetooth-sdk/build/_internal" +import {useGlassesStore} from "../stores/glasses" + +export const ota = { + // --- actions --- + /** + * Start the firmware install with the resolved OTA manifest URL. Progress lands on + * `status()`/`onStatus()`. (The host resolves the manifest URL — dev-override/env/prod + * resolution stays with the OTA config; the host-side stuck/retry watchdog is its own + * resilience layer on top of this command.) + */ + install: (...args: Parameters) => BluetoothSdk.startOtaUpdate(...args), + /** Re-query the glasses for an available update (e.g. after a clock-fix or failure). */ + retry: () => BluetoothSdk.checkForOtaUpdate(), + + /** Current available-update info (versionName/updates/totalSize), or null. */ + updateAvailable: (): OtaUpdateInfo | null => useGlassesStore.getState().otaUpdateAvailable, + /** Current OTA install status (stepType/phase/percent/status/error), or null. */ + status: (): OtaStatus | null => useGlassesStore.getState().otaStatus, + + /** Subscribe to "an update became available" (null → info). Returns an unsubscribe. */ + onUpdateAvailable: (cb: (info: OtaUpdateInfo) => void): (() => void) => { + let last = useGlassesStore.getState().otaUpdateAvailable + return useGlassesStore.subscribe(() => { + const info = useGlassesStore.getState().otaUpdateAvailable + if (info === last) return + last = info + if (info) cb(info) + }) + }, + + /** Subscribe to OTA install status changes. Returns an unsubscribe. */ + onStatus: (cb: (status: OtaStatus | null) => void): (() => void) => { + let last = useGlassesStore.getState().otaStatus + return useGlassesStore.subscribe(() => { + const status = useGlassesStore.getState().otaStatus + if (status === last) return + last = status + cb(status) + }) + }, +} diff --git a/mobile/modules/island/src/facades/pairing.ts b/mobile/modules/island/src/facades/pairing.ts new file mode 100644 index 0000000000..879838378a --- /dev/null +++ b/mobile/modules/island/src/facades/pairing.ts @@ -0,0 +1,51 @@ +/** + * pairing facade — `toolkit.pairing`: scan for nearby glasses + pair. Scan state + * (searching + results) comes from the island core store; scan/connect go over the + * bluetooth-sdk; pair-failure / not-ready are the bluetooth-sdk events. + * + * (Connect-the-already-paired-default is `toolkit.glasses.connectDefault()`; this + * facade is the first-time discovery + pair flow.) + */ +import BluetoothSdk from "@mentra/bluetooth-sdk" +import type {PairFailureEvent, GlassesNotReadyEvent} from "@mentra/bluetooth-sdk" +import {useCoreStore} from "../stores/core" +import {pushAllBluetoothSettings} from "../services/GlassesSettingsSync" + +export const pairing = { + /** Start scanning for nearby glasses. Results land on `searchResults()`/`onFound()`. */ + scan: (...args: Parameters) => BluetoothSdk.startScan(...args), + /** Whether a scan is currently in progress. */ + scanning: (): boolean => useCoreStore.getState().searching, + /** The current scan results (snapshot). */ + searchResults: () => useCoreStore.getState().searchResults, + /** Subscribe to scan-result changes; fires only when they change. Returns an unsubscribe. */ + onFound: (cb: (results: ReturnType["searchResults"]) => void): (() => void) => { + let last = JSON.stringify(useCoreStore.getState().searchResults) + return useCoreStore.subscribe(() => { + const results = useCoreStore.getState().searchResults + const key = JSON.stringify(results) + if (key === last) return + last = key + cb(results) + }) + }, + + /** Pair with (connect to) a discovered device. */ + pair: async (...args: Parameters): Promise => { + await pushAllBluetoothSettings() + return BluetoothSdk.connect(...args) + }, + /** Set a device as the default for subsequent `glasses.connectDefault()`. */ + setDefault: (...args: Parameters) => BluetoothSdk.setDefaultDevice(...args), + + /** Subscribe to pairing failures; returns an unsubscribe. */ + onPairFailure: (cb: (event: PairFailureEvent) => void): (() => void) => { + const sub = BluetoothSdk.addListener("pair_failure", cb) + return () => sub.remove() + }, + /** Subscribe to glasses-not-ready events during pairing; returns an unsubscribe. */ + onGlassesNotReady: (cb: (event: GlassesNotReadyEvent) => void): (() => void) => { + const sub = BluetoothSdk.addListener("glasses_not_ready", cb) + return () => sub.remove() + }, +} diff --git a/mobile/modules/island/src/facades/permissions.ts b/mobile/modules/island/src/facades/permissions.ts new file mode 100644 index 0000000000..de4c5b29c5 --- /dev/null +++ b/mobile/modules/island/src/facades/permissions.ts @@ -0,0 +1,121 @@ +/** + * permissions facade — `toolkit.permissions`: the raw OS-permission ops the runtime + * needs, mapped from feature keys to react-native-permissions. This is the LEAN + * surface (check/request/openSettings/requirementsForMiniapp) — the host keeps its + * own UI helpers (rationale dialogs, denied-warnings) on top; island stays free of + * i18n/theme/AlertUtils. + */ +import {Platform, Linking, PermissionsAndroid, type Permission as AndroidPermission} from "react-native" +import {check, request, PERMISSIONS, RESULTS, type Permission} from "react-native-permissions" + +import appRegistry from "../services/AppRegistry" +import type {AppletPermission} from "../types" + +/** OEM-facing feature keys (mapped to OS permissions per platform). */ +export const PermissionFeatures = { + MICROPHONE: "microphone", + CAMERA: "camera", + CALENDAR: "calendar", + LOCATION: "location", + BACKGROUND_LOCATION: "background_location", + BLUETOOTH: "bluetooth", + PHONE_STATE: "phone_state", + POST_NOTIFICATIONS: "post_notifications", +} as const + +const ANDROID = PermissionsAndroid.PERMISSIONS +const apiLevel = typeof Platform.Version === "number" ? Platform.Version : parseInt(String(Platform.Version), 10) + +function iosPerms(feature: string): Permission[] { + switch (feature) { + case PermissionFeatures.MICROPHONE: + return [PERMISSIONS.IOS.MICROPHONE] + case PermissionFeatures.CAMERA: + return [PERMISSIONS.IOS.CAMERA] + case PermissionFeatures.CALENDAR: + return [PERMISSIONS.IOS.CALENDARS] + case PermissionFeatures.LOCATION: + return [PERMISSIONS.IOS.LOCATION_WHEN_IN_USE] + case PermissionFeatures.BACKGROUND_LOCATION: + // iOS won't grant ALWAYS without WHEN_IN_USE first — request both (matches + // the host PermissionsUtils mapping), else a cold request can't escalate. + return [PERMISSIONS.IOS.LOCATION_WHEN_IN_USE, PERMISSIONS.IOS.LOCATION_ALWAYS] + case PermissionFeatures.BLUETOOTH: + return [PERMISSIONS.IOS.BLUETOOTH] + default: + return [] + } +} + +function androidPerms(feature: string): AndroidPermission[] { + switch (feature) { + case PermissionFeatures.MICROPHONE: + return [ANDROID.RECORD_AUDIO] + case PermissionFeatures.CAMERA: + return [ANDROID.CAMERA] + case PermissionFeatures.CALENDAR: + return [ANDROID.READ_CALENDAR, ANDROID.WRITE_CALENDAR] + case PermissionFeatures.LOCATION: + case PermissionFeatures.BACKGROUND_LOCATION: + return [ANDROID.ACCESS_FINE_LOCATION] + case PermissionFeatures.PHONE_STATE: + return [ANDROID.READ_PHONE_STATE] + case PermissionFeatures.BLUETOOTH: + return apiLevel >= 31 ? [ANDROID.BLUETOOTH_SCAN, ANDROID.BLUETOOTH_CONNECT, ANDROID.BLUETOOTH_ADVERTISE] : [] + case PermissionFeatures.POST_NOTIFICATIONS: + return apiLevel >= 33 ? [ANDROID.POST_NOTIFICATIONS] : [] + default: + return [] + } +} + +export const permissions = { + /** Is the given feature's OS permission currently granted? */ + async check(feature: string): Promise { + if (Platform.OS === "android") { + const perms = androidPerms(feature) + if (perms.length === 0) return true // nothing to ask on this OS/version + // Every permission in the group must be granted (e.g. bluetooth = + // scan+connect+advertise, calendar = read+write) — matches the iOS branch. + for (const p of perms) { + if (!(await PermissionsAndroid.check(p))) return false + } + return true + } + const perms = iosPerms(feature) + if (perms.length === 0) return true + for (const p of perms) { + const status = await check(p) + if (status !== RESULTS.GRANTED && status !== RESULTS.LIMITED) return false + } + return true + }, + + /** Request the given feature's OS permission. Resolves to whether it's granted. */ + async request(feature: string): Promise { + if (Platform.OS === "android") { + const perms = androidPerms(feature) + if (perms.length === 0) return true + const results = await PermissionsAndroid.requestMultiple(perms) + // Granted only when EVERY permission in the group was granted. + return perms.every((p) => results[p] === PermissionsAndroid.RESULTS.GRANTED) + } + const perms = iosPerms(feature) + if (perms.length === 0) return true + let granted = true + for (const p of perms) { + const result = await request(p) + if (result !== RESULTS.GRANTED && result !== RESULTS.LIMITED) granted = false + } + return granted + }, + + /** Open the OS settings app (so the user can grant a blocked permission). */ + openSettings: (): Promise => Linking.openSettings(), + + /** The permissions a given installed miniapp declares it needs. */ + async requirementsForMiniapp(packageName: string): Promise { + const apps = await appRegistry.getInstalledMiniapps() + return apps.find((a) => a.packageName === packageName)?.permissions ?? [] + }, +} diff --git a/mobile/modules/island/src/facades/phoneNotifications.ts b/mobile/modules/island/src/facades/phoneNotifications.ts new file mode 100644 index 0000000000..5d68f6bd07 --- /dev/null +++ b/mobile/modules/island/src/facades/phoneNotifications.ts @@ -0,0 +1,50 @@ +/** + * phoneNotifications facade — `toolkit.phoneNotifications`: forward phone + * notifications to the glasses, with a per-app blocklist. Android-only (the + * NotificationListenerService); the getters return sensible defaults on iOS. + * + * Enabled-state + blocklist are island settings (auto-synced to the native + * listener); the installed-apps list + the listener permission are CrustModule. + */ +import {Platform} from "react-native" +import CrustModule from "@mentra/crust" +import {useSettingsStore, SETTINGS} from "../stores/settings" + +export const phoneNotifications = { + /** Is phone-notification forwarding enabled? (false on iOS) */ + enabled: (): boolean => { + if (Platform.OS !== "android") return false + return useSettingsStore.getState().getSetting(SETTINGS.notifications_enabled.key) ?? true + }, + /** Enable/disable phone-notification forwarding. (no-op on iOS) */ + setEnabled: (enabled: boolean) => { + if (Platform.OS !== "android") return + return useSettingsStore.getState().setSetting(SETTINGS.notifications_enabled.key, enabled) + }, + + /** The phone's installed apps (package, name, icon). (empty on iOS) */ + installedApps: (): Promise> => { + if (Platform.OS !== "android") return Promise.resolve([]) + return CrustModule.getInstalledApps() + }, + + /** The per-app blocklist (package names whose notifications are NOT forwarded). */ + blocklist: (): string[] => { + const b = useSettingsStore.getState().getSetting(SETTINGS.notifications_blocklist.key) + return Array.isArray(b) ? b : [] + }, + /** Replace the per-app blocklist. */ + setBlocklist: (packageNames: string[]) => + useSettingsStore.getState().setSetting(SETTINGS.notifications_blocklist.key, packageNames), + + /** Does the app have the Android notification-listener permission? (false on iOS) */ + hasListenerPermission: (): Promise => { + if (Platform.OS !== "android") return Promise.resolve(false) + return CrustModule.hasNotificationListenerPermission() + }, + /** Open the OS settings to grant the notification-listener permission. (no-op on iOS) */ + requestListenerPermission: () => { + if (Platform.OS !== "android") return + return CrustModule.openNotificationListenerSettings() + }, +} diff --git a/mobile/modules/island/src/facades/session.ts b/mobile/modules/island/src/facades/session.ts new file mode 100644 index 0000000000..26a52779d4 --- /dev/null +++ b/mobile/modules/island/src/facades/session.ts @@ -0,0 +1,48 @@ +/** + * session facade — `toolkit.session`: the cloud-v2 live-session surface island + * now owns (keystone #5). Exposes the connection status read-model (status + + * audio transport, projected from the island-owned cloud-status store) and the + * liveness flag. + * + * `account.*` operations route through the now-island RestComms (the account REST + * calls moved in with the settings+RestComms keystone), so they're adapter-free. + * (`requestExport` is a host UI navigation flow with no backend method yet, so it + * is not surfaced here.) + */ +import restComms from "../services/RestComms" +import {cloudClientService} from "../services/CloudClientService" +import {useCloudClientStatusStore} from "../stores/cloudClientStatus" +import type {CloudClientStatusSnapshot} from "../runtime/config" + +function project(): CloudClientStatusSnapshot { + const s = useCloudClientStatusStore.getState() + return {status: s.status, audioTransport: s.audioTransport} +} + +export const session = { + /** Current cloud live-session status (connection state + audio transport). */ + status: (): CloudClientStatusSnapshot => project(), + /** Subscribe to cloud session-status changes; returns an unsubscribe. */ + onStatus: (cb: (status: CloudClientStatusSnapshot) => void): (() => void) => { + // Dedupe: fire only when the projected {status, audioTransport} changes. + let last = JSON.stringify(project()) + return useCloudClientStatusStore.subscribe(() => { + const snap = project() + const key = JSON.stringify(snap) + if (key === last) return + last = key + cb(snap) + }) + }, + /** Whether the live-session handshake has completed. */ + isConnected: (): boolean => cloudClientService.isConnected(), + + /** Identity / account operations (island-owned RestComms). */ + account: { + /** Request account deletion — the backend emails a confirmation code. */ + delete: () => restComms.requestAccountDeletion(), + /** Confirm a pending account deletion with the emailed code. */ + confirmDelete: (requestId: string, confirmationCode: string) => + restComms.confirmAccountDeletion(requestId, confirmationCode), + }, +} diff --git a/mobile/modules/island/src/facades/settings.ts b/mobile/modules/island/src/facades/settings.ts new file mode 100644 index 0000000000..3667362ba4 --- /dev/null +++ b/mobile/modules/island/src/facades/settings.ts @@ -0,0 +1,27 @@ +/** + * settings facade — `toolkit.settings`: the typed keyed user-settings surface over + * the island-owned settings store. This is the (A) OEM contract; the raw store at + * `toolkit.stores.settings` is the Mentra-app escape hatch. + * + * Keys + their schema live in `SETTINGS` (theme, devMode, metric/twelveHourTime, + * notifications, onboarding flags, …); `descriptor(key)`/`keys()` expose them. + */ +import {useSettingsStore, SETTINGS} from "../stores/settings" + +export const settings = { + /** Read a setting by key (the current value, or its default). */ + get: (key: string): T | undefined => useSettingsStore.getState().getSetting(key) as T | undefined, + /** + * Write a setting. `syncToServer` (default true) also pushes the change to the + * backend; pass false for device-local-only writes. + */ + set: (key: string, value: T, syncToServer = true) => + useSettingsStore.getState().setSetting(key, value, syncToServer), + /** Subscribe to changes for one key; returns an unsubscribe. */ + onChanged: (key: string, cb: (value: T | undefined) => void): (() => void) => + useSettingsStore.subscribe((s) => s.getSetting(key) as T | undefined, cb), + /** The schema descriptor for a key (type, default, options…), or undefined. */ + descriptor: (key: string) => SETTINGS[key], + /** All known setting keys. */ + keys: (): string[] => Object.keys(SETTINGS), +} diff --git a/mobile/modules/island/src/facades/speech.ts b/mobile/modules/island/src/facades/speech.ts new file mode 100644 index 0000000000..f68c8a011b --- /dev/null +++ b/mobile/modules/island/src/facades/speech.ts @@ -0,0 +1,38 @@ +/** + * speech facade — `toolkit.speech.{stt,tts}`: on-device STT/TTS model management, + * wrapping the in-island model managers. Thin passthrough (args forwarded with + * exact types via Parameters<>), so the facade never drifts from the managers. + * STT additionally exposes the auto-download status stream (the TTS path has none). + */ +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import sttModelManager from "../services/STTModelManager" +import ttsModelManager from "../services/TTSModelManager" +import offlineSpeechModelService from "../services/OfflineSpeechModelService" + +export const speech = { + /** Restart the glasses' on-device transcriber (e.g. after switching STT language). */ + restartTranscriber: (): Promise => BluetoothSdk.restartTranscriber(), + stt: { + currentLanguage: () => sttModelManager.getCurrentLanguage(), + languages: () => sttModelManager.getAvailableLanguages(), + languageInfo: () => sttModelManager.getAllLanguageInfo(), + download: (...args: Parameters) => sttModelManager.downloadModel(...args), + activate: (code: string) => sttModelManager.setCurrentLanguage(code), + cancelDownload: () => sttModelManager.cancelDownload(), + deleteModel: (...args: Parameters) => sttModelManager.deleteModel(...args), + /** Current auto-download status of the offline STT model. */ + status: () => offlineSpeechModelService.getStatus(), + /** Subscribe to auto-download status changes; returns an unsubscribe. */ + onStatusChanged: (...args: Parameters) => + offlineSpeechModelService.subscribe(...args), + }, + tts: { + currentLanguage: () => ttsModelManager.getCurrentLanguage(), + languages: () => ttsModelManager.getAvailableLanguages(), + languageInfo: () => ttsModelManager.getAllLanguageInfo(), + download: (...args: Parameters) => ttsModelManager.downloadModel(...args), + activate: (code: string) => ttsModelManager.setCurrentLanguage(code), + cancelDownload: () => ttsModelManager.cancelDownload(), + deleteModel: (...args: Parameters) => ttsModelManager.deleteModel(...args), + }, +} diff --git a/mobile/modules/island/src/index.ts b/mobile/modules/island/src/index.ts index e44bdffb79..8153c487bb 100644 --- a/mobile/modules/island/src/index.ts +++ b/mobile/modules/island/src/index.ts @@ -20,8 +20,6 @@ export {default as localDisplayManager, type DisplayPayload} from "./services/Lo export {default as localMiniappRuntime, type InstalledMiniappManifest} from "./services/LocalMiniappRuntime" export { miniappLauncher, - configureLauncher, - type LauncherDeps, type LaunchHints, type LaunchResult, type ResolvedBundle, @@ -40,6 +38,7 @@ export { type CrashOutcome, type CrashControllerOptions, } from "./services/MentraJSCrashController" +export {ensureMiniappEngine, getMiniappEngine, type MiniappEngine} from "./services/MiniappEngine" export { redactSecrets, MentraJSLogThrottle, @@ -48,6 +47,36 @@ export { } from "./services/MentraJSLogPipeline" export {default as localSttFallbackCoordinator} from "./services/LocalSttFallbackCoordinator" export {default as micStateCoordinator} from "./services/MicStateCoordinator" +export {default as navigationService} from "./services/NavigationService" +export {default as audioPlaybackService} from "./services/AudioPlaybackService" +// Phone GPS — the background location task + tier control moved into island (was the +// host locationTier hook + a MantleManager defineTask). Exported so the barrel load +// registers the background task at @mentra/island import time. +export {phoneLocationService, stopPhoneLocation} from "./services/PhoneLocationService" +// Clock-skew utils (used by OTA + the host gallery sync) moved into island. +export {fixGlassesClockIfSkewed, maybeFixGlassesClockFromVersionInfo} from "./services/glassesClockSync" +// OTA manifest-URL resolution (dev-override/legacy-build/env/prod) moved into island. +// Re-exported through the host @/services/asg/asgOtaVersionUrl shim. +export {getAsgOtaVersionUrl} from "./services/asgOtaVersionUrl" +export {detectClockSkew, isSyncManifestEmpty, CLOCK_SKEW_TOLERANCE_MS} from "./services/gallerySyncClock" +// Gallery cluster — the sync orchestrator + glasses-camera HTTP API + media/storage, +// moved into island. Host Gallery UI + tests import these from @mentra/island. +export {gallerySyncService} from "./services/asg/gallerySyncService" +export {asgCameraApi} from "./services/asg/asgCameraApi" +export {localStorageService} from "./services/asg/localStorageService" +export {mediaProcessingQueue} from "./services/asg/mediaProcessingQueue" +export {gallerySettingsService} from "./services/asg/gallerySettingsService" +export { + INVALID_DOWNLOADED_MEDIA, + validateCaptureMetadataForDownload, + validateDownloadedMediaFile, +} from "./services/asg/galleryMediaValidation" +export {emitGalleryNotice, onGalleryNotice, type GalleryNotice, type GalleryNoticeCode} from "./services/asg/galleryNotices" +export {MediaLibraryPermissions} from "./utils/permissions/MediaLibraryPermissions" +export {deriveGalleryDisplayName} from "./utils/permissions/galleryDisplayName" +export type {CaptureFile, CaptureGroup, GalleryResponse, ServerStatus, HealthResponse, GalleryEvent} from "./types/asg" +export {phonePhotoCoordinator} from "./services/PhonePhotoCoordinator" +export {phoneStreamCoordinator} from "./services/PhoneStreamCoordinator" export { default as sttModelManager, STTModelManager, @@ -78,39 +107,90 @@ export { type ConnectButtonAction, } from "./services/ConnectionCoordinator" -// Bluetooth SDK internal compatibility passthrough. The island package uses the -// declared @mentra/bluetooth-sdk/internal subpath so it does not reach into SDK -// build output directly. Prefer an island facade when one exists; this remains -// an internal escape hatch for app-only compatibility needs. -export {default as BluetoothSdk} from "@mentra/bluetooth-sdk/internal" -export type {PairFailureEvent, GlassesNotReadyEvent} from "@mentra/bluetooth-sdk/internal" +// The namespaced OEM-facing toolkit API (the "(A) host API"). Additive — grows one +// facade at a time alongside the flat exports below. See ./island. +export {toolkit} from "./island" +export type {IslandNotification, IslandNotificationKind} from "./facades/notifications" +export type {WifiSearchResult} from "./facades/glassesWifi" +export type {DisplayMirrorEvent} from "./facades/displayMirror" +export type { + IslandConfigureOptions, + IslandAuth, + IslandConfigValues, + IslandAnalytics, + SubjectTokenType, +} from "./runtime/bootstrap" + +// Glasses device-state store — moved into island so island owns device state. +// Re-exported through the host's @/stores/glasses shim so the app keeps its +// imports. Also surfaced as toolkit.glassesStore (the escape hatch). Predicates +// (isGlassesConnected/…) come from GlassesReadiness above, not re-exported here. +export { + useGlassesStore, + selectGlassesConnected, + selectGlassesReady, + getGlasesInfoPartial, + getGlassesSystemTimeMs, + waitForGlassesState, +} from "./stores/glasses" +// Display/mirror store — also moved into island; re-exported through the host's +// @/stores/display shim (and toolkit.displayStore). Read it via toolkit.display.mirror. +export {useDisplayStore} from "./stores/display" +// More device/runtime stores moved into island (re-exported via host shims + +// toolkit.stores.*). WebSocketStatus + PhotoInfo were relocated off host modules. +export {useCoreStore} from "./stores/core" +export {useConnectionStore, WebSocketStatus} from "./stores/connection" +export { + useGallerySyncStore, + selectSyncProgress, + selectIssyncing, + selectGlassesGalleryStatus, +} from "./stores/gallerySync" +export type {PhotoInfo, SyncState, HotspotInfo, SyncQueue, GallerySyncInfo} from "./stores/gallerySync" +// Cloud-client runtime status store + secure credential store — moved into +// island as part of owning the cloud-v2 client. Re-exported via host shims +// (@/stores/cloudClientStatus, @/utils/cloudClient/MmkvSecureStore). +export {useCloudClientStatusStore} from "./stores/cloudClientStatus" +export type {RuntimeAudioTransport, RuntimeSnapshot, RuntimeStatus} from "./stores/cloudClientStatus" +export {cloudSecureStore} from "./utils/cloudClient/cloudSecureStore" +// The cloud-client singleton now lives in island (keystone #5). Construction and +// live runtime surfaces happen here; the host's @/services/cloudClient is a thin +// delegating wrapper that keeps endpoint resolution (dev/settings) host-side. +export {cloudClientService} from "./services/CloudClientService" +// Settings store + RestComms — the mutually-coupled v1-comms pair, moved into +// island together (settings needs RestComms for cloud-sync; RestComms needs +// settings for the backend URL). Re-exported via host shims (@/stores/settings, +// @/services/RestComms). v1-transitional: deleted in place when v1 retires. +export {SETTINGS, OFFLINE_APPLETS, useSettingsStore, useSetting} from "./stores/settings" +export {default as restComms} from "./services/RestComms" + +// Bluetooth SDK passthrough — the full @mentra/bluetooth-sdk surface re-exported +// so the app reaches the SDK through island instead of importing it directly. +// This is the escape hatch for what no island facade models yet (OTA, raw device +// control); prefer a facade when one exists. The curated public +// @mentra/bluetooth-sdk entry is for external enterprise SDK consumers — inside +// the app we use the complete (internal) surface. +// Resolved by relative path to bluetooth-sdk's built output (postinstall builds +// bluetooth-sdk before island). "@mentra/bluetooth-sdk-internal" is only a +// mobile-app tsconfig alias — it does not resolve in island's own standalone +// build (whose tsconfig is regenerated by expo-module). The path resolves +// identically from island's src (typecheck) and build (runtime), since both sit +// one level under modules/island. +export {default as BluetoothSdk} from "../../bluetooth-sdk/build/_internal" +export type {PairFailureEvent, GlassesNotReadyEvent} from "../../bluetooth-sdk/build/_internal" -// Runtime config (host-injected adapters) +// Runtime-shared constants and DTOs export { - configureRuntime, - getRuntimeHooks, ISLAND_SETTINGS_KEYS, - type RuntimeHooks, - type SocketCommsAdapter, type CloudClientStatusSnapshot, - type CloudRuntimeAdapter, - type MiniappAuthAdapter, type MiniappAuthToken, - type AudioPlaybackAdapter, - type AudioPlayRequest, - type SettingsAccessor, - type StoreAccessor, - type GlassesSnapshot, - type StreamingAdapter, - type PhotoAdapter, - type InteropAdapter, type InteropAuditEvent, + type TtsSynthesisResult, } from "./runtime/config" // Stores export { useAppStatusStore, - configureIsland, DUMMY_APPLET, saveAppsOrder, getAppsOrder, @@ -134,7 +214,6 @@ export { useForegroundApp, useActiveBackgroundAppsCount, useLocalMiniApps, - type IslandHostHooks, type StartOptions, type OrderMap, } from "./stores/apps" @@ -153,6 +232,9 @@ export {createCloudUdpSocket} from "./utils/cloudClient/RnUdpAdapter" export {HardwareCompatibility, type CompatibilityResult} from "./utils/hardware" export {BgTimer, throttle, debounce} from "./utils/timers" export {storage, printDirectory} from "./utils/storage" +// Process-wide event bus — one shared instance for island services + host +// (re-exported via the @/utils/GlobalEventEmitter shim). +export {default as GlobalEventEmitter} from "./utils/GlobalEventEmitter" // Types (copied from @mentra/types — keep in sync with cloud/packages/types/src) export { diff --git a/mobile/modules/island/src/island.ts b/mobile/modules/island/src/island.ts new file mode 100644 index 0000000000..24be90288d --- /dev/null +++ b/mobile/modules/island/src/island.ts @@ -0,0 +1,138 @@ +/** + * `toolkit` — the namespaced OEM-facing toolkit API (the "(A) host API" from the + * Phase-1 contract). Host UI calls `toolkit..()`; each domain is a + * typed facade over the runtime. (Exported from the `@mentra/island` module, whose + * name stays `island` in code; the public API surface is `toolkit`.) + * + * This is additive: it grows one facade at a time and lives *alongside* the flat + * named exports in `index.ts` during the migration. The flat exports (and the + * `BluetoothSdk` passthrough) stay until every screen has moved onto `island.*`. + */ +import {configure, start as bootstrapStart, stop as bootstrapStop} from "./runtime/bootstrap" +import {cloudClientService} from "./services/CloudClientService" +import {startGlassesSettingsSync, stopGlassesSettingsSync} from "./services/GlassesSettingsSync" +import {startGlassesStatusProjection, stopGlassesStatusProjection} from "./services/GlassesStatusProjection" +import {startOtaService, stopOtaService} from "./services/OtaService" +import {startAudioCloudUplink, stopAudioCloudUplink} from "./services/AudioCloudUplink" +import {startDeviceEventRouter, stopDeviceEventRouter} from "./services/DeviceEventRouter" +import {startPhoneNotificationsSync, stopPhoneNotificationsSync} from "./services/PhoneNotificationsSync" +import {ensureMiniappEngine} from "./services/MiniappEngine" +import localMiniappRuntime from "./services/LocalMiniappRuntime" +import displayProcessor from "./services/DisplayProcessor" +import {gallerySyncService} from "./services/asg/gallerySyncService" +import {glasses} from "./facades/glasses" +import {display} from "./facades/display" +import {speech} from "./facades/speech" +import {session} from "./facades/session" +import {settings} from "./facades/settings" +import {dev} from "./facades/dev" +import {incidents} from "./facades/incidents" +import {ota} from "./facades/ota" +import {gallery} from "./facades/gallery" +import {miniapps} from "./facades/miniapps" +import {pairing} from "./facades/pairing" +import {phoneNotifications} from "./facades/phoneNotifications" +import {permissions} from "./facades/permissions" +import {notifications} from "./facades/notifications" +import {useGlassesStore} from "./stores/glasses" +import {useDisplayStore} from "./stores/display" +import {useCoreStore} from "./stores/core" +import {useConnectionStore} from "./stores/connection" +import {useGallerySyncStore} from "./stores/gallerySync" +import {useCloudClientStatusStore} from "./stores/cloudClientStatus" +import {useSettingsStore} from "./stores/settings" + +export const toolkit = { + /** Front door — hand island auth + config, then start/stop the runtime. */ + configure, + /** Start the runtime: mark started + construct/connect the cloud client + begin + * syncing device settings to the glasses. */ + async start() { + await bootstrapStart() + // Construct + connect the cloud client so the documented configure()+start() + // lifecycle yields a live toolkit.session for OEMs. Idempotent for repeated + // start() calls. + cloudClientService.init() + // Project native device status -> the island stores (the inbound feed the rest + // of the runtime reads). Established first so the stores are live before the + // syncs below react to them. + startGlassesStatusProjection() + // Route the rest of the inbound device events (wifi/hotspot/gallery -> stores+bus, + // photo/stream -> coordinators, button/touch/accel/head -> miniapps, save_setting -> + // store, miniapp_selected -> launcher) so a bare OEM gets device data, not just the + // Mentra app's MantleManager. + startDeviceEventRouter() + // Project the glasses' OTA events into the store for the toolkit.ota read surface. + startOtaService() + // Forward glasses mic_lc3 frames to the v2 cloud session so cloud transcription + // works for any host (not just the Mentra app's host-side MantleManager fork). + startAudioCloudUplink() + // Push device-setting changes to the glasses for ANY host, so + // toolkit.glasses.settings.set() reaches the device (not just the Mentra app). + startGlassesSettingsSync() + // Same for phone-notification config -> the native listener (Android). + startPhoneNotificationsSync() + // Bring up the local-miniapp engine so a bare OEM can run MentraJS miniapps: + // the LocalMiniappRuntime (registry + WebView bridge), the MentraJS router + // (crust-bound spawn/dispatch pump + launcher wiring), the DisplayProcessor + // (layout arbitration over the island stores), and the gallery-sync service. + // All idempotent — when the Mentra app's host bootstrap also calls these, the + // second call is a no-op. Host-only crashloop telemetry is attached + // separately by the Mentra app. + localMiniappRuntime.initialize() + ensureMiniappEngine() + displayProcessor.attachToRuntime() + gallerySyncService.initialize() + }, + /** Stop the runtime: tear down the settings sync + cloud client + mark stopped. */ + async stop() { + stopGlassesSettingsSync() + stopGlassesStatusProjection() + stopDeviceEventRouter() + stopOtaService() + stopAudioCloudUplink() + stopPhoneNotificationsSync() + cloudClientService.stop() + await bootstrapStop() + }, + glasses, + speech, + /** Cloud (cloud-v2) live-session status + account ops — island owns the client. */ + session, + /** Typed keyed user settings over the island-owned settings store. */ + settings, + /** Developer/debug surface (backend+cloud URL overrides, reconnect, min version). */ + dev, + /** Bug-report / feedback submission (island-owned RestComms). */ + incidents, + /** Firmware OTA read/observe surface (updateAvailable/status + on* subscriptions). */ + ota, + /** Gallery sync: status/onStatus · onNotice (host renders) · sync/cancel. */ + gallery, + /** Miniapp lifecycle (the WebView bridge primitives are exported separately). */ + miniapps, + /** First-time glasses discovery + pairing. */ + pairing, + /** Phone→glasses notification forwarding (Android). */ + phoneNotifications, + /** OS permissions (check/request/openSettings + per-miniapp requirements). */ + permissions, + /** Inbound alerts island→host (crashloop, version-incompatible, connection loss). */ + notifications, + display, + /** + * Escape hatches — the raw device-state zustand stores, grouped under `stores`, + * exposed so the Mentra app keeps using them directly instead of rewriting every + * screen onto typed facades. These are Mentra-app convenience, NOT the OEM + * contract — OEMs use the typed facades above. Prefer a facade where one exists. + */ + stores: { + glasses: useGlassesStore, + display: useDisplayStore, + core: useCoreStore, + connection: useConnectionStore, + gallerySync: useGallerySyncStore, + cloudClientStatus: useCloudClientStatusStore, + settings: useSettingsStore, + }, +} diff --git a/mobile/modules/island/src/runtime/bootstrap.ts b/mobile/modules/island/src/runtime/bootstrap.ts new file mode 100644 index 0000000000..238f30e945 --- /dev/null +++ b/mobile/modules/island/src/runtime/bootstrap.ts @@ -0,0 +1,81 @@ +/** + * Bootstrap — the toolkit's single front door (`toolkit.configure` / `start` / `stop`). + * + * The host hands island auth + config + analytics in one call; island owns the + * runtime construction behind `start()`. Internal services read this holder + * directly instead of exposing host-facing adapter bags. + */ + +export type SubjectTokenType = "supabase" | "authing" | (string & {}) + +export interface IslandAuth { + /** Returns the host's current (auto-refreshed) subject token for the backend. */ + getSubjectToken: () => Promise<{token: string; type: SubjectTokenType}> +} + +export interface IslandConfigValues { + /** cloud-v2 core service base URL (defaults resolved by the cloud client). */ + coreUrl?: string + /** cloud-v2 runtime service base URL. */ + runtimeUrl?: string + /** OEM identifier (Mentra is OEM #0); reserved for OEM auth/telemetry. */ + oemId?: string + /** + * LC3 frame size (bytes) the phone's mic encoder emits — announced to the + * cloud on connect (20 for G1, 40 for G2, …). Defaults to 20 if unset. + */ + audioFrameSizeBytes?: number + /** + * Dev-only live Metro/LAN host used to repair persisted local-miniapp dev URLs + * after the laptop changes networks. Omitted in production/OEM builds. + */ + devServerHost?: () => string | undefined +} + +export type IslandAnalytics = (event: string, props?: Record) => void + +export interface IslandConfigureOptions { + /** REQUIRED — the only must-have seam. The host owns login; island owns the rest. */ + auth: IslandAuth + config?: IslandConfigValues + analytics?: IslandAnalytics +} + +let options: IslandConfigureOptions | null = null +let started = false + +/** Hand island its auth + config + analytics. Call once, before `start()`. */ +export function configure(opts: IslandConfigureOptions): void { + options = opts +} + +/** Mark the runtime started. Idempotent. */ +export async function start(): Promise { + if (started) return + if (!options) { + throw new Error("toolkit.start() called before toolkit.configure()") + } + started = true +} + +/** Tear down. Idempotent. */ +export async function stop(): Promise { + started = false +} + +export function isStarted(): boolean { + return started +} + +/** The host-supplied auth provider, or null if not configured yet. */ +export function getAuth(): IslandAuth | null { + return options?.auth ?? null +} + +export function getConfigValues(): IslandConfigValues { + return options?.config ?? {} +} + +export function getAnalytics(): IslandAnalytics | null { + return options?.analytics ?? null +} diff --git a/mobile/modules/island/src/runtime/config.ts b/mobile/modules/island/src/runtime/config.ts index 386ab50f4e..9aab63c088 100644 --- a/mobile/modules/island/src/runtime/config.ts +++ b/mobile/modules/island/src/runtime/config.ts @@ -1,17 +1,10 @@ /** - * Runtime configuration — host-injected accessors used by services that - * cannot be fully self-contained inside the island module (LocalMiniappRuntime, - * LocalDisplayManager, LocalSttFallbackCoordinator, DisplayProcessor). + * Runtime-shared constants and DTOs. * - * The mobile manager calls `configureRuntime(...)` early at boot to wire in - * the manager's own stores and adapters. OEM hosts implement the same shape - * with their own backing. - * - * Keep this surface tight — every entry here is a coupling point between - * the host and the runtime. Prefer pushing data IN over pulling it via a - * getter when reasonable. + * The old host-injected runtime adapter surface has moved behind island-owned + * services and `toolkit.configure({auth, config, analytics})`. Keep this file + * limited to stable types that are shared across those services. */ -import type {AudioSubscription, TranscriptionData, TranslationData} from "@mentra/cloud-runtime/protocol" export type CloudClientConnectionStatus = "connected" | "connecting" | "reconnecting" | "disconnected" export type CloudClientAudioTransport = "udp" | "ws" | "offline" | "none" @@ -21,25 +14,6 @@ export interface CloudClientStatusSnapshot { audioTransport: CloudClientAudioTransport } -export interface CloudRuntimeTtsSpeakOptions { - voiceId?: string - voice_id?: string - modelId?: string - model_id?: string - voiceSettings?: Record - voice_settings?: Record -} - -export interface CloudRuntimeTtsSpeechSource { - audioUrl: string - contentType: string - source: "cloud" -} - -export interface CloudRuntimeTtsAdapter { - speak: (text: string, options?: CloudRuntimeTtsSpeakOptions) => Promise -} - export interface MiniappAuthToken { mentraUserId: string oemId?: string @@ -47,101 +21,6 @@ export interface MiniappAuthToken { expiresAt: number } -export interface MiniappAuthAdapter { - /** Mint or return a cached token scoped to one miniapp packageName. */ - getToken: (packageName: string, opts?: {minTtlMs?: number}) => Promise -} - -import type {ClientApp} from "../types/applet" - -/** - * Snapshot the host exposes about the connected glasses. The host's full - * glasses store is too rich for the runtime — these are the fields the - * runtime actually reads. Extra fields are passed through verbatim on the - * `glasses_connection` stream snapshot. - */ -export interface GlassesSnapshot { - connected: boolean - deviceModel?: string - modelName?: string - batteryLevel?: number - charging?: boolean - headUp?: boolean - /** Extra host-defined fields surfaced on glasses_connection. */ - [key: string]: unknown -} - -export interface SocketCommsAdapter { - sendMessage: (message: object) => void - updatePhoneSubscriptions: (subscriptions: string[]) => void -} - -/** - * Cloud-v2 (`@mentra/cloud-client`) runtime surface, wired in alongside the v1 - * `socketComms` path during the dual-cloud transition. The host owns the - * singleton CloudClient; this adapter is the thin slice the island runtime - * needs to drive transcription/translation subscriptions and fan results back - * to local miniapps. - * - * Typed against `@mentra/cloud-runtime/protocol` so the subscription/result - * shapes are the real wire types, not loosely-typed mirrors. Optional on - * `RuntimeHooks`: hosts still on v1-only leave it unset and the runtime keeps - * driving cloud transcription purely through `socketComms`. - */ -export interface CloudRuntimeAdapter { - /** Replace the v2 cloud's audio subscription set for the live session. */ - setSubscriptions: (subs: AudioSubscription[]) => Promise - /** Encrypt + send one LC3 (or PCM) audio frame over the v2 UDP path. */ - sendAudioFrame: (frame: Uint8Array) => void - /** Subscribe to v2 transcription results. Returns an unsubscribe fn. */ - onTranscript: (cb: (d: TranscriptionData) => void) => () => void - /** Subscribe to v2 translation results. Returns an unsubscribe fn. */ - onTranslation: (cb: (d: TranslationData) => void) => () => void - /** Current cloud-client runtime status, without host UI labels. */ - getStatus: () => CloudClientStatusSnapshot - /** Subscribe to cloud-client runtime status changes. Returns an unsubscribe fn. */ - onStatusChanged: (cb: (snapshot: CloudClientStatusSnapshot) => void) => () => void - /** Runtime TTS API. The cloud-client owns endpoint paths and validation. */ - tts: CloudRuntimeTtsAdapter - /** - * Whether any transcription/translation subscription is currently set on v2. - * The host's audio-capture site gates `sendAudioFrame` on this so we don't - * burn UDP bandwidth when nobody is subscribed on the v2 cloud. - */ - hasAudioSubscriptions: () => boolean - /** Whether the v2 live session is connected (handshake completed). */ - isConnected: () => boolean -} - -export interface AudioPlayRequest { - requestId: string - audioUrl: string - appId?: string - volume?: number - stopOtherAudio?: boolean -} - -export interface AudioPlaybackAdapter { - /** - * Play audio for a specific app. Calls onComplete when playback finishes - * or errors. Returns a promise that resolves once playback is dispatched. - */ - play: ( - request: AudioPlayRequest, - onComplete: (requestId: string, success: boolean, error: string | null, duration: number | null) => void, - ) => Promise | void - /** - * Stop playback for an app (e.g. on disconnect / close). - */ - stopForApp: (packageName: string) => void -} - -export interface MicRequirements { - shouldSendPcm: boolean - shouldSendLc3: boolean - shouldSendTranscript: boolean -} - /** * Result of an offline TTS synthesis. The island's TTSModelManager produces * this directly; the type is kept exported for hosts that wrap it. @@ -151,24 +30,6 @@ export interface TtsSynthesisResult { cleanup?: () => Promise | void } -/** - * Generic store accessor. The host wraps its Zustand / Redux / etc. selector - * so the island module never imports the host's store implementation. - */ -export interface StoreAccessor { - get: () => T -} - -export interface SettingsAccessor { - getSetting: (key: string) => T | undefined - setSetting: (key: string, value: T, persistImmediately?: boolean) => void - /** - * Subscribe to changes for one setting key. Returns an unsubscribe fn. - * Optional — coordinators that only read settings on demand can skip it. - */ - subscribeKey?: (key: string, onChange: (value: T | undefined) => void) => () => void -} - /** * Stable settings keys read by island services. Hosts must wire their own * settings store keys to these names. Mobile already uses these strings. @@ -237,81 +98,23 @@ export type NavTripSnapshot = { speedLimitMps?: number | null } -/** - * Navigation adapter — surface of the host's NavigationService that the - * runtime needs to wire `navigation_*` streams + request handlers for - * miniapps. - */ -export interface NavigationAdapter { - getState: () => "idle" | "navigating" | "rerouting" | "arrived" - getSnapshot: () => NavTripSnapshot | null - addListener: (l: (u: NavUpdate) => void) => () => void - addLocationListener: (l: (loc: NavLocation) => void) => () => void - addRouteListener: (l: (route: NavRoute) => void) => () => void - start: ( - coords: {lat: number; lng: number}, - options?: { - simulate?: boolean - speedMultiplier?: number - stops?: Array<{lat: number; lng: number}> - mode?: string - avoid?: {highways?: boolean; tolls?: boolean; ferries?: boolean} - missedTurnRerouteMeters?: number - }, - ) => Promise<{ok: boolean; error?: string}> - stop: () => Promise<{ok: boolean; error?: string}> - simulateDeviation: (offsetMeters?: number) => Promise<{ok: boolean; error?: string}> - setWrongSidewalkOffset: (enabled: boolean) => Promise<{ok: boolean; error?: string}> - setSkipCrossings: (enabled: boolean) => Promise<{ok: boolean; error?: string}> - requestPermission: () => Promise<{ok: boolean; accepted: boolean; error?: string}> - computeRoute: (payload: Record) => Promise<{ - ok: boolean - error?: string - routes?: Array<{ - points: Array<{lat: number; lng: number}> - totalDistanceMeters: number - totalDurationSeconds: number - summary?: string - steps?: NavRouteStep[] - }> - }> - /** - * Reverse-geocode a coordinate into a short road/route name. Used by - * the SDK pivot engine as a last-resort fallback when the Routes-API - * step's instruction text didn't yield a clean road name. Optional — - * hosts that don't implement it leave the SDK without a fallback, - * and pivots with no parseable instruction stay unlabeled. - */ - reverseGeocodeRoad?: (coord: {lat: number; lng: number}) => Promise<{ - ok: boolean - road?: string | null - error?: string - }> -} +// Navigation moved into island (NavigationService, called directly by the runtime's +// NavigationHandlers) — no longer a host-provided hook. The Nav* data types below stay +// (the runtime handlers use them). /** * Heading adapter — compass / device heading subscription. The host's * HeadingService wraps native sensor output; the runtime only needs the * subscribe-and-unsubscribe surface. */ -export interface HeadingAdapter { - addListener: (l: (degrees: number) => void) => () => void -} +// Location-tier control + the background GPS task moved into island +// (PhoneLocationService, driven directly by the runtime's recomputeLocation) — no +// longer a host-provided hook. -/** - * Location-tier control. Used by the runtime to request a higher GPS - * sample rate while a miniapp is subscribed to `location_update`. - * Implemented by the host's MantleManager (or equivalent). - */ -export interface LocationTierAdapter { - setLocationTier: (rate: "off" | "passive" | "low" | "high" | "realtime") => void -} +// Streaming (RTMP/SRT/WHIP publishing) moved into island (PhoneStreamCoordinator, +// called directly by the runtime) — no longer a host-provided hook. The stream config +// data types stay here (shared by runtime/streamConfig.ts + the island coordinator). -/** - * Streaming adapter — owns RTMP/SRT/WHIP publishing on the phone for local - * miniapps. The runtime calls these from its stream request handlers; the - * host's PhoneStreamCoordinator implements them. - */ export interface StreamVideoConfig { width?: number height?: number @@ -326,57 +129,6 @@ export interface StreamAudioConfig { noiseSuppression?: boolean } -export interface StreamingAdapter { - /** Glasses-confirmed publisher start result. */ - startUnmanaged: ( - packageName: string, - opts: { - streamUrl: string - video?: StreamVideoConfig - audio?: StreamAudioConfig - sound?: boolean - }, - ) => Promise - startManaged: ( - packageName: string, - opts: { - restreamDestinations?: Array - video?: StreamVideoConfig - audio?: StreamAudioConfig - sound?: boolean - /** "srt" (default; HLS playback + recording) or "whip" (sub-second WHEP, no HLS/recording). */ - ingest?: "srt" | "whip" - }, - ) => Promise - stop: (packageName: string, streamId?: string) => Promise - /** - * Subscribe to status updates produced by the coordinator (BLE-originated - * status, Cloudflare poll, lifecycle errors). Called per-(packageName, - * update) pair so the runtime can fan an EVENT into the right miniapp(s). - */ - setStatusSubscriber: ( - cb: ( - packageName: string, - update: {streamId: string; status: string; data?: Record; source: string}, - ) => void, - ) => void -} - -export interface StreamPublisherStartResult { - streamId: string - status: string - resolvedConfig?: Record -} - -export interface ManagedStreamStartResult extends StreamPublisherStartResult { - liveInputId: string - /** "hls" (SRT/RTMP ingest) or "webrtc" (WHIP ingest -> WHEP playback). */ - mode: "hls" | "webrtc" - hlsUrl: string - dashUrl: string - webrtcUrl?: string -} - export type CameraRoiPosition = "center" | "bottom" | "top" export type CameraFovPreset = "narrow" | "standard" | "wide" @@ -396,13 +148,9 @@ export interface CameraFovResult { timestamp: number } -/** - * Camera settings adapter. Used for local miniapp camera.setFov so the - * miniapp Promise resolves from the glasses-side hardware-applied ack flow. - */ -export interface CameraSettingsAdapter { - setFov: (packageName: string, request: CameraFovRequest) => Promise -} +// Camera FOV moved into island — LocalMiniappRuntime calls BluetoothSdk.setCameraFov +// directly (a pure passthrough with no host coupling) — no longer a host hook. +// CameraFovRequest/CameraFovResult below stay: they're the miniapp-facing payload types. /** One audited inter-miniapp call. An LLM caller (Mentra AI) will eventually do * something a user wants to trace — every interop op emits one of these. */ @@ -419,160 +167,3 @@ export interface InteropAuditEvent { /** MiniappErrorCode when ok is false. */ errorCode?: string } - -/** - * Inter-miniapp interop adapter — backs `session.miniapps` (list/start/stop) - * and `session.actions.invoke`. The host provides the system-app *policy* and - * the app-store operations so the runtime stays decoupled from the host's - * store and its hardcoded SYSTEM_APPS list. Wired by the host at bootstrap. - */ -export interface InteropAdapter { - /** Is this package a system app — allowed to use the SYSTEM-only interop APIs? */ - isSystemApp: (packageName: string) => boolean - /** Snapshot of all installed miniapps (the host's app-store state). */ - listApps: () => ClientApp[] - /** - * Start (and foreground) another miniapp — user-tap semantics. Resolves true - * if it actually started; false if a host gate aborted it (incompatible - * hardware, captions STT gate, …) or its JS context failed to spawn. - */ - startApp: (packageName: string) => Promise - /** Stop another miniapp. */ - stopApp: (packageName: string) => Promise - /** - * Headless-wake a miniapp's background context AND wait for its CONNECT - * handshake (for action invoke). No foreground, no arbitration. Rejects on - * spawn/connect failure. - */ - wakeMiniapp: (packageName: string) => Promise - /** Optional audit sink — one event per interop call (caller, op, outcome). */ - audit?: (event: InteropAuditEvent) => void -} - -export interface RuntimeHooks { - socketComms?: SocketCommsAdapter - /** - * Cloud-v2 (`@mentra/cloud-client`) runtime adapter. Additive alongside - * `socketComms` during the dual-cloud transition; unset on v1-only hosts. - */ - cloud?: CloudRuntimeAdapter - /** - * Package-scoped backend auth for local miniapps. The host owns the real - * Core/runtime credentials; this adapter returns only miniapp tokens. - */ - miniappAuth?: MiniappAuthAdapter - audioPlayback?: AudioPlaybackAdapter - /** Returns the connected glasses' status snapshot. */ - glassesStatus?: StoreAccessor - settings?: SettingsAccessor - /** Google Navigation SDK adapter (turn-by-turn + computeRoute). */ - navigation?: NavigationAdapter - /** Device heading / compass adapter. */ - heading?: HeadingAdapter - /** Location-tier escalation (e.g. realtime GPS when a trip is active). */ - locationTier?: LocationTierAdapter - /** - * The dev machine's live LAN host (no port), derived by the host from - * Metro's `hostUri` — the address this dev bundle was actually served from, - * so it is always current for whatever network the phone is on. Used to - * repair persisted dev-miniapp URLs that froze a previous network's IP. - * Unset outside Metro-served dev builds. - */ - devServerHost?: () => string | undefined - /** - * Forward processed display events into the host's mirror store. The - * default no-op skips the mirror — installed-only hosts (no UI mirror) - * can leave this unset. - */ - setDisplayEvent?: (event: string) => void - /** - * Send a processed display event to the connected glasses through the host's - * native Bluetooth bridge. - */ - sendDisplayEvent?: (event: Record) => Promise | void - /** - * Subscribe to host glasses-status changes when the runtime needs live - * device-model updates. Returns an unsubscribe function. - */ - subscribeGlassesStatus?: (onChange: (changed: Partial) => void) => () => void - /** - * Restart the host-managed local transcriber. The runtime only decides when - * local STT is needed; the host owns the native STT implementation. - */ - restartTranscriber?: () => Promise | void - /** - * Apply the union of cloud and local microphone requirements through the - * host's native Bluetooth bridge. - */ - setMicRequirements?: (requirements: MicRequirements) => Promise | void - /** Phone-orchestrated photo capture (session.camera.takePhoto). */ - photo?: PhotoAdapter - /** Phone-orchestrated video recording (session.camera.startVideoRecording). */ - videoRecording?: VideoRecordingAdapter - /** Phone-orchestrated camera settings (session.camera.setFov). */ - cameraSettings?: CameraSettingsAdapter - /** Phone-orchestrated RTMP/SRT/WHIP publishing. */ - streaming?: StreamingAdapter - /** Inter-miniapp interop (session.miniapps + session.actions.invoke). */ - interop?: InteropAdapter -} - -/** - * Video recording adapter — start/stop a local video recording on the glasses. - * The runtime calls these from its handleVideoRecordingStart/Stop handlers; the - * host's PhoneVideoCoordinator implements them (drives the glasses over BLE via - * the bluetooth-sdk startVideoRecording/stopVideoRecording). Unlike photo, this - * returns recording control status only — no uploaded URL is returned. - */ -export interface VideoRecordingAdapter { - startRecording: ( - packageName: string, - opts: { - width?: number - height?: number - fps?: number - sound?: boolean - save?: boolean - }, - ) => Promise<{recordingId: string}> - stopRecording: (packageName: string, recordingId?: string) => Promise - /** - * Stop any recordings still owned by an app (e.g. on miniapp disconnect/crash) - * so the glasses don't keep recording until the max-recording timeout. - */ - stopForApp?: (packageName: string) => Promise -} - -/** - * Photo adapter — end-to-end takePhoto(). The runtime calls `takePhoto` - * from its handlePhoto handler; the host's PhonePhotoCoordinator implements - * it (mints upload token via the v2 cloud route, drives glasses over BLE, - * long-polls for the download URL). - */ -export interface PhotoAdapter { - takePhoto: ( - packageName: string, - opts: { - size?: "low" | "medium" | "high" | "max" | "small" | "large" | "full" - compress?: "none" | "low" | "medium" | "high" - sound?: boolean - saveToGallery?: boolean - exposureTimeNs?: number - }, - ) => Promise<{ - photoUrl: string - mimeType: string - size: number - requestId: string - }> -} - -let hooks: RuntimeHooks = {} - -export function configureRuntime(next: RuntimeHooks): void { - hooks = {...hooks, ...next} -} - -export function getRuntimeHooks(): RuntimeHooks { - return hooks -} diff --git a/mobile/modules/island/src/services/AppRegistry.ts b/mobile/modules/island/src/services/AppRegistry.ts index 0eabcfb3e8..8c1d2a19f4 100644 --- a/mobile/modules/island/src/services/AppRegistry.ts +++ b/mobile/modules/island/src/services/AppRegistry.ts @@ -24,7 +24,7 @@ import {AsyncResult, Result, result as Res} from "typesafe-ts" import type {AppletPermission, AppPermissionType, AppletType, ClientApp, DeclaredAction} from "../types/applet" import {HardwareRequirement, HardwareRequirementLevel, HardwareType} from "../types" -import {getRuntimeHooks} from "../runtime/config" +import {getConfigValues} from "../runtime/bootstrap" import {storage} from "../utils/storage/storage" import {printDirectory} from "../utils/storage/zip" import {checkManifestVersions} from "./manifestVersionGate" @@ -936,7 +936,7 @@ function configuredDevHost(): string | undefined { if (/^[\w.-]+$/.test(explicit)) return explicit } } - return getRuntimeHooks().devServerHost?.() + return getConfigValues().devServerHost?.() } function isPrivateLanHost(hostname: string): boolean { diff --git a/mobile/modules/island/src/services/AudioCloudUplink.ts b/mobile/modules/island/src/services/AudioCloudUplink.ts new file mode 100644 index 0000000000..570a2edc1b --- /dev/null +++ b/mobile/modules/island/src/services/AudioCloudUplink.ts @@ -0,0 +1,34 @@ +/** + * Audio cloud uplink — island-owned. Subscribes to the glasses' `mic_lc3` BLE frames + * and forwards each one to the v2 cloud session (`cloud.sendAudioFrame`), gated on the + * cloud being connected AND having active audio subscriptions so we don't burn the + * UDP/encrypt path when nothing upstream is listening. + * + * This is the device→cloud audio data-plane for the v2 runtime. It used to live in the + * host's MantleManager `mic_lc3` handler (the "cloud fork"); moved here so ANY host — + * including a bare OEM that never runs the v1 SocketComms plane — gets cloud + * transcription/translation, not just the first-party Mentra app. + * + * The old v1 UDP/SocketComms upload legs were removed; MantleManager only keeps + * host-side debug mic-activity tracking and on-device PCM fan-out. + * + * Started by `toolkit.start()`. Idempotent. + */ +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import {cloudClientService} from "./CloudClientService" + +let sub: {remove: () => void} | null = null + +export function startAudioCloudUplink(): void { + if (sub) return + sub = BluetoothSdk.addListener("mic_lc3", (event) => { + if (cloudClientService.isConnected() && cloudClientService.hasAudioSubscriptions()) { + cloudClientService.sendAudioFrame(new Uint8Array(event.lc3)) + } + }) +} + +export function stopAudioCloudUplink(): void { + sub?.remove() + sub = null +} diff --git a/mobile/src/services/AudioPlaybackService.ts b/mobile/modules/island/src/services/AudioPlaybackService.ts similarity index 99% rename from mobile/src/services/AudioPlaybackService.ts rename to mobile/modules/island/src/services/AudioPlaybackService.ts index c267dc975f..6d4ad43af9 100644 --- a/mobile/src/services/AudioPlaybackService.ts +++ b/mobile/modules/island/src/services/AudioPlaybackService.ts @@ -1,6 +1,6 @@ import {createAudioPlayer, AudioPlayer, AudioStatus, setAudioModeAsync} from "expo-audio" -import BluetoothSdk from "@mentra/bluetooth-sdk" -import {BgTimer} from "@mentra/island" +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import {BgTimer} from "../utils/timers" interface AudioPlayRequest { requestId: string diff --git a/mobile/modules/island/src/services/CloudClientService.ts b/mobile/modules/island/src/services/CloudClientService.ts new file mode 100644 index 0000000000..955a45ccbc --- /dev/null +++ b/mobile/modules/island/src/services/CloudClientService.ts @@ -0,0 +1,480 @@ +/** + * Owns the singleton `@mentra/cloud-client` (CloudClient) for the island runtime + * — the keystone move that pulls backend comms INTO island. + * + * island constructs the client from island-owned pieces (the native UDP socket, + * the MMKV secure store, the cloud-status store) plus the two things the host + * provides through the toolkit front door: `auth.getSubjectToken` (the one + * permanent seam — the OEM owns login) and the resolved cloud endpoints + * (`config.coreUrl/runtimeUrl`, computed host-side from dev/settings). On init it + * exposes cloud runtime methods directly from this service, so the host no longer + * injects a cloud adapter — island owns construction and wiring. + * + * The local-miniapp path drives the cloud's transcription/translation through + * this service. + */ +import {CloudClient, setNativeUdp, setSecureStorage} from "@mentra/cloud-client/react-native" +import type {RuntimeSnapshot} from "@mentra/cloud-client/react-native" +import type {SubjectTokenType} from "@mentra/cloud-client" +import type {AudioSubscription, TranscriptionData, TranslationData} from "@mentra/cloud-runtime/protocol" + +import {getAuth, getConfigValues} from "../runtime/bootstrap" +import {useSettingsStore, SETTINGS} from "../stores/settings" +import {type CloudClientStatusSnapshot, type MiniappAuthToken} from "../runtime/config" + +/** Core/cloud-client may speak Unix seconds while the miniapp SDK uses JS millis. */ +function normalizeExpiresAt(expiresAt: number): number { + return expiresAt < 10_000_000_000 ? expiresAt * 1000 : expiresAt +} +import {createCloudUdpSocket} from "../utils/cloudClient/RnUdpAdapter" +import {cloudSecureStore} from "../utils/cloudClient/cloudSecureStore" +import {useCloudClientStatusStore} from "../stores/cloudClientStatus" +import {islandNotifications} from "./NotificationsEmitter" +import {BgTimer} from "../utils/timers" + +const LOG_TAG = "cloudClient" + +// Persistent cloud-disconnect detector: the cloud-client reconnect loop is infinite +// and never surfaces a "giving up" signal, so "persistent failure" = the session has +// been continuously NOT connected for this long. A quick reconnect cancels it; brief +// flaps that recover don't fire. Raised as toolkit.notifications(connection_failed_persistent). +const CLOUD_PERSISTENT_FAILURE_MS = 60_000 + +/** Cancel the pending persistent-failure alarm + re-arm for the next outage. */ +function clearPersistentFailureAlarm(): void { + if (persistentFailureTimer) { + BgTimer.clearTimeout(persistentFailureTimer) + persistentFailureTimer = null + } + persistentFailureNotified = false +} + +type Lc3FrameSizeBytes = 20 | 40 | 60 + +// Neutral last-ditch fallbacks (reachable under `adb reverse`) for when the host +// passes no endpoints. The host normally resolves the real URLs (dev override / +// Metro host / env) and hands them in via `toolkit.configure({config})`. +const FALLBACK_CORE_URL = "http://localhost:3000" +const FALLBACK_RUNTIME_URL = "http://localhost:3001" + +let client: CloudClient | null = null +let connected = false +let persistentFailureTimer: ReturnType | null = null +let persistentFailureNotified = false +let audioSubscriptions: AudioSubscription[] = [] +let transportsReady = false +/** Endpoints to build with — seeded from config, overridable via reconnect(). */ +let endpointsOverride: {core: string; runtime: string} | null = null +let runtimeStatusUnsubscribe: (() => void) | null = null +let transcriptUnsubscribe: (() => void) | null = null +let translationUnsubscribe: (() => void) | null = null + +const transcriptListeners = new Set<(d: TranscriptionData) => void>() +const translationListeners = new Set<(d: TranslationData) => void>() +const statusListeners = new Set<(snapshot: CloudClientStatusSnapshot) => void>() +const connectionListeners = new Set<(connected: boolean) => void>() + +function resolveEndpoints(): {core: string; runtime: string} { + if (endpointsOverride) return endpointsOverride + const cfg = getConfigValues() + return { + core: cfg.coreUrl?.trim() || FALLBACK_CORE_URL, + runtime: cfg.runtimeUrl?.trim() || FALLBACK_RUNTIME_URL, + } +} + +function frameSizeBytes(): Lc3FrameSizeBytes { + // Read LIVE from the island settings store (the source of truth) so a + // reconnect after a glasses swap (G1=20 ↔ G2=40) picks up the new frame size, + // not the one-time toolkit.configure({config}) snapshot. Falls back to the + // config value (set at boot) then 20 when the setting is unset. + const fromSettings = useSettingsStore.getState().getSetting(SETTINGS.lc3_frame_size.key) + const size = fromSettings ?? getConfigValues().audioFrameSizeBytes + return size === 20 || size === 40 || size === 60 ? size : 20 +} + +async function getSubjectToken(): Promise<{token: string; type: SubjectTokenType}> { + const a = getAuth() + if (!a) throw new Error("cloudClient: toolkit.configure({auth}) not called") + const r = await a.getSubjectToken() + // IslandAuth's SubjectTokenType is intentionally open (`string & {}`) so OEMs + // can use other token kinds; cloud-client's is a closed union. The host's + // actual value ("supabase") is valid — narrow at this boundary. + return {token: r.token, type: r.type as SubjectTokenType} +} + +// Local-dev runtime token: when the runtime endpoint is a local cloud (localhost / +// emulator loopback) in a __DEV__ build, the runtime can't verify a real Supabase +// subject token, so fetch a dev runtime token from the local auth server instead of +// exchanging the core token. Ported from the host cloudClient when the client moved +// into island; a no-op for any real (non-local) cloud or release build. +const LOCAL_AUTH_PORT = 3002 +let localDevRuntimeToken: {token: string; expiresAtMs: number} | null = null + +function shouldUseLocalDevRuntimeToken(endpoints: {runtime: string}): boolean { + if (!__DEV__) return false + try { + const host = new URL(endpoints.runtime).hostname + return host === "localhost" || host === "127.0.0.1" || host === "10.0.2.2" + } catch { + return false + } +} + +async function getLocalDevRuntimeToken(opts?: {forceRefresh?: boolean}): Promise { + const now = Date.now() + if (!opts?.forceRefresh && localDevRuntimeToken && localDevRuntimeToken.expiresAtMs - now > 60_000) { + return localDevRuntimeToken.token + } + const base = new URL(resolveEndpoints().runtime) + base.port = String(LOCAL_AUTH_PORT) + base.pathname = "/api/dev/runtime-token" + base.search = new URLSearchParams({userId: "local-phone-user", oemId: "mentra"}).toString() + const res = await fetch(base.toString()) + if (!res.ok) { + throw new Error(`cloudClient: local dev runtime token failed (${res.status})`) + } + const body = (await res.json()) as {access_token?: string; expires_in?: number} + if (!body.access_token) { + throw new Error("cloudClient: local dev runtime token response missing access_token") + } + localDevRuntimeToken = {token: body.access_token, expiresAtMs: now + (body.expires_in ?? 300) * 1000} + return body.access_token +} + +function toCloudClientStatusSnapshot(snapshot: RuntimeSnapshot): CloudClientStatusSnapshot { + return {status: snapshot.status, audioTransport: snapshot.audioTransport} +} + +function currentRuntimeStatus(): CloudClientStatusSnapshot { + const state = useCloudClientStatusStore.getState() + return {status: state.status, audioTransport: state.audioTransport} +} + +function setRuntimeStatus(snapshot: RuntimeSnapshot): void { + useCloudClientStatusStore.getState().setSnapshot(snapshot) + emitStatus(toCloudClientStatusSnapshot(snapshot)) +} + +function resetRuntimeStatus(): void { + useCloudClientStatusStore.getState().reset() + emitStatus(currentRuntimeStatus()) +} + +function stringifyMeta(meta: unknown): string { + if (!meta || typeof meta !== "object") return "" + try { + return ` ${JSON.stringify(meta)}` + } catch { + return "" + } +} + +const cloudLogger = { + debug: (msg: string, meta?: unknown) => console.log(`${LOG_TAG}: debug: ${msg}${stringifyMeta(meta)}`), + info: (msg: string, meta?: unknown) => console.log(`${LOG_TAG}: info: ${msg}${stringifyMeta(meta)}`), + warn: (msg: string, meta?: unknown) => console.warn(`${LOG_TAG}: warn: ${msg}${stringifyMeta(meta)}`), + error: (msg: string, meta?: unknown) => console.warn(`${LOG_TAG}: error: ${msg}${stringifyMeta(meta)}`), +} + +function notifyConnectionListeners(next: boolean): void { + for (const l of connectionListeners) { + try { + l(next) + } catch (err) { + console.warn(`${LOG_TAG}: connection listener threw: ${(err as Error)?.message ?? err}`) + } + } +} + +function emitTranscript(data: TranscriptionData): void { + for (const l of transcriptListeners) { + try { + l(data) + } catch (err) { + console.warn(`${LOG_TAG}: transcript listener threw: ${(err as Error)?.message ?? err}`) + } + } +} + +function emitTranslation(data: TranslationData): void { + for (const l of translationListeners) { + try { + l(data) + } catch (err) { + console.warn(`${LOG_TAG}: translation listener threw: ${(err as Error)?.message ?? err}`) + } + } +} + +function emitStatus(snapshot: CloudClientStatusSnapshot): void { + for (const l of statusListeners) { + try { + l(snapshot) + } catch (err) { + console.warn(`${LOG_TAG}: status listener threw: ${(err as Error)?.message ?? err}`) + } + } +} + +function ensureTransports(): void { + if (transportsReady) return + transportsReady = true + setNativeUdp(() => createCloudUdpSocket()) + setSecureStorage(cloudSecureStore) +} + +function clearRuntimeEventSubscriptions(): void { + runtimeStatusUnsubscribe?.() + runtimeStatusUnsubscribe = null + transcriptUnsubscribe?.() + transcriptUnsubscribe = null + translationUnsubscribe?.() + translationUnsubscribe = null +} + +function construct(): void { + ensureTransports() + + const endpoints = resolveEndpoints() + console.log(`${LOG_TAG}: endpoints ${JSON.stringify(endpoints)}`) + + const coreAuth = {getSubjectToken} + const auth = shouldUseLocalDevRuntimeToken(endpoints) + ? {core: coreAuth, runtime: {getToken: getLocalDevRuntimeToken}} + : {core: coreAuth, runtime: {source: "core" as const}} + + client = new CloudClient({ + endpoints, + // The phone LC3-encodes mic audio (even in phone/simulated mode); announce + // LC3 at 16 kHz with the frame size the encoder emits. + audio: {codec: "lc3", sampleRate: 16000, frameSizeBytes: frameSizeBytes()}, + auth, + logger: cloudLogger, + }) + + const c = client + clearRuntimeEventSubscriptions() + runtimeStatusUnsubscribe = c.runtime.onStatusChanged((status) => { + if (c !== client) return + setRuntimeStatus(status) + }) + transcriptUnsubscribe = c.runtime.onTranscript((data) => { + if (c !== client) return + emitTranscript(data) + }) + translationUnsubscribe = c.runtime.onTranslation((data) => { + if (c !== client) return + emitTranslation(data) + }) + setRuntimeStatus(c.runtime.getStatus()) + + c.runtime.onConnected(() => { + if (c !== client) return + connected = true + console.log(`${LOG_TAG}: runtime connected`) + // Cloud is up — cancel any pending persistent-failure alarm and re-arm for the + // next outage. + clearPersistentFailureAlarm() + // Re-apply the authoritative subscription set, including the empty set, so + // a fresh phone launch clears any stale server-side audio subscriptions. + c.runtime + .setSubscriptions(audioSubscriptions) + .catch((err) => + console.warn(`${LOG_TAG}: re-applying queued subscriptions failed: ${(err as Error)?.message ?? err}`), + ) + notifyConnectionListeners(true) + }) + c.runtime.onDisconnected((info) => { + if (c !== client) return + connected = false + console.log(`${LOG_TAG}: runtime disconnected (${info.reason})`) + // Arm the persistent-failure alarm once; if we're still down when it fires, raise + // the notification. A reconnect within the window cancels it (onConnected above). + if (!persistentFailureTimer && !persistentFailureNotified) { + persistentFailureTimer = BgTimer.setTimeout(() => { + persistentFailureTimer = null + if (connected) return + persistentFailureNotified = true + islandNotifications.emit({ + kind: "connection_failed_persistent", + reason: `Cloud session has been disconnected for over ${Math.round(CLOUD_PERSISTENT_FAILURE_MS / 1000)}s`, + metadata: {downForMs: CLOUD_PERSISTENT_FAILURE_MS, lastReason: info.reason}, + timestamp: Date.now(), + }) + }, CLOUD_PERSISTENT_FAILURE_MS) + } + notifyConnectionListeners(false) + }) + c.runtime.onError((err) => { + console.warn(`${LOG_TAG}: runtime error: ${err.code}`) + }) + + // Best-effort connect. Do not crash the app if the dev cloud is unreachable. + c.runtime + .connect() + .then(() => console.log(`${LOG_TAG}: connect() resolved`)) + .catch((err) => console.warn(`${LOG_TAG}: connect() failed: ${err?.message ?? err}`)) +} + +/** + * island's cloud client. Owns the singleton CloudClient, its connection state, + * and the runtime-hook wiring. + */ +export const cloudClientService = { + /** + * Construct (once) + connect the client. Idempotent. Best-effort connect — a + * failure is logged and the app keeps running. Requires + * `toolkit.configure({auth, config})` first. + */ + init(): void { + if (client) return + construct() + }, + + /** + * Tear down + rebuild. Pass new endpoints to switch URLs; pass `null` to CLEAR + * a prior override and fall back to the boot config (so cleared/default cloud + * URLs don't keep reconnecting to a stale override); omit to keep the current. + */ + reconnect(endpoints?: {core: string; runtime: string} | null): void { + if (endpoints !== undefined) endpointsOverride = endpoints + try { + client?.runtime.close() + } catch (err) { + console.warn(`${LOG_TAG}: reconnect close() failed: ${(err as Error)?.message ?? err}`) + } + clearRuntimeEventSubscriptions() + clearPersistentFailureAlarm() + + const wasConnected = connected + client = null + localDevRuntimeToken = null + connected = false + resetRuntimeStatus() + if (wasConnected) notifyConnectionListeners(false) + + construct() + }, + + /** Tear down the client + connection (the toolkit.stop() lifecycle). */ + stop(): void { + try { + client?.runtime.close() + } catch (err) { + console.warn(`${LOG_TAG}: stop close() failed: ${(err as Error)?.message ?? err}`) + } + clearRuntimeEventSubscriptions() + clearPersistentFailureAlarm() + const wasConnected = connected + client = null + localDevRuntimeToken = null + connected = false + resetRuntimeStatus() + if (wasConnected) notifyConnectionListeners(false) + }, + + /** Package-scoped backend auth token for a local miniapp (cloud-v2 auth). The host + * owns the Core/runtime credentials; this returns only the miniapp token. */ + async getMiniappAuthToken(packageName: string, opts?: {minTtlMs?: number}): Promise { + if (!client) construct() + const c = client + if (!c) throw new Error("cloud client not initialized") + const {token, expiresAt} = await c.auth.getMiniappToken(packageName, opts) + const identity = c.auth.identity + return { + mentraUserId: identity.mentraUserId, + oemId: identity.oemId, + token, + expiresAt: normalizeExpiresAt(expiresAt), + } + }, + + /** Device-side managed photo (cloud-v2): presign now, deliver bytes, await ready. */ + startManagedPhoto(opts: Record = {}) { + if (!client) throw new Error("cloud client not connected") + return client.runtime.startManagedPhoto(opts) + }, + awaitManagedPhotoReady(requestId: string) { + if (!client) throw new Error("cloud client not connected") + return client.runtime.awaitManagedPhotoReady(requestId) + }, + + /** Managed stream (cloud-v2): provision ingest+playback on the runtime. */ + startManagedStream(opts: Record = {}) { + if (!client) throw new Error("cloud client not connected") + return client.runtime.startManagedStream(opts) + }, + getManagedStreamStatus(streamId: string) { + if (!client) throw new Error("cloud client not connected") + return client.runtime.getManagedStreamStatus(streamId) + }, + stopManagedStream(streamId: string) { + if (!client) throw new Error("cloud client not connected") + return client.runtime.stopManagedStream(streamId) + }, + + /** Current live-session connection state (handshake completed). */ + isConnected(): boolean { + return connected + }, + + /** Subscribe to connection-state transitions. Returns an unsubscribe fn. */ + onConnectionChange(listener: (connected: boolean) => void): () => void { + connectionListeners.add(listener) + return () => { + connectionListeners.delete(listener) + } + }, + + /** Replace the v2 cloud's audio subscription set for the live session. */ + async setSubscriptions(subs: AudioSubscription[]): Promise { + // Cache unconditionally so the desired set survives a not-yet-connected + // session and reconnects; the onConnected handler re-applies the cached set. + audioSubscriptions = subs + const c = client + if (connected && c) { + await c.runtime.setSubscriptions(subs) + } + }, + + sendAudioFrame(frame: Uint8Array): void { + client?.runtime.sendAudioFrame(frame) + }, + + onTranscript(cb: (d: TranscriptionData) => void): () => void { + transcriptListeners.add(cb) + return () => { + transcriptListeners.delete(cb) + } + }, + + onTranslation(cb: (d: TranslationData) => void): () => void { + translationListeners.add(cb) + return () => { + translationListeners.delete(cb) + } + }, + + getStatus(): CloudClientStatusSnapshot { + return currentRuntimeStatus() + }, + + onStatusChanged(cb: (snapshot: CloudClientStatusSnapshot) => void): () => void { + statusListeners.add(cb) + return () => { + statusListeners.delete(cb) + } + }, + + tts: { + speak(text: string, options?: Parameters[1]) { + if (!client) throw new Error("cloud client not connected") + return client.runtime.tts.speak(text, options) + }, + }, + + hasAudioSubscriptions(): boolean { + return audioSubscriptions.length > 0 + }, +} diff --git a/mobile/modules/island/src/services/DeviceEventRouter.ts b/mobile/modules/island/src/services/DeviceEventRouter.ts new file mode 100644 index 0000000000..ff09fecafd --- /dev/null +++ b/mobile/modules/island/src/services/DeviceEventRouter.ts @@ -0,0 +1,188 @@ +/** + * Device-event router — island-owned. Subscribes to the inbound native BLE/device + * events and routes them into the island runtime: device stores, the process event + * bus, the photo/stream coordinators, and local miniapps (via forwardEvent). + * + * Why this exists: island owns the stores, coordinators, miniapp runtime, and facades, + * but it never *subscribed to the device* for most events — the host MantleManager was + * still the event router the whole runtime secretly depended on. So a bare OEM that + * imported island + called toolkit.start() got a connected runtime with almost no device + * data flowing in (no miniapp input, dead gallery sync, starved coordinators). This + * service moves those inbound bridges into island so ANY host gets them. + * + * Scope: only the NON-v1 legs move here. The v1 SocketComms forwards (touch→cloud, + * battery→cloud, the cloud-SDK stream/photo legs, etc.) stay in MantleManager and die at + * v1 retirement. For events MantleManager also forwards over v1, it keeps its v1 leg and + * this router adds the island leg (no double-handling — owned-stream/photo legs are + * `owns()`-gated; the forwardEvent legs were removed from MantleManager). + * + * Started by `toolkit.start()`. Idempotent. + */ +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" + +import restComms from "./RestComms" +import localMiniappRuntime from "./LocalMiniappRuntime" +import localSttFallbackCoordinator from "./LocalSttFallbackCoordinator" +import {phonePhotoCoordinator} from "./PhonePhotoCoordinator" +import {phoneStreamCoordinator} from "./PhoneStreamCoordinator" +import {useGlassesStore} from "../stores/glasses" +import {useSettingsStore} from "../stores/settings" +import {useAppStatusStore} from "../stores/apps" +import GlobalEventEmitter from "../utils/GlobalEventEmitter" + +let subs: Array<{remove: () => void}> = [] + +export function startDeviceEventRouter(): void { + if (subs.length) return + + // --- device state → island stores --- + + // Standalone WiFi status → glasses store. + subs.push( + BluetoothSdk.addListener("wifi_status_change", (event) => { + const {type: _type, ...wifi} = event + useGlassesStore.getState().setGlassesInfo({wifi}) + }), + ) + + // Hotspot status → glasses store + event bus. island's own gallerySyncService listens + // on the bus for these, so without this bridge island's gallery sync is dead. + subs.push( + BluetoothSdk.addListener("hotspot_status_change", (event) => { + const enabled = event.state === "enabled" + const ssid = enabled ? event.ssid : "" + const password = enabled ? event.password : "" + const localIp = enabled ? event.localIp : "" + useGlassesStore.getState().setHotspotInfo(enabled, ssid, password, localIp) + GlobalEventEmitter.emit("hotspot_status_change", {enabled, ssid, password, local_ip: localIp}) + }), + ) + subs.push( + BluetoothSdk.addListener("hotspot_error", (event) => { + GlobalEventEmitter.emit("hotspot_error", {error_message: event.errorMessage, timestamp: event.timestamp}) + }), + ) + + // Glasses gallery content counts → event bus (consumed by gallerySyncService). + subs.push( + BluetoothSdk.addListener("gallery_status", (event) => { + GlobalEventEmitter.emit("gallery_status", { + photos: event.photos, + videos: event.videos, + total: event.total, + has_content: event.hasContent, + camera_busy: event.cameraBusy, + }) + }), + ) + + // Hardware-originated setting changes (user changes a setting ON the glasses) → store. + // The inbound complement to GlassesSettingsSync (which only pushes store→device). + subs.push( + BluetoothSdk.addListener("save_setting", async (event) => { + await useSettingsStore.getState().setSetting(event.key, event.value) + }), + ) + + // --- coordinators (owns()-gated; MantleManager keeps the cloud-SDK v1 legs) --- + + // Phone-owned photo errors settle the in-flight long-poll fast (vs. timeout). Cloud-app + // photos forward via restComms (island REST). MantleManager no longer handles this. + subs.push( + BluetoothSdk.addListener("photo_response", (event) => { + if (event.requestId && phonePhotoCoordinator.owns(event.requestId)) { + if (event.state === "error") { + phonePhotoCoordinator.handlePhotoError( + event.requestId, + event.errorCode ?? "GLASSES_ERROR", + event.errorMessage ?? "Glasses reported an error", + ) + } + return + } + restComms.sendPhotoResponse(event) + }), + ) + + // Phone-owned stream status / keep-alive → the stream coordinator. Non-owned (cloud-SDK) + // streams stay on MantleManager's v1 SocketComms leg. + subs.push( + BluetoothSdk.addListener("stream_status", (event) => { + if (event.streamId && phoneStreamCoordinator.owns(event.streamId)) { + phoneStreamCoordinator.handleGlassesStatus(event) + } + }), + ) + subs.push( + BluetoothSdk.addListener("keep_alive_ack", (event) => { + if (event.streamId && phoneStreamCoordinator.owns(event.streamId)) { + phoneStreamCoordinator.handleKeepAliveAck(event) + } + }), + ) + + // --- device input → local miniapps (forwardEvent is subscriber-gated; a no-op when + // no miniapp listens). MantleManager keeps the v1 SocketComms legs for these. --- + + subs.push( + BluetoothSdk.addListener("button_press", (event) => { + localMiniappRuntime.forwardEvent("button_press", event) + }), + ) + subs.push( + BluetoothSdk.addListener("touch_event", (event) => { + localMiniappRuntime.forwardEvent("touch_event", event) + }), + ) + // G2 IMU accelerometer — payload already matches the miniapp AccelData shape. + subs.push( + BluetoothSdk.addListener("accel_event", (event) => { + localMiniappRuntime.forwardEvent("accel_event", { + x: event.x, + y: event.y, + z: event.z, + timestamp: typeof event.timestamp === "number" ? event.timestamp : Date.now(), + }) + }), + ) + // Head position — translate native {up:boolean} → SDK {position:"up"|"down"}. + subs.push( + BluetoothSdk.addListener("head_up", (event) => { + localMiniappRuntime.forwardEvent("head_up", {position: event.up ? "up" : "down", timestamp: Date.now()}) + }), + ) + // On-device STT transcripts → local miniapps, but ONLY when local STT fallback is the + // active engine (cloud STT is down). When the cloud WS is up, cloud transcripts reach + // miniapps independently via the same forwardEvent, so gating on isActive() avoids + // double-delivery. forwardEvent is subscriber-gated — a no-op when no miniapp listens. + // (Was MantleManager.handle_local_transcription; its offline-captions display branch + // went away with the pseudo captions renderer.) + subs.push( + BluetoothSdk.addListener("local_transcription", (event) => { + if (!localSttFallbackCoordinator.isActive()) return + const lang = event.transcribeLanguage ?? localSttFallbackCoordinator.getActiveLanguage() ?? "en-US" + localMiniappRuntime.forwardEvent(`transcription:${lang}`, event) + }), + ) + + // --- glasses swipe-menu app launcher (G2) --- + subs.push( + BluetoothSdk.addListener("miniapp_selected", (event) => { + const packageName = event.packageName as string + if (!packageName) return + const app = useAppStatusStore.getState().apps.find((a) => a.packageName === packageName) + if (!app) return + // Toggle: stop if running, else start. + if (app.running) { + useAppStatusStore.getState().stop(packageName) + } else { + useAppStatusStore.getState().start(app, {skipNavigation: true}) + } + }), + ) +} + +export function stopDeviceEventRouter(): void { + for (const sub of subs) sub.remove() + subs = [] +} diff --git a/mobile/modules/island/src/services/DisplayProcessor.ts b/mobile/modules/island/src/services/DisplayProcessor.ts index c1460068e7..383d15e448 100644 --- a/mobile/modules/island/src/services/DisplayProcessor.ts +++ b/mobile/modules/island/src/services/DisplayProcessor.ts @@ -27,7 +27,10 @@ import { type BreakMode, } from "../utils/display" -import {getRuntimeHooks, ISLAND_SETTINGS_KEYS} from "../runtime/config" +import {ISLAND_SETTINGS_KEYS} from "../runtime/config" +import {useGlassesStore} from "../stores/glasses" +import {useSettingsStore} from "../stores/settings" +import {isGlassesConnected} from "./GlassesReadiness" // ============================================================================= // Types @@ -155,12 +158,13 @@ function getPlaceholderValues(): PlaceholderValues { const day = now.getDate() const DATE = `${month}/${day}` - // Get battery level from glasses store - const batteryLevel = getRuntimeHooks().glassesStatus?.get().batteryLevel ?? -1 + // Battery + connection read from the island glasses store directly (was a host + // glassesStatus hook). + const batteryLevel = useGlassesStore.getState().batteryLevel ?? -1 const GBATT = batteryLevel === -1 ? "" : `${batteryLevel}%` // Connection status - const connected = getRuntimeHooks().glassesStatus?.get().connected ?? false + const connected = isGlassesConnected(useGlassesStore.getState().connection) const CONNECTION_STATUS = connected ? "Connected" : "" return {TIME12, TIME24, DATE, GBATT, CONNECTION_STATUS} @@ -292,39 +296,38 @@ export class DisplayProcessor { this.composer = new ColumnComposer(toolkit.profile, this.options.breakMode) this.profile = toolkit.profile - // Runtime hooks may not be populated yet at module-load time; attachToRuntime() - // is called by the host after configureRuntime() runs. + // Island stores may not be hydrated yet at module-load time; attachToRuntime() + // is called again by the host after app services start. this.attachToRuntime() } /** - * Read the current default wearable and subscribe to host glasses-status changes. - * Idempotent — safe to call again after configureRuntime() once hooks are wired. + * Read the current default wearable and subscribe to island glasses-status changes. + * Idempotent — safe to call again after app services start. */ public attachToRuntime(): void { if (this.runtimeAttached) { return } - const hooks = getRuntimeHooks() - if (!hooks.settings && !hooks.subscribeGlassesStatus) { - // Hooks not configured yet; bail and let the host call us back after configureRuntime(). - return - } - - const defaultWearable = hooks.settings?.getSetting(ISLAND_SETTINGS_KEYS.defaultWearable) as string | undefined + // Seed the device profile from the saved default wearable, read off the island + // settings store directly (was a host settings hook). + const defaultWearable = useSettingsStore.getState().getSetting(ISLAND_SETTINGS_KEYS.defaultWearable) as + | string + | undefined if (defaultWearable) { this.setDeviceModel(defaultWearable) console.log(`DISPLAY_PROCESSOR: Initialized DisplayProcessor with default wearable: ${defaultWearable}`) } - if (hooks.subscribeGlassesStatus) { - this.unsubscribeGlassesStatus = hooks.subscribeGlassesStatus((changed) => { - if (changed.deviceModel) { - this.setDeviceModel(String(changed.deviceModel)) - } - }) - } + // Track device-model changes off the island glasses store directly (was a host + // subscribeGlassesStatus hook). + this.unsubscribeGlassesStatus = useGlassesStore.subscribe( + (s) => s.deviceModel, + (deviceModel) => { + if (deviceModel) this.setDeviceModel(String(deviceModel)) + }, + ) this.runtimeAttached = true } diff --git a/mobile/modules/island/src/services/GlassesSettingsSync.ts b/mobile/modules/island/src/services/GlassesSettingsSync.ts new file mode 100644 index 0000000000..7c9d34be30 --- /dev/null +++ b/mobile/modules/island/src/services/GlassesSettingsSync.ts @@ -0,0 +1,72 @@ +/** + * Glasses settings sync — island-owned. Keeps the connected glasses in sync with + * the phone's device settings (`BLUETOOTH_SETTING_KEYS`) over the bluetooth-sdk, so + * `toolkit.glasses.settings.set()` (and any other settings-store write) reaches the + * device for ANY host — not just the first-party Mentra app, where this used to live + * in MantleManager. + * + * Two triggers: + * 1. on change — push the keys that changed. + * 2. on (re)connect — push the FULL set, so a freshly-connected device gets the + * phone's current settings (not just future changes). + * + * Started by `toolkit.start()`. Idempotent. + */ +import {shallow} from "zustand/shallow" + +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import {useSettingsStore} from "../stores/settings" +import {useGlassesStore} from "../stores/glasses" +import {isGlassesConnected} from "./GlassesReadiness" + +let unsubChange: (() => void) | null = null +let unsubConnect: (() => void) | null = null + +/** + * Push the FULL current device-settings set to native over the bluetooth-sdk. + * Used both by the on-connect transition below and as a pre-connect seed by + * `toolkit.glasses.connectDefault()`, so native has the phone's settings primed + * before the connect handshake replays them to the glasses. + */ +export async function pushAllBluetoothSettings(): Promise { + // Returns the native write promise so callers can await the seed before the + // connect handshake replays settings to the glasses (otherwise the handshake + // can race ahead and replay stale native settings). + await BluetoothSdk.updateBluetoothSettings(useSettingsStore.getState().getBluetoothSettings()) +} + +export function startGlassesSettingsSync(): void { + if (unsubChange || unsubConnect) return + + // 1. Push only the keys that changed (matching the prior MantleManager sync). + unsubChange = useSettingsStore.subscribe( + (state) => state.getBluetoothSettings(), + (settings: Record, previous: Record) => { + const changed: Record = {} + for (const key in settings) { + if (settings[key] !== previous[key]) changed[key] = settings[key] + } + if (Object.keys(changed).length > 0) { + BluetoothSdk.updateBluetoothSettings(changed) + } + }, + {equalityFn: shallow}, + ) + + // 2. Push the full set whenever the glasses transition to connected. + let wasConnected = isGlassesConnected(useGlassesStore.getState().connection) + unsubConnect = useGlassesStore.subscribe(() => { + const connected = isGlassesConnected(useGlassesStore.getState().connection) + if (connected && !wasConnected) { + pushAllBluetoothSettings() + } + wasConnected = connected + }) +} + +export function stopGlassesSettingsSync(): void { + unsubChange?.() + unsubConnect?.() + unsubChange = null + unsubConnect = null +} diff --git a/mobile/modules/island/src/services/GlassesStatusProjection.ts b/mobile/modules/island/src/services/GlassesStatusProjection.ts new file mode 100644 index 0000000000..94f2165097 --- /dev/null +++ b/mobile/modules/island/src/services/GlassesStatusProjection.ts @@ -0,0 +1,47 @@ +/** + * Glasses status projection — island-owned. Subscribes to the native bluetooth-sdk + * status events and projects them into the island device stores, so island's own + * services (which react to those stores) work for ANY host — not just the first-party + * Mentra app, where this used to live in MantleManager. + * + * This is the INBOUND mirror of GlassesSettingsSync: that pushes the settings store + * OUT to the device; this pulls device status IN to the stores. It's the feed that + * the rest of the island runtime depends on (MicStateCoordinator, the settings-sync + * on-connect trigger, the facade read-models all read these stores). + * + * Started by `toolkit.start()`. Idempotent. + */ +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import {useCoreStore} from "../stores/core" +import {useGlassesStore} from "../stores/glasses" +import localMiniappRuntime from "./LocalMiniappRuntime" + +let unsubs: Array<() => void> = [] + +export function startGlassesStatusProjection(): void { + if (unsubs.length) return + + // Bluetooth-adapter status -> core store. + unsubs.push( + BluetoothSdk.onBluetoothStatus((changed) => { + useCoreStore.getState().setCoreInfo(changed) + }), + ) + + // Glasses status -> glasses store (+ forward to local miniapps; clear any stale + // OTA-available flag on disconnect). + unsubs.push( + BluetoothSdk.onGlassesStatus((changed) => { + useGlassesStore.getState().setGlassesInfo(changed) + localMiniappRuntime.forwardEvent("glasses_connection_state", changed) + if (changed.connection?.state === "disconnected") { + useGlassesStore.getState().setOtaUpdateAvailable(null) + } + }), + ) +} + +export function stopGlassesStatusProjection(): void { + unsubs.forEach((unsub) => unsub()) + unsubs = [] +} diff --git a/mobile/src/services/HeadingService.ts b/mobile/modules/island/src/services/HeadingService.ts similarity index 100% rename from mobile/src/services/HeadingService.ts rename to mobile/modules/island/src/services/HeadingService.ts diff --git a/mobile/modules/island/src/services/LocalDisplayManager.ts b/mobile/modules/island/src/services/LocalDisplayManager.ts index bc9a58e293..b5c7fae955 100644 --- a/mobile/modules/island/src/services/LocalDisplayManager.ts +++ b/mobile/modules/island/src/services/LocalDisplayManager.ts @@ -19,8 +19,9 @@ * is off. */ +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" import displayProcessor from "./DisplayProcessor" -import {getRuntimeHooks} from "../runtime/config" +import {useDisplayStore} from "../stores/display" import {BgTimer} from "../utils/timers" // ============================================================================= @@ -381,13 +382,14 @@ class LocalDisplayManager { } try { - const sendDisplayEvent = getRuntimeHooks().sendDisplayEvent - if (sendDisplayEvent) { - void Promise.resolve(sendDisplayEvent(processedEvent)).catch((err) => { - console.error(`${LOG_TAG}: native display failed:`, err) - }) - } - getRuntimeHooks().setDisplayEvent?.(JSON.stringify(processedEvent)) + // The display output path is a direct btsdk call now (was a host sendDisplayEvent + // hook) so a bare OEM renders to the glasses without wiring it. + void Promise.resolve(BluetoothSdk.displayEvent(processedEvent)).catch((err) => { + console.error(`${LOG_TAG}: native display failed:`, err) + }) + // The mirror store also lives in island, so display requests update the + // phone-side preview without any host hook. + useDisplayStore.getState().setDisplayEvent(JSON.stringify(processedEvent)) } catch (err) { console.error(`${LOG_TAG}: native display failed:`, err) } diff --git a/mobile/modules/island/src/services/LocalMiniappRuntime.ts b/mobile/modules/island/src/services/LocalMiniappRuntime.ts index f02f090da7..d7b52565c8 100644 --- a/mobile/modules/island/src/services/LocalMiniappRuntime.ts +++ b/mobile/modules/island/src/services/LocalMiniappRuntime.ts @@ -33,22 +33,33 @@ import {DeviceTypes, getModelCapabilities} from "../types" import {storage as mmkvStorage} from "../utils/storage/storage" import {BgTimer} from "../utils/timers" import devServerBridge from "./DevServerBridge" +import {islandNotifications} from "./NotificationsEmitter" +import {isGlassesConnected} from "./GlassesReadiness" +import audioPlaybackService from "./AudioPlaybackService" +import {phoneLocationService} from "./PhoneLocationService" +import {useGlassesStore} from "../stores/glasses" +import {useSettingsStore} from "../stores/settings" import localDisplayManager from "./LocalDisplayManager" import type {DisplayPayload} from "./LocalDisplayManager" +import headingService from "./HeadingService" import localSttFallbackCoordinator from "./LocalSttFallbackCoordinator" import micStateCoordinator from "./MicStateCoordinator" +import {phonePhotoCoordinator} from "./PhonePhotoCoordinator" +import {phoneStreamCoordinator} from "./PhoneStreamCoordinator" +import {phoneVideoCoordinator} from "./PhoneVideoCoordinator" +import {cloudClientService} from "./CloudClientService" +import {miniappLauncher} from "./MiniappLauncher" import { - getRuntimeHooks, ISLAND_SETTINGS_KEYS, type CameraFovPreset, type CameraFovRequest, type CameraRoiPosition, type CloudClientStatusSnapshot, - type CloudRuntimeAdapter, type InteropAuditEvent, type MiniappAuthToken, type TtsSynthesisResult, } from "../runtime/config" +import {getAnalytics} from "../runtime/bootstrap" import {normalizeStreamAudioConfig, normalizeStreamVideoConfig} from "../runtime/streamConfig" import type { AudioSubscription, @@ -59,6 +70,8 @@ import type { import ttsModelManager from "./TTSModelManager" import {NavigationHandlers} from "./NavigationHandlers" import type {ClientApp} from "../types/applet" +import {useAppStatusStore} from "../stores/apps" +import {DEV_APP_PACKAGE_NAME, getDevAppSourcePackage} from "./AppRegistry" // ============================================================================= // Types @@ -125,6 +138,18 @@ function withTimeout(promise: Promise, timeoutMs: number): Promise { - console.warn(`${LOG_TAG}: failed to stop stream for ${packageName} on unregister`, error) + void phoneStreamCoordinator.stop(packageName).catch((error) => { + console.warn(`${LOG_TAG}: failed to stop stream for ${packageName} on unregister`, error) }) // Stop any phone-owned video recordings for this app. A miniapp that // closes/crashes mid-recording loses its recordingId, so without this the // glasses keep recording until the max-recording timeout or thermal shutdown. - void getRuntimeHooks() - .videoRecording?.stopForApp?.(packageName) - .catch((error) => { - console.warn(`${LOG_TAG}: failed to stop video recording for ${packageName} on unregister`, error) - }) + void phoneVideoCoordinator.stopForApp(packageName).catch((error) => { + console.warn(`${LOG_TAG}: failed to stop video recording for ${packageName} on unregister`, error) + }) // Detach the per-app nav event forwarder but leave the native nav session // running. The user may have just closed the mini-app UI and will reopen @@ -1000,7 +1001,7 @@ class LocalMiniappRuntime { existing.lastPongAt = Date.now() // Read current glasses capabilities from the settings store - const defaultWearable = getRuntimeHooks().settings?.getSetting(ISLAND_SETTINGS_KEYS.defaultWearable) + const defaultWearable = (useSettingsStore.getState().getSetting(ISLAND_SETTINGS_KEYS.defaultWearable) as DeviceTypes | undefined) const capabilities = getModelCapabilities(defaultWearable || DeviceTypes.NONE) // Build the declared-permission record for the SDK's session.permissions @@ -1048,9 +1049,9 @@ class LocalMiniappRuntime { } private async requestMiniappAuth(packageName: string, opts?: {minTtlMs?: number}): Promise { - const auth = getRuntimeHooks().miniappAuth - if (!auth) return null - return auth.getToken(packageName, opts) + const authPackageName = packageName === DEV_APP_PACKAGE_NAME ? getDevAppSourcePackage() : packageName + if (!authPackageName) return null + return cloudClientService.getMiniappAuthToken(authPackageName, opts) } private async handleAuthRefresh(packageName: string, payload: Record, requestId?: string): Promise { @@ -1223,7 +1224,14 @@ class LocalMiniappRuntime { * are pure event streams with no "current value" to snapshot. */ private emitInitialSnapshots(packageName: string, streams: string[]): void { - const glassesState = getRuntimeHooks().glassesStatus?.get() ?? {connected: false} + // Read the island glasses store directly (was a host glassesStatus hook). + const gs = useGlassesStore.getState() + const glassesState = { + connected: isGlassesConnected(gs.connection), + deviceModel: gs.deviceModel, + batteryLevel: gs.batteryLevel, + charging: gs.charging, + } const isSimulated = (glassesState.deviceModel || "").toLowerCase().includes("simulated") for (const stream of streams) { @@ -1405,7 +1413,7 @@ class LocalMiniappRuntime { const stopOtherAudio = payload.stopOtherAudio !== false this.setSpeakerState(packageName, "loading") - getRuntimeHooks().audioPlayback?.play( + audioPlaybackService.play( {requestId: audioRequestId, audioUrl, appId: packageName, volume, stopOtherAudio}, (_respId, success, error, duration) => { if (success) { @@ -1439,7 +1447,7 @@ class LocalMiniappRuntime { } private handleStopAudio(packageName: string, _payload: Record, requestId?: string): void { - getRuntimeHooks().audioPlayback?.stopForApp(packageName) + audioPlaybackService.stopForApp(packageName) this.setSpeakerState(packageName, "stopped") this.sendResult(packageName, requestId, true) } @@ -1467,21 +1475,6 @@ class LocalMiniappRuntime { this.setSpeakerState(packageName, "loading") - const hooks = getRuntimeHooks() - const audioPlayback = hooks.audioPlayback - if (!audioPlayback) { - const message = "audio playback unavailable" - this.setSpeakerState(packageName, "error", { - errorCode: MiniappErrorCode.INTERNAL, - errorMessage: message, - }) - this.sendResult(packageName, requestId, false, undefined, { - code: MiniappErrorCode.INTERNAL, - message, - }) - return - } - const voiceExplicit = payload.voice_id !== undefined || payload.voice !== undefined const offlineSupportsVoice = !voiceExplicit || @@ -1489,10 +1482,9 @@ class LocalMiniappRuntime { ttsModelManager.getAvailableLanguages().some((l) => l.code === voice) const modelId = typeof payload.model_id === "string" ? payload.model_id : undefined - const cloud = hooks.cloud - const cloudConnected = cloud?.isConnected() === true + const cloudConnected = cloudClientService.isConnected() console.log( - `${LOG_TAG}: TTS decision for ${packageName}: cloudConnected=${cloudConnected}, runtimeTts=${cloud?.tts ? "yes" : "no"}, offlineVoice=${offlineSupportsVoice}`, + `${LOG_TAG}: TTS decision for ${packageName}: cloudConnected=${cloudConnected}, runtimeTts=yes, offlineVoice=${offlineSupportsVoice}`, ) let terminalSent = false @@ -1543,7 +1535,7 @@ class LocalMiniappRuntime { } const generated = offlineGenerated - audioPlayback.play( + audioPlaybackService.play( {requestId: audioRequestId, audioUrl: generated.audioUrl, appId: packageName, volume, stopOtherAudio}, (_respId, success, error, duration) => { void Promise.resolve(generated.cleanup?.()).catch((cleanupError) => { @@ -1557,11 +1549,9 @@ class LocalMiniappRuntime { } const playCloudTts = async (fallbackToOffline: boolean): Promise => { - if (!cloud?.tts) return false - - let source: Awaited> + let source: Awaited> try { - source = await cloud.tts.speak(text, { + source = await cloudClientService.tts.speak(text, { ...(voiceExplicit && voice !== "default" ? {voice_id: voice} : {}), ...(modelId ? {model_id: modelId} : {}), ...(voiceSettings ? {voice_settings: voiceSettings} : {}), @@ -1577,7 +1567,7 @@ class LocalMiniappRuntime { } await Promise.resolve( - audioPlayback.play( + audioPlaybackService.play( {requestId: audioRequestId, audioUrl: source.audioUrl, appId: packageName, volume, stopOtherAudio}, (_respId, success, error, duration) => { if (!success && fallbackToOffline) { @@ -1719,14 +1709,10 @@ class LocalMiniappRuntime { private recomputeHeadingSubscription(): void { const wantsHeading = this.streamSubscribers.has(MiniappStreamType.HEADING_UPDATE) if (wantsHeading && !this.headingUnsub) { - // Heading is host-supplied via runtime hooks; if the host hasn't wired - // it the subscription is a no-op and no events fire. - const heading = getRuntimeHooks().heading - if (heading) { - this.headingUnsub = heading.addListener((degrees: number) => { - this.forwardEvent(MiniappStreamType.HEADING_UPDATE, {degrees}) - }) - } + // Heading sensor lives in island (HeadingService) — subscribe directly. + this.headingUnsub = headingService.addListener((degrees: number) => { + this.forwardEvent(MiniappStreamType.HEADING_UPDATE, {degrees}) + }) } else if (!wantsHeading && this.headingUnsub) { this.headingUnsub() this.headingUnsub = null @@ -1783,7 +1769,7 @@ class LocalMiniappRuntime { if (next === this.lastAppliedLocationRate) return this.lastAppliedLocationRate = next console.log(`[LOCATION] aggregate tier → ${next}`) - getRuntimeHooks().locationTier?.setLocationTier(next) + void phoneLocationService.setLocationTier(next) } // --------------------------------------------------------------------------- @@ -1791,7 +1777,7 @@ class LocalMiniappRuntime { // --------------------------------------------------------------------------- private getStorageKeyPrefix(packageName: string): string { - const userId = getRuntimeHooks().settings?.getSetting(ISLAND_SETTINGS_KEYS.coreToken) || "anonymous" + const userId = (useSettingsStore.getState().getSetting(ISLAND_SETTINGS_KEYS.coreToken) as string | undefined) || "anonymous" return `mentraos_localstorage_${userId}_${packageName}_` } @@ -2008,24 +1994,17 @@ class LocalMiniappRuntime { return } - const cameraSettings = getRuntimeHooks().cameraSettings - if (!cameraSettings) { - this.sendResult(packageName, requestId, false, undefined, { - code: MiniappErrorCode.NOT_IMPLEMENTED, - message: "Camera FOV settings are not configured on this host", - }) - return - } - try { const request = normalizeCameraFovPayload(payload) const description = "preset" in request ? `preset=${request.preset}` : `fov=${request.fov} roi=${request.roiPosition ?? "center"}` console.log(`${LOG_TAG}: camera_fov_set ${description}`) - const result = await cameraSettings.setFov(packageName, request) + // Camera FOV is a direct btsdk call now (was the host-injected `cameraSettings` + // runtime hook — a pure BluetoothSdk.setCameraFov passthrough with no host coupling). + const result = await BluetoothSdk.setCameraFov(request) - getRuntimeHooks().settings?.setSetting( + useSettingsStore.getState().setSetting( ISLAND_SETTINGS_KEYS.cameraFov, {fov: result.fov, roi_position: CAMERA_ROI_POSITION_VALUES[result.roiPosition]}, false, @@ -2206,17 +2185,8 @@ class LocalMiniappRuntime { return } - const photo = getRuntimeHooks().photo - if (!photo) { - this.sendResult(packageName, requestId, false, undefined, { - code: MiniappErrorCode.NOT_IMPLEMENTED, - message: "Photo capture is not configured on this host", - }) - return - } - try { - const result = await photo.takePhoto(packageName, { + const result = await phonePhotoCoordinator.takePhoto(packageName, { size: payload.size as | "low" | "medium" @@ -2261,17 +2231,8 @@ class LocalMiniappRuntime { return } - const video = getRuntimeHooks().videoRecording - if (!video) { - this.sendResult(packageName, requestId, false, undefined, { - code: MiniappErrorCode.NOT_IMPLEMENTED, - message: "Video recording is not configured on this host", - }) - return - } - try { - const result = await video.startRecording(packageName, { + const result = await phoneVideoCoordinator.startRecording(packageName, { width: payload.width as number | undefined, height: payload.height as number | undefined, fps: payload.fps as number | undefined, @@ -2292,17 +2253,8 @@ class LocalMiniappRuntime { payload: Record, requestId?: string, ): Promise { - const video = getRuntimeHooks().videoRecording - if (!video) { - this.sendResult(packageName, requestId, false, undefined, { - code: MiniappErrorCode.NOT_IMPLEMENTED, - message: "Video recording is not configured on this host", - }) - return - } - try { - await video.stopRecording(packageName, payload.recordingId as string | undefined) + await phoneVideoCoordinator.stopRecording(packageName, payload.recordingId as string | undefined) this.sendResult(packageName, requestId, true) } catch (err) { this.sendResult(packageName, requestId, false, undefined, { @@ -2313,8 +2265,8 @@ class LocalMiniappRuntime { } /** - * Stream handlers — dispatched to the host's StreamingAdapter. For managed - * streams the adapter additionally calls the v2 client REST route to + * Stream handlers — dispatched to the island PhoneStreamCoordinator. For managed + * streams the coordinator additionally calls the v2 client REST route to * provision Cloudflare. Cloud-SDK apps (third-party developers) use a * separate cloud-side path that does not pass through here. */ @@ -2323,7 +2275,7 @@ class LocalMiniappRuntime { payload: Record, requestId?: string, ): Promise { - const streaming = getRuntimeHooks().streaming + const streaming = phoneStreamCoordinator if (!streaming) { this.sendResult(packageName, requestId, false, undefined, { code: MiniappErrorCode.NOT_IMPLEMENTED, @@ -2354,7 +2306,7 @@ class LocalMiniappRuntime { payload: Record, requestId?: string, ): Promise { - const streaming = getRuntimeHooks().streaming + const streaming = phoneStreamCoordinator if (!streaming) { // No adapter wired — treat as already-stopped. this.sendResult(packageName, requestId, true) @@ -2378,7 +2330,7 @@ class LocalMiniappRuntime { payload: Record, requestId?: string, ): Promise { - const streaming = getRuntimeHooks().streaming + const streaming = phoneStreamCoordinator if (!streaming) { this.sendResult(packageName, requestId, false, undefined, { code: MiniappErrorCode.NOT_IMPLEMENTED, @@ -2414,12 +2366,12 @@ class LocalMiniappRuntime { } /** - * Wire the StreamingAdapter's status callback so coordinator-emitted + * Wire the PhoneStreamCoordinator's status callback so coordinator-emitted * updates become `EVENT { streamType: "stream_status" }` envelopes on the * subscribing miniapp(s). */ public wireStreamingStatusFanout(): void { - const streaming = getRuntimeHooks().streaming + const streaming = phoneStreamCoordinator if (!streaming) return streaming.setStatusSubscriber((packageName, update) => { this.sendToMiniapp(packageName, { @@ -2510,12 +2462,10 @@ class LocalMiniappRuntime { } /** - * Recompute the aggregated subscription list across all local miniapps and - * send PHONE_SUBSCRIPTION_UPDATE to the cloud so TranscriptionManager / - * TranslationManager deliver data to the __phone__ subscriber. + * Recompute the aggregated v2 cloud subscription list across all local miniapps. * * Cloud-dependent streams (transcription:*, translation:*) only flow if the - * cloud knows the phone wants them. Local-only streams (button_press, etc.) + * v2 runtime knows the phone wants them. Local-only streams (button_press, etc.) * are NOT sent — they come from the Bluetooth SDK, not from cloud. */ private updateCloudSubscriptions(): void { @@ -2533,23 +2483,19 @@ class LocalMiniappRuntime { transcriptionLang = stream.substring("transcription:".length) } } - getRuntimeHooks().socketComms?.updatePhoneSubscriptions(Array.from(cloudStreams)) localSttFallbackCoordinator.onSubscriptionChange(transcriptionLang !== null, transcriptionLang) // Mirror the same set as typed AudioSubscription[] and push it to the cloud // runtime. - const cloud = getRuntimeHooks().cloud - if (cloud) { - this.ensureCloudResultsWired(cloud) - const subs = this.buildCloudAudioSubscriptions(cloudStreams) - console.log( - `${LOG_TAG}: updateCloudSubscriptions streams=[${Array.from(cloudStreams).join(", ")}] cloudSubs=${subs.length}`, - ) - cloud.setSubscriptions(subs).catch((err) => { - // Best-effort: the cloud may not be connected yet. Logging is enough. - console.warn(`${LOG_TAG}: cloud setSubscriptions failed: ${(err as Error)?.message ?? err}`) - }) - } + this.ensureCloudResultsWired() + const subs = this.buildCloudAudioSubscriptions(cloudStreams) + console.log( + `${LOG_TAG}: updateCloudSubscriptions streams=[${Array.from(cloudStreams).join(", ")}] cloudSubs=${subs.length}`, + ) + cloudClientService.setSubscriptions(subs).catch((err) => { + // Best-effort: the cloud may not be connected yet. Logging is enough. + console.warn(`${LOG_TAG}: cloud setSubscriptions failed: ${(err as Error)?.message ?? err}`) + }) } /** @@ -2592,11 +2538,11 @@ class LocalMiniappRuntime { * `transcription:` / `translation::` stream strings, so * subscribed miniapps receive identical envelopes. */ - private ensureCloudResultsWired(cloud: CloudRuntimeAdapter): void { + private ensureCloudResultsWired(): void { if (this.cloudResultsWired) return this.cloudResultsWired = true - cloud.onTranscript((d: TranscriptionData) => { + cloudClientService.onTranscript((d: TranscriptionData) => { this.forwardEvent(`transcription:${d.resolvedLanguage}`, { type: "transcription", text: d.text, @@ -2613,7 +2559,7 @@ class LocalMiniappRuntime { }) }) - cloud.onTranslation((d: TranslationData) => { + cloudClientService.onTranslation((d: TranslationData) => { this.forwardEvent(`translation:${d.source.language}:${d.target.language}`, { type: "translation", text: d.text, @@ -2637,25 +2583,22 @@ class LocalMiniappRuntime { if (this.cloudStatusWired) return this.cloudStatusWired = true - getRuntimeHooks().cloud?.onStatusChanged((status) => { + cloudClientService.onStatusChanged((status) => { if (status.status === "connected") { this.updateCloudSubscriptions() } this.broadcastCloudStatus() }) - getRuntimeHooks().settings?.subscribeKey?.(ISLAND_SETTINGS_KEYS.localSttFallbackActive, () => { + useSettingsStore.subscribe((s) => s.getSetting(ISLAND_SETTINGS_KEYS.localSttFallbackActive), () => { this.broadcastCloudStatus() }) } private currentCloudStatus(): CloudClientStatusSnapshot { - const base = getRuntimeHooks().cloud?.getStatus() ?? { - status: "disconnected", - audioTransport: "none", - } + const base = cloudClientService.getStatus() const fallbackActive = - getRuntimeHooks().settings?.getSetting(ISLAND_SETTINGS_KEYS.localSttFallbackActive) === true + (useSettingsStore.getState().getSetting(ISLAND_SETTINGS_KEYS.localSttFallbackActive) as boolean | undefined) === true return { status: base.status, audioTransport: fallbackActive ? "offline" : base.audioTransport, @@ -2897,10 +2840,44 @@ class LocalMiniappRuntime { >() private actionCallSeq = 0 + private interopApps(): ClientApp[] { + return useAppStatusStore.getState().apps + } + + private isSystemPackage(packageName: string): boolean { + if (SYSTEM_MINIAPP_PACKAGES.has(packageName)) return true + return this.interopApps().find((app) => app.packageName === packageName)?.isMiniappDev === true + } + + private async startInteropApp(packageName: string): Promise { + const app = this.interopApps().find((a) => a.packageName === packageName) + if (!app) return false + if (app.local) { + await miniappLauncher.ensureConnected(packageName) + return true + } + return useAppStatusStore.getState().start(app, {skipNavigation: true}) + } + + private stopInteropApp(packageName: string): Promise { + return useAppStatusStore.getState().stop(packageName) + } + + private wakeInteropApp(packageName: string): Promise { + return miniappLauncher.ensureConnected(packageName) + } + /** Emit an interop audit event (best-effort — never let telemetry break a call). */ private auditInterop(event: InteropAuditEvent): void { try { - getRuntimeHooks().interop?.audit?.(event) + getAnalytics()?.("miniapp_interop", { + caller: event.caller, + op: event.op, + target: event.target ?? "", + actionId: event.actionId ?? "", + ok: event.ok, + errorCode: event.errorCode ?? "", + }) } catch { /* telemetry must never break an interop call */ } @@ -2913,7 +2890,7 @@ class LocalMiniappRuntime { op: InteropAuditEvent["op"], target?: string, ): boolean { - if (getRuntimeHooks().interop?.isSystemApp(packageName)) return true + if (this.isSystemPackage(packageName)) return true this.auditInterop({caller: packageName, op, target, ok: false, errorCode: MiniappErrorCode.NOT_PERMITTED}) this.sendResult(packageName, requestId, false, undefined, { code: MiniappErrorCode.NOT_PERMITTED, @@ -2942,17 +2919,8 @@ class LocalMiniappRuntime { private handleMiniappsList(packageName: string, payload: Record, requestId?: string): void { if (!this.requireSystemCaller(packageName, requestId, "list")) return - const interop = getRuntimeHooks().interop - if (!interop) { - this.sendResult(packageName, requestId, false, undefined, { - code: MiniappErrorCode.INTERNAL, - message: "interop adapter not configured", - }) - return - } const includeIncompatible = payload.includeIncompatible === true - const infos = interop - .listApps() + const infos = this.interopApps() .filter( (a) => a.packageName && @@ -2972,9 +2940,8 @@ class LocalMiniappRuntime { ): Promise { const target = payload.packageName as string | undefined if (!this.requireSystemCaller(packageName, requestId, "start", target)) return - const interop = getRuntimeHooks().interop - const app = target ? interop?.listApps().find((a) => a.packageName === target) : undefined - if (!target || !app || !interop) { + const app = target ? this.interopApps().find((a) => a.packageName === target) : undefined + if (!target || !app) { this.sendResult(packageName, requestId, false, undefined, { code: MiniappErrorCode.APP_NOT_FOUND, message: `Miniapp not found: ${target ?? "(missing packageName)"}`, @@ -2995,6 +2962,13 @@ class LocalMiniappRuntime { code: MiniappErrorCode.APP_NOT_COMPATIBLE, message: `${target} is not compatible with the connected glasses`, }) + islandNotifications.emit({ + kind: "version_incompatible", + packageName: target, + reason: `${target} is not compatible with the connected glasses`, + metadata: {missingRequired: app.compatibility.missingRequired ?? [], via: "miniapps.start"}, + timestamp: Date.now(), + }) this.auditInterop({ caller: packageName, op: "start", @@ -3005,9 +2979,9 @@ class LocalMiniappRuntime { return } try { - // startApp resolves to false when the host gate (beforeStart) rejected the + // startInteropApp resolves to false when the app-store gate rejected the // launch or the background context failed to spawn — don't report success. - const started = await interop.startApp(target) + const started = await this.startInteropApp(target) if (started) { this.sendResult(packageName, requestId, true) this.auditInterop({caller: packageName, op: "start", target, ok: true}) @@ -3046,8 +3020,7 @@ class LocalMiniappRuntime { ): Promise { const target = payload.packageName as string | undefined if (!this.requireSystemCaller(packageName, requestId, "stop", target)) return - const interop = getRuntimeHooks().interop - if (!target || !interop) { + if (!target) { this.sendResult(packageName, requestId, false, undefined, { code: MiniappErrorCode.APP_NOT_FOUND, message: "stop requires a packageName", @@ -3055,7 +3028,7 @@ class LocalMiniappRuntime { return } try { - await interop.stopApp(target) + await this.stopInteropApp(target) this.sendResult(packageName, requestId, true) this.auditInterop({caller: packageName, op: "stop", target, ok: true}) } catch (e) { @@ -3099,7 +3072,6 @@ class LocalMiniappRuntime { // with no caller left to receive its result. 6s = that 5s buffer + 1s for the // ACTION_RESULT/NO_ACTION_HANDLER reply to travel back. Keep these in sync. const timeoutMs = Math.min(Math.max(Number(payload.timeoutMs) || 30_000, 6_000), 120_000) - const interop = getRuntimeHooks().interop if (this.actionPayloadTooLarge(params)) { this.sendResult(callerPackageName, requestId, false, undefined, { @@ -3109,8 +3081,8 @@ class LocalMiniappRuntime { return } - const app = target ? interop?.listApps().find((a) => a.packageName === target) : undefined - if (!target || !app || !interop) { + const app = target ? this.interopApps().find((a) => a.packageName === target) : undefined + if (!target || !app) { this.sendResult(callerPackageName, requestId, false, undefined, { code: MiniappErrorCode.APP_NOT_FOUND, message: `Miniapp not found: ${target ?? "(missing targetPackageName)"}`, @@ -3129,6 +3101,13 @@ class LocalMiniappRuntime { code: MiniappErrorCode.APP_NOT_COMPATIBLE, message: `${target} is not compatible with the connected glasses`, }) + islandNotifications.emit({ + kind: "version_incompatible", + packageName: target, + reason: `${target} is not compatible with the connected glasses`, + metadata: {missingRequired: app.compatibility.missingRequired ?? [], via: "miniapps.invoke"}, + timestamp: Date.now(), + }) return } // Declared-action gate — only once the host populates app.actions (Phase 2); @@ -3143,7 +3122,7 @@ class LocalMiniappRuntime { // Headless wake + wait for CONNECT (idempotent / fast if already connected). try { - await interop.wakeMiniapp(target) + await this.wakeInteropApp(target) } catch (e) { this.sendResult(callerPackageName, requestId, false, undefined, { code: MiniappErrorCode.WAKE_FAILED, diff --git a/mobile/modules/island/src/services/LocalSttFallbackCoordinator.ts b/mobile/modules/island/src/services/LocalSttFallbackCoordinator.ts index f36760b62d..0297692f76 100644 --- a/mobile/modules/island/src/services/LocalSttFallbackCoordinator.ts +++ b/mobile/modules/island/src/services/LocalSttFallbackCoordinator.ts @@ -1,4 +1,7 @@ -import {getRuntimeHooks, ISLAND_SETTINGS_KEYS} from "../runtime/config" +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import {ISLAND_SETTINGS_KEYS} from "../runtime/config" +import {useSettingsStore} from "../stores/settings" +import {cloudClientService} from "./CloudClientService" import sttModelManager from "./STTModelManager" /** @@ -22,8 +25,7 @@ class LocalSttFallbackCoordinator { private activeLanguage: string | null = null /** * Default to "cloud is up" so we never accidentally activate local STT before - * the host has had a chance to wire the cloud-client adapter via - * `configureRuntime`. The adapter is attached lazily on the first + * the cloud client has had a chance to start. We attach lazily on the first * subscription/reconcile pass; once attached, this field reflects the real * cloud-client runtime status. */ @@ -32,26 +34,22 @@ class LocalSttFallbackCoordinator { private cloudAdapterAttached = false private constructor() { - const settings = getRuntimeHooks().settings // Reset the persisted mirror flag on boot — the in-memory state in this // coordinator is the source of truth, and a stale "true" left from the // previous session would cause native to feed Sherpa before any miniapp // registered a subscription. - settings?.setSetting(ISLAND_SETTINGS_KEYS.localSttFallbackActive, false) + useSettingsStore.getState().setSetting(ISLAND_SETTINGS_KEYS.localSttFallbackActive, false) } /** - * Lazy-attach the cloud connection adapter. The coordinator is constructed - * at module load (singleton import), before `configureRuntime` runs on the - * host. We defer reading the adapter until the first place we actually - * need cloud state — which is reconcile(). + * Lazy-attach the island cloud connection listener. The coordinator is + * constructed at module load, before toolkit.start() brings up the cloud + * client, so we defer until the first reconcile. */ private attachCloudAdapterIfReady(): void { if (this.cloudAdapterAttached) return - const cloud = getRuntimeHooks().cloud - if (!cloud) return - this.cloudConnected = cloud.isConnected() - cloud.onStatusChanged((status) => { + this.cloudConnected = cloudClientService.isConnected() + cloudClientService.onStatusChanged((status) => { const connected = status.status === "connected" if (this.cloudConnected === connected) return this.log(`cloud connection -> ${connected ? "up" : "down"}`) @@ -114,7 +112,8 @@ class LocalSttFallbackCoordinator { return } try { - await getRuntimeHooks().restartTranscriber?.() + // Direct btsdk call now (was a host restartTranscriber hook). + await BluetoothSdk.restartTranscriber() } catch (err) { this.log(`restartTranscriber failed: ${err}`) } @@ -122,13 +121,13 @@ class LocalSttFallbackCoordinator { this.log("local stt activation skipped: cloud recovered during transcriber restart") return } - getRuntimeHooks().settings?.setSetting(ISLAND_SETTINGS_KEYS.localSttFallbackActive, true) + useSettingsStore.getState().setSetting(ISLAND_SETTINGS_KEYS.localSttFallbackActive, true) this.localActive = true } private stopLocalStt(reason: string): void { this.log(`stopping local stt: ${reason}`) - getRuntimeHooks().settings?.setSetting(ISLAND_SETTINGS_KEYS.localSttFallbackActive, false) + useSettingsStore.getState().setSetting(ISLAND_SETTINGS_KEYS.localSttFallbackActive, false) this.localActive = false } diff --git a/mobile/modules/island/src/services/MentraJSRouter.ts b/mobile/modules/island/src/services/MentraJSRouter.ts index 9eea6c6315..dff996293a 100644 --- a/mobile/modules/island/src/services/MentraJSRouter.ts +++ b/mobile/modules/island/src/services/MentraJSRouter.ts @@ -41,6 +41,7 @@ import type localMiniappRuntime from "./LocalMiniappRuntime" import type {InstalledMiniappManifest} from "./LocalMiniappRuntime" import type {MentraJSCrashController} from "./MentraJSCrashController" import {MentraJSLogRingBuffer, MentraJSLogThrottle, redactSecrets} from "./MentraJSLogPipeline" +import {islandNotifications} from "./NotificationsEmitter" import type {MentraUIRouter} from "./MentraUIRouter" import {miniappRunningRegistry} from "./MiniappRunningRegistry" @@ -288,6 +289,7 @@ export class MentraJSRouter { const outcome = controller.onCrash(packageName, reason) if (outcome.surfaceCrashloopBanner) { this.onCrashloop?.(packageName, reason) + islandNotifications.emit({kind: "miniapp_crashloop", packageName, reason, timestamp: Date.now()}) return } if (outcome.showRestartToast) { diff --git a/mobile/modules/island/src/services/MicStateCoordinator.ts b/mobile/modules/island/src/services/MicStateCoordinator.ts index 433eed87dc..491ea7daff 100644 --- a/mobile/modules/island/src/services/MicStateCoordinator.ts +++ b/mobile/modules/island/src/services/MicStateCoordinator.ts @@ -1,26 +1,20 @@ /** * MicStateCoordinator * - * Unions cloud-driven and local-miniapp-driven microphone requirements. + * Owns local-miniapp-driven microphone requirements. * - * The cloud sends mic state changes via SocketComms (e.g., "pcm", "transcription"). * Local miniapps subscribe to audio_chunk / transcription streams. - * This coordinator merges both sets of requirements and pushes the union to - * BluetoothSdk so the mic runs whenever at least one consumer needs it. + * This coordinator pushes the aggregate local requirement set to BluetoothSdk + * so the mic runs whenever at least one local consumer needs it. */ -import {getRuntimeHooks} from "../runtime/config" +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" const LOG_TAG = "MIC_COORDINATOR" class MicStateCoordinator { private static instance: MicStateCoordinator | null = null - // Cloud requirements (set by SocketComms on mic_state_change) - private cloudWantsPcm = false - private cloudWantsLc3 = false - private cloudWantsTranscript = false - // Local miniapp requirements (set when miniapps subscribe to audio streams) private localWantsPcm = false private localWantsLc3 = false @@ -34,20 +28,6 @@ class MicStateCoordinator { return MicStateCoordinator.instance } - /** - * Update cloud-side requirements. Called by SocketComms when the cloud - * sends a mic_state_change message. - */ - public setCloudRequirements(req: {pcm: boolean; lc3: boolean; transcript: boolean}): void { - this.cloudWantsPcm = req.pcm - this.cloudWantsLc3 = req.lc3 - this.cloudWantsTranscript = req.transcript - // console.log( - // `${LOG_TAG}: cloud requirements updated — pcm=${req.pcm} lc3=${req.lc3} transcript=${req.transcript}`, - // ) - this.applyUnion() - } - /** * Update local miniapp requirements. Called by LocalMiniappRuntime when * the aggregated set of local subscriptions changes. @@ -60,35 +40,25 @@ class MicStateCoordinator { } /** - * Compute the union of cloud and local requirements and push to BluetoothSdk. - * - * Wire-format note: the cloud only ever receives LC3 over the binary - * WebSocket. Its `requiredData=["pcm"]` is a logical "I need audio" - * request — we always answer it with LC3. So neither cloudWantsPcm nor - * cloudWantsLc3 ever flips `should_send_pcm` on; both map to - * `should_send_lc3`. `should_send_pcm` is strictly for on-device PCM - * consumers (local miniapps' audio_chunk listeners and Sherpa STT). + * Push local requirements to BluetoothSdk. `should_send_pcm` is strictly for + * on-device PCM consumers; cloud audio uses LC3 through AudioCloudUplink. */ private applyUnion(): void { const shouldSendPcm = this.localWantsPcm - const shouldSendLc3 = this.cloudWantsPcm || this.cloudWantsLc3 || this.localWantsLc3 - const shouldSendTranscript = this.cloudWantsTranscript + const shouldSendLc3 = this.localWantsLc3 // console.log( - // `${LOG_TAG}: applying union — pcm=${shouldSendPcm} lc3=${shouldSendLc3} transcript=${shouldSendTranscript}`, + // `${LOG_TAG}: applying requirements — pcm=${shouldSendPcm} lc3=${shouldSendLc3}`, // ) - const setMicRequirements = getRuntimeHooks().setMicRequirements - if (!setMicRequirements) { - return - } - + // The mic control plane is a direct btsdk call now (was a host setMicRequirements + // hook) so a bare OEM streams audio without wiring it. try { void Promise.resolve( - setMicRequirements({ - shouldSendPcm, - shouldSendLc3, - shouldSendTranscript, + BluetoothSdk.updateBluetoothSettings({ + should_send_pcm: shouldSendPcm, + should_send_lc3: shouldSendLc3, + should_send_transcript: false, }), ).catch((err) => { console.error(`${LOG_TAG}: failed to apply mic requirements:`, err) @@ -102,9 +72,6 @@ class MicStateCoordinator { * Reset all requirements to off. Called during cleanup. */ public reset(): void { - this.cloudWantsPcm = false - this.cloudWantsLc3 = false - this.cloudWantsTranscript = false this.localWantsPcm = false this.localWantsLc3 = false this.applyUnion() diff --git a/mobile/modules/island/src/services/MiniappEngine.ts b/mobile/modules/island/src/services/MiniappEngine.ts new file mode 100644 index 0000000000..734cd97ec6 --- /dev/null +++ b/mobile/modules/island/src/services/MiniappEngine.ts @@ -0,0 +1,114 @@ +/** + * MiniappEngine — constructs the MentraJS router singletons (crash controller, + * UI router, JS router), binds them to the native Crust module + the launcher, + * wires the dev-server background-respawn signal, and starts the message pump. + * + * Island owns this so a bare OEM's `toolkit.start()` brings up the local-miniapp + * engine with no host construction step. Host-only concerns that remain outside + * island, such as Mentra-app crashloop telemetry, are attached by the host after + * this returns via `router.onCrashloop` / `router.onRestartToast`. + * + * Idempotent — `ensureMiniappEngine()` returns the same singletons on every call, + * so it's safe to call from both `toolkit.start()` and the host bootstrap. + */ + +import CrustModule from "@mentra/crust" + +import devServerBridge from "./DevServerBridge" +import localMiniappRuntime from "./LocalMiniappRuntime" +import {miniappLauncher} from "./MiniappLauncher" +import {MentraJSCrashController} from "./MentraJSCrashController" +import {MentraJSRouter, type MentraJSCrustBinding} from "./MentraJSRouter" +import {MentraUIRouter} from "./MentraUIRouter" + +export interface MiniappEngine { + router: MentraJSRouter + uiRouter: MentraUIRouter + crashController: MentraJSCrashController +} + +let engine: MiniappEngine | null = null + +/** + * Construct (once) and return the MentraJS engine singletons. Starts the router's + * message pump on first call. Subsequent calls return the same instances. + */ +export function ensureMiniappEngine(): MiniappEngine { + if (engine) return engine + + // The Crust native module's published TS typing doesn't include the new + // mentraJs* functions until expo prebuild regenerates it; cast to the binding + // shapes the routers expect. + const crust = CrustModule as unknown as MentraJSCrustBinding + + const crashController = new MentraJSCrashController({ + maxRetries: 3, + }) + const uiRouter = new MentraUIRouter({ + mentraJsDispatchToJs: (packageName: string, envelope: Record) => + ( + CrustModule as unknown as { + mentraJsDispatchToJs: (p: string, e: Record) => Promise + } + ).mentraJsDispatchToJs(packageName, envelope), + }) + const router = new MentraJSRouter(localMiniappRuntime, crust) + router.crashController = crashController + router.uiRouter = uiRouter + + // Hand the router to the island MiniappLauncher so headless launch/teardown + // (apps.ts start/stop, the action broker, the WebView mount path) all spawn + // through one place. + miniappLauncher.configure({router}) + + // Wire up the dev server's "respawn-bg" signal so a touch under + // src/background/ kills + re-spawns the JSContext with the latest bundle. + // The WebView reload path stays separate (devServerBridge.onReload). + // + // Dev miniapps fetch the fresh background JS straight off the dev server over + // HTTP (mirrors LocalMiniappView's HTTP-direct launch). Released miniapps fall + // back to reading the installed file:// snapshot. + devServerBridge.onRespawnBackground(async (packageName) => { + try { + // Re-resolve the freshly built bundle (dev: HTTP off the dev server; + // released: file:// snapshot) via the launcher's shared recipe. Carries + // the declared permissions + manifest across the respawn — omitting them + // would respawn the JSContext with no permissions, so SUBSCRIBE gates and + // per-call dispatch checks would start rejecting after a background + // hot-reload. + const resolved = await miniappLauncher.resolveBundle(packageName) + if (!resolved) { + console.warn(`MentraJS: respawn-bg could not resolve bundle for ${packageName}`) + return + } + // Force a respawn (kill + spawn) rather than launcher.ensureRunning, + // which is idempotent and would no-op an already-registered context. + await router.unregister(packageName) + const ok = await router.spawnAndRegister(packageName, resolved.bgSource, { + permissions: resolved.declaredPermissions, + installedManifest: resolved.installedManifest, + }) + if (!ok) { + console.warn(`MentraJS: respawn-bg failed for ${packageName}`) + return + } + // The respawned JSContext is a fresh MiniappSession with ui.bound false. + // The mounted WebView won't re-fire mentra.ready() (it's latched), so + // re-announce it so RPC replies (mentra.request) reach the UI again + // instead of being dropped while unbound. + uiRouter.notifyReopen(packageName) + } catch (e) { + console.warn(`MentraJS: respawn-bg threw for ${packageName}:`, e) + } + }) + + router.start() + + engine = {router, uiRouter, crashController} + return engine +} + +/** Returns the engine singletons if already constructed, else null. */ +export function getMiniappEngine(): MiniappEngine | null { + return engine +} diff --git a/mobile/modules/island/src/services/MiniappLauncher.ts b/mobile/modules/island/src/services/MiniappLauncher.ts index 2f65e3389e..a809323a7c 100644 --- a/mobile/modules/island/src/services/MiniappLauncher.ts +++ b/mobile/modules/island/src/services/MiniappLauncher.ts @@ -13,11 +13,10 @@ * - the dev hot-reload respawn path and the WebView mount path share one * resolve+spawn recipe instead of three copies. * - * The launcher is **headless**: no React, no RN components. It is configured - * once at host bootstrap with the {@link MentraJSRouter} instance (the router - * needs the native Crust binding, so it is constructed host-side and injected - * here — the same DI seam as `configureRuntime`). OEMs embedding the runtime - * as a native library drive miniapp lifecycle through this surface. + * The launcher is **headless**: no React, no RN components. The island-owned + * MiniappEngine hands it the {@link MentraJSRouter} instance once during + * engine construction. OEMs embedding the runtime drive miniapp lifecycle + * through the toolkit facade and stores, not a separate launcher configure API. * * Boundary: the launcher owns the **background** context. WebView *rendering* * (the UI layer) stays with the host — the launcher just hands back the @@ -33,7 +32,7 @@ import devServerBridge from "./DevServerBridge" import localMiniappRuntime, {type InstalledMiniappManifest} from "./LocalMiniappRuntime" import type {MentraJSRouter} from "./MentraJSRouter" -export interface LauncherDeps { +interface LauncherDeps { /** The host-constructed router (needs the native Crust binding). */ router: MentraJSRouter } @@ -75,14 +74,14 @@ class MiniappLauncher { */ private readonly inFlight = new Map>() - /** Wire the router in. Called once from host bootstrap. */ + /** Wire the router in. Called once from MiniappEngine construction. */ configure(deps: LauncherDeps): void { this.deps = deps } private requireRouter(): MentraJSRouter { if (!this.deps) { - throw new Error("MiniappLauncher not configured — call configureLauncher() at host bootstrap") + throw new Error("MiniappLauncher not configured — ensureMiniappEngine() has not run") } return this.deps.router } @@ -295,8 +294,3 @@ class MiniappLauncher { } export const miniappLauncher = new MiniappLauncher() - -/** Wire the router into the launcher. Called once at host bootstrap. */ -export function configureLauncher(deps: LauncherDeps): void { - miniappLauncher.configure(deps) -} diff --git a/mobile/modules/island/src/services/NavigationHandlers.ts b/mobile/modules/island/src/services/NavigationHandlers.ts index 6e23b04c6c..920c8b733c 100644 --- a/mobile/modules/island/src/services/NavigationHandlers.ts +++ b/mobile/modules/island/src/services/NavigationHandlers.ts @@ -23,7 +23,7 @@ */ import type {NavLocation, NavRoute, NavUpdate} from "../runtime/config" -import {getRuntimeHooks} from "../runtime/config" +import navigationService from "./NavigationService" import {MiniappErrorCode, MiniappResponseType, MiniappStreamType} from "@mentra/miniapp" const LOG_TAG = "LocalMiniappRuntime" @@ -121,7 +121,7 @@ export class NavigationHandlers { const missedRaw = Number(payload.missedTurnRerouteMeters) const missedTurnRerouteMeters = Number.isFinite(missedRaw) && missedRaw > 0 ? missedRaw : undefined - const navigation = getRuntimeHooks().navigation + const navigation = navigationService if (!navigation) { this.sendResult(packageName, requestId, false, undefined, { code: MiniappErrorCode.INTERNAL, @@ -213,7 +213,7 @@ export class NavigationHandlers { this.activeNavApps.delete(packageName) // Only stop the native trip when no other miniapp is still navigating. if (this.activeNavApps.size === 0) { - const navigation = getRuntimeHooks().navigation + const navigation = navigationService const result = navigation ? await navigation.stop() : {ok: true} this.sendResult(packageName, requestId, result.ok, result, undefined) } else { @@ -232,7 +232,7 @@ export class NavigationHandlers { try { const offsetNum = Number(payload.offsetMeters) const offsetMeters = Number.isFinite(offsetNum) && offsetNum > 0 ? offsetNum : 20 - const navigation = getRuntimeHooks().navigation + const navigation = navigationService const result = navigation ? await navigation.simulateDeviation(offsetMeters) : {ok: false, error: "navigation adapter not configured"} @@ -253,7 +253,7 @@ export class NavigationHandlers { ): Promise { try { const enabled = payload.enabled === true - const navigation = getRuntimeHooks().navigation + const navigation = navigationService const result = navigation ? await navigation.setWrongSidewalkOffset(enabled) : {ok: false, error: "navigation adapter not configured"} @@ -274,7 +274,7 @@ export class NavigationHandlers { ): Promise { try { const enabled = payload.enabled === true - const navigation = getRuntimeHooks().navigation + const navigation = navigationService const result = navigation ? await navigation.setSkipCrossings(enabled) : {ok: false, error: "navigation adapter not configured"} @@ -289,13 +289,13 @@ export class NavigationHandlers { } handleGetState(packageName: string, requestId?: string): void { - const snapshot = getRuntimeHooks().navigation?.getSnapshot() ?? null + const snapshot = navigationService.getSnapshot() ?? null this.sendResult(packageName, requestId, true, {state: snapshot}) } async handleRequestPermission(packageName: string, requestId?: string): Promise { try { - const navigation = getRuntimeHooks().navigation + const navigation = navigationService const result = navigation ? await navigation.requestPermission() : {ok: false, accepted: false, error: "navigation adapter not configured"} @@ -315,7 +315,7 @@ export class NavigationHandlers { requestId?: string, ): Promise { try { - const navigation = getRuntimeHooks().navigation + const navigation = navigationService const result = navigation ? await navigation.computeRoute(payload) : {ok: false as const, error: "navigation adapter not configured"} @@ -358,7 +358,7 @@ export class NavigationHandlers { }) return } - const navigation = getRuntimeHooks().navigation + const navigation = navigationService const result = navigation?.reverseGeocodeRoad ? await navigation.reverseGeocodeRoad({lat, lng}) : {ok: false as const, error: "navigation adapter not configured"} diff --git a/mobile/src/services/NavigationService.ts b/mobile/modules/island/src/services/NavigationService.ts similarity index 99% rename from mobile/src/services/NavigationService.ts rename to mobile/modules/island/src/services/NavigationService.ts index da02ca0226..fe2fe6c56f 100644 --- a/mobile/src/services/NavigationService.ts +++ b/mobile/modules/island/src/services/NavigationService.ts @@ -16,7 +16,7 @@ import CrustModule from "@mentra/crust" import {decodePolyline, parseDurationSeconds} from "./navigation/routesApiCodec" import {resolveStepRoads} from "./navigation/roadNameResolver" import restComms from "./RestComms" -import {useSettingsStore} from "@/stores/settings" +import {useSettingsStore} from "../stores/settings" const LOG_TAG = "NAV_SERVICE" diff --git a/mobile/modules/island/src/services/NotificationsEmitter.ts b/mobile/modules/island/src/services/NotificationsEmitter.ts new file mode 100644 index 0000000000..49da4a7edf --- /dev/null +++ b/mobile/modules/island/src/services/NotificationsEmitter.ts @@ -0,0 +1,47 @@ +/** + * Island notifications emitter — the island→host alert channel behind + * `toolkit.notifications`. Island-detected conditions the host should surface + * (miniapp crashloop, version-incompatible miniapp, persistent connection loss) + * are pushed here; the facade fans them out to host subscribers. + * + * A plain listener set (not Node's EventEmitter) to avoid the `events` typings + * gap in island's standalone build. + */ +export type IslandNotificationKind = + | "miniapp_crashloop" + | "version_incompatible" + | "connection_failed_persistent" + +export interface IslandNotification { + kind: IslandNotificationKind + /** The miniapp this concerns, when applicable. */ + packageName?: string + /** Epoch millis when raised. */ + timestamp: number + /** Human-readable summary the host can show. */ + reason: string + /** Kind-specific extras (versions, attempt counts, …). */ + metadata?: Record +} + +const listeners = new Set<(n: IslandNotification) => void>() + +export const islandNotifications = { + /** Raise a notification to all host subscribers. */ + emit(notification: IslandNotification): void { + for (const l of listeners) { + try { + l(notification) + } catch (err) { + console.warn(`islandNotifications: listener threw: ${(err as Error)?.message ?? err}`) + } + } + }, + /** Subscribe to island notifications; returns an unsubscribe. */ + subscribe(cb: (n: IslandNotification) => void): () => void { + listeners.add(cb) + return () => { + listeners.delete(cb) + } + }, +} diff --git a/mobile/modules/island/src/services/OtaService.ts b/mobile/modules/island/src/services/OtaService.ts new file mode 100644 index 0000000000..0b9309c2cf --- /dev/null +++ b/mobile/modules/island/src/services/OtaService.ts @@ -0,0 +1,88 @@ +/** + * OTA service — island-owned. Subscribes to the glasses' OTA BLE events and projects + * them into the island glasses store (otaUpdateAvailable / otaStatus / otaProgress), so + * the OTA read surface works for ANY host — not just the first-party Mentra app, where + * these handlers used to live in MantleManager. + * + * This is the inbound projection half of `toolkit.ota` (read/observe). The install + * ORCHESTRATION (startOtaUpdate/poll/ping) is host-side for now — Phase 2 moves it here + * behind `toolkit.ota.{check,install,retry}` (bootloop-risk; needs on-device verification). + * + * Started by `toolkit.start()`. Idempotent. + */ +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import type {OtaStatus} from "../../../bluetooth-sdk/build/_internal" +import GlobalEventEmitter from "../utils/GlobalEventEmitter" +import {useGlassesStore} from "../stores/glasses" +import {handleOtaClockSkewFromGlasses} from "./glassesClockSync" +import { + legacyOtaProgressFromOtaStatusEvent, + normalizeOtaStatusEvent, + otaStatusFromNormalized, +} from "./otaLegacyMapping" + +let subs: Array<{remove: () => void}> = [] + +export function startOtaService(): void { + if (subs.length) return + + // (The push-based `ota_update_available` BLE event was removed in the OTA-simplify + // pass — availability is now resolved by the host OTA check flow via + // BluetoothSdk.checkForOtaUpdate(), which calls setOtaUpdateAvailable(...) directly. + // OtaService no longer projects an availability event; it owns status/progress below.) + + // MTK firmware update finished (self power-cycle path). + subs.push( + BluetoothSdk.addListener("mtk_update_complete", (event) => { + GlobalEventEmitter.emit("mtk_update_complete", {message: event.message, timestamp: event.timestamp}) + }), + ) + + // Glasses acknowledged the install start. + subs.push( + BluetoothSdk.addListener("ota_start_ack", (event) => { + GlobalEventEmitter.emit("ota_start_ack", {timestamp: event.timestamp}) + }), + ) + + // Install progress / lifecycle. Updates both the new (otaStatus) and the legacy + // (otaProgress) store shapes; auto-fixes clock skew on the relevant failures. + subs.push( + BluetoothSdk.addListener("ota_status", (event) => { + const normalized = normalizeOtaStatusEvent(event as Record) + const status: OtaStatus = otaStatusFromNormalized(normalized) + useGlassesStore.getState().setOtaStatus(status) + // Emit before legacy progress: setOtaProgress can throw (e.g. JSON.stringify in store); + // native logs would still show while RN UI would stay on "Starting update…". + GlobalEventEmitter.emit("ota_status", status) + try { + useGlassesStore.getState().setOtaProgress(legacyOtaProgressFromOtaStatusEvent(normalized)) + } catch (err) { + console.warn("OTA: ota_status legacy otaProgress mapping failed", err) + } + + if (status.status === "failed") { + const raw = event as Record + const glassesTimeMs = Number(raw.glasses_time_ms ?? raw.glassesTimeMs ?? 0) || undefined + const errorCode = normalized.error_message + if ( + errorCode === "clock_skew" || + (errorCode === "ssl_error" && typeof glassesTimeMs === "number" && Number.isFinite(glassesTimeMs)) + ) { + handleOtaClockSkewFromGlasses(errorCode, glassesTimeMs).catch((err: unknown) => { + console.warn("OTA: clock skew auto-fix failed", err) + }) + } + } + + if (status.status === "complete" || status.status === "failed") { + useGlassesStore.getState().setOtaUpdateAvailable(null) + } + }), + ) +} + +export function stopOtaService(): void { + subs.forEach((s) => s.remove()) + subs = [] +} diff --git a/mobile/modules/island/src/services/PhoneLocationService.ts b/mobile/modules/island/src/services/PhoneLocationService.ts new file mode 100644 index 0000000000..e658f513ba --- /dev/null +++ b/mobile/modules/island/src/services/PhoneLocationService.ts @@ -0,0 +1,116 @@ +/** + * Phone location service — island-owned. Owns the background phone-GPS task: the + * accuracy-tier control (`setLocationTier`), the accuracy mapping, and the + * `expo-task-manager` background task that fans each fix to the v2 cloud + * (`restComms.sendLocationData`) + local miniapps (`location_update`). + * + * This used to be the host-injected `locationTier` runtime hook + a MantleManager + * `TaskManager.defineTask`. It's device/OS plumbing (no UI), so it moved into island: + * the runtime drives `setLocationTier` directly off miniapp demand, and any host — + * including a bare OEM — gets phone location without wiring a hook. The host keeps the + * permission *UI* (it gates `setLocationTier` on the OS permission before calling). + * + * The task is registered at module load (the export from `index.ts` evaluates this + * file as soon as `@mentra/island` is imported), matching the old MantleManager + * top-level `defineTask` timing — including on a headless background relaunch. + */ +import * as Location from "expo-location" +import * as TaskManager from "expo-task-manager" + +import restComms from "./RestComms" +import localMiniappRuntime from "./LocalMiniappRuntime" + +export const LOCATION_TASK_NAME = "handleLocationUpdates" + +// Background location task — sends each fix to the v2 cloud (RestComms) + local +// miniapps. (The v1 SocketComms leg was already removed/commented before the move.) +TaskManager.defineTask<{locations?: Location.LocationObject[]}>(LOCATION_TASK_NAME, async ({data, error}) => { + if (error) return + const locs = data?.locations ?? [] + if (locs.length === 0) { + console.log("ISLAND: LOCATION: No locations received") + return + } + const first = locs[0]! + restComms.sendLocationData(first) + // Cloud path (relayMessageToApps) never reaches __phone__, so local miniapps rely + // on this direct push. + localMiniappRuntime.forwardEvent("location_update", { + lat: first.coords.latitude, + lng: first.coords.longitude, + accuracy: first.coords.accuracy ?? undefined, + timestamp: first.timestamp, + }) +}) + +/** Map a MentraOS location tier/accuracy string to an expo-location accuracy. */ +export function getLocationAccuracy(accuracy: string | undefined): Location.LocationAccuracy { + switch (accuracy) { + // Aggregate miniapp tiers (LocalMiniappRuntime.recomputeLocation → "passive" | + // "low" | "high" | "realtime"). Previously only "realtime" mapped; "passive"/"low"/ + // "high" fell through to Lowest, so a miniapp asking for high-rate GPS got the + // coarsest accuracy. + case "realtime": + return Location.LocationAccuracy.BestForNavigation + case "high": + return Location.LocationAccuracy.High + case "low": + return Location.LocationAccuracy.Low + case "passive": + return Location.LocationAccuracy.Lowest + // Accuracy-string vocabulary (persisted location_tier setting / boot path). + case "tenMeters": + return Location.LocationAccuracy.High + case "hundredMeters": + return Location.LocationAccuracy.Balanced + case "kilometer": + return Location.LocationAccuracy.Low + case "threeKilometers": + return Location.LocationAccuracy.Lowest + case "reduced": + return Location.LocationAccuracy.Lowest + default: + return Location.LocationAccuracy.Lowest + } +} + +/** + * Apply a location tier (the aggregate of what miniapps request). "off" means no app + * is asking — stop the task so the OS can power GPS down; anything else (re)starts the + * background task at the matching accuracy. Callers gate this on the OS location + * permission (host UI concern). + */ +export async function setLocationTier( + tier: "off" | "passive" | "low" | "high" | "realtime" | string, +): Promise { + console.log("ISLAND: setLocationTier()", tier) + try { + const isRegistered = await Location.hasStartedLocationUpdatesAsync(LOCATION_TASK_NAME).catch(() => false) + if (isRegistered) { + await Location.stopLocationUpdatesAsync(LOCATION_TASK_NAME) + } + if (tier === "off") { + console.log("ISLAND: setLocationTier() stopped — no active subscribers") + return + } + await Location.startLocationUpdatesAsync(LOCATION_TASK_NAME, { + accuracy: getLocationAccuracy(tier), + pausesUpdatesAutomatically: false, + }) + console.log("ISLAND: setLocationTier() success —", tier) + } catch (error) { + console.log("ISLAND: Error setting location tier", error) + } +} + +/** Stop the background location task (host cleanup). */ +export function stopPhoneLocation(): void { + Location.stopLocationUpdatesAsync(LOCATION_TASK_NAME).catch(() => {}) +} + +export const phoneLocationService = { + LOCATION_TASK_NAME, + getLocationAccuracy, + setLocationTier, + stopPhoneLocation, +} diff --git a/mobile/modules/island/src/services/PhoneNotificationsSync.ts b/mobile/modules/island/src/services/PhoneNotificationsSync.ts new file mode 100644 index 0000000000..a9a1952087 --- /dev/null +++ b/mobile/modules/island/src/services/PhoneNotificationsSync.ts @@ -0,0 +1,44 @@ +/** + * Phone-notifications sync — island-owned. Pushes the notification-forwarding + * config (`notifications_enabled` + `notifications_blocklist`) to the native + * NotificationListener via `CrustModule.setNotificationConfig`, so + * `toolkit.phoneNotifications.setEnabled()/setBlocklist()` actually reach the + * listener for ANY host — not just the Mentra app, where this used to live in + * MantleManager. Android-only; a no-op elsewhere. Started by `toolkit.start()`. + */ +import {shallow} from "zustand/shallow" +import CrustModule from "@mentra/crust" + +import {useSettingsStore, SETTINGS} from "../stores/settings" + +let unsubscribe: (() => void) | null = null + +function pushConfig(): void { + const s = useSettingsStore.getState() + const enabled = Boolean(s.getSetting(SETTINGS.notifications_enabled.key)) + const blocklist = s.getSetting(SETTINGS.notifications_blocklist.key) + // Unconditional (matches the prior MantleManager sync) — the listener is + // Android-only, but CrustModule.setNotificationConfig is a safe no-op elsewhere; + // the Android-only gating lives on the facade's setEnabled/setBlocklist. + CrustModule.setNotificationConfig(enabled, Array.isArray(blocklist) ? blocklist : []).catch((err) => + console.warn(`PhoneNotificationsSync: setNotificationConfig failed: ${(err as Error)?.message ?? err}`), + ) +} + +export function startPhoneNotificationsSync(): void { + if (unsubscribe) return + pushConfig() // initial sync so the listener has the current config + unsubscribe = useSettingsStore.subscribe( + (state) => ({ + enabled: state.getSetting(SETTINGS.notifications_enabled.key), + blocklist: state.getSetting(SETTINGS.notifications_blocklist.key), + }), + () => pushConfig(), + {equalityFn: shallow}, + ) +} + +export function stopPhoneNotificationsSync(): void { + unsubscribe?.() + unsubscribe = null +} diff --git a/mobile/src/services/photo/PhonePhotoCoordinator.ts b/mobile/modules/island/src/services/PhonePhotoCoordinator.ts similarity index 88% rename from mobile/src/services/photo/PhonePhotoCoordinator.ts rename to mobile/modules/island/src/services/PhonePhotoCoordinator.ts index fe90334ee3..8aee947141 100644 --- a/mobile/src/services/photo/PhonePhotoCoordinator.ts +++ b/mobile/modules/island/src/services/PhonePhotoCoordinator.ts @@ -5,10 +5,10 @@ * miniapp → SDK → LocalMiniappRuntime → photo runtime hook * → coordinator.takePhoto(packageName, opts) * ├── (precheck) glasses connected + hasCamera - * ├── cloudClient.startManagedPhoto → {requestId, uploadUrl, readUrl} (cloud-v2 runtime presign) + * ├── cloudClientService.startManagedPhoto → {requestId, uploadUrl, readUrl} (cloud-v2 runtime presign) * ├── BluetoothSdk.requestPhoto(requestId, packageName, size, uploadUrl, compress, sound) * └── race: - * - cloudClient.awaitManagedPhotoReady(requestId) resolves on photo.ready push + * - cloudClientService.awaitManagedPhotoReady(requestId) resolves on photo.ready push * - BluetoothSdk.requestPhoto rejects if terminal photo_response is an error * - handlePhotoError(requestId, code, message) rejects if MantleManager observes * the same BLE photo_response error before the native promise crosses the bridge @@ -18,12 +18,32 @@ * etc.) instead of waiting 30s for cloud's timeout. */ -import BluetoothSdk from "@mentra/bluetooth-sdk" -import {getRuntimeHooks} from "@mentra/island" +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import type {PhotoSize} from "../../../bluetooth-sdk/build/_internal" +import {cloudClientService} from "./CloudClientService" +import {isGlassesConnected} from "./GlassesReadiness" +import {useGlassesStore} from "../stores/glasses" -import {normalizePhotoSize} from "@/services/SocketComms.normalizers" -import {cloudClient} from "@/services/cloudClient" -import {freePhoto, pollUntilReady, requestPhoto, type PhotoResult} from "./v2PhotoApi" +interface PhotoResult { + photoUrl: string + mimeType: string + size: number +} + +/** Map legacy/cloud size names onto the native take_photo enum. */ +function normalizePhotoSize(value: unknown): PhotoSize { + if (typeof value !== "string") return "medium" + switch (value) { + case "small": + return "low" + case "large": + return "high" + case "full": + return "max" + default: + return (["low", "medium", "high", "max"] as const).includes(value as PhotoSize) ? (value as PhotoSize) : "medium" + } +} export interface PhotoOpts { /** Legacy cloud size names are normalized before the native take_photo command. */ @@ -93,8 +113,7 @@ export class PhonePhotoCoordinator { // handler will return a photo_response error within ~1s and the gated // photo_response listener in MantleManager will short-circuit our // long-poll. Slower but correct. - const glasses = getRuntimeHooks().glassesStatus?.get() - if (!glasses?.connected) { + if (!isGlassesConnected(useGlassesStore.getState().connection)) { throw new PhotoError("GLASSES_NOT_CONNECTED", "Glasses are not connected", "command", "ble") } @@ -106,7 +125,7 @@ export class PhonePhotoCoordinator { let uploadUrl: string let readUrl: string try { - const r = await cloudClient.startManagedPhoto({size: opts.size ?? "medium"}) + const r = await cloudClientService.startManagedPhoto({size: opts.size ?? "medium"}) requestId = r.requestId uploadUrl = r.uploadUrl readUrl = r.readUrl @@ -182,7 +201,7 @@ export class PhonePhotoCoordinator { // 4) Await the runtime's photo.ready push (replaces the legacy long-poll). // handlePhotoError races against it and uses the same entry to reject // first. - cloudClient + cloudClientService .awaitManagedPhotoReady(requestId) .then((res) => { const e = this.activeRequests.get(requestId) diff --git a/mobile/src/services/streaming/PhoneStreamCoordinator.ts b/mobile/modules/island/src/services/PhoneStreamCoordinator.ts similarity index 99% rename from mobile/src/services/streaming/PhoneStreamCoordinator.ts rename to mobile/modules/island/src/services/PhoneStreamCoordinator.ts index 93f2dfecd8..88332bdbbc 100644 --- a/mobile/src/services/streaming/PhoneStreamCoordinator.ts +++ b/mobile/modules/island/src/services/PhoneStreamCoordinator.ts @@ -28,14 +28,15 @@ * the legacy path. */ -import BluetoothSdk from "@mentra/bluetooth-sdk-internal" +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" import type { KeepAliveAckEvent, StreamResolvedConfig, StreamStartRequest, StreamStatusEvent, -} from "@mentra/bluetooth-sdk-internal" -import {getRuntimeHooks} from "@mentra/island" +} from "../../../bluetooth-sdk/build/_internal" +import {isGlassesConnected} from "./GlassesReadiness" +import {useGlassesStore} from "../stores/glasses" import {StreamLifecycleController, type LifecycleLogger} from "./StreamLifecycleController" import { @@ -179,8 +180,7 @@ export class StreamConflictError extends Error { * tear the input down again: a slow, billable no-op with a confusing error. */ function assertGlassesConnected(): void { - const glasses = getRuntimeHooks().glassesStatus?.get() - if (!glasses?.connected) { + if (!isGlassesConnected(useGlassesStore.getState().connection)) { throw new StreamConflictError( "GLASSES_NOT_CONNECTED", "Glasses are not connected", diff --git a/mobile/src/services/video/PhoneVideoCoordinator.ts b/mobile/modules/island/src/services/PhoneVideoCoordinator.ts similarity index 96% rename from mobile/src/services/video/PhoneVideoCoordinator.ts rename to mobile/modules/island/src/services/PhoneVideoCoordinator.ts index fa4c3bc670..322c128ff4 100644 --- a/mobile/src/services/video/PhoneVideoCoordinator.ts +++ b/mobile/modules/island/src/services/PhoneVideoCoordinator.ts @@ -15,8 +15,9 @@ * closed/crashed miniapp be cleaned up on unregister. */ -import BluetoothSdk from "@mentra/bluetooth-sdk" -import {getRuntimeHooks} from "@mentra/island" +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import {useGlassesStore} from "../stores/glasses" +import {isGlassesConnected} from "./GlassesReadiness" export interface VideoRecordingOpts { width?: number @@ -57,8 +58,7 @@ export class PhoneVideoCoordinator { async startRecording(packageName: string, opts: VideoRecordingOpts): Promise { // Fail fast if glasses aren't connected — the BLE command would otherwise // be sent into the void. - const glasses = getRuntimeHooks().glassesStatus?.get() - if (!glasses?.connected) { + if (!isGlassesConnected(useGlassesStore.getState().connection)) { throw new VideoError("GLASSES_NOT_CONNECTED", "Glasses are not connected") } diff --git a/mobile/modules/island/src/services/RestComms.ts b/mobile/modules/island/src/services/RestComms.ts new file mode 100644 index 0000000000..df0276af6b --- /dev/null +++ b/mobile/modules/island/src/services/RestComms.ts @@ -0,0 +1,596 @@ +import axios, {AxiosInstance, AxiosRequestConfig} from "axios" +import {AsyncResult, Result, result as Res} from "typesafe-ts" + +// Internal btsdk surface (RestComms needs updateBluetoothSettings, not on the +// public entry). Relative path — `@mentra/bluetooth-sdk-internal` is a mobile-app +// tsconfig alias that doesn't resolve in island's standalone build. Same path +// island's index.ts uses, one level deeper from services/. +import BluetoothSdk, {type PhotoResponseEvent} from "../../../bluetooth-sdk/build/_internal" +import {SETTINGS, useSettingsStore} from "../stores/settings" +import {useConnectionStore} from "../stores/connection" +import {WebSocketStatus} from "../stores/connection" +import GlobalEventEmitter from "../utils/GlobalEventEmitter" +import {BgTimer} from "../utils/timers" + +interface RequestConfig { + method: "GET" | "POST" | "DELETE" + endpoint: string + data?: any + params?: any + requiresAuth?: boolean +} + +class RestComms { + private static instance: RestComms + private readonly TAG = "RestComms" + private coreToken: string | null = null + private axiosInstance: AxiosInstance + + private constructor() { + this.axiosInstance = axios.create({ + headers: { + "Content-Type": "application/json", + }, + }) + } + + public static getInstance(): RestComms { + if (!RestComms.instance) { + RestComms.instance = new RestComms() + } + return RestComms.instance + } + + // Token Management + public setCoreToken(token: string | null): void { + this.coreToken = token + const tokenLen = token?.length ?? 0 + console.log( + `${this.TAG}: Core token ${token ? "set" : "cleared"} - Length: ${tokenLen} - First 20 chars: ${ + token?.substring(0, 20) || "null" + }`, + ) + + // Sync to native DeviceStore (and persist to SharedPreferences in BluetoothSdkModule when bridge runs) + const value = token ?? "" + const updateResult = BluetoothSdk.updateBluetoothSettings({core_token: value}) + if (updateResult != null && typeof (updateResult as Promise).then === "function") { + ;(updateResult as Promise).catch(() => {}) + } + + if (token) { + console.log(`${this.TAG}: Core token set, emitting CORE_TOKEN_SET event`) + GlobalEventEmitter.emit("CORE_TOKEN_SET") + } + } + + public getCoreToken(): string | null { + return this.coreToken + } + + // Helper Methods + private validateToken(): Result { + if (!this.coreToken) { + return Res.error(new Error("No core token available for authentication")) + } + return Res.ok(undefined) + } + + private createAuthHeaders(): Record { + return { + "Content-Type": "application/json", + "Authorization": `Bearer ${this.coreToken}`, + } + } + + private makeRequest(config: RequestConfig): AsyncResult { + const {method, endpoint, data, params, requiresAuth = true} = config + + const baseUrl = useSettingsStore.getState().getRestUrl() + const url = `${baseUrl}${endpoint}` + // console.log(`REST: ${method}:${url}`) + + const headers = requiresAuth ? this.createAuthHeaders() : {"Content-Type": "application/json"} + + const axiosConfig: AxiosRequestConfig = { + method, + url, + headers, + data, + params, + } + + return Res.try_async(async () => { + try { + const res = await this.axiosInstance.request(axiosConfig) + return res.data + } catch (error) { + if (!this.isNoActiveSessionError(error)) { + throw error + } + + // Cloud pod has no session for this user (we reconnected to a different + // pod, or the prior session was cleaned up). Trigger a WS reconnect, + // wait for it to land, then retry the request exactly once. + // + // Subscribe BEFORE emitting so we don't miss the DISCONNECTED → CONNECTED + // transition triggered by handleNoActiveSession → reconnectNow. + const waitPromise = this.waitForNextConnected(8_000) + GlobalEventEmitter.emit("NO_ACTIVE_SESSION") + try { + await waitPromise + } catch (waitErr) { + console.log(`${this.TAG}: Retry skipped — WS didn't reconnect in time:`, waitErr) + throw error + } + + const retryHeaders = requiresAuth ? this.createAuthHeaders() : {"Content-Type": "application/json"} + const retryRes = await this.axiosInstance.request({...axiosConfig, headers: retryHeaders}) + return retryRes.data + } + }) + } + + /** + * Resolves on the NEXT CONNECTED transition of the WS (or rejects after + * timeoutMs). Does NOT short-circuit when already CONNECTED — callers + * invoke this after a 503 NO_ACTIVE_SESSION when we know the current + * connection is landing on the wrong pod; we need to wait for the + * post-reconnect CONNECTED event, not the current one. + * + * Uses the connection store directly rather than WebSocketManager to avoid + * a circular import (WebSocketManager → RestComms → WebSocketManager). + */ + private waitForNextConnected(timeoutMs: number): Promise { + return new Promise((resolve, reject) => { + let settled = false + const timer = BgTimer.setTimeout(() => { + if (settled) return + settled = true + unsub() + reject(new Error(`waitForNextConnected timed out after ${timeoutMs}ms`)) + }, timeoutMs) + + let sawNonConnected = useConnectionStore.getState().status !== WebSocketStatus.CONNECTED + const unsub = useConnectionStore.subscribe((state) => { + if (settled) return + if (state.status !== WebSocketStatus.CONNECTED) { + sawNonConnected = true + return + } + // Only resolve on a CONNECTED transition that follows a non-CONNECTED + // state. This guarantees we waited for a real reconnect rather than + // resolving on the stale pre-reconnect CONNECTED state. + if (sawNonConnected) { + settled = true + BgTimer.clearTimeout(timer) + unsub() + resolve() + } + }) + }) + } + + private isNoActiveSessionError(error: unknown): boolean { + if (!axios.isAxiosError(error)) { + return false + } + + return error.response?.status === 503 && error.response?.data?.error === "NO_ACTIVE_SESSION" + } + + private authenticatedRequest(config: RequestConfig): AsyncResult { + let res = this.validateToken() + if (res.is_error()) { + return Res.error_async(res.error) + } + return this.makeRequest({...config}) + } + + private unauthenticatedRequest(config: RequestConfig): AsyncResult { + return this.makeRequest({...config, requiresAuth: false}) + } + + // Public API Methods + + public getMinimumClientVersion(): AsyncResult<{required: string; recommended: string}, Error> { + interface Response { + success: boolean + data: {required: string; recommended: string} + } + const config: RequestConfig = { + method: "GET", + endpoint: "/api/client/min-version", + } + const res = this.unauthenticatedRequest(config) + return res.map((response) => response.data) + } + + public retry(fn: () => AsyncResult, attempts: number, delayMs: number = 0): AsyncResult { + return Res.try_async(async () => { + let lastError: Error | null = null + + for (let i = 0; i < attempts; i++) { + const result: Result = await fn() + if (result.is_ok()) { + return result.value + } + lastError = result.error + if (i < attempts - 1 && delayMs > 0) { + await new Promise((resolve) => setTimeout(resolve, delayMs)) + } + } + throw lastError + }) + } + + public updateGlassesState(state: Record): AsyncResult { + const config: RequestConfig = { + method: "POST", + endpoint: "/api/client/device/state", + data: state, + } + interface Response { + success: boolean + } + const res = this.authenticatedRequest(config) + return res.map(() => undefined) + } + + public exchangeToken(token: string): AsyncResult { + const isChina: string = useSettingsStore.getState().getSetting(SETTINGS.china_deployment.key) + + const config: RequestConfig = { + method: "POST", + endpoint: "/auth/exchange-token", + data: { + supabaseToken: !isChina ? token : undefined, + authingToken: isChina ? token : undefined, + }, + } + interface Response { + coreToken: string + } + let res = this.makeRequest(config) + const coreTokenResult: AsyncResult = res.map((response) => response.coreToken) + + // set the core token in the store: + return coreTokenResult.and_then((coreToken: string) => { + this.setCoreToken(coreToken) + return Res.ok(coreToken) + }) + } + + // Account Management + public requestAccountDeletion(): AsyncResult { + const config: RequestConfig = { + method: "POST", + endpoint: "/api/account/request-deletion", + } + interface Response { + success: boolean + } + const res = this.authenticatedRequest(config) + return res.map(() => undefined) + } + + public confirmAccountDeletion(requestId: string, confirmationCode: string): AsyncResult { + const config: RequestConfig = { + method: "DELETE", + endpoint: "/api/account/confirm-deletion", + data: {requestId, confirmationCode}, + } + interface Response { + success: boolean + } + const res = this.authenticatedRequest(config) + return res + } + + public getLivekitUrlAndToken(): AsyncResult<{url: string; token: string}, Error> { + const config: RequestConfig = { + method: "GET", + endpoint: "/api/client/livekit/token", + } + interface Response { + // url: string + // token: string + success: boolean + data: {url: string; token: string} + } + const res = this.authenticatedRequest(config) + + // ;(async () => { + // console.log("result@@@@@", await result) + // // const response = await Res.value + // // return {url: response.url, token: response.token} + // })() + + return res.map((response) => response.data) + } + + // User Feedback & Incidents + + /** + * Create a new incident for a bug report. + * Returns incidentId for subsequent log/attachment uploads. + */ + public createIncident( + feedback: object, + phoneState?: Record, + ): AsyncResult<{success: boolean; incidentId: string}, Error> { + const config: RequestConfig = { + method: "POST", + endpoint: "/api/incidents", + data: { + feedback, + ...(phoneState && {phoneState}), + }, + } + interface Response { + success: boolean + incidentId: string + } + return this.authenticatedRequest(config) + } + + /** + * Submit feedback (feature requests only). + * For bug reports, use createIncident instead. + */ + public sendFeedback( + feedbackBody: string | object, + phoneState?: Record, + ): AsyncResult<{success: boolean}, Error> { + const config: RequestConfig = { + method: "POST", + endpoint: "/api/client/feedback", + data: { + feedback: feedbackBody, + ...(phoneState && {phoneState}), + }, + } + interface Response { + success: boolean + } + return this.authenticatedRequest(config) + } + + /** + * Upload phone logs to an incident. + * Called after createIncident returns an incidentId. + */ + public uploadIncidentLogs( + incidentId: string, + logs: Array<{timestamp: number; level: string; message: string; source?: string}>, + ): AsyncResult { + const config: RequestConfig = { + method: "POST", + endpoint: `/api/incidents/${incidentId}/logs`, + data: { + source: "phone", + logs, + }, + } + interface Response { + success: boolean + } + const res = this.authenticatedRequest(config) + return res.map(() => undefined) + } + + /** + * Upload screenshot attachments to an incident. + * Called after createIncident returns an incidentId. + */ + public uploadIncidentAttachments( + incidentId: string, + images: Array<{uri: string; fileName?: string | null; mimeType?: string | null}>, + ): AsyncResult<{uploaded: number; errors: number}, Error> { + const uploadPromise = async (): Promise> => { + try { + const coreToken = this.getCoreToken() + if (!coreToken) { + return Res.error(new Error("Not authenticated")) + } + + const baseUrl = useSettingsStore.getState().getRestUrl() + const formData = new FormData() + + for (const image of images) { + const filename = image.fileName || `screenshot-${Date.now()}.jpg` + const mimeType = image.mimeType || "image/jpeg" + + // React Native FormData expects this format + formData.append("files", { + uri: image.uri, + name: filename, + type: mimeType, + } as unknown as Blob) + } + + const response = await axios.post(`${baseUrl}/api/incidents/${incidentId}/attachments`, formData, { + headers: { + "Authorization": `Bearer ${coreToken}`, + "Content-Type": "multipart/form-data", + }, + timeout: 60000, // 60 second timeout for uploads + }) + + const data = response.data as { + success: boolean + uploaded?: Array<{filename: string}> + errors?: Array<{filename: string; error: string}> + } + + return Res.ok({ + uploaded: data.uploaded?.length || 0, + errors: data.errors?.length || 0, + }) + } catch (err) { + return Res.error(err instanceof Error ? err : new Error(String(err))) + } + } + + return new AsyncResult(uploadPromise()) + } + + public writeUserSettings(settings: any): AsyncResult { + const config: RequestConfig = { + method: "POST", + endpoint: "/api/client/user/settings", + data: {settings}, + } + interface Response { + success: boolean + } + const res = this.authenticatedRequest(config) + return res.map(() => undefined) + } + + public loadUserSettings(): AsyncResult { + const config: RequestConfig = { + method: "GET", + endpoint: "/api/client/user/settings", + } + interface Response { + success: boolean + data: {settings: Record} + } + const res = this.authenticatedRequest(config) + return res.map((response) => response.data.settings) + } + + // Error Reporting + public sendErrorReport(reportData: any): AsyncResult { + const config: RequestConfig = { + method: "POST", + endpoint: "/app/error-report", + data: reportData, + } + interface Response { + success: boolean + data: any + } + const res = this.authenticatedRequest(config) + return res.map(() => undefined) + } + + // Calendar + public sendCalendarData(data: any): AsyncResult { + const config: RequestConfig = { + method: "POST", + endpoint: "/api/client/calendar", + data: data, + } + interface Response { + success: boolean + data: any + } + const res = this.authenticatedRequest(config) + return res.map(() => undefined) + } + + // Location + public sendLocationData(data: any): AsyncResult { + const config: RequestConfig = { + method: "POST", + endpoint: "/api/client/location", + data: data, + } + interface Response { + success: boolean + data: any + } + const res = this.authenticatedRequest(config) + return res.map(() => undefined) + } + + // Phone Notifications + public sendPhoneNotification(data: { + notificationId: string + app: string + title: string + content: string + priority: string + timestamp: number + packageName: string + }): AsyncResult { + const config: RequestConfig = { + method: "POST", + endpoint: "/api/client/notifications", + data: data, + } + interface Response { + success: boolean + data: any + } + const res = this.authenticatedRequest(config) + return res.map(() => undefined) + } + + public sendPhoneNotificationDismissed(data: { + notificationId: string + notificationKey: string + packageName: string + }): AsyncResult { + const config: RequestConfig = { + method: "POST", + endpoint: "/api/client/notifications/dismissed", + data: data, + } + interface Response { + success: boolean + data: any + } + const res = this.authenticatedRequest(config) + return res.map(() => undefined) + } + + public sendPhotoResponse(data: PhotoResponseEvent): AsyncResult { + const response = + data.state === "success" + ? { + type: data.type, + requestId: data.requestId, + photoUrl: data.uploadUrl, + timestamp: data.timestamp, + success: true, + } + : { + type: data.type, + requestId: data.requestId, + timestamp: data.timestamp, + success: false, + errorCode: data.errorCode, + errorMessage: data.errorMessage, + } + const config: RequestConfig = { + method: "POST", + endpoint: "/api/client/photo/response", + data: response, + } + interface Response { + success: boolean + data: any + } + const res = this.authenticatedRequest(config) + return res.map(() => undefined) + } + + public goodbye(): AsyncResult { + const config: RequestConfig = { + method: "POST", + endpoint: "/api/client/goodbye", + } + interface Response { + success: boolean + data: any + } + const res = this.authenticatedRequest(config) + return res.map(() => undefined) + } +} + +const restComms = RestComms.getInstance() +export default restComms diff --git a/mobile/src/services/streaming/StreamLifecycleController.ts b/mobile/modules/island/src/services/StreamLifecycleController.ts similarity index 100% rename from mobile/src/services/streaming/StreamLifecycleController.ts rename to mobile/modules/island/src/services/StreamLifecycleController.ts diff --git a/mobile/modules/island/src/services/__tests__/LocalDisplayManager.test.ts b/mobile/modules/island/src/services/__tests__/LocalDisplayManager.test.ts index 88f46dd78f..3988aff2c9 100644 --- a/mobile/modules/island/src/services/__tests__/LocalDisplayManager.test.ts +++ b/mobile/modules/island/src/services/__tests__/LocalDisplayManager.test.ts @@ -1,3 +1,7 @@ +/// + +import {afterEach, beforeEach, describe, expect, jest, mock, test} from "bun:test" + /** * Unit tests for LocalDisplayManager. * @@ -5,21 +9,34 @@ * arbitration. Uses jest fake timers + an injected clock. */ -const displayEventMock = jest.fn() +const displayEventMock = mock(() => {}) // DisplayProcessor: pass through unchanged so we can assert on the raw event. -jest.doMock("../DisplayProcessor", () => ({ +mock.module("../DisplayProcessor", () => ({ __esModule: true, default: { processDisplayEvent: (e: Record) => ({...e, _processed: true}), }, })) -const setDisplayEventMock = jest.fn() +mock.module("../../../../bluetooth-sdk/build/_internal", () => ({ + __esModule: true, + default: { + displayEvent: displayEventMock, + }, +})) + +mock.module("../../utils/timers", () => ({ + BgTimer: { + setTimeout: (callback: () => void, delay: number) => setTimeout(callback, delay) as unknown as number, + clearTimeout: (timeoutId: number) => clearTimeout(timeoutId), + setInterval: (callback: () => void, delay: number) => setInterval(callback, delay) as unknown as number, + clearInterval: (intervalId: number) => clearInterval(intervalId), + }, +})) // Import AFTER mocks -const {configureRuntime} = require("../../runtime/config") const {LocalDisplayManager} = require("../LocalDisplayManager") type Mgr = InstanceType @@ -56,11 +73,6 @@ describe("LocalDisplayManager", () => { beforeEach(() => { jest.useFakeTimers() displayEventMock.mockClear() - setDisplayEventMock.mockClear() - configureRuntime({ - sendDisplayEvent: displayEventMock, - setDisplayEvent: setDisplayEventMock, - }) now = 1_000_000 // Fresh singleton per test. mgr = LocalDisplayManager.getInstance() diff --git a/mobile/modules/island/src/services/__tests__/MicStateCoordinator.test.ts b/mobile/modules/island/src/services/__tests__/MicStateCoordinator.test.ts index 03dcd1e2ae..52da9ffa73 100644 --- a/mobile/modules/island/src/services/__tests__/MicStateCoordinator.test.ts +++ b/mobile/modules/island/src/services/__tests__/MicStateCoordinator.test.ts @@ -1,73 +1,80 @@ -const mockSetMicRequirements = jest.fn() +/// -// Import AFTER the mock is registered +import {beforeEach, describe, expect, mock, test} from "bun:test" + +const mockUpdateBluetoothSettings = mock(() => Promise.resolve()) + +mock.module("../../../../bluetooth-sdk/build/_internal", () => ({ + __esModule: true, + default: { + updateBluetoothSettings: mockUpdateBluetoothSettings, + }, +})) -const {configureRuntime} = require("../../runtime/config") +// Import AFTER the mock is registered const MicStateCoordinator = require("../MicStateCoordinator").default describe("MicStateCoordinator", () => { beforeEach(() => { - configureRuntime({setMicRequirements: mockSetMicRequirements}) MicStateCoordinator.reset() - mockSetMicRequirements.mockClear() + mockUpdateBluetoothSettings.mockClear() }) - test("cloud-only PCM requirement", () => { - MicStateCoordinator.setCloudRequirements({pcm: true, lc3: false, transcript: false}) - expect(mockSetMicRequirements).toHaveBeenCalledWith( + test("local PCM requirement", () => { + MicStateCoordinator.setLocalRequirements({pcm: true, lc3: false}) + expect(mockUpdateBluetoothSettings).toHaveBeenCalledWith( expect.objectContaining({ - shouldSendPcm: false, - shouldSendLc3: true, + should_send_pcm: true, + should_send_lc3: false, + should_send_transcript: false, }), ) }) - test("local-only LC3 requirement", () => { + test("local LC3 requirement", () => { MicStateCoordinator.setLocalRequirements({pcm: false, lc3: true}) - expect(mockSetMicRequirements).toHaveBeenCalledWith( + expect(mockUpdateBluetoothSettings).toHaveBeenCalledWith( expect.objectContaining({ - shouldSendLc3: true, + should_send_pcm: false, + should_send_lc3: true, + should_send_transcript: false, }), ) }) - test("union of cloud + local", () => { - MicStateCoordinator.setCloudRequirements({ - pcm: true, - lc3: false, - transcript: true, - }) - MicStateCoordinator.setLocalRequirements({pcm: false, lc3: true}) - const lastCall = mockSetMicRequirements.mock.calls[mockSetMicRequirements.mock.calls.length - 1] + test("local PCM and LC3 can be enabled together", () => { + MicStateCoordinator.setLocalRequirements({pcm: true, lc3: true}) + const lastCall = mockUpdateBluetoothSettings.mock.calls[mockUpdateBluetoothSettings.mock.calls.length - 1] expect(lastCall[0]).toEqual( expect.objectContaining({ - shouldSendPcm: false, - shouldSendLc3: true, - shouldSendTranscript: true, + should_send_pcm: true, + should_send_lc3: true, + should_send_transcript: false, }), ) }) test("both off means all false", () => { - MicStateCoordinator.setCloudRequirements({pcm: false, lc3: false, transcript: false}) MicStateCoordinator.setLocalRequirements({pcm: false, lc3: false}) - const lastCall = mockSetMicRequirements.mock.calls[mockSetMicRequirements.mock.calls.length - 1] + const lastCall = mockUpdateBluetoothSettings.mock.calls[mockUpdateBluetoothSettings.mock.calls.length - 1] expect(lastCall[0]).toEqual( expect.objectContaining({ - shouldSendPcm: false, - shouldSendLc3: false, + should_send_pcm: false, + should_send_lc3: false, + should_send_transcript: false, }), ) }) - test("local unsubscribe doesn't kill cloud mic", () => { - MicStateCoordinator.setCloudRequirements({pcm: false, lc3: true, transcript: true}) + test("local unsubscribe turns mic requirements off", () => { MicStateCoordinator.setLocalRequirements({pcm: false, lc3: true}) MicStateCoordinator.setLocalRequirements({pcm: false, lc3: false}) - const lastCall = mockSetMicRequirements.mock.calls[mockSetMicRequirements.mock.calls.length - 1] + const lastCall = mockUpdateBluetoothSettings.mock.calls[mockUpdateBluetoothSettings.mock.calls.length - 1] expect(lastCall[0]).toEqual( expect.objectContaining({ - shouldSendLc3: true, + should_send_pcm: false, + should_send_lc3: false, + should_send_transcript: false, }), ) }) diff --git a/mobile/modules/island/src/services/__tests__/MiniappLauncher.test.ts b/mobile/modules/island/src/services/__tests__/MiniappLauncher.test.ts index 5e36f7f776..54fd95711e 100644 --- a/mobile/modules/island/src/services/__tests__/MiniappLauncher.test.ts +++ b/mobile/modules/island/src/services/__tests__/MiniappLauncher.test.ts @@ -46,12 +46,10 @@ mock.module("expo-file-system", () => ({ })) let miniappLauncher: typeof import("../MiniappLauncher").miniappLauncher -let configureLauncher: typeof import("../MiniappLauncher").configureLauncher beforeAll(async () => { const mod = await import("../MiniappLauncher") miniappLauncher = mod.miniappLauncher - configureLauncher = mod.configureLauncher }) // Fresh router (mutable registered set) per test. @@ -81,7 +79,7 @@ describe("MiniappLauncher", () => { activeVersion = "1.0.0" waitForConnectCalls = [] mockRouter = buildMockRouter() - configureLauncher({router: mockRouter.router}) + miniappLauncher.configure({router: mockRouter.router}) }) test("ensureRunning spawns the background context when not registered", async () => { diff --git a/mobile/src/services/streaming/PhoneStreamCoordinator.test.ts b/mobile/modules/island/src/services/__tests__/PhoneStreamCoordinator.test.ts similarity index 96% rename from mobile/src/services/streaming/PhoneStreamCoordinator.test.ts rename to mobile/modules/island/src/services/__tests__/PhoneStreamCoordinator.test.ts index 3a1b77257f..ba3ad786cb 100644 --- a/mobile/src/services/streaming/PhoneStreamCoordinator.test.ts +++ b/mobile/modules/island/src/services/__tests__/PhoneStreamCoordinator.test.ts @@ -15,7 +15,7 @@ const startExternallyManagedStream = mock(async (req: unknown) => streamStatusFo const stopStream = mock(async () => {}) const sendExternallyManagedStreamKeepAlive = mock(async (_req: unknown) => {}) -mock.module("@mentra/bluetooth-sdk-internal", () => ({ +mock.module("../../../../bluetooth-sdk/build/_internal", () => ({ default: {startStream, startExternallyManagedStream, stopStream, sendExternallyManagedStreamKeepAlive}, })) @@ -35,17 +35,22 @@ const getManagedStreamStatus = mock(async (_id: string) => ({ })) const teardownManagedStream = mock(async (_id: string) => {}) -mock.module("./cloudStreamApi", () => ({ +mock.module("../cloudStreamApi", () => ({ provisionManagedStream, getManagedStreamStatus, teardownManagedStream, })) -// The coordinator's glasses-connected precheck reads island runtime hooks; -// mocking the module also keeps bun from parsing react-native's flow types. -mock.module("@mentra/island", () => ({ - getRuntimeHooks: () => ({glassesStatus: {get: () => ({connected: true})}}), +// The coordinator's glasses-connected precheck reads the island glasses store via +// isGlassesConnected. Mock both (the real store transitively drags react-native, +// which bun can't parse) so the precheck passes deterministically. +mock.module("../../stores/glasses", () => ({ + useGlassesStore: {getState: () => ({connection: {state: "connected"}})}, })) +mock.module("../GlassesReadiness", () => ({ + isGlassesConnected: () => true, +})) + // Patch global fetch so the HLS readiness HEAD probe is deterministic. let hlsHeadResponder: () => Response = () => new Response(null, {status: 200}) @@ -70,7 +75,7 @@ afterEach(() => { ;(globalThis as {fetch: typeof fetch}).fetch = realFetch }) -const {PhoneStreamCoordinator, StreamConflictError} = await import("./PhoneStreamCoordinator") +const {PhoneStreamCoordinator, StreamConflictError} = await import("../PhoneStreamCoordinator") describe("PhoneStreamCoordinator", () => { describe("unmanaged", () => { diff --git a/mobile/src/services/streaming/StreamLifecycleController.test.ts b/mobile/modules/island/src/services/__tests__/StreamLifecycleController.test.ts similarity index 99% rename from mobile/src/services/streaming/StreamLifecycleController.test.ts rename to mobile/modules/island/src/services/__tests__/StreamLifecycleController.test.ts index 802da6da8a..41e30f1536 100644 --- a/mobile/src/services/streaming/StreamLifecycleController.test.ts +++ b/mobile/modules/island/src/services/__tests__/StreamLifecycleController.test.ts @@ -2,7 +2,7 @@ import {beforeEach, describe, expect, mock, test} from "bun:test" -import {StreamLifecycleController, type LifecycleLogger} from "./StreamLifecycleController" +import {StreamLifecycleController, type LifecycleLogger} from "../StreamLifecycleController" const noopLogger: LifecycleLogger = { child: () => noopLogger, diff --git a/mobile/src/services/asg/asgCameraApi.ts b/mobile/modules/island/src/services/asg/asgCameraApi.ts similarity index 99% rename from mobile/src/services/asg/asgCameraApi.ts rename to mobile/modules/island/src/services/asg/asgCameraApi.ts index f09c3e3cb9..ed6b414c3c 100644 --- a/mobile/src/services/asg/asgCameraApi.ts +++ b/mobile/modules/island/src/services/asg/asgCameraApi.ts @@ -5,8 +5,8 @@ import * as RNFS from "@dr.pogodin/react-native-fs" -import {PhotoInfo, CaptureGroup, GalleryResponse, ServerStatus, HealthResponse} from "@/types/asg" -import {BgTimer} from "@mentra/island" +import {PhotoInfo, CaptureGroup, GalleryResponse, ServerStatus, HealthResponse} from "../../types/asg" +import {BgTimer} from "../../utils/timers" import {localStorageService} from "./localStorageService" import {validateDownloadedMediaFile} from "./galleryMediaValidation" @@ -15,8 +15,6 @@ export class AsgCameraApiClient { private baseUrl: string private port: number private lastRequestTime: number = 0 - private requestQueue: Array<() => Promise> = [] - private isProcessingQueue: boolean = false constructor(serverUrl?: string, port: number = 8089) { this.port = port diff --git a/mobile/src/services/asg/galleryMediaValidation.ts b/mobile/modules/island/src/services/asg/galleryMediaValidation.ts similarity index 100% rename from mobile/src/services/asg/galleryMediaValidation.ts rename to mobile/modules/island/src/services/asg/galleryMediaValidation.ts diff --git a/mobile/modules/island/src/services/asg/galleryNotices.ts b/mobile/modules/island/src/services/asg/galleryNotices.ts new file mode 100644 index 0000000000..d5eb43bb4f --- /dev/null +++ b/mobile/modules/island/src/services/asg/galleryNotices.ts @@ -0,0 +1,43 @@ +/** + * Gallery notices — island-owned. The gallery sync emits STRUCTURED notice codes + * (never localized strings or alerts) when it hits a user-actionable precondition; + * the host renders its own alert / settings deep-link from the code. This keeps the + * sync (island runtime) free of host UI/i18n/navigation — exposed as + * `toolkit.gallery.onNotice(cb)`. + */ +export type GalleryNoticeCode = + | "glasses_disconnected" + | "insufficient_storage" + | "wifi_initializing" + | "wifi_off" + | "location_services_off" + | "connect_to_glasses" + +export interface GalleryNotice { + code: GalleryNoticeCode + /** Optional extra context (e.g. ssid/platform) for the host's message. */ + data?: Record + /** + * Present only on notices the sync BLOCKS on (currently the one-time wifi-join + * explanation): the host calls this when the user acknowledges, letting the sync + * proceed. If no host is listening, the emitter's return value lets the sync fall + * through instead of hanging. + */ + ack?: () => void +} + +type Listener = (notice: GalleryNotice) => void + +const listeners = new Set() + +/** Emit a notice to all subscribers; returns how many received it (0 = nobody listening). */ +export function emitGalleryNotice(notice: GalleryNotice): number { + listeners.forEach((l) => l(notice)) + return listeners.size +} + +/** Subscribe to gallery notices; returns an unsubscribe. */ +export function onGalleryNotice(cb: Listener): () => void { + listeners.add(cb) + return () => listeners.delete(cb) +} diff --git a/mobile/src/services/asg/gallerySettingsService.ts b/mobile/modules/island/src/services/asg/gallerySettingsService.ts similarity index 97% rename from mobile/src/services/asg/gallerySettingsService.ts rename to mobile/modules/island/src/services/asg/gallerySettingsService.ts index 80759f9d7a..3f4b9c7a73 100644 --- a/mobile/src/services/asg/gallerySettingsService.ts +++ b/mobile/modules/island/src/services/asg/gallerySettingsService.ts @@ -1,4 +1,4 @@ -import {storage} from "@/utils/storage/storage" +import {storage} from "../../utils/storage/storage" export interface GallerySettings { autoSaveToCameraRoll: boolean diff --git a/mobile/src/services/asg/gallerySyncNotifications.ts b/mobile/modules/island/src/services/asg/gallerySyncNotifications.ts similarity index 81% rename from mobile/src/services/asg/gallerySyncNotifications.ts rename to mobile/modules/island/src/services/asg/gallerySyncNotifications.ts index 0a2a5393cd..86d8803ec8 100644 --- a/mobile/src/services/asg/gallerySyncNotifications.ts +++ b/mobile/modules/island/src/services/asg/gallerySyncNotifications.ts @@ -11,10 +11,6 @@ // import * as Notifications from "expo-notifications" import {Platform} from "react-native" -// Notification IDs -const _SYNC_NOTIFICATION_ID = "gallery-sync-progress" -const _CHANNEL_ID = "gallery-sync" - // iOS throttling - only update every N seconds to avoid banner spam const IOS_UPDATE_THROTTLE_MS = 10000 // 10 seconds between updates on iOS @@ -32,7 +28,6 @@ const IOS_UPDATE_THROTTLE_MS = 10000 // 10 seconds between updates on iOS class GallerySyncNotifications { private static instance: GallerySyncNotifications - private channelCreated = false private notificationActive = false private lastUpdateTime = 0 // For iOS throttling @@ -65,8 +60,6 @@ class GallerySyncNotifications { // showBadge: false, // }) // } - - this.channelCreated = true } /** @@ -112,17 +105,6 @@ class GallerySyncNotifications { console.log(`[SyncNotifications] Started sync notification for ${totalFiles} files (notifications disabled)`) } - /** - * Create a visual progress bar (Android only) - */ - private createProgressBar(progress: number, width: number = 15): string { - const filled = Math.round((progress / 100) * width) - const empty = width - filled - const filledBar = "●".repeat(filled) - const emptyBar = "○".repeat(empty) - return `${filledBar}${emptyBar}` - } - /** * Update sync progress notification * On iOS, updates are throttled to avoid spamming the user with banner notifications @@ -151,18 +133,8 @@ class GallerySyncNotifications { await this.ensureChannel() - // Calculate overall progress (completed files + current file progress) - const overallProgress = Math.round(((currentFile - 1 + fileProgress / 100) / totalFiles) * 100) - - // Build notification body - progress bar only on Android - let _body: string - if (Platform.OS === "android") { - const progressBar = this.createProgressBar(overallProgress) - _body = `${progressBar} ${overallProgress}%\nDownloading ${currentFile} of ${totalFiles}` - } else { - // iOS: simple text only, no progress bar - _body = `Downloading ${currentFile} of ${totalFiles} (${overallProgress}%)` - } + // Notifications are disabled; the overall-progress + body computation that fed + // scheduleNotificationAsync (below, commented) is omitted to keep this a no-op. // await Notifications.scheduleNotificationAsync({ // identifier: SYNC_NOTIFICATION_ID, @@ -182,19 +154,8 @@ class GallerySyncNotifications { async showSyncComplete(downloadedCount: number, failedCount: number = 0): Promise { await this.ensureChannel() - let _title: string - let _body: string - - if (failedCount === 0) { - _title = "Sync complete" - _body = `Downloaded ${downloadedCount} ${downloadedCount === 1 ? "file" : "files"} from your glasses` - } else if (downloadedCount === 0) { - _title = "Sync failed" - _body = `Failed to download ${failedCount} ${failedCount === 1 ? "file" : "files"}` - } else { - _title = "Sync complete with errors" - _body = `Downloaded ${downloadedCount}, failed ${failedCount}` - } + // Notifications are disabled; the title/body computation that fed + // scheduleNotificationAsync (below, commented) is omitted to keep this a no-op. // await Notifications.scheduleNotificationAsync({ // identifier: SYNC_NOTIFICATION_ID, diff --git a/mobile/src/services/asg/gallerySyncService.ts b/mobile/modules/island/src/services/asg/gallerySyncService.ts similarity index 90% rename from mobile/src/services/asg/gallerySyncService.ts rename to mobile/modules/island/src/services/asg/gallerySyncService.ts index fb9728d586..210ee9d4bd 100644 --- a/mobile/src/services/asg/gallerySyncService.ts +++ b/mobile/modules/island/src/services/asg/gallerySyncService.ts @@ -5,36 +5,30 @@ import * as RNFS from "@dr.pogodin/react-native-fs" import NetInfo from "@react-native-community/netinfo" -import BluetoothSdk from "@mentra/bluetooth-sdk" +import BluetoothSdk from "../../../../bluetooth-sdk/build/_internal" +import CrustModule from "@mentra/crust" import {AppState, AppStateStatus, Platform} from "react-native" import WifiManager from "react-native-wifi-reborn" -import {useGallerySyncStore, HotspotInfo} from "@/stores/gallerySync" -import {isGlassesConnected, selectGlassesConnected, useGlassesStore} from "@/stores/glasses" -import {SETTINGS, useSettingsStore} from "@/stores/settings" -import {PhotoInfo, CaptureGroup} from "@/types/asg" -import {showAlert} from "@/utils/AlertUtils" -import GlobalEventEmitter from "@/utils/GlobalEventEmitter" -import {SettingsNavigationUtils} from "@/utils/SettingsNavigationUtils" -import {BgTimer} from "@mentra/island" -import {MediaLibraryPermissions} from "@/utils/permissions/MediaLibraryPermissions" - -import {translate} from "@/i18n" +import {useGallerySyncStore, HotspotInfo} from "../../stores/gallerySync" +import {selectGlassesConnected, useGlassesStore} from "../../stores/glasses" +import {isGlassesConnected} from "../GlassesReadiness" +import {SETTINGS, useSettingsStore} from "../../stores/settings" +import {PhotoInfo, CaptureGroup} from "../../types/asg" +import GlobalEventEmitter from "../../utils/GlobalEventEmitter" +import {BgTimer} from "../../utils/timers" +import {fixGlassesClockIfSkewed} from "../glassesClockSync" +import {detectClockSkew, isSyncManifestEmpty} from "../gallerySyncClock" +import {MediaLibraryPermissions} from "../../utils/permissions/MediaLibraryPermissions" +import {permissions, PermissionFeatures} from "../../facades/permissions" + import {asgCameraApi} from "./asgCameraApi" -import {fixGlassesClockIfSkewed} from "./glassesClockSync" -import {detectClockSkew, isSyncManifestEmpty} from "./gallerySyncClock" import {gallerySettingsService} from "./gallerySettingsService" import {gallerySyncNotifications} from "./gallerySyncNotifications" import {localStorageService} from "./localStorageService" import {mediaProcessingQueue} from "./mediaProcessingQueue" import {validateCaptureMetadataForDownload} from "./galleryMediaValidation" -import { - checkConnectivityRequirementsUI, - checkFeaturePermissions, - requestFeaturePermissions, - PermissionFeatures, - isLocationServicesEnabled, -} from "@/utils/PermissionsUtils" +import {emitGalleryNotice} from "./galleryNotices" // Timing constants const TIMING = { @@ -48,6 +42,9 @@ const TIMING = { IOS_WIFI_MAX_RETRIES: 5, // Retry multiple times to give user time to accept // WiFi initialization cooldown - prevents repeated "enable WiFi" alerts while WiFi is initializing WIFI_COOLDOWN_MS: 3000, // Wait 3 seconds after user visits WiFi settings before showing alert again + // Cap the native location-services (GPS) check so a hung CrustModule call can't freeze + // the sync pre-flight (mirrors the host's former PermissionsUtils guard). + LOCATION_SERVICES_CHECK_TIMEOUT_MS: 5000, } as const type SyncManifestData = { @@ -375,6 +372,34 @@ class GallerySyncService { return this.syncStartPromise } + /** + * Native location-services (GPS on/off) check, raced against a timeout so a hung + * CrustModule call can't freeze the sync pre-flight. Mirrors the host's former + * PermissionsUtils.isLocationServicesEnabled exactly: on timeout assume enabled + * (proceed); on any other native error assume disabled (block + prompt). Android-only + * callers; iOS doesn't need location for BLE since iOS 13. + */ + private async isLocationServicesEnabled(): Promise { + try { + return await Promise.race([ + CrustModule.isLocationServicesEnabled(), + new Promise((_, reject) => + setTimeout( + () => reject(new Error("Location services check timed out")), + TIMING.LOCATION_SERVICES_CHECK_TIMEOUT_MS, + ), + ), + ]) + } catch (error) { + console.error("[GallerySyncService] Error checking location services:", error) + if (error instanceof Error && error.message.includes("timed out")) { + console.warn("[GallerySyncService] Location services check timed out — assuming enabled so sync can proceed") + return true + } + return false + } + } + private shouldAbortPreFlight(): boolean { if (this.startAborted) { this.startAborted = false @@ -418,23 +443,12 @@ class GallerySyncService { return } - // Reuse shared connectivity gate (BT + Android location); shows the right alert if not ready - const connectivityOk = await checkConnectivityRequirementsUI() - if (this.shouldAbortPreFlight()) { - console.log("[GallerySyncService] Pre-flight aborted after connectivity check") - return - } - if (!connectivityOk) { - console.warn("[GallerySyncService] Sync aborted - connectivity requirements not met") - store.setSyncError("Connectivity requirements not met") - return - } - - // Check if glasses are connected (store-based, secondary check) + // The BT + Android-location connectivity gate now runs host-side before sync() is + // triggered (it shows host UI); island only does the store-based glasses check. if (!glassesConnected) { console.warn("[GallerySyncService] Sync aborted - Glasses not connected") store.setSyncError("Glasses not connected") - showAlert("Glasses Disconnected", "Please connect your glasses before syncing the gallery.", [{text: "OK"}]) + emitGalleryNotice({code: "glasses_disconnected"}) return } @@ -461,10 +475,10 @@ class GallerySyncService { // 2. Location permission (required to read WiFi SSID for hotspot verification) console.log("[GallerySyncService] 📍 Checking location permission...") - const hasLocationPermission = await checkFeaturePermissions(PermissionFeatures.LOCATION) + const hasLocationPermission = await permissions.check(PermissionFeatures.LOCATION) if (!hasLocationPermission) { console.log("[GallerySyncService] ⚠️ Location permission not granted - requesting...") - const granted = await requestFeaturePermissions(PermissionFeatures.LOCATION) + const granted = await permissions.request(PermissionFeatures.LOCATION) if (!granted) { console.warn("[GallerySyncService] ❌ Location permission denied - WiFi SSID verification may fail") // Don't block sync - we'll try anyway and fall back to IP-based verification if needed @@ -511,11 +525,7 @@ class GallerySyncService { console.log(`[GallerySyncService] 💾 Free disk space: ${freeSpaceMB.toFixed(0)} MB`) if (fsInfo.freeSpace < 500 * 1024 * 1024) { console.error("[GallerySyncService] ❌ Insufficient disk space (<500MB)") - showAlert( - "Insufficient Storage", - `Only ${freeSpaceMB.toFixed(0)} MB free. Please free up at least 500 MB before syncing.`, - [{text: "OK"}], - ) + emitGalleryNotice({code: "insufficient_storage", data: {freeSpaceMB: Math.round(freeSpaceMB)}}) store.setSyncError("Insufficient storage space") return } @@ -544,10 +554,7 @@ class GallerySyncService { )}s remaining) - showing wait message`, ) - showAlert("Please Wait", "WiFi is initializing. Please wait a moment before trying to sync again.", [ - {text: "OK"}, - ]) - + emitGalleryNotice({code: "wifi_initializing"}) return } else { // Cooldown expired, clear the timestamp @@ -576,35 +583,21 @@ class GallerySyncService { if (!wifiEnabled) { console.error("[GallerySyncService] ❌ WiFi is disabled - cannot sync") - // Mark that we're waiting for WiFi so we can auto-retry when user returns - this.waitingForWifiRetry = true - - // Show styled alert with option to open settings - showAlert( - "WiFi is Disabled", - "Please enable WiFi to sync photos from your glasses. Would you like to open WiFi settings?", - [ - { - text: "Cancel", - style: "cancel", - onPress: () => { - this.waitingForWifiRetry = false - this.wifiSettingsOpenedAt = null - store.setSyncError("WiFi disabled - enable WiFi and try again") - }, - }, - { - text: "Open Settings", - onPress: async () => { - // Set timestamp so we can enforce cooldown on next sync attempt - this.wifiSettingsOpenedAt = Date.now() - await SettingsNavigationUtils.openWifiSettings() - store.setSyncError("Enable WiFi and try sync again") - }, - }, - ], - {cancelable: false}, - ) + // Don't arm the auto-retry / "WiFi initializing" cooldown yet — whether to arm + // them depends on the user's choice in the host's prompt, which only the host + // learns. The host renders an "enable WiFi → open settings" alert from this + // notice and calls ack() ONLY when the user opts to open settings; on Cancel it + // does nothing, so a declined prompt leaves no pending retry or phantom cooldown. + // The island still owns what arming MEANS (this closure) — the host only reports + // that the user took the affirmative action. + emitGalleryNotice({ + code: "wifi_off", + ack: () => { + this.waitingForWifiRetry = true + this.wifiSettingsOpenedAt = Date.now() + }, + }) + store.setSyncError("WiFi disabled - enable WiFi and try again") // Return early - do NOT proceed with sync return @@ -630,36 +623,16 @@ class GallerySyncService { if (Platform.OS === "android") { console.log("[GallerySyncService] 📍 Checking Location Services status...") try { - const locationServicesEnabled = await isLocationServicesEnabled() + const locationServicesEnabled = await this.isLocationServicesEnabled() console.log("[GallerySyncService] 📍 Location Services enabled:", locationServicesEnabled) if (!locationServicesEnabled) { console.error("[GallerySyncService] ❌ Location Services is OFF - cannot sync") console.error("[GallerySyncService] ❌ Android requires Location Services for WiFi operations") - // Show styled alert with option to enable location services - showAlert( - "Location Services Required", - "Android requires Location Services to be enabled to connect to your glasses WiFi hotspot. Would you like to enable it?", - [ - { - text: "Cancel", - style: "cancel", - onPress: () => { - store.setSyncError("Location Services disabled - enable in Settings and try again") - }, - }, - { - text: "Enable", - onPress: async () => { - // Use the native dialog for better UX (shows in-app prompt on supported devices) - await SettingsNavigationUtils.showLocationServicesDialog() - store.setSyncError("Enable Location Services and try sync again") - }, - }, - ], - {cancelable: false}, - ) + // Emit a notice; the host shows the "enable Location Services" alert + dialog. + emitGalleryNotice({code: "location_services_off"}) + store.setSyncError("Location Services disabled - enable in Settings and try again") // Return early - do NOT proceed with sync return @@ -790,29 +763,22 @@ class GallerySyncService { console.log("[GallerySyncService] First sync - showing WiFi join explanation") - return new Promise((resolve) => { - const message = - Platform.OS === "ios" - ? translate("glasses:wifiJoinExplanationIos", {ssid}) - : translate("glasses:wifiJoinExplanationAndroid", {ssid}) - - showAlert(translate("glasses:connectToGlassesTitle"), message, [ - { - text: translate("common:ok"), - onPress: async () => { - console.log("[GallerySyncService] User acknowledged WiFi explanation") - // Mark as explained so we don't show again - settingsStore.setSetting(SETTINGS.gallery_sync_explained.key, true, false) - // Android: the native WiFi connect call freezes the UI thread (screen dims - // instantly on Samsung), leaving the dialog mid-fade. Wait for the fade to - // complete before resolving so the dialog fully dismisses first. - if (Platform.OS === "android") { - await new Promise((r) => setTimeout(r, 150)) - } - resolve(true) - }, - }, - ]) + return new Promise((resolve) => { + const proceed = () => { + // Mark as explained so we don't show again. + settingsStore.setSetting(SETTINGS.gallery_sync_explained.key, true, false) + resolve(true) + } + // Emit the one-time wifi-join explanation as a notice the host renders + acknowledges. + // The host owns the alert + its dismiss timing (Android's native WiFi connect freezes + // the UI thread, so it should let the dialog fully dismiss before calling ack). + const delivered = emitGalleryNotice({ + code: "connect_to_glasses", + data: {ssid, platform: Platform.OS}, + ack: proceed, + }) + // No host listening (e.g. an OEM that hasn't wired onNotice) — don't hang the sync. + if (delivered === 0) proceed() }) } @@ -1576,7 +1542,6 @@ class GallerySyncService { } // Enqueue for background processing (non-blocking) - const _isPhoto = downloadedFile.name?.match(/\.(jpg|jpeg|png)$/i) const isVideo = downloadedFile.name?.match(/\.(mp4|mov)$/i) const leaf = downloadedFile.name?.includes("/") ? downloadedFile.name.substring(downloadedFile.name.lastIndexOf("/") + 1) @@ -1951,98 +1916,6 @@ class GallerySyncService { } } - /** - * Auto-save downloaded files to camera roll - * - * ⚠️ DEPRECATED: This method is no longer used. Photos are now saved to camera roll - * immediately after each download completes (see executeDownload method). - * - * NOTE: Files now download in chronological order (oldest first), so the immediate-save - * approach will also save them in chronological order to the system gallery. - */ - private async autoSaveToCameraRoll(downloadedFiles: PhotoInfo[]): Promise { - const shouldAutoSave = await gallerySettingsService.getAutoSaveToCameraRoll() - if (!shouldAutoSave || downloadedFiles.length === 0) return - - console.log( - `[GallerySyncService] Auto-saving ${downloadedFiles.length} files to camera roll in chronological order...`, - ) - - const hasPermission = await MediaLibraryPermissions.checkPermission() - if (!hasPermission) { - const granted = await MediaLibraryPermissions.requestPermission() - if (!granted) { - console.warn("[GallerySyncService] Camera roll permission denied") - return - } - } - - // CRITICAL: Sort all downloaded files by capture time BEFORE saving to gallery - // This ensures gallery displays them in chronological order, not download order - // (photos download first by size, videos second, but we want chronological capture order) - const sortedFiles = [...downloadedFiles].sort((a, b) => { - // Parse capture timestamps - handle both string and number formats - // Use Number.MAX_SAFE_INTEGER for invalid/missing timestamps to push them to the end - const parseTime = (modified: string | number | undefined): number => { - if (modified === undefined || modified === null) return Number.MAX_SAFE_INTEGER - if (typeof modified === "number") return isNaN(modified) ? Number.MAX_SAFE_INTEGER : modified - const parsed = parseInt(modified, 10) - return isNaN(parsed) ? Number.MAX_SAFE_INTEGER : parsed - } - - const timeA = parseTime(a.modified) - const timeB = parseTime(b.modified) - - // Sort oldest first (ascending) so they're added to gallery in chronological order - return timeA - timeB - }) - - console.log(`[GallerySyncService] Sorted ${sortedFiles.length} files by capture time:`) - sortedFiles.slice(0, 5).forEach((file, idx) => { - const captureTime = typeof file.modified === "string" ? parseInt(file.modified, 10) : file.modified || 0 - const captureDate = new Date(captureTime) - const fileType = file.is_video ? "video" : "photo" - console.log(` ${idx + 1}. ${file.name} - ${captureDate.toISOString()} (${fileType})`) - }) - if (sortedFiles.length > 5) { - console.log(` ... and ${sortedFiles.length - 5} more files`) - } - - let savedCount = 0 - let failedCount = 0 - - // Save files in chronological order (oldest first) - for (const photoInfo of sortedFiles) { - const filePath = photoInfo.filePath || localStorageService.getPhotoFilePath(photoInfo.name) - - // Parse the capture timestamp from the photo metadata - // The 'modified' field contains the original capture time from the glasses - let captureTime: number | undefined - if (photoInfo.modified) { - captureTime = typeof photoInfo.modified === "string" ? parseInt(photoInfo.modified, 10) : photoInfo.modified - if (isNaN(captureTime)) { - console.warn(`[GallerySyncService] Invalid modified timestamp for ${photoInfo.name}:`, photoInfo.modified) - captureTime = undefined - } - } - - // Save to camera roll with capture time for logging - const success = await MediaLibraryPermissions.saveToLibrary(filePath, captureTime) - if (success) { - savedCount++ - } else { - failedCount++ - } - } - - console.log( - `[GallerySyncService] Saved ${savedCount}/${sortedFiles.length} files to camera roll in chronological order`, - ) - if (failedCount > 0) { - console.warn(`[GallerySyncService] Failed to save ${failedCount} files to camera roll`) - } - } - /** * Handle sync completion */ diff --git a/mobile/src/services/asg/localStorageService.ts b/mobile/modules/island/src/services/asg/localStorageService.ts similarity index 99% rename from mobile/src/services/asg/localStorageService.ts rename to mobile/modules/island/src/services/asg/localStorageService.ts index f7f168b913..4f1b7fca92 100644 --- a/mobile/src/services/asg/localStorageService.ts +++ b/mobile/modules/island/src/services/asg/localStorageService.ts @@ -5,9 +5,9 @@ import * as RNFS from "@dr.pogodin/react-native-fs" -import {PhotoInfo} from "@/types/asg" -import {BgTimer} from "@mentra/island" -import {storage} from "@/utils/storage" +import {PhotoInfo} from "../../types/asg" +import {BgTimer} from "../../utils/timers" +import {storage} from "../../utils/storage" export interface DownloadedFile { name: string diff --git a/mobile/src/services/asg/mediaProcessingQueue.ts b/mobile/modules/island/src/services/asg/mediaProcessingQueue.ts similarity index 97% rename from mobile/src/services/asg/mediaProcessingQueue.ts rename to mobile/modules/island/src/services/asg/mediaProcessingQueue.ts index 9f4eda5a73..4f726aa522 100644 --- a/mobile/src/services/asg/mediaProcessingQueue.ts +++ b/mobile/modules/island/src/services/asg/mediaProcessingQueue.ts @@ -8,12 +8,12 @@ import * as RNFS from "@dr.pogodin/react-native-fs" import CrustModule from "@mentra/crust" -import {asgCameraApi} from "@/services/asg/asgCameraApi" -import {localStorageService} from "@/services/asg/localStorageService" -import {INVALID_DOWNLOADED_MEDIA, validateDownloadedMediaFile} from "@/services/asg/galleryMediaValidation" -import {useGallerySyncStore} from "@/stores/gallerySync" -import {BgTimer} from "@mentra/island" -import {MediaLibraryPermissions} from "@/utils/permissions/MediaLibraryPermissions" +import {asgCameraApi} from "./asgCameraApi" +import {localStorageService} from "./localStorageService" +import {INVALID_DOWNLOADED_MEDIA, validateDownloadedMediaFile} from "./galleryMediaValidation" +import {useGallerySyncStore} from "../../stores/gallerySync" +import {BgTimer} from "../../utils/timers" +import {MediaLibraryPermissions} from "../../utils/permissions/MediaLibraryPermissions" const TAG = "[MediaProcessingQueue]" diff --git a/mobile/modules/island/src/services/asgOtaVersionUrl.ts b/mobile/modules/island/src/services/asgOtaVersionUrl.ts new file mode 100644 index 0000000000..9d20d68d7c --- /dev/null +++ b/mobile/modules/island/src/services/asgOtaVersionUrl.ts @@ -0,0 +1,45 @@ +import {SETTINGS, useSettingsStore} from "../stores/settings" + +// Mirrors @/config/ota OTA_VERSION_URL_PROD. Inlined here so island owns its OTA +// manifest resolution without reaching back into the host config. Production +// remains the MentraOS compiled fallback because current production glasses +// either advertise that URL or, on older builds, ignore the ota_start manifest +// override and install from their compiled default. +const OTA_VERSION_URL_PROD = "https://ota.mentraglass.com/prod_live_version.json" + +function isLegacyAsgOtaStartBuild(glassesBuildNumber?: string | null): boolean { + const buildNumber = Number.parseInt(glassesBuildNumber ?? "", 10) + // Pre-wall-clock ASG builds ignore ota_start.ota_version_url, so compare against the URL they will actually use. + return Number.isFinite(buildNumber) && buildNumber < 100000 +} + +function getOtaVersionUrlDevOverride(): string | null { + // Super mode only: a wrong OTA manifest can brick glasses, so a saved + // override is inert unless super mode is currently enabled. + if (!useSettingsStore.getState().getSetting(SETTINGS.super_mode.key)) { + return null + } + const value = useSettingsStore.getState().getSetting(SETTINGS.ota_version_url.key) + const trimmed = typeof value === "string" ? value.trim() : "" + return trimmed || null +} + +export function getAsgOtaVersionUrl(glassesUrl?: string | null, glassesBuildNumber?: string | null): string { + const deviceUrl = glassesUrl?.trim() + if (isLegacyAsgOtaStartBuild(glassesBuildNumber)) { + // Legacy glasses ignore ota_start.ota_version_url and install from their compiled + // default, so the developer override does not apply to them either. + return deviceUrl || OTA_VERSION_URL_PROD + } + + const devOverrideUrl = getOtaVersionUrlDevOverride() + if (devOverrideUrl) { + return devOverrideUrl + } + + const envUrl = process.env.EXPO_PUBLIC_ASG_OTA_VERSION_URL?.trim() + if (envUrl) { + return envUrl + } + return deviceUrl || OTA_VERSION_URL_PROD +} diff --git a/mobile/src/services/streaming/cloudStreamApi.ts b/mobile/modules/island/src/services/cloudStreamApi.ts similarity index 98% rename from mobile/src/services/streaming/cloudStreamApi.ts rename to mobile/modules/island/src/services/cloudStreamApi.ts index 628f8d85aa..de4cce8004 100644 --- a/mobile/src/services/streaming/cloudStreamApi.ts +++ b/mobile/modules/island/src/services/cloudStreamApi.ts @@ -14,7 +14,7 @@ * "cloud-rest" so a miniapp sees exactly where and on which leg it broke. */ -import {cloudClient} from "@/services/cloudClient" +import {cloudClientService as cloudClient} from "./CloudClientService" /** * Restream destination. The full stream key is part of the URL — Cloudflare diff --git a/mobile/src/services/asg/gallerySyncClock.ts b/mobile/modules/island/src/services/gallerySyncClock.ts similarity index 100% rename from mobile/src/services/asg/gallerySyncClock.ts rename to mobile/modules/island/src/services/gallerySyncClock.ts diff --git a/mobile/src/services/asg/glassesClockSync.ts b/mobile/modules/island/src/services/glassesClockSync.ts similarity index 95% rename from mobile/src/services/asg/glassesClockSync.ts rename to mobile/modules/island/src/services/glassesClockSync.ts index 0e6f99cd6a..2c27d3a822 100644 --- a/mobile/src/services/asg/glassesClockSync.ts +++ b/mobile/modules/island/src/services/glassesClockSync.ts @@ -2,10 +2,9 @@ * Push phone time to glasses only when clock skew is detected (gallery sync, OTA, etc.). */ -import BluetoothSdk from "@mentra/bluetooth-sdk-internal" -import {BgTimer} from "@mentra/island" - -import {useGlassesStore} from "@/stores/glasses" +import BluetoothSdk from "../../../bluetooth-sdk/build/_internal" +import {BgTimer} from "../utils/timers" +import {useGlassesStore} from "../stores/glasses" import {getAsgOtaVersionUrl} from "./asgOtaVersionUrl" import {detectClockSkew} from "./gallerySyncClock" diff --git a/mobile/src/services/navigation/roadNameResolver.ts b/mobile/modules/island/src/services/navigation/roadNameResolver.ts similarity index 100% rename from mobile/src/services/navigation/roadNameResolver.ts rename to mobile/modules/island/src/services/navigation/roadNameResolver.ts diff --git a/mobile/src/services/navigation/routesApiCodec.ts b/mobile/modules/island/src/services/navigation/routesApiCodec.ts similarity index 100% rename from mobile/src/services/navigation/routesApiCodec.ts rename to mobile/modules/island/src/services/navigation/routesApiCodec.ts diff --git a/mobile/src/utils/otaLegacyMapping.ts b/mobile/modules/island/src/services/otaLegacyMapping.ts similarity index 96% rename from mobile/src/utils/otaLegacyMapping.ts rename to mobile/modules/island/src/services/otaLegacyMapping.ts index e1d634efcf..86655fd643 100644 --- a/mobile/src/utils/otaLegacyMapping.ts +++ b/mobile/modules/island/src/services/otaLegacyMapping.ts @@ -1,4 +1,4 @@ -import type {OtaProgress, OtaProgressStatus, OtaStatus} from "@mentra/bluetooth-sdk-internal" +import type {OtaProgress, OtaProgressStatus, OtaStatus} from "../../../bluetooth-sdk/build/_internal" /** Normalized ota_status fields (snake_case) after bridging Core/expo (snake or camel). */ export type NormalizedOtaStatusEvent = { diff --git a/mobile/modules/island/src/stores/apps.ts b/mobile/modules/island/src/stores/apps.ts index 0b4fc7d001..9eeef251fe 100644 --- a/mobile/modules/island/src/stores/apps.ts +++ b/mobile/modules/island/src/stores/apps.ts @@ -1,18 +1,13 @@ /** - * Island apps store — runtime state of installed and remote applets. + * Island apps store — runtime state of installed and local applets. * * The OEM-facing API: subscribe to the running set, install/uninstall * miniapps, start/stop them. The store delegates install plumbing to * AppRegistry, and exposes hooks (`useApps`, `useStart`, `useStop`, * `useRefresh`, `useStopAll`) that the host UI can read. * - * Side-effects the host needs (cloud REST calls, navigation, alerts) are - * injected via `configureIsland`. The store invokes those hooks at the - * right moments — but never imports them directly. - * * Source of `apps`: * - Local: appRegistry.getInstalledMiniapps() (always) - * - Extra: hostHooks.loadExtraApps?.() (e.g. cloud applets) */ import {useMemo} from "react" @@ -20,14 +15,15 @@ import {AsyncResult, Result, result as Res} from "typesafe-ts" import {create} from "zustand" import type {ClientApp} from "../types/applet" -import type {Capabilities} from "../types/hardware" import {DeviceTypes} from "../types/enums" import {getModelCapabilities} from "../types/hardware" import {HardwareCompatibility} from "../utils/hardware/hardware" import {storage} from "../utils/storage/storage" import appRegistry from "../services/AppRegistry" +import {islandNotifications} from "../services/NotificationsEmitter" import {miniappLauncher} from "../services/MiniappLauncher" import {miniappRunningRegistry} from "../services/MiniappRunningRegistry" +import {SETTINGS, useSettingsStore} from "./settings" import BluetoothSdk from "@mentra/bluetooth-sdk" // --------------------------------------------------------------------------- @@ -38,27 +34,6 @@ export interface StartOptions { skipNavigation?: boolean } -export interface IslandHostHooks { - /** Return host-provided extra apps (e.g. cloud applets). Called on every refresh. */ - loadExtraApps?: () => Promise - /** Return the connected device's capabilities for compatibility checks. */ - getCapabilities?: () => Capabilities | null - /** Called by start() before applet.onStart. Return false to abort the start. */ - beforeStart?: (app: ClientApp, opts?: StartOptions) => Promise | boolean - /** Called by stop() before applet.onStop. */ - beforeStop?: (app: ClientApp) => Promise | void - /** Called by uninstall() before appRegistry.uninstall — e.g. for cloud-side cleanup. */ - onUninstall?: (app: ClientApp) => Promise | void - /** Called after the apps array is rebuilt — host can mutate / re-sort. */ - postProcessApps?: (apps: ClientApp[]) => ClientApp[] | Promise -} - -let hostHooks: IslandHostHooks = {} - -export function configureIsland(hooks: IslandHostHooks): void { - hostHooks = {...hostHooks, ...hooks} -} - // --------------------------------------------------------------------------- // Store // --------------------------------------------------------------------------- @@ -172,15 +147,13 @@ const startStopApp = async (app: ClientApp, status: boolean): Promise => { } /** - * Build the final `apps` array for the store from the two source lists - * (local + cloud). Pure — same inputs → same outputs. Used by `refresh` - * to emit twice (local-only, then merged) without duplicating the - * dedupe/carry-over/compat/hidden/postProcess pipeline. + * Build the final `apps` array for the store from the installed/offline app + * source. Pure — same inputs -> same outputs. */ -function projectApps(previousState: AppStatusState, localApps: ClientApp[], extraApps: ClientApp[]): ClientApp[] { - // Dedupe by packageName, keep first occurrence (extra/cloud wins). +function projectApps(previousState: AppStatusState, localApps: ClientApp[]): ClientApp[] { + // Dedupe by packageName, keep first occurrence. const byPackage = new Map() - for (const app of [...extraApps, ...localApps]) { + for (const app of localApps) { if (!byPackage.has(app.packageName)) byPackage.set(app.packageName, app) } @@ -190,7 +163,10 @@ function projectApps(previousState: AppStatusState, localApps: ClientApp[], extr previousByPackage.set(oldApp.packageName, oldApp) } - const capabilities = hostHooks.getCapabilities?.() ?? getModelCapabilities(DeviceTypes.NONE) + const defaultWearable = + (useSettingsStore.getState().getSetting(SETTINGS.default_wearable.key) as DeviceTypes | undefined) || + DeviceTypes.NONE + const capabilities = getModelCapabilities(defaultWearable) // Single pass: dedupe → screenshot carry-over → compat → hidden, all into // fresh objects. The previous in-place mutation re-used object references @@ -221,55 +197,12 @@ export const useAppStatusStore = create((set, get) => ({ foregroundedPackage: null, refresh: async () => { - // Two-pass: local apps first (fast, no network), then merge cloud - // applets when they arrive. Without this the home tray waits for - // `loadExtraApps` to return — and when cloud is slow/503 (e.g. on - // a fresh boot), the just-installed dev miniapp takes 10+ seconds - // to show up because the local entry is held back behind the cloud - // fetch. - // - // Pass 1 carries over the PREVIOUS snapshot's cloud apps so the - // tray doesn't flicker — empty cloud list on first render would - // blank out tiles for a frame before pass 2 re-merges them. On - // first-ever refresh (state.apps is empty) we skip pass 1 entirely - // and let pass 2 own the only emit; there are no rendered tiles to - // flicker yet. const previousState = get() const localApps = await appRegistry.getInstalledMiniapps() - const hasPriorSnapshot = previousState.apps.length > 0 - if (hasPriorSnapshot) { - const previousCloudApps = previousState.apps.filter((a) => !a.local) - let pass1 = projectApps(previousState, localApps, previousCloudApps) - if (hostHooks.postProcessApps) { - try { - pass1 = await hostHooks.postProcessApps(pass1) - } catch (e) { - console.warn("ISLAND: postProcessApps threw on local-only pass:", e) - } - } - set({apps: pass1}) - } - - // Pass 2: fetch cloud applets, merge, re-emit. - let extraApps: ClientApp[] = [] - try { - extraApps = (await hostHooks.loadExtraApps?.()) ?? [] - } catch { - // Cloud failures shouldn't blow away the local list we just published. - extraApps = [] - } - let pass2 = projectApps(get(), localApps, extraApps) - if (hostHooks.postProcessApps) { - try { - pass2 = await hostHooks.postProcessApps(pass2) - } catch (e) { - console.warn("ISLAND: postProcessApps threw on merged pass:", e) - } - } - set({apps: pass2}) + set({apps: projectApps(previousState, localApps)}) }, - start: async (clientApp: ClientApp, opts?: StartOptions) => { + start: async (clientApp: ClientApp, _opts?: StartOptions) => { const state = get() const packageName = clientApp.packageName const app = state.apps.find((a) => a.packageName === packageName) @@ -284,10 +217,22 @@ export const useAppStatusStore = create((set, get) => ({ return false } - // Host gate (incompatible alerts, offline-mode rejection, etc.). - if (hostHooks.beforeStart) { - const proceed = await hostHooks.beforeStart(app, opts) - if (!proceed) return false + if (!app.offline && !app.local) { + console.warn(`ISLAND: cloud-v1 app entries are no longer supported: ${packageName}`) + return false + } + + // Island-native incompatibility gate. Block the launch and raise a structured + // notification the host can render off toolkit.notifications. + if (app.compatibility?.isCompatible === false) { + islandNotifications.emit({ + kind: "version_incompatible", + packageName, + reason: `${app.name ?? packageName} is not compatible with the connected glasses`, + metadata: {missingRequired: app.compatibility.missingRequired ?? []}, + timestamp: Date.now(), + }) + return false } // Foreground-only-one rule: stop other running standard apps. @@ -300,9 +245,8 @@ export const useAppStatusStore = create((set, get) => ({ } } - const shouldLoad = !app.offline && !app.local set((s) => ({ - apps: s.apps.map((a) => (a.packageName === packageName ? {...a, running: true, loading: shouldLoad} : a)), + apps: s.apps.map((a) => (a.packageName === packageName ? {...a, running: true, loading: false} : a)), })) saveLastOpenTime(packageName) @@ -344,14 +288,9 @@ export const useAppStatusStore = create((set, get) => ({ return } - if (hostHooks.beforeStop) { - await hostHooks.beforeStop(app) - } - - const shouldLoad = !app.offline && !app.local set((s) => ({ apps: s.apps.map((a) => - a.packageName === packageName ? {...a, running: false, screenshot: undefined, loading: shouldLoad} : a, + a.packageName === packageName ? {...a, running: false, screenshot: undefined, loading: false} : a, ), })) @@ -363,10 +302,8 @@ export const useAppStatusStore = create((set, get) => ({ } await startStopApp(app, false) - // Tear down the background JS context for local miniapps. Previously the - // host's beforeStop hook (MiniappCatalog) called router.unregister; that - // now flows through the launcher so lifecycle lives in one place. No-op for - // native offline built-ins / cloud apps (no JS context). + // Tear down the background JS context for local miniapps. No-op for native + // offline built-ins. if (app.local) { try { await miniappLauncher.stop(packageName) @@ -426,12 +363,6 @@ export const useAppStatusStore = create((set, get) => ({ uninstall: (packageName, version) => { return Res.try_async(async () => { - if (hostHooks.onUninstall) { - const app = get().apps.find((a) => a.packageName === packageName) - if (app) { - await hostHooks.onUninstall(app) - } - } const res = await appRegistry.uninstall(packageName, version) if (res.is_error()) throw res.error set((s) => ({apps: s.apps.filter((a) => a.packageName !== packageName)})) @@ -496,6 +427,14 @@ appRegistry.subscribe(() => { void useAppStatusStore.getState().refresh() }) +// Re-evaluate hardware compatibility when the paired/default wearable changes. +useSettingsStore.subscribe( + (state) => state.getSetting(SETTINGS.default_wearable.key), + () => { + void useAppStatusStore.getState().refresh() + }, +) + // --------------------------------------------------------------------------- // Public hooks // --------------------------------------------------------------------------- diff --git a/mobile/modules/island/src/stores/cloudClientStatus.ts b/mobile/modules/island/src/stores/cloudClientStatus.ts new file mode 100644 index 0000000000..6512a08f25 --- /dev/null +++ b/mobile/modules/island/src/stores/cloudClientStatus.ts @@ -0,0 +1,37 @@ +/** + * Cloud-client runtime status store — moved into island so island owns the + * cloud-v2 runtime status (connection status + audio transport). Re-exported + * through the host's `@/stores/cloudClientStatus` shim so the app keeps its + * imports, and surfaced as `toolkit.stores.cloudClientStatus`. + */ +import {create} from "zustand" + +import type {RuntimeAudioTransport, RuntimeSnapshot, RuntimeStatus} from "@mentra/cloud-client/react-native" + +const initialSnapshot: RuntimeSnapshot = { + status: "disconnected", + audioTransport: "none", +} + +export interface CloudClientStatusState extends RuntimeSnapshot { + lastChangedAt: Date | null + setSnapshot: (snapshot: RuntimeSnapshot) => void + reset: () => void +} + +export const useCloudClientStatusStore = create((set) => ({ + ...initialSnapshot, + lastChangedAt: null, + setSnapshot: (snapshot) => + set({ + ...snapshot, + lastChangedAt: new Date(), + }), + reset: () => + set({ + ...initialSnapshot, + lastChangedAt: null, + }), +})) + +export type {RuntimeAudioTransport, RuntimeSnapshot, RuntimeStatus} diff --git a/mobile/modules/island/src/stores/connection.ts b/mobile/modules/island/src/stores/connection.ts new file mode 100644 index 0000000000..d1168b846d --- /dev/null +++ b/mobile/modules/island/src/stores/connection.ts @@ -0,0 +1,68 @@ +import {create} from "zustand" + +// Relocated from the host @/services/ws-types so this store is self-contained +// inside island. The host @/services/ws-types now re-exports this (a shim), so its +// other importers are unchanged. +export enum WebSocketStatus { + DISCONNECTED = "disconnected", + CONNECTING = "connecting", + CONNECTED = "connected", + ERROR = "error", +} + +export interface ConnectionState { + status: WebSocketStatus + url: string | null + error: string | null + lastConnectedAt: Date | null + lastDisconnectedAt: Date | null + reconnectAttempts: number + + setStatus: (status: WebSocketStatus) => void + setUrl: (url: string | null) => void + setError: (error: string | null) => void + incrementReconnectAttempts: () => void + resetReconnectAttempts: () => void + reset: () => void +} + +export const useConnectionStore = create((set) => ({ + status: WebSocketStatus.DISCONNECTED, + url: null, + error: null, + lastConnectedAt: null, + lastDisconnectedAt: null, + reconnectAttempts: 0, + + setStatus: (status) => + set((state) => ({ + status, + error: status === WebSocketStatus.ERROR ? state.error : null, + lastConnectedAt: status === WebSocketStatus.CONNECTED ? new Date() : state.lastConnectedAt, + lastDisconnectedAt: + status === WebSocketStatus.DISCONNECTED || status === WebSocketStatus.ERROR + ? new Date() + : state.lastDisconnectedAt, + })), + + setUrl: (url) => set({url}), + + setError: (error) => set({error, status: WebSocketStatus.ERROR, lastDisconnectedAt: new Date()}), + + incrementReconnectAttempts: () => + set((state) => ({ + reconnectAttempts: state.reconnectAttempts + 1, + })), + + resetReconnectAttempts: () => set({reconnectAttempts: 0}), + + reset: () => + set({ + status: WebSocketStatus.DISCONNECTED, + url: null, + error: null, + lastConnectedAt: null, + lastDisconnectedAt: null, + reconnectAttempts: 0, + }), +})) diff --git a/mobile/modules/island/src/stores/core.ts b/mobile/modules/island/src/stores/core.ts new file mode 100644 index 0000000000..e367193ba8 --- /dev/null +++ b/mobile/modules/island/src/stores/core.ts @@ -0,0 +1,32 @@ +import {create} from "zustand" +import {subscribeWithSelector} from "zustand/middleware" +import type {BluetoothStatus} from "../../../bluetooth-sdk/build/_internal" + +export interface CoreState extends BluetoothStatus { + setCoreInfo: (info: Partial) => void + reset: () => void +} + +const initialState: BluetoothStatus = { + // state: + searching: false, + searchingController: false, + micRanking: ["glasses", "phone", "bluetooth", "bluetoothClassic"], + systemMicUnavailable: false, + currentMic: null, + searchResults: [], + wifiScanResults: [], + lastLog: [], + otherBtConnected: false, + galleryModeEnabled: true, +} + +export const useCoreStore = create()( + subscribeWithSelector((set) => ({ + ...initialState, + + setCoreInfo: (info) => set((state) => ({...state, ...info})), + + reset: () => set(initialState), + })), +) diff --git a/mobile/modules/island/src/stores/display.ts b/mobile/modules/island/src/stores/display.ts new file mode 100644 index 0000000000..d5c4930de9 --- /dev/null +++ b/mobile/modules/island/src/stores/display.ts @@ -0,0 +1,96 @@ +import {create} from "zustand" +import {subscribeWithSelector} from "zustand/middleware" + +// Inlined from the host @/utils/e2eMetrics (both are pure) so this store is +// self-contained inside island. Emits the same `E2E_METRIC` console lines the +// e2e harness reads; the host util keeps emitting its non-display metrics. +const E2E_METRICS_ENABLED = process.env.EXPO_PUBLIC_ENABLE_E2E_METRICS === "true" + +function logE2EMetric(event: string, payload: Record = {}): void { + if (!E2E_METRICS_ENABLED) return + try { + console.log(`E2E_METRIC ${JSON.stringify({event, ts_ms: Date.now(), ...payload})}`) + } catch (error) { + console.warn("E2E_METRIC: failed to serialize payload", error) + } +} + +function extractDisplayText(displayEvent: any): string[] { + const layout = displayEvent?.layout + if (!layout || typeof layout !== "object") { + return [] + } + switch (layout.layoutType) { + case "text_wall": + case "text_line": + return typeof layout.text === "string" ? layout.text.split("\n") : [] + case "double_text_wall": + return [layout.topText, layout.bottomText].filter((value): value is string => typeof value === "string") + case "text_rows": + return Array.isArray(layout.text) + ? layout.text.filter((value: unknown): value is string => typeof value === "string") + : [] + default: + return [] + } +} + +export interface DisplayStore { + currentEvent: any + dashboardEvent: any + mainEvent: any + setDisplayEvent: (eventString: string) => void + view: string + setView: (view: string) => void +} + +export const useDisplayStore = create()( + subscribeWithSelector((set, get) => ({ + currentEvent: {} as any, + dashboardEvent: {} as any, + mainEvent: {} as any, + view: "main", + setDisplayEvent: (eventString: string) => { + const event = JSON.parse(eventString) + const currentView = get().view + const targetBucket = event.view === "dashboard" ? "dashboardEvent" : "mainEvent" + + const updates: any = { + [targetBucket]: event, + } + + // also update the current event if the view is the same: + if (event.view === currentView) { + updates.currentEvent = event + } + + const visibleEvent = updates.currentEvent ?? event + const textLines = extractDisplayText(visibleEvent) + if (textLines.some((line) => line.trim() !== "")) { + logE2EMetric("display_store_update", { + view: visibleEvent.view ?? currentView, + layout_type: visibleEvent.layout?.layoutType ?? "", + text_lines: textLines, + }) + } + + set(updates) + }, + setView: (view: string) => { + const currentView = get().view + if (view === currentView) { + return + } + + // update the view and the currentEvent with the corresponding event: + let newEvent + if (view === "dashboard") { + newEvent = get().dashboardEvent + } else { + newEvent = get().mainEvent + } + logE2EMetric("display_view_changed", {view}) + set({view, currentEvent: newEvent}) + }, + })), +) diff --git a/mobile/modules/island/src/stores/gallerySync.ts b/mobile/modules/island/src/stores/gallerySync.ts new file mode 100644 index 0000000000..533003ceab --- /dev/null +++ b/mobile/modules/island/src/stores/gallerySync.ts @@ -0,0 +1,380 @@ +/** + * Gallery Sync Store + * Manages gallery sync state independently of UI lifecycle + */ + +import {create} from "zustand" +import {subscribeWithSelector} from "zustand/middleware" + +// Relocated from the host @/types/asg so this store is self-contained inside +// island. The host barrel re-exports this (a shim), so its importers are unchanged. +export interface PhotoInfo { + name: string + url: string + download: string + size: number + modified: string | number // Unix timestamp (milliseconds) - can be string or number from API + mime_type?: string + is_video?: boolean + thumbnail_data?: string + downloaded_at?: number + // Video duration in milliseconds (from glasses sync response) + duration?: number + // New fields for filesystem storage + filePath?: string + thumbnailPath?: string + // Glasses model that captured this media + glassesModel?: string +} + +// Sync state machine states +export type SyncState = + | "idle" + | "requesting_hotspot" + | "connecting_wifi" + | "syncing" + | "complete" + | "error" + | "cancelled" + +export interface HotspotInfo { + ssid: string + password: string + ip: string +} + +export interface SyncQueue { + files: PhotoInfo[] + currentIndex: number + startedAt: number + hotspotInfo: HotspotInfo +} + +export interface GallerySyncInfo { + // State machine + syncState: SyncState + + // Progress tracking + currentFile: string | null + currentFileProgress: number // 0-100 + completedFiles: number + totalFiles: number + failedFiles: string[] + + // Queue (persisted separately via localStorageService) + queue: PhotoInfo[] + queueIndex: number + + // Hotspot info + hotspotInfo: HotspotInfo | null + syncServiceOpenedHotspot: boolean + + // Gallery status from glasses + glassesPhotoCount: number + glassesVideoCount: number + glassesTotalCount: number + glassesHasContent: boolean + + // Error tracking + lastError: string | null +} + +export interface GallerySyncState extends GallerySyncInfo { + // State transitions + setSyncState: (state: SyncState) => void + setRequestingHotspot: () => void + setConnectingWifi: () => void + setSyncing: (files: PhotoInfo[]) => void + setSyncComplete: () => void + setSyncError: (error: string) => void + setSyncCancelled: () => void + + // Progress updates + setCurrentFile: (fileName: string | null, progress: number) => void + onFileProgress: (fileName: string, progress: number) => void + onFileComplete: (fileName: string) => void + onFileFailed: (fileName: string, error?: string) => void + onFileProcessing: (fileName: string) => void + onFileProcessed: (fileName: string) => void + updateFileInQueue: (fileName: string, updatedFile: PhotoInfo) => void + removeFilesFromQueue: (fileNames: string[]) => void + + // Processing queue tracking + processingFiles: Set + processedFiles: number + + // Hotspot management + setHotspotInfo: (info: HotspotInfo | null) => void + setSyncServiceOpenedHotspot: (opened: boolean) => void + + // Gallery status from glasses + setGlassesGalleryStatus: (photos: number, videos: number, total: number, hasContent: boolean) => void + clearGlassesGalleryStatus: () => void + + // Queue management (for resume) + setQueue: (files: PhotoInfo[], startIndex?: number) => void + advanceQueue: () => void + clearQueue: () => void + + // Full reset + reset: () => void +} + +const initialState: GallerySyncInfo & {processingFiles: Set; processedFiles: number} = { + syncState: "idle", + currentFile: null, + currentFileProgress: 0, + completedFiles: 0, + totalFiles: 0, + failedFiles: [], + queue: [], + queueIndex: 0, + hotspotInfo: null, + syncServiceOpenedHotspot: false, + glassesPhotoCount: 0, + glassesVideoCount: 0, + glassesTotalCount: 0, + glassesHasContent: false, + lastError: null, + processingFiles: new Set(), + processedFiles: 0, +} + +export const useGallerySyncStore = create()( + subscribeWithSelector((set, get) => ({ + ...initialState, + + // State transitions + setSyncState: (syncState: SyncState) => set({syncState}), + + setRequestingHotspot: () => + set({ + syncState: "requesting_hotspot", + lastError: null, + failedFiles: [], + }), + + setConnectingWifi: () => + set({ + syncState: "connecting_wifi", + }), + + setSyncing: (files: PhotoInfo[]) => + set({ + syncState: "syncing", + // C4: Strip thumbnail_data (base64) from store to prevent OOM + queue: files.map(({thumbnail_data: _thumbnailData, ...rest}) => rest), + queueIndex: 0, + totalFiles: files.length, + completedFiles: 0, + currentFile: files.length > 0 ? files[0].name : null, + currentFileProgress: 0, + failedFiles: [], + lastError: null, + processedFiles: 0, + processingFiles: new Set(), + }), + + setSyncComplete: () => + set({ + syncState: "complete", + currentFile: null, + currentFileProgress: 0, + // Keep queue intact so photos remain visible after sync + // Don't clear: queue: [], queueIndex: 0 + }), + + setSyncError: (error: string) => + set({ + syncState: "error", + lastError: error, + currentFile: null, + currentFileProgress: 0, + }), + + setSyncCancelled: () => + set({ + syncState: "cancelled", + currentFile: null, + currentFileProgress: 0, + queue: [], + queueIndex: 0, + }), + + // Progress updates + setCurrentFile: (fileName: string | null, progress: number) => + set({ + currentFile: fileName, + currentFileProgress: Math.max(0, Math.min(100, progress)), + }), + + onFileProgress: (fileName: string, progress: number) => { + const state = get() + const clampedProgress = Math.max(0, Math.min(100, progress)) + // Throttle: skip update if same file and same percentage (prevents Zustand flood) + if (clampedProgress === state.currentFileProgress && fileName === state.currentFile) { + return + } + set({ + currentFile: fileName, + currentFileProgress: clampedProgress, + }) + }, + + onFileComplete: (_fileName: string) => { + const state = get() + const newCompletedFiles = state.completedFiles + 1 + const newQueueIndex = state.queueIndex + 1 + const nextFile = state.queue[newQueueIndex] + + set({ + completedFiles: newCompletedFiles, + queueIndex: newQueueIndex, + currentFile: nextFile?.name || null, + currentFileProgress: 0, + }) + }, + + onFileFailed: (fileName: string, _error?: string) => { + const state = get() + const newQueueIndex = state.queueIndex + 1 + const nextFile = state.queue[newQueueIndex] + + set({ + failedFiles: [...state.failedFiles, fileName], + queueIndex: newQueueIndex, + currentFile: nextFile?.name || null, + currentFileProgress: 0, + }) + }, + + onFileProcessing: (fileName: string) => { + const state = get() + const newSet = new Set(state.processingFiles) + newSet.add(fileName) + set({processingFiles: newSet}) + }, + + onFileProcessed: (fileName: string) => { + const state = get() + const newSet = new Set(state.processingFiles) + newSet.delete(fileName) + set({processingFiles: newSet, processedFiles: state.processedFiles + 1}) + }, + + updateFileInQueue: (fileName: string, updatedFile: PhotoInfo) => { + const state = get() + const updatedQueue = state.queue.map((file) => (file.name === fileName ? updatedFile : file)) + set({queue: updatedQueue}) + }, + + removeFilesFromQueue: (fileNames: string[]) => { + if (fileNames.length === 0) return + + const filesToRemove = new Set(fileNames) + const state = get() + const failedFilesSet = new Set(state.failedFiles) + const filesBeforeQueueIndex = state.queue + .slice(0, state.queueIndex) + .filter((file) => filesToRemove.has(file.name)) + const removedBeforeQueueIndex = filesBeforeQueueIndex.length + const removedCompletedBeforeQueueIndex = filesBeforeQueueIndex.filter( + (file) => !failedFilesSet.has(file.name), + ).length + const filteredQueue = state.queue.filter((file) => !filesToRemove.has(file.name)) + const removedCount = state.queue.length - filteredQueue.length + + if (removedCount === 0) return + + const nextQueueIndex = state.queueIndex - removedBeforeQueueIndex + const currentFileRemoved = state.currentFile !== null && filesToRemove.has(state.currentFile) + + set({ + queue: filteredQueue, + totalFiles: state.totalFiles - removedCount, + completedFiles: Math.max(0, state.completedFiles - removedCompletedBeforeQueueIndex), + queueIndex: nextQueueIndex, + failedFiles: state.failedFiles.filter((fileName) => !filesToRemove.has(fileName)), + processingFiles: new Set(Array.from(state.processingFiles).filter((fileName) => !filesToRemove.has(fileName))), + currentFile: currentFileRemoved ? filteredQueue[nextQueueIndex]?.name || null : state.currentFile, + }) + }, + + // Hotspot management + setHotspotInfo: (info: HotspotInfo | null) => set({hotspotInfo: info}), + + setSyncServiceOpenedHotspot: (opened: boolean) => set({syncServiceOpenedHotspot: opened}), + + // Gallery status from glasses + setGlassesGalleryStatus: (photos: number, videos: number, total: number, hasContent: boolean) => { + const state = get() + // Reset to idle if sync is in a terminal state and new content is available + // This allows syncing again after taking new photos + const shouldResetToIdle = + hasContent && (state.syncState === "complete" || state.syncState === "error" || state.syncState === "cancelled") + + set({ + glassesPhotoCount: photos, + glassesVideoCount: videos, + glassesTotalCount: total, + glassesHasContent: hasContent, + ...(shouldResetToIdle ? {syncState: "idle" as SyncState} : {}), + }) + }, + + clearGlassesGalleryStatus: () => + set({ + glassesPhotoCount: 0, + glassesVideoCount: 0, + glassesTotalCount: 0, + glassesHasContent: false, + }), + + // Queue management + setQueue: (files: PhotoInfo[], startIndex: number = 0) => + set({ + // C4: Strip thumbnail_data (base64) from store to prevent OOM + queue: files.map(({thumbnail_data: _thumbnailData, ...rest}) => rest), + queueIndex: startIndex, + totalFiles: files.length, + completedFiles: startIndex, + }), + + advanceQueue: () => { + const state = get() + set({queueIndex: state.queueIndex + 1}) + }, + + clearQueue: () => + set({ + queue: [], + queueIndex: 0, + totalFiles: 0, + completedFiles: 0, + }), + + // Full reset + reset: () => set({...initialState, processingFiles: new Set()}), + })), +) + +// Selector helpers for common subscriptions +export const selectSyncProgress = (state: GallerySyncState) => ({ + syncState: state.syncState, + currentFile: state.currentFile, + currentFileProgress: state.currentFileProgress, + completedFiles: state.completedFiles, + totalFiles: state.totalFiles, + failedFiles: state.failedFiles, + processingFiles: state.processingFiles, +}) + +export const selectIssyncing = (state: GallerySyncState) => + state.syncState === "syncing" || state.syncState === "requesting_hotspot" || state.syncState === "connecting_wifi" + +export const selectGlassesGalleryStatus = (state: GallerySyncState) => ({ + photos: state.glassesPhotoCount, + videos: state.glassesVideoCount, + total: state.glassesTotalCount, + hasContent: state.glassesHasContent, +}) diff --git a/mobile/modules/island/src/stores/glasses.ts b/mobile/modules/island/src/stores/glasses.ts new file mode 100644 index 0000000000..c7e0985beb --- /dev/null +++ b/mobile/modules/island/src/stores/glasses.ts @@ -0,0 +1,309 @@ +import type { + GlassesConnectionStatus, + GlassesStatus, + HotspotStatus, + OtaProgress, + OtaStatus, + OtaUpdateInfo, + WifiStatus, +} from "../../../bluetooth-sdk/build/_internal" +import {isGlassesConnected, isGlassesLinkLayerBusy, isGlassesReady} from "../services/GlassesReadiness" +import {create} from "zustand" +import {subscribeWithSelector} from "zustand/middleware" + +// Re-export the connection predicates (single source of truth lives in +// GlassesReadiness) so importers of this store keep getting them. +export {isGlassesConnected, isGlassesLinkLayerBusy, isGlassesReady} + +export const selectGlassesConnected = (state: {connection: GlassesConnectionStatus}) => + isGlassesConnected(state.connection) + +export const selectGlassesReady = (state: {connection: GlassesConnectionStatus}) => isGlassesReady(state.connection) + +export interface GlassesState extends GlassesStatus { + systemTimeMs: number + wifiStatusKnown: boolean + setGlassesInfo: (info: GlassesInfoUpdate) => void + setBatteryInfo: (batteryLevel: number, charging: boolean, caseBatteryLevel: number, caseCharging: boolean) => void + setWifiInfo: (connected: boolean, ssid: string) => void + setHotspotInfo: (enabled: boolean, ssid: string, password: string, ip: string) => void + // OTA methods + otaStatus: OtaStatus | null + setOtaStatus: (status: OtaStatus | null) => void + setOtaUpdateAvailable: (info: OtaUpdateInfo | null) => void + setOtaProgress: (progress: OtaProgress | null) => void + setOtaInProgress: (inProgress: boolean) => void + setMtkUpdatedThisSession: (updated: boolean) => void + clearOtaState: () => void + reset: () => void + mtkUpdatedThisSession: boolean +} + +type LegacyWifiFields = { + wifiConnected?: boolean + wifiSsid?: string + wifiLocalIp?: string +} + +type LegacyHotspotFields = { + hotspotEnabled?: boolean + hotspotSsid?: string + hotspotPassword?: string + hotspotGatewayIp?: string + hotspotLocalIp?: string +} + +type GlassesInfoUpdate = Partial & LegacyWifiFields & LegacyHotspotFields + +function wifiFromLegacyFields(info: LegacyWifiFields): WifiStatus | null { + if (info.wifiConnected === true) { + const ssid = info.wifiSsid?.trim() + const localIp = info.wifiLocalIp?.trim() + return ssid ? {state: "connected", ssid, ...(localIp ? {localIp} : {})} : null + } + if (info.wifiConnected === false) { + return {state: "disconnected"} + } + return null +} + +function hotspotFromLegacyFields(info: LegacyHotspotFields): HotspotStatus | null { + if (info.hotspotEnabled === true) { + const ssid = info.hotspotSsid?.trim() + const password = info.hotspotPassword?.trim() + const localIp = (info.hotspotGatewayIp ?? info.hotspotLocalIp)?.trim() + return ssid && password && localIp ? {state: "enabled", ssid, password, localIp} : null + } + if (info.hotspotEnabled === false) { + return {state: "disabled"} + } + return null +} + +export const getGlasesInfoPartial = (state: GlassesStatus) => { + const wifi = state.wifi + const connected = isGlassesConnected(state.connection) + return { + batteryLevel: state.batteryLevel, + charging: state.charging, + caseBatteryLevel: state.caseBatteryLevel, + caseCharging: state.caseCharging, + connected, + wifiConnected: wifi.state === "connected", + wifiSsid: wifi.state === "connected" ? wifi.ssid : "", + deviceModel: state.deviceModel, + // Cloud GlassesInfo uses modelName, map from deviceModel so the cloud + // knows which device is connected when it receives connection state updates + modelName: state.deviceModel || null, + } +} + +interface GlassesStore extends GlassesStatus { + systemTimeMs: number + mtkUpdatedThisSession: boolean + wifiStatusKnown: boolean + otaStatus: OtaStatus | null +} + +const initialState: GlassesStore = { + // state: + connection: {state: "disconnected"}, + micEnabled: false, + bluetoothClassicConnected: false, + signalStrength: -1, + signalStrengthUpdatedAt: 0, + voiceActivityDetectionEnabled: true, + // device info + deviceModel: "", + androidVersion: "", + firmwareVersion: "", + bluetoothMacAddress: "", + leftMacAddress: "", + rightMacAddress: "", + buildNumber: "", + systemTimeMs: 0, + otaVersionUrl: "", + appVersion: "", + bluetoothName: "", + serialNumber: "", + style: "", + color: "", + mtkFirmwareVersion: "", + besFirmwareVersion: "", + // wifi info + wifi: {state: "disconnected"}, + wifiStatusKnown: false, + // battery info + batteryLevel: -1, + charging: false, + caseBatteryLevel: -1, + caseCharging: false, + caseOpen: false, + caseRemoved: true, + // hotspot info + hotspot: {state: "disabled"}, + // OTA update info + otaStatus: null, + otaUpdateAvailable: null, + otaProgress: null, + otaInProgress: false, + mtkUpdatedThisSession: false, + // ring: + controllerConnected: false, + controllerFullyBooted: false, + controllerMacAddress: "", + controllerBatteryLevel: -1, + controllerSignalStrength: -1, +} + +export const useGlassesStore = create()( + subscribeWithSelector((set) => ({ + ...initialState, + + setGlassesInfo: (info) => + set((state) => { + const { + wifiConnected, + wifiSsid, + wifiLocalIp, + hotspotEnabled, + hotspotSsid, + hotspotPassword, + hotspotGatewayIp, + hotspotLocalIp, + wifi, + hotspot, + ...sdkInfo + } = info + const wifiUpdate = wifi ?? wifiFromLegacyFields({wifiConnected, wifiSsid, wifiLocalIp}) + const hotspotUpdate = + hotspot ?? + hotspotFromLegacyFields({hotspotEnabled, hotspotSsid, hotspotPassword, hotspotGatewayIp, hotspotLocalIp}) + const hasWifiInfoUpdate = + Object.prototype.hasOwnProperty.call(info, "wifi") || + Object.prototype.hasOwnProperty.call(info, "wifiConnected") || + Object.prototype.hasOwnProperty.call(info, "wifiSsid") || + Object.prototype.hasOwnProperty.call(info, "wifiLocalIp") + const next = { + ...state, + ...sdkInfo, + ...(wifiUpdate ? {wifi: wifiUpdate} : {}), + ...(hotspotUpdate ? {hotspot: hotspotUpdate} : {}), + ...(hasWifiInfoUpdate ? {wifiStatusKnown: true} : {}), + } + if (!isGlassesConnected(next.connection)) { + next.wifiStatusKnown = false + } + return next + }), + + setBatteryInfo: (batteryLevel, charging, caseBatteryLevel, caseCharging) => + set({ + batteryLevel, + charging, + caseBatteryLevel, + caseCharging, + }), + + setWifiInfo: (connected, ssid) => + set(() => { + const trimmedSsid = ssid.trim() + if (connected && !trimmedSsid) { + return {} + } + const wifi: WifiStatus = connected ? {state: "connected", ssid: trimmedSsid} : {state: "disconnected"} + return { + wifi, + wifiStatusKnown: true, + } + }), + + setHotspotInfo: (enabled: boolean, ssid: string, password: string, ip: string) => + set(() => { + const hotspot = hotspotFromLegacyFields({ + hotspotEnabled: enabled, + hotspotSsid: ssid, + hotspotPassword: password, + hotspotGatewayIp: ip, + }) + return hotspot ? {hotspot} : {} + }), + + // OTA methods + setOtaStatus: (status: OtaStatus | null) => set({otaStatus: status}), + + setOtaUpdateAvailable: (info: OtaUpdateInfo | null) => set({otaUpdateAvailable: info}), + + setOtaProgress: (progress: OtaProgress | null) => + set((state) => { + const otaInProgress = progress !== null && progress.status !== "FINISHED" && progress.status !== "FAILED" + console.log("🔍 GLASSES STORE: setOtaProgress called with:", JSON.stringify(progress)) + console.log("🔍 GLASSES STORE: otaInProgress =", otaInProgress) + + // Never allow progress to regress within the same stage+currentUpdate — except a new + // work wave (STARTED) or the step after FINISHED (multi-hop APK: 27→31→36 re-downloads from 0). + const prev = state.otaProgress + const sameWave = + progress && + prev && + progress.stage === prev.stage && + progress.currentUpdate === prev.currentUpdate && + progress.progress < prev.progress + if (sameWave) { + const nextIsNewWave = progress.status === "STARTED" || prev.status === "FINISHED" + if (!nextIsNewWave) { + return {otaProgress: {...progress, progress: prev.progress}, otaInProgress} + } + } + + return {otaProgress: progress, otaInProgress} + }), + + setOtaInProgress: (inProgress: boolean) => set({otaInProgress: inProgress}), + + setMtkUpdatedThisSession: (updated: boolean) => set({mtkUpdatedThisSession: updated}), + + clearOtaState: () => + set({ + otaUpdateAvailable: null, + otaProgress: null, + otaInProgress: false, + // Note: mtkUpdatedThisSession is NOT cleared here - it stays true until glasses disconnect/reboot + }), + + reset: () => set(initialState), + })), +) + +export function getGlassesSystemTimeMs(): number { + return useGlassesStore.getState().systemTimeMs ?? 0 +} + +export const waitForGlassesState = ( + key: K, + predicate: (value: GlassesState[K]) => boolean, + timeoutMs = 1000, +): Promise => { + return new Promise((resolve) => { + const state = useGlassesStore.getState() + if (predicate(state[key])) { + resolve(true) + return + } + + const unsubscribe = useGlassesStore.subscribe( + (s) => s[key], + (value) => { + if (predicate(value)) { + unsubscribe() + resolve(true) + } + }, + ) + + setTimeout(() => { + unsubscribe() + resolve(predicate(useGlassesStore.getState()[key])) + }, timeoutMs) + }) +} diff --git a/mobile/modules/island/src/stores/settings.ts b/mobile/modules/island/src/stores/settings.ts new file mode 100644 index 0000000000..65569001db --- /dev/null +++ b/mobile/modules/island/src/stores/settings.ts @@ -0,0 +1,935 @@ +import {getTimeZone} from "react-native-localize" +import {AsyncResult, result as Res, Result} from "typesafe-ts" +import {create} from "zustand" +import {subscribeWithSelector} from "zustand/middleware" + +import restComms from "../services/RestComms" +import {storage} from "../utils/storage" + +export interface Setting { + key: string + defaultValue: () => any + writable: boolean + saveOnServer: boolean + // change the key to a different key based on the indexer + // NEVER do any network calls in the indexer (or performance will suffer greatly + indexer?: (key: string) => string + // optionally override the value of the setting when it's accessed + override?: () => any + // onWrite?: () => void + persist: boolean +} + +export const SETTINGS: Record = { + // feature flags / mantle settings: + dev_mode: {key: "dev_mode", defaultValue: () => __DEV__, writable: true, saveOnServer: true, persist: true},// deprecated + debug_mode: {key: "debug_mode", defaultValue: () => __DEV__, writable: true, saveOnServer: true, persist: true}, + super_mode: {key: "super_mode", defaultValue: () => false, writable: true, saveOnServer: true, persist: true}, + miniapp_dev_mode: {key: "miniapp_dev_mode", defaultValue: () => false, writable: true, saveOnServer: true, persist: true}, + enable_squircles: { + key: "enable_squircles", + defaultValue: () => true, + writable: true, + saveOnServer: true, + persist: true, + }, + android_blur: { + key: "android_blur", + defaultValue: () => { + return false + }, + writable: true, + saveOnServer: true, + persist: true, + }, + android_inner_shadow: { + key: "android_inner_shadow", + defaultValue: () => { + return false + }, + writable: true, + saveOnServer: true, + persist: true, + }, + ios_glass_effect: { + key: "ios_glass_effect", + defaultValue: () => true, + writable: true, + saveOnServer: true, + persist: true, + }, + ios_app_switcher_bottom_swipe: { + key: "ios_app_switcher_bottom_swipe", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + debug_console: { + key: "debug_console", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + debug_navigation_history: { + key: "debug_navigation_history", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + debug_core_status_bar: { + key: "debug_core_status_bar", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + // Mentra Nex feature flags (off by default; toggled from Nex Developer Settings). + // When on, the Nex display skips ASCII-only text sanitization so CJK/Chinese + // captions render on glasses. Synced to the Bluetooth SDK via BLUETOOTH_SETTING_KEYS. + nex_chinese_captions: { + key: "nex_chinese_captions", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + // When on, LC3 audio received from Nex glasses is played back (Android only). + nex_audio_playback: { + key: "nex_audio_playback", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + china_deployment: { + key: "china_deployment", + defaultValue: () => (process.env.EXPO_PUBLIC_DEPLOYMENT_REGION === "china" ? true : false), + override: () => (process.env.EXPO_PUBLIC_DEPLOYMENT_REGION === "china" ? true : false), + writable: false, + saveOnServer: false, + persist: true, + }, + backend_url: { + key: "backend_url", + defaultValue: () => { + if (process.env.EXPO_PUBLIC_BACKEND_URL_OVERRIDE) { + return process.env.EXPO_PUBLIC_BACKEND_URL_OVERRIDE + } + if (process.env.EXPO_PUBLIC_DEPLOYMENT_REGION === "china") { + return "https://api.mentraglass.cn:443" + } + return "https://api.mentra.glass" + }, + // If env var is set, always use it (on every boot) + override: () => process.env.EXPO_PUBLIC_BACKEND_URL_OVERRIDE, + writable: true, + saveOnServer: false, + persist: true, + }, + // Cloud V2 endpoint OVERRIDES. Empty = no override; cloudClient's resolveUrl + // owns the full precedence (override -> env -> Metro-derived dev default). + // The value may be an explicit URL or the METRO_AUTO sentinel ("my dev + // laptop", resolved live from Metro so it survives network changes). Never + // bake an env var or a personal LAN IP into the default here: that makes the + // override branch always-truthy and strands devs on a stale address. + cloud_core_url: { + key: "cloud_core_url", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: true, + }, + cloud_runtime_url: { + key: "cloud_runtime_url", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: true, + }, + saved_backend_urls: { + key: "saved_backend_urls", + defaultValue: () => [], + writable: true, + saveOnServer: true, + persist: true, + }, + // Developer override for the ASG OTA manifest URL. null/empty = no override; + // the normal selection applies (legacy-glasses gate, EXPO_PUBLIC_ASG_OTA_VERSION_URL, + // glasses-reported URL, then production). See getAsgOtaVersionUrl. + ota_version_url: { + key: "ota_version_url", + defaultValue: () => null, + writable: true, + saveOnServer: false, + persist: true, + }, + saved_ota_version_urls: { + key: "saved_ota_version_urls", + defaultValue: () => [], + writable: true, + saveOnServer: false, + persist: true, + }, + reconnect_on_app_foreground: { + key: "reconnect_on_app_foreground", + defaultValue: () => true, + writable: true, + saveOnServer: true, + persist: true, + }, + location_tier: {key: "location_tier", defaultValue: () => "", writable: true, saveOnServer: true, persist: true}, + // state: + core_token: {key: "core_token", defaultValue: () => "", writable: true, saveOnServer: true, persist: true}, + auth_email: {key: "auth_email", defaultValue: () => "", writable: true, saveOnServer: false, persist: true}, + pending_wearable: { + key: "pending_wearable", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: false, + }, + // Device/pairing identity is per-phone state, not an account setting: a user with + // two phones may have each paired to different glasses. These are never synced to the + // server (saveOnServer: false) so a stale server copy can't clobber the locally paired + // device on relaunch. MantleManager also strips them from any server payload as a guard + // against legacy values uploaded before this flag was flipped. + default_wearable: { + key: "default_wearable", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: true, + }, + device_name: {key: "device_name", defaultValue: () => "", writable: true, saveOnServer: false, persist: true}, + device_address: { + key: "device_address", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: true, + }, + default_controller: { + key: "default_controller", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: true, + }, + pending_controller: { + key: "pending_controller", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: true, + }, + controller_device_name: { + key: "controller_device_name", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: true, + }, + controller_address: { + key: "controller_address", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: true, + }, + // ui state: + home_background: { + key: "home_background", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: true, + }, + theme_preference: { + key: "theme_preference", + defaultValue: () => (__DEV__ ? "system" : "light"), + // Force light mode - i mode is not complete yet + // override: () => "light", + writable: true, + saveOnServer: true, + persist: true, + }, + enable_phone_notifications: { + key: "enable_phone_notifications", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + settings_access_count: { + key: "settings_access_count", + defaultValue: () => 0, + writable: true, + saveOnServer: true, + persist: true, + }, + show_advanced_settings: { + key: "show_advanced_settings", + defaultValue: () => false, + writable: true, + saveOnServer: false, + persist: true, + }, + onboarding_completed: { + key: "onboarding_completed", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + onboarding_live_completed: { + key: "onboarding_live_completed", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + onboarding_os_completed: { + key: "onboarding_os_completed", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + has_ever_activated_app: { + key: "has_ever_activated_app", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + + // Bluetooth SDK settings: + sensing_enabled: { + key: "sensing_enabled", + defaultValue: () => true, + writable: true, + saveOnServer: true, + persist: true, + }, + power_saving_mode: { + key: "power_saving_mode", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + voice_activity_detection_enabled: { + key: "voice_activity_detection_enabled", + defaultValue: () => true, + writable: true, + saveOnServer: false, + persist: false, + }, + always_on_status_bar: { + key: "always_on_status_bar", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + // Legacy cloud/mobile setting name. Locally it maps to glasses-side Voice Activity Detection. + bypass_vad_for_debugging: { + key: "bypass_vad_for_debugging", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + bypass_audio_encoding_for_debugging: { + key: "bypass_audio_encoding_for_debugging", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + metric_system: { + key: "metric_system", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + twelve_hour_time: { + key: "twelve_hour_time", + defaultValue: () => true, + writable: true, + saveOnServer: true, + persist: true, + }, + enforce_local_transcription: { + key: "enforce_local_transcription", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + // LC3 audio quality setting (frame size in bytes) + // 20 = 16kbps (low bandwidth), 40 = 32kbps (balanced), 60 = 48kbps (high quality) + lc3_frame_size: { + key: "lc3_frame_size", + defaultValue: () => 60, + writable: true, + saveOnServer: false, + persist: true, + }, + preferred_mic: { + key: "preferred_mic", + defaultValue: () => "auto", + writable: true, + indexer: (key: string) => { + const glasses = useSettingsStore.getState().getSetting(SETTINGS.default_wearable.key) + if (glasses) { + return `${key}:${glasses}` + } + return key + }, + saveOnServer: true, + persist: true, + }, + screen_disabled: { + key: "screen_disabled", + defaultValue: () => false, + writable: true, + saveOnServer: false, + persist: true, + }, + // glasses settings: + contextual_dashboard: { + key: "contextual_dashboard", + defaultValue: () => true, + writable: true, + saveOnServer: true, + persist: true, + }, + use_native_dashboard: { + key: "use_native_dashboard", + defaultValue: () => true, + writable: true, + saveOnServer: true, + persist: true, + }, + head_up_angle: {key: "head_up_angle", defaultValue: () => 45, writable: true, saveOnServer: true, persist: true}, + brightness: {key: "brightness", defaultValue: () => 50, writable: true, saveOnServer: true, persist: true}, + auto_brightness: { + key: "auto_brightness", + defaultValue: () => true, + writable: true, + saveOnServer: true, + persist: true, + }, + dashboard_height: { + key: "dashboard_height", + defaultValue: () => 4, + writable: true, + saveOnServer: true, + persist: true, + }, + dashboard_depth: { + key: "dashboard_depth", + defaultValue: () => 2, + writable: true, + saveOnServer: true, + persist: true, + }, + menu_apps: { + key: "menu_apps", + defaultValue: () => null, + writable: true, + saveOnServer: true, + persist: true, + }, + calendar_events: { + key: "calendar_events", + defaultValue: () => [], + writable: true, + saveOnServer: false, + persist: false, + }, + // button settings + // Legacy persisted/cloud key; hardware behavior is now controlled by gallery_mode plus capture settings. + button_mode: {key: "button_mode", defaultValue: () => "photo", writable: true, saveOnServer: true, persist: true}, + button_photo_size: { + key: "button_photo_size", + defaultValue: () => "max", + writable: true, + saveOnServer: true, + persist: true, + }, + button_video_settings: { + key: "button_video_settings", + defaultValue: () => ({width: 1920, height: 1080, fps: 30}), + writable: true, + saveOnServer: true, + persist: true, + }, + button_max_recording_time: { + key: "button_max_recording_time", + defaultValue: () => 10, + writable: true, + saveOnServer: true, + persist: true, + }, + camera_fov: { + key: "camera_fov", + defaultValue: () => ({fov: 118, roi_position: 0}), + writable: true, + saveOnServer: true, + persist: true, + }, + media_post_processing: { + key: "media_post_processing", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + + // time zone settings + time_zone: { + key: "time_zone", + defaultValue: () => "", + writable: true, + override: () => { + const override = useSettingsStore.getState().getSetting(SETTINGS.time_zone_override.key) + if (override) { + return override + } + return getTimeZone() + }, + saveOnServer: true, + persist: true, + }, + time_zone_override: { + key: "time_zone_override", + defaultValue: () => "", + writable: true, + saveOnServer: true, + persist: true, + }, + // offline applets + offline_mode: {key: "offline_mode", defaultValue: () => false, writable: true, saveOnServer: true, persist: true}, + offline_captions_running: { + key: "offline_captions_running", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + // Runtime flag: coordinator flips this on when cloud STT has failed and fallback is active. + // Native GlassesStore watches it to gate PCM → Sherpa feeding. Not user-facing. + local_stt_fallback_active: { + key: "local_stt_fallback_active", + defaultValue: () => false, + writable: true, + saveOnServer: false, + persist: false, + }, + gallery_mode: {key: "gallery_mode", defaultValue: () => true, writable: true, saveOnServer: true, persist: true}, + gallery_sync_explained: { + key: "gallery_sync_explained", + defaultValue: () => false, + writable: true, + saveOnServer: false, + persist: true, + }, + offline_camera_running: { + key: "offline_camera_running", + defaultValue: () => false, + writable: true, + saveOnServer: false, + persist: true, + }, + // offline translation + offline_translation_running: { + key: "offline_translation_running", + defaultValue: () => false, + writable: true, + saveOnServer: true, + persist: true, + }, + offline_translation_source: { + key: "offline_translation_source", + defaultValue: () => "en", + writable: true, + saveOnServer: true, + persist: true, + }, + offline_translation_target: { + key: "offline_translation_target", + defaultValue: () => "es", + writable: true, + saveOnServer: true, + persist: true, + }, + // button action settings + default_button_action_enabled: { + key: "default_button_action_enabled", + defaultValue: () => true, + writable: true, + saveOnServer: true, + persist: true, + }, + default_button_action_app: { + key: "default_button_action_app", + defaultValue: () => "com.mentra.camera", + writable: true, + saveOnServer: true, + persist: true, + }, + // notifications + notifications_enabled: { + key: "notifications_enabled", + defaultValue: () => true, + writable: true, + saveOnServer: true, + persist: true, + }, + notifications_blocklist: { + key: "notifications_blocklist", + defaultValue: () => [], + writable: true, + saveOnServer: true, + persist: true, + }, + // Cached required version from server - used to enforce updates even when offline + cached_required_version: { + key: "cached_required_version", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: true, + }, + // OTA update dismissal - stores the version code user dismissed (not persisted so resets on app restart) + dismissed_ota_version: { + key: "dismissed_ota_version", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: false, + }, + // Contact email for feedback (persisted for Apple private relay users) + contact_email: { + key: "contact_email", + defaultValue: () => "", + writable: true, + saveOnServer: false, + persist: true, + }, +} as const + +export const OFFLINE_APPLETS: string[] = ["com.mentra.livecaptions", "com.mentra.camera"] + +// These settings are automatically synced to the Bluetooth SDK. +// Keep this list hardware-facing; app/UI/cloud-only preferences should stay in JS/Crust. +export const BLUETOOTH_SETTING_KEYS: string[] = [ + // Bluetooth settings: + SETTINGS.sensing_enabled.key, + SETTINGS.power_saving_mode.key, + SETTINGS.voice_activity_detection_enabled.key, + SETTINGS.lc3_frame_size.key, + SETTINGS.preferred_mic.key, + SETTINGS.screen_disabled.key, + SETTINGS.auth_email.key, + SETTINGS.core_token.key, + // glasses settings: + SETTINGS.contextual_dashboard.key, + SETTINGS.head_up_angle.key, + SETTINGS.brightness.key, + SETTINGS.auto_brightness.key, + SETTINGS.dashboard_height.key, + SETTINGS.dashboard_depth.key, + SETTINGS.menu_apps.key, + SETTINGS.calendar_events.key, + SETTINGS.use_native_dashboard.key, + SETTINGS.twelve_hour_time.key, + SETTINGS.metric_system.key, + // button: + SETTINGS.button_photo_size.key, + // Legacy MentraLive native code reads the object form when syncing video settings. + SETTINGS.button_video_settings.key, + SETTINGS.button_max_recording_time.key, + SETTINGS.camera_fov.key, + // device / pairing: + SETTINGS.pending_wearable.key, + SETTINGS.default_wearable.key, + SETTINGS.device_name.key, + SETTINGS.device_address.key, + SETTINGS.default_controller.key, + SETTINGS.pending_controller.key, + SETTINGS.controller_device_name.key, + SETTINGS.controller_address.key, + // offline applets: + SETTINGS.offline_mode.key, + SETTINGS.offline_captions_running.key, + // Runtime flag flipped by LocalSttFallbackCoordinator. Native reads it from + // GlassesStore to gate PCM → Sherpa feeding in handlePcm and to keep the + // mic on while local STT is the active engine. + SETTINGS.local_stt_fallback_active.key, + SETTINGS.gallery_mode.key, + // Mentra Nex feature flags: + SETTINGS.nex_chinese_captions.key, + SETTINGS.nex_audio_playback.key, +] + +// const PER_GLASSES_SETTINGS_KEYS: string[] = [SETTINGS.preferred_mic.key] + +export interface SettingsState { + // Settings values + settings: Record + // Loading states + isInitialized: boolean + // Actions + setSetting: (key: string, value: any, updateServer?: boolean) => AsyncResult + setManyLocally: (settings: Record) => AsyncResult + getSetting: (key: string) => any + // loadSetting: (key: string) => AsyncResult + loadAllSettings: () => AsyncResult + // Utility methods + getRestUrl: () => string + getWsUrl: () => string + getBluetoothSettings: () => Record + resetAllSettingsLocally: () => void +} + +const getDefaultSettings = () => + Object.keys(SETTINGS).reduce((acc, key) => { + acc[key] = SETTINGS[key].defaultValue() + return acc + }, {} as Record) + +export const useSettingsStore = create()( + subscribeWithSelector((set, get) => ({ + settings: getDefaultSettings(), + isInitialized: false, + loadingKeys: new Set(), + setSetting: (key: string, value: any, updateServer = true): AsyncResult => { + return Res.try_async(async () => { + const setting = SETTINGS[key] + const originalKey = key + + if (!setting) { + throw new Error(`SETTINGS: SET: ${originalKey} is not a valid setting!`) + } + + if (setting.indexer) { + key = setting.indexer(originalKey) + } + + if (!setting.writable) { + throw new Error(`SETTINGS: ${originalKey} is not writable!`) + } + + // Update store immediately for optimistic UI + console.log(`SETTINGS: SET: ${key} = ${value}`) + set((state) => ({ + settings: {...state.settings, [key]: value}, + })) + + if (setting.persist) { + let res = await storage.save(key, value) + if (res.is_error()) { + throw new Error(`SETTINGS: couldn't save setting to storage: ${res.error}`) + } + + // Sync with server if needed + if (updateServer) { + const result = await restComms.writeUserSettings({[key]: value}) + if (result.is_error()) { + throw new Error(`SETTINGS: couldn't sync setting to server: ${result.error}`) + } + } + } + }) + }, + getSetting: (key: string) => { + const state = get() + const originalKey = key + const setting = SETTINGS[originalKey] + + if (!setting) { + console.error(`SETTINGS: GET: ${originalKey} is not a valid setting!`) + return undefined + } + + if (setting.override) { + let override = setting.override() + if (override !== undefined) { + return override + } + } + + if (setting.indexer) { + key = setting.indexer(originalKey) + } + + // console.log(`GET SETTING: ${key} = ${state.settings[key]}`) + + try { + const raw = state.settings[key] ?? SETTINGS[originalKey].defaultValue() + return raw + } catch (e) { + // for dynamically created settings, we need to create a new setting in SETTINGS: + console.log(`Failed to get setting, creating new setting:(${key}):`, e) + SETTINGS[key] = {key: key, defaultValue: () => undefined, writable: true, saveOnServer: false, persist: true} + return SETTINGS[key].defaultValue() + } + }, + // batch update many settings from the server: + setManyLocally: (settings: Record): AsyncResult => { + return Res.try_async(async () => { + const settingsToLoad: Record = {} + // if a setting should not persist, don't load it: + for (const [key, value] of Object.entries(settings)) { + const stg: Setting | undefined = SETTINGS[key.toLowerCase()] + if (!stg) { + continue + } + if (!stg.persist) { + continue + } + settingsToLoad[key.toLowerCase()] = value + } + // console.log("SETTINGS: SET MANY LOCALLY: ", settingsToLoad) + + set((state) => ({ + settings: {...state.settings, ...settingsToLoad}, + })) + + // save to storage: + await Promise.all(Object.entries(settingsToLoad).map(([key, value]) => storage.save(key, value))) + }) + }, + // loads any preferences that have been changed from the default and saved to DISK! + loadAllSettings: (): AsyncResult => { + console.log("SETTINGS: loadAllSettings()") + return Res.try_async(async () => { + const state = get() + let loadedSettings: Record = {} + + if (state.isInitialized) { + return undefined + } + + for (const setting of Object.values(SETTINGS)) { + // if the settings should not persist, don't load it: + if (!setting.persist) { + continue + } + + // load all subkeys for an indexed setting: + if (setting?.indexer) { + console.log(`SETTINGS: LOAD: ${setting.key} with indexer!`) + + let res: Result, Error> = storage.loadSubKeys(setting.key) + if (res.is_error()) { + console.log(`SETTINGS: LOAD: ${setting.key}`, res.error) + continue + } + + let subKeys: Record = res.value + console.log(`SETTINGS: LOAD: ${setting.key} subkeys are set!`, subKeys) + loadedSettings = {...loadedSettings, ...subKeys} + continue + } + + let res = storage.load(setting.key) + if (res.is_error()) { + console.log(`SETTINGS: LOAD: ${setting.key} is not set!`, res.error) + // this setting isn't set from the default, so we don't load anything + continue + } + // normal key:value pair: + let value = res.value + console.log(`SETTINGS: LOAD: ${setting.key} = ${value.value}`) + loadedSettings[setting.key] = value + } + + // console.log("##############################################") + // console.log(loadedSettings) + // console.log("##############################################") + + set((state) => ({ + isInitialized: true, + settings: {...state.settings, ...loadedSettings}, + })) + + // One-time migration: force android_blur=false for existing users. + // The setting's default is already false; this migration covers users + // who explicitly opted into Android blur effects before we discovered + // they're a major source of frame drops on cheap Android phones. + // The dimezisBlurViewSdk31Plus blur each costs ~5-10ms/frame; with + // multiple blurs on home (top fade + AppSwitcherButton x2) a low-end + // device misses the 16ms budget consistently. Users can turn it back + // on under Settings → Appearance once we've optimized further. + // + // The setSetting call also pushes to the server (saveOnServer: true) + // so the server-stored value flips too — otherwise the next sync + // from the user's server-stored prefs would re-enable blur. + // Best-effort: a server failure (offline, 5xx) shouldn't block boot; + // we still mark the migration done locally so we don't loop. + const MIGRATION_KEY = "migration:android_blur_default_false_v1" + const migrationDone = storage.load(MIGRATION_KEY) + if (migrationDone.is_error() || !migrationDone.value) { + const current = get().getSetting(SETTINGS.android_blur.key) + if (current === true) { + const result = await get().setSetting(SETTINGS.android_blur.key, false, true) + if (result.is_error()) { + // Server push failed (offline / 5xx). Local storage was still + // updated, so the user immediately gets the new behavior. The + // server-side stale `true` will be overwritten the next time + // the user opens Appearance settings and the auto-sync runs. + console.log("SETTINGS: android_blur migration server-push failed:", result.error) + } + } + // Mark done unconditionally — even on server-push failure we don't + // want to retry the migration on every boot. The local value is + // already correct. + storage.save(MIGRATION_KEY, true) + } + }) + }, + getRestUrl: () => { + const serverUrl = get().getSetting(SETTINGS.backend_url.key) + // console.log("GET REST URL: serverUrl:", serverUrl) + const url = new URL(serverUrl) + const secure = url.protocol === "https:" + return `${secure ? "https" : "http"}://${url.hostname}:${url.port || (secure ? 443 : 80)}` + }, + getWsUrl: () => { + const serverUrl = get().getSetting(SETTINGS.backend_url.key) + const url = new URL(serverUrl) + const secure = url.protocol === "https:" + return `${secure ? "wss" : "ws"}://${url.hostname}:${url.port || (secure ? 443 : 80)}/glasses-ws` + }, + getBluetoothSettings: () => { + const state = get() + const bluetoothSettings: Record = {} + Object.values(SETTINGS).forEach((setting) => { + if (BLUETOOTH_SETTING_KEYS.includes(setting.key)) { + bluetoothSettings[setting.key] = state.getSetting(setting.key) + } + }) + return bluetoothSettings + }, + resetAllSettingsLocally: () => { + set((_state) => ({ + settings: getDefaultSettings(), + isInitialized: true, + })) + }, + })), +) + +export const useSetting = (key: string): [T, (value: T) => AsyncResult] => { + const value = useSettingsStore((state) => state.getSetting(key)) + const setSetting = useSettingsStore((state) => state.setSetting) + return [value, (newValue: T) => setSetting(key, newValue)] +} diff --git a/mobile/modules/island/src/types/asg.ts b/mobile/modules/island/src/types/asg.ts new file mode 100644 index 0000000000..dfd3503b76 --- /dev/null +++ b/mobile/modules/island/src/types/asg.ts @@ -0,0 +1,50 @@ +/** + * Gallery / ASG HTTP-API types (relocated from the host `@/types/asg`). `PhotoInfo` + * lives with the gallerySync store that owns it; the rest describe the glasses' + * on-device camera HTTP API that asgCameraApi talks to. The host `@/types/asg` + * re-exports these from `@mentra/island` so existing importers stay unchanged. + */ +import type {PhotoInfo} from "../stores/gallerySync" +export type {PhotoInfo} + +export interface CaptureFile { + name: string // "IMG_xxx/base.jpg" or "IMG_xxx.jpg" (legacy) + size: number + role: "primary" | "bracket" | "sidecar" +} + +export interface CaptureGroup { + capture_id: string // folder name: "IMG_20250302_143022_456_123" + type: "photo" | "video" + timestamp: number + total_size: number + files: CaptureFile[] + thumbnail_data?: string + duration?: number // video only +} + +export interface GalleryResponse { + status: "success" | "error" + data: { + photos: PhotoInfo[] + } +} + +export interface ServerStatus { + status: string + uptime: number + version: string + timestamp: string +} + +export interface HealthResponse { + status: "healthy" | "unhealthy" + timestamp: string + version: string +} + +export interface GalleryEvent { + type: "photo_added" | "photo_deleted" | "gallery_updated" + photo?: PhotoInfo + timestamp: string +} diff --git a/mobile/modules/island/src/utils/GlobalEventEmitter.ts b/mobile/modules/island/src/utils/GlobalEventEmitter.ts new file mode 100644 index 0000000000..ede75c7830 --- /dev/null +++ b/mobile/modules/island/src/utils/GlobalEventEmitter.ts @@ -0,0 +1,14 @@ +// @ts-nocheck — `events` (node builtin, RN-polyfilled) ships no types in island's +// standalone build; the host build resolves them fine. +import {EventEmitter} from "events" + +/** + * Process-wide event bus — moved into island so island-owned services (RestComms) + * and the host share ONE emitter instance across the island↔host boundary. + * Re-exported through the host's `@/utils/GlobalEventEmitter` shim. + * + * @deprecated Use BluetoothSDK subscriptions directly instead. + */ +const GlobalEventEmitter = new EventEmitter() + +export default GlobalEventEmitter diff --git a/mobile/modules/island/src/utils/cloudClient/RnUdpAdapter.ts b/mobile/modules/island/src/utils/cloudClient/RnUdpAdapter.ts index 0119a3bf79..5b1181ae61 100644 --- a/mobile/modules/island/src/utils/cloudClient/RnUdpAdapter.ts +++ b/mobile/modules/island/src/utils/cloudClient/RnUdpAdapter.ts @@ -3,8 +3,7 @@ * * The cloud-client encrypts each audio frame in shared code (NaCl secretbox) and * hands the raw bytes here; this adapter only owns the native dgram socket and - * sends them. Mirrors v1's UdpManager socket setup (create udp4, bind to any - * port, send). Audio bytes never round-trip through extra JS here beyond the + * sends them. Audio bytes never round-trip through extra JS here beyond the * Buffer wrap the native module needs. */ import dgram from "react-native-udp" diff --git a/mobile/modules/island/src/utils/cloudClient/cloudSecureStore.ts b/mobile/modules/island/src/utils/cloudClient/cloudSecureStore.ts new file mode 100644 index 0000000000..d7d3ecc9b3 --- /dev/null +++ b/mobile/modules/island/src/utils/cloudClient/cloudSecureStore.ts @@ -0,0 +1,28 @@ +/** + * MMKV-backed KeyValueStore for @mentra/cloud-client — moved into island so + * island owns the cloud-v2 credential storage (the v2 refresh token, so a + * relaunch does not force a new login). A dedicated MMKV instance keeps + * cloud-v2 credentials out of the app's general store. Re-exported through the + * host's `@/utils/cloudClient/MmkvSecureStore` shim. + * + * NOTE: MMKV is not hardware-backed. Before production handling of the + * long-lived refresh token, swap this for a Keychain/Keystore-backed store + * (react-native-keychain). It is fine for dev / simulator e2e. + */ +import {createMMKV, type MMKV} from "react-native-mmkv" +import type {KeyValueStore} from "@mentra/cloud-client" + +const store: MMKV = createMMKV({id: "mentra-cloud-v2"}) + +export const cloudSecureStore: KeyValueStore = { + async get(key: string): Promise { + return store.getString(key) ?? null + }, + async set(key: string, value: string): Promise { + store.set(key, value) + }, + async delete(key: string): Promise { + // react-native-mmkv v4's instance method is `remove`, not `delete`. + store.remove(key) + }, +} diff --git a/mobile/src/utils/permissions/MediaLibraryPermissions.ts b/mobile/modules/island/src/utils/permissions/MediaLibraryPermissions.ts similarity index 100% rename from mobile/src/utils/permissions/MediaLibraryPermissions.ts rename to mobile/modules/island/src/utils/permissions/MediaLibraryPermissions.ts diff --git a/mobile/src/utils/permissions/galleryDisplayName.ts b/mobile/modules/island/src/utils/permissions/galleryDisplayName.ts similarity index 100% rename from mobile/src/utils/permissions/galleryDisplayName.ts rename to mobile/modules/island/src/utils/permissions/galleryDisplayName.ts diff --git a/mobile/modules/island/src/utils/storage/storage.ts b/mobile/modules/island/src/utils/storage/storage.ts index 2f0b603c85..c6aa3b1956 100644 --- a/mobile/modules/island/src/utils/storage/storage.ts +++ b/mobile/modules/island/src/utils/storage/storage.ts @@ -51,6 +51,30 @@ class MMKVStorage { return Res.ok(undefined) } + public loadSubKeys(key: string): Result, Error> { + return Res.try(() => { + // return the key value pair of any keys that start with the given key and contain a colon: + const keys = this.store.getAllKeys() + + const subKeys = keys.filter((k) => k.startsWith(key) && k.includes(":")) + + if (subKeys.length === 0) { + throw new Error(`No subkeys found for ${key}`) + } + + let subKeysObject: Record = {} + + for (const subKey of subKeys) { + const res = this.load(subKey) + if (res.is_ok()) { + subKeysObject[subKey] = res.value + } + } + + return subKeysObject + }) + } + public remove(key: string): Result { this.store.remove(key) return Res.ok(undefined) diff --git a/mobile/src/__tests__/app/pairing/loading.test.tsx b/mobile/src/__tests__/app/pairing/loading.test.tsx index 59d68ed01b..12ec459a16 100644 --- a/mobile/src/__tests__/app/pairing/loading.test.tsx +++ b/mobile/src/__tests__/app/pairing/loading.test.tsx @@ -1,7 +1,7 @@ import {act, render, waitFor} from "@testing-library/react-native" import type {ReactNode} from "react" -import BluetoothSdk from "@mentra/bluetooth-sdk" +import {toolkit} from "@mentra/island" import {useRoute} from "@react-navigation/native" import {useNavigationStore} from "@/stores/navigation" import {submitAutomaticBugIncident} from "@/services/bugReport/automaticBugReport" @@ -123,7 +123,7 @@ describe("pairing loading screen", () => { }) await waitFor(() => { - expect(BluetoothSdk.forget).toHaveBeenCalled() + expect(toolkit.glasses.forget).toHaveBeenCalled() expect(replace).toHaveBeenCalledWith("/pairing/failure", { error: "pairing:failed", deviceModel: "Mentra Live", diff --git a/mobile/src/__tests__/app/pairing/scan.test.tsx b/mobile/src/__tests__/app/pairing/scan.test.tsx index 39943ceddf..ef1166f22a 100644 --- a/mobile/src/__tests__/app/pairing/scan.test.tsx +++ b/mobile/src/__tests__/app/pairing/scan.test.tsx @@ -138,7 +138,7 @@ import {render, fireEvent, waitFor} from "@testing-library/react-native" import type {ReactNode} from "react" import {Platform} from "react-native" -import BluetoothSdk from "@mentra/bluetooth-sdk" +import {toolkit} from "@mentra/island" import {useLocalSearchParams} from "expo-router" import {usePushUnder} from "@/contexts/NavigationHistoryContext" import {useNavigationStore} from "@/stores/navigation" @@ -189,7 +189,7 @@ describe("pairing scan screen", () => { const {getByText} = render() await waitFor(() => { - expect(BluetoothSdk.startScan).toHaveBeenCalledWith("Mentra Live") + expect(toolkit.pairing.scan).toHaveBeenCalledWith("Mentra Live") }) fireEvent.press(getByText("001")) @@ -202,7 +202,7 @@ describe("pairing scan screen", () => { }) }) - expect(BluetoothSdk.setDefaultDevice).toHaveBeenCalledWith({ + expect(toolkit.pairing.setDefault).toHaveBeenCalledWith({ id: "a", model: "Mentra Live", name: "MENTRA_LIVE_BLE_001", diff --git a/mobile/src/__tests__/glassesFacade.test.ts b/mobile/src/__tests__/glassesFacade.test.ts new file mode 100644 index 0000000000..804737f7da --- /dev/null +++ b/mobile/src/__tests__/glassesFacade.test.ts @@ -0,0 +1,67 @@ +// Imports the real glasses facade by path (not via "@mentra/island", which jest +// mocks) so the actual projection + delegation run under the mobile jest runner. +import BluetoothSdk from "@mentra/bluetooth-sdk-internal" +import {glasses} from "../../modules/island/src/facades/glasses" +import {useGlassesStore} from "../../modules/island/src/stores/glasses" + +describe("glasses facade", () => { + beforeEach(() => { + jest.clearAllMocks() + useGlassesStore.getState().reset() + }) + + it("status() projects the glasses store into the read-model", () => { + useGlassesStore.setState({ + connection: {state: "connected", fullyBooted: true} as never, + batteryLevel: 80, + charging: true, + signalStrength: -50, + micEnabled: true, + bluetoothClassicConnected: true, + }) + const st = glasses.status() + expect(st.state).toBe("connected") + expect(st.fullyBooted).toBe(true) + expect(st.battery).toBe(80) + expect(st.charging).toBe(true) + expect(st.micEnabled).toBe(true) + expect(st.btClassic).toBe(true) + }) + + it("info() projects device info from the store", () => { + useGlassesStore.setState({deviceModel: "Even Realities G1", firmwareVersion: "1.2.3", serialNumber: "SN1"}) + const info = glasses.info() + expect(info.model).toBe("Even Realities G1") + expect(info.firmwareVersion).toBe("1.2.3") + expect(info.serialNumber).toBe("SN1") + }) + + it("connectDefault / disconnect / forget delegate to bluetooth-sdk", async () => { + await glasses.connectDefault() + expect(BluetoothSdk.connectDefault).toHaveBeenCalled() + await glasses.disconnect() + expect(BluetoothSdk.disconnect).toHaveBeenCalled() + await glasses.forget() + expect(BluetoothSdk.forget).toHaveBeenCalled() + }) + + it("connectDefault() seeds the phone's device settings to native before connecting", async () => { + await glasses.connectDefault() + // The pre-connect seed (moved out of the host Reconnect flow) must land first. + expect(BluetoothSdk.updateBluetoothSettings).toHaveBeenCalled() + expect((BluetoothSdk.updateBluetoothSettings as jest.Mock).mock.invocationCallOrder[0]).toBeLessThan( + (BluetoothSdk.connectDefault as jest.Mock).mock.invocationCallOrder[0], + ) + }) + + it("onStatus() fires on store changes and stops after unsubscribe", () => { + const cb = jest.fn() + const unsub = glasses.onStatus(cb) + useGlassesStore.setState({batteryLevel: 42}) + expect(cb).toHaveBeenCalled() + unsub() + cb.mockClear() + useGlassesStore.setState({batteryLevel: 10}) + expect(cb).not.toHaveBeenCalled() + }) +}) diff --git a/mobile/src/__tests__/glassesWifi.test.ts b/mobile/src/__tests__/glassesWifi.test.ts new file mode 100644 index 0000000000..9c39935094 --- /dev/null +++ b/mobile/src/__tests__/glassesWifi.test.ts @@ -0,0 +1,51 @@ +// Imports the real glasses.wifi facade by path (not via "@mentra/island", which +// jest mocks) so the actual delegation runs under the mobile jest CI runner. +// @mentra/bluetooth-sdk is globally mocked in jest.setup.js — the facade calls it. +import BluetoothSdk from "@mentra/bluetooth-sdk" +import {glassesWifi} from "../../modules/island/src/facades/glassesWifi" +import {useGlassesStore} from "../../modules/island/src/stores/glasses" + +describe("glassesWifi facade", () => { + beforeEach(() => { + jest.clearAllMocks() + }) + + it("scan() delegates to requestWifiScan and returns its results", async () => { + const networks = [{ssid: "home", requiresPassword: true, signalStrength: -40}] + ;(BluetoothSdk.requestWifiScan as jest.Mock).mockResolvedValueOnce(networks) + await expect(glassesWifi.scan()).resolves.toEqual(networks) + expect(BluetoothSdk.requestWifiScan).toHaveBeenCalledTimes(1) + }) + + it("connect() delegates to sendWifiCredentials with ssid + password", async () => { + await glassesWifi.connect("home", "hunter2") + expect(BluetoothSdk.sendWifiCredentials).toHaveBeenCalledWith("home", "hunter2") + }) + + it("connect() propagates the bluetooth-sdk coded error unchanged", async () => { + const err = Object.assign(new Error("bt off"), {code: "bluetooth_powered_off"}) + ;(BluetoothSdk.sendWifiCredentials as jest.Mock).mockRejectedValueOnce(err) + await expect(glassesWifi.connect("home", "x")).rejects.toBe(err) + }) + + it("forget() delegates to forgetWifiNetwork", async () => { + await glassesWifi.forget("home") + expect(BluetoothSdk.forgetWifiNetwork).toHaveBeenCalledWith("home") + }) + + it("status() reads the glasses store's wifi state", () => { + useGlassesStore.setState({wifi: {state: "connected", ssid: "home"}}) + expect(glassesWifi.status()).toEqual({state: "connected", ssid: "home"}) + }) + + it("onStatus() fires on wifi change and stops after unsubscribe", () => { + useGlassesStore.setState({wifi: {state: "disconnected"}}) + const cb = jest.fn() + const unsub = glassesWifi.onStatus(cb) + useGlassesStore.setState({wifi: {state: "connected", ssid: "home"}}) + expect(cb).toHaveBeenCalledWith({state: "connected", ssid: "home"}, {state: "disconnected"}) + unsub() + useGlassesStore.setState({wifi: {state: "disconnected"}}) + expect(cb).toHaveBeenCalledTimes(1) + }) +}) diff --git a/mobile/src/__tests__/islandBootstrapMirror.test.ts b/mobile/src/__tests__/islandBootstrapMirror.test.ts new file mode 100644 index 0000000000..362470b557 --- /dev/null +++ b/mobile/src/__tests__/islandBootstrapMirror.test.ts @@ -0,0 +1,52 @@ +// Imports the real bootstrap + displayMirror by path (not via "@mentra/island", +// which jest mocks) so the actual logic runs under the mobile jest CI runner. +import * as bootstrap from "../../modules/island/src/runtime/bootstrap" +import {displayMirror} from "../../modules/island/src/facades/displayMirror" +import {useDisplayStore} from "../../modules/island/src/stores/display" + +describe("island bootstrap front door", () => { + beforeEach(async () => { + await bootstrap.stop() + }) + + it("start() before configure() rejects (this test runs first, nothing configured yet)", async () => { + await expect(bootstrap.start()).rejects.toThrow(/configure/) + }) + + it("configure() stores auth/config/analytics for island to read", () => { + const auth = {getSubjectToken: async () => ({token: "t", type: "supabase" as const})} + const analytics = jest.fn() + bootstrap.configure({auth, config: {coreUrl: "http://core", oemId: "mentra"}, analytics}) + expect(bootstrap.getAuth()).toBe(auth) + expect(bootstrap.getConfigValues()).toEqual({coreUrl: "http://core", oemId: "mentra"}) + expect(bootstrap.getAnalytics()).toBe(analytics) + }) + + it("start()/stop() toggle isStarted (idempotent)", async () => { + bootstrap.configure({auth: {getSubjectToken: async () => ({token: "t", type: "supabase"})}}) + await bootstrap.start() + await bootstrap.start() + expect(bootstrap.isStarted()).toBe(true) + await bootstrap.stop() + expect(bootstrap.isStarted()).toBe(false) + }) +}) + +describe("toolkit.display.mirror read facade over the display store", () => { + it("current() returns the store's current display event", () => { + const event = {view: "main", layout: {layoutType: "text_wall", text: "hi"}} + useDisplayStore.getState().setDisplayEvent(JSON.stringify(event)) + expect(displayMirror.current()).toEqual(event) + }) + + it("onMirror() fires on display change and stops after unsubscribe", () => { + const cb = jest.fn() + const unsub = displayMirror.onMirror(cb) + useDisplayStore.getState().setDisplayEvent(JSON.stringify({view: "main", layout: {layoutType: "text_wall", text: "x"}})) + expect(cb).toHaveBeenCalled() + unsub() + cb.mockClear() + useDisplayStore.getState().setDisplayEvent(JSON.stringify({view: "main", layout: {layoutType: "text_wall", text: "y"}})) + expect(cb).not.toHaveBeenCalled() + }) +}) diff --git a/mobile/src/__tests__/otaService.test.ts b/mobile/src/__tests__/otaService.test.ts new file mode 100644 index 0000000000..a1a428d41d --- /dev/null +++ b/mobile/src/__tests__/otaService.test.ts @@ -0,0 +1,61 @@ +// Exercises the real island OtaService by path (not via "@mentra/island", which jest +// mocks): emitted OTA BLE events must project into the island glasses store, which is +// the toolkit.ota read surface. +// +// NOTE: the push-based `ota_update_available` BLE event was removed in the OTA-simplify +// pass — availability is now resolved by the host OTA check flow (checkForOtaUpdate + +// setOtaUpdateAvailable), so OtaService only projects status/progress here. +import {startOtaService, stopOtaService} from "../../modules/island/src/services/OtaService" +import {useGlassesStore} from "../../modules/island/src/stores/glasses" +import {emitBluetoothSdkEvent, resetBluetoothSdkMock} from "@/test-utils/mockBluetoothSdk" + +describe("OtaService projection", () => { + beforeEach(() => { + resetBluetoothSdkMock() + useGlassesStore.getState().reset() + useGlassesStore.setState({connection: {state: "connected", fullyBooted: true}} as never) + startOtaService() + }) + + afterEach(() => { + stopOtaService() + }) + + it("projects ota_status into the store and clears the available flag on completion", () => { + useGlassesStore.getState().setOtaUpdateAvailable({ + available: true, + versionCode: 1, + versionName: "x", + updates: [], + totalSize: 0, + }) + emitBluetoothSdkEvent("ota_status", { + session_id: "s1", + total_steps: 1, + current_step: 1, + step_type: "apk", + phase: "install", + step_percent: 100, + overall_percent: 100, + status: "complete", + }) + expect(useGlassesStore.getState().otaStatus?.status).toBe("complete") + // complete/failed clears the available flag (matches the prior MantleManager handler). + expect(useGlassesStore.getState().otaUpdateAvailable).toBeNull() + }) + + it("stops projecting after stopOtaService()", () => { + stopOtaService() + emitBluetoothSdkEvent("ota_status", { + session_id: "s1", + total_steps: 1, + current_step: 1, + step_type: "apk", + phase: "install", + step_percent: 50, + overall_percent: 50, + status: "downloading", + }) + expect(useGlassesStore.getState().otaStatus).toBeNull() + }) +}) diff --git a/mobile/src/__tests__/speechFacade.test.ts b/mobile/src/__tests__/speechFacade.test.ts new file mode 100644 index 0000000000..b1685c9ddd --- /dev/null +++ b/mobile/src/__tests__/speechFacade.test.ts @@ -0,0 +1,55 @@ +// Mock the in-island model managers (they pull native deps) so the real speech +// facade's forwarding can be exercised under the mobile jest runner. +jest.mock("../../modules/island/src/services/STTModelManager", () => ({ + __esModule: true, + default: { + getCurrentLanguage: jest.fn(() => "en"), + getAvailableLanguages: jest.fn(() => [{code: "en"}]), + getAllLanguageInfo: jest.fn(() => Promise.resolve([])), + downloadModel: jest.fn(() => Promise.resolve()), + setCurrentLanguage: jest.fn(), + cancelDownload: jest.fn(() => Promise.resolve()), + deleteModel: jest.fn(() => Promise.resolve()), + }, +})) +jest.mock("../../modules/island/src/services/TTSModelManager", () => ({ + __esModule: true, + default: { + getCurrentLanguage: jest.fn(() => "en"), + getAvailableLanguages: jest.fn(() => []), + getAllLanguageInfo: jest.fn(() => Promise.resolve([])), + downloadModel: jest.fn(() => Promise.resolve()), + setCurrentLanguage: jest.fn(), + cancelDownload: jest.fn(() => Promise.resolve()), + deleteModel: jest.fn(() => Promise.resolve()), + }, +})) +jest.mock("../../modules/island/src/services/OfflineSpeechModelService", () => ({ + __esModule: true, + default: {getStatus: jest.fn(() => null), subscribe: jest.fn(() => () => {})}, +})) + +import sttModelManager from "../../modules/island/src/services/STTModelManager" +import ttsModelManager from "../../modules/island/src/services/TTSModelManager" +import {speech} from "../../modules/island/src/facades/speech" + +describe("speech facade", () => { + beforeEach(() => jest.clearAllMocks()) + + it("stt.activate delegates to setCurrentLanguage", () => { + speech.stt.activate("fr") + expect(sttModelManager.setCurrentLanguage).toHaveBeenCalledWith("fr") + }) + + it("stt.download forwards its args to downloadModel", async () => { + const onProgress = jest.fn() + await speech.stt.download("de", onProgress) + expect(sttModelManager.downloadModel).toHaveBeenCalledWith("de", onProgress) + }) + + it("tts methods delegate to the TTS manager", () => { + expect(speech.tts.currentLanguage()).toBe("en") + speech.tts.activate("es") + expect(ttsModelManager.setCurrentLanguage).toHaveBeenCalledWith("es") + }) +}) diff --git a/mobile/src/app/applet/settings.tsx b/mobile/src/app/applet/settings.tsx deleted file mode 100644 index 8789e9f9bb..0000000000 --- a/mobile/src/app/applet/settings.tsx +++ /dev/null @@ -1,846 +0,0 @@ -import {useFocusEffect, useLocalSearchParams} from "expo-router" -import {useCallback, useEffect, useMemo, useRef, useState} from "react" -import {Animated, BackHandler, Platform, TextStyle, View, ViewStyle} from "react-native" -import {useSaferAreaInsets} from "@/contexts/SaferAreaContext" - -import {Header, Icon, PillButton, Screen, Text} from "@/components/ignite" -import AppIcon from "@/components/home/AppIcon" -import LoadingOverlay from "@/components/ui/LoadingOverlay" -import SettingsSkeleton from "@/components/settings/SettingsSkeleton" -import GroupTitle from "@/components/settings/GroupTitle" -import MultiSelectSetting from "@/components/settings/MultiSelectSetting" -import NumberSetting from "@/components/settings/NumberSetting" -import SelectSetting from "@/components/settings/SelectSetting" -import SelectWithSearchSetting from "@/components/settings/SelectWithSearchSetting" -import SliderSetting from "@/components/settings/SliderSetting" -import TextSettingNoSave from "@/components/settings/TextSettingNoSave" -import TimeSetting from "@/components/settings/TimeSetting" -import TitleValueSetting from "@/components/settings/TitleValueSetting" -import ToggleSetting from "@/components/settings/ToggleSetting" -import Divider from "@/components/ui/Divider" -import InfoCardSection from "@/components/ui/InfoCard" -import {RouteButton} from "@/components/ui/RouteButton" -import {focusEffectPreventBack} from "@/contexts/NavigationHistoryContext" -import {useAppTheme} from "@/contexts/ThemeContext" -import {useNavigationStore} from "@/stores/navigation" -import {translate} from "@/i18n" -import restComms from "@/services/RestComms" -import {useApps, useAppStatusStore, useRefresh, useStart, useStop} from "@mentra/island" - -import {SYSTEM_APPS} from "@/constants/miniapps" -import {ThemedStyle} from "@/theme" -import {showAlert} from "@/utils/AlertUtils" -import {askPermissionsUI} from "@/utils/PermissionsUtils" -import {storage} from "@/utils/storage" -import {captureRef} from "react-native-view-shot" - -export default function AppSettings() { - const {packageName, appName: appNameParam} = useLocalSearchParams<{packageName: string; appName?: string}>() - const [isUninstalling, setIsUninstalling] = useState(false) - const {theme, themed} = useAppTheme() - const {goBack, replaceAll} = useNavigationStore.getState() - const insets = useSaferAreaInsets() - const hasLoadedData = useRef(false) - - // Use appName from params or default to empty string - const [appName, setAppName] = useState(appNameParam || "") - - // Animation values for collapsing header - const scrollY = useRef(new Animated.Value(0)).current - const headerOpacity = scrollY.interpolate({ - inputRange: [0, 50, 100], - outputRange: [0, 0, 1], - extrapolate: "clamp", - }) - - // State to hold the complete configuration from the server. - const [serverAppInfo, setServerAppInfo] = useState(null) - // Local state to track current values for each setting. - const [settingsState, setSettingsState] = useState<{[key: string]: any}>({}) - - const startApp = useStart() - const applets = useApps() - const refreshApplets = useRefresh() - const stopApp = useStop() - - const appInfo = useMemo(() => { - return applets.find((app) => app.packageName === packageName) || null - }, [applets, packageName]) - - const SETTINGS_CACHE_KEY = (packageName: string) => `app_settings_cache_${packageName}` - const [settingsLoading, setSettingsLoading] = useState(true) - const [hasCachedSettings, setHasCachedSettings] = useState(false) - - const viewShotRef = useRef(null) - const saveScreenshot = async () => { - try { - const uri = await captureRef(viewShotRef, { - format: "jpg", - quality: 0.5, - }) - console.log("saving screenshot for", packageName) - await useAppStatusStore.getState().saveScreenshot(packageName as string, uri) - } catch (e) { - console.warn("screenshot failed:", e) - } - } - const handleExit = async () => { - await saveScreenshot() - goBack() - } - focusEffectPreventBack(() => { - console.log("SETTINGS: focusEffectPreventBack() called") - // On iOS, iosDontPreventBack=true lets the native gesture handle navigation, - // so we only need to capture the screenshot. On Android, preventBack is still - // true so we must also call goBack() to navigate. - saveScreenshot() - if (Platform.OS === "android") { - goBack() - } - }, true) - - // Handle app start/stop actions with debouncing - const handleStartStopApp = async () => { - if (!appInfo) return - - try { - if (appInfo.running) { - stopApp(packageName) - return - } - - // If the app appears offline, confirm before proceeding - if (!appInfo.healthy) { - const developerName = ( - " " + - ((serverAppInfo as any)?.organization?.name || - (appInfo as any).orgName || - (appInfo as any).developerId || - "") + - " " - ).replace(" ", " ") - const proceed = await new Promise((resolve) => { - // Use the shared alert utility - showAlert( - translate("appSettings:appDownForMaintenance"), - translate("appSettings:appOfflineMessage", {appName: appInfo.name, developerName}), - [ - {text: translate("common:cancel"), style: "cancel", onPress: () => resolve(false)}, - {text: translate("appSettings:tryAnyway"), onPress: () => resolve(true)}, - ], - {iconName: "alert-circle-outline", iconColor: theme.colors.palette.angry500}, - ) - }) - if (!proceed) return - } - - const health = await restComms.checkAppHealthStatus(appInfo.packageName) - if (health.is_error() || !health.value) { - showAlert(translate("errors:appNotOnlineTitle"), translate("errors:appNotOnlineMessage"), [ - {text: translate("common:ok")}, - ]) - return - } - - // ask for needed perms: - const result = await askPermissionsUI(appInfo, theme) - if (result === -1) { - return - } else if (result === 0) { - handleStartStopApp() // restart this function - return - } - - startApp(appInfo) - } catch (error) { - // Refresh the app status to get the accurate state from the server - refreshApplets() - - console.error(`Error ${appInfo.running ? "stopping" : "starting"} app:`, error) - } - } - - const handleUninstallApp = () => { - console.log(`Uninstalling app: ${packageName}`) - - showAlert( - translate("appSettings:uninstallApp"), - translate("appSettings:uninstallConfirm", {appName: appInfo?.name || appName}), - [ - { - text: translate("common:cancel"), - style: "cancel", - }, - { - text: translate("appSettings:uninstall"), - style: "destructive", - onPress: async () => { - try { - setIsUninstalling(true) - // First stop the app if it's running - if (appInfo?.running) { - // Optimistically update UI first - stopApp(packageName) - await restComms.stopApp(packageName) - } - - // Then uninstall it - await restComms.uninstallApp(packageName) - - // Show success message and navigate after dismissal - showAlert( - translate("common:success"), - translate("appSettings:uninstalledSuccess", {appName: appInfo?.name || appName}), - [{text: translate("common:ok"), onPress: () => replaceAll("/home")}], - ) - } catch (error: any) { - console.error("Error uninstalling app:", error) - refreshApplets() - showAlert( - translate("common:error"), - translate("appSettings:uninstallError", {error: error.message || "Unknown error"}), - [{text: translate("common:ok")}], - ) - } finally { - setIsUninstalling(false) - } - }, - }, - ], - { - iconName: "trash", - iconSize: 48, - }, - ) - } - - const fetchUpdatedSettingsInfo = async () => { - // Only show skeleton if there are no cached settings - if (!hasCachedSettings) setSettingsLoading(true) - const startTime = Date.now() // For profiling - try { - const res = await restComms.getAppSettings(packageName) - - const elapsed = Date.now() - startTime - console.log(`[PROFILE] getTpaSettings for ${packageName} took ${elapsed}ms`) - console.log("GOT TPA SETTING") - // TODO: Profile backend and optimize if slow - // If no data is returned from the server, create a minimal app info object - if (res.is_error()) { - setServerAppInfo({ - name: appInfo?.name || appName, - description: translate("appSettings:noDescription"), - settings: [], - uninstallable: !SYSTEM_APPS.includes(packageName), - }) - setSettingsState({}) - setHasCachedSettings(false) - setSettingsLoading(false) - return - } - const data: any = res.value - setServerAppInfo(data) - - console.log("GOT TPA SETTING", JSON.stringify(data)) - - // Update appName if we got it from server - if (data.name) { - setAppName(data.name) - } - - // Initialize local state using the "selected" property. - if (data.settings && Array.isArray(data.settings)) { - const initialState: {[key: string]: any} = {} - data.settings.forEach((setting: any) => { - if (setting.type !== "group") { - // Use cached value if it exists (user has interacted with this setting before) - // Otherwise use 'selected' from backend (which includes defaultValue for new settings) - initialState[setting.key] = setting.selected - } - }) - setSettingsState(initialState) - // Cache the settings - storage.save(SETTINGS_CACHE_KEY(packageName), { - serverAppInfo: data, - settingsState: initialState, - }) - setHasCachedSettings(data.settings.length > 0) - } else { - setHasCachedSettings(false) - } - setSettingsLoading(false) - - // TACTICAL BYPASS: Execute immediate webview redirect if webviewURL detected - // if (data.webviewURL && fromWebView !== "true") { - // replace("/applet/webview", { - // webviewURL: data.webviewURL, - // appName: appName, - // packageName: packageName, - // }) - // return - // } - } catch (err) { - setSettingsLoading(false) - setHasCachedSettings(false) - console.error("Error fetching App settings:", err) - setServerAppInfo({ - name: appInfo?.name || appName, - description: translate("appSettings:noDescription"), - settings: [], - uninstallable: true, - }) - setSettingsState({}) - } - } - - // When a setting changes, update local state and send the full updated settings payload. - const handleSettingChange = (key: string, value: any) => { - setSettingsState((prevState) => ({ - ...prevState, - [key]: value, - })) - - void restComms.updateAppSetting(packageName, {key, value}).then((result) => { - if (result.is_error()) { - console.error("Error updating setting on server:", result.error) - return - } - console.log("Server update response:", result.value) - }) - } - - // Pre-process settings into groups for proper isFirst/isLast styling - const processedSettings = useMemo(() => { - if (!serverAppInfo?.settings) return [] - - const settings = serverAppInfo.settings - const result: Array<{setting: any; isFirst: boolean; isLast: boolean; isGrouped: boolean}> = [] - let currentGroupStart = -1 - - for (let i = 0; i < settings.length; i++) { - const setting = settings[i] - - if (setting.type === "group") { - // Close previous group if exists - if (currentGroupStart !== -1 && result.length > 0) { - // Find last non-group setting and mark as last - for (let j = result.length - 1; j >= 0; j--) { - if (result[j].isGrouped) { - result[j].isLast = true - break - } - } - } - // Add group title (not styled as grouped) - result.push({setting, isFirst: false, isLast: false, isGrouped: false}) - currentGroupStart = result.length - } else { - // Check if this is the first setting after a group title or at the start - const isFirstInGroup = - currentGroupStart === result.length || - (currentGroupStart === -1 && result.filter((r) => r.isGrouped).length === 0) - - // Check if next is a group or end - const nextSetting = settings[i + 1] - const isLastInGroup = !nextSetting || nextSetting.type === "group" - - result.push({ - setting, - isFirst: isFirstInGroup, - isLast: isLastInGroup, - isGrouped: true, - }) - } - } - - return result - }, [serverAppInfo?.settings]) - - // Render each setting. - const renderSetting = (setting: any, isFirst: boolean, isLast: boolean, index: number) => { - switch (setting.type) { - case "group": - return - case "toggle": - return ( - handleSettingChange(setting.key, val)} - isFirst={isFirst} - isLast={isLast} - /> - ) - case "text": - return ( - handleSettingChange(setting.key, text)} - settingKey={setting.key} - isFirst={isFirst} - isLast={isLast} - /> - ) - case "text_no_save_button": - return ( - handleSettingChange(setting.key, text)} - settingKey={setting.key} - isFirst={isFirst} - isLast={isLast} - /> - ) - case "slider": - return ( - - setSettingsState((prevState) => ({ - ...prevState, - [setting.key]: val, - })) - } - onValueSet={(val) => handleSettingChange(setting.key, val)} - isFirst={isFirst} - isLast={isLast} - /> - ) - case "select": - return ( - handleSettingChange(setting.key, val)} - isFirst={isFirst} - isLast={isLast} - /> - ) - case "select_with_search": - return ( - handleSettingChange(setting.key, val)} - isFirst={isFirst} - isLast={isLast} - /> - ) - case "numeric_input": - return ( - handleSettingChange(setting.key, val)} - isFirst={isFirst} - isLast={isLast} - /> - ) - case "time_picker": - return ( - handleSettingChange(setting.key, val)} - isFirst={isFirst} - isLast={isLast} - /> - ) - case "multiselect": - return ( - handleSettingChange(setting.key, vals)} - isFirst={isFirst} - isLast={isLast} - /> - ) - case "titleValue": - return ( - - ) - default: - return null - } - } - - // Reset hasLoadedData when packageName changes - useEffect(() => { - hasLoadedData.current = false - }, [packageName]) - - // Fetch App settings on mount - useEffect(() => { - // Skip if we've already loaded data for this packageName - if (hasLoadedData.current) { - return - } - - let isMounted = true - let debounceTimeout: NodeJS.Timeout - - const loadCachedSettings = async () => { - const res = await storage.load(SETTINGS_CACHE_KEY(packageName)) - if (res.is_error()) { - setHasCachedSettings(false) - setSettingsLoading(true) - return - } - const cached: any = res.value - if (!isMounted) { - return - } - - setServerAppInfo(cached.serverAppInfo) - setSettingsState(cached.settingsState) - setHasCachedSettings(!!(cached.serverAppInfo?.settings && cached.serverAppInfo.settings.length > 0)) - setSettingsLoading(false) - - // Update appName from cached data if available - if (cached.serverAppInfo?.name) { - setAppName(cached.serverAppInfo.name) - } - - // TACTICAL BYPASS: If webviewURL exists in cached data, execute immediate redirect - // if (cached.serverAppInfo?.webviewURL && fromWebView !== "true") { - // replace("/applet/webview", { - // webviewURL: cached.serverAppInfo.webviewURL, - // appName: appName, - // packageName: packageName, - // }) - // return - // } - } - - // Load cached settings immediately - loadCachedSettings() - - // Debounce fetch to avoid redundant calls - debounceTimeout = setTimeout(() => { - fetchUpdatedSettingsInfo() - hasLoadedData.current = true - }, 150) - - return () => { - isMounted = false - clearTimeout(debounceTimeout) - } - }, []) - - if (!appInfo) { - // Optionally, you could render a fallback error or nothing - return null - } - - if (!packageName || typeof packageName !== "string") { - console.error("No packageName found in params") - return null - } - - return ( - - {isUninstalling && } - - -
- - - {appInfo?.name || (appName as string)} - - - - - {/* */} - - - {/* Combined App Info and Action Section */} - - - - - - {appInfo.name} - {serverAppInfo?.version && ( - {serverAppInfo?.version || "1.0.0"} - )} - - {/* - - */} - - - - {!appInfo.healthy && !appInfo.offline && ( - - - {translate("appSettings:appOfflineWarning")} - - )} - - - - {/* Description Section */} - - - {serverAppInfo?.description || translate("appSettings:noDescription")} - - - - - - {/* App Instructions Section */} - {serverAppInfo?.instructions && ( - - {translate("appSettings:aboutThisApp")} - {serverAppInfo.instructions} - - )} - - {/* App Settings Section */} - - { - settingsLoading && (!serverAppInfo?.settings || typeof serverAppInfo.settings === "undefined") ? ( - - ) : processedSettings.length > 0 ? ( - processedSettings.map(({setting, isFirst, isLast}, index) => - renderSetting(setting, isFirst, isLast, index), - ) - ) : null /* ( - {translate("appSettings:noSettings")} - ) */ - } - - - {/* Additional Information Section */} - - {translate("appSettings:appInfo")} - - - - {/* Uninstall Button at the bottom */} - { - if (serverAppInfo?.uninstallable && !SYSTEM_APPS.includes(packageName)) { - handleUninstallApp() - } else { - showAlert(translate("appSettings:cannotUninstall"), translate("appSettings:cannotUninstallMessage"), [ - {text: translate("common:ok"), style: "default"}, - ]) - } - }} - disabled={!serverAppInfo?.uninstallable || SYSTEM_APPS.includes(packageName)} - /> - - {/* Bottom safe area padding */} - - - - {/* */} - - ) -} - -const $topSection: ThemedStyle = ({spacing}) => ({ - flexDirection: "row", - gap: spacing.s6, - alignItems: "center", -}) - -const $rightColumn: ThemedStyle = () => ({ - flex: 1, - justifyContent: "space-between", -}) - -const $textContainer: ThemedStyle = ({spacing}) => ({ - gap: spacing.s1, -}) - -const $buttonContainer: ThemedStyle = ({spacing}) => ({ - alignSelf: "flex-start", - marginTop: spacing.s3, -}) - -const $appIconLarge: ThemedStyle = ({spacing}) => ({ - width: 90, - height: 90, - borderRadius: spacing.s6, // Squircle-friendly radius -}) - -const $appNameSmall: ThemedStyle = ({colors}) => ({ - fontSize: 24, - fontWeight: "600", - fontFamily: "Montserrat-Bold", - color: colors.text, -}) - -const $versionText: ThemedStyle = ({colors}) => ({ - fontSize: 16, - fontFamily: "Montserrat-Regular", - color: colors.textDim, -}) - -const $descriptionSection: ThemedStyle = ({spacing}) => ({ - paddingVertical: spacing.s2, - paddingHorizontal: spacing.s4, -}) - -const $descriptionText: ThemedStyle = ({colors}) => ({ - fontSize: 16, - fontFamily: "Montserrat-Regular", - lineHeight: 22, - color: colors.text, -}) - -const $sectionContainer: ThemedStyle = ({colors, spacing}) => ({ - borderRadius: spacing.s3, - borderWidth: 1, - padding: spacing.s4, - elevation: 2, - shadowColor: "#000", - shadowOffset: {width: 0, height: 2}, - shadowOpacity: 0.1, - shadowRadius: spacing.s1, - backgroundColor: colors.background, - borderColor: colors.border, -}) - -const $sectionTitle: ThemedStyle = ({colors, spacing}) => ({ - fontSize: 18, - fontWeight: "bold", - fontFamily: "Montserrat-Bold", - marginBottom: spacing.s3, - color: colors.text, -}) - -const $instructionsText: ThemedStyle = ({colors}) => ({ - fontSize: 14, - lineHeight: 22, - fontFamily: "Montserrat-Regular", - color: colors.text, -}) - -const $settingsContainer: ThemedStyle = () => ({ - // Gap is handled by individual settings via isFirst/isLast marginBottom -}) - -const $noSettingsText: ThemedStyle = ({colors, spacing}) => ({ - fontSize: 14, - fontFamily: "Montserrat-Regular", - fontStyle: "italic", - textAlign: "center", - padding: spacing.s4, - color: colors.textDim, -}) - -const _$groupTitle: ThemedStyle = () => ({}) - -const $sectionTitleText: ThemedStyle = ({colors, spacing}) => ({ - fontSize: 14, - color: colors.text, - lineHeight: 20, - letterSpacing: 0, - marginBottom: spacing.s2, - marginTop: spacing.s3, -}) diff --git a/mobile/src/app/applet/webview.tsx b/mobile/src/app/applet/webview.tsx deleted file mode 100644 index 9f7ad40f67..0000000000 --- a/mobile/src/app/applet/webview.tsx +++ /dev/null @@ -1,534 +0,0 @@ -import {useLocalSearchParams} from "expo-router" -import {useRef, useState, useEffect, useCallback} from "react" -import {Platform, View} from "react-native" -import {WebView} from "react-native-webview" -import Animated, {useSharedValue, useAnimatedStyle, withTiming} from "react-native-reanimated" - -import {Header, Screen, Text} from "@/components/ignite" -import MiniappErrorScreen from "@/components/miniapps/MiniappErrorScreen" -import LoadingOverlay from "@/components/ui/LoadingOverlay" -import {useAppTheme} from "@/contexts/ThemeContext" -import {useNavigationStore} from "@/stores/navigation" -import restComms from "@/services/RestComms" -import {webviewBridge as miniComms} from "@mentra/island" -import {WebSocketStatus} from "@/services/ws-types" -import {SETTINGS, useSetting, useSettingsStore} from "@/stores/settings" -import {useAppStatusStore} from "@mentra/island" - -import miniappCatalog from "@/services/miniapps/MiniappCatalog" -import {useConnectionStore} from "@/stores/connection" -import {captureScreenshot} from "@/effects/CapsuleMenu" -import AppIcon from "@/components/home/AppIcon" -import {useSaferAreaInsets} from "@/contexts/SaferAreaContext" -import {buildMiniappGlobalsScript} from "@mentra/island" -import {useRegisterCapsule} from "@/stores/capsule" - -export default function AppWebView() { - const {webviewURL, appName, packageName, isLocal: isLocalParam} = useLocalSearchParams() - const isLocal = isLocalParam === "true" - const [hasError, setHasError] = useState(false) - const webViewRef = useRef(null) - - const [finalUrl, setFinalUrl] = useState(null) - const [isLoadingToken, setIsLoadingToken] = useState(!isLocal) - const [tokenError, setTokenError] = useState(null) - const [retryTrigger, setRetryTrigger] = useState(0) - const {goBack, push} = useNavigationStore.getState() - const viewShotRef = useRef(null) - const insets = useSaferAreaInsets() - const {theme} = useAppTheme() - const colorScheme = theme.isDark ? "dark" : "light" - - // Track if the server-side app start failed - const [appStartFailed, setAppStartFailed] = useState(false) - - // Track whether the WebView has back navigation history - const [webViewCanGoBack, setWebViewCanGoBack] = useState(false) - - // Allow back to exit if route params are invalid (no X button on that screen) - const hasValidParams = - typeof webviewURL === "string" && typeof appName === "string" && typeof packageName === "string" - - const {setForceGestureEnabled} = useNavigationStore.getState() - - // Back press handler for CapsuleMenu/Header buttons and Android back button. - const handleWebViewBack = useCallback(async () => { - console.log("WEBVIEW: handleWebViewBack()") - if (Platform.OS === "ios") { - await captureScreenshot(viewShotRef, packageName.toString(), insets.top) - } - if (!hasValidParams) { - if (Platform.OS === "android") { - goBack() - } - return - } - if (webViewCanGoBack && webViewRef.current) { - webViewRef.current.goBack() - } else { - if (Platform.OS === "android") { - captureScreenshot(viewShotRef, packageName.toString(), insets.top) - goBack() - } - } - }, [webViewCanGoBack, hasValidParams, goBack]) - - // Block native back gesture/button — route through handleWebViewBack for Android. - // focusEffectPreventBack(handleWebViewBack, false) - - // Dynamically toggle gesture handling based on webview navigation state: - // - Page 0 (no history): disable WebView's gesture, force-enable React Navigation's - // native swipe-back so user can exit miniapp with the real iOS animation. - // - Has history: enable WebView's gesture for in-webview navigation, - // React Navigation's gesture stays blocked by focusEffectPreventBack. - useEffect(() => { - if (!webViewCanGoBack) { - // Page 0: force React Navigation gesture on, WebView gesture off - setForceGestureEnabled(true) - } else { - // Has history: let focusEffectPreventBack handle it (gesture disabled), - // WebView's allowsBackForwardNavigationGestures handles in-webview swipe - setForceGestureEnabled(false) - } - - return () => setForceGestureEnabled(false) - }, [webViewCanGoBack, setForceGestureEnabled]) - - useRegisterCapsule({ - packageName: packageName as string, - viewShotRef, - visibleOnRoutes: ["/applet/webview"], - onBackPress: handleWebViewBack, - }) - - // Two conditions for showing the webview content: - // 1. WebView HTML has loaded (onLoadEnd fired) - const [isWebViewLoaded, setIsWebViewLoaded] = useState(false) - // 2. Server confirmed the app is running (loading=false, running=true in store) - // Local miniapps skip the server handshake, so they are confirmed immediately. - const [isServerConfirmed, setIsServerConfirmed] = useState(isLocal) - // Splash screen stays up until BOTH are true - const isWebViewReady = isWebViewLoaded && isServerConfirmed - - const webViewOpacity = useSharedValue(0) - const loadingOpacity = useSharedValue(1) - - const webViewAnimatedStyle = useAnimatedStyle(() => ({ - opacity: webViewOpacity.value, - })) - - const loadingAnimatedStyle = useAnimatedStyle(() => ({ - opacity: loadingOpacity.value, - })) - - if (!hasValidParams) { - return Missing required parameters - } - - // Watch the applet's store state for server confirmation. - // startApplet() sets loading=true, then refreshApplets() (at ~2s) fetches - // the real state from the server which sets loading=false. - // If running=false after server confirms, the app failed to start. - // Local miniapps skip this entirely — they don't need server confirmation. - // - // Un-latch on positive confirmation: running=true clears a prior failure - // (e.g. if the WS briefly dropped and the store observed running=false - // during a cross-pod reconnect before the cloud re-hydrated the session). - // - // Suppress failure latching while the WS isn't CONNECTED or during a short - // grace window after reconnect — the store is inherently stale in that - // window and false-negatives there were causing "Cannot reach" mid-session. - useEffect(() => { - if (isLocal) return - - const POST_RECONNECT_GRACE_MS = 5_000 - // The island store's `loading: true` stamp lands ~1 frame AFTER nav - // (beforeStart awaits an alert/network call before island.start() sets - // it). Without this grace, the screen mounts seeing stale loading=false - // running=false and immediately latches appStartFailed, causing the - // "Can't connect" screen to flash for ~500ms before the real load. - const MOUNT_GRACE_MS = 3_000 - const mountedAt = Date.now() - - const checkApplet = (state: {apps: Array<{packageName: string; loading: boolean; running: boolean}>}) => { - const applet = state.apps.find((a) => a.packageName === packageName) - if (!applet) return - if (applet.loading) return - - if (applet.running) { - setIsServerConfirmed(true) - setAppStartFailed(false) - return - } - - if (Date.now() - mountedAt < MOUNT_GRACE_MS) return - - const connState = useConnectionStore.getState() - if (connState.status !== WebSocketStatus.CONNECTED) return - const lastConnectedAt = connState.lastConnectedAt?.getTime() ?? 0 - if (lastConnectedAt && Date.now() - lastConnectedAt < POST_RECONNECT_GRACE_MS) return - - setAppStartFailed(true) - } - - checkApplet(useAppStatusStore.getState()) - - const unsubApplets = useAppStatusStore.subscribe(checkApplet) - const unsubConn = useConnectionStore.subscribe(() => { - checkApplet(useAppStatusStore.getState()) - }) - // Re-run once the mount grace expires so a failure that latched silently - // during the grace gets re-evaluated. - const graceTimer = setTimeout(() => { - checkApplet(useAppStatusStore.getState()) - }, MOUNT_GRACE_MS + 50) - return () => { - unsubApplets() - unsubConn() - clearTimeout(graceTimer) - } - }, [packageName, isLocal]) - - // Fade in webview once both conditions are met - useEffect(() => { - if (isWebViewReady) { - webViewOpacity.value = withTiming(1, {duration: 200}) - loadingOpacity.value = withTiming(0, {duration: 400}) - } - }, [isWebViewReady]) - - useEffect(() => { - // Local miniapps don't need token generation — use the URL directly. - if (isLocal) { - if (webviewURL) { - setFinalUrl(webviewURL as string) - console.log(`WEBVIEW: local miniapp URL: ${webviewURL}`) - } else { - setTokenError("Webview URL is missing.") - } - return - } - - const generateTokenAndSetUrl = async () => { - console.log("WEBVIEW: generateTokenAndSetUrl()") - setIsLoadingToken(true) - setTokenError(null) - - if (!packageName) { - setTokenError("App package name is missing. Cannot authenticate.") - setIsLoadingToken(false) - return - } - if (!webviewURL) { - setTokenError("Webview URL is missing.") - setIsLoadingToken(false) - return - } - - let res = await restComms.generateWebviewToken(packageName) - if (res.is_error()) { - console.error("Error generating webview token:", res.error) - setTokenError(`Couldn't securely connect to ${appName}. Please try again.`) - setIsLoadingToken(false) - return - } - - let tempToken = res.value - - res = await restComms.generateWebviewToken(packageName, "generate-webview-signed-user-token") - if (res.is_error()) { - console.warn("Failed to generate signed user token:", res.error) - } - let signedUserToken: string = res.value_or("") - - const cloudApiUrl = useSettingsStore.getState().getRestUrl() - - const url = new URL(webviewURL) - url.searchParams.set("aos_temp_token", tempToken) - if (signedUserToken) { - url.searchParams.set("aos_signed_user_token", signedUserToken) - } - if (cloudApiUrl) { - res = await restComms.hashWithApiKey(cloudApiUrl, packageName) - if (res.is_error()) { - console.error("Error hashing cloud API URL:", res.error) - setIsLoadingToken(false) - return - } - const checksum = res.value - url.searchParams.set("cloudApiUrl", cloudApiUrl) - url.searchParams.set("cloudApiUrlChecksum", checksum) - } - - setFinalUrl(url.toString()) - console.log(`Constructed final webview URL: ${url.toString()}`) - - setIsLoadingToken(false) - } - - generateTokenAndSetUrl() - }, [packageName, webviewURL, appName, retryTrigger, isLocal]) - - // Register with MiniComms for bridge messaging - useEffect(() => { - const sendToWebView = (message: string) => { - if (webViewRef.current) { - webViewRef.current.injectJavaScript(` - window.receiveNativeMessage(${message}); - `) - } - } - miniComms.setWebViewMessageHandler(packageName, sendToWebView) - return () => { - miniComms.setWebViewMessageHandler(packageName, undefined) - } - }, [packageName]) - - // Push color scheme changes into the WebView so miniapps using - // useColorScheme() can react. Skipped until the WebView has loaded. - useEffect(() => { - if (!webViewRef.current || !isWebViewLoaded) return - const envelope = JSON.stringify({ - payload: {type: "miniapp_color_scheme_change", colorScheme}, - }) - try { - webViewRef.current.injectJavaScript( - `window.dispatchEvent(new MessageEvent('message', {data: ${JSON.stringify(envelope)}})); true;`, - ) - } catch { - // noop - } - }, [colorScheme, isWebViewLoaded]) - - const handleWebViewMessage = (event: any) => { - // Cloud app webviews don't send miniapp SDK envelopes; local - // miniapp WebViews live in /applet/local and route via MentraUIRouter. - const _data = event.nativeEvent.data - } - - const handleLoadStart = () => { - // android tries to load the webview twice for some reason, and this does nothning so it's safe to disable: - console.log("WEBVIEW: handleLoadStart()") - // Reset states when starting to load - // setIsWebViewReady(false) - // webViewOpacity.value = 0 - // loadingOpacity.value = 1 - } - - const handleLoadEnd = () => { - console.log("WEBVIEW: handleLoadEnd()") - setHasError(false) - setIsWebViewLoaded(true) - setIsLoadingToken(false) - } - - const handleError = (syntheticEvent: any) => { - console.log("WEBVIEW: handleError()") - const {nativeEvent} = syntheticEvent - console.warn("WebView error: ", nativeEvent) - setHasError(true) - - const errorDesc = nativeEvent.description || "" - let friendlyMessage = `Unable to load ${appName}` - - if ( - errorDesc.includes("ERR_INTERNET_DISCONNECTED") || - errorDesc.includes("ERR_NETWORK_CHANGED") || - errorDesc.includes("ERR_CONNECTION_FAILED") || - errorDesc.includes("ERR_NAME_NOT_RESOLVED") - ) { - friendlyMessage = "No internet connection. Please check your network settings and try again." - } else if (errorDesc.includes("ERR_CONNECTION_TIMED_OUT") || errorDesc.includes("ERR_TIMED_OUT")) { - friendlyMessage = "Connection timed out. Please check your internet connection and try again." - } else if (errorDesc.includes("ERR_CONNECTION_REFUSED")) { - friendlyMessage = `Unable to connect to ${appName}. Please try again later.` - } else if (errorDesc.includes("ERR_SSL") || errorDesc.includes("ERR_CERT")) { - friendlyMessage = "Security error. Please check your device's date and time settings." - } else if (errorDesc) { - friendlyMessage = `Unable to load ${appName}. Please try again.` - } - - setTokenError(friendlyMessage) - } - - // const screenshotComponent = () => { - // const screenshot = useAppStatusStore.getState().apps.find((a) => a.packageName === packageName)?.screenshot - // if (screenshot) { - // return - // } - // return null - // } - - const renderLoadingOverlay = () => { - const app = useAppStatusStore.getState().apps.find((a) => a.packageName === packageName) - - // disabled for now: - // const screenshot = screenshotComponent() - // if (screenshot) { - // return ( - // - // {screenshot} - // - // ) - // } - - if (!app) { - return ( - - - - ) - } - - // force loading to false for the app icon: - let appCopy = {...app, loading: false} - - return ( - - {/* show the app icon and app name */} - - - - {/* */} - - - - ) - } - - // Show error screen if: server-side start failed, token generation failed, or webview failed to load - const showError = appStartFailed || (tokenError && !isLoadingToken) || hasError - const errorMessage = appStartFailed - ? `${appName} couldn't be started. The miniapp may be temporarily unavailable.` - : tokenError || `Unable to load ${appName}. Please check your connection and try again.` - - if (showError) { - return ( - <> - - { - setAppStartFailed(false) - setHasError(false) - setTokenError(null) - setFinalUrl(null) - setIsWebViewLoaded(false) - setIsServerConfirmed(false) - webViewOpacity.value = 0 - loadingOpacity.value = 1 - // Re-send the start request and poll for confirmation - void miniappCatalog.retryStart(packageName as string) - setRetryTrigger((prev) => prev + 1) - }} - /> - - - ) - } - - // Build the window.MentraOS globals (safeAreaInsets, capsuleMenu, etc.) via - // the shared util so cloud and local miniapps see identical shapes. - const miniappGlobalsScript = buildMiniappGlobalsScript({ - packageName: packageName as string, - miniappLocal: isLocal, - safeAreaInsets: { - top: insets.top, - bottom: Platform.OS === "android" ? insets.bottom : 0, - left: insets.left, - right: insets.right, - }, - colorScheme, - }) - - return ( - - {/* rainbow bars for debugging insets / screenshots */} - {/* - - - - - - - - - - - - - - - - - - - - - - - - - - - */} - - {renderLoadingOverlay()} - {finalUrl && ( - - setWebViewCanGoBack(navState.canGoBack)} - automaticallyAdjustContentInsets={false} - contentInsetAdjustmentBehavior="never" - injectedJavaScriptBeforeContentLoaded={miniappGlobalsScript} - injectedJavaScript={` - const meta = document.createElement('meta'); - meta.setAttribute('name', 'viewport'); - meta.setAttribute('content', 'width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no'); - document.getElementsByTagName('head')[0].appendChild(meta); - true; - `} - /> - - )} - - - ) -} diff --git a/mobile/src/app/asg/gallery-settings.tsx b/mobile/src/app/asg/gallery-settings.tsx index 1266a14b55..4bd6a33af8 100644 --- a/mobile/src/app/asg/gallery-settings.tsx +++ b/mobile/src/app/asg/gallery-settings.tsx @@ -9,8 +9,7 @@ import {RouteButton} from "@/components/ui/RouteButton" import {useAppTheme} from "@/contexts/ThemeContext" import {useNavigationStore} from "@/stores/navigation" import {translate} from "@/i18n" -import {gallerySettingsService} from "@/services/asg/gallerySettingsService" -import {localStorageService} from "@/services/asg/localStorageService" +import {gallerySettingsService, localStorageService} from "@mentra/island" import {useGallerySyncStore} from "@/stores/gallerySync" import {SETTINGS, useSetting} from "@/stores/settings" import {ThemedStyle} from "@/theme" diff --git a/mobile/src/app/miniapps/gallery/gallery-settings.tsx b/mobile/src/app/miniapps/gallery/gallery-settings.tsx index 999770c310..ba2619597c 100644 --- a/mobile/src/app/miniapps/gallery/gallery-settings.tsx +++ b/mobile/src/app/miniapps/gallery/gallery-settings.tsx @@ -9,8 +9,7 @@ import {RouteButton} from "@/components/ui/RouteButton" import {useAppTheme} from "@/contexts/ThemeContext" import {useNavigationStore} from "@/stores/navigation" import {translate} from "@/i18n" -import {gallerySettingsService} from "@/services/asg/gallerySettingsService" -import {localStorageService} from "@/services/asg/localStorageService" +import {gallerySettingsService, localStorageService} from "@mentra/island" import {useGallerySyncStore} from "@/stores/gallerySync" import {SETTINGS, useSetting} from "@/stores/settings" import {ThemedStyle} from "@/theme" diff --git a/mobile/src/app/miniapps/settings/controller.tsx b/mobile/src/app/miniapps/settings/controller.tsx index edbf28c772..f6ccc624d5 100644 --- a/mobile/src/app/miniapps/settings/controller.tsx +++ b/mobile/src/app/miniapps/settings/controller.tsx @@ -13,7 +13,7 @@ import {Group} from "@/components/ui" import {RouteButton} from "@/components/ui/RouteButton" import {DeviceTypes} from "@/../../cloud/packages/types/src" -import BluetoothSdk from "@mentra/bluetooth-sdk-internal" +import {toolkit} from "@mentra/island" import {EmptyState} from "@/components/glasses/info/EmptyState" import {showAlert} from "@/contexts/ModalContext" @@ -35,7 +35,7 @@ function DeviceSettings() { }) if (result === 1) { try { - BluetoothSdk.forgetController() + toolkit.glasses.controller.forget() } catch (e) { console.log(e) } @@ -55,7 +55,7 @@ function DeviceSettings() { }) if (result === 1) { - BluetoothSdk.disconnectController() + toolkit.glasses.controller.disconnect() } } diff --git a/mobile/src/app/miniapps/settings/debug.tsx b/mobile/src/app/miniapps/settings/debug.tsx index a20ab41d20..c6c5e296d1 100644 --- a/mobile/src/app/miniapps/settings/debug.tsx +++ b/mobile/src/app/miniapps/settings/debug.tsx @@ -5,7 +5,6 @@ import {ScrollView, View} from "react-native" import BackendUrl from "@/components/dev/BackendUrl" import CloudUrl from "@/components/dev/CloudUrl" import OtaVersionUrl from "@/components/dev/OtaVersionUrl" -import StoreUrl from "@/components/dev/StoreUrl" import {Header, Icon, Screen, Text} from "@/components/ignite" import SelectSetting from "@/components/settings/SelectSetting" import ToggleSetting from "@/components/settings/ToggleSetting" @@ -16,7 +15,7 @@ import {useAppTheme} from "@/contexts/ThemeContext" import {useNavigationStore} from "@/stores/navigation" import {translate} from "@/i18n" import {SETTINGS, useSetting} from "@/stores/settings" -import navigationService from "@/services/NavigationService" +import {navigationService} from "@mentra/island" import ws from "@/services/WebSocketManager" import socketComms from "@/services/SocketComms" import showAlert from "@/utils/AlertUtils" @@ -275,12 +274,6 @@ export default function DebugSettingsScreen() { onValueChange={async (value) => { const frameSize = parseInt(value, 10) setLc3FrameSize(frameSize) - // Apply immediately to native encoder and cloud - try { - await socketComms.configureAudioFormat() - } catch (err) { - console.error("Failed to apply LC3 frame size:", err) - } }} description="Higher bitrates improve transcription quality but use more bandwidth." /> @@ -292,8 +285,6 @@ export default function DebugSettingsScreen() { - - {/* Super mode only: a wrong OTA manifest can brick glasses */} {superMode && } diff --git a/mobile/src/app/miniapps/settings/glasses.tsx b/mobile/src/app/miniapps/settings/glasses.tsx index f0a9d370e5..66872589a9 100644 --- a/mobile/src/app/miniapps/settings/glasses.tsx +++ b/mobile/src/app/miniapps/settings/glasses.tsx @@ -14,14 +14,13 @@ import {Group} from "@/components/ui" import {RouteButton} from "@/components/ui/RouteButton" import {Capabilities, DeviceTypes, getModelCapabilities} from "@/../../cloud/packages/types/src" -import BluetoothSdk from "@mentra/bluetooth-sdk" import OtaProgressSection from "@/components/glasses/OtaProgressSection" import {BatteryStatus} from "@/components/glasses/info/BatteryStatus" import {EmptyState} from "@/components/glasses/info/EmptyState" import {ButtonSettings} from "@/components/glasses/settings/ButtonSettings" import BrightnessSetting from "@/components/settings/BrightnessSetting" -import {useApps} from "@mentra/island" +import {toolkit, useApps} from "@mentra/island" // import showAlert from "@/utils/AlertUtils" import {showAlert} from "@/contexts/ModalContext" @@ -56,7 +55,7 @@ function DeviceSettings() { options: {allowDismiss: false}, }) if (result === 1) { - BluetoothSdk.forget() + toolkit.glasses.forget() // useAppletStatusStore.getState().stopAllApplets() // useAppletStatusStore.getState().refreshApplets() // // give us a second to forget the glasses before going back @@ -75,7 +74,7 @@ function DeviceSettings() { }) if (result === 1) { - BluetoothSdk.disconnect() + toolkit.glasses.disconnect() } } diff --git a/mobile/src/app/miniapps/settings/speech.tsx b/mobile/src/app/miniapps/settings/speech.tsx index c923aecdaa..ff63695e78 100644 --- a/mobile/src/app/miniapps/settings/speech.tsx +++ b/mobile/src/app/miniapps/settings/speech.tsx @@ -1,5 +1,4 @@ import {useFocusEffect} from "@react-navigation/native" -import BluetoothSdk from "@mentra/bluetooth-sdk" import {useCallback, useEffect, useRef, useState} from "react" import {ActivityIndicator, BackHandler, Platform, ScrollView, View} from "react-native" @@ -12,6 +11,7 @@ import {translate} from "@/i18n" import { offlineSpeechModelService, sttModelManager as STTModelManager, + toolkit, ttsModelManager as TTSModelManager, useStopAll, type OfflineModelDownloadStatus as DownloadStatus, @@ -242,7 +242,7 @@ export default function SpeechSettingsScreen() { if (target == null) return try { await STTModelManager.activateLanguage(target) - await BluetoothSdk.restartTranscriber() + await toolkit.speech.restartTranscriber() } catch (error: any) { console.error("STT activation failed:", error) showAlert("Error", error?.message ?? "Failed to switch language", [{text: "OK"}]) @@ -262,7 +262,7 @@ export default function SpeechSettingsScreen() { ) await refreshLists() await STTModelManager.activateLanguage(code) - await BluetoothSdk.restartTranscriber() + await toolkit.speech.restartTranscriber() setSttCurrent(code) STTModelManager.setCurrentLanguage(code) sttDesiredRef.current = code diff --git a/mobile/src/app/miniapps/settings/stress-test.tsx b/mobile/src/app/miniapps/settings/stress-test.tsx index ac8431194d..07af2e3a1f 100644 --- a/mobile/src/app/miniapps/settings/stress-test.tsx +++ b/mobile/src/app/miniapps/settings/stress-test.tsx @@ -16,7 +16,7 @@ import {useNavigationStore} from "@/stores/navigation" import {useStressTestStore} from "@/stores/stressTest" import {buildDummyMiniappHtml} from "@/utils/stressTest/dummyHtml" import BluetoothSdk from "@mentra/bluetooth-sdk-internal" -import {miniappRunningRegistry} from "@mentra/island" +import {miniappRunningRegistry, toolkit} from "@mentra/island" const POLL_MS = 1000 @@ -28,7 +28,7 @@ export default function StressTest() { let id: ReturnType | null = null const tick = () => { try { - const mb = BluetoothSdk.getMemoryMB() + const mb = toolkit.dev.getMemoryMB() setResidentMB(mb) if (active) { // eslint-disable-next-line no-console @@ -161,7 +161,7 @@ export default function StressTest() { label="Spawn 1 JSContext" subtitle="Measures per-context memory cost" onPress={async () => { - const baseline = BluetoothSdk.getMemoryMB() + const baseline = toolkit.dev.getMemoryMB() const result = (BluetoothSdk as any).jscSpawnAndMeasure(1, baseline) console.log("STRESS: jsc-spike", JSON.stringify(result)) }} @@ -169,7 +169,7 @@ export default function StressTest() { { - const baseline = BluetoothSdk.getMemoryMB() + const baseline = toolkit.dev.getMemoryMB() const result = (BluetoothSdk as any).jscSpawnAndMeasure(10, baseline) console.log("STRESS: jsc-spike", JSON.stringify(result)) }} @@ -177,7 +177,7 @@ export default function StressTest() { { - const baseline = BluetoothSdk.getMemoryMB() + const baseline = toolkit.dev.getMemoryMB() const result = (BluetoothSdk as any).jscSpawnAndMeasure(50, baseline) console.log("STRESS: jsc-spike", JSON.stringify(result)) }} diff --git a/mobile/src/app/miniapps/store/store.tsx b/mobile/src/app/miniapps/store/store.tsx deleted file mode 100644 index f6d2104d30..0000000000 --- a/mobile/src/app/miniapps/store/store.tsx +++ /dev/null @@ -1,265 +0,0 @@ -import {useFocusEffect} from "@react-navigation/native" -import {useLocalSearchParams} from "expo-router" -import {useState, useCallback, useMemo, useEffect, useRef} from "react" -import {View, ViewStyle, ActivityIndicator, BackHandler, TextStyle} from "react-native" -import {WebView} from "react-native-webview" - -import {MentraLogoStandalone} from "@/components/brands/MentraLogoStandalone" -import {Text, Screen, Header} from "@/components/ignite" -import InternetConnectionFallbackComponent from "@/components/ui/InternetConnectionFallbackComponent" -import {useAppStoreWebviewPrefetch} from "@/contexts/AppStoreContext" -import {useAppTheme} from "@/contexts/ThemeContext" -import {useNavigationStore} from "@/stores/navigation" -import {useRefresh} from "@mentra/island" -import {ThemedStyle} from "@/theme" -import {useRegisterCapsule} from "@/stores/capsule" -export default function AppStoreWeb() { - const [_webviewLoading, setWebviewLoading] = useState(true) - const [hasError, setHasError] = useState(false) - const [errorMessage, setErrorMessage] = useState("") - const {packageName} = useLocalSearchParams() - const [canGoBack, setCanGoBack] = useState(false) - const [isAuthReady, setIsAuthReady] = useState(false) - const {push} = useNavigationStore.getState() - const {appStoreUrl, webViewRef: prefetchedWebviewRef} = useAppStoreWebviewPrefetch() - const refreshApplets = useRefresh() - const {theme, themed} = useAppTheme() - const viewShotRef = useRef(null) - - useRegisterCapsule({ - packageName: "com.mentra.store", - viewShotRef, - visibleOnRoutes: ["/miniapps/store"], - offsetRight: theme.spacing.s2, - }) - - // Construct the final URL with packageName if provided - const finalUrl = useMemo(() => { - if (!appStoreUrl) return null - - const url = new URL(appStoreUrl) - console.log("AppStoreWeb: appStoreUrl", appStoreUrl) - console.log("AppStoreWeb: packageName", packageName) - if (packageName && typeof packageName === "string") { - // If packageName is provided, update the path to point to the app details page - url.pathname = `/package/${packageName}` - } - url.searchParams.set("theme", theme.isDark ? "dark" : "light") - console.log("AppStoreWeb: finalUrl", url.toString()) - return url.toString() - }, [appStoreUrl, packageName]) - - // Reset auth ready state when URL changes (e.g., new tokens, theme change) - useEffect(() => { - setIsAuthReady(false) - }, [finalUrl]) - - // prevents the auth getting stuck after a hot-reload during development: - useEffect(() => { - // reload the webview when auth is false: - if (!isAuthReady) { - if (prefetchedWebviewRef.current) { - prefetchedWebviewRef.current.reload() - } - } - }, [isAuthReady]) - - const handleError = (syntheticEvent: any) => { - const {nativeEvent} = syntheticEvent - console.error("WebView error:", nativeEvent) - setWebviewLoading(false) - setHasError(true) - - // Parse error message to show user-friendly text - const errorDesc = nativeEvent.description || "" - let friendlyMessage = "Unable to load the App Store" - - if ( - errorDesc.includes("ERR_INTERNET_DISCONNECTED") || - errorDesc.includes("ERR_NETWORK_CHANGED") || - errorDesc.includes("ERR_CONNECTION_FAILED") || - errorDesc.includes("ERR_NAME_NOT_RESOLVED") - ) { - friendlyMessage = "No internet connection. Please check your network settings and try again." - } else if (errorDesc.includes("ERR_CONNECTION_TIMED_OUT") || errorDesc.includes("ERR_TIMED_OUT")) { - friendlyMessage = "Connection timed out. Please check your internet connection and try again." - } else if (errorDesc.includes("ERR_CONNECTION_REFUSED")) { - friendlyMessage = "Unable to connect to the App Store server. Please try again later." - } else if (errorDesc.includes("ERR_SSL") || errorDesc.includes("ERR_CERT")) { - friendlyMessage = "Security error. Please check your device's date and time settings." - } else if (errorDesc) { - // For any other errors, just show a generic message without the technical error - friendlyMessage = "Unable to load the App Store. Please try again." - } - - setErrorMessage(friendlyMessage) - } - - const handleRetry = () => { - setHasError(false) - setErrorMessage("") - if (prefetchedWebviewRef.current) { - prefetchedWebviewRef.current.reload() - } - } - - // Handle messages from WebView - const handleWebViewMessage = (event: any) => { - try { - const data = JSON.parse(event.nativeEvent.data) - - // Handle auth ready message from store - hides loading overlay - if (data.type === "AUTH_READY") { - console.log("AppStoreWeb: Received AUTH_READY from store") - setIsAuthReady(true) - return - } - - if ((data.type === "OPEN_APP_SETTINGS" || data.type === "OPEN_TPA_SETTINGS") && data.packageName) { - // Navigate to TPA settings page - push("/applet/settings", {packageName: data.packageName}) - } - } catch (error) { - console.error("Error handling WebView message:", error) - } - } - - // Handle Android back button press - useFocusEffect( - useCallback(() => { - const onBackPress = () => { - if (prefetchedWebviewRef.current && canGoBack) { - prefetchedWebviewRef.current.goBack() - return true // Prevent default back action - } - return false // Allow default back action (close screen) - } - - const subscription = BackHandler.addEventListener("hardwareBackPress", onBackPress) - - return () => subscription.remove() // Cleanup listener on blur - }, [canGoBack, prefetchedWebviewRef]), // Re-run effect if canGoBack or ref changes - ) - - // propagate any changes in app lists when this screen is unmounted: - useFocusEffect( - useCallback(() => { - return async () => { - await refreshApplets() - } - }, []), - ) - - // Show loading state while getting the URL - if (!finalUrl) { - return ( - -
} /> - - - - - - ) - } - - if (hasError) { - return ( - -
} /> - - - ) - } - - // If the prefetched WebView is ready, show it in the correct style - return ( - - - {/* Show the prefetched WebView, but now visible and full size */} - setWebviewLoading(true)} - onLoadEnd={() => { - setWebviewLoading(false) - setIsAuthReady(true) - }} - onError={handleError} - onNavigationStateChange={(navState) => setCanGoBack(navState.canGoBack)} - onMessage={handleWebViewMessage} - javaScriptEnabled={true} - domStorageEnabled={true} - startInLoadingState={false} - scalesPageToFit={false} - bounces={false} - scrollEnabled={true} - // Inject CSS/JS to disable zoom and selection - injectedJavaScript={` - document.body.style.userSelect = 'none'; - document.body.style.webkitUserSelect = 'none'; - document.body.style.webkitTouchCallout = 'none'; - - document.addEventListener('gesturestart', function(e) { - e.preventDefault(); - }); - - const meta = document.createElement('meta'); - meta.name = 'viewport'; - meta.content = 'width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no'; - document.head.appendChild(meta); - true; - - true; - `} - /> - {/* Loading overlay - stays visible until store confirms auth ready */} - {!isAuthReady && ( - - - - - )} - - - ) -} - -// Themed styles using ThemedStyle pattern -const $loadingContainer: ThemedStyle = ({colors}) => ({ - flex: 1, - alignItems: "center", - justifyContent: "center", - backgroundColor: colors.background, -}) - -const $loadingOverlay: ThemedStyle = () => ({ - alignItems: "center", - backgroundColor: "rgba(0, 0, 0, 0.3)", - bottom: 0, - justifyContent: "center", - left: 0, - position: "absolute", - right: 0, - top: 0, -}) - -const $loadingText: ThemedStyle = ({colors, spacing}) => ({ - fontSize: spacing.s4, - marginTop: 10, - color: colors.text, -}) - -const $webView: ThemedStyle = ({colors}) => ({ - flex: 1, - backgroundColor: colors.background, -}) diff --git a/mobile/src/app/ota/check-for-updates.tsx b/mobile/src/app/ota/check-for-updates.tsx index 66905de1e3..f3c755bd2a 100644 --- a/mobile/src/app/ota/check-for-updates.tsx +++ b/mobile/src/app/ota/check-for-updates.tsx @@ -1,7 +1,6 @@ import {useFocusEffect} from "expo-router" import {useEffect, useState, useCallback, useRef} from "react" import {View, ActivityIndicator} from "react-native" -import BluetoothSdk from "@mentra/bluetooth-sdk" import {MINIMUM_OTA_STATUS_BUILD} from "@/app/ota/otaProgressTimeouts" import {MentraLogoStandalone} from "@/components/brands/MentraLogoStandalone" @@ -14,7 +13,7 @@ import {getAsgOtaVersionUrl} from "@/services/asg/asgOtaVersionUrl" import {translate} from "@/i18n/translate" import {isGlassesConnected, selectGlassesConnected, useGlassesStore, waitForGlassesState} from "@/stores/glasses" import {SETTINGS, useSetting} from "@/stores/settings" -import {BgTimer} from "@mentra/island" +import {BgTimer, toolkit} from "@mentra/island" type CheckState = "checking" | "update_available" | "no_update" | "error" @@ -101,7 +100,7 @@ export default function OtaCheckForUpdatesScreen() { // Request version info since we don't have it yet console.log("OTA: Requesting version_info from glasses") - void BluetoothSdk.requestVersionInfo().catch((error) => { + void toolkit.glasses.requestVersionInfo().catch((error) => { console.warn("OTA: Failed to request version_info from glasses:", error) }) @@ -122,7 +121,7 @@ export default function OtaCheckForUpdatesScreen() { } // Match OtaUpdateChecker home path: BES often arrives late in version_info_3 (chip init after reflash). - void BluetoothSdk.requestVersionInfo().catch((error) => { + void toolkit.glasses.requestVersionInfo().catch((error) => { console.warn("OTA: Failed to refresh version_info before BES wait:", error) }) @@ -175,7 +174,7 @@ export default function OtaCheckForUpdatesScreen() { // Refresh version_info (build / fw) in case the store still held values from a prior session // before the native clear-on-connect + glasses_ready re-query completed. console.log("OTA: Requesting fresh version_info from glasses before HTTP compare") - void BluetoothSdk.requestVersionInfo().catch((error) => { + void toolkit.glasses.requestVersionInfo().catch((error) => { console.warn("OTA: Failed to refresh version_info before OTA compare:", error) }) diff --git a/mobile/src/app/ota/progress.tsx b/mobile/src/app/ota/progress.tsx index 504d1fcc58..c1e0a39234 100644 --- a/mobile/src/app/ota/progress.tsx +++ b/mobile/src/app/ota/progress.tsx @@ -1,5 +1,6 @@ import BluetoothSdk from "@mentra/bluetooth-sdk-internal" import type {OtaProgress, OtaStatus} from "@mentra/bluetooth-sdk-internal" +import {toolkit} from "@mentra/island" import {useCallback, useEffect, useRef, useState} from "react" import {View, ActivityIndicator} from "react-native" @@ -343,7 +344,7 @@ export default function OtaProgressScreen() { const glassesState = useGlassesStore.getState() const otaVersionUrl = getAsgOtaVersionUrl(glassesState.otaVersionUrl, glassesState.buildNumber) console.log(`[OTA_PROGRESS] sending ota_start with manifest URL: ${otaVersionUrl}`) - await BluetoothSdk.startOtaUpdate(otaVersionUrl) + await toolkit.ota.install(otaVersionUrl) } catch (err) { console.warn("[OTA_PROGRESS] sendOtaStart threw", err) clearRetryTimeout() diff --git a/mobile/src/app/pairing/btclassic.tsx b/mobile/src/app/pairing/btclassic.tsx index a6f4fb689f..36470cf280 100644 --- a/mobile/src/app/pairing/btclassic.tsx +++ b/mobile/src/app/pairing/btclassic.tsx @@ -4,7 +4,7 @@ import {OnboardingGuide, OnboardingStep} from "@/components/onboarding/Onboardin import {translate} from "@/i18n" import {focusEffectPreventBack, usePushPrevious} from "@/contexts/NavigationHistoryContext" import {useGlassesStore} from "@/stores/glasses" -import BluetoothSdk from "@mentra/bluetooth-sdk" +import {toolkit} from "@mentra/island" import {SETTINGS, useSetting} from "@/stores/settings" import {SettingsNavigationUtils} from "@/utils/SettingsNavigationUtils" import {useCoreStore} from "@/stores/core" @@ -24,7 +24,7 @@ export default function BtClassicPairingScreen() { focusEffectPreventBack() const handleSuccess = () => { - BluetoothSdk.connectDefault().catch((error) => { + toolkit.glasses.connectDefault().catch((error) => { console.error("Failed to connect default glasses after Bluetooth Classic pairing:", error) }) pushPrevious() diff --git a/mobile/src/app/pairing/failure.tsx b/mobile/src/app/pairing/failure.tsx index 6b799c8346..058d040932 100644 --- a/mobile/src/app/pairing/failure.tsx +++ b/mobile/src/app/pairing/failure.tsx @@ -1,4 +1,4 @@ -import BluetoothSdk from "@mentra/bluetooth-sdk" +import {toolkit} from "@mentra/island" import {useLocalSearchParams} from "expo-router" import {useEffect} from "react" import {View} from "react-native" @@ -30,7 +30,7 @@ export default function PairingFailureScreen() { }, []) const handleRetry = () => { - BluetoothSdk.forget() + toolkit.glasses.forget() clearHistoryAndGoHome() push("/pairing/select-glasses-model") } diff --git a/mobile/src/app/pairing/loading.tsx b/mobile/src/app/pairing/loading.tsx index d8cb27b05f..8ad6ef84f7 100644 --- a/mobile/src/app/pairing/loading.tsx +++ b/mobile/src/app/pairing/loading.tsx @@ -1,5 +1,5 @@ import {useRoute} from "@react-navigation/native" -import {waitForGlassesReady, BluetoothSdk} from "@mentra/island" +import {waitForGlassesReady, toolkit} from "@mentra/island" import type {PairFailureEvent, GlassesNotReadyEvent} from "@mentra/island" import {useCallback, useEffect, useRef, useState} from "react" import {View} from "react-native" @@ -26,11 +26,11 @@ export default function GlassesPairingLoadingScreen() { const [showGlassesBooting, setShowGlassesBooting] = useState(false) useEffect(() => { - let sub = BluetoothSdk.addListener("glasses_not_ready", (_event: GlassesNotReadyEvent) => { + let unsub = toolkit.pairing.onGlassesNotReady((_event: GlassesNotReadyEvent) => { setShowGlassesBooting(true) }) return () => { - sub.remove() + unsub() } }, []) @@ -42,7 +42,7 @@ export default function GlassesPairingLoadingScreen() { const handlePairFailure = useCallback( (error: string) => { - BluetoothSdk.forget() + toolkit.glasses.forget() if (error === "errors:pairNeedDisconnect") { replace("/pairing/unpair-even", {deviceModel: deviceModel}) return @@ -53,11 +53,11 @@ export default function GlassesPairingLoadingScreen() { ) useEffect(() => { - let sub = BluetoothSdk.addListener("pair_failure", (event: PairFailureEvent) => { + let unsub = toolkit.pairing.onPairFailure((event: PairFailureEvent) => { handlePairFailure(event.error) }) return () => { - sub.remove() + unsub() } }, [handlePairFailure]) diff --git a/mobile/src/app/pairing/prep-controller.tsx b/mobile/src/app/pairing/prep-controller.tsx index cdb3e4c89a..ac77253990 100644 --- a/mobile/src/app/pairing/prep-controller.tsx +++ b/mobile/src/app/pairing/prep-controller.tsx @@ -12,7 +12,7 @@ import {showAlert} from "@/utils/AlertUtils" import {PermissionFeatures, checkConnectivityRequirementsUI, requestFeaturePermissions} from "@/utils/PermissionsUtils" import {useState} from "react" import GlassesTroubleshootingModal from "@/components/glasses/GlassesTroubleshootingModal" -import BluetoothSdk from "@mentra/bluetooth-sdk-internal" +import {toolkit} from "@mentra/island" type BluetoothPermission = Permission | "android.permission.BLUETOOTH" | "android.permission.BLUETOOTH_ADMIN" @@ -195,7 +195,7 @@ export default function PairingPrepScreen() { // skip pairing for simulated glasses: if (deviceModel.startsWith(DeviceTypes.SIMULATED)) { - await BluetoothSdk.connectSimulated() + await toolkit.glasses.connectSimulated() clearHistoryAndGoHome() return } diff --git a/mobile/src/app/pairing/prep.tsx b/mobile/src/app/pairing/prep.tsx index 088404e0a7..14c1eba123 100644 --- a/mobile/src/app/pairing/prep.tsx +++ b/mobile/src/app/pairing/prep.tsx @@ -15,8 +15,7 @@ import {useState} from "react" import GlassesTroubleshootingModal from "@/components/glasses/GlassesTroubleshootingModal" import {OnboardingGuide, OnboardingStep} from "@/components/onboarding/OnboardingGuide" import {CDN_BASE_URL} from "@/constants/appConfig" -import {useAppStatusStore} from "@mentra/island" -import BluetoothSdk from "@mentra/bluetooth-sdk-internal" +import {toolkit, useAppStatusStore} from "@mentra/island" type BluetoothPermission = Permission | "android.permission.BLUETOOTH" | "android.permission.BLUETOOTH_ADMIN" @@ -202,7 +201,7 @@ export default function PairingPrepScreen() { // skip pairing for simulated glasses: if (deviceModel.startsWith(DeviceTypes.SIMULATED)) { - await BluetoothSdk.connectSimulated() + await toolkit.glasses.connectSimulated() clearHistoryAndGoHome() return } diff --git a/mobile/src/app/pairing/scan-controller.tsx b/mobile/src/app/pairing/scan-controller.tsx index 0f57a64fcc..4f9ac6a7c2 100644 --- a/mobile/src/app/pairing/scan-controller.tsx +++ b/mobile/src/app/pairing/scan-controller.tsx @@ -1,4 +1,5 @@ -import BluetoothSdk, {type Device, type DeviceModel} from "@mentra/bluetooth-sdk-internal" +import {type Device, type DeviceModel} from "@mentra/bluetooth-sdk-internal" +import {toolkit} from "@mentra/island" import {useLocalSearchParams} from "expo-router" import {useEffect, useState} from "react" import {ActivityIndicator, Image, Platform, ScrollView, TouchableOpacity, View} from "react-native" @@ -37,8 +38,8 @@ export default function SelectGlassesBluetoothScreen() { if (event && event.actionType !== "GO_BACK" && event.actionType !== "POP") { return } - BluetoothSdk.disconnectController() - BluetoothSdk.forgetController() + toolkit.glasses.controller.disconnect() + toolkit.glasses.controller.forget() goBack() }, true) @@ -53,7 +54,7 @@ export default function SelectGlassesBluetoothScreen() { useEffect(() => { const initializeAndSearchForDevices = async () => { try { - await BluetoothSdk.startScan(deviceModel) + await toolkit.pairing.scan(deviceModel) } catch (error) { console.error("Failed to start controller scan:", error) } @@ -92,7 +93,7 @@ export default function SelectGlassesBluetoothScreen() { const startPairing = async (device: Device) => { setTimeout(() => { - BluetoothSdk.connect(device).catch((error) => { + toolkit.pairing.pair(device).catch((error) => { console.error("Failed to connect to controller:", error) }) }, 2000) diff --git a/mobile/src/app/pairing/scan.tsx b/mobile/src/app/pairing/scan.tsx index 233b695e4a..7e90b030c9 100644 --- a/mobile/src/app/pairing/scan.tsx +++ b/mobile/src/app/pairing/scan.tsx @@ -1,4 +1,5 @@ -import BluetoothSdk, {type Device, type DeviceModel} from "@mentra/bluetooth-sdk" +import {type Device, type DeviceModel} from "@mentra/bluetooth-sdk" +import {toolkit} from "@mentra/island" import {useLocalSearchParams} from "expo-router" import {useEffect, useState} from "react" import {ActivityIndicator, Image, Platform, ScrollView, TouchableOpacity, View} from "react-native" @@ -44,8 +45,8 @@ export default function SelectGlassesBluetoothScreen() { if (event && event.actionType !== "GO_BACK" && event.actionType !== "POP") { return } - BluetoothSdk.disconnect() - BluetoothSdk.forget() + toolkit.glasses.disconnect() + toolkit.glasses.forget() goBack() }, true) @@ -60,7 +61,7 @@ export default function SelectGlassesBluetoothScreen() { useEffect(() => { const initializeAndSearchForDevices = async () => { try { - await BluetoothSdk.startScan(deviceModel) + await toolkit.pairing.scan(deviceModel) } catch (error) { console.error("Failed to start glasses scan:", error) } @@ -101,7 +102,7 @@ export default function SelectGlassesBluetoothScreen() { const deviceTypesWithBtClassic = [DeviceTypes.LIVE] if (Platform.OS === "android" || bluetoothClassicConnected || !deviceTypesWithBtClassic.includes(device.model as DeviceTypes)) { setTimeout(() => { - BluetoothSdk.connect(device).catch((error) => { + toolkit.pairing.pair(device).catch((error) => { console.error("Failed to connect to glasses:", error) }) }, 2000) @@ -110,7 +111,7 @@ export default function SelectGlassesBluetoothScreen() { return } - await BluetoothSdk.setDefaultDevice(device) + await toolkit.pairing.setDefault(device) setDeviceName(device.name) // pair bt classic first: replace("/pairing/btclassic") diff --git a/mobile/src/app/pairing/select-glasses-model.tsx b/mobile/src/app/pairing/select-glasses-model.tsx index 533ca8a8fb..8f1a2bf675 100644 --- a/mobile/src/app/pairing/select-glasses-model.tsx +++ b/mobile/src/app/pairing/select-glasses-model.tsx @@ -1,5 +1,5 @@ import {DeviceTypes} from "@/../../cloud/packages/types/src" -import BluetoothSdk from "@mentra/bluetooth-sdk" +import {toolkit} from "@mentra/island" import {useFocusEffect} from "expo-router" import {useCallback} from "react" import {View, TouchableOpacity, Platform, ScrollView, Image} from "react-native" @@ -27,7 +27,7 @@ export default function SelectGlassesModelScreen() { // when this screen is focused, forget any glasses that may be paired: useFocusEffect( useCallback(() => { - BluetoothSdk.forget() + toolkit.glasses.forget() return () => {} }, []), ) diff --git a/mobile/src/app/pairing/unpair-even.tsx b/mobile/src/app/pairing/unpair-even.tsx index 8f82eb1b67..206736f553 100644 --- a/mobile/src/app/pairing/unpair-even.tsx +++ b/mobile/src/app/pairing/unpair-even.tsx @@ -1,6 +1,6 @@ import {useRoute} from "@react-navigation/native" import {View} from "react-native" -import BluetoothSdk from "@mentra/bluetooth-sdk" +import {toolkit} from "@mentra/island" import {Button, Screen} from "@/components/ignite" import {OnboardingGuide, OnboardingStep} from "@/components/onboarding/OnboardingGuide" @@ -24,7 +24,7 @@ export default function UnpairEvenScreen() { } const handleTryAgain = () => { - BluetoothSdk.forget() + toolkit.glasses.forget() clearHistory() replace("/pairing/prep", {deviceModel}) } diff --git a/mobile/src/app/wifi/connecting.tsx b/mobile/src/app/wifi/connecting.tsx index f362959324..13a050c748 100644 --- a/mobile/src/app/wifi/connecting.tsx +++ b/mobile/src/app/wifi/connecting.tsx @@ -1,4 +1,4 @@ -import BluetoothSdk from "@mentra/bluetooth-sdk" +import {toolkit} from "@mentra/island" import {useLocalSearchParams} from "expo-router" import {useEffect, useState, useCallback} from "react" import {ActivityIndicator, View} from "react-native" @@ -81,7 +81,7 @@ export default function WifiConnectingScreen() { const attemptConnection = async () => { try { console.log("Attempting to send wifi credentials to Core", ssid, password) - await BluetoothSdk.sendWifiCredentials(ssid, password) + await toolkit.glasses.wifi.connect(ssid, password) // Save credentials ONLY on successful connection if checkbox was checked. // This ensures we never save wrong passwords. diff --git a/mobile/src/app/wifi/scan.tsx b/mobile/src/app/wifi/scan.tsx index 0620a061e7..f08ade03a7 100644 --- a/mobile/src/app/wifi/scan.tsx +++ b/mobile/src/app/wifi/scan.tsx @@ -1,4 +1,4 @@ -import BluetoothSdk, {WifiSearchResult} from "@mentra/bluetooth-sdk" +import {toolkit, type WifiSearchResult} from "@mentra/island" import {useFocusEffect} from "expo-router" import {useCallback, useEffect, useMemo, useState} from "react" import {ActivityIndicator, ScrollView, TouchableOpacity, View} from "react-native" @@ -81,7 +81,7 @@ export default function WifiScanScreen() { setNetworks([]) try { - const scanResults = await BluetoothSdk.requestWifiScan() + const scanResults = await toolkit.glasses.wifi.scan() console.log(`WIFI_SCAN: Received ${scanResults.length} WiFi scan results`) setNetworks( scanResults.map((network) => ({ @@ -118,7 +118,7 @@ export default function WifiScanScreen() { onPress: async () => { try { console.log(`WIFI_SCAN: Forgetting network: ${selectedNetwork.ssid}`) - await BluetoothSdk.forgetWifiNetwork(selectedNetwork.ssid) + await toolkit.glasses.wifi.forget(selectedNetwork.ssid) // Also remove from local saved credentials WifiCredentialsService.removeCredentials(selectedNetwork.ssid) setSavedNetworks((prev) => prev.filter((ssid) => ssid !== selectedNetwork.ssid)) diff --git a/mobile/src/components/dev/CoreStatusBar.tsx b/mobile/src/components/dev/CoreStatusBar.tsx index 9583fb846b..6ba2949d10 100644 --- a/mobile/src/components/dev/CoreStatusBar.tsx +++ b/mobile/src/components/dev/CoreStatusBar.tsx @@ -10,8 +10,8 @@ import {useDebugStore} from "@/stores/debug" import {selectGlassesConnected, selectGlassesReady, useGlassesStore} from "@/stores/glasses" import {SETTINGS, useSetting} from "@/stores/settings" import {useSaferAreaInsets} from "@/contexts/SaferAreaContext" -import BluetoothSdk, {TouchEvent} from "@mentra/bluetooth-sdk" -import {BgTimer} from "@mentra/island" +import {TouchEvent} from "@mentra/bluetooth-sdk" +import {BgTimer, toolkit} from "@mentra/island" function Tag({icon, label, bg}: {icon: IconTypes; label: string; bg: string}) { const {theme} = useAppTheme() @@ -57,7 +57,7 @@ export default function CoreStatusBar() { const touchEventTimer = useRef(null) useEffect(() => { - let sub = BluetoothSdk.addListener("touch_event", (event: TouchEvent) => { + let unsub = toolkit.glasses.onTouchGesture((event: TouchEvent) => { setTouchEvent(event) BgTimer.clearTimeout(touchEventTimer.current ?? 0) touchEventTimer.current = BgTimer.setTimeout(() => { @@ -66,7 +66,7 @@ export default function CoreStatusBar() { // console.log("touch_event", event) }) return () => { - sub.remove() + unsub() } }, []) diff --git a/mobile/src/components/dev/StoreUrl.tsx b/mobile/src/components/dev/StoreUrl.tsx deleted file mode 100644 index 9392361068..0000000000 --- a/mobile/src/components/dev/StoreUrl.tsx +++ /dev/null @@ -1,277 +0,0 @@ -import {useState} from "react" -import {TextInput, View, ViewStyle, TextStyle, TouchableOpacity} from "react-native" - -import {Button, Text} from "@/components/ignite" -import GlassView from "@/components/ui/GlassView" -import {useAppTheme} from "@/contexts/ThemeContext" -import {useNavigationStore} from "@/stores/navigation" -import {translate} from "@/i18n" -import {SETTINGS, useSetting} from "@/stores/settings" -import {ThemedStyle} from "@/theme" -import showAlert from "@/utils/AlertUtils" - -interface SavedUrl { - label: string - url: string -} - -export default function StoreUrl() { - const {theme, themed} = useAppTheme() - const {replace} = useNavigationStore.getState() - const [customUrlInput, setCustomUrlInput] = useState("") - const [storeUrl, setStoreUrl] = useSetting(SETTINGS.store_url.key) - const [savedUrls, setSavedUrls] = useSetting(SETTINGS.saved_store_urls.key) - - // Ensure savedUrls is always an array - const bookmarks: SavedUrl[] = Array.isArray(savedUrls) ? savedUrls : [] - - const generateLabel = (url: string): string => { - try { - const parsed = new URL(url) - return parsed.host - } catch { - return url - } - } - - const handleBookmark = () => { - const urlToSave = customUrlInput.trim().replace(/\/+$/, "") - - if (!urlToSave) { - showAlert("No URL", "Enter a URL in the text field first.", [{text: "OK"}]) - return - } - - if (!urlToSave.startsWith("http://") && !urlToSave.startsWith("https://")) { - showAlert("Invalid URL", "Please enter a valid URL starting with http:// or https://", [{text: "OK"}]) - return - } - - // Check for duplicates - if (bookmarks.some((b) => b.url === urlToSave)) { - showAlert("Already Bookmarked", "This URL is already in your saved list.", [{text: "OK"}]) - return - } - - const label = generateLabel(urlToSave) - const updated = [...bookmarks, {label, url: urlToSave}] - setSavedUrls(updated) - showAlert("Bookmarked", `Saved "${label}" to your URLs.`, [{text: "OK"}]) - } - - const handleDeleteBookmark = (index: number) => { - const bookmark = bookmarks[index] - showAlert("Remove Bookmark", `Remove "${bookmark.label}" from your saved URLs?`, [ - {text: "Cancel", style: "cancel"}, - { - text: "Remove", - style: "destructive", - onPress: () => { - const updated = bookmarks.filter((_, i) => i !== index) - setSavedUrls(updated) - }, - }, - ]) - } - - const handleSaveUrl = async () => { - const urlToTest = customUrlInput.trim().replace(/\/+$/, "") - - // Basic validation - if (!urlToTest) { - showAlert("Empty URL", "Please enter a URL or reset to default.", [{text: "OK"}]) - return - } - - if (!urlToTest.startsWith("http://") && !urlToTest.startsWith("https://")) { - showAlert("Invalid URL", "Please enter a valid URL starting with http:// or https://", [{text: "OK"}]) - return - } - - await setStoreUrl(urlToTest) - - await showAlert( - "Success", - "Custom store URL saved and verified. It will be used on the next connection attempt or app restart.", - [ - { - text: translate("common:ok"), - onPress: () => { - replace("/") - }, - }, - ], - ) - } - - const handleResetUrl = async () => { - setStoreUrl(null) - setCustomUrlInput("") - showAlert("Success", "Reset store URL to default.", [ - { - text: "OK", - onPress: () => { - replace("/") - }, - }, - ]) - } - - return ( - - - Custom Store URL - - Override the default store server URL. Leave blank to use default. - {storeUrl && `\nCurrently using: ${storeUrl}`} - - - -