Skip to content

feat: synchronized audio + haptics (iOS, Android, RN, Flutter, KMP) - #162

Open
piaskowyk wants to merge 6 commits into
mainfrom
feat/audio-haptics-sync
Open

feat: synchronized audio + haptics (iOS, Android, RN, Flutter, KMP)#162
piaskowyk wants to merge 6 commits into
mainfrom
feat/audio-haptics-sync

Conversation

@piaskowyk

Copy link
Copy Markdown
Member

What

Adds synchronized audio + haptics to Pulsar: play a short authored sound locked to a haptic pattern, via a new parsePatternWithSound entry point on the Pattern Composer. Implemented across native iOS, native Android, React Native, Flutter, and KMP.

Why

Pulsar could vibrate but not play a paired sound in sync. The existing AudioSimulator only synthesizes an audible preview of the haptic envelope (a dev/simulator aid) — it can't play a real audio file. This PR adds true sound playback aligned to the haptics.

Design (per platform)

  • iOS (Core Haptics): the sound file is registered with CHHapticEngine.registerAudioResource and folded into the discrete CHHapticPattern as a CHHapticEvent. Audio and haptics share one player/timeline → sample-accurate.
  • Android (SoundPool): an .ogg with baked haptic channels drives the vibrator directly (audio-coupled, perfect sync) when the device supports it. Otherwise — including any non-.ogg file (.wav/.mp3), auto-detected — it falls back to playing the audio while Pulsar's own generated VibrationEffect fires in parallel.
  • RN / Flutter / KMP: thin wrappers over the native modules. The app-facing API exposes only { uri, volume, offset }; hapticChannels is native-default (true) + auto-detection, so it isn't surfaced to app developers.
  • KMP iOS was a separate Kotlin/Native reimplementation with no audio path — this syncs it with the Swift design (adds registerAudioResource interop + the audio-event fold).

Public API

// React Native
usePatternComposer({
  discretePattern: [...], continuousPattern: {...},
  sound: { uri: require('./boom.wav'), volume: 1, offset: 0 },  // require() or a uri string
});
// iOS (Swift)
composer.parsePatternWithSound(hapticsData: pattern, uri: "boom.caf", volume: 1, offset: 0)
// Android / KMP
composer.parsePatternWithSound(pattern, SoundData(uri = "boom.ogg"))
// Flutter
await composer.parsePatternWithSound(pattern, Sound(uri: 'boom'));

Providing audio files

  • iOS: .caf/.wav (PCM), local/bundled; Core Haptics can't register remote URLs. .mp3 works if the OS registers it, else falls back to haptics-only.
  • Android: .ogg with haptic channels for coupled sync; any other format is played as plain audio + Pulsar haptics.
  • RN: sound.uri accepts a require() (resolved via Image.resolveAssetSource) or a string. Note a require() resolves to a remote Metro URL in dev but a local file in release — test synced sound on a release build / device.

Verification

  • KMP compileKotlinIosSimulatorArm64 (common + Kotlin/Native Core Haptics interop) — ✅ (surfaced & fixed a real NSURL forward-declaration mismatch).
  • KMP compileAndroidMain with -PUSE_LOCAL_PULSAR_ANDROID=1 (also recompiles the canonical Android/Pulsar changes) — ✅.
  • Flutter dart analyze lib — ✅ clean.
  • RN tsc — ✅.

Reviewer notes

  • KMP Android artifact: KMP defaults to the published com.swmansion:pulsar Maven artifact, which won't have this API until it's published (or built with -PUSE_LOCAL_PULSAR_ANDROID=1).
  • Not compiled in CI-equivalent here: the Flutter native plugins (Kotlin/Swift) and a full iOS xcodebuild — they call the same canonical APIs that did compile.
  • iOS Simulator has no Core Haptics, so the sound path is device-only there; Android emulator won't vibrate (audio still plays via fallback).
  • The unrelated docs/.../config.ts change in the working tree is intentionally not part of this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant