Skip to content

Documentation010_recording_and_inputs.md

Latest commit

 

History

History
101 lines (65 loc) · 7.93 KB

Documentation010_recording_and_inputs.md

File metadata and controls

101 lines (65 loc) · 7.93 KB
Raw

::v3.0::1o.2o22::2019.4.0::

Audio input

To stream audio from any available system recording input - just attach an AudioStreamInput component to an empty game object and from custom script access audio buffer data of AudioSource which it automatically creates. See how to interact with it in the AudioStreamInputDemo/AudioStreamInput2DDemo scene. Latency is rather high for Unity input streams since it has to go via full Unity audio buffer processing (including optional spatialization). For significantly lower latency you can use AudioStreamInput2D component - with downside that it is 2D only. ResonanceInput is fully spatialized and has near realtime latency (it doesn't use Unity spatialization at all)

All input components can stream/record from almost any connected input with a recording interface and they support broader class of hardware compared to Unity Microphone class. See also Audio input interfaces below.

As for latency currently this is best option I could come up with; for even lower latency a native plugin is needed ( such as https://github.com/keijiro/Lasp ), but with the same 2D limitation since it uses just OnAudioFilterRead [ OnAudioFilterRead has limitation of not being able to support 3D sound ]. For LASP interop with Unity AudioSource see this gist: https://gist.github.com/r618/d74f07b6049fce20f1dc0dfa546bda89 ( LASP has to be patched though currently since it can only be run from the main thread, and frequency is not exposed - those are just minor changes). AudioStreamInput2D latency is very usable though and e.g. on iOS and recent phones it has almost immediate response to audio input in the scene.

AudioStreamInput* components will use DSP buffer setting for output #0, if specified, since FMOD's system for output #0 is also used as recording system. See Devices config in Documentation.md

For latency you can also refer to answers to this post: http://www.fmod.org/questions/question/delaylatency-when-playing-sounds/

Since the way the resampling of input signal by just setting the pitch on the AudioSource was handled led to big drifting over longer time if input and output sample rates were significantly different, in 1.7.7 I added option to not use Unity's built-in AudioSource resampling and channel conversion this way. Instead a custom resampler is used, with possibility to specify custom mix matrix to provide mapping between input and output channels. Default mix matrix is computed by FMOD if not customised by user, on iOS default mix matrix is computed manually (might not be best fit for all scenarios). See demo scene for reference, and call SetCustomMixMatrix on component before starting the recording, if needed.

The custom resampler is based on simple interpolation of original audio/samplerate for target/output, and is probably lower quality than Unity's builtin one - but is stable when running long/er.

! For iOS/mobiles please see specific mobile recording notes. Documentation060_mobiles

With ResonanceInput it's possible to capture any input device and play it back in 3D - see in 3D spatialisation.

! Please be aware that each component which manipulates and manages its own AudioSource makes the Unity AudioSource component on the GameObject unusable by the user/Unity/other components at runtime So in general it's advisable to place each AudioStream* component which uses AudioSource on its own exclusive GameObject

Audio input interfaces

On desktops/standalones it is possible to capture and listen to the system output

Windows specific:

  • any device/s which can be opened for recording, not designated as a microphone/audio input by system, is postfixed with "[loopback]" in its name by FMOD
  • for example on Windows a VirtualCable port/channel can be opened and streamed into Unity this way -

Note: that device access like this is only possible on desktops, mobile OSs typically don't expose this functionality of main audio output. Note: don't use loopback interfaces for signals which should be audible/heard - this will immediately cause feedback loop -

HW compatibility:

  • FMOD currently has limit MAX_CHANNEL_WIDTH which at the time of writing is =32= for number of channels of a given interface - so it doesn't support more than 32 inputs/outputs on a soundcard/interface
  • multichannel input/microphones are supported as opposed to Unity's Microphone class Unity support for multichannel inputs might have got better with latest (2023) updates though

Android recording permissions

there is a Microphone class referenced in Android specific code of AudioStreamInputBase.cs, which causes Unity to automatically include recording permission in the manifest If you don't want/need this permission and recording in your Android build, please delete the whole Scripts/AudioStreamInput folder and demos in Demo/AudioStreamInput folder.

Android OpenSL ES support

FMOD requires OpenSL ES support on Android in order to enable recording - this is currently enabled for all cases/devices, even when recording is not used in the app

  • contact me if you need just playback in the app w/o recording enabled, alternatively you can just remove the requirement in FMOD_System.cs: comment out/remove outputType = FMOD.OUTPUTTYPE.OPENSL;

'resampleInput' setting

  • AudioStreamInput* 's setting for better interoperability with other plugins/assets which need original sample rate of the input signal preserved (such as AVProMediaRecorder at that time) when they want to do their own resampling/encoding. Default is ON and the input signal is resampled to current output sample rate so it can be e.g. heard normally on speakers. When OFF no resampling is done and input signal can be manually forwarded as needed - for this custom scripting is needed if it is e.g. not being captured from attached AudioSource.

'hotplugging'

Hotplugging - dynamic updates of attached input devices - can be enabled by including separate AudioStreamDevicesChangedNotify component anywhere in the scene Devices are checked for changes in Update If a change is detected, a user Unity event is invoked where the new list can be queried/updated via AvailableInputs again. AudioStreamDevicesChangedNotify is included in few demo scenes together with event handler.

'AudioStream InputDevice' mixer effect

To record directly into an AudioMixer group add AudioStream InputDevice effect and set InputDeviceID to 0 (default system input) or above. The effect will start recording immediately and pass the recording along in the mixer Set InputDeviceID to -1 to stop recording

The recording will be running depending on the setting of the mixer:

  • if Auto Mixer Suspend is ON (default) the recording will be running (recording from the input) as long as the signal is above the Threshold Volume (default -80 dB) of the mixer note: this is regardless of playmode, so if the recording was started during play time after exiting play mode in Editor, it will still be running if the volume is above the threshold

  • if Auto Mixer Suspend is OFF the recording will start and be running regardless of e.g. play mode

Moreover, the demo scene for the effect - InputDeviceUnityMixerDemo - has additional AudioSource in the scene, which outputs to (a single) mixer group, is played automatically and has its volume set to 0. This is in order to activate the mixer processing by Unity, since Auto Mixer Suspend is by default ON and the group wouldn't be even processed otherwise

Note: * macOS * Please note the recording effect currently doesn't work on macOS -- this should be hopefully fixed later

  • in any case all sources for the plugin are included in AudioPlugin.zip archive