A Python-based framework for automated audio hardware testing and validation.
This project demonstrates QA automation, hardware testing, configuration-driven validation, CI-friendly test design, and cross-platform audio device discovery. It is being developed as a portfolio project targeting Audio QA Engineer, Hardware QA Engineer, Embedded QA Engineer, and Software QA Engineer roles within the audio technology industry.
Phase 5.1 complete — split-endpoint physical loopback validation
Completed capabilities include:
- Cross-platform audio device discovery
- Host API-aware device matching
- Input-only, output-only, and duplex stream validation
- PortAudio capability checks
- Safe stream construction and closure
- Finite-duration recording
- Finite-buffer playback
- Framework-owned audio buffers
- WAV import and export
- Deterministic sine-wave and silence generation
- RMS, peak and DC offset analysis
- Silence and clipping detection
- Per-channel sample-domain metrics
- Configuration-driven metric thresholds
- Structured threshold failure reasons
- Duplex playback and capture execution
- Configurable physical loopback signal routing
- Silence-padded loopback test signals
- Captured-signal alignment
- Dominant-frequency measurement
- End-to-end loopback metric validation
- Structured per-channel loopback failures
validate-loopbackCLI command- Loopback WAV and JSON evidence export
- WAV analysis through the
analyse-audioCLI command - Backend-independent recording and playback services
- Rich CLI and structured JSON reporting
- Windows and Linux hardware validation
Next development: analyse the frequency-domain characteristics of recorded signals and compare them against expected behaviour.
See the Roadmap for planned development.
The framework aims to provide a reusable foundation for automated testing of audio hardware such as:
- Audio interfaces
- USB audio devices
- Embedded audio hardware
- Audio drivers
- Recording and playback systems
The current foundation includes reliable device discovery, configuration management, hardware identification, stream-capability validation, finite recording and playback, WAV handling, deterministic signal generation, sample-domain analysis, configuration-driven metric validation, duplex loopback execution, captured-signal alignment, frequency validation, and evidence export.
Future releases will build on the current loopback-validation foundation with measured sample-rate verification, deeper audio-quality measurements, latency observation, and stability testing.
Enumerate audio devices through PortAudio using Python's sounddevice package.
The framework collects:
- Device name
- Host API name
- Host API index
- Input channel count
- Output channel count
- Default sample rate
Example:
Index Device Host API Inputs Outputs Default Rate
0 Analogue 1 + 2 (Focusrite USB Audio) WASAPI 2 0 48000 Hz
1 Speakers (Focusrite USB Audio) WASAPI 0 2 48000 Hz
...
Example output shortened for readability.
Supported matching rules:
- Exact device name
- Partial device name
- Host API matching
- Minimum input channels
- Minimum output channels
Example:
device:
name_contains: "Focusrite USB Audio"
host_api_contains: "WASAPI"
minimum_input_channels: 2This allows the framework to distinguish between multiple entries representing the same physical audio interface across different host APIs.
Inspect available audio devices:
audio-hw inspect-devicesInspect devices using a configuration:
audio-hw inspect-devices --config configs/scarlett_windows_wasapi_input.yamlOutput JSON:
audio-hw inspect-devices --jsonValidate a configured stream:
audio-hw validate-stream \
--config configs/scarlett_windows_wasapi_input.yamlOutput validation results as JSON:
audio-hw validate-stream \
--config configs/scarlett_windows_wasapi_input.yaml \
--jsonTo create a WAV file for playback validation, first ensure the recording configuration contains an output path:
execution:
duration_seconds: 1.0
timeout_seconds: 5.0
output_file: recordings/test.wavThen run recording validation:
audio-hw validate-recording \
--config configs/scarlett_windows_wasapi_input.yamlThen validate playback using the generated recording:
audio-hw validate-playback \
--config configs/scarlett_windows_wasapi_output.yaml \
--input recordings/test.wavOutput recording validation results as JSON:
audio-hw validate-recording \
--config configs/scarlett_windows_wasapi_input.yaml \
--jsonOutput playback validation results as JSON:
audio-hw validate-playback \
--config configs/scarlett_windows_wasapi_output.yaml \
--input recordings/test.wav \
--jsonAnalyse a WAV file using configured sample-domain thresholds:
audio-hw analyse-audio \
--config configs/example_analysis.yaml \
--input recordings/test.wavOutput analysis results as JSON:
audio-hw analyse-audio \
--config configs/example_analysis.yaml \
--input recordings/test.wav \
--jsonRun physical loopback validation using the verified Linux ALSA configuration:
audio-hw validate-loopback \
--config configs/example_loopback_alsa.yamlOutput the loopback result as JSON:
audio-hw validate-loopback \
--config configs/example_loopback_alsa.yaml \
--jsonSave the playback, raw capture, aligned analysis audio and JSON report:
audio-hw validate-loopback \
--config configs/example_loopback_alsa.yaml \
--evidence-dir evidence/linux-alsavalidate-stream checks whether PortAudio accepts the requested stream settings and whether the requested stream can be constructed and closed safely.
validate-recording performs a finite recording using the configured duration and timeout. If execution.output_file is configured, the captured audio is exported as a WAV file.
validate-playback reads a WAV file, verifies that its sample rate and channel count match the configured output stream, and performs finite playback through the selected device.
analyse-audio reads the supplied WAV file into a framework-owned AudioBuffer, calculates RMS, peak and DC offset, performs silence and clipping detection, and applies configured thresholds per channel.
validate-loopback generates a deterministic sine wave, routes it to the configured output channel, executes duplex playback and capture, aligns the selected captured input against the reference signal, measures dominant frequency, applies configured sample-domain thresholds, and returns structured pass/fail results.
(When --evidence-dir is supplied, the command retains playback.wav, captured.wav, analysed.wav and report.json for both passing and failing validations.)
Recording, playback and stream-validation commands require exactly one matching device.
analyse-audio is file-based and does not perform device discovery or matching.
Loopback validation can resolve either one shared PortAudio duplex device or separate input and output endpoints.
Separate endpoints are useful on platforms such as Windows where one physical audio interface may be exposed as distinct input-only and output-only PortAudio devices.
Support for separate endpoints does not imply that arbitrary physical devices are sample-clock synchronized. For reliable hardware validation, prefer endpoints belonging to the same physical interface and compatible host API/driver path.
Configuration-driven validation supports:
- Input-only streams
- Output-only streams
- Duplex streams
- Sample-rate capability checks
- Input and output channel checks
- Strongly typed sample formats
- Configurable or backend-selected block sizes
- Safe stream construction and closure
- Human-readable and JSON results
Validation is implemented through a backend-independent application service. This separates workflow orchestration from PortAudio-specific behaviour and allows deterministic hardware-independent testing.
A successful stream-opening result does not start recording or playback and does not yet prove end-to-end audio signal quality.
See docs/stream-validation.md for validation scope, limitations, exit codes and the hardware test procedure.
Phase 3 adds finite audio execution while preserving the framework's backend-independent architecture.
Recording validation supports:
- Configuration-driven input-device selection
- Finite-duration capture
- Exact requested frame-count validation
- Configurable execution timeout
- Framework-owned
AudioBufferresults - Optional WAV export
- Human-readable and JSON CLI results
Playback validation supports:
- Configuration-driven output-device selection
- WAV input
- Sample-rate compatibility validation
- Channel-count compatibility validation
- Finite-buffer playback
- Configurable execution timeout
- Human-readable and JSON CLI results
Recording and playback failures are translated through framework-owned backend exceptions.
Phase 3 validates execution and data flow only. It does not yet make claims about:
- Audio quality
- Frequency response
- Distortion
- Noise level
- Measured sample-rate accuracy
- Latency
- Loopback correctness
Phase 4 adds deterministic signal generation and basic audio analysis.
Signal generation supports:
- Sine waves
- Silence
- Configurable sample rate
- Configurable duration
- Configurable channel count
- Configurable sine frequency and amplitude
- Framework-owned
AudioBufferoutput - Deterministic, hardware-independent execution
Sample-domain analysis supports:
- RMS level
- Absolute peak level
- DC offset
- Silence detection
- Clipping detection
- Overall metrics
- Per-channel metrics
- Configuration-driven validation thresholds
- Structured per-channel failure reasons
- Human-readable and JSON CLI reporting
Example threshold configuration:
thresholds:
minimum_rms: 0.01
maximum_rms: 0.8
maximum_peak: 0.95
maximum_abs_dc_offset: 0.02
silence_threshold: 0.0001
clipping_threshold: 1.0
fail_on_silence: true
fail_on_clipping: truePhase 4 analysis operates exclusively on framework-owned AudioBuffer instances.
Default detection thresholds assume normalized floating-point audio. When analysing raw integer AudioBuffer data directly, threshold values are interpreted in the buffer's native sample units and must be configured accordingly.
It is sample-domain analysis only and does not currently measure frequency response, distortion, signal-to-noise ratio, inter-sample peaks, latency or end-to-end signal-path correctness.
See docs/signal-generation-analysis.md for metric definitions, threshold behaviour, CLI usage and analysis limitations.
Phase 5 builds on deterministic signal generation and sample-domain analysis with physical duplex loopback validation.
Loopback validation supports:
- Configurable output and input channel routing
- Deterministic sine-wave playback
- Leading and trailing silence padding
- Backend-independent duplex execution
- Captured-signal alignment
- Dominant-frequency measurement
- RMS, peak and DC-offset validation
- Silence and clipping detection
- Structured metric and channel failures
- Human-readable and JSON CLI output
- Optional WAV and JSON evidence retention
- Backward-compatible shared-device duplex selection
- Independent input and output device selectors
- Host-API-aware endpoint matching
- Direction and channel-capability validation
- Separate PortAudio input and output device indexes
- Exact input/output endpoint reporting in CLI and JSON evidence
Split-endpoint execution is intended to support host APIs that expose one physical interface through separate capture and playback entries. Whether a particular endpoint pair can operate simultaneously remains dependent on PortAudio, the host API, the driver and the hardware clock topology.
The framework does not provide cross-device clock synchronization or drift compensation.
A physical loopback test requires a line-level connection between the selected hardware output and input.
For the current Linux Focusrite validation environment, direct ALSA at 48 kHz is the verified unattended physical loopback path.
Post-Phase 5 diagnostics also demonstrated successful physical loopback through JACK at 48 kHz. However, the tested JACK graph requires the original PortAudio capture routing to remain in place while the physical Input 1 capture_MONO source is additionally connected manually to PortAudio:in_0.
The PortAudio JACK client is transient, so this manual connection disappears when the validation stream closes and must currently be recreated for each run.
JACK physical loopback is therefore verified but is not yet an unattended framework-only workflow. Software monitor_* routes can also produce passing digital loopback results and are documented separately from physical hardware evidence.
Dedicated ALSA and JACK configurations are retained because host APIs can expose materially different device, routing and duplex behaviour.
See docs/loopback-validation.md for physical setup, safety guidance, host-API considerations, CLI usage, evidence export and validation scope.
The framework uses strongly typed Pydantic models for:
- Audio devices
- Device matching rules
- Stream configuration
- Signal-generation configuration
- Metric-threshold configuration
- Framework configuration
This provides validation and predictable configuration handling.
Current development targets:
- Windows 11
- Ubuntu Linux
Audio backends are abstracted to support future expansion.
The project includes:
- Unit tests
- Device matching tests
- Configuration validation tests
- Backend abstraction tests
- Deterministic fake-backend tests
- PortAudio capability-validation tests
- Stream-opening tests
- Recording and playback backend tests
- Audio-buffer tests
- WAV import/export tests
- Recording-service orchestration tests
- Playback-service orchestration tests
- Signal-generation model tests
- Deterministic sine and silence generation tests
- RMS analysis tests
- Peak analysis tests
- DC-offset analysis tests
- Silence and clipping detection tests
- Metric-threshold validation tests
- Non-finite sample and threshold validation tests
- Analysis failure-path tests
- CLI analysis tests
- Duplex backend contract tests
- Deterministic fake duplex tests
- PortAudio duplex execution tests
- Signal routing and padding tests
- Captured-signal alignment tests
- Dominant-frequency analysis tests
- Loopback validation-service tests
- Loopback CLI tests
- Loopback evidence-reporting tests
GitHub Actions runs hardware-independent tests, linting, formatting checks, type checking, and coverage reporting on Ubuntu.
Current coverage target:
95%+
The framework resolves host API information reported by PortAudio.
Examples:
- WASAPI
- MME
- DirectSound
- Windows WDM-KS
- ALSA
- JACK
- PulseAudio
This enables more reliable device matching on systems where the same physical interface appears multiple times through different audio subsystems.
audio-hardware-automation-framework/
│
├── configs/
│ ├── example_analysis.yaml
│ ├── example_duplex_device.yaml
│ ├── example_loopback.yaml
│ ├── example_loopback_alsa.yaml
│ ├── example_loopback_jack.yaml
│ │
│ ├── scarlett_windows_mme_input.yaml
│ ├── scarlett_windows_mme_output.yaml
│ ├── scarlett_windows_directsound_input.yaml
│ ├── scarlett_windows_directsound_output.yaml
│ ├── scarlett_windows_wasapi_input.yaml
│ ├── scarlett_windows_wasapi_output.yaml
│ ├── scarlett_windows_wdmks_input.yaml
│ ├── scarlett_windows_wdmks_output.yaml
│ │
│ ├── scarlett_windows_mme_loopback.yaml
│ ├── scarlett_windows_directsound_loopback.yaml
│ ├── scarlett_windows_wasapi_loopback.yaml
│ │
│ ├── scarlett_linux_alsa.yaml
│ ├── scarlett_linux_jack.yaml
│ ├── scarlett_linux_pulseaudio_input1.yaml
│ ├── scarlett_linux_pulseaudio_input2.yaml
│ └── scarlett_linux_pulseaudio_output.yaml
│
├── docs/
│ ├── discovery.md
│ ├── focusrite-loopback-validation.md
│ ├── jack-loopback-routing.md
│ ├── loopback-validation.md
│ ├── platform-support.md
│ ├── recording-playback-validation.md
│ ├── signal-generation-analysis.md
│ ├── stream-validation.md
│ └── images/
│ ├── phase-1-github-actions-passing.png
│ ├── phase-1-linux-device-discovery.png
│ ├── phase-1-pytest-coverage.png
│ ├── phase-1-wdmks-hotplug-verification.png
│ ├── phase-1-windows-device-discovery.png
│ ├── phase-2-linux-duplex-validation.png
│ ├── phase-2-negative-validation.png
│ ├── phase-2-stream-validation.png
│ ├── phase-3-playback-validation.png
│ ├── phase-3-pytest-coverage.png
│ ├── phase-3-recording-validation.png
│ ├── phase-4-analysis-json.png
│ ├── phase-4-analysis-sine-wave.png
│ ├── phase-4-pytest-coverage.png
│ ├── phase-4-threshold-failure.png
│ ├── phase-5-alsa-physical-loopback.png
│ ├── phase-5-evidence-bundle.png
│ ├── phase-5-jack-loopback-fail.png
│ ├── phase-5-jack-loopback-default-fail.png
│ ├── phase-5-jack-loopback-manual-pass.png
│ ├── phase-5-jack-routing.png
│ ├── phase-5-jack-routing-default-fail.png
│ ├── phase-5-jack-routing-manual-pass.png
│ ├── phase-5-loopback-json.png
│ ├── phase-5-pytest-coverage.png
│ ├── phase-5.1-windows-device-discovery.png
│ ├── phase-5.1-windows-wasapi-pass.png
│ ├── phase-5.1-windows-mme-pass.png
│ ├── phase-5.1-windows-directsound-timeout.png
│ └── phase-5.1-windows-wdmks-blocking-api.png
│
├── src/
│ └── audio_hw_framework/
│ ├── analysis/
│ │ ├── __init__.py
│ │ ├── _samples.py
│ │ ├── alignment.py
│ │ ├── dc_offset.py
│ │ ├── detection.py
│ │ ├── exceptions.py
│ │ ├── frequency.py
│ │ ├── models.py
│ │ ├── peak.py
│ │ └── rms.py
│ │
│ ├── audio/
│ │ ├── __init__.py
│ │ ├── audio_buffer.py
│ │ └── wav.py
│ │
│ ├── backend/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── fake_backend.py
│ │ └── sounddevice_backend.py
│ │
│ ├── configuration/
│ │ ├── __init__.py
│ │ ├── loader.py
│ │ ├── loopback.py
│ │ └── thresholds.py
│ │
│ ├── device/
│ │ ├── __init__.py
│ │ ├── matcher.py
│ │ └── models.py
│ │
│ ├── playback/
│ │ ├── __init__.py
│ │ └── service.py
│ │
│ ├── recording/
│ │ ├── __init__.py
│ │ └── service.py
│ │
│ ├── reporting/
│ │ ├── __init__.py
│ │ └── loopback.py
│ │
│ ├── signal/
│ │ ├── __init__.py
│ │ ├── generator.py
│ │ ├── models.py
│ │ └── transforms.py
│ │
│ ├── validation/
│ │ ├── __init__.py
│ │ ├── audio_metrics.py
│ │ ├── duplex_endpoints.py
│ │ ├── loopback_models.py
│ │ ├── loopback_service.py
│ │ └── service.py
│ │
│ ├── __init__.py
│ ├── __main__.py
│ └── cli.py
│
├── tests/
│ ├── __init__.py
│ └── unit/
│ ├── __init__.py
│ ├── cli_helpers.py
│ ├── fake_backend_helpers.py
│ ├── sounddevice_backend_helpers.py
│ ├── test_analysis_models.py
│ ├── test_analysis_non_finite.py
│ ├── test_audio_buffer.py
│ ├── test_audio_metric_validation.py
│ ├── test_backend_contract.py
│ ├── test_cli_analysis.py
│ ├── test_cli_app.py
│ ├── test_cli_inspect_devices.py
│ ├── test_cli_loopback.py
│ ├── test_cli_playback.py
│ ├── test_cli_recording.py
│ ├── test_cli_stream_validation.py
│ ├── test_configuration.py
│ ├── test_dc_offset.py
│ ├── test_detection.py
│ ├── test_device_matching.py
│ ├── test_duplex_endpoint_resolution.py
│ ├── test_fake_backend_duplex.py
│ ├── test_fake_backend_playback.py
│ ├── test_fake_backend_recording.py
│ ├── test_fake_backend_stream.py
│ ├── test_frequency.py
│ ├── test_loopback_models.py
│ ├── test_loopback_reporting.py
│ ├── test_loopback_service.py
│ ├── test_models.py
│ ├── test_package.py
│ ├── test_peak.py
│ ├── test_playback_service.py
│ ├── test_recording_service.py
│ ├── test_rms.py
│ ├── test_signal_alignment.py
│ ├── test_signal_generator.py
│ ├── test_signal_models.py
│ ├── test_signal_transforms.py
│ ├── test_sounddevice_backend_device.py
│ ├── test_sounddevice_backend_duplex.py
│ ├── test_sounddevice_backend_playback.py
│ ├── test_sounddevice_backend_recording.py
│ ├── test_sounddevice_backend_stream.py
│ ├── test_validation_service.py
│ └── test_wav.py
│
├── pyproject.toml
├── README.md
└── LICENSE
Clone the repository:
git clone <repository-url>
cd audio-hardware-automation-frameworkCreate a virtual environment:
python -m venv .venvActivate it:
Windows:
.venv\Scripts\activateLinux:
source .venv/bin/activateInstall the project:
pip install -e ".[dev]"audio-hw inspect-devicesaudio-hw inspect-devices \
--config configs/scarlett_windows_wasapi_input.yamlaudio-hw inspect-devices --jsonWhen no audio devices are detected:
audio-hw inspect-devicesThe command displays:
No audio devices detected.
and exits with code 1.
audio-hw inspect-devices --jsonThe command returns:
{
"backend": {
"name": "portaudio",
"library": "sounddevice",
"library_version": "..."
},
"devices": []
}and exits successfully.
This behaviour allows automated tooling to distinguish between command failures and valid enumeration results containing no devices.
device:
name_contains: "Focusrite USB Audio"
host_api_contains: "WASAPI"
minimum_input_channels: 2
minimum_output_channels: 0
stream:
sample_rate: 48000
input_channels: 2
output_channels: 0
block_size: null
dtype: "float32"
execution:
duration_seconds: 1.0
timeout_seconds: 5.0
output_file: null
thresholds:
minimum_rms: null
maximum_rms: null
maximum_peak: null
maximum_abs_dc_offset: null
silence_threshold: 0.0001
clipping_threshold: 1.0
fail_on_silence: true
fail_on_clipping: trueCurrent validation:
- Device enumeration
- Host API discovery
- Host API filtering
- Device matching
- CLI inspection
- Configuration loading
- Cross-platform discovery
- Windows host API validation (MME, DirectSound, WASAPI, WDM-KS)
- Linux host API validation (ALSA, JACK, PulseAudio)
- Audio-stack-specific device matching
- Configuration-driven unique device selection
- PortAudio input/output capability validation
- Input-only, output-only or duplex stream construction
- Configured block-size application during stream construction
- Safe stream construction and closure
- Finite recording through the Scarlett input
- WAV export of recorded audio
- Finite playback through the Scarlett output
- Structured CLI and JSON validation reporting
- Physical end-to-end loopback validation
- Verified physical duplex loopback through ALSA at 48 kHz
- JACK software-loopback validation and physical-routing diagnostics
- Configurable loopback input/output routing
- Captured-signal alignment against a generated reference
- Dominant-frequency validation using a 1 kHz test tone
- RMS, peak, DC-offset, silence and clipping validation
- WAV and JSON loopback evidence retention
- JACK and ALSA host-API comparison
- Verified Windows physical split-endpoint loopback through WASAPI
- Verified Windows physical split-endpoint loopback through MME
- Three-run repeatability validation for WASAPI and MME
- DirectSound split-endpoint duplex-timeout characterization
- WDM-KS blocking-stream API limitation characterization
Phase 5 and follow-up loopback findings:
| Environment | Result | Observation |
|---|---|---|
| Linux / ALSA / 48 kHz / physical cable | PASS | Verified unattended physical Scarlett loopback |
| Linux / JACK / 48 kHz / software monitor route | PASS | Digital/software route only; not physical hardware evidence |
| Linux / JACK / 48 kHz / default routing | FAIL | Expected physical return not captured at a valid level |
Linux / JACK / 48 kHz / capture_MONO replacing default route |
FAIL | Physical source alone was insufficient in the tested graph |
Linux / JACK / 48 kHz / default routing + capture_MONO |
PASS | Verified physical loopback; requires manual per-run graph modification |
| Windows / Phase 5 | Not executed | Scarlett input/output were exposed separately and the original architecture required one duplex device |
| Windows / WASAPI / Phase 5.1 | PASS | Verified physical split-endpoint loopback; three consecutive passing runs |
| Windows / MME / Phase 5.1 | PASS | Verified physical split-endpoint loopback; three consecutive passing runs |
| Windows / DirectSound / Phase 5.1 | FAIL | Individual endpoints passed, but paired duplex execution timed out |
| Windows / WDM-KS / Phase 5.1 | BLOCKED | PortAudio reported Blocking API not supported yet during stream opening |
The direct ALSA 48 kHz run remains the Phase 5 physical hardware acceptance result because it operates without manual host-routing intervention.
The JACK follow-up demonstrates that physical loopback is also possible through JACK, while identifying a remaining transient-client routing limitation.
Phase 5.1 additionally verifies physical Windows split-endpoint loopback through WASAPI and MME.
The Windows results also demonstrate that host APIs exposing the same physical hardware can behave differently: DirectSound timed out during paired execution and WDM-KS was blocked before duplex execution.
See docs/focusrite-loopback-validation.md for the detailed Phase 5 hardware results and interpretation.
Platforms tested:
- Windows 11 25H2
- Ubuntu Linux 26.04 LTS
Additional documentation is available in:
docs/discovery.md
Explains:
- What device discovery does
- What it does not prove
- Device matching behaviour
- PortAudio limitations
docs/platform-support.md
Explains:
- Supported operating systems
- Host APIs
- CI limitations
- Hardware validation scope
docs/stream-validation.md
Explains:
- Validation workflow
- Supported stream directions
- CLI and JSON output
- Exit codes
- Validation scope and limitations
- Focusrite hardware-validation procedure
docs/recording-playback-validation.md
Explains:
- Recording execution workflow
- Playback execution workflow
- Execution configuration
- WAV import and export
- CLI and JSON output
- Failure handling
- Validation scope and limitations
docs/signal-generation-analysis.md
Explains:
- Deterministic sine-wave and silence generation
- RMS, peak and DC-offset calculations
- Silence and clipping detection
- Per-channel metric behaviour
- Configurable validation thresholds
- Structured threshold failures
analyse-audioCLI and JSON output- Exit codes
- Sample-domain analysis limitations
docs/loopback-validation.md
Explains:
- Physical loopback signal flow
- Hardware and duplex-device requirements
- Safe gain and monitoring setup
- Loopback signal routing and padding
- JACK and ALSA configuration separation
validate-loopbackCLI usage- JSON output and exit codes
- WAV and JSON evidence retention
- Captured-signal alignment
- Frequency-result interpretation
- Platform and host-API limitations
- Current validation scope
docs/focusrite-loopback-validation.md
Records:
- Focusrite Scarlett 2i2 physical loopback validation
- ALSA 48 kHz passing unattended physical hardware result
- JACK 48 kHz software-loopback result
- initial JACK physical-routing failures
- post-Phase 5 successful JACK physical loopback with manual runtime routing
- cable A/B/A physical-path verification
- analogue gain/output-level observations
- transient PortAudio JACK client limitation
- correction of invalid preliminary incomplete-loopback ALSA results
- Windows Phase 5 single-device duplex limitation
- Phase 5.1 Windows split-endpoint physical validation
- WASAPI and MME physical split-endpoint PASS results
- DirectSound duplex-timeout characterization
- WDM-KS blocking-API limitation
- measured frequency and sample-domain results
- host-API and routing comparison
- evidence interpretation
- Phase 5 acceptance criteria and limitations
docs/jack-loopback-routing.md
Records:
- PipeWire-backed JACK environment observations
- JACK-visible Scarlett capture and monitor paths
- default PortAudio routing failure
capture_MONO-only routing failure- successful combined default +
capture_MONOtopology - transient PortAudio JACK client behaviour
- cable A/B/A physical-path verification
- gain-dependent physical capture behaviour
- distinction between physical and software-monitor loopback
- remaining routing-automation questions
The Focusrite Scarlett 2i2 successfully completed end-to-end physical loopback validation through direct ALSA at 48 kHz.
A representative physical capture measured approximately:
Frequency: 1000.000 Hz
RMS: 0.014242
Peak: 0.021235
DC offset: 0.000005
Silence: False
Clipping: False
The physical signal path was:
framework
↓
Scarlett output
↓
physical line-level cable
↓
Scarlett input
↓
framework capture
This is the primary Phase 5 hardware acceptance result.
The validation result is also available as structured machine-readable JSON containing backend, device, stream, routing, frequency, metric and artifact information.
A completed validation retains playback, raw capture, aligned analysis audio and the structured report.
Post-Phase 5 diagnostics established that physical loopback through JACK is possible, but the tested environment requires manual runtime graph modification.
With the default PortAudio routing alone, the expected physical return was absent.
Replacing the default Input 1 connection with the Scarlett physical capture_MONO source also failed.
The successful topology retained the original PortAudio capture routing and additionally connected the physical Input 1 capture_MONO source to PortAudio:in_0.
A representative physical JACK PASS measured:
Expected frequency: 1000.000 Hz
Measured frequency: 999.965 Hz
RMS: 0.032995
Peak: 0.046741
Silence: False
Clipping: False
Cable A/B/A testing produced:
disconnected → FAIL
connected → PASS
disconnected → FAIL
confirming that the successful result depended on the physical Scarlett output-to-input path.
The PortAudio JACK client exists only while validation is running, so the required additional connection disappears after each run and must currently be recreated manually.
Direct ALSA therefore remains the primary unattended Phase 5 hardware acceptance path.
See docs/jack-loopback-routing.md for the full routing investigation.
The Phase 5 hardware-independent regression suite completed with:
400 passed
99% coverage
See docs/focusrite-loopback-validation.md for the complete result matrix and interpretation.
Complete
- Backend abstraction
- PortAudio device enumeration
- Typed YAML configuration
- Host API-aware device matching
- Rich and JSON device inspection
- Cross-platform hardware discovery
Complete
- Strongly typed stream configuration
- Input-only, output-only, and duplex validation
- PortAudio capability checks
- Stream construction and safe closure
- Validation orchestration service
validate-streamCLI command- Structured validation reporting
- Windows and Linux hardware validation
Complete
- Finite-duration recording
- Finite-buffer playback
- Framework-owned audio buffer abstraction
- WAV import and export
- Deterministic fake recording and playback
- PortAudio recording and playback
- Recording execution service
- Playback execution service
validate-recordingCLI commandvalidate-playbackCLI command- Human-readable and JSON execution reporting
Complete
- Deterministic sine-wave generation
- Deterministic silence generation
- Typed signal-generation configuration
- Framework-owned generated audio buffers
- RMS level analysis
- Peak level analysis
- DC offset analysis
- Silence detection
- Clipping detection
- Overall and per-channel metrics
- Configurable metric thresholds
- Structured threshold failure reasons
- Defensive validation of empty and non-finite analysis inputs
- Validation of non-finite signal-generation and detection parameters
analyse-audioCLI command- Human-readable and JSON analysis reporting
- Documented sample-domain analysis limitations
Complete
- Typed loopback validation configuration
- Typed loopback result and failure models
- Backend duplex execution contract
- Deterministic fake-backend duplex execution
- PortAudio duplex playback and capture
- Output-channel routing
- Signal padding for hardware latency
- Captured-signal alignment
- Dominant-frequency measurement
- End-to-end loopback validation service
validate-loopbackCLI command- Human-readable and structured JSON reporting
- Playback, capture and analysed WAV evidence retention
- Structured JSON evidence export
- Physical loopback setup and safety documentation
- Focusrite Scarlett 2i2 hardware validation
- JACK and ALSA host-API comparison
- More advanced signal analysis
- Measured round-trip latency and stream-health validation
- Stability and recovery testing
- Validation profiles and richer report generation
- Python 3.13
- NumPy
- Pydantic
- PyYAML
- sounddevice
- soundfile
- PortAudio
- pytest
- mypy
- Ruff
- Typer
- Rich
This project demonstrates:
- Test automation design
- Configuration-driven testing
- Hardware abstraction
- Audio subsystem awareness (WASAPI, ALSA, JACK)
- Configuration-driven hardware selection
- API design
- Defensive programming
- Cross-platform testing considerations
- CI-friendly test strategies
- Structured logging and reporting preparation
- Python QA tooling
- Automated validation framework development
- Layered application architecture
- Backend-independent workflow orchestration
- Framework-owned exception translation
- Domain-specific analysis exception design
- Deterministic test doubles
- Stream lifecycle validation
- Finite audio execution testing
- Immutable framework-owned audio data models
- WAV file I/O validation
- PortAudio stream lifecycle management
- Timeout, overflow, and underflow handling
- Backend contract enforcement
- Recording and playback service orchestration
- Deterministic synthetic test-signal generation
- Numerical audio-analysis implementation
- Boundary-value testing for audio metrics
- Non-finite numerical input validation
- Defensive error-path testing
- Per-channel validation design
- Configuration-driven metric thresholds
- Structured QA failure reporting
- Sample-domain analysis and limitation documentation
- Human-readable and machine-readable reporting
- Duplex audio execution and validation
- Physical audio loopback automation
- Configurable channel-routing validation
- Captured-signal alignment using normalized correlation
- FFT-based dominant-frequency analysis
- End-to-end hardware/software signal-path validation
- Host-API behaviour comparison
- Evidence-preserving validation workflows
- Structured hardware-validation result reporting
MIT License




















