Skip to content

About

Cross-platform Python framework for audio hardware validation, recording/playback, deterministic signal generation, and sample-domain analysis.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Audio Hardware Automation Framework

CI

A Python-based framework for automated audio hardware testing and validation.

This project demonstrates QA automation, hardware testing, configuration-driven validation, CI-friendly test design, and cross-platform audio device discovery. It is being developed as a portfolio project targeting Audio QA Engineer, Hardware QA Engineer, Embedded QA Engineer, and Software QA Engineer roles within the audio technology industry.


Project Status

Phase 5.1 complete — split-endpoint physical loopback validation

Completed capabilities include:

  • Cross-platform audio device discovery
  • Host API-aware device matching
  • Input-only, output-only, and duplex stream validation
  • PortAudio capability checks
  • Safe stream construction and closure
  • Finite-duration recording
  • Finite-buffer playback
  • Framework-owned audio buffers
  • WAV import and export
  • Deterministic sine-wave and silence generation
  • RMS, peak and DC offset analysis
  • Silence and clipping detection
  • Per-channel sample-domain metrics
  • Configuration-driven metric thresholds
  • Structured threshold failure reasons
  • Duplex playback and capture execution
  • Configurable physical loopback signal routing
  • Silence-padded loopback test signals
  • Captured-signal alignment
  • Dominant-frequency measurement
  • End-to-end loopback metric validation
  • Structured per-channel loopback failures
  • validate-loopback CLI command
  • Loopback WAV and JSON evidence export
  • WAV analysis through the analyse-audio CLI command
  • Backend-independent recording and playback services
  • Rich CLI and structured JSON reporting
  • Windows and Linux hardware validation

Next development: analyse the frequency-domain characteristics of recorded signals and compare them against expected behaviour.

See the Roadmap for planned development.


Project Goals

The framework aims to provide a reusable foundation for automated testing of audio hardware such as:

  • Audio interfaces
  • USB audio devices
  • Embedded audio hardware
  • Audio drivers
  • Recording and playback systems

The current foundation includes reliable device discovery, configuration management, hardware identification, stream-capability validation, finite recording and playback, WAV handling, deterministic signal generation, sample-domain analysis, configuration-driven metric validation, duplex loopback execution, captured-signal alignment, frequency validation, and evidence export.

Future releases will build on the current loopback-validation foundation with measured sample-rate verification, deeper audio-quality measurements, latency observation, and stability testing.


Current Features

Device Discovery

Enumerate audio devices through PortAudio using Python's sounddevice package.

The framework collects:

  • Device name
  • Host API name
  • Host API index
  • Input channel count
  • Output channel count
  • Default sample rate

Example:

Index  Device                                Host API  Inputs  Outputs  Default Rate
0      Analogue 1 + 2 (Focusrite USB Audio)  WASAPI    2       0        48000 Hz
1      Speakers (Focusrite USB Audio)        WASAPI    0       2        48000 Hz
...

Example output shortened for readability.

Supported matching rules:

  • Exact device name
  • Partial device name
  • Host API matching
  • Minimum input channels
  • Minimum output channels

Example:

device:
  name_contains: "Focusrite USB Audio"
  host_api_contains: "WASAPI"
  minimum_input_channels: 2

This allows the framework to distinguish between multiple entries representing the same physical audio interface across different host APIs.


Command-Line Interface

Inspect available audio devices:

audio-hw inspect-devices

Inspect devices using a configuration:

audio-hw inspect-devices --config configs/scarlett_windows_wasapi_input.yaml

Output JSON:

audio-hw inspect-devices --json

Validate a configured stream:

audio-hw validate-stream \
  --config configs/scarlett_windows_wasapi_input.yaml

Output validation results as JSON:

audio-hw validate-stream \
  --config configs/scarlett_windows_wasapi_input.yaml \
  --json

To create a WAV file for playback validation, first ensure the recording configuration contains an output path:

execution:
  duration_seconds: 1.0
  timeout_seconds: 5.0
  output_file: recordings/test.wav

Then run recording validation:

audio-hw validate-recording \
  --config configs/scarlett_windows_wasapi_input.yaml

Then validate playback using the generated recording:

audio-hw validate-playback \
  --config configs/scarlett_windows_wasapi_output.yaml \
  --input recordings/test.wav

Output recording validation results as JSON:

audio-hw validate-recording \
  --config configs/scarlett_windows_wasapi_input.yaml \
  --json

Output playback validation results as JSON:

audio-hw validate-playback \
  --config configs/scarlett_windows_wasapi_output.yaml \
  --input recordings/test.wav \
  --json

Analyse a WAV file using configured sample-domain thresholds:

audio-hw analyse-audio \
  --config configs/example_analysis.yaml \
  --input recordings/test.wav

Output analysis results as JSON:

audio-hw analyse-audio \
  --config configs/example_analysis.yaml \
  --input recordings/test.wav \
  --json

Run physical loopback validation using the verified Linux ALSA configuration:

audio-hw validate-loopback \
  --config configs/example_loopback_alsa.yaml

Output the loopback result as JSON:

audio-hw validate-loopback \
  --config configs/example_loopback_alsa.yaml \
  --json

Save the playback, raw capture, aligned analysis audio and JSON report:

audio-hw validate-loopback \
  --config configs/example_loopback_alsa.yaml \
  --evidence-dir evidence/linux-alsa

validate-stream checks whether PortAudio accepts the requested stream settings and whether the requested stream can be constructed and closed safely.

validate-recording performs a finite recording using the configured duration and timeout. If execution.output_file is configured, the captured audio is exported as a WAV file.

validate-playback reads a WAV file, verifies that its sample rate and channel count match the configured output stream, and performs finite playback through the selected device.

analyse-audio reads the supplied WAV file into a framework-owned AudioBuffer, calculates RMS, peak and DC offset, performs silence and clipping detection, and applies configured thresholds per channel.

validate-loopback generates a deterministic sine wave, routes it to the configured output channel, executes duplex playback and capture, aligns the selected captured input against the reference signal, measures dominant frequency, applies configured sample-domain thresholds, and returns structured pass/fail results.

(When --evidence-dir is supplied, the command retains playback.wav, captured.wav, analysed.wav and report.json for both passing and failing validations.)

Recording, playback and stream-validation commands require exactly one matching device.

analyse-audio is file-based and does not perform device discovery or matching.

Loopback validation can resolve either one shared PortAudio duplex device or separate input and output endpoints.

Separate endpoints are useful on platforms such as Windows where one physical audio interface may be exposed as distinct input-only and output-only PortAudio devices.

Support for separate endpoints does not imply that arbitrary physical devices are sample-clock synchronized. For reliable hardware validation, prefer endpoints belonging to the same physical interface and compatible host API/driver path.


Stream Validation

Configuration-driven validation supports:

  • Input-only streams
  • Output-only streams
  • Duplex streams
  • Sample-rate capability checks
  • Input and output channel checks
  • Strongly typed sample formats
  • Configurable or backend-selected block sizes
  • Safe stream construction and closure
  • Human-readable and JSON results

Validation is implemented through a backend-independent application service. This separates workflow orchestration from PortAudio-specific behaviour and allows deterministic hardware-independent testing.

A successful stream-opening result does not start recording or playback and does not yet prove end-to-end audio signal quality.

See docs/stream-validation.md for validation scope, limitations, exit codes and the hardware test procedure.


Recording and Playback Validation

Phase 3 adds finite audio execution while preserving the framework's backend-independent architecture.

Recording validation supports:

  • Configuration-driven input-device selection
  • Finite-duration capture
  • Exact requested frame-count validation
  • Configurable execution timeout
  • Framework-owned AudioBuffer results
  • Optional WAV export
  • Human-readable and JSON CLI results

Playback validation supports:

  • Configuration-driven output-device selection
  • WAV input
  • Sample-rate compatibility validation
  • Channel-count compatibility validation
  • Finite-buffer playback
  • Configurable execution timeout
  • Human-readable and JSON CLI results

Recording and playback failures are translated through framework-owned backend exceptions.

Phase 3 validates execution and data flow only. It does not yet make claims about:

  • Audio quality
  • Frequency response
  • Distortion
  • Noise level
  • Measured sample-rate accuracy
  • Latency
  • Loopback correctness

Signal Generation and Sample-Domain Analysis

Phase 4 adds deterministic signal generation and basic audio analysis.

Signal generation supports:

  • Sine waves
  • Silence
  • Configurable sample rate
  • Configurable duration
  • Configurable channel count
  • Configurable sine frequency and amplitude
  • Framework-owned AudioBuffer output
  • Deterministic, hardware-independent execution

Sample-domain analysis supports:

  • RMS level
  • Absolute peak level
  • DC offset
  • Silence detection
  • Clipping detection
  • Overall metrics
  • Per-channel metrics
  • Configuration-driven validation thresholds
  • Structured per-channel failure reasons
  • Human-readable and JSON CLI reporting

Example threshold configuration:

thresholds:
  minimum_rms: 0.01
  maximum_rms: 0.8
  maximum_peak: 0.95
  maximum_abs_dc_offset: 0.02
  silence_threshold: 0.0001
  clipping_threshold: 1.0
  fail_on_silence: true
  fail_on_clipping: true

Phase 4 analysis operates exclusively on framework-owned AudioBuffer instances.

Default detection thresholds assume normalized floating-point audio. When analysing raw integer AudioBuffer data directly, threshold values are interpreted in the buffer's native sample units and must be configured accordingly.

It is sample-domain analysis only and does not currently measure frequency response, distortion, signal-to-noise ratio, inter-sample peaks, latency or end-to-end signal-path correctness.

See docs/signal-generation-analysis.md for metric definitions, threshold behaviour, CLI usage and analysis limitations.


End-to-End Loopback Validation

Phase 5 builds on deterministic signal generation and sample-domain analysis with physical duplex loopback validation.

Loopback validation supports:

  • Configurable output and input channel routing
  • Deterministic sine-wave playback
  • Leading and trailing silence padding
  • Backend-independent duplex execution
  • Captured-signal alignment
  • Dominant-frequency measurement
  • RMS, peak and DC-offset validation
  • Silence and clipping detection
  • Structured metric and channel failures
  • Human-readable and JSON CLI output
  • Optional WAV and JSON evidence retention
  • Backward-compatible shared-device duplex selection
  • Independent input and output device selectors
  • Host-API-aware endpoint matching
  • Direction and channel-capability validation
  • Separate PortAudio input and output device indexes
  • Exact input/output endpoint reporting in CLI and JSON evidence

Split-endpoint execution is intended to support host APIs that expose one physical interface through separate capture and playback entries. Whether a particular endpoint pair can operate simultaneously remains dependent on PortAudio, the host API, the driver and the hardware clock topology.

The framework does not provide cross-device clock synchronization or drift compensation.

A physical loopback test requires a line-level connection between the selected hardware output and input.

For the current Linux Focusrite validation environment, direct ALSA at 48 kHz is the verified unattended physical loopback path.

Post-Phase 5 diagnostics also demonstrated successful physical loopback through JACK at 48 kHz. However, the tested JACK graph requires the original PortAudio capture routing to remain in place while the physical Input 1 capture_MONO source is additionally connected manually to PortAudio:in_0.

The PortAudio JACK client is transient, so this manual connection disappears when the validation stream closes and must currently be recreated for each run.

JACK physical loopback is therefore verified but is not yet an unattended framework-only workflow. Software monitor_* routes can also produce passing digital loopback results and are documented separately from physical hardware evidence.

Dedicated ALSA and JACK configurations are retained because host APIs can expose materially different device, routing and duplex behaviour.

See docs/loopback-validation.md for physical setup, safety guidance, host-API considerations, CLI usage, evidence export and validation scope.


Structured Device Models

The framework uses strongly typed Pydantic models for:

  • Audio devices
  • Device matching rules
  • Stream configuration
  • Signal-generation configuration
  • Metric-threshold configuration
  • Framework configuration

This provides validation and predictable configuration handling.


Cross-Platform Design

Current development targets:

  • Windows 11
  • Ubuntu Linux

Audio backends are abstracted to support future expansion.


Automated Testing

The project includes:

  • Unit tests
  • Device matching tests
  • Configuration validation tests
  • Backend abstraction tests
  • Deterministic fake-backend tests
  • PortAudio capability-validation tests
  • Stream-opening tests
  • Recording and playback backend tests
  • Audio-buffer tests
  • WAV import/export tests
  • Recording-service orchestration tests
  • Playback-service orchestration tests
  • Signal-generation model tests
  • Deterministic sine and silence generation tests
  • RMS analysis tests
  • Peak analysis tests
  • DC-offset analysis tests
  • Silence and clipping detection tests
  • Metric-threshold validation tests
  • Non-finite sample and threshold validation tests
  • Analysis failure-path tests
  • CLI analysis tests
  • Duplex backend contract tests
  • Deterministic fake duplex tests
  • PortAudio duplex execution tests
  • Signal routing and padding tests
  • Captured-signal alignment tests
  • Dominant-frequency analysis tests
  • Loopback validation-service tests
  • Loopback CLI tests
  • Loopback evidence-reporting tests

GitHub Actions runs hardware-independent tests, linting, formatting checks, type checking, and coverage reporting on Ubuntu.

Current coverage target:

95%+


Host API Awareness

The framework resolves host API information reported by PortAudio.

Examples:

  • WASAPI
  • MME
  • DirectSound
  • Windows WDM-KS
  • ALSA
  • JACK
  • PulseAudio

This enables more reliable device matching on systems where the same physical interface appears multiple times through different audio subsystems.


Current Architecture

audio-hardware-automation-framework/
│
├── configs/
│   ├── example_analysis.yaml
│   ├── example_duplex_device.yaml
│   ├── example_loopback.yaml
│   ├── example_loopback_alsa.yaml
│   ├── example_loopback_jack.yaml
│   │
│   ├── scarlett_windows_mme_input.yaml
│   ├── scarlett_windows_mme_output.yaml
│   ├── scarlett_windows_directsound_input.yaml
│   ├── scarlett_windows_directsound_output.yaml
│   ├── scarlett_windows_wasapi_input.yaml
│   ├── scarlett_windows_wasapi_output.yaml
│   ├── scarlett_windows_wdmks_input.yaml
│   ├── scarlett_windows_wdmks_output.yaml
│   │
│   ├── scarlett_windows_mme_loopback.yaml
│   ├── scarlett_windows_directsound_loopback.yaml
│   ├── scarlett_windows_wasapi_loopback.yaml
│   │
│   ├── scarlett_linux_alsa.yaml
│   ├── scarlett_linux_jack.yaml
│   ├── scarlett_linux_pulseaudio_input1.yaml
│   ├── scarlett_linux_pulseaudio_input2.yaml
│   └── scarlett_linux_pulseaudio_output.yaml
│
├── docs/
│   ├── discovery.md
│   ├── focusrite-loopback-validation.md
│   ├── jack-loopback-routing.md
│   ├── loopback-validation.md
│   ├── platform-support.md
│   ├── recording-playback-validation.md
│   ├── signal-generation-analysis.md
│   ├── stream-validation.md
│   └── images/
│       ├── phase-1-github-actions-passing.png
│       ├── phase-1-linux-device-discovery.png
│       ├── phase-1-pytest-coverage.png
│       ├── phase-1-wdmks-hotplug-verification.png
│       ├── phase-1-windows-device-discovery.png
│       ├── phase-2-linux-duplex-validation.png
│       ├── phase-2-negative-validation.png
│       ├── phase-2-stream-validation.png
│       ├── phase-3-playback-validation.png
│       ├── phase-3-pytest-coverage.png
│       ├── phase-3-recording-validation.png
│       ├── phase-4-analysis-json.png
│       ├── phase-4-analysis-sine-wave.png
│       ├── phase-4-pytest-coverage.png
│       ├── phase-4-threshold-failure.png
│       ├── phase-5-alsa-physical-loopback.png
│       ├── phase-5-evidence-bundle.png
│       ├── phase-5-jack-loopback-fail.png
│       ├── phase-5-jack-loopback-default-fail.png
│       ├── phase-5-jack-loopback-manual-pass.png
│       ├── phase-5-jack-routing.png
│       ├── phase-5-jack-routing-default-fail.png
│       ├── phase-5-jack-routing-manual-pass.png
│       ├── phase-5-loopback-json.png
│       ├── phase-5-pytest-coverage.png
│       ├── phase-5.1-windows-device-discovery.png
│       ├── phase-5.1-windows-wasapi-pass.png
│       ├── phase-5.1-windows-mme-pass.png
│       ├── phase-5.1-windows-directsound-timeout.png
│       └── phase-5.1-windows-wdmks-blocking-api.png
│
├── src/
│   └── audio_hw_framework/
│       ├── analysis/
│       │   ├── __init__.py
│       │   ├── _samples.py
│       │   ├── alignment.py
│       │   ├── dc_offset.py
│       │   ├── detection.py
│       │   ├── exceptions.py
│       │   ├── frequency.py
│       │   ├── models.py
│       │   ├── peak.py
│       │   └── rms.py
│       │
│       ├── audio/
│       │   ├── __init__.py
│       │   ├── audio_buffer.py
│       │   └── wav.py
│       │
│       ├── backend/
│       │   ├── __init__.py
│       │   ├── base.py
│       │   ├── fake_backend.py
│       │   └── sounddevice_backend.py
│       │
│       ├── configuration/
│       │   ├── __init__.py
│       │   ├── loader.py
│       │   ├── loopback.py
│       │   └── thresholds.py
│       │
│       ├── device/
│       │   ├── __init__.py
│       │   ├── matcher.py
│       │   └── models.py
│       │
│       ├── playback/
│       │   ├── __init__.py
│       │   └── service.py
│       │
│       ├── recording/
│       │   ├── __init__.py
│       │   └── service.py
│       │
│       ├── reporting/
│       │   ├── __init__.py
│       │   └── loopback.py
│       │
│       ├── signal/
│       │   ├── __init__.py
│       │   ├── generator.py
│       │   ├── models.py
│       │   └── transforms.py
│       │
│       ├── validation/
│       │   ├── __init__.py
│       │   ├── audio_metrics.py
│       │   ├── duplex_endpoints.py
│       │   ├── loopback_models.py
│       │   ├── loopback_service.py
│       │   └── service.py
│       │
│       ├── __init__.py
│       ├── __main__.py
│       └── cli.py
│
├── tests/
│   ├── __init__.py
│   └── unit/
│       ├── __init__.py
│       ├── cli_helpers.py
│       ├── fake_backend_helpers.py
│       ├── sounddevice_backend_helpers.py
│       ├── test_analysis_models.py
│       ├── test_analysis_non_finite.py
│       ├── test_audio_buffer.py
│       ├── test_audio_metric_validation.py
│       ├── test_backend_contract.py
│       ├── test_cli_analysis.py
│       ├── test_cli_app.py
│       ├── test_cli_inspect_devices.py
│       ├── test_cli_loopback.py
│       ├── test_cli_playback.py
│       ├── test_cli_recording.py
│       ├── test_cli_stream_validation.py
│       ├── test_configuration.py
│       ├── test_dc_offset.py
│       ├── test_detection.py
│       ├── test_device_matching.py
│       ├── test_duplex_endpoint_resolution.py
│       ├── test_fake_backend_duplex.py
│       ├── test_fake_backend_playback.py
│       ├── test_fake_backend_recording.py
│       ├── test_fake_backend_stream.py
│       ├── test_frequency.py
│       ├── test_loopback_models.py
│       ├── test_loopback_reporting.py
│       ├── test_loopback_service.py
│       ├── test_models.py
│       ├── test_package.py
│       ├── test_peak.py
│       ├── test_playback_service.py
│       ├── test_recording_service.py
│       ├── test_rms.py
│       ├── test_signal_alignment.py
│       ├── test_signal_generator.py
│       ├── test_signal_models.py
│       ├── test_signal_transforms.py
│       ├── test_sounddevice_backend_device.py
│       ├── test_sounddevice_backend_duplex.py
│       ├── test_sounddevice_backend_playback.py
│       ├── test_sounddevice_backend_recording.py
│       ├── test_sounddevice_backend_stream.py
│       ├── test_validation_service.py
│       └── test_wav.py
│
├── pyproject.toml
├── README.md
└── LICENSE

Installation

Clone the repository:

git clone <repository-url>
cd audio-hardware-automation-framework

Create a virtual environment:

python -m venv .venv

Activate it:

Windows:

.venv\Scripts\activate

Linux:

source .venv/bin/activate

Install the project:

pip install -e ".[dev]"

Usage

List Devices

audio-hw inspect-devices

Match Devices Using Configuration

audio-hw inspect-devices \
    --config configs/scarlett_windows_wasapi_input.yaml

JSON Output

audio-hw inspect-devices --json

No Devices Found

When no audio devices are detected:

Table Output

audio-hw inspect-devices

The command displays:

No audio devices detected.

and exits with code 1.

JSON Output

audio-hw inspect-devices --json

The command returns:

{
  "backend": {
    "name": "portaudio",
    "library": "sounddevice",
    "library_version": "..."
  },
  "devices": []
}

and exits successfully.

This behaviour allows automated tooling to distinguish between command failures and valid enumeration results containing no devices.


Example Configuration

device:
  name_contains: "Focusrite USB Audio"
  host_api_contains: "WASAPI"
  minimum_input_channels: 2
  minimum_output_channels: 0

stream:
  sample_rate: 48000
  input_channels: 2
  output_channels: 0
  block_size: null
  dtype: "float32"

execution:
  duration_seconds: 1.0
  timeout_seconds: 5.0
  output_file: null

thresholds:
  minimum_rms: null
  maximum_rms: null
  maximum_peak: null
  maximum_abs_dc_offset: null
  silence_threshold: 0.0001
  clipping_threshold: 1.0
  fail_on_silence: true
  fail_on_clipping: true

Tested Hardware

Focusrite Scarlett 2i2 (3rd Gen)

Current validation:

  • Device enumeration
  • Host API discovery
  • Host API filtering
  • Device matching
  • CLI inspection
  • Configuration loading
  • Cross-platform discovery
  • Windows host API validation (MME, DirectSound, WASAPI, WDM-KS)
  • Linux host API validation (ALSA, JACK, PulseAudio)
  • Audio-stack-specific device matching
  • Configuration-driven unique device selection
  • PortAudio input/output capability validation
  • Input-only, output-only or duplex stream construction
  • Configured block-size application during stream construction
  • Safe stream construction and closure
  • Finite recording through the Scarlett input
  • WAV export of recorded audio
  • Finite playback through the Scarlett output
  • Structured CLI and JSON validation reporting
  • Physical end-to-end loopback validation
  • Verified physical duplex loopback through ALSA at 48 kHz
  • JACK software-loopback validation and physical-routing diagnostics
  • Configurable loopback input/output routing
  • Captured-signal alignment against a generated reference
  • Dominant-frequency validation using a 1 kHz test tone
  • RMS, peak, DC-offset, silence and clipping validation
  • WAV and JSON loopback evidence retention
  • JACK and ALSA host-API comparison
  • Verified Windows physical split-endpoint loopback through WASAPI
  • Verified Windows physical split-endpoint loopback through MME
  • Three-run repeatability validation for WASAPI and MME
  • DirectSound split-endpoint duplex-timeout characterization
  • WDM-KS blocking-stream API limitation characterization

Phase 5 and follow-up loopback findings:

Environment Result Observation
Linux / ALSA / 48 kHz / physical cable PASS Verified unattended physical Scarlett loopback
Linux / JACK / 48 kHz / software monitor route PASS Digital/software route only; not physical hardware evidence
Linux / JACK / 48 kHz / default routing FAIL Expected physical return not captured at a valid level
Linux / JACK / 48 kHz / capture_MONO replacing default route FAIL Physical source alone was insufficient in the tested graph
Linux / JACK / 48 kHz / default routing + capture_MONO PASS Verified physical loopback; requires manual per-run graph modification
Windows / Phase 5 Not executed Scarlett input/output were exposed separately and the original architecture required one duplex device
Windows / WASAPI / Phase 5.1 PASS Verified physical split-endpoint loopback; three consecutive passing runs
Windows / MME / Phase 5.1 PASS Verified physical split-endpoint loopback; three consecutive passing runs
Windows / DirectSound / Phase 5.1 FAIL Individual endpoints passed, but paired duplex execution timed out
Windows / WDM-KS / Phase 5.1 BLOCKED PortAudio reported Blocking API not supported yet during stream opening

The direct ALSA 48 kHz run remains the Phase 5 physical hardware acceptance result because it operates without manual host-routing intervention.

The JACK follow-up demonstrates that physical loopback is also possible through JACK, while identifying a remaining transient-client routing limitation.

Phase 5.1 additionally verifies physical Windows split-endpoint loopback through WASAPI and MME.

The Windows results also demonstrate that host APIs exposing the same physical hardware can behave differently: DirectSound timed out during paired execution and WDM-KS was blocked before duplex execution.

See docs/focusrite-loopback-validation.md for the detailed Phase 5 hardware results and interpretation.

Platforms tested:

  • Windows 11 25H2
  • Ubuntu Linux 26.04 LTS

Documentation

Additional documentation is available in:

Discovery

docs/discovery.md

Explains:

  • What device discovery does
  • What it does not prove
  • Device matching behaviour
  • PortAudio limitations

Platform Support

docs/platform-support.md

Explains:

  • Supported operating systems
  • Host APIs
  • CI limitations
  • Hardware validation scope

Stream Validation

docs/stream-validation.md

Explains:

  • Validation workflow
  • Supported stream directions
  • CLI and JSON output
  • Exit codes
  • Validation scope and limitations
  • Focusrite hardware-validation procedure

Recording and Playback Validation

docs/recording-playback-validation.md

Explains:

  • Recording execution workflow
  • Playback execution workflow
  • Execution configuration
  • WAV import and export
  • CLI and JSON output
  • Failure handling
  • Validation scope and limitations

Signal Generation and Sample-Domain Analysis

docs/signal-generation-analysis.md

Explains:

  • Deterministic sine-wave and silence generation
  • RMS, peak and DC-offset calculations
  • Silence and clipping detection
  • Per-channel metric behaviour
  • Configurable validation thresholds
  • Structured threshold failures
  • analyse-audio CLI and JSON output
  • Exit codes
  • Sample-domain analysis limitations

Physical Loopback Validation

docs/loopback-validation.md

Explains:

  • Physical loopback signal flow
  • Hardware and duplex-device requirements
  • Safe gain and monitoring setup
  • Loopback signal routing and padding
  • JACK and ALSA configuration separation
  • validate-loopback CLI usage
  • JSON output and exit codes
  • WAV and JSON evidence retention
  • Captured-signal alignment
  • Frequency-result interpretation
  • Platform and host-API limitations
  • Current validation scope

Focusrite Loopback Hardware Validation

docs/focusrite-loopback-validation.md

Records:

  • Focusrite Scarlett 2i2 physical loopback validation
  • ALSA 48 kHz passing unattended physical hardware result
  • JACK 48 kHz software-loopback result
  • initial JACK physical-routing failures
  • post-Phase 5 successful JACK physical loopback with manual runtime routing
  • cable A/B/A physical-path verification
  • analogue gain/output-level observations
  • transient PortAudio JACK client limitation
  • correction of invalid preliminary incomplete-loopback ALSA results
  • Windows Phase 5 single-device duplex limitation
  • Phase 5.1 Windows split-endpoint physical validation
  • WASAPI and MME physical split-endpoint PASS results
  • DirectSound duplex-timeout characterization
  • WDM-KS blocking-API limitation
  • measured frequency and sample-domain results
  • host-API and routing comparison
  • evidence interpretation
  • Phase 5 acceptance criteria and limitations

JACK Loopback Routing Investigation

docs/jack-loopback-routing.md

Records:

  • PipeWire-backed JACK environment observations
  • JACK-visible Scarlett capture and monitor paths
  • default PortAudio routing failure
  • capture_MONO-only routing failure
  • successful combined default + capture_MONO topology
  • transient PortAudio JACK client behaviour
  • cable A/B/A physical-path verification
  • gain-dependent physical capture behaviour
  • distinction between physical and software-monitor loopback
  • remaining routing-automation questions

Evidence

Phase 1

Windows device discovery

Windows device discovery

Linux device discovery

Linux device discovery

GitHub Actions

GitHub Actions passing

pytest coverage

pytest coverage

WDM-KS hotplug verification

WDM-KS hotplug verification

Phase 2

Stream validation (JSON)

stream validation (JSON)

Linux duplex validation

linux duplex validation

Negative stream validation

negative stream validation

Phase 3

Recording validation

recording validation

Playback validation

playback validation

pytest coverage

pytest coverage

Phase 4

Known sine-wave analysis

known sine-wave analysis

Structured JSON analysis

structured JSON analysis

Threshold failure reporting

threshold failure reporting

pytest coverage

pytest coverage

Phase 5

ALSA physical loopback validation

The Focusrite Scarlett 2i2 successfully completed end-to-end physical loopback validation through direct ALSA at 48 kHz.

ALSA physical loopback validation

A representative physical capture measured approximately:

Frequency:      1000.000 Hz
RMS:            0.014242
Peak:           0.021235
DC offset:      0.000005
Silence:        False
Clipping:       False

The physical signal path was:

framework
    ↓
Scarlett output
    ↓
physical line-level cable
    ↓
Scarlett input
    ↓
framework capture

This is the primary Phase 5 hardware acceptance result.

Structured JSON evidence

The validation result is also available as structured machine-readable JSON containing backend, device, stream, routing, frequency, metric and artifact information.

Loopback JSON evidence

Evidence bundle

A completed validation retains playback, raw capture, aligned analysis audio and the structured report.

Loopback evidence bundle

JACK routing follow-up

Post-Phase 5 diagnostics established that physical loopback through JACK is possible, but the tested environment requires manual runtime graph modification.

With the default PortAudio routing alone, the expected physical return was absent.

Replacing the default Input 1 connection with the Scarlett physical capture_MONO source also failed.

The successful topology retained the original PortAudio capture routing and additionally connected the physical Input 1 capture_MONO source to PortAudio:in_0.

A representative physical JACK PASS measured:

Expected frequency:  1000.000 Hz
Measured frequency:  999.965 Hz
RMS:                 0.032995
Peak:                0.046741
Silence:             False
Clipping:            False

Cable A/B/A testing produced:

disconnected → FAIL
connected    → PASS
disconnected → FAIL

confirming that the successful result depended on the physical Scarlett output-to-input path.

The PortAudio JACK client exists only while validation is running, so the required additional connection disappears after each run and must currently be recreated manually.

Direct ALSA therefore remains the primary unattended Phase 5 hardware acceptance path.

See docs/jack-loopback-routing.md for the full routing investigation.

JACK physical loopback routing

JACK physical loopback pass

Automated regression suite

Phase 5 pytest coverage

The Phase 5 hardware-independent regression suite completed with:

400 passed
99% coverage

See docs/focusrite-loopback-validation.md for the complete result matrix and interpretation.


Roadmap

Phase 1 — Device discovery and configuration

Complete

  • Backend abstraction
  • PortAudio device enumeration
  • Typed YAML configuration
  • Host API-aware device matching
  • Rich and JSON device inspection
  • Cross-platform hardware discovery

Phase 2 — Stream capability and opening validation

Complete

  • Strongly typed stream configuration
  • Input-only, output-only, and duplex validation
  • PortAudio capability checks
  • Stream construction and safe closure
  • Validation orchestration service
  • validate-stream CLI command
  • Structured validation reporting
  • Windows and Linux hardware validation

Phase 3 — Recording and playback foundations

Complete

  • Finite-duration recording
  • Finite-buffer playback
  • Framework-owned audio buffer abstraction
  • WAV import and export
  • Deterministic fake recording and playback
  • PortAudio recording and playback
  • Recording execution service
  • Playback execution service
  • validate-recording CLI command
  • validate-playback CLI command
  • Human-readable and JSON execution reporting

Phase 4 — Signal generation and basic audio analysis

Complete

  • Deterministic sine-wave generation
  • Deterministic silence generation
  • Typed signal-generation configuration
  • Framework-owned generated audio buffers
  • RMS level analysis
  • Peak level analysis
  • DC offset analysis
  • Silence detection
  • Clipping detection
  • Overall and per-channel metrics
  • Configurable metric thresholds
  • Structured threshold failure reasons
  • Defensive validation of empty and non-finite analysis inputs
  • Validation of non-finite signal-generation and detection parameters
  • analyse-audio CLI command
  • Human-readable and JSON analysis reporting
  • Documented sample-domain analysis limitations

Phase 5 — End-to-end physical loopback validation

Complete

  • Typed loopback validation configuration
  • Typed loopback result and failure models
  • Backend duplex execution contract
  • Deterministic fake-backend duplex execution
  • PortAudio duplex playback and capture
  • Output-channel routing
  • Signal padding for hardware latency
  • Captured-signal alignment
  • Dominant-frequency measurement
  • End-to-end loopback validation service
  • validate-loopback CLI command
  • Human-readable and structured JSON reporting
  • Playback, capture and analysed WAV evidence retention
  • Structured JSON evidence export
  • Physical loopback setup and safety documentation
  • Focusrite Scarlett 2i2 hardware validation
  • JACK and ALSA host-API comparison

Later phases

  • More advanced signal analysis
  • Measured round-trip latency and stream-health validation
  • Stability and recovery testing
  • Validation profiles and richer report generation

Technologies

  • Python 3.13
  • NumPy
  • Pydantic
  • PyYAML
  • sounddevice
  • soundfile
  • PortAudio
  • pytest
  • mypy
  • Ruff
  • Typer
  • Rich

QA Engineering Skills Demonstrated

This project demonstrates:

  • Test automation design
  • Configuration-driven testing
  • Hardware abstraction
  • Audio subsystem awareness (WASAPI, ALSA, JACK)
  • Configuration-driven hardware selection
  • API design
  • Defensive programming
  • Cross-platform testing considerations
  • CI-friendly test strategies
  • Structured logging and reporting preparation
  • Python QA tooling
  • Automated validation framework development
  • Layered application architecture
  • Backend-independent workflow orchestration
  • Framework-owned exception translation
  • Domain-specific analysis exception design
  • Deterministic test doubles
  • Stream lifecycle validation
  • Finite audio execution testing
  • Immutable framework-owned audio data models
  • WAV file I/O validation
  • PortAudio stream lifecycle management
  • Timeout, overflow, and underflow handling
  • Backend contract enforcement
  • Recording and playback service orchestration
  • Deterministic synthetic test-signal generation
  • Numerical audio-analysis implementation
  • Boundary-value testing for audio metrics
  • Non-finite numerical input validation
  • Defensive error-path testing
  • Per-channel validation design
  • Configuration-driven metric thresholds
  • Structured QA failure reporting
  • Sample-domain analysis and limitation documentation
  • Human-readable and machine-readable reporting
  • Duplex audio execution and validation
  • Physical audio loopback automation
  • Configurable channel-routing validation
  • Captured-signal alignment using normalized correlation
  • FFT-based dominant-frequency analysis
  • End-to-end hardware/software signal-path validation
  • Host-API behaviour comparison
  • Evidence-preserving validation workflows
  • Structured hardware-validation result reporting

License

MIT License

About

Cross-platform Python framework for audio hardware validation, recording/playback, deterministic signal generation, and sample-domain analysis.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages