Zipformer is a streaming Automatic Speech Recognition (ASR) model that runs on the Snapdragon NPU (HTP) via QAI AppBuilder. It transcribes Mandarin Chinese and English (mixed) speech using a 3-stage RNN-T pipeline (encoder → joiner → decoder) with chunked streaming inference.
- Task: Automatic Speech Recognition (ASR)
- Language: Mandarin Chinese + English (mixed)
- Input: WAV audio file (16 kHz mono preferred; auto-resampled if needed)
- Output: Transcribed text string
- Platform: Windows on Snapdragon (WoS), Snapdragon X Elite / X2 Elite
- Runtime: HTP (Hexagon NPU) with native data types (float32 / int32)
- AI Hub Model: Zipformer
Zipformer uses a streaming RNN-T architecture split into three QNN context binaries.
All three run on the NPU via qai_appbuilder.QNNContext.
| Stage | Model File | I/O | Description |
|---|---|---|---|
| 1. Encoder | encoder.bin |
float32 / int32 (native) | Chunked streaming Zipformer encoder; 35 cache tensors I/O + encoder_out [1,16,512] |
| 2. Decoder | decoder.bin |
int32 → float32 (native) | RNN-T prediction network; 2-token context → decoder_out [1,512] |
| 3. Joiner | joiner.bin |
float32 (native) | encoder_out [1,512] × decoder_out [1,512] → logit [1,6254] |
All three
.binfiles useinput_data_type="native"+output_data_type="native"because the encoder cache mixes int32 (cached_len_*) and float32 tensors.
pip install qai_appbuilder numpy soundfile scipy kaldi_native_fbank
kaldi_native_fbankis strongly recommended for accurate feature extraction (matches the sherpa-onnx training recipe). If unavailable, the script falls back to a pure-numpy STFT implementation.Install the ARM64 wheel matching your Python version:
pip install kaldi_native_fbank
ffmpeg(optional, on PATH) enables the silence-detection VAD chunking that is on by default — it skips silent regions and splits long audio at natural pauses before decoding. Ifffmpegis not found (or fails), the script silently falls back to decoding the whole waveform in one pass. Pass--no-use_vadto disable VAD explicitly.
cd qai-appbuilder\samples
# Run with the auto-downloaded test audio (Chinese speech)
python run_inference.py --model zipformer
# Run with a custom audio file
python run_inference.py --model zipformer --args "--audio path\to\audio_16khz.wav"Or run the script directly:
python models\audio\speech_recognition\zipformer\python\zipformer.py
python models\audio\speech_recognition\zipformer\python\zipformer.py --audio path\to\audio.wav| Argument | Default | Description |
|---|---|---|
--audio |
models/test_zh.wav (auto-downloaded) |
Path to input WAV (16 kHz mono preferred) |
--model_dir |
Auto-downloaded model directory | Directory holding encoder.bin / decoder.bin / joiner.bin |
--tokens |
tokens.txt inside model directory |
Path to sherpa-onnx tokens.txt vocabulary file |
--use_vad / --no-use_vad |
on | Enable/disable silence-detection VAD chunking (requires ffmpeg on PATH; falls back to a single-segment decode if unavailable) |
--vad_noise_db |
-35.0 |
[--use_vad] silencedetect noise floor (dB) |
--vad_min_silence |
0.6 |
[--use_vad] Minimum silence duration (s) to split at |
--vad_overlap |
0.3 |
[--use_vad] Segment overlap (s) |
--vad_min_speech |
1.0 |
[--use_vad] Discard speech segments shorter than this (s) |
--vad_max_segment |
25.0 |
[--use_vad] Max segment length (s); longer segments are split with overlap |
All downloaded files are stored in the working directory (WORK_DIR).
By default this is the models/ folder that sits next to the python/ folder:
zipformer/
├── python/
│ └── zipformer.py ← inference script
└── models/ ← WORK_DIR (auto-created on first run)
├── tokens.txt ← vocabulary (auto-downloaded from GitHub)
├── test_en.wav ← English test audio (auto-downloaded from AI Hub S3)
├── test_zh.wav ← Chinese test audio (auto-downloaded from HuggingFace)
└── zipformer-qnn_context_binary-float-qualcomm_snapdragon_x_elite/
├── encoder.bin
├── decoder.bin
├── joiner.bin
└── metadata.json
To use a different location, set the environment variable before running:
set ZIPFORMER_WORK_DIR=C:\MyModels\zipformer
python zipformer.pyOn first run the script automatically downloads:
-
Model ZIP — Detects your device via
install.detect_device_model()(Snapdragon X Elite / X2 Elite), picks the matching AI Hub ZIP, downloads and extracts it viainstall.download_url()(the shared helper insamples/shared/python/install.py). The ZIP containsencoder.bin/decoder.bin/joiner.bin/metadata.json. -
tokens.txt(vocabulary, 6254 BPE tokens) — NOT included in the ZIP, downloaded separately from the officialqualcomm/ai-hub-modelsGitHub repo:src/qai_hub_models/models/zipformer/tokens.txt→ saved toWORK_DIR/tokens.txt. Without this file the output will be raw token ID numbers instead of text. -
Test audio — see Test Audio Download below.
⚠️ The.binfiles are chipset-specific — do not mix X Elite / X2 Elite binaries. The script picks the right ZIP for the detected chipset automatically.
The script also auto-downloads a test audio file on first run. Two sources are tried in order (first successful download wins):
| Priority | File | Source | URL |
|---|---|---|---|
| 1 (primary, default) | test_zh.wav |
HuggingFace / sherpa-onnx — Chinese speech, ~5.6 s, 16 kHz mono | https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23/resolve/main/test_wavs/0.wav |
| 2 (fallback) | test_en.wav |
AI Hub public S3 — English (Common Voice), ~15 s, 241 KB. This is the same audio used by the official qai_hub_models demo (CachedWebModelAsset.from_asset_store("hf_whisper_asr_shared", "1", "audio/common_voice_en_19653650.wav")) |
https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/hf_whisper_asr_shared/v1/audio/common_voice_en_19653650.wav |
Both files are 16 kHz mono WAV. test_zh.wav is the default — if HuggingFace is
unreachable, the script falls back to test_en.wav from the AI Hub public S3
bucket (no authentication required).
| Chipset | URL |
|---|---|
| Snapdragon X Elite | https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/zipformer/releases/v0.55.0/zipformer-qnn_context_binary-float-qualcomm_snapdragon_x_elite.zip |
| Snapdragon X2 Elite | https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/zipformer/releases/v0.55.0/zipformer-qnn_context_binary-float-qualcomm_snapdragon_x2_elite.zip |
| Format | Description | |
|---|---|---|
| Input | WAV file | 16 kHz mono preferred; auto-resampled via scipy.signal if needed |
| Output | String | Transcribed Mandarin Chinese / English text |
Audio file (WAV, 16 kHz mono)
↓ soundfile.read + optional scipy resample
↓ VAD segmentation (ffmpeg silencedetect; on by default, --no-use_vad to skip)
→ list of speech segments, silent regions skipped, long segments split
↓ per segment: kaldi_native_fbank log-mel (80 bins, dither=0, snip_edges=False, high_freq=-400)
Features [T, 80] float32 (per segment; each decoded on its own encoder state)
↓ for each 71-frame chunk (stride=64):
Encoder (NPU, native I/O)
inputs : chunk [1,71,80] + 35 cache tensors (int32/float32)
outputs: 35 new cache tensors + encoder_out [1,16,512]
for t in 0..15:
Joiner (NPU, native I/O)
inputs : encoder_out[t] [1,512] + decoder_out [1,512]
outputs: logit [1,6254]
argmax → if not blank(0): append token
Decoder (NPU, native I/O)
inputs : last 2 token ids [1,2] int32
outputs: decoder_out [1,512]
Token IDs
↓ tokens.txt (sherpa-onnx format)
Transcribed text (string)
On Snapdragon X2 Elite (verified):
- RTF ≈ 0.02 (50× faster than real-time)
- Example output for
test_zh.wav:「对我做了介绍啊那么我想说的是呢大家如果对我的研究感兴趣」
[INFO] Checking / downloading Zipformer models ...
[INFO] Detected device model: snapdragon_x2_elite
[INFO] Model dir : ...\models\zipformer-qnn_context_binary-float-qualcomm_snapdragon_x2_elite
[INFO] Tokens : ...\models\zipformer-qnn_context_binary-float-qualcomm_snapdragon_x2_elite\tokens.txt
============================================================
Zipformer streaming ASR — qai_appbuilder / QNNContext (NPU)
============================================================
[1] Loading audio: ...\models\test_zh.wav
Duration : 5.63 s | Samples: 90080 | SR: 16000
[2] VAD segmentation (ffmpeg silencedetect) ...
Segments : 1 (0.043s)
[3] Extracting 80-dim log-mel features (per segment) ...
Total frames: 351 (0.012s)
[4] Loading vocabulary ...
Vocab size: 6254
[5] Loading QNN context binaries (encoder/decoder/joiner.bin) on HTP ...
Load time: 1.23s
[6] Running streaming greedy search ...
============================================================
RECOGNITION RESULT
============================================================
Text : 对我做了介绍啊那么我想说的是呢大家如果对我的研究感兴趣
Tokens : 31
Segments : 1 (VAD on)
Audio : 5.63s
Infer : 0.11s
RTF : 0.0195 (< 1.0 = faster than real-time)
============================================================
- Cache input order (Issue Z-1): The encoder inputs are ordered by layer
(not by type):
x, then for each of the 5 layers:len/avg/key/val/val2/conv1/conv2. Wrong order causes silent hang or garbled output. - encoder_out is the last output (Issue Z-2):
encoder_out [1,16,512]is at index 35 (the last output), not the first. The first 35 outputs are the new cache. - Native dtype (Issue Z-3):
cached_len_*are int32; all other cache tensors andxare float32. All three models useinput_data_type="native"/output_data_type="native"— do not uniformly cast to float. - Use real speech (Issue Z-4): Synthetic audio (sine wave / silence) produces
empty output. The auto-downloaded
test_zh.wavis a real Chinese speech sample. - VAD chunking is on by default and uses
ffmpeg'ssilencedetectfilter to skip silent regions and split long audio at natural pauses; each speech segment is decoded on its own streaming encoder state so context doesn't leak across pauses. It requiresffmpegon PATH — ifffmpegis missing or fails, the script silently falls back to decoding the whole waveform in one pass. Disable it with--no-use_vadand tune it with the--vad_*arguments. - A harmless
<E> Error 0x200: failed to close queue ...may print on exit. It is a QNN HTP context-teardown warning and does not affect the output. - Files in this directory:
python/zipformer.py— the inference script (auto-downloads models on first run).NOTES.md— detailed model notes: I/O shapes, download URLs, pipeline geometry, known issues (Z-1 through Z-4), and the reference implementation.
- NOTES.md — full technical reference for this model
- whisper_base_en — English-only ASR (Whisper Base)
- whisper_tiny_en — English-only ASR (Whisper Tiny, faster)
- MeloTTS-ZH — Chinese TTS (text → speech)