PESTE (PErsian Speech to TExt) is a benchmark and leaderboard for Persian automatic speech recognition.
- Release:
v2.1.0 - Suite:
fleurs-fa-ir-v1, 871 test recordings - Normalization: immutable
fa-v1 - Accuracy: corpus CER (primary), WER, deterministic bootstrap uncertainty, and paired CER comparisons
- Speed: steady-state end-to-end audio throughput and real-time factor (RTF)
- Official profile: one NVIDIA RTX 6000 Ada Generation 48 GB GPU (
rtx-6000-ada-v1) - Inference: deterministic per-model batching, checkpoint-native precision, automatic native Whisper long-form decoding, offline execution, and read-only dataset/checkpoint caches
- Runtime: digest-pinned public GHCR image with isolated modern and NeMo environments
The tables show the top 10 models. See the full leaderboard for complete standings.
| Order | Model | CER | WER | Word accuracy |
|---|---|---|---|---|
| 1 | whisper-persian-v4-nezamisafa | 0.0493 95% CI: 0.0443–0.0550 |
0.1312 95% CI: 0.1233–0.1393 |
86.88% |
| 2 | whisper-large-v2-fa-aictsharif | 0.0510 95% CI: 0.0445–0.0589 |
0.2326 95% CI: 0.2233–0.2424 |
76.74% |
| 3 | visualears-fastconformer-fa-full-ab | 0.0518 95% CI: 0.0475–0.0565 |
0.1552 95% CI: 0.1481–0.1624 |
84.48% |
| 4 | whisper-large-fa-v1-vhdm | 0.0535 95% CI: 0.0473–0.0611 |
0.1448 95% CI: 0.1361–0.1542 |
85.52% |
| 5 | whisper-large-persian-steja | 0.0584 95% CI: 0.0538–0.0636 |
0.2643 95% CI: 0.2554–0.2732 |
73.57% |
| 6 | whisper-large-v3 | 0.0605 95% CI: 0.0560–0.0655 |
0.2044 95% CI: 0.1961–0.2129 |
79.56% |
| 7 | persian-whisper-large-v3-10-percent-17-0-one-epoch-mohammadreza-halakoo | 0.0623 95% CI: 0.0574–0.0675 |
0.2743 95% CI: 0.2654–0.2831 |
72.57% |
| 8 | whisper-large-v3-turbo | 0.0629 95% CI: 0.0570–0.0699 |
0.2042 95% CI: 0.1951–0.2135 |
79.58% |
| 9 | shenava-rizeh-v1-0 | 0.0640 95% CI: 0.0589–0.0694 |
0.1555 95% CI: 0.1470–0.1638 |
84.45% |
| 10 | whisper-medium-fa-aictsharif | 0.0658 95% CI: 0.0517–0.0836 |
0.2604 95% CI: 0.2434–0.2804 |
73.96% |
CER is primary because Persian WER is sensitive to word segmentation. The 95% intervals use a deterministic 10,000-replicate utterance bootstrap with seed 20250731; point-estimate order alone does not establish a significant difference.
| Rank | Model | Batch | Throughput | RTF | Processing s | Audio s |
|---|---|---|---|---|---|---|
| 1 | nvidia-fastconformer-fa | 32 | 1199.914× | 0.00083 | 11.102 | 13321.860 |
| 2 | shenava-rizeh-v1-0 | 16 | 675.031× | 0.00148 | 19.735 | 13321.860 |
| 3 | wav2vec2-large-xlsr-53-persian | 1 | 470.167× | 0.00213 | 28.334 | 13321.860 |
| 4 | visualears-fastconformer-fa-full-ab | 32 | 378.349× | 0.00264 | 35.211 | 13321.860 |
| 5 | wav2vec2-large-xlsr-persian-v3-masoumehb | 1 | 296.335× | 0.00337 | 44.955 | 13321.860 |
| 6 | wav2vec2-xls-r-300m-fa-alifarokh | 1 | 291.422× | 0.00343 | 45.713 | 13321.860 |
| 7 | xls-r-1b-fa-cv8-ghofrani | 1 | 289.030× | 0.00346 | 46.092 | 13321.860 |
| 8 | wav2vec2-large-xlsr-persian-v2-m3hrdadfi | 1 | 288.740× | 0.00346 | 46.138 | 13321.860 |
| 9 | persian-speech-transcription-wav2vec2-v1-seyedali | 1 | 286.634× | 0.00349 | 46.477 | 13321.860 |
| 10 | wav2vec2-large-xlsr-persian-shemo-m3hrdadfi | 1 | 285.490× | 0.00350 | 46.663 | 13321.860 |
Throughput is total audio seconds divided by measured processing seconds; RTF is its reciprocal. Resumed runs retain accuracy but are excluded here.
| Model | CER (95% CI) | Throughput (× real time) |
|---|---|---|
| nvidia-fastconformer-fa | 0.2646 (0.2552–0.2741) | 1199.914× |
| shenava-rizeh-v1-0 | 0.0640 (0.0589–0.0694) | 675.031× |
| visualears-fastconformer-fa-full-ab | 0.0518 (0.0475–0.0565) | 378.349× |
| whisper-large-v2-fa-aictsharif | 0.0510 (0.0445–0.0589) | 4.597× |
| whisper-persian-v4-nezamisafa | 0.0493 (0.0443–0.0550) | 2.445× |
A model is Pareto-efficient when no other speed-valid model has equal-or-lower CER and equal-or-higher throughput with at least one strict advantage. This is a trade-off classification, not a composite score; JSON and CSV artifacts retain the full dominance analysis.
Install the locked environment, configure Vast.ai, and use the immutable image reference from a
successful Runtime image workflow artifact:
uv sync --frozen --all-groups
uv run vastai set api-key <key>
image_ref=$(jq -r .image_reference runtime-image.json)
uv run peste cloud up --image "$image_ref" --max-dph <maximum-hourly-price>
uv run peste dataset prepare --suite fleurs-fa-ir-v1 --host <ssh-url>
uv run peste model validate --model <model-id> --host <ssh-url>
uv run peste run --suite fleurs-fa-ir-v1 --model <model-id> --host <ssh-url>
uv run peste leaderboard --suite fleurs-fa-ir-v1
uv run peste cloud downAlways destroy the instance after copying results. See the maintainer guide for image builds, batch calibration, campaigns, recovery, and publication checks.
The suite verifies pinned FLEURS audio against its committed manifest and applies immutable fa-v1
normalization to references and predictions. CER ignores normalized whitespace; WER preserves word
boundaries. See the benchmark contract for the complete rules.
- Benchmark contract
- Adding a model
- Maintainer guide
- Contributing source changes
- Version roadmap
- Changelog
FLEURS is public read speech and may overlap model training data. It does not represent all conversational, noisy, accented, domain-specific, streaming, or long-form Persian. The benchmark does not evaluate timestamps, diarization, punctuation quality, training uncertainty, deployment robustness, or production suitability. Prompts, hotwords, quantization, offload, compilation, external language models, and policy-changing fallbacks are excluded.
Accuracy intervals describe test-utterance sampling uncertainty. Speed is one uninterrupted pass per model and has no repeated-run confidence interval. PESTE 2.1 regenerates the 29 affected Whisper results while retaining 13 unchanged non-Whisper 2.0 bundles; the generated JSON/CSV and individual result bundles carry the exact model, image, source, and hardware provenance.
The Shenava collection includes FLEURS-fa evaluation artifacts, and the
visualears-fastconformer-fa-full-ab
card reports external-language-model calibration on a FLEURS-256 slice. PESTE used the fixed
default RNNT decoder without that language model, but the published VisualEars score should not be
treated as clean held-out evidence.
PESTE code and definitions are Apache-2.0. FLEURS content is CC BY 4.0; see NOTICE.
@software{jafarnezhad_peste_persian_speech_to_text_benchmark,
author = {Jafarnezhad, Arman},
title = {PESTE: Persian Speech to Text benchmark},
year = {2026},
url = {https://github.com/ArmanJR/PESTE-Benchmark},
version = {2.1.0}
}