This is Part 2 of a series. Part 1 covered the initial model selection and design decisions. Part 3 will have the actual test results.
Part 1 ended with an ensemble of five models and a GPU layout. This part is a step back: before running anything, it's worth understanding what the benchmark numbers actually mean, whether the leaderboard rankings translate to real meeting recordings, and — most honestly — which model I'd missed entirely the first time around.
How the Open ASR Leaderboard works
The Open ASR Leaderboard is a collaboration between Hugging Face, NVIDIA, Mistral AI, and the University of Cambridge. It evaluates 86 models (open-source and commercial APIs) across 12 standardised datasets, producing reproducible results using open-source toolkits (ESPNet, NeMo, SpeechBrain, HuggingFace Transformers). The accompanying paper (arXiv:2510.06961) describes the methodology in detail.
The 12 datasets span several distinct conditions:
| Dataset | What it tests |
|---|---|
| LibriSpeech test-clean | Read speech from audiobooks. Studio-quality, single speaker, no noise. |
| LibriSpeech test-other | Same audiobook data but harder accents and recording conditions. |
| TED-LIUM 3 | Conference talks. Scripted or semi-scripted, moderate noise. |
| AMI meeting corpus | Real multi-speaker meetings. Overlapping speech, room acoustics, variable mic quality. |
| GigaSpeech | Web audio: podcasts, YouTube, diverse conditions. |
| Earnings-22 | Financial earnings calls. Domain-specific vocabulary, telephone audio quality. |
| CallHome | Conversational telephone calls between family members. |
| VoxPopuli | European Parliament speech (multilingual track). |
| Common Voice | Crowdsourced speech across many languages. |
| FLEURS | Read sentences across 102 languages. |
| MLS (Multilingual LibriSpeech) | Audiobook data for European languages. |
| CV 16 | Common Voice, specific to 16 languages. |
The English average WER that headlines the leaderboard is a mean across a subset of these. LibriSpeech, TED-LIUM, AMI, GigaSpeech, Earnings-22, CallHome, and Common Voice all feed into it. A model's headline figure is therefore a blend of clean and noisy conditions — but the blend is dominated by the cleaner datasets, which makes the headline look better than meeting-only performance would suggest.
The gap between benchmarks and meeting recordings
This is the most important thing to understand before picking models for this use case.
LibriSpeech test-clean is read speech from audiobooks, recorded in quiet conditions, by a single speaker per file, with a professional microphone. It is the easiest English ASR benchmark by a considerable margin. NVIDIA Parakeet-TDT-0.6b-v2 achieves 1.69% WER on LibriSpeech test-clean. On the AMI meeting corpus — which captures real multi-speaker meetings in rooms — the same model scores 11.16% WER. That's a 6.6× degradation.
Whisper large-v3 degrades less sharply (around 3.5×) on noisy conditions because it was trained on noisier, more diverse data. But it starts from a higher WER on clean benchmarks, so it ends up in a similar place in practice. IBM Granite Speech 3.3 reports only 7.54% relative degradation from clean to noisy conditions — unusually low, suggesting deliberate noise-robustness training.
The upshot: for selecting models for poor-quality office recordings, the AMI score and any available noise-condition figures are more relevant than the LibriSpeech headline. The leaderboard presents both where available; it's worth checking rather than reading only the top-line average.
A more complete model comparison
Here's the picture when you include all the serious contenders, with the metrics that matter most for this use case:
| Model | HF repo | Avg WER | AMI WER | Noise notes | Timestamps | Licence | VRAM (float16) |
|---|---|---|---|---|---|---|---|
| Canary-Qwen-2.5B | nvidia/canary-qwen-2.5b |
5.63% | — | 2.41% at 10 dB SNR | forced align | CC-BY-4.0 | ~8 GB |
| IBM Granite 3.3 8B | ibm-granite/granite-speech-3.3-8b |
5.74% | — | 7.54% relative degradation | forced align | Apache-2.0 | ~18 GB |
| IBM Granite 3.3 2B | ibm-granite/granite-speech-3.3-2b |
~6.00% | — | similar noise profile | forced align | Apache-2.0 | ~5 GB |
| Qwen3-ASR-1.7B | Qwen/Qwen3-ASR-1.7B |
~6% | — | 16.17% vs Whisper's 63.17% under extreme noise | native | Apache-2.0 | ~4 GB |
| Phi-4-multimodal | microsoft/Phi-4-multimodal-instruct |
6.14% | — | — | no native (40-sec cap) | MIT | ~12 GB |
| Parakeet-TDT-0.6b-v2 | nvidia/parakeet-tdt-0.6b-v2 |
6.05% | 11.16% | — | native | CC-BY-4.0 | ~2 GB |
| CrisperWhisper | nyrahealth/CrisperWhisper |
6.67% | — | disfluency-aware | specialised | — | ~6 GB |
| Kyutai STT-2.6b-en | kyutai/stt-2.6b-en |
6.4% | — | 2-hr files, streaming | token-level | CC-BY-4.0 | ~6 GB |
| Voxtral-Mini-3B | mistralai/Voxtral-Mini-3B-2507 |
— | — | beats Whisper-v3 | no native (30-min cap) | Apache-2.0 | ~9.5 GB |
| Whisper large-v3 (WhisperX) | openai/whisper-large-v3 |
7.4% | ~9% | diverse training data | wav2vec2 align | MIT | ~6 GB |
| Gemma 4 E4B | google/gemma-4-E4B-it |
— | — | — | no native (30-sec cap) | Apache-2.0 | ~8 GB |
The commercial-API leaders (ElevenLabs on long-form, RevAI, Speechmatics) don't appear here because there's no self-hosted option. For transcribing client-confidential recordings, sending audio to a third-party API isn't on the table regardless of the accuracy numbers.
The model I missed: Qwen3-ASR
Qwen3-ASR (GitHub: QwenLM/Qwen3-ASR) was released by Alibaba Cloud's Qwen team at the end of January 2026, under Apache-2.0, covering 52 languages. The technical report is arXiv:2601.21337. It has over 700 million downloads on HuggingFace, which suggests it's found a real audience.
The number that makes it relevant here: under extreme noise conditions, Qwen3-ASR-1.7B achieves 16.17% WER compared to Whisper large-v3's 63.17% — a 3.9× advantage. In moderate noise the gap narrows, but the model was specifically designed to handle acoustic environments that simpler models fall apart on: significant ambient noise, background music, overlapping voices, regional dialects.
For recordings made in server rooms, open-plan offices, and cafes — which describes several of the recordings I want to transcribe — this is the most relevant differentiator on the whole leaderboard.
Two variants: Qwen3-ASR-1.7B (flagship, maximum accuracy) and Qwen3-ASR-0.6B (lightweight, ~600M parameters). The 1.7B fits at ~4 GB in float16, small enough to join the dedicated ASR pack on GPU 2.
The reason it wasn't in Part 1: it surfaced only in the follow-up review pass, after the initial research agents had already completed. That's now corrected.
Some honest caveats about the plan
On the LLM reconciler. The GenSEC paper (arXiv:2409.09785) is unambiguous that zero-shot prompt-only LLMs frequently make ASR output worse — they're fluent but they hallucinate. The literature's wins on GER come from fine-tuned models (Whispering LLaMA, arXiv:2310.06434, being the canonical example). I'm using a general-purpose Qwen3-14B-AWQ with a carefully constrained prompt. That's a reasonable starting point but not the same thing, and I'll be watching the per-segment hypotheses in the JSON output early on to see whether the LLM is actually helping or just confidently rearranging the furniture.
On pyannote diarisation. Pyannote works well on clean two-speaker audio with good microphone separation. Meeting recordings — overlapping speech, varying acoustic distance, sometimes three or four speakers — are harder. Diarisation error rates on real meetings can be 20–30% even with good tooling. The timestamps will be accurate (wav2vec2 forced alignment is robust); the speaker attribution will be best-effort. The auditable transcript.json with per-segment hypotheses lets you check and correct speaker assignments manually if needed.
On the short audio caps. Phi-4-multimodal handles 40 seconds per ASR call; Voxtral handles 30 minutes; Gemma 4 handles 30 seconds. All three need the audio chunked with overlapping boundaries before submission. The preprocessing step handles this — but it's worth noting that the "just POST the file" simplicity only applies to Parakeet, Canary-Qwen, WhisperX, and Qwen3-ASR.
The revised ensemble
With Qwen3-ASR added and whisper-large-v3-turbo (which was in the GPU table in Part 1 but never properly introduced) replaced:
| GPU | Role | Models |
|---|---|---|
| GPU 0 | Voice LLM transcription | Voxtral-Mini-3B-2507 (~9.5 GB bf16) |
| GPU 1 | Voice LLM transcription | Phi-4-multimodal-instruct (~12 GB) |
| GPU 2 | Dedicated ASR pack | WhisperX (large-v3) + Parakeet-TDT-0.6b-v2 + Qwen3-ASR-1.7B |
| GPU 3 | Reconciliation + summary | Qwen3-14B-AWQ via vLLM |
WhisperX remains the diarisation and timing backbone. Parakeet contributes native word timestamps and extraordinary throughput. Qwen3-ASR adds the noise-robustness the ensemble was previously missing. Voxtral and Phi-4 give two architecturally distinct voice-LLM hypotheses. Kyutai STT, Canary-Qwen, and IBM Granite 3.3 2B are on deck as optional additions if GPU 2 VRAM headroom allows — at build time I'll measure rather than guess.
What we're about to test
The recordings I'm working with include:
- 60–90 minute meetings in university offices (reasonable quiet, 2–3 speakers, formal vocabulary)
- Site visits at client premises (background machinery, 2–4 speakers, more variable)
- A couple of recordings from what I can only describe as "enthusiastically air-conditioned corridors"
For each, the pipeline will produce five hypotheses per segment. Part 3 will show where the models agreed, where they disagreed, how the LLM reconciler resolved it, and whether the final transcript reads better or worse than any individual engine's output. I'll also have the diarisation accuracy to report — at least for the recordings where I know who was speaking when.
The architecture is built. Time to find out if the benchmarks mean anything on real audio.
References
- Open ASR Leaderboard
- Open ASR Leaderboard paper (arXiv:2510.06961)
- HuggingFace blog: Open ASR Leaderboard trends
- Qwen3-ASR GitHub (QwenLM/Qwen3-ASR)
- Qwen3-ASR-1.7B model card
- Qwen3-ASR-0.6B model card
- Qwen3-ASR technical report (arXiv:2601.21337)
- Qwen3-ASR overview (Algo-Mania)
- NVIDIA Canary-Qwen-2.5B model card
- IBM Granite Speech 3.3 8B model card
- IBM Granite tops Hugging Face leaderboard (IBM Research)
- NVIDIA Parakeet-TDT-0.6b-v2 model card
- microsoft/Phi-4-multimodal-instruct model card
- mistralai/Voxtral-Mini-3B-2507 model card
- nyrahealth/CrisperWhisper model card
- kyutai/stt-2.6b-en model card
- google/gemma-4-E4B-it model card
- Google Gemma 4 announcement
- GenSEC challenge paper (arXiv:2409.09785)
- Whispering LLaMA GER framework (arXiv:2310.06434)
- Best open-source STT models 2026 with benchmarks (Northflank)
- ASR in 2025–2026: A Deep Dive (Ruoqi Jin)
- NVIDIA, Microsoft, ElevenLabs top ASR leaderboard (Slator)
- pyannote-audio (GitHub)
Part 3 will have the actual numbers. Et volia — or not. We'll see.
