We tested a new Amharic speech dataset against seven ASR systems

Most ASR benchmarks start with the model. We started with a more basic question: can we trust the dataset?

Dataset.ET Amharic Speech v0.1.0 is a new open corpus with 22.7 hours of read speech, 7,405 clips, and 320 contributors. Its test split contains 718 clips from 34 speakers, totaling 2.22 hours. No speaker or prompt in the test split appears in training or validation.

That design looks good on paper. We wanted evidence from the audio itself. So we ran the same test split through seven speech-to-text systems: Addis AI STT, Google Chirp 3, three Addis Scribe variants, a compact 0.6B baseline, and Meta MMS-1B.

Every system received identical audio and references. Every output went through the same Amharic normalization and jiwer scoring pipeline. None of the systems was trained or tuned on this test set.

The full results

We ranked the systems by Word Error Rate. Lower CER and WER are better. The table reports the benchmark result after applying the same normalization to every system.

Addis AI STT finished 7.9 WER points ahead of Chirp 3. Addis Scribe md led the compact tier at 41.8, 16.8 points behind the top result. That is a real separation, but the leaderboard is only half of what this run tells us.

The dataset passed the harder test

A best WER of 25.0 can trigger an obvious suspicion: maybe the reference labels are wrong. We checked that directly.

The references are clean, formal Amharic. More importantly, strong recognizers track them closely at the character level. When two independent production systems reach CER 8.2 and 14.3 on the same external split, the audio and labels are not broadly drifting apart.

The corpus construction supports that result. Contributors read known prompts, validators checked whether each recording matched, clips with any reject vote were excluded, and the remaining audio went through acoustic screening. The test split is also speaker-disjoint and prompt-disjoint.

That is enough evidence for us to treat this as a credible neutral test set. It is not a perfect representation of every kind of Amharic speech, but the ground truth is usable.

WER is also measuring transcription style

The raw WER numbers are honest, but they need context. A meaningful share of the errors comes from formatting differences rather than failed recognition.

The content is the same. The reference spells out 4 ነጥብ 7 and 7 ነጥብ 6; the recognizer writes 4.7 and 7.6. WER counts those as errors even though the transcript remains correct for a reader.

The same thing happens with joined and split compounds, hyphens, punctuation, and common spelling variants. In this test set, 21% of references contain digits and 16% contain punctuation that compact models were not trained to emit consistently.

We kept the reported scores unchanged. A benchmark becomes less useful when the normalization is quietly tuned after seeing the outputs. A second score with explicit number and punctuation normalization would still be useful, but it should sit next to the reported result, not replace it.

What the model spread tells us

Production systems generalize better

Addis AI STT and Google Chirp 3 occupy the top tier on external Amharic audio. Neither system saw this test set during training or tuning. The result is evidence of broader coverage, not familiarity with one benchmark distribution.

The compact tier is useful, but there is still a gap

Addis Scribe md is the best compact model measured at WER 41.8. The sm and base variants follow at 44.0 and 48.1. These models can produce useful Amharic transcripts, but out-of-distribution coverage still separates them from the production APIs.

CER and WER expose different failures

Meta MMS-1B reaches CER 16.21, better than every Addis Scribe variant on characters, yet finishes last on WER at 52.0. The compact 0.6B baseline shows the same pattern with CER 19.30 and WER 49.6.

Those systems are hearing much of the acoustic content, but word boundaries and transcript formatting remain weak. CER alone makes them look stronger than the transcripts feel. WER alone hides how much of the underlying speech they captured. For Amharic, we need both.

What this dataset can and cannot measure

The full release is useful because it is open, licensed under CC BY 4.0, and explicit about how it was collected. The 7,405 clips are 16 kHz audio with speaker metadata, acoustic measurements, peer-validation votes, and speaker-disjoint splits.

Its limits are equally important. This is read speech, not conversation. The prompts lean formal and toward current affairs. Contributors skew young and urban: 75.1% of clips come from people aged 18 to 24, and 47.5% come from Addis Ababa. The original recordings also passed through Telegram voice messages, so device and codec artifacts are part of the corpus.

A model that scores well here has handled clean, prompted Amharic across held-out speakers. That does not automatically tell us how it handles phone calls, overlapping speech, rural accents, code-switching, or spontaneous conversation.

Method

Two answers from one benchmark

The model result is straightforward. Addis AI STT led the field at WER 25.0 and CER 8.2, with Google Chirp 3 second. Addis Scribe md led the compact tier.

The dataset result matters beyond our own models. Independent systems align closely with its references, the test design avoids speaker and prompt overlap, and the remaining errors are explainable. snapwre/amharic-speech is a dataset we can train on and benchmark against with confidence, as long as we stay honest about the type of speech it represents.