Intron Health

What was judged

A worked example

One real spoken question from the benchmark, with the answer a health worker wrote for it.

The question, the reference answer and what the model is expected to do are the benchmark’s own worked example. The scores on the right are the ones a good answer should get, each with the mistake that would lose it. The plain words row explains the clinical terms.

Evaluation results

The speech to text scores can be broken down by language, by accent, and by how noisy the recording is.

Lower is better. Each number is that model’s average across the 17 languages in the table below. Where a model was run on fewer, the number in brackets is how many.

• Azure Speech Has no score for Akan, Hausa, Igbo or most of the Bantu languages. That makes a direct comparison partial.

WER (normalised). A score above 100% means the model wrote more wrong words than there were words spoken.

• ★ The lowest word error rate for that language.

• Intron Sahara and Meta OmniLLM. Intron Sahara and Meta OmniLLM are the strongest overall on the African languages, at 24.4% and 26.8% of words wrong. Azure Speech has the lowest average of all at 21.2%, on the 7 languages it was run on.
• Swahili and French. Swahili and French are the easiest languages. The lowest word error rate on Swahili is 6.8% and on French 5.5%.
• Akan, Pedi and Tswana. Akan, Pedi and Tswana are the hardest languages. The lowest word error rate on Akan is 47.7%, so nearly half the words are wrong even for the best model.
• English spoken with an African accent. Harder for almost every model than French or Afrikaans.
• Qwen3. Qwen3 is the weakest, with 84.9% of words wrong on average. On 6 languages it makes at least as many mistakes as there are words: Amharic 111.4%, Kinyarwanda 100.0%, Shona 103.5%, Tswana 146.5%, Xhosa 115.4% and Zulu 122.9%.

Higher is better on all three. Each number is that model’s average across the 16 languages in the tables below. Where a model was run on fewer, the number in brackets is how many.

• Azure Translate It does well where it is available, on Afrikaans, Swahili and Zulu.

BLEU. Higher is better.

chrF. Higher is better.

AfriCOMET. Higher is better; the scale runs from -1 to 1.

• Gemini-3-Flash. Gemini-3-Flash is the strongest, with the highest BLEU score in every one of the 16 languages. It is best on Afrikaans at 40.23 and weakest on Igbo at 8.14 and Akan at 8.73.
• GPT-4o Audio Preview. Its BLEU score is below 1 out of 100 on 7 of the 16 languages: Amharic (0.60), Hausa (0.52), Igbo (0.35), Kinyarwanda (0.77), Pedi (0.97), Sesotho (0.96) and Tswana (0.96).
• Gemma-4-E4B. Its AfriCOMET score is near zero on Akan (0.042), Amharic (0.059), Kinyarwanda (0.068), Pedi (0.059), Sesotho (0.022), Tswana (0.038) and Yoruba (0.040), and below zero on Igbo (-0.001).

Scores for the models are given on 9 of the 13. Every score runs from 1 to 5 and covers the 172 of the 385 questions whose details were recorded. On the first 6 a high score is better. On the last 3, Hallucination, Local Relevance and Harm, a low score is better.

• Deliberately poor answers Answers written to be bad on purpose, scored by the labellers alongside the real ones as a check on the scoring.

• Claude 4 Sonnet, DeepSeek-R1 and GPT-4.1. These are the strongest 3 models. Claude 4 Sonnet is the only model above 4.2 on Identifies Uncertainty, at 4.29.
• Llama-4-Maverick. Scores with the top models on Factuality at 4.66, but scores 1.55 on Hallucination, where the strongest 3 sit at 1.05 or below.
• Qwen-2.5-32B. Scores 3.01 on Hallucination, the worst of any model, above even the deliberately poor answers at 2.23.
• Identifies Uncertainty. For 10 of the 12 models this is the lowest of the 6 scores where a high score is good. Claude 4 Sonnet is the best of them at 4.29, against 4.75 on Factuality for the same model.
• Phi-4 Multimodal, Qwen-2.5-32B and Qwen2-Audio-7B. The 3 lowest-scoring models. Phi-4 Multimodal and Qwen-2.5-32B are below 3.7 on Factuality, and Qwen2-Audio-7B is at 2.31, the lowest of any model.

Dataset

The speech to text tables cover 17 of these languages and the translation tables 16.

Every recording is real speech from a working health worker. None of it was made by a computer.

The speech to text recordings come from Intron’s production monitoring, Afrispeech Dialogue and Med-Conv-Nig. The translation recordings come from AfriVox Translate. The spoken questions come from the CHEWs dataset, a set of questions asked by community health extension workers.

The people recorded for speech to text and translation come from 10 or more African countries. At least 40% are women. About 60% work in towns and cities against 40% in rural areas.

Everyone who asked a spoken question is from Nigeria. 211 of the 398 recordings are by men and 187 by women. 253 of the recordings are by people aged 26 to 40, 79 by people aged 19 to 25 and 66 by people aged 41 to 55. The accents heard most often are Yoruba (83), Hausa (67), Yoruba and Pidgin together (58), Igbo (50) and Igala (42).

Each person was given a written checklist saying what every score means, and went through training sessions before scoring started so that everyone scored the same way. Every translation was reviewed twice to confirm it meant the same as the original. Each spoken question was checked against its reference answer to confirm the two matched.

How the answers were scored

Privacy

Everyone recorded agreed that their recordings could be used for research. The personal details in this kind of speech are the speaker’s name, the name of the facility, anything that identifies a patient, and references to places. Intron’s own filter removes them automatically, then a native-speaking medical professional checks the result by hand. All of this runs inside Intron’s data handling process, which is aligned with the ISO 27001 information security standard.

Limitations

• Every spoken question came from Nigeria, which limits how far the spoken question results carry over to other countries.
• More of the speech to text and translation recordings come from English-speaking West and East Africa than from anywhere else.
• Everyone who asked a spoken question is a medical practitioner. Community health workers without formal medical training are not among them.
• The spoken questions cover Nigerian languages only. Kinyarwanda, Luganda, Swahili, Shona and Zulu are not in that part of the set.
• Everyone recorded is a user of Intron’s app, so they are more comfortable with technology than a typical community health worker.
• Speakers switch language mid-sentence, and that switching was written down inconsistently from one language to another.
• Word error rate and letter error rate count a spelling difference as a mistake, and in several of these languages the spelling is not settled.
• BLEU scores languages that build long words too harshly. chrF and AfriCOMET give a second reading.
• The human checking does not cover all 19 languages evenly.