Global South Health AI BenchmarkseHealth Africa
Voices of Care measures the two things a Hausa voice tool has to get right before anyone can build one: turning what a person says into written Hausa, and working out what that person is asking for, so the request can be sent to the right next step. This is a benchmark, not a product people use, so nothing here reaches a patient. In the use it is built for, models like these would sit inside voice tools for community health workers in northern Nigeria, on their phones and over voice phone lines, taking down symptoms, helping them work through triage, and writing up notes. A community health worker or a clinician still makes every decision.
What was judged
Nothing here is scored by a model. Every number comes from comparing what the model produced against the sentence the speaker read, or against the group a person put that sentence in.
Does the model write down exactly what the speaker said, in Hausa, keeping the hooked letters and the marks that change what a word means? The model is scored against the written sentence the speaker was reading.
What is the speaker trying to do: report a symptom, ask for information, disclose something in their history, raise a worry about a medicine? The answer decides whether the request needs triage, counselling, an escalation or a follow-up appointment.
Does the model do as well for women as for men, for each of the 4 dialects, and for older speakers as for younger ones? This has to be broken out and published rather than reported as one average.
- Word Error Rate: out of every 100 words in the sentence the speaker read, how many the model gets wrong. Lower is better.
- Character Error Rate: out of every 100 letters, how many are wrong. In Hausa a single letter can change the meaning of a word, so this is scored on its own.
- Accuracy: out of every 100 sentences, how many the model put in the right one of the 11 groups.
- Macro-F1: the same test scored again so that a rare group counts for as much as a common one.
The 11 groups a sentence can fall into:
What a wrong letter does: a tool that mishears zazzaɓin cizon sauro, malaria fever, cannot triage it. The text is put through a fixed clean-up step before scoring.
A worked example
The benchmark tests 2 things: turning a recording into Hausa text, and putting a sentence into 1 of the 11 groups.
tari na ya ƙaru tun da hayaƙi ya fara a kewayen gidana
tari na ya karu tunda hayaki ya fara kewaye gidana
That counts as 4 word errors. The hooked letters are gone, so ƙaru is written karu and hayaƙi is written hayaki; tun da is run together as tunda; and a kewayen has become kewaye.
ina jin ƙarancin numfashi… belongs in Symptom reporting.
A sentence discussing a treatment plan, labelled Prevention and counselling. In one check of which groups get confused with each other, 10 of the 25 sentences that were really about treatment landed there.
Evaluation results
Measured on the 5,375 recordings held back from training. Each number is the average of 5 training runs, and the number after ± says how far the runs spread around that average, in percentage points (pp). The last row is a single run.
| Model | Word Error Rate | Character Error Rate |
|---|---|---|
| Whisper Large-v3 | 15.14% ± 0.20pp | 3.66% ± 0.05pp |
| XLSR-53 | 15.59% ± 0.12pp | 3.60% ± 0.04pp |
| MMS-1B-all | 22.81% ± 0.95pp | 5.43% ± 0.33pp |
| MMS-1B-fl102 | 30.74% | 7.12% |
- The top two cannot be separated. Whisper Large-v3 and XLSR-53 are 0.51 percentage points apart, with a range running from −0.19 to +1.26. The ranking should not be over-read.
- Where the mistakes are. Under 4 letters wrong in every 100 means more than 96 come out right. The mistakes are in where one word ends and the next begins, and in the small parts of words: the marks on letters and doubled letters. They are not in hearing the sounds wrong.
Measured on the 749 sentences held back from training. Afro-XLMR-large’s numbers are the average of 4 training runs, mBERT’s of 5.
| Model | Accuracy | Macro-F1 |
|---|---|---|
| Afro-XLMR-large | 87.34% ± 1.11pp | 68.98% ± 1.83pp |
| mBERT | 81.38% ± 1.17pp | 60.71% ± 1.22pp |
The gap between those two columns is the central finding about the groups. Groups with fewer than about 250 sentences to train on are recognised much less reliably than the larger ones. The way to close the gap is collecting more sentences for the small groups, not changing the model.
3 groups are not yet reliably recognised:
The rare requests are the ones the model gets wrong, and they are the ones a routing tool exists to catch. The common things a person says, asking for information or describing a symptom, are recognised well. Any deployment has to send the 3 unreliable groups to a person rather than act on them.
Measured on a different set from the table above: 4,837 test sentences rather than 749.
With the other speech models in front of it, the loss is 5.71 percentage points for XLSR-53, 7.09 for MMS-1B-all and 16.06 for MMS-1B-fl102. The loss grows faster than the transcription errors do: MMS-1B-fl102 gets 30.74% of words wrong against 22.81% for MMS-1B-all, but costs 16.06 points against 7.09.
| Speaker group | Word Error Rate | Likely range |
|---|---|---|
| Female speakers | 13.3% (6) | 12.4–14.2 |
| Male speakers | 18.6% (3) | 14.3–24.7 |
| Kananci, the Kano dialect | 16.9% | — |
| Katsinanci, the Katsina dialect | 14.8% | — |
| Speakers aged 15 to 29 | 14.1% | — |
| Speakers aged 45 and over | 17.8% | — |
Every number in this table is Whisper Large-v3, the best model. The number in brackets is how many speakers are in that group. The likely range is where the true number probably sits given so few speakers; there is one for the gender rows. Any group with fewer than 30 recordings or fewer than 3 speakers is published as a range only and is left out of any ranking. Sakkwatanci and Zazzaganci speakers are 2 of the groups below that floor.
The error rate is not the same for every speaker. The ranges are published rather than smoothed over, and these results come from the 9 speakers who were tested.
Dataset
15,000 sentences were written to read like conversations between community health workers and patients: symptom reports, questions asking for information, counselling, and referral and treatment discussions, across 10 health topics. Clinicians wrote them with help from AI, and duplicates were removed. Native Hausa speakers then read them aloud, which came to about 50 hours of audio.
| Group a sentence can fall into | Recordings |
|---|---|
| Information seeking | 7,313 |
| Clinical assessment | 6,978 |
| Prevention and counselling | 6,435 |
| Symptom reporting | 4,492 |
| Risk / history / context disclosure | 2,046 |
| Treatment | 914 |
| Follow-up / status update | 540 |
| Vaccine recommendation / administration | 462 |
| Referral | 364 |
| Diagnosis / results communication | 356 |
| Medication concern | 100 |
About 50 native Hausa speakers, 25 men and 25 women, each made about 600 recordings, roughly an hour of audio each. Every sentence was read by 2 speakers of different genders, and no sentence was read twice by the same speaker: 14,928 of the 14,966 sentences carry exactly that pairing. Men and women recorded 15,000 each. Speakers span 3 age bands, 15 to 29, 30 to 45, and 45 and over.
Speakers by area:
The dialect of each: Kananci in Kano, Zazzaganci in Zaria, Katsinanci in Katsina, Sakkwatanci in Sokoto.
Speakers were recruited across northern Nigeria and chosen to cover the 4 main Hausa dialect areas, with even numbers of men and women, the 3 age bands, and a mix of secondary and tertiary education. The benchmark covers adult native Hausa speakers in northern Nigeria talking about community health.
The recordings were split so that no speaker and no sentence appears on both sides: 16,717 to train on, 1,855 to tune on, and 5,375 held back for the test, read by 9 speakers, 6 women and 3 men. Enforcing that cost 5,913 recordings, which are published rather than absorbed. The sentences for the group task were split so that no sentence appears on both sides: 13,469 to train on, 748 to tune on, and 749 held back.
Clinicians checked the sentences were medically appropriate across the 10 health topics. Native-Hausa reviewers with at least 5 years of community health experience checked the Hausa. 10 trained native-Hausa reviewers nominated from the EHA Group checked the written sentence against every recording. Trained Hausa-speaking labellers with community health backgrounds gave each sentence its group.
Privacy
The sentences hold no personal details of any kind: no names, no dates, no places, no clinic names, no record numbers. Because every sentence was written in advance and then read aloud, no real person’s health, history or condition appears anywhere in the recordings.
Everyone who recorded gave consent in Hausa, was told the recordings would be released for health AI research, could withdraw at any time, and was paid. Nobody under 18 took part. The study was cleared by the Kano State Ministry of Health research ethics committee on 13 May 2026. The dataset comes with use conditions: no working out who a speaker is, no voice cloning, no biometric or surveillance use.
The release is pseudonymous, not anonymous: speakers appear only under codes, but a copy of the source data held by the technical partner, Data Science Nigeria / EqualyzAI, included speaker identifiers.
Limitations
- People read sentences aloud rather than talking to a health worker. The scores show the best these models could do on real conversation.
- Every headline number comes from 9 speakers, and 3 speaker groups fall below the level at which a single number is published.
- The recordings were made in conditions set up to resemble the field, not on the phones and networks community health workers actually use.
- Some groups have almost no test sentences. Medication concern has 1, so it cannot yet be tested in any meaningful way.
- Speed was measured on a server, not on the kind of device a tool would run on.
- The recordings have far more Kano speakers than Sokoto speakers.
- About 50 people recorded, which is a modest number. This is offset by about an hour of audio from each person, an exact balance of men and women, and coverage of 4 dialect areas.
- The benchmark does not judge whether any advice is medically right, whether a system avoids harm, or whether it knows when to escalate. It measures only how well a model hears what is said and works out what is being asked.