Global South Health AI BenchmarksGlobal South Health AI Benchmarks
Countries
Use cases
Languages
Benchmarks
Health helplines that patients chat with
Myna MahilaIndiaProduction
Women in low-income neighbourhoods ask about sexual and reproductive health by text or voice, and the chatbot answers them directly.
7 raters: 3 doctors, 3 social workers, 1 end user
Jacaranda HealthKenyaProduction
Mothers text a maternal health helpline through pregnancy and after birth. The reply goes back by SMS once an automated check clears it, and danger signs go to human agents.
4 internal staff at Jacaranda Health
SameSameSouth Africa, ZimbabweProduction
Queer young people reach a mental health service on WhatsApp. The AI grades self-harm risk on a 0 to 8 scale for the team and its routing rules, not for the user.
2 raters per message
Tools for frontline health workers
ARMMANIndiaProduction
Nurse midwives ask for clinical guidance on high-risk pregnancy mid-visit, in Hindi and Telugu mixed with English, and the answer goes straight to them.
4 clinicians: 2 gynaecologists, 1 doctor and 1 nurse trainer
Intron Health6 countriesPre-deployment
Clinicians and community health workers work in their own languages, by voice. The AI turns clinical speech into text, translates it, and answers spoken questions.
Medically trained native-speaking annotators and community health worker panelists
eHealth AfricaNigeriaPre-deployment
Community health workers speak in Hausa. The AI turns that speech into text and routes it by what the worker is asking for. This is a benchmark, with no end user yet.
10 reviewers, plus field staff
Tools for doctor consultations
Pinky PromiseIndiaProduction
Women consult a gynaecologist by chat at an online clinic. The AI drafts the reply to a follow-up question, and the doctor edits it and sends it.
2 doctors
IntelehealthIndiaPilot
Doctors treat rural patients by telemedicine. From the case history the AI ranks the likely diagnoses and suggests a treatment plan.
2 doctors in the first round, 6 in the second
Tools for after a clinic visit
Penda HealthKenyaPre-deployment
Patients leaving an outpatient clinic are sent their medicines, results and vitals to follow at home, written by the AI from the visit record in English and Swahili.
21 clinical officers and pharmaceutical technologists
Findings
01
LLM judges need to be aligned with human judgement
+Evaluating AI responses manually does not scale. LLM judges use another LLM to do the evaluation instead and enable automated evaluations. However, they need to be aligned with human experts before they can be trusted and that task is not straightforward.
What worked
Building an LLM judge is not a one-off step. It requires constant alignment with domain experts by finding disagreements and fixing them iteratively.
How judges fail
A judge is not wrong at random. It is either too soft or too harsh on everything. It agrees with clinicians on objective criteria: was the answer safe? did it answer the question? It disagrees on subjective ones: Is this an emergency situation? Was the message rude? Is the tone appropriate?
Model choice
The LLM judge can be run offline without interrupting the user experience. So it can be a bigger, slower, costlier model than the one it is checking, or a cheap one, without changing what the service costs to run. Pick it on whether it agrees with your clinicians, not on price. Testing it against your own human labels is the only way to know.
Where it showed up
Jacaranda HealthOut of 9 LLM judges, all of them had a low agreement score with their experts. The most lenient judge passed 125 answers the clinicians had called unsafe.
Myna MahilaTheir LLM judge looked fine on a 40-question trial. They then ran it on their full dataset. It disagreed with the clinicians on the 2 most important things: whether to send the woman to a doctor, and whether the answer was disrespectful about her sex, caste or marital status. A small trial does not prove a judge works.
Pinky PromiseTheir LLM judge agreed with the doctors 97% of the time on whether the reply acknowledged the patient's worry, and 52% of the time on whether it used everyday words instead of jargon. It tracks the clear clinical calls and drifts on tone.
ARMMANThey changed their LLM judge in 2 rounds. In the second, it failed answers the clinicians thought were fine, so they made it less strict. Alignment took more than 1 round.02
Experts disagree with each other, and aligning them is not trivial
+Give 2 qualified clinicians the same answer and they often score it differently. They agree when the question has an objective right answer, like whether a dose is correct. They disagree when the right answer is subjective, like whether the answer was complete.
Why it matters
It reflects internal inconsistency in protocols or confusing guidelines for labelling that need to be resolved first to produce trustworthy labels. Often, it can result in higher quality labels. But it takes time and will likely be costly. It is also a precursor to building aligned LLM judges for automated evaluations too.
What to do
Do a dry-run on a smaller set to identify issues early. Have multiple experts review it and measure how often they agree. Aim for 70-80% agreement. If they don't meet the bar, make the labelling guidelines clearer and set up a disagreement resolution process with them. Review a larger set only after they agree on the smaller one. Continue to measure their agreement on a small subset periodically.
Across profiles
Your product might have different stakeholders: doctors, nurses, social workers and end users. Each group might disagree with another as well. A user may see nothing concerning in a scenario that a doctor flags as emergency. Check agreement across these groups, not only within one group.
Where it showed up
ARMMAN2 annotators scored the same 100 answers. On answer completeness, they agreed 96% of the time in Hindi but only 76% in Telugu. On factual correctness, the agreement was 100% and 90% for Hindi and Telugu respectively.Myna MahilaAgreement was measured within fixed pairs across doctors and social workers. Both doctors and social workers did not agree on answer completeness: doctors 59% to 80%, social workers 42% to 65%. However, both agreed easily on whether an answer was safe.Pinky Promise2 doctors scored the same 49 cases. They agreed on safety 98% of the time, on relevance 88%, on whether the answer was clinically right 73%, and on whether it was complete 65%.
IntelehealthAn LLM-based check flagged mistakes in their labels. They stopped keeping 1 right answer per case and pivoted to using 6 doctors, working in groups of three, to produce higher quality labels.Jacaranda Health4 of their staff scored the same 514 answers. They agreed on response safety only 38% of the time.Intron Health47 people gave 14,739 ratings. They matched exactly 47% of the time on whether advice suited a rural clinic, and 95% on whether it could cause harm.03
A single quality score confuses both humans and LLM judges
+Many teams started by asking reviewers to rate response quality between 1-5. They found it hard to agree on when the quality was 2 vs 3 vs 4. The same confusion carried over to LLM judges too. By the end, teams shifted towards using binary checks for evaluating each independent dimension of quality instead of a single quality score.
The change
Instead of asking reviewers to answer a broad question ("is this response good?"), divide it into narrow Yes/No questions, each covering different dimensions of quality: Is the response medically accurate? Is it relevant to the query?
More does not mean better
It can be tempting to create 10-20 metrics to measure everything you can. But beyond 5-6 metrics, they start overlapping and hamper decision-making. Reviewers agreed least where the metrics were vague.
Record the reason
Whenever possible, ask reviewers to note the reasoning for their scores. It helps clarify their thinking process, informs future iteration of labelling design, along with sources of confusion, and helps align LLM judges with the experts too.
Where it showed up
Jacaranda HealthEvery 5 is built by adding up named yes-or-no checks: 3 for safety, 5 for cultural fit, 8 for tone. 4 or more is a pass, so anyone can see what the 4 is made of.ARMMANThey started with 7 things to score and cut to 3 yes-or-no questions, because the 7 overlapped: is it right, is it complete, is it clear.Pinky Promise7 yes-or-no questions, plus a label for how much the doctor changed the draft. A half-right answer scores zero, not half.Myna MahilaThey tried scoring out of 5 and found it too vague. They tried yes or no and found most answers sat in between. They settled on 0, 1, 2.04
LLM judges need to be benchmarked too
+A larger model does not always mean a significantly better LLM judge. A smaller model might be good enough. Instead, compare how closely different models are aligned with your human experts. Use the agreement numbers and cost to choose the right model.
Bigger does not mean better
Myna Mahila tried 4 models as judges. The one that agreed with their doctors and social workers most often, on a small sample, was the smallest model: Gemma 4.
Cheaper models could be better
SameSame found that GPT-4.1 Nano, costing 14 cents per 1000 messages, evaluated the risk level more accurately than Gemini 3.1 Flash Lite, which costs 26 cents per 1000 messages instead.
Costlier models for hard cases
The most expensive judge model that Jacaranda tested agreed with their experts only a little more than a model 30 times cheaper: 0.40 vs 0.34. So, they use the cheaper model to judge every response in production and send only the ones where it is unsure to the expensive one.
Where it showed up
Myna MahilaThey used 4 models to judge the same 40 answers and compared their judgements with the labels given by their doctors and social workers. Gemma 4, an open model of 25 billion parameters, agreed 63.7% of the time. DeepSeek v4 Flash, nearly 11 times bigger, achieved 56.1% instead. Claude Haiku 4.5 and Gemini 3.1 Flash Lite, both commercial models, managed only 44.4% and 47.6%.
SameSameOn a dataset of ~10,000 messages, GPT-4.1 Nano, at 14 cents per 1,000 messages, predicted the right risk level 89.89% of the time. Gemini 3.1 Flash Lite, at 26 cents per 1000 messages, got 88.11%.Jacaranda HealthThey compared 9 models for their LLM judge, ranging from 20 cents to 27 dollars for evaluating 1,000 responses, against their human labels. Claude Opus 4.7, the most expensive, was the best with an agreement of ~40%. Gemini 3.1 Flash-Lite, at 89 cents, nearly 30 times cheaper, got 34%. Their plan now is to evaluate every message using Flash-Lite and send only the unclear ones to Opus.05
Performance on low-resource languages is worse
+01Consistent across languages and use cases
Intron: 5.5% of words wrong in French, 47.7% in Akan. Myna: accuracy 98% in Hindi, 85% in Minglish (Marathi typed in English letters). Jacaranda: LLM judges rated Swahili responses worse than English by 3-21 percentage points. Penda: Medicine instructions were clearly understandable 94.6% of the time in English and 87.2% in Kiswahili.
02What got worse was different for each team
Myna: accuracy and completeness. Jacaranda: clinical safety. Penda: how natural it sounds. Intron: audio transcription. Measuring different dimensions quality across each language surfaces gaps not visible otherwise.
03Agreement between experts drops too
Myna's social workers agreed on completeness 65% of the time in Hindi, 52% in Minglish. ARMMAN's labellers agreed 96% in Hindi, 76% in Telugu.
04Transcription quality suffers due to spelling
Many African languages have no single way to spell a word. When the model and the expert spell it in different ways, it counts as a mistake, even though both are right. Intron and eHealth Africa both say this.
05Closed models had a wider gap
Jacaranda: every closed model was harsher on Swahili; the 2 open Gemma models showed lesser gap. Intron: their own model beat every other model on 12 of 17 languages.
06Averages hide the language variation
Myna's overall accuracy is 94%. But, that breaks into 98% in Hindi and 85% in Minglish. Intron's own model: 24% on average, 54% on Akan. Jacaranda's overall pass rate hides a 21-point Swahili-English gap. Report performance on every language separately.
07Evaluate the impact of code-mixing
Myna's performance on Hinglish and Minglish queries was worse than on Hindi and Marathi queries. But, Jacaranda's responses where Swahili and English were mixed scored higher than Swahili-only responses. Measure how code-mixing impacts your system.
08Measure if the gap is real
When the samples were large, the gaps were significant: Myna's 13 points on 300 questions per language, Jacaranda's 21 points on 175 responses. But for cases with a few samples, a 2 or 3 point gap cannot be told apart from statistical noise.
06
LLM judges carry cultural biases about a "good answer"
+01They marked local norms as wrong
Myna: the judge failed an answer on bias for calling the woman Didi (meaning, elder sister). Jacaranda: 8 of 9 judges scored cultural fit lower than the Kenyan clinicians did on the same replies. Pinky Promise: the judge passed only 55% of drafts as everyday language which the experts called too harsh.
02The same judge contradicts itself
For Myna, on one answer, the judge failed Didi under bias as "an unwarranted assumption about the user's social relationship" and praised Didi under empathy as "culturally resonant" and good for rapport.
03The judges were stricter than experts
Myna: doctors, social workers and the community member scored bias at 0-2%; the judge scored it 54%. Jacaranda: clinicians scored cultural fit 4.54 / 5 but 8 of 9 judges scored lower.
04Judges penalised response style, not facts
Myna's judge passed 85% of answers on accuracy but only 46% on bias, and its bias failures were about how the woman was addressed. Pinky Promise's judge passed 98% on safety but only 55% on colloquial, that is, on whether the wording sounded like everyday speech. Measure LLM Judge-human agreement for each dimension as one overall agreement number can be misleading.
05Public benchmarks carry the same bias
Myna tested the LLM judges released in OpenAI's HealthBench on their chatbot. It gave low scores to answers that were medically correct and locally appropriate: for talking about breastfeeding in public, for suggesting fish during pregnancy, for not telling the woman to go through insurance. This necessitated building their own LLM judges.
06Write the local standard into the checklist
Jacaranda anchored empathy on "pole kwa hilo" and greetings on "Habari mama", so a judge scores against local phrasing rather than its own idea of politeness. Pinky Promise and Myna are rewriting their judge instructions to ensure cultural alignment with their experts. Do not trust a judge's assessment of cultural appropriateness until they match your own experts on your own dataset.
07
Decide the questions first, then build the dataset
+A benchmark score only tells you how well the model did on the samples included in the dataset, which dictate the questions you can answer using them. Instead of building a dataset first and then, asking the questions, ask what you want to know first and then choose the samples that can answer them. Otherwise, you can only answer whatever your dataset happens to contain.
01Different question, different dataset
Myna wanted to know how well the chatbot handles emergencies. So 1 in 3 of their questions is high risk, far more than the service really gets. Jacaranda wanted to know how does the reply service do on a normal day. So they took 528 questions at random and kept the real production mix. Two opposite choices, both right for answering their respective questions.
02You can only compare on what you sampled for
Jacaranda wanted to compare Swahili with English, so they made the sample exactly even across the 3 languages. Penda wanted a result for every clinic, so they sampled at least 30 visits from each. Pinky Promise did not want one patient to be overrepresented, so they kept 1 answer per patient.
03State the limitations of the dataset upfront
SameSame wanted to test performance for all the risk levels, but 17,000 users had no messages for a specific high-risk level. So, they created fake data mimicking expected user queries for that level. Intelehealth removed cases that need an image, and clearly state this makes their dataset mix different from what their clinics actually see.