Global South Health AI Benchmarks
Auxiliary Nurse Midwives (ANMs) in Uttar Pradesh and Telangana type questions and get on-demand guidance on high-risk pregnancy conditions, clinical protocols and case management during antenatal care and follow-up. The answer goes straight to the nurse midwife, and the bot redirects her to a medical person when a question is out of scope or it is unsure.
What was judged
Is the response factually and medically correct, with no incorrect or misleading information?
YesNoDoes the response provide all the important information needed to answer the question appropriately?
YesNoIs the response clear, understandable, and appropriately phrased for the intended user, the nurse midwife?
YesNoThe team started with seven things to score and cut to three. The seven overlapped too much. The three that stayed cover the most ground and are the easiest to write a rule for.
A worked example
How much iron and calcium should a pregnant woman be given?
- गर्भवती महिला को आयरन और फोलिक एसिड (Iron and Folic Acid - IFA) टैबलेट्स पहले तिमाही (First Trimester) में एक बार रोज़ाना दी जाती हैं, यदि सहन हो।
- दूसरी या तीसरी तिमाही (Second or Third Trimester) में IFA टैबलेट्स दिन में दो बार दी जाती हैं जब तक कि Haemoglobin (Hb) स्तर 11 g/dL तक न पहुँच जाए।
- जब Haemoglobin (Hb) 11 g/dL हो जाए, तब 180 दिनों तक IFA टैबलेट्स एक बार रोज़ाना जारी रखें।
- कैल्शियम टैबलेट्स भी दी जाती हैं, लेकिन IFA और कैल्शियम टैबलेट्स एक साथ नहीं लेनी चाहिए।
- IFA टैबलेट्स भोजन के एक घंटे बाद और कैल्शियम टैबलेट्स दो घंटे बाद लें।
- साथ ही, संतुलित आहार और विटामिन C युक्त खाद्य पदार्थों का सेवन बढ़ाएं ताकि आयरन का अवशोषण अच्छा हो।
- In the first trimester, a pregnant woman is given iron and folic acid (IFA) tablets once a day, if she can take them.
- In the second or third trimester, IFA tablets are given twice a day until her haemoglobin (Hb) reaches 11 g/dL.
- Once her haemoglobin reaches 11 g/dL, keep giving IFA tablets once a day for 180 days.
- Calcium tablets are also given, but IFA and calcium tablets should not be taken together.
- Take IFA tablets one hour after a meal, and calcium tablets two hours after.
- Also eat a balanced diet and more foods with vitamin C, so the body takes up the iron better.
The response correctly states IFA once daily in 1st trimester, twice daily in 2nd/3rd trimester until Hb reaches 11 g/dL, then once daily for 180 days. It also correctly advises not to take IFA with calcium and to separate calcium later. It does not invent unsupported calcium dose details.
It includes the essential trimester-based IFA schedule and spacing from calcium with meal timing, but it misses the essential advice to avoid taking IFA with milk/tea/coffee.
The answer is organized in simple bullet points with clear timing and frequency, making it easy to understand.
Evaluation results
Model tested: gpt-4.1
- Completeness. The weakest of the three. Where it fails, the answer leaves out something the rubric says has to be there.
- Factual correctness. Where an answer fails, it is almost always because something is left out, not because something is stated wrongly.
- Clarity. The strongest of the three. In the few answers that fail, the wording is too formal or uses medical terms a nurse midwife would not use, in Hindi and in Telugu.
- The answers were usually correct and easy to read.
- Most often the answer left out something the rubric asked for. Usually that was the signs that mean the woman needs to see a doctor. The clinicians who checked did not count these as serious mistakes.
- The model was built to keep replies short and easy to read, which is the likely reason it leaves things out. That is the trade-off: the short answer is the one a nurse midwife will actually use in front of a patient, and the complete answer is what the clinical guidance asks for.
Agreement between labellers
2 clinicians labelled the same 100 pairs, 50 in each language. How often they gave the same answer:
| Dimension | Hindi | Telugu |
|---|---|---|
| Factual correctness | 100% | 90% |
| Completeness | 96% | 76% |
| Clarity | 84% | 80% |
The 2 clinicians agreed less often in Telugu than in Hindi on all three, and least of all on completeness in Telugu, at 76%.
Dataset
The 1,010 questions come from 6 common risk topics, which together account for more than 80% of what nurse midwives actually ask. Questions shorter than 10 characters were left out.
The clinicians did not label all 1,010 questions. They labelled a sample of 100 questions, half in Hindi and half in Telugu. Every pair of 2 clinicians scored each question/answer pair, adding comments when they found an answer wrong.
The team gave the labellers written guidelines before they started.
Building the LLM judge
2 clinicians read 100 question and answer pairs. They marked each answer acceptable or not, and wrote a comment whenever they marked one down.
Those judgements were turned into a checklist for each individual question. Asking whether an answer is complete only means something against what that particular question needed, so one general rule would not work. The judge model wrote the checklists itself, from the answers the clinicians had marked.
gpt-5.4 does the judging. It is a stronger model than gpt-4.1, which writes the answers. It reads the question, the answer and that question's checklist, and returns a yes or no on each of the three, with its reasoning.
The team read 50 questions and their checklists in each language by hand, and rewrote the instructions given to the judge.
After the full run they compared the judge against the clinicians again. Where the judge was stricter than the clinicians, they softened either the checklist or the judge.
The benchmark is harsh. It fails an answer for a small gap a clinician would have accepted, and it almost never passes an answer that is actually wrong.
Privacy
No personal details are in the dataset. No nurse midwife name, no phone number.
Limitations
Only typed questions were included. That may leave out questions from nurse midwives who are less comfortable typing and prefer to send voice notes.