Global South Health AI Benchmarks
Intelehealth
Ayu 2.0 is Intelehealth’s clinical decision support tool for last-mile telemedicine in low resource rural settings. It is being piloted in 30 villages of Peth and Surgana, in Nashik district, Maharashtra. A health worker sits with the patient and fills in a structured history form in Marathi, covering the complaint, the patient’s past illnesses and the vital signs. Ayu 2.0 reads that written history and returns a list of up to 5 diagnoses in order, the most likely one first, with its reasoning, and a suggested treatment plan of medicines, tests, advice and referral. It is built to assist the doctor and the health worker, not to replace them.
What was judged
The confirmed diagnosis is the single diagnosis one doctor settled on for a case, starting from what the treating doctor wrote and checked afterwards by a more senior doctor. The first 3 measures below, and Comprehensiveness and Completeness, compare what the model produced against it. The rest judge the model’s output on its own.
The share of cases in which the confirmed diagnosis is the first one the model names. Higher is better.
0% never first100% always firstThe share of cases in which the confirmed diagnosis appears anywhere in the 5 diagnoses the model names. Higher is better.
0% never in the 5100% always in the 5One score for where the confirmed diagnosis sits in the list. A case where it is first counts 1, second counts a half, third counts a third, and a case where it is missing counts 0. Higher is better.
0 never in the list1 always firstHow appropriate is the list of diagnoses the model wrote for this patient?
5 Very appropriate4 Appropriate3 Neutral2 Inappropriate0 Very inappropriateHow comprehensive is the list compared with the confirmed diagnosis?
5 The confirmed diagnosis is in the list0 Nothing in the list is related to itEvery medicine is scored on 10 questions, each carrying its own weight in the total: is there an indication for the drug (weight 3), is it effective for the condition (3), is the dosage correct (2), are the directions correct (2), are the directions practical (1), is there a clinically significant interaction with another drug (2), is there a clinically significant interaction with the patient’s condition (2), is there unnecessary duplication with another drug (1), is the duration acceptable (1), and is this the least expensive alternative compared with others of equal use (1). Each answer is 0 appropriate, 1 marginally appropriate or 2 inappropriate. This is the one measure here where a low score is the good one.
0 appropriate1 marginally appropriate2 inappropriateDoes the treatment plan include every medicine the patient needed?
5 Includes all4 Almost all3 About half2 Only one0 Not at allAre the investigations the plan asks for relevant and necessary?
Is the advice in the plan clinically relevant, clear and safe?
Is the decision to refer, or not to refer, necessary and appropriate for this patient?
The list of diagnoses is written as the DDx, so these 3 measures are named Appropriateness of AI generated DDx, Comprehensiveness of AI generated DDx and Completeness of Medication list.
A worked example
Female, 43
- Fever for 1 day, all day and every few hours, in the evening and at night, low severity. No epidemic in the community and no treatment tried.
- Pain in both legs for 2 months, in the hips, buttocks, thighs, knees and calves, and in all the joints. Dull aching with tingling numbness and sharp shooting and burning pain, worse on movement and on climbing stairs, and worse on rest, which activity relieves. Recent and constant overuse reported.
Joints not tender, not swollen, full range of movement and no pain on movement. No paleness, no yellowing, no rash and no swelling of the ankles. Back of the throat normal, nothing abnormal in the abdomen or the back, and no injury to the head.
Not pregnant, no known allergies, no alcohol, no smoking and no recent medicines.
Blood pressure 120 over 80, pulse 85, temperature 35.61 °C, breathing rate 20, oxygen in the blood 98%, weight 42 kg, height 155 cm.
- 1. Viral Fever, which is the confirmed diagnosis
- 2. Osteoarthritis
- 3. Upper Respiratory Tract Infection
- 4. Rheumatoid Arthritis
- 5. Scrub Typhus
Viral Fever, the diagnosis the doctor settled on, is the first name on the list.
It is also somewhere in the 5.
The confirmed diagnosis sits first, which is the highest this measure goes.
Very appropriate, the top of the 5 to 0 scale.
The confirmed diagnosis is in the list, the top of the 5 to 0 scale.
The case history the health worker typed in is scored too, not only the model’s output.
How much clinical detail the case carries.
- Cetirizine 10 mg, 1 tablet at bedtime, once daily for 3 days
- Paracetamol 500 mg, 1 tablet after food, three times daily for 3 days
- B-Complex capsule, 1 capsule after food, once daily for 7 days
- Oral Rehydration Salts, 1 sachet in 200 ml of boiled and cooled water, as needed for 5 days
- Ibuprofen 400 mg, 1 tablet after food, twice daily for 3 days
- Rest and keep her fluids up, drinking plenty of water, juice and broth.
- Be careful taking Paracetamol and Ibuprofen together, as both can affect the liver and the kidneys, and do not go above the recommended doses.
- Check her temperature regularly, and if the fever lasts beyond 3 days or gets worse, seek further medical advice.
Almost all, one below the top of the 5 to 0 scale.
The plan for this patient names no test.
For the 3 pieces of advice above.
The plan for this patient names no referral.
| Medicine | What the 10 questions flagged |
|---|---|
| Cetirizine 10 mg | Nothing. All 10 questions scored 0, appropriate. |
| Paracetamol 500 mg | Nothing. All 10 questions scored 0, appropriate. |
| B-Complex capsule | Possibly indicated rather than clearly indicated, questionable effectiveness, and moderately more expensive than a drug that works as well. |
| Oral Rehydration Salts | Possibly indicated rather than clearly indicated, questionable effectiveness, and a dose slightly outside the recommended range. |
| Ibuprofen 400 mg | Possibly indicated rather than clearly indicated, questionable effectiveness, and partial duplication of another drug in the plan. |
One real visit from the set, a woman of 43 seen for fever and for pain in her legs, knees and hips, with the scores given for it.
A case can be scored at the top on the diagnosis and still carry problems in the prescription, which is why the treatment plan is scored medicine by medicine rather than as a whole.
Evaluation results
More than 12 models were compared for the diagnosis list. Llama 4 Maverick was chosen for the first phase of the pilot.
- Top-1 accuracy. In 18% of cases the confirmed diagnosis is somewhere in the 5 but is not the one named first. That is a potential opportunity to improve ranking and calibration.
- Top-5 accuracy. In 5% of cases the confirmed diagnosis does not appear in the 5 at all.
Llama 4 Maverick was chosen for the first phase of the pilot because it was cost-effective and did well on the team’s own comparisons. MedGemma 27B is the current model and does better at naming the confirmed diagnosis first.
Dataset
The health worker fills in the history form in Marathi. Every field on that form has an English version, so the doctor reads the same history in English. Marathi phrases typed into the free-text boxes are captured from the patient’s notes in the record.
| Diagnosis | Cases | Share of the set |
|---|---|---|
| Acute Gastroenteritis | 174 | 17.40% |
| Upper Respiratory Tract Infection | 124 | 12.40% |
| Viral Fever | 121 | 12.10% |
| Acute Gastritis | 102 | 10.20% |
| Osteoarthritis | 97 | 9.70% |
These case histories cover 40 different diagnoses. The 5 above are the commonest.
669 of the patients were women and 331 were men. Ages run from 0 to 98, with a median of 32. 316 of the cases are children under 18.
Cases were picked to mirror the mix of illnesses the health workers actually see in these communities, so the proportions in the set are close to what is common there. None of the cases are made up: all of them are real visits.
- Cases where the doctor recorded more than one diagnosis were left out. Every case in the set resolves to a single confirmed diagnosis.
- Cases where a picture matters to the diagnosis, skin conditions for example, were left out, because the set is text only. A small number of tooth decay cases were kept.
- Dropping the cases that need a picture, without putting other cases in their place, leaves the diagnoses that remain with a slightly larger share of the set than they have in the community.
All the results on this page come from the first round. All 6 doctors in the second round hold an MBBS, the standard medical degree in India, with 8 to 40 years of experience in primary healthcare.
- One right answer per case does not hold up in a real clinic. 2 doctors looking at the same patient can write different diagnoses and both be right.
- A patient with a cough and sneezing might be written down by one doctor as an Upper Respiratory Tract Infection and by another as Acute Rhinitis. One is broader and one is more exact, and both lead to the same treatment.
- The second round is meant to strengthen the labels, and it does not pick one answer per case. The 6 doctors are split into 2 groups of 3, with 500 case histories each. Each group agrees a set of diagnoses that would all be reasonable for that patient, and a set of reasonable treatment plans, and the model is scored against those sets.
- Getting 3 doctors to agree on that set is slow. The team describes it as work that goes back and forth many times before they settle.
Building the LLM judge
Each case history goes to Ayu 2.0, which returns its 5 diagnoses in order with its reasoning. Comparing that list against the confirmed diagnosis gives Top-1 accuracy, Top-5 accuracy and the mean reciprocal rank, with no judge involved.
An LLM judge then reads each case and the list Ayu 2.0 produced, and scores where the confirmed diagnosis sits in the list and how good the reasoning behind it is. Version 2 of this judge was used for the runs on the model in the pilot, and version 3 is being built.
The treatment plan is scored by 11 judges: one for each of the 10 questions in the Medication Appropriateness Index, plus one that looks for a medicine the plan should have included and did not. Those scores exist for every case in the set.
Gemini 3 Flash was pointed at the labels themselves rather than at Ayu 2.0, to check whether each confirmed diagnosis fitted the case history it was attached to. It flagged 41 cases where the confirmed diagnosis looked wrong, 14 where the notes were unclear, and 16 where the patient’s presentation was unclear.
Scoring the whole set of 1,000 cases costs about $3 to $4.
The check on the labels found problems in about 7% of the cases: the diagnosis attached to the case looked wrong, or the notes or the patient’s presentation were unclear. That check sits behind the second round of labelling.
Privacy
Patient and visit identifiers are stripped out when the data is prepared, and links to any video or picture are left out, so a case history cannot be traced back to the person.
Limitations
- While working on the second round the team found that many of the case histories describe symptoms that point to a further diagnosis beyond the single confirmed one.
- Every case comes from Peth and Surgana, so the mix of illnesses may not match other places or other kinds of care.
- A patient may not be able to describe their symptoms properly.
- A health worker may make a mistake typing in what the patient described.
- Common illnesses dominate the set, so there are few examples of rare ones.
- The 4 babies under a year old cannot be reliably separated into newborns and older babies, because age is recorded only in whole years. The team plans to record age more finely next time.
- The first round relies primarily on 2 doctors for the confirmed diagnosis, and the results should be read with that in mind.
- The benchmark does not measure what the doctor does with the answer, whether the treatment worked for the patient, or the effect on the health system.