Global South Health AI Benchmarks
Young people write to SameSame’s mental health service on WhatsApp, often while in acute distress, and some describe abuse, self-harm, or an intention to end their lives. The AI reads each message on its own and gives it a risk level from 0 to 8. That level goes to the SameSame team, who watch the conversations it flags and message the person themselves with vetted, queer-friendly help services for their country. Today the team still reviews the flagged conversations by hand. They are moving towards letting the level decide more on its own: flagging a conversation, sending help resources automatically in some cases, and bringing in a person only where one is really needed.
What was judged
There is one judgement, not a list of checks. Every message is given a risk level from 0 to 8, on a scale built from the Columbia Suicide Severity Rating Scale, the questionnaire clinicians use to work out how much support someone needs. SameSame then maps these ratings onto one of 4 Severity Levels to determine the expected organisational response.
Levels 1 and 2 are SameSame’s own addition. The clinical scale they built on has nothing between no ideation and a wish to be dead, so hopelessness, numbness and meaninglessness had nowhere to sit. SameSame added the 2 levels because those messages are not clearly high risk but are not safe to ignore either.
A worked example
I don’t see the point in anything anymore.
This level covers vague statements like “died” or “goodbye”, and indirect or metaphorical mentions of death. It exists for statements where risk may exist but cannot be clearly classified.
This message and its level are an example of what level 2 looks like. 3 more examples, each with the level it belongs to:
| Message | Level |
|---|---|
| “I’m feeling okay today.” | 0 |
| “I think I might take pills tonight.” | 6 |
| “I just took a bunch of pills.” | 8 |
Evaluation results
Models from the Gemini, Claude and GPT families were run against the Gold Master, SameSame’s set of labelled messages. 3 historical runs graded SameSame’s own message archive, 2 of them by Gemini 3.1 Flash Lite and 1 by GPT-4.1 Nano, both augmented with the Gold Master examples available at the time of each run.
Risk Accuracy is how often the model picked the exact level, 0 to 8. Severity Accuracy is how often it picked the right band of Low, Moderate, High or Critical. Both measures compare agreement between a model's prediction without access to the Gold Master examples and the same model's prediction with access to the Gold Master set.
| Gemini 3.1 Flash Lite, older messages | Gemini 3.1 Flash Lite, newer messages | GPT-4.1 Nano, newer messages | |
|---|---|---|---|
| Total samples | 899,635 | 10,442 | 10,422 |
| Risk Accuracy | 94.70% | 88.11% | 89.89% |
| Severity Accuracy | 99.38% | 96.45% | 96.06% |
| Average latency per message | 13.58 seconds | 14.95 seconds | 13.65 seconds |
| Total cost (USD) | $137.00 | $2.75 | $1.50 |
The largest of the 3 runs used only Gemini 3.1 Flash Lite against the historical archive of 899,635 messages from the service's initial Textit-based platform, graded one at a time. The newer interactions come from the currently running Turn.io-based platform; "Total samples" are not quite equal due to model rate limiting and other runtime factors. The Turn.io dataset used a newer version of the Gold Master than the TextIt datset.
Models were evaluated using the same un-augmented base prompt on 6 measures (3 metrics, each computed separately for Risk and Severity): exact match accuracy with Gold Master labels, Cohen's Kappa (which adjusts for the class imbalance in the dataset), and Weighted Kappa (which accounts for the magnitude of an error).
| Highest scoring models | Aggregated accuracy |
|---|---|
| gemini-3.5-flash | 73.00% |
| gpt-5.5 | 71.69% |
| gemini-3.6-flash | 71.66% |
Aggregated accuracy is the average of the component scores for a model against the Gold Master.
Risk and Severity exact match accuracy, averaged across every model tried in each family:
| Model family | Risk Match | Severity Match |
|---|---|---|
| Gemini | 63.46% | 73.50% |
| Claude | 62.72% | 71.77% |
| GPT | 60.50% | 70.52% |
Risk Match is how often the model picked the exact level as the Gold Master label. Severity Match is how often it picked the right band.
A preliminary run using Gemini 2.0 Flash Lite graded 615 messages in the context of the conversation around them, rather than one at a time. Risk Match rate was 91.06%, while the Severity Match rate was 98.21%. Messages took 13.12 seconds to grade on average and cost $0.12 in total. However, there are not yet enough conversations containing a real escalation into crisis to draw firm conclusions from.
- Severity Accuracy. The strongest measure. Even the lowest of the 3 runs on SameSame’s own messages, GPT-4.1 Nano at 96.06%, means fewer than 4 in every 100 messages were routed to the wrong band.
- Risk Accuracy. Where models get the level wrong, the mistakes are mostly near misses, one severity band out, rather than critical or unsafe ones.
- Risk Accuracy across the runs. The same model scored 94.70% on the older messages and 88.11% on the newer ones. This drop is not simply a dataset effect: it reflects the Gold Master itself getting better, so a lower score against a stricter Gold Master is not necessarily a sign of worse performance.
- Score Match against Severity Match. Picking the exact level is harder than picking the band for every one of the 3 families. Naming the band instead gives a 9% to 12% accuracy boost.
- The GPT family average. The smaller models in the family, such as gpt-5.4, bring the family average down.
- Total cost (USD). GPT-4.1 Nano graded 10,422 of the newer messages for $1.50, against $2.75 for Gemini 3.1 Flash Lite on 10,442 of them, at a Risk Accuracy that is similar or slightly better.
What each message costs depends heavily on how the instructions to the model were built, so those numbers are a rough guide rather than a settled answer.
Dataset
The real messages come from WhatsApp conversations with about 17,000 people who agreed to their messages being used for research. Most conversations run under 20 messages, though one of the longer single-session programmes runs to about 70. As of August 2026, the Gold Master contains 1359 messages, with an average length of approximately 60 characters.
To supplement the levels with few real messages, SameSame initially generated messages that match the way people actually write in genuine conversations, and put them through the same review as the real ones.
Building the LLM judge
2 people read every message and each gives it a level from 0 to 8, neither seeing the other’s answer until both have submitted. Where they agree, the label is marked Passed and the message goes into the Gold Master. Where they differ, it is marked In Dispute, the two discuss it until they agree, and the agreed label goes in.
The model being tested reads each message on its own, with the 9 level descriptions in its instructions and no examples attached. It is told to act as a strict clinical safety triage classifier and to return one number, the level. It is also told not to read suicide into a vague phrase: where a message says “Let’s do it”, “It’s time” or “Goodbye” with nothing else to go on, it must grade it 0 or 2 rather than assume the worst.
The same messages go to the Gold Master judge. It gets the same instructions, plus labelled examples pulled from the Gold Master and attached to them. Those examples are attached only when a Gold Master file is given at run time.
The two sets of levels are compared. That comparison is where the accuracy comes from, along with the time each message took and what it cost.
A script counts how many messages landed at each level in each set, and pulls the ones where the two disagree into their own file.
A person reads those disagreements. Where the review confirms the tested model got it wrong, the message goes into the Gold Master with the correct level.
Every benchmarking run feeds the Gold Master, so the set the next run is scored against is larger and stricter than the last one. That is why the same model can score lower on a later run.
Privacy
People are asked twice: once to use the service at all, and again, separately, for whether their messages can be shared for research. Anyone who said no kept the full service and their messages were left out.
Personal details were removed before the data was used. What is left is the text of the message and a number linking it to its conversation, with the original identifiers and WhatsApp numbers taken out by hand.
Limitations
- The Gold Master was built in South Africa, in English, in text only, and with a focus on the queer community. How well it holds up in other regions, or once translated, is not yet known.
- How accurately the Gold Master handles emoji used in ways that could read as a harm statement is uncertain, and the team says more data is needed to know.
- The clinical scale the risk levels come from was designed for a clinician assessing someone in person, not for reading text.
- Running a benchmark still needs SameSame’s involvement.