A few months ago, a Syrian psychosocial-support organization we work with asked us to build something digital that could extend psychological first aid beyond the handful of counselors who could ever deliver it in person. We agreed and got started. Within weeks we hit a wall that had nothing to do with engineering: we couldn't tell whether the tool was detecting distress correctly, or whether it helped anyone at all. The instruments and models available to us have never been shown to measure distress validly in the language our users actually speak.
That wall is what this essay is about. The problem is not a shortage of therapists, funding, or apps, though those shortages are real too. It is something more basic and much harder to see. The instruments used to measure psychological distress in Syria (both the translated clinical questionnaires and the language models now being built on top of them) have never been validated against how Syrians actually express suffering. Measurement is a kind of infrastructure, and in Syrian Arabic that infrastructure is largely uncalibrated. A great deal of downstream work is resting on it anyway.
A specific problem
Take one concrete, high-stakes case: post-traumatic stress in displaced and conflict-affected Syrian adults. Prevalence figures get published, screening happens in clinics and camps, organizations run psychosocial programs and report their outcomes. Almost all of it depends on Arabic translations of instruments built and validated in English-speaking clinical populations — the PCL-5, the Harvard Trauma Questionnaire, the PHQ-9. Translating an instrument makes it readable in Arabic. It does not make it valid.
The gap is conceptual, not linguistic. In colloquial Arabic for example Syrian (Shami), distress tends to be expressed somatically and through metaphor rather than introspection: a tightness in the chest (ضاق صدري), a heavy heart, a burdened soul, suffering routed through shame and family obligation. An English-language item like "feeling down, depressed, or hopeless" does not map cleanly onto any of these. Someone can be suffering profoundly and still answer "no" to the literal question, or report bodily complaints that a symptom checklist quietly files under the wrong organ system.
This is old ground in cross-cultural psychiatry, and we want to say so plainly. Kleinman warned in 1977 that importing a Western diagnostic category into another culture and assuming it measures the same thing commits what he called a "category fallacy." The idioms-of-distress literature that followed showed how suffering gets encoded in culturally specific terms. The DSM-5 added a Cultural Formulation Interview for exactly this reason: translated criteria are known to travel badly. So the caution is well established. What is missing is a MENA-specific measurement pipeline that connects idioms of distress, psychometric validity, and computational representation, and turns that caution into something testable and usable by the people running services now. Nobody has built it. Until someone does, the prevalence figures, the screening decisions, and the reported intervention effects in this population all carry an error whose size no one has measured.
What is technically hard, not just missing
It would be easy to read this as a translation task, or a routine validation chore. It is neither. The failure is plausibly happening in three layers at once, and no one has established which layers are failing, or by how much. We propose treating it as three testable hypotheses, ordered from cheapest to hardest to answer. The psychometric layer is the scientific core; the representational layer is both the cheapest way to show that something is wrong and, in the end, part of the repair.
Layer 1 — Representation (testable immediately)
Modern Arabic NLP tools, including openly available models like AraBERT and CAMeLBERT, represent text as points in a high-dimensional vector space where proximity is supposed to track meaning. Our first hypothesis is that Shami somatic idioms of distress land in the wrong place. We expect ضاق صدري to sit near respiratory or cardiac vocabulary rather than near psychological distress, because these models were trained overwhelmingly on Modern Standard Arabic and formal text, where such idioms either do not appear or appear only in their literal sense. Prior research indicates that how close a model's training variant is to the target text matters more than how much training data it has. That predicts that even a dialect-aware model trained on ordinary social media would misplace a clinical register of distress it has never seen. If so, any classifier or chatbot built on these embeddings is partly blind to the signal it most needs to detect. This is a precise, falsifiable claim about geometry, and it can be tested on a laptop with public models and a curated list of idioms. No patients required.
Layer 2 — Psychometrics (the core R&D)
Set language models aside completely, and there is still an open empirical question: do translated instruments measure the same latent construct here that they measure in the population they were designed for? Psychometrics has a mature framework for this. Measurement invariance is tested as a hierarchy — configural, then metric, then scalar — using multi-group confirmatory factor or item-response models and differential item functioning (DIF) analysis. The hierarchy matters because each level that fails has its own concrete consequence.
If configural invariance fails, the items do not even cohere into the same construct for Arabe respondents.
If metric invariance fails, items contribute unequally, which distorts any correlation or regression built on the score.
If scalar invariance fails, which is the most common case and the most damaging, item thresholds have shifted, so the scores cannot be compared against a normative cutoff or tracked over time. And comparing against a cutoff, or across time, is exactly what every prevalence survey and every pre/post evaluation quietly does.
A concrete example helps. The PHQ-9 item "feeling down, depressed, or hopeless" may show DIF if for example Syrian respondents route the same underlying depression through somatic channels (fatigue, appetite, bodily pain) leaving the explicitly affective item under-endorsed at any given true severity. The total score can still look plausible while measuring something subtly different from what the same number means in the original validation sample. Establishing this is hard. It needs a sample with a defensible clinical anchor, and the conclusion will not be that the instrument is useless; it almost certainly keeps some validity. The conclusion will be a quantified map of where the instrument bends and by how much. That map, rather than an overclaim, is the scientific target.
Layer 3 — Criterion validity (the hardest, and explicitly later)
Do instrument scores, or model outputs, actually agree with expert clinical judgment in this population? This is the gold standard, and the most resource-intensive layer, because it spends clinician time that is desperately scarce. We include it not because we can do it at the start, but because a serious account has to say where the road eventually leads.
The genuinely novel part is joining Layers 1 and 2: for instance, a Palestinian-grounded account of distress expression that is at once a psychometric object, with items whose behavior has been measured, and a computational object, with idioms whose positions in representation space have been measured. Cross-cultural psychiatry gives you the first instinct and NLP gives you the second, but almost no one has built the bridge for any Arabic dialect. The hard part is not writing better Arabic. It is building, for the first time, a representation of Arabic distress that a measurement model and a language model can both share.
How a small team would start
We are two people: a medical student (Bahaa) and an AI engineer (Anas). The only reason this problem is visible to us at all is that it sits right on the seam between our two skill sets. A clinician working alone sees the cultural mismatch but not the vector space; an engineer working alone sees the model failure but not why the idioms matter. That seam is our advantage, and it is why a small mixed team is the right unit to start with.
Mental-health language is about as sensitive as data gets, which is exactly why we work through an established organization instead of around it: it provides the consent framework and clinical oversight we could not responsibly improvise. We keep our claims at the level of idioms and items, never individuals, and we describe the partnership honestly — an early, willing collaboration, not yet a signed program. The first deliverable covers one disorder, one dialect, one population, because a small credible result is worth more than a large unbelievable one.
What success looks like
What this produces is not an app but measurement infrastructure: a validated inventory of Arabic idioms of post-traumatic distress, a quantified account of how the existing translated instruments behave in this population, and a small open benchmark others can test their own tools against. The support tool our partner organization first asked for, and tools like it across the region, could then be built on measurement that has been checked instead of assumed.
The point reaches well beyond PTSD and beyond Syria. Across the Arab world, mental-health policy, prevalence surveillance, and the evaluation of expensive humanitarian programs all rest on instruments that were imported and translated, and increasingly on language technology trained on a register that few people actually speak. A bent ruler gives a wrong reading every time, in a direction no one has mapped. Straightening one ruler, for one population, in a way that clinicians and machines can both use, is a small job that a lot of other work depends on. It is also the kind of slow, foundational work that has to happen before the louder promises about AI and mental health in the region can mean anything.