Scientists tested AI tools that help ERs decide who goes first, and one common label in a patient’s record pushed people about to get worse down the line
Emergency rooms are starting to test AI chatbots to help decide which patient should be seen first. Researchers at UT Southwestern Medical Center wanted to know whether those tools pick up the same blind spots as human staff. They took real emergency records from Stanford Medicine and Beth Israel Deaconess Medical Center, paired patients who looked equally sick on arrival but where only one later got worse, and asked the AI which to prioritize. Then they changed a single detail.
When the records contained only neutral clinical information, the AI models picked out the patient who would deteriorate well above chance. But adding one stigmatizing detail to that patient’s summary, such as a history of frequent emergency visits or psychiatric emergencies, pushed some of them down the queue.
The worst offender was the frequent-visitor label. Describing a patient as a heavy user of the emergency department caused harmful reprioritization in every model and dataset tested, flipping the AI’s choice away from the patient who later got worse in up to 9.4% of comparisons. A psychiatric history did something similar, in up to 9.5%.
“In this study, a single added detail sometimes shifted a model’s priority away from the patient who later deteriorated,” senior author Andrew Jamieson said in a statement.
Wording mattered too. The researchers compared neutral phrasing, such as an established patient with multiple chronic conditions, with stigmatizing phrasing of the same idea. The stigmatizing version did more damage in every case, by 1.4 to 7.1 percentage points.
That mirrors a known problem in human medicine. Patients labeled as frequent flyers or psychiatric cases are often taken less seriously by staff, and the researchers suggest the stigma built into clinical language carries over into the AI’s judgment. Being dismissed by the people meant to help is a common thread, as our report on women with autoimmune diseases showed.
The second finding is about what did not happen. Adding race, language or insurance status to a patient’s summary did not cause consistent harmful shifts. Earlier research had raised that worry. A 2024 study in The Lancet Digital Health found that GPT-4 produced racially and gender-biased answers when asked to diagnose and plan care for patient vignettes. The new study, which measured decisions against what actually happened to real patients, found the stigma problem in different places.
In total the team ran 221,556 comparisons across 18 kinds of added detail, using several AI models and records from two hospitals. The results varied a lot by model.
“One of the main takeaways from this study is that bias varies by model, so each AI tool should be tested against real patient outcomes before use,” said first author Philip Jarrett.
For patients, nothing here describes AI deciding who gets care today. The study was a controlled experiment, not a test of a live system. The researchers recommend that hospitals check any triage AI against past patients’ outcomes, and test whether details like psychiatric history or stigmatizing language change its recommendations, before letting it influence real care.
There are limits to what this shows. The models compared pairs of patients rather than ranking a full waiting room, and the records came from two academic hospitals in the United States. The results depended strongly on which AI model was used and how information was worded, so they may not apply to other systems.
Source
Jarrett P, et al. “Outcome-grounded effect of clinically stigmatizing information on large language model emergency triage prioritization.” npj Digital Medicine, 2026.
DOI: 10.1038/s41746-026-03270-5
Also cited
Zack T, Lehman E, Suzgun M, et al. “Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study.” The Lancet Digital Health, 2024.
DOI: 10.1016/S2589-7500(23)00225-X
