By Angela Spivey, Duke School of Medicine. First published in Duke Magnify, February 11, 2026.
Most people have heard about “hallucinations,” when AI models make up facts. But Agrawal’s research highlights a less-obvious risk: answers that are technically correct but medically inappropriate because they lack context.
To study the problem, Agrawal and her team created HealthChat-11K, a dataset of 11,000 real-world health-related conversations (about 25,000 user messages) across 21 medical specialties. They analyzed these interactions using a clinician-developed framework and made the dataset available so other researchers can explore it too.
They found that the way patients ask questions looks nothing like the way these models were evaluated.
