AI Health
Friday Roundup
The AI Health Friday Roundup highlights the week’s news and publications related to artificial intelligence, data science, public health, and clinical research.
January 9, 2025
In this week’s Duke AI Health Friday Roundup: the pinball-like nature of LLM parameters; off-target benefits of vaccines; incorporating humility and curiosity in LLMs; the potential benefits of author-level publication metrics; OpenAI unveils ChatGPT Health; international alternatives to PubMed; the relationship between flu and heart attacks; why LLMs remain vulnerable to prompt injection; much more:
AI, STATISTICS & DATA SCIENCE
- “A large language model’s parameters are often said to be the dials and levers that control how it behaves. Think of a planet-size pinball machine that sends its balls pinging from one end to the other via billions of paddles and bumpers set just so. Tweak those settings and the balls will behave in a different way. …But the basics of what parameters are and how they make LLMs do the remarkable things that they do are the same across different models.” At MIT Technology Review, Will Douglas Heaven offers a crash-course on just what LLM “parameters” are and do.
- “Despite the known accuracy issues with AI chatbots, OpenAI’s new Health feature will allow users to connect medical records and wellness apps like Apple Health and MyFitnessPal so that ChatGPT can provide personalized health responses like summarizing care instructions, preparing for doctor appointments, and understanding test results…. The company worked with more than 260 physicians over two years to develop ChatGPT Health and says conversations in the new section will not be used to train its AI models.” Ars Technica’s Benj Edwards reports on this week’s announcement by OpenAI that it is planning a new offering: a ChatGPT Health LLM chatbot that can connect with your personal electronic health records.
- “…the root cause is the inability to distinguish valid instructions in prompts from users and those embedded into emails or other documents that anyone—including attackers—can send to the target. When the user configures the AI agent to summarize an email, the LLM interprets instructions incorporated into a message as a valid prompt….AI developers have so far been unable to devise a means for LLMs to distinguish between the sources of the directives.” Also at Ars: Dan Goodin reports on why ChatGPT, as well as other LLMs, remain vulnerable to prompt injection attacks, and why that problem may be difficult or impossible to fully solve.
- “…we explore a simple idea: medical artificial intelligence should not only aim to be accurate, but also to recognise when it might be wrong. Today, many systems speak with great confidence even when faced with situations they do not fully understand. This can mislead clinicians and put patients at risk. We argue that AI used in healthcare should instead show two familiar human qualities: curiosity and humility.” A research article published in PLOS Digital Health by Cajas Ordóñez and colleagues explores a vexed attribute of many LLMs – the tendency for their output to be couched in terms that express an unwarranted sense of confidence (H/T Erich Huang, MD, PhD).
- “Patient-specific fact verification of LLM-generated text can be automated using VeriFact, with over 93% agreement with consensus clinician chart review using locally hosted, open-source AI models without task-specific or medical-specific training….VeriFact’s judgments are more consistent than those of individual human clinicians performing the same task A research article published in NEJM AI by Chung and colleagues presents a benchmark dataset and accompanying retrieval-augmented generation AI model designed to provide a layer of fact-checking for LLM-generated clinical documentation.
BASIC SCIENCE, CLINICAL RESEARCH & PUBLIC HEALTH
- “We show that a working line–specific haplotype of EPHB1 is associated with elevated levels of chase-bite motor patterns based on a well-validated behavior survey. These findings indicate that functional selection has shaped the genetic architecture of herding breeds, which may relate to their proficiency in addressing diverse tasks and challenges in maintaining control over the herd.” Maybe these good boys are born with it: A genomics study published in Science Advances by Jeong and colleagues offers evidence that the signature abilities of some breeds of herding dogs have a genetic basis.
- “What causes such vaccine bonuses? Most hypotheses focus on the inflammation that arises when the immune system mobilizes to fight off an infection. ‘You have damage to the surrounding environment” in the body, “and that takes time to calm down,’ Dr. Chu said. The effects of inflammation can far outlast the initial illness. It may allow other infections to take hold, or cause heart attacks and strokes when clots form in narrowed blood vessels.” The New York Times’ Paul Span reports on recent research demonstrating a number of “off-target” benefits that come along with getting vaccinated.
- “The connection between influenza and a host of cardiovascular conditions is not a surprise. Scientists have long known that the virus affects more than the respiratory tract. It triggers a powerful inflammatory response, sending the immune system into overdrive. This response can activate blood platelets, increasing the risk of blood-clot formation. Fever elevates heart rate and energy demands, and dehydration adds further strain. The result is a physiological storm that can tip vulnerable people — especially those with underlying cardiovascular disease — into crisis.” In a Spotlight article at Nature, Ashish Jha makes the case for treating vaccination for flu as an opportunity for protection not just from viral infection, but from cardiovascular problems as well.
COMMUNICATIONS & Policy
- “We find that, using a variety of impact factor thresholds, researchers overwhelmingly receive more recognition with article-level indicators. This advantage cuts across racial, gender, and career groups. Even under zero-sum rank-ordered conditions, we observe that many more researchers would receive recognition using article-level indicators.” A research article published in PLOS Biology by Arabi and colleagues presents findings that suggest that replacing journal-level with article-level impact metrics would be beneficial to authors.
- “A concept known as the Szilard point helps to contextualize the issues arising from excessive competition for grants. Named after the Hungarian-born physicist Leo Szilard…this metric describes the threshold at which the total cost of competing for a grant equals (or surpasses) the value of the available funding. These costs are incurred by scientists in writing proposals, by their peers in reviewing them and by the administrative systems that run the process.” A “careers column” article in Nature by Gerald Schweiger looks at what happens when pursuing grant funding literally becomes more trouble than it’s worth.
- “PubMed has never really stood alone: It already coexisted alongside Europe PMC, Embase, and other specialized databases. But it has long held symbolic and functional dominance over its field. It has also traditionally supported global accessibility and standardized searchability. Yet recent debates have raised concerns about PubMed’s reliability, stability, and openness.” An editorial published in STAT News by Sara Rubinelli, Rebecca Ivic, Kenneth H. Rabin, Lawrence O. Gostin, and Scott C. Ratzan looks at the rise of international alternatives to PubMed as a scientific library.
