AI Health
Friday Roundup
The AI Health Friday Roundup highlights the week’s news and publications related to artificial intelligence, data science, public health, and clinical research.
April 18, 2025
In this week’s Duke AI Health Friday Roundup: How to read an article about AI; probing deficiencies in heart failure care; AlphaFold2 sometimes posits unlikely protein structures; challenges of implementing health AI in under-resourced health systems; teachers wrestle with AI in the classroom (and lesson planning); computational analysis shows decline in evidence-based appeals in Congressional speeches; AI hallucinations offer opportunity for malware; much more:
AI, STATISTICS & DATA SCIENCE
- “Running that code should result in an error when importing a non-existent package. But miscreants have realized that they can hijack the hallucination for their own benefit. All that’s required is to create a malicious software package under a hallucinated package name and then upload the bad package to a package registry or index like PyPI or npm for distribution. Thereafter, when an AI code assistant re-hallucinates the co-opted name, the process of installing dependencies and executing the code will run the malware. The recurrence appears to follow a bimodal pattern…suggesting certain prompts reliably produce the same phantom packages.” A post by Thomas Claburn at The Register outlines the growing dangers posed by the combination of hallucination and “slop-squatting” in AI-assisted code development.
- “Prompted by a highly confident prediction for a biologically meaningless, randomly permuted repeat sequence, we assessed AF2 performance on sequences composed of perfect repeats of random sequences of different lengths. AF2 frequently folds such sequences into β-solenoids which, while ascribed high confidence, contain unusual and implausible features such as internally stacked and uncompensated charged residues). A research article by Pratt and colleagues, published in the Computational and Structural Biotechnology Journal reports that the protein-structure-predicting AlphaFold2 (but not its successor AlphaFold3) has a tendency to confidently predict implausible structures in certain scenarios.
- “In the remote parts of Northern Arizona where North Country has its clinics, broadband infrastructure is spotty. Access to high-speed Internet powers the clinics’ access to digitally stored patient data, the electronic health record (EHR) system and telehealth visits…While all the clinics theoretically have Internet access and WiFi networks, Cortes said her clinics still have issues connecting to the internet almost daily. ‘When you think about that, who cares about AI? If you can’t keep this thing connected, it’s not going to make people happy,’ she said.” An article in Fierce Healthcare by Emma Bevins (featuring Duke’s Health AI Partnership) offers a detailed look at the challenges of implementing health AI in lower-resourced hospitals and healthcare systems.
- “Much of the IT and cybersecurity infrastructure underpinning the US health system is in danger of a possible collapse following a purge of IT staff and leadership at the Department of Health and Human Services (HHS), four current and former agency workers tell WIRED. This could put vast troves of public health data, including the sensitive health records of hundreds of millions of Americans, clinical trial data, and more, at risk of exposure.” An article by Wired’s David Gilbert reports on concerns about the security of troves of sensitive data stored by the Department of Health and Human Services.
- “In this paper, we aim to provide novice readers with an introduction to the many uses of AI in healthcare and how to begin to critically appraise these studies. Our objectives were to: introduce and define key concepts; outline some ethical principles for health related use of AI; give examples of reporting guidelines and checklists for appraising papers reporting AI studies; and propose some initial questions to ask about papers describing machine learning decision support systems (one of the best established uses of AI in clinical practice).” A timely article published in the BMJ by Dijkstra and colleagues walks readers through the critical appraisal of research articles reporting results from studies involving AI.
BASIC SCIENCE, CLINICAL RESEARCH & PUBLIC HEALTH
- “Two major themes emerged; the first theme was centered on hospital readmissions, which often uncovered gaps in communication and deficiencies in discharge education. Patients expressed frustration with recurrent admissions, identified communication challenges while inpatient, and highlighted the shortcomings of the current discharge education models. The second theme explored the ways in which fragmented understanding of HF as a result of gaps in effective care delivery impacted multiple domains of care spanning the spectrum from initial diagnosis to prognosis.” A qualitative research article published in the Journal of Hospital Medicine by Latif and colleagues combined patient interviews and chart notes to reveal shortcomings in the treatment of patients with heart failure.
- “Deciphering the genetic basis of height provides understanding into the biology of growth and is also of relevance to disease, as increased or decreased height relative to population averages has been epidemiologically and genetically associated with an altered risk of cancer or cardiometabolic diseases. With recent large-scale genome-wide association studies of human height reaching saturation, its genetic architecture has become clearer.” A review article published in Nature Reviews Genetics by Bicknell and colleagues examines the complexities of genetic factors in determining human height.
- “We have made substantive changes to the CONSORT checklist. We added 7 new checklist items, revised 3 items, deleted 1 item, and integrated several items from key CONSORT extensions. We also restructured the CONSORT checklist, with a new section on open science. The CONSORT 2025 statement consists of a 30-item checklist of essential items that should be included when reporting the results of a randomized trial and a diagram for documenting the flow of participants through the trial.” An updated version of the CONSORT guidelines for reporting clinical trial results in journal articles has been published this week in multiple journals.
COMMUNICATION, Health Equity & Policy
- “Many individual teachers are finding themselves navigating this terrain on their own, determining when A.I. is the enemy, and also, how it could be a friend. Mike Sullivan, a middle school math teacher in Brockton, Mass., estimated about half his students are using problem-solvers like Google Lens. Some constrain their use to homework help. But he has also caught students using A.I. tools during in-class quizzes.’It’s just too easy,’ Mr. Sullivan said. The experience has made him rethink the wisdom of relying on computers so heavily in the classroom.” A New York Times article by Dana Goldstein explores the ethically and professionally challenging territory of teachers adapting to a growing number of AI applications crowding the educational marketplace.
- “…what makes the US ecosystem so powerful is what then happens to the university research: it’s the engine for creating start-ups and jobs. In 2023, US universities licensed 3,000 patents, 3,200 copyrights and 1,600 other licences to technology start-ups and existing companies. Such firms spin off more than 1,100 science-based start-ups each year, which lead to countless products…It all adds up to a virtuous circle of discovery and innovation.” A Nature opinion article by Steve Blank traces the origins of American dominance in science and innovation, and warns that recent changes may be imperiling that preeminence.
- “…we explore a continuum of perspectives from evidence-based reasoning, rooted in ascertainable facts and data, at one end, to intuitive decisions that are driven by feelings and subjective interpretations, at the other. We analyse the linguistic traces of those contrasting perspectives in congressional speeches from 1879 to 2022. We find that evidence-based language has continued to decline since the mid-1970s, together with a decline in legislative productivity.” A research article published in Nature Human Behavior by Aroyehun and colleagues traces the evolution of evidence-based rhetoric in Congressional speeches over time.
- “The aim of this commentary is twofold: to summarize the discrepancies in the extent to which patient information leaflets (PILs) list potential benefits and harms of trial interventions; and to highlight subsequent ethical issues that may result from failure to disclose potential benefits or harms . A review of 247 patient information leaflets (PILs) found that the extent to which potential benefits and harms are described varies, with 28 (11%) not describing potential benefits and 23 (9%) not describing potential harms.” A commentary published in the journal Trials by Howick and colleagues examines ethical issues in informational leaflets provided to patients as part of the informed consent process.
