AI Health

Friday Roundup

The AI Health Friday Roundup highlights the week’s news and publications related to artificial intelligence, data science, public health, and clinical research.

January 17, 2025

In this week’s Duke AI Health Friday Roundup: amount of misinformation needed to “poison” an AI; projected burden of dementia may be greater than previously thought; will LLMs lighten clinician loads – or add to them?; how exposure to red light may be related to thrombosis risk; learning to do better with communicating science; leveraging LLMs to improve health equity; health systems scramble to assure compliance with algorithmic nondiscrimination requirements; much more:

AI, STATISTICS & DATA SCIENCE

Three clear glasses filled with water, with blue ink diffusing in streaks and clouds through the water. Image credit: Chaozzy Lin/Unsplash
Image credit: Chaozzy Lin/Unsplash
  • “Ideally, having substantially higher volumes of accurate information might overwhelm the lies. But is that really the case? A new study by researchers at New York University examines how much medical information can be included in a large language model (LLM) training set before it spits out inaccurate answers. While the study doesn’t identify a lower bound, it does show that by the time misinformation accounts for 0.001 percent of the training data, the resulting LLM is compromised.” Ars Technica’s John Timmer reports on a recent study by Alber and colleagues evaluating just what proportions of misinformation are needed to “poison” a large language model’s accuracy.
  • “Unfortunately, while the field aims to reduce the administrative burden for physicians, it simultaneously exposes them to risk and creates a new cognitive task in their workflow. The result of the first principle creates a new and tedious burden of high-stakes proofreading for physicians, when the use of LLMs promised to reduce their cognitive and administrative burden. If we expect physicians to thoroughly proofread and edit LLM-generated content, we must acknowledge this as a new cognitive and administrative task for them. Proofreading content that was neither written nor dictated by the user is difficult to do well, especially while under pressure.” In a viewpoint article published in NEJM AI, Ohde and colleagues address the potential for large language model AIs – touted as potential administrative labor-savors and burden-lifters for clinicians – to actually create additional work for physicians.
  • “The review identified three key issues in current MMR applications: 1. Insufficient articulation of methodological contributions. 2. Limited visualization of quantitative and qualitative data integration. 3. Minimal engagement with recent MMR advancements. To address these gaps, a targeted To-Do List was created, offering actionable strategies for improving methodological rigor.” A review article published in the journal Accountability in Research by Gengyan Tang evaluates the use of mixed-methods approaches for evaluating research integrity in the published literature.
  • “Overall, the Phi-3 Mini fine-tuned model and RAG demonstrated strong resource-to- performance results, matching the performance of highly sophisticated flagship models such as GPT-4o. While direct comparison is difficult,40,41 our models show competitive results to common industry tools…Importantly, both configurations were constructed with economic and integration viability as a priority. Their straightforward design and low resource requirements make reproducibility more accessible across institutions.” A preprint by Rollman and colleagues, available at arXiv, describes an evaluation of LLM models tasked with generating surgical claim codes under “real-world” resource constraints.

BASIC SCIENCE, CLINICAL RESEARCH & PUBLIC HEALTH

Closeup photograph taken from directly underneath a lighting fixture with a red-light incandescent bulb. Image credit: Mateo Avila Chinchilla/Unsplash
Image credit: Mateo Avila Chinchilla/Unsplash
  • “The present study reports that selective exposure to long-wavelength red light results in a significant reduction in thrombosis. This is consistent across differential exposure and animal breeds. We found reduced platelet aggregation and activation leading to a reduction in platelet-driven NET release by neutrophils, together resulting in a lower burden of thrombosis. This represents a unique way to modulate platelet–neutrophil interactions and subsequent thrombosis. In both venous and arterial pathologies, we observed significant reductions in thrombus burden with prophylactic selective exposure to long-wavelength red light.” In a research article published in the Journal of Thrombosis and Haemostasis, Andraska and colleagues explore the effects of exposure to longer (redder) wavelengths of light on coagulation and the development of thrombosis in mouse models (H/T @DanBuckland.me).
  • “People living further from MFMs or in rural areas had significantly lower MFM involvement in pregnancy care. Rates of any MFM involvement increased from 2016 to 2021, by 8.1 percentage points for urban residents, and 6.1 percentage points for rural residents, but there was low use of telemedicine-enabled MFM care over the whole study period, which included the COVID-19 pandemic…For many pregnancies, care from an MFM is clinically recommended and can potentially avert adverse pregnancy outcomes. In our sample, nearly half of at-risk pregnancies did not have any MFM involvement in care.” An analysis published in JAMA Network Open by Sullivan and colleagues examines patterns of access to maternal-fetal medicine specialists by pregnant women with private insurance coverage.
  • “Among the initial ED visits by included patients, there was no difference in the rates of hospital admission, subsequent health care use, or survival at 6 months for the preintervention period vs the postintervention period. In contrast to previous studies demonstrating the effectiveness of specialty palliative care, the palliative care intervention for emergency clinicians used in the current trial did not demonstrate an effect on health care use.” A stepped-wedge pragmatic trial, reported in JAMA by Gruzden and colleagues, evaluated a palliative care intervention designed to be implemented in emergency settings.
  • “Here, we report 6,589 contest outcomes for boxing, taekwondo, and wrestling from seven Summer Olympic Games (1996–2020) and nine World Boxing Championships (2005–2021). Using meta-analytic techniques, we found 50.5% wins by red for the overall data, which was not a statistically significant bias. Analyses of close contests resulted in 51.5% red wins, also not significantly different from the null expectation of equal proportions.” So much for seeing red: a meta-analysis published in Scientific Reports by Peperkoom and colleagues casts doubt on the ostensible “advantage” thought to accrue to combat-sports athletes who wear red outfits.
  • “Our results suggest that the current lifetime risk of dementia may be substantially higher than previously thought, emphasizing the importance of prevention throughout the life course. Policies focused on optimizing cardiovascular health and preserving hearing may be particularly important 12 . Accumulating data from clinical trials have linked healthy lifestyle behaviors, the absence of vascular risk factors and hearing rehabilitation with improved cognitive outcomes. However, only approximately 20% of US adults are meeting recommended lifestyle and cardiovascular health targets, and only approximately 30% of older adults with hearing loss are using a hearing aid.” An article published this week in Nature Medicine by Fang and colleagues offers a sobering estimate regarding the future burden of dementia in the United States.

COMMUNICATION, Health Equity & Policy

CRT-style television monitor with static on the screen. Image credit: TopSphere Media/Unsplash
Image credit: TopSphere Media/Unsplash
  • “Our expertise is valuable, but it’s not a substitute for connection. As current events continue to demonstrate, the gap between scientific understanding and public action isn’t due to a lack of facts or credentials – it’s due to our failure to create meaningful relationships with the communities we serve. The sooner we recognize this, the sooner we can begin to make real progress on the critical challenges facing our world.” In an essay available on Substack’s Unbiased Science, Jessica Steier and Sarah Scheinman examine the effects of misguided, misaligned, or absent efforts at science communication (h/t @meganranney.bsky.social).
  • “As the research community’s discussion of the impact of AI on equity has centered on risks, the policy landscape has largely followed suit; AI research has been framed as having significant potential for growth but potentially harmful effects on marginalized groups. As such, policy has focused on audits and regulation with an eye toward possible harms. Our main policy recommendation thus echoes our call for researchers: policy makers should seek not only to reduce equity-related harms, but also to incentivize equity-promoting use cases.” In an article published in NEJM AI, Pierson and colleagues explore ways to use large language models – a technology that has sparked considerable worry about fairness and equity in its various applications – to improve health equity.
  • “…Section 1557 also protects patients against discrimination on the basis of color, national origin, sex, age, and disability. That means an enormous range of artificial intelligence algorithms, clinical risk prediction calculators, and resource allocation tools used by health systems receiving federal funds — typically through government programs such as Medicare and Medicaid — will soon need to be evaluated for their potential to discriminate against patients.” STAT News’ Katie Palmer reports on US health systems’ push to demonstrate to federal government that their algorithmic technologies are not running afoul of anti-discrimination requirements before a deadline in May of 2025.