AI Health
Friday Roundup
The AI Health Friday Roundup highlights the week’s news and publications related to artificial intelligence, data science, public health, and clinical research.
July 11, 2025
In this week’s Duke AI Health Friday Roundup: LLM writing crops up in the biomedical literature; declining health outcomes for US children; reprieve for AI regulation efforts; reproducibility in Drosophila research; jailbreaking LLMs by flooding them with information; Medicaid cuts imperil NC hospitals; HHS sued by professional organizations over changes to vaccine policy; cybersecurity for medical devices; much more:
AI, STATISTICS & DATA SCIENCE
- “…yes, that means just what you think it means. That is a chatbot prompt, of course, and its presence indicates that the journals involved are using this technology to provide reviews of submitted papers. And why the hell not, since some of the papers have probably been extensive chatbotted during their preparation? Let the plagiarism machines clean up their own messes, I guess.” At In the Pipeline, Derek Lowe highlights a recent example demonstrating that some authors are starting to salt their manuscripts with basic steganographic prompts meant to counter review by LLMs employed by journals.
- “Life and being an ‘I’ is about having experiences in the physical world, about suffering and joy and curiosity and protectiveness and fascination and humor and lack of understanding and an underlying (if only vague) sense of profound loss and fear of death (one’s own and of one’s loved ones). It is not the glib throwing-about of technical phrases to make scientific-sounding claims, nor is it about virtuosically combining words like ‘love’ and ‘compassion’ and ‘psychopathy’ and ‘ontological’ and ‘recursion’ and so forth and so on.” In a letter forwarded by Gary Marcus, cognitive scientist and author Douglas Hofstadter gently lets some air out of the LLM “sentience” balloon (H/T @ Shelley.Rusincovitch).
- “…we leveraged excess word usage to show how LLMs have affected scientific writing in biomedical research. We found that the effect was unprecedented in quality and quantity: Hundreds of words have abruptly increased their frequency after ChatGPT-like LLMs became available. In contrast to previous shifts in word popularity, the 2023–2024 excess words were not content-related nouns but rather style-affecting verbs and adjectives that LLMs prefer.” A research article published by Kobak and colleagues in Science Advances offers findings from a textual analysis that suggests a major influx of LLM-based writing influences in recent biomedical research papers.
- “InfoFlood follows a three-stage approach: When a malicious query is input, InfoFlood employs Linguistic Saturation to restructure the query using linguistic transformations to evade safety filters and increase perceivable linguistic complexity. The restructured query is then passed to the target LLM, and we evaluate the model response to determine whether it is jailbroken. If the jailbreak attempt is unsuccessful, InfoFlood uses Rejection Analysis to detect the primary cause for failure in producing the jailbreak by analyzing the LLM response. Once the cause has been found, Saturation Refinement subtly refines the restructured query to prevent the discovered failure cause.” A preprint by Yadav and colleagues, available from arXiv, demonstrates a technique for “jailbreaking” LLMs – that is, overriding built-in constraints on producing potentially harmful content – that succeeds by flooding the LLM with extraneous information during prompting.
BASIC SCIENCE, CLINICAL RESEARCH & PUBLIC HEALTH
- “That sense of desperation is acutely felt in Martin County, where roughly 22,000 people, more than a quarter of whom are older than 65, live in a health care desert. Two physicians remain there. The nearest hospital with robust services is 40 minutes away, in Greenville. Some people who cannot afford to drive there take buses on slow routes, residents said.” The New York Times’ Eduardo Medina reports on the looming hospital funding crisis facing North Carolina in the wake of legislation that cuts funding for Medicaid.
- “Hospitals are currently fielding medical devices on their networks with limited insight into their security risks and even less market power to demand improvements. Patients, hospitals, and insurance companies foot the bill for cyber incidents, whereas device manufacturers have limited legal or financial incentive to improve security. We consider these incidents to be cybersecurity ‘never events,’ just as preventable and unforgivable as wrong-site surgery.” A viewpoint article by Kramer and colleagues, published in JAMA, scrutinizes risks to patients arising from cybersecurity lapses in medical devices.
- “Without decisive action to change the trajectory, the US health disadvantage is likely to worsen. Policymakers who are invested in enhancing children’s health must intervene. They could increase social media protections, invest in antipoverty measures such as child tax credits and income transfers, broaden health insurance coverage, invest in primary care, and pass firearm safety laws.” An editorial published in JAMA by Wolf and colleagues enlarges on a paper by Forrest and colleagues that adds to a growing array of evidence pointing to a downward trend in key health outcomes for US children.
- “The lawsuit filed today is the latest among a handful of efforts to push back against Kennedy’s assault on vaccines. Others include the launch of the Vaccine Integrity Project (VIP), a group of leading public health experts that defend vaccination and make recommendations on how nongovernmental agencies can help make sure that vaccine recommendations remain grounded in the best available science, free of external influence.” An article by Lisa Schnirring at the University of Minnesota CIDRAP reports on a lawsuit filed by professional medical organizations (including CIDRAP) against the Department of Health and Human Services over recent changes to vaccination policy.
COMMUNICATION, Health Equity & Policy
- “Contrary to the more dramatic narratives surrounding the reproducibility crisis…our study shows that a substantial proportion of claims—61%— can be considered as verified. The proportion of claims that have been directly challenged remains relatively low (7%), though likely underestimated. This indicates that research in the Drosophila immunity field is rather solid, possibly due to the power of Drosophila genetics, the collective ethos of sharing within the community, a less competitive environment, and a lack of influence associated with translational research….A novel and important contribution of our work is the identification of a large fraction of unchallenged claims —findings that, despite being published, have never been independently tested.” A systematic examination of reproducibility issues in Drosophila research by Lemaitre and colleagues, available from bioRxiv, finds that reproducibility in that domain are not evenly distributed.
- “Integrating seven different large-scale data sources, this work reveals central facts about research trajectories before and after tenure, demonstrating widespread turning points for individual scientists. Interestingly, although tenure represents a universal milestone across fields, post-tenure trajectories diverge in notable ways. On average, some disciplines experience declining publication rates, while others sustain high rates of research output. Moreover, receipt of tenure is associated with a shift toward more novel but less impactful work.” In what is sure to be an entirely uncontroversial study, Tripodi and colleagues present a preprint, available from arXiv, the sifts patterns in academic tenure and research productivity.
- “The F.D.A. was deeply imperfect. It was lumbering and often opaque, and it had gotten many things wrong in its long history. But the agency had also been built around a set of principles — scientific inquiry, impartial judgment, collective responsibility — that represented the best of what a functioning government could do for its people.” An in-depth story by Jeneen Interlandi for the New York Times Magazine examines how the FDA is being aggressively transformed as HHS Secretary Kennedy enacts sweeping changes.
- “Wiener and other state lawmakers can now get back to work writing and passing AI policy, at least for the time being—with the tailwind of a major moral victory at their backs. The movement to defeat the moratorium was impressively bipartisan: 40 state attorneys general signed a letter to Congress opposing the measure, as did a group of over 250 Republican and Democratic state lawmakers.” MIT Technology Review’s Grace Huckins reports on the recent defeat of a federal spending provision that would have placed a ten-year moratorium on AI regulation.
- “…The dust has settled a bit since April 1, multiple employees told STAT. But the persistent departures of both top and mid-level staff, as well as the overruling of scientists by Makary and biologics director Vinay Prasad, have continued to deplete morale…The mood has plummeted in the drug and biologics centers in particular.” In his first speech to FDA staff, he called the agency a “sock puppet” for the industries it regulates.” A STAT News article by Lizzy Lawrence reports on disruptions and declining staff morale at the FDA.
