AI Health

Friday Roundup

The AI Health Friday Roundup highlights the week’s news and publications related to artificial intelligence, data science, public health, and clinical research.

October 4, 2024

In this week’s Duke AI Health Friday Roundup: FTC takes aim at misleading AI claims; study suggests greater LLM size and instructability correlates with less reliability; evidence of misconduct surfaces in Alzheimer, Parkinson research; worries that the antimicrobial “bubble” may be about to pop; harnessing AI for better disaster preparedness; increasing numbers of female docs entering high-compensation specialties; much more:

AI, STATISTICS & DATA SCIENCE

“The increasing use of AI systems in the disaster-management domain brings promise but also risks. For example, because there tend to be more ground radar systems in wealthier regions, there can be biases in the data sets that AI algorithms are trained on to predict precipitation patterns. Such biases can put poorer regions at a disadvantage…To address these risks, specialists and stakeholders must come together to provide standards — internationally agreed best practices — to govern AI-infused disaster-management tools. These standards should address everything from how data are collected and handled, to how algorithms are trained, tested and used.” A Nature commentary by Kuglitsch and colleagues proposes ways to responsibly harness AI for better natural disaster warning systems.
“…we show that easy instances for human participants are also easy for the models, but scaled-up, shaped-up models do not secure areas of low difficulty in which either the model does not err or human supervision can spot the errors. We also find that early models often avoid user questions but scaled-up, shaped-up models tend to give an apparently sensible yet wrong answer much more often, including errors on difficult questions that human supervisors frequently overlook.” A research article published in Nature by Zhou and colleagues describes an inverse relationship between the size and “instructability” of large language models and the reliability of their output.
“Automatic metrics are extensively used to evaluate natural language processing systems. However, there has been increasing focus on how they are used and reported by practitioners within the field. In this paper, we have conducted a survey on the use of automatic metrics, focusing particularly on natural language generation (NLG) tasks. We inspect which metrics are used as well as why they are chosen and how their use is reported. Our findings from this survey reveal significant shortcomings, including inappropriate metric usage, lack of implementation details and missing correlations with human judgements.” A preprint by Schmidtová and colleagues, available as unreviewed preprint from arXiv, evaluates current practices with regard to the evaluation of natural language processing AI systems, and finds the state of play to be lacking.
“…we firmly believe AI policy should be informed by scientific understanding of AI risks and how to successfully mitigate them. Therefore, if policymakers pursue highly committal policy, the evidence of the associated AI risks should meet a high evidentiary standard. Advancing significant legislation without clear understanding of risks it intends to address may lead to more negative consequences than positive outcomes….We support evidence-based policy and recognize current scientific understanding is quite limited. As the highly scrutinized and potentially influential California bill that would have regulated AI in that state is vetoed, a group of AI researchers has proposed an evidence-based framework for developing AI policy.

BASIC SCIENCE, CLINICAL RESEARCH & PUBLIC HEALTH

“I can’t help but think of the era of antibiotics as a bubble that is about to burst….I have been fortunate to live my entire life within this bubble. Most of us were born into it. Living in the bubble has made us comfortable and isolated from the historical threats of infectious diseases. We don’t fear contracting a deadly infection during surgery, which carried a 40% mortality rate in the pre-antibiotic era. It has deprived us of understanding the true threat that infectious diseases pose to humanity and has made us complacent.” An expert viewpoint article in Science by Andreas J. Bäumler details the growing danger of a collapse in the effectiveness of antimicrobial therapeutics as a resistant strains flourish while the pipeline of new drugs stagnates.
“For populations at high risk of exposure, protective measures against pathogens and antimicrobial resistance are paramount. Environmental and health authorities should expand existing food-safety programmes — which already monitor contaminants such as heavy metals, pesticides and bacterial pathogens — to include microplastics and communicate these potential risks to consumers.” A commentary published in Nature by Li and colleagues describes the rise of the “plastisphere” – a new domain created by enormous amounts of plastic waste that are now home to numerous microbes, including pathogenic ones.
“…over the past 2 years questions have arisen about some of Masliah’s research. A Science investigation has now found that scores of his lab studies at UCSD and NIA are riddled with apparently falsified Western blots—images used to show the presence of proteins—and micrographs of brain tissue. Numerous images seem to have been inappropriately reused within and across papers, sometimes published years apart in different journals, describing divergent experimental conditions.” A shocking article by Science’s Charles Pillar details mounting evidence of research misconduct with potentially enormous ramifications for Alzheimer and Parkinson disease research.
“In terms of figuring out how to do it [achieve an effective HIV vaccine], there have been about four amazing papers out, largely from Scripps Research Institute and Duke University, in the last year. And I think the field is getting really exciting in terms of understanding and training the body’s ability to elicit broadly neutralizing antibodies. I don’t know if it’s going to be two years, three years, five years, but I do feel like there are some incipient breakthroughs. So I don’t think it’s time to give up.” STAT News’ Helen Branswell talks with incoming National Institute of Allergy and Infectious Disease director Jeanne Marrazzo about the agency’s priorities – and the outlook for an array of concerning viral diseases.
“…female physicians were underrepresented among residents entering high-compensation specialties compared with non–high-compensation specialties. However, while high-compensation surgical specialties experienced a steady increase in the proportion of female applicants and matriculants over time, high-compensation nonsurgical specialties experienced an overall decrease in the proportion of female applicants and no significant changes in the proportion of female matriculants.” A research letter published in JAMA by Pereira-Lima and colleagues examines recent trends in female physician representation in higher-paying medical specialties, including surgical specialties.

COMMUNICATION, Health Equity & Policy

“Claims around artificial intelligence have become more prevalent in the marketplace, including frequent promises about the ways it could potentially enhance people’s lives through automation and problem solving. The cases included in this sweep show that firms have seized on the hype surrounding AI and are using it to lure consumers into bogus schemes, and are also providing AI powered tools that can turbocharge deception.” A recent news release from the Federal Trade Commission highlights enforcement activity taken against a number of purveyors of misleading AI hype.
“The tool will generate recommendations based on hearing transcripts and evidentiary documents, supplying its own analysis of whether a person’s unemployment claim should be approved, denied, or modified. At least one human referee will then review each recommendation….If the referee agrees with the recommendation, they will sign and issue the decision. If they don’t agree, the referee will revise the document and DETR will investigate the discrepancy.” An article by Gizmodo’s Todd Feathers details plans by the state of Nevada to use a generative AI system developed by Google to process unemployment benefit claims, in hopes of more rapidly taming a large backlog.
“We have reported frequently on Cureus in the past. The journal, a Springer Nature title that prides itself on speedy publication, has drawn flak for publishing low-quality studies and hosting “channels” that allow questionable organizations to hand-pick their own editors. Earlier this year, it retracted 55 papers from Saudi Arabia for dubious authorship.” Retraction Watch reports that two high-volume, rapid-review, open-access journals – Cureus and Heliyon – are being scrutinized by Clarivate over worry about the quality of their publications and have had their listings on the Web of Science “paused.”