Duke Authors Introduce Framework for Clinical Algorithm Oversight
A group of Duke Health researchers recently shared their insights on approaches to managing the complex issues that are emerging as “algorithmic medicine” increasingly becomes part of clinical care at hospitals and health systems. The authors, who comprise faculty and staff from Duke AI Health, Duke Health Technology Solutions, the Duke Institute for Health Innovation, and other physicians and researchers from Duke University and Duke University Health System, published an account of their approach to evaluating and monitoring the use of algorithmic predictive models at Duke Health hospitals and clinics.
The article, titled “A framework for the oversight and local deployment of safe and high-quality prediction models,” was published on May 31 in the Journal of the American Medical Informatics Association (JAMIA). It showcases the processes and procedures by which an expert group at Duke Health known as Algorithm-Based Clinical Decision Support (ABCDS) Oversight reviews, approves, and manages predictive models intended for use in patient care settings.
Predictive models have long been used in medicine, where physicians use them to estimate a patient’s degree of risk for a given outcome or to help make decisions about patient care. However, advances in the availability of data and the recent emergence of sophisticated analytics and machine-learning approaches to prediction have prompted a surge in the development and use of clinical algorithms, many of them drawing on data from electronic health records. But this surge itself carries risks as well. For this reason, Duke Health has made it an institutional priority to carefully review medical algorithms for issues related to accuracy, safety, usability, transparency, effectiveness, stability, and bias.
Actual frameworks for evaluating and governing predictive models in clinical settings are in relatively short supply, and many are still in their early stages. Duke Health’s ABCDS Oversight aims to bridge existing gaps by combining regulatory review processes with best practices from that draw upon Duke’s rich research expertise and statistical insight.
“What’s distinctive about the ABCDS Oversight approach is that it considers the algorithm or predictive model throughout its whole lifecycle,” says lead author Armando Bedoya, MD, Duke Health’s associate chief medical informatics officer and a physician in the Department of Medicine. “ABCDS doesn’t evaluate models on the basis of a narrow window of time, but starts with model development and continues all the way through validation, implementation in the real world, and the entire time it’s in use in patient care.”
Such close attention to the workings of multiple clinical algorithms – a total of 52 at the time of the article’s writing – requires the combined effort of multiple experts to ensure a thorough and timely review. The ABCDS Oversight Committee and three specialized subcommittees monitor the process continuously and provide input and feedback to the different teams presenting their models for review and evaluation. The ABCDS Oversight Committee is co-chaired by Michael Pencina, the Vice Dean of Data Science at the Duke School of Medicine, and Eric Poon, the Chief Health Information Officer for Duke Medicine.
“We’re trying to strike the best possible balance between fostering innovation in algorithmic medicine, while also making sure that the output of these models is accurate, safe, impactful, and equitable,” says co-author and ABCDS Oversight Program Director Nicoleta Economou, PhD. “Keeping pace with an extremely fast-moving field like clinical prediction is challenging in a lot of different ways, but we believe our approach has a great deal to offer people at Duke and beyond who want to be sure that their models translate into better outcomes for patients.”
Other authors on the JAMIA paper include Benjamin A. Goldstein, Allison Young, J. Eric Jelovsek, Cara O’Brien, Amanda B. Parrish, Scott Elengold, Kay Lytle, Suresh Balu, Erich Huang, Eric G. Poon, and Michael J. Pencina.
Bedoya AD, Economou-Zavlanos NJ, Goldstein BA, Young A, Jelovsek JE, O’Brien C, Parrish AB, Elengold S, Lytle K, Balu S, Huang E, Poon EG, Pencina MJ. A framework for the oversight and local deployment of safe and high-quality prediction models. J Am Med Inform Assoc. 2022 May 31:ocac078. doi: 10.1093/jamia/ocac078. Epub ahead of print. PMID: 35641123.
