The Problem
Adults over the age of sixty-five account for nearly 40% of inpatient surgical procedures in the US, and their hospital stays often require heavy resource utilization.1,2 They are a highly vulnerable patient population at increased risk of perioperative morbidity and mortality. As the mean age of the United States population rises, the ability of a surgeon to make decisions by accurately incorporating goals of care, the utility of surgery, and the risk of harm is becoming an increasingly critical skill. To effectively identify which patients are at high risk for surgery, physicians require objective measures of the clinical factors that confer this elevated risk status. At project inception, there were no standardized geriatric-specific perioperative risk stratification tools that incorporated pertinent variables such as function, cognition, nutrition, polypharmacy, and code status.
Our Solution
We trained separate models to predict surgical outcomes at the time the surgical team decided to schedule a surgery and the day before surgery. Separate models were needed as the surgery and scheduling decisions are made well before the preoperative interventions. The model fitting our use case informs the prediction of surgery at the time of scheduling. The other informs the prediction of surgical outcomes on the day before surgery. We studied how the scheduling-to-surgery time interval and interventions impact model performance and decided to present both models’ results to perioperative teams at their respective dates. Furthermore, we redesigned the user interface to display model results for any combination of medical record number (MRN) and current procedure terminology (CPT, the codes by which surgeries are identified for billing) within seconds (Figure 1). This allowed the surgical team to explore surgical opportunities for any patient at any time.
Impact
We learned that the optimal utility of surgical predictive models required accurate model results well before the date of surgery and even before the surgical case was scheduled. We realized that the original model required substantial adaption to meet this need. Statistics from the thirty-day all-surgery case creation and admission models designed to this end are presented in Figure 2. The AUC for the 30-day mortality model trained with data before case creation and evaluated on patients 65 years old or older was 0.78. Furthermore, we learned that structured data supporting Mini-Cognitive Assessment scores, the Duke Activity Status Index (DASI) score, and Best Evaluation Systems Test (MiniBEST) scores was not consistent through the model training and validation cohort from 2015-2019, principally because the scoring and documentation processes were implemented at Duke more consistently in the years (2019-2022). In the spirit of continuous improvement, we recommend strident maintenance of score documentation, exploring enhanced model versions with 2018-2022 data, and consideration of processing language from surgical progress notes. Starting September 2022, steps were also be taken to increase data availability through a remote patient monitoring study.
Next Steps
We will present and silently test the mobile and desktop applications with surgeons and clinicians who were high users of previous models. Without influencing clinical decision-making, the surgeons will provide feedback about model performance on recent retrospective patients, application design, and ease of use.
Subsequently, the user applicability of the models will be studied and tested in the surgical outpatient and preoperative anesthesia surgical screening clinics. This continued study is supported by Duke Health leadership and the DIHI-funded project, Improving Perioperative Care Coordination via Enhanced Decision Support Tools, led by Dr. Jeanna Blitz.
Academic Output
References
- McDermott KW (IBM Watson Health), Freeman WJ (AHRQ), ElixhauserA (AHRQ). Overview of Operating Room Procedures During Inpatient Stays in U.S. Hospitals, 2014. HCUP Statistical Brief #233. December 2017. Agency for Healthcare Research and Quality, Rockville, MD.
- Dall T.M., Gallo P.D., Chakrabarti R., et. al. An aging population and growing disease burden will require a large and specialized health care workforce by 2025.Health Aff 2013; 32: pp. 2013-2020.
- Corey KM, Kashyap S, Lorenzi E, et al. Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study. PLoS Med. 2018;15(11):e1002701. Published 2018 Nov 27. doi:10.1371/journal.pmed.100270
