Brief
Behavioral emergencies by patients during admission represent a growing problem for health systems. At Duke, there is no good way to identify patients at risk of such emergencies to mitigate that risk. To help our clinical colleagues who respond to these emergencies, including our behavioral response teams (BRTs), our team developed a machine-learning solution that alerts clinicians of high-risk patients at the time of their inpatient admission and 24 hours after admission. We have implemented the model in real-time and are evaluating its performance using metrics such as behavioral emergency event rate, use of restraints and antipsychotic meds, patient length of stay, and employee injury rates.
The Problem
Nearly half of all hospitalized patients in the US have comorbid psychiatric disorders.1 Behavioral health crises are associated with most patient-perpetrated assaults and physical threats.2 However, the current practice for patients experiencing a behavioral emergency is reactive rather than proactive, focusing more on containing the patient and suppressing violence rather than improving treatment and outcomes. Containment-focused responses to behavioral emergencies use chemical sedation and physical restraints, which contribute to longer lengths of stay and increased rates of injury and nosocomial infections for patients.3 There is a call to health systems to progress the standard of care for behavioral emergencies by forming trained behavioral response teams (BRTs).4
Our Solution
To address this gap, Duke University’s Departments of Medicine and Psychiatry and the Duke Institute for Health Innovation (DIHI) formed a transdisciplinary team to develop a machine-learning model for real-time detection of behavioral emergencies using electronic health record (EHR) data. Our goal is to support an established BRT’s proactive monitoring of and intervention in patients’ psychiatric destabilizations to improve patient care and safety for clinicians.
We used data from 310873 inpatient encounters for 179,416 unique adult patients at three Duke University Health System (DUHS) hospitals from 1/1/2017 to 12/31/2021. We excluded patients under 18 years old at the time of admission. A behavioral emergency outcome was defined as the occurrence of any of three interventions during the encounter while the patient was on an intermediate or step-down unit (excluding emergency and perioperative units): a violent restraint order placed, a medical hold order placed, or a non-violent restraint order placed plus the ordering or administration of antipsychotic medication. We compared the outcome to a limited data set of ground truth clinical note documentation of a behavioral emergency or consultation or dual physician adjudication. We designed the machine learning model to predict the first occurrence of a behavioral crisis up to 12 hours before the event. Model inputs included historical event data and 31 temporal data inputs related to patient assessment scores, medication ordering and administration, and toxicology screening labs. We trained the model using a light gradient boosting machine (LGBM), with 70% of the data used for training and 15% each for validation and testing. We evaluated the model performance based on sensitivity, specificity, positive predictive value (PPV), and area under the curve (AUC) of the receiver operating characteristic (ROC) curve.
Outcomes
Of the 310,873 encounters for 179,416 patients, 5,261 encounters (1.69%) for 4,759 patients (2.65%) met the behavioral emergency outcome definition during the encounter. A May-November 2022 comparison of the behavioral emergency outcome (n=203) with behavioral emergency clinical note documentation (n=506) and dual physician adjudication (n=31) yielded a precision of 90% and sensitivity of 36% for the behavioral emergency outcome. The clinical workflow applies the real-time behavioral emergency outcome, the predictive model, and secure paging technology to alert the BRT nurse, prompting a chart review and bedside assessment as needed to identify and intervene in imminent behavioral emergencies. We set up the behavioral emergency outcome label to run in real-time, with a bed view visualization tool and event paging system in place. Our next steps are to complete temporal validation of the model and integrate it into the BRT team workflow (Figure 1). We aim to evaluate the solution’s potential to decrease the use of violent restraints, antipsychotic medications, physician hold, suicide precautions, length of stay, and incidence of patient violence.
ACADEMIC OUTPUT
- Our work was submitted to, presented at, and selected as the winner of the GSA Applied AI Healthcare Challenge, which took place in May 2023 (virtual). GSA awarded our team a $25K cash prize, which we are using to support the project.
- Our work contributed to an abstract, poster, and spotlight presentation at the Machine Learning for Healthcare Conference, which was held in August 2023 at Columbia University in NY.
- Ratliff, W., Gao, M., Tang, L., Sendak, M., Nichols, M., Gardner, M., Saleh, S., Balu, S., Dunn, C., Tart, D., Kester, K., Shirey, K., O’Brien, C. Predicting Behavioral Emergencies in the Hospital [Poster presented]. Machine Learning for Healthcare (MLHC) Conference. August 2023. New York, NY, USA.
Steps after Dec 2023
We are piloting the behavioral emergency prediction solution in the Duke University Hospital (DUH) inpatient units to evaluate its performance. The clinical workflow utilizes push notifications and snoozing logic to alert the BRT team when a patient is at high risk of meeting or has met the real-time behavioral emergency phenotype. After the pilot period, we will evaluate the impact of the solution on behavioral emergency event rate, use of restraints and antipsychotic meds, patient length of stay, and employee injury rates.
