Brief
Telehealth in rheumatology remains vastly underutilized in the post-pandemic era. We created a predictive model to identify future rheumatology visits that could be safely performed via telehealth. Implementing this predictive model will improve patient access to appropriate, efficient, and effective care.
The Problem
Before this DIHI project began, David Leverenz, MD, MEd, and Jayanth Doss, MD, MPH, led a team to develop a novel scoring system called the Encounter Appropriateness Score for You (EASY), in which providers document their perception of the appropriateness of telehealth or in-person care after every encounter. An analysis of over 12,000 EASY scores in 2021 demonstrated that our providers think telehealth is acceptable for approximately 30% of all follow-up visits in rheumatology. However, Duke Rheumatology (which cares for over 13,000 patients in over 24,000 annual encounters) currently performs only about 15% of its visits by telehealth. As such, telehealth remains vastly underutilized at Duke Rheumatology in the post-pandemic era. (Figure 1 on page 10 of Impact Volume 25) If Duke rheumatologists can better identify patients who are appropriate for telehealth rather than in-person care, we can expand access to telehealth by over a thousand visits each year. The team used EASY scores to create a predictive model to identify future rheumatology visits appropriate for telehealth care. The initial model was a logistic regression algorithm based on data stored in the Duke Protected Analytics Computing Environment (PACE). Before collaborating with DIHI, the team could not fully implement this model without converting the model to a more actionable interface.
Our Solution
Our collaboration with DIHI was instrumental in putting our model to use. Their unique skills and understanding were crucial in designing the model for an impactful fit within our clinical workflow. DIHI are experts in shifting from a retrospective to a prospective mindset: model feature and operational design become dynamically different when you move from using a static dataset of the past to a continuously updating one that periodically re-accounts for the past, present, and future. Together, we have created a predictive model that can be accessed in real-time and implemented in clinical practice, providing healthcare professionals with a powerful tool for shared decision-making.
Impact
The reconfigured predictive model uses real-time data that can be accessed through a Tableau dashboard. DIHI’s updated design allows us to utilize this data in real-time for shared decision-making with patients. Additionally, periodic reviews of patient lists allow us to offer visit modality switches to patients identified by the model as appropriate for telehealth. The predictive model, with its ability to identify future visits appropriate for telehealth, has the potential to significantly improve patient care and practice efficiency.
To visualize the model’s performance, AUROC and PRC curves comparing the 2021 and 2023 models are presented. While AUROC and recall may show improvement, precision is expected to be lower in 2023 due to the drop in telehealth usage from 2021 to 2023. This reflects the current challenge of predicting telehealth-appropriate visits in an environment with low actual telehealth usage.
(Figures on page 11 of Impact Volume 25)
Future steps involve implementing the model within clinical care, measuring its impact, and disseminating it to other rheumatology practices. If our model was used in the last year, telehealth use would have been 18% rather than 5%. If used as intended, we can expect to significantly and satisfactorily expand telehealth access from 3% currently scheduled to 20% over the next year.
The DIHI team also contributed to other ongoing work on understanding the perspectives of rheumatology patients with regards to the appropriateness of telehealth visits. In particular, DIHI used our EASY data to help clinicians identify patients for qualitative interviews, which clinicians completed. Two manuscripts describing patient and provider perspectives of telehealth appropriateness are in preparation.
Academic output
The initial model development (before this DIHI grant) resulted in several published manuscripts. Academic output from the novel version of the model is pending final implementation. Prior works include the following:
- Smith ID, Coles TM, Howe C, Overton R, Economou-Zavlanos N, Solomon MJ, Zhao R, Adagarla B, Doss J, Henao R, Clowse MEB, Leverenz DL. Telehealth Made EASY: Understanding Provider Perceptions of Telehealth Appropriateness in Outpatient Rheumatology Encounters. ACR Open Rheumatol. 2022 Oct;4(10):845-852. doi: 10.1002/acr2.11470. Epub 2022 Jul 19. PMID: 35855564; PMCID: PMC9555194.
- Solomon M, Henao R, Economau-Zavlanos N, Smith I, Adagarla B, Overton AJ, Howe C, Doss J, Clowse M, Leverenz DL. Encounter Appropriateness Score for You Model: Development and Pilot Implementation of a Predictive Model to Identify Visits Appropriate for Telehealth in Rheumatology. Arthritis Care Res (Hoboken). 2024 Jan;76(1):63-71. doi: 10.1002/acr.25247. Epub 2023 Nov 13. PMID: 37781782.
- Smith ID, Solomon MJ, Mulder H, Sims C, Coles TM, Overton R, Economou-Zavlanos N, Zhao R, Adagarla B, Doss J, Henao R, Clowse MEB, Bosworth H, Leverenz DL. Evaluating factors associated with telehealth appropriateness in outpatient rheumatoid arthritis encounters using the Encounter Appropriateness Score for You (EASY). J Rheumatol. 2024 May 15:jrheum.2024-0014. doi: 10.3899/jrheum.2024-0014. Epub ahead of print. PMID: 38749564.
