Corey, Kristin, Joshua Helmkamp, Allan D. Kirk, Suresh Balu, David Thompson, Leila Mureebe, Joshua Watson, Keith Marsolo, Lesley Curtis, and Mark Sendak. “Assessing Quality of Real-World Data Supplied by an Automated Surgical Data Pipeline.” Journal of the American College of Surgeons 229, no. 4 (October 25, 2019): S89. https://doi.org/10.1016/j.jamcollsurg.2019.08.203
Introduction
We developed a fully automated surgical data pipeline curating electronic health record (EHR) data from an Epic platform for >400,000 procedures at a quaternary healthcare system to support research, quality improvement, and innovation. Due to significant efforts to improve the quality of surgical research, we sought to validate real-world data quality so that surgical research and quality improvement studies could be performed with robust face validity.
Methods
Data quality was assessed using the harmonized data quality framework. The NSQIP database was used for extrinsic validation. We performed conformance checks to confirm that data elements adhered to definitions, completeness checks to estimate completeness of data capture, and plausibility checks to test relational, temporal, and atemporal relationships. Checks evaluated expected stability of population measures, expected data shift, and expected data drift.
Results
A total of 527 data quality analyses were completed over 13 domains in a fully automated EHR data pipeline (330 conformance checks and 170 completeness checks). Additionally, 27 clinical plausibility checks were completed demonstrating strong surgical data integrity (Table). Extrinsic comparison to NSQIP demonstrated similar data shift as diagnosis codes transitioned from ICD9 to ICD10 (Figure).
Conclusion
We developed a framework to evaluate the quality of automatically curated real-world EHR data for surgical patients. This framework is not meant to assess the quality of clinical care, but to validate the substrate used for downstream analyses of clinical care. To our knowledge, this is the first rigorously developed and evaluated framework to assess real-world data for surgical research.