In today’s healthcare landscape, organizations often rely on a blend of automated and manual methods to compile, assess, and categorize patient safety reports. Front-line clinicians manually input these reports into the RL Datix reporting system, which includes both discrete data and free-text narratives. While the data collection might start digitally, it usually becomes inaccessible for real-time analysis and trending once entered. Reporters can only view the adverse events they personally reported, while unit and file managers have broader access relevant to their authority. However, the raw format of the data, particularly the narrative descriptions, often prevents comprehensive insights. Consequently, trends across the organization, such as spikes in infections or medication errors, tend to be perceived as isolated incidents due to their dependence on specific units or service lines.
Current analysis is performed via a mix of built-in reports, manual data manipulation, and discrete field displays. This siloed analysis limits organization-wide insights and typically necessitates multiple patient safety analysts and data specialists. In academic medical centers (AMCs), this process consumes valuable time and resources, highlighting the need for a technology solution that automates analytical processes and reallocates resources towards critical patient care improvements.
As a Proof of Concept (POC), our focus on the automated analysis of medication-related patient safety reports aims to minimize manual analytical workloads, enhance insights from daily reports, and uncover patterns across the organization. Collaborating with the University of Utah Health, we employed five years of medication-related patient safety reports to optimize generalized and domain-specific language models using Amazon SageMaker. This approach enables the classification of error severity through discrete fields, identifies high-risk medications from narrative texts, and visualizes high-risk medication-related events according to their harm levels.
Solution Overview
We utilized Amazon Comprehend Medical to pinpoint high-risk medications, summarizing results in an interactive dashboard built on Amazon QuickSight. The entire data processing pipeline was automated using event-driven, serverless architecture via AWS Lambda. Given the sensitive nature of patient safety reports, all services utilized in this solution are HIPAA eligible, conducted within a HIPAA-compliant environment. The de-identification of reports was achieved using the Amazon Comprehend Medical DetectPH API, as discussed in another blog post here.
To enhance the efficiency of the patient safety reporting process, we refined and compared various transformer-based LLMs in collaboration with AWS partner Huggingface, effectively classifying high-risk medications based on free-text descriptions (see Table 1). A sample Jupyter notebook has been prepared for academic medical centers to facilitate further customization. The architectural diagram provided outlines the potential steps for patient safety professionals to implement this solution on AWS.
Additionally, to ensure a secure and compliant machine learning (ML) environment, we set up Amazon SageMaker with data encryption, network isolation, authentication, and authorization as defaults. Key features include:
- Encryption of data at rest in an Amazon Simple Storage Service (Amazon S3) bucket, managed with your own key stored in AWS Key Management Service (AWS KMS).
- Encryption enabled for data at rest in Amazon Elastic File System (Amazon EFS) using the default AWS KMS key.
- An Amazon SageMaker Studio environment launched within a private VPC, with network isolation and access to other AWS services via AWS PrivateLink.
- Role-based access control using Amazon Identity and Access Management (IAM) to regulate SageMaker user permissions.
For organizations seeking a secure research environment via a lockdown Virtual Desktop Infrastructure (VDI) without screen copying, Amazon AppStream 2.0 or Amazon Workspaces can be utilized to access Amazon SageMaker through a presigned URL.
This solution effectively leverages AWS Analytics and AI/ML services for automatic data processing, information extraction, and predictive analytics on patient safety reports. High-alert medications, derived from the Institute for Safe Medication Practices (ISMP) high-risk medication list, have been consolidated into RxNorm concepts, which were mapped with alternative synonyms extracted by Amazon Comprehend Medical. The analysis culminated in a visually rich dashboard on Amazon QuickSight, showcasing various data visualizations that combine both discrete and textual sources.
Outcomes
The AI-driven approach yielded promising results, as detailed in Table 1, with metrics ranging from a Precision of .881 to .901, Recall from .874 to .899, Accuracy from .874 to .899, and an F1 score between .873 and .899, depending on the application.
Conclusion
The success of this POC project paves the way for further collaboration with an AWS partner to explore additional use case applications and develop a production-ready system incorporating comprehensive clinical data. This initiative may lead to further metrics, models, and enhancements. With the manual entry of clinical data into the patient safety reporting system, efforts are ongoing to integrate electronic health record (EHR) information into the analytical framework.
Machine learning serves as a powerful tool to boost efficiency, reduce time to insight, and reveal potentially obscured information in medication-related patient safety reports. Given these findings, it is essential to continue enhancing outcome scores and extend these efforts to other facets of patient safety reporting. For an excellent resource on this topic, check out here.
Location: Amazon IXD – VGT2, 6401 E Howdy Wells Ave, Las Vegas, NV 89115.
Leave a Reply