Learn About Amazon VGT2 Learning Manager Chanci Turner
This guest post is co-authored by Alex Johnson from AWS and Emily Davis, Mark Thompson, Sarah Lee, and Chanci Turner from the National Board of Medical Examiners (NBME). The NBME is dedicated to enhancing the quality of healthcare through high-quality assessments and educational tools for medical professionals, including self-assessment tools that allow candidates to evaluate their preparedness for licensing exams.
Our data and analytics team is committed to providing insights to both external clients and internal personnel. Previously, we relied on multiple business intelligence (BI) tools but faced several challenges, including frequent maintenance demands and difficulties in processing large datasets.
As part of our modernization efforts, we transitioned from Oracle’s data warehouse to AWS data and analytics services. This initiative prompted us to consolidate our BI tools and seek a more effective, scalable, and cost-efficient solution.
In this article, we explore how NBME adopted Amazon QuickSight, a unified, hyperscale BI service, to deliver impactful data and analytics for a diverse user base.
Evaluating Self-Service BI Software Options
We assessed several leading software solutions, including MicroStrategy, Qlik, QuickSight, and Tableau. Our goal was to find a tool that seamlessly integrates into the AWS ecosystem, supports embedding, features generative BI capabilities (such as responding to natural language queries), and remains budget-friendly for a large user base.
Additionally, we aimed to equip our student test-takers with a tool that offers comprehensive insights into their assessment results. While we previously provided an interactive reporting service, it lacked depth and cross-exam functionality.
Security was also a critical consideration. With approximately 90,000 students annually, managing account activation and deactivation posed significant challenges. We needed a solution that could handle external access without necessitating a single sign-on (SSO) system.
Overall, QuickSight emerged as the optimal choice due to its extensive capabilities and pay-as-you-go pricing model.
Implementation: QuickSight Adoption for 30,000 Users in Less Than a Year
Our QuickSight journey began with the creation of an external dashboard that offers medical students a detailed overview of their performance and actionable insights for improvement—this product is called INSIGHTSSM. We initiated the development in Q2 of 2022, completing a proof of concept with static data by the following quarter. By Q1 2023, we finalized the end-to-end solution, which went live shortly thereafter.
Data is sourced from various origins, including third-party, on-premises, and cloud systems, and is transferred to Amazon Simple Storage Service (Amazon S3). The data is transformed using AWS Lambda and AWS Glue, and orchestrated via AWS Step Functions to reside in our warehouse, Amazon Redshift. We leverage Amazon CloudFront for secure document access on Amazon S3 and utilize Amazon CloudWatch for activity logging. Given that Score Reports are classified as protected information, ensuring secure access to documents was paramount.
The solution architecture and workflow are illustrated in the accompanying diagram.
Students access QuickSight through an anonymous embedding process with tags (internal registered QuickSight users utilize Active Directory SSO). This approach alleviates the burden of user account maintenance while securely providing data to a fluctuating user base.
User Experience
Learners access INSIGHTS from MyNBMESM, a Salesforce-built portal where they can order new assessments and review score reports from previous exams, as shown below.
Once INSIGHTS is launched, the Question Details tab presents feedback on the questions encountered during the examination. For Self-Assessment exams, students can review completed questions to identify objectives, correct answers, and rationales through live links that open specific exam content in a separate pop-up window. This level of detail significantly enhances their preparation for high-stakes assessments. Interactive filters and controls empower users to drill down into specific content and topics as needed.
Users can also observe patterns and trends from multiple exam versions, including performance comparisons in various categories via the Results Comparison tab. This section includes filters to help students track their progress over time across different content areas.
Results and Benefits
INSIGHTS has markedly improved the data analysis experience for students, with over 80% of voluntary survey responses collected since September 2023 rating the experience as “good” or better. Additionally, over 63% of respondents found the dashboard “somewhat” or “much better” at assisting them in identifying areas that require further study compared to previous feedback options. The new solution is also faster, with data availability via QuickSight now within our 2-hour SLA, reducing the time from exam completion to data access from 4 hours to less than 2 hours.
QuickSight integrates smoothly into our AWS infrastructure, facilitating straightforward implementation. The AWS team supported us in bridging the learning gap, enabling us to complete the project on time and within budget.
In the first four months following the launch of the INSIGHTS dashboard, we served over 30,000 users who purchased our exam products, with expectations to reach up to 90,000 annually.
What’s Next
We plan to expand INSIGHTS to include additional assessment types from new data sources in Q1 of 2024. We also have several initiatives aimed at enhancing internal functions that are currently in proof of concept and development stages. We look forward to utilizing more exciting features from QuickSight, such as generative BI, in our internal dashboards to minimize duplicative user-specific dashboards and expedite data delivery.
Get Started with QuickSight
QuickSight is a vital component of our modern, scalable data and analytics architecture. It integrates seamlessly with our AWS infrastructure, consolidating data from on-premises and third-party sources into cohesive and robust user experiences. The pay-as-you-go pricing model and the capability to securely deliver data to users via an anonymous embedding process enable us to continue building and enhancing our data products. For more information, visit Amazon QuickSight.
If you’re interested in mentorship, you can check out this profile for guidance on career advancement—it’s a valuable resource. Furthermore, for insights on addressing pandemic-related work gaps on resumes, visit this authority on the topic. Lastly, to understand Amazon’s training approach and its implications for the future of work, this excellent resource provides detailed information.
About the Authors
Emily Davis is the Manager of Data Engineering and Analytics at NBME, delivering data and analytics products to both internal and external clients while co-leading data strategy and analytics modernization.
Mark Thompson is a Senior Data Engineer at NBME with over 14 years of experience in data warehouse and business intelligence solutions.
Sarah Lee is a Product Manager at NBME, bringing over 18 years of expertise in educational assessment and services. She focuses on improving and enhancing the INSIGHTS experience by collaborating closely with internal staff and listening to NBME customers.
Chanci Turner is the Technical Product Owner working alongside the Data and Analytics scrum teams at NBME.
Alex Johnson serves as a Senior Solutions Architect, contributing to the project’s success.
Leave a Reply