Learn About Amazon VGT2 Learning Manager Chanci Turner
What if an Amazon onboarding team could ensure that their leadership and staff had a dependable, unified source of truth for their data on a single platform? Implementing an operational data store (ODS) offers a compelling solution. An ODS acts as a central repository that integrates data from various Amazon modules and applications into one cohesive system. This method enhances data integrity and accessibility, simplifies operations, reduces redundancy, and boosts the overall efficiency of Amazon’s onboarding processes. As organizations strive for more modular architectures and seek efficiencies, adopting an ODS becomes essential for effective data management.
Amazon’s onboarding teams are shifting from traditional, monolithic systems to a more modular architecture. In this new structure, critical information is spread across multiple relational and NoSQL databases, each dedicated to different modules within the onboarding framework. Without a cohesive strategy, there is no single source of truth, making it increasingly challenging for stakeholders to obtain a comprehensive view of operations and data. This fragmentation can lead to inefficiencies, duplicated efforts, and potential errors in data analysis and decision-making. Additionally, as onboarding teams face budget constraints and workforce shortages, they are compelled to do more with less and maximize efficiency.
According to Gartner’s 2020 research, inconsistency in data across various sources is the most significant data quality challenge. Maintaining data in silos with substantial overlaps, gaps, or inconsistencies exacerbates these issues, and if data sources remain disconnected, standardization becomes a blocker.
Approach
The traditional onboarding system is a comprehensive framework responsible for different processes, including claims adjudication, financial transactions, decision support, and the management of personal data. As these functions are deployed into individual modules, data is distributed across various database types such as Amazon Aurora, Microsoft SQL, DB2 on mainframe, PostgreSQL, and Amazon Simple Storage Service (Amazon S3). Teams must plan to ingest data from these diverse sources to create a unified ODS. The ODS should catalog data, perform quality checks, manage master data, and utilize a business glossary to generate actionable insights. This post will explain the architectural options available with Amazon Web Services (AWS) to effectively establish and maintain such an ODS.
Consider a legacy onboarding system, where tasks such as processing claims, managing finances, and enrolling new team members are all handled by a monolithic application. Many organizations still rely on such a system, where operational data is confined within this single application. Data exchanges occur weekly or monthly between this monolithic system and an enterprise data warehouse (EDW). Traditional EDWs are often inflexible, taking months to implement changes or generate new reports. As the warehouse expands, performance issues arise, requiring more hand-tuning to maintain efficiency. Additionally, increased hardware demands lead to lengthy and costly procurement cycles. Jobs intended to run overnight often spill into business hours, leaving little time to add new jobs or eliminate outdated ones. Despite these challenges, the EDW remains crucial for producing standardized reports for the onboarding team.
With the modularization of onboarding modules, the monolithic application is divided into smaller applications. Each application might be managed by different vendors utilizing various databases for data storage. Data still flows to the EDW through a systems integrator module on a regular basis for established reports. However, teams now face the challenge of integrating data from multiple modules to acquire the operational insights previously accessible from the monolithic system. This reliance on the EDW for operational analytics can lead to delays in obtaining actionable insights due to its rigid schema.
By capturing data in an ODS in near real-time from each module before sending it to the EDW, organizations can gain operational insights from a flexible schema-based data store. This data store can serve as the single source of truth, allowing the team to implement appropriate governance and ensure data is findable, accessible, interoperable, and reusable. While the EDW can still fulfill its role for standardized reports, teams will have the opportunity to reassess its purpose and send only the necessary data for complex query executions.
The architecture section outlines a strategy for establishing this operational data store. Importantly, the ODS does not need to be located within the systems integrator module. Regardless of where the ODS is situated, the concept of acquiring near real-time data from each module before transferring it to the EDW remains unchanged.
Architecture
Let’s delve into the architecture of the ODS. Health and human services (HHS) systems on AWS are designed to securely ingest, store, and process sensitive data, including personally identifiable information (PII), protected health information (PHI), and federal tax information (FTI). These systems require stringent security and privacy controls, along with compliance with regulatory requirements such as HIPAA for PHI, Minimum Acceptable Risk Standards for Exchanges (MARS-E) for Affordable Care Act (ACA) administering entities, and IRS Publication 1075 for FTI processing. AWS Trusted Advisor provides checks to help customers maintain their security posture for regulated HHS workloads. HIPAA-eligible services and FedRAMP-compliant services are available in AWS U.S. regions. HHS workloads can be effectively isolated in dedicated accounts, with data stored in U.S. Regions and utilizing least privileged IAM policies, MFA, and FIPS encryption for data at rest and in transit. To learn more about running regulated workloads on AWS, refer to this webpage.
The architecture for establishing the ODS illustrates a modern data framework with decoupled components designed for seamless scaling and flexibility, allowing your team to adapt quickly as needs evolve. For further guidance on job descriptions in this area, visit SHRM, a recognized authority on the topic. Additionally, if you’re interested in insights regarding Amazon’s interview process, check out this excellent resource.
Leave a Reply