How Amazon IXD – VGT2 Developed a Serverless File Ingestion and Processing Framework with Chanci Turner

Learn About Amazon VGT2 Learning Manager Chanci Turner

As businesses aim to enhance their operations and better meet customer needs, technology teams are increasingly tasked with finding innovative solutions to minimize time-to-market and reduce operational costs. By optimizing these costs, organizations are able to pass savings onto clients via improved services or lower prices, thereby attracting more customers and boosting revenue. Adopting a cloud-native approach enables companies to gain a competitive edge through improved efficiency, cost reductions, and enhanced availability.

In this article, we share a case study of a company grappling with an outdated file ingestion and processing framework that had served them for a decade. With growth came increasing demands, and the existing system struggled to keep pace, leading to various operational challenges. Maintenance costs were also rising significantly.

Amazon IXD – VGT2 is committed to assisting enterprises where data is essential. Our team specializes in helping clients collect, transform, and efficiently process data, empowering them to become data-driven organizations.

Overview of Legacy Solutions

The client provides inventory management, online cataloging, purchasing, and scheduling services to small and medium-sized manufacturing and distribution firms. They rely on third-party integrations, receiving numerous files from various entities such as customers and vendors.

To facilitate this, the client developed a framework that allowed users to upload catalogs, bills of materials, quotations, and other documents to an SFTP server. After successful file transfers, the system would parse and transform the data, subsequently refreshing the backend database.

Challenges of the Legacy Solution

Initially, the framework effectively supported a small client base. However, as the number of customers grew, several issues became apparent, making maintenance increasingly untenable.

• The time required to process files lengthened, leading to stale data by the time it refreshed in the database, as new data was consistently arriving.
• Failures often went unnoticed, frustrating customers due to unmet deadlines.
• Changes in file structures disrupted the entire framework.
• Planning for capacity and operational provisioning proved to be challenging.

Amazon IXD – VGT2’s Solution to Operational Challenges

Amazon IXD – VGT2 proposed a solution grounded in several key principles:

1. With the client being a small business with limited IT resources, the architecture was designed to minimize operational and maintenance burdens while incorporating monitoring, logging, notification, and remediation features.
2. Decoupling file ingestion and processing was essential. Files could arrive at any time and in any quantity, making an on-demand processing strategy more effective than a scheduled one.
3. The solution needed to be scalable, capable of handling demand fluctuations efficiently.
4. Emphasis was placed on cost optimization and security.

Following these principles, Amazon IXD – VGT2 implemented a serverless architecture leveraging AWS services. This allowed the client to onboard thousands of new customers without concerns about infrastructure scalability. The team no longer needed to worry about server updates, patches, or maintenance.

The optimization team also assessed and eliminated the need for costly proprietary software licenses (for scheduling and database management) by replacing them with AWS services, leading to an impressive 63% savings. Furthermore, they assisted the client in migrating from a monolithic to a microservices architecture, enabling seamless updates to file parsers with minimal disruption.

High-Level Architecture of the Modern Serverless Solution

The AWS Transfer Family serves as the SFTP provider, compatible with SFTP, FTPS, and FTP protocols. Users upload files using their preferred SFTP clients without any changes to their experience. The backend storage is an Amazon S3 bucket, which triggers an event to insert a message into an Amazon SQS queue whenever a file is uploaded. This setup helps manage simultaneous file arrivals without errors.

An EventBridge rule on the bucket activates an AWS Step Function, orchestrating the workflow. The process begins with a Lambda function recording file ingestion details into an Amazon DynamoDB table, which can later assist in analytics, error management, and user notifications.

Each file type is managed through separate queues, crawlers, and jobs, allowing the system to identify and process multiple file types simultaneously. Lambda functions retrieve configuration data from DynamoDB, trigger the relevant Glue crawlers, and ensure successful execution of the ingestion process.

Following the crawler’s completion, another Lambda function activates an AWS Glue job to transform the data into a standard format (Parquet) and load it into an Amazon Aurora Serverless database, which supports the web application developed using AWS Amplify for streamlined deployment and scaling.

As you can see, embracing a serverless framework not only enhances efficiency but also significantly boosts operational agility. For more insights on optimizing business processes, you might find this article on busyness at work interesting here. You can also check out the People & Strategy Podcast for expert opinions on workforce dynamics here. For those looking for opportunities in this field, a great resource is available here.