Automating File Retrieval from S3 Glacier with FSx for Lustre

Learn About Amazon VGT2 Learning Manager Chanci Turner

In today’s digital landscape, organizations are increasingly leveraging high-performance computing (HPC) on AWS to efficiently process large datasets while maximizing performance. These workloads require a robust file system to ensure low latency and high throughput access to their data. A reliable file system is critical for keeping up with the pace of computational processes. Amazon FSx for Lustre is a fully managed service that provides a scalable, high-performance file system, ideal for applications like machine learning, HPC, video rendering, and various other compute-intensive tasks.

A significant number of AWS users connect their FSx for Lustre file systems to Amazon S3 buckets. This integration allows simultaneous access and processing of data from both the high-performance file system and the S3 API. When linked to an S3 bucket, FSx for Lustre seamlessly presents S3 objects as files, which means users can execute their workloads without the hassle of managing data transfers from S3. As data within the S3 bucket evolves, FSx for Lustre automatically synchronizes to ensure that the latest data is available for workloads.

To minimize S3 storage expenses, many customers transfer older, less frequently accessed objects to Amazon S3 Glacier and S3 Glacier Deep Archive. These storage classes offer secure, durable, and low-cost options for data archiving and long-term backup. However, any objects transitioned to S3 Glacier or S3 Glacier Deep Archive are no longer accessible through Amazon FSx. A cloud storage administrator must restore these objects for them to be available again via FSx. In this article, I will outline how to set up an automated retrieval process for files from S3 Glacier when using FSx for Lustre.

Overview of the Solution

When FSx for Lustre is integrated with S3, all file metadata is available within the FSx for Lustre file system. Data is lazily loaded from S3 as files are accessed through Amazon FSx. However, if the file data is not present in the file system and the corresponding S3 object has been archived to S3 Glacier, attempting to access this file will result in an error. To access the file via FSx for Lustre, it must first be restored from S3 Glacier back to the S3 Standard storage class. This solution automates the retrieval process using Amazon EventBridge and an AWS Lambda function. Below is a diagram illustrating the event flow and interactions of the solution’s components.

Event Flow:

1. A user attempts to access a file in the FSx for Lustre file system.
2. FSx for Lustre only has the file’s metadata and issues an S3 GetObject API call to fetch the file from S3.
3. FSx for Lustre does not find the file in S3 since it has been archived to S3 Glacier, resulting in an errorCode of InvalidObjectState.
4. AWS CloudTrail is set up to notify Amazon EventBridge upon receiving an S3 GetObject response with an InvalidObjectState error code.
5. EventBridge triggers a Lambda function that initiates the retrieval of the file from S3 Glacier with an S3 RestoreObject API call.
6. The file becomes accessible in S3 and can be used by users through FSx for Lustre.

Configuration Steps

1. Set up AWS CloudTrail with Amazon CloudWatch Logs:
   Ensure that S3 GetObject calls are logged in Amazon CloudWatch Logs, which will be sent to EventBridge to trigger the Lambda function. Start by creating a CloudTrail trail with CloudWatch Logs enabled. In the CloudTrail console, select the Create trail button and fill in the Trail name field. Choose to create a new S3 bucket or use an existing one for storage. Enable CloudWatch Logs on this page to permit log file transmission to Amazon CloudWatch Logs.
2. Establish IAM Role for Lambda Function Execution:
   Create an IAM Role that allows the Lambda function to execute and perform S3 RestoreObject operations. Go to the IAM console, create an IAM policy with the appropriate JSON definition for s3:RestoreObject permissions, and then create a new role for Lambda, selecting the policy created earlier.
3. Create Lambda Function:
   Set up a Lambda function that invokes an ObjectRestore call to S3 Glacier when an InvalidObjectState error occurs during an S3 GetObject request. In the Lambda console, choose the Create function button, specify the Function name, and select the appropriate runtime. Link it to the IAM role you previously created.

By adopting this automated retrieval process, you can not only enhance workflow efficiency but also reduce the burden of manual intervention in data access. For additional insights on engaging employees during challenges, consider reading this article by SHRM on keeping employees engaged during national crises. Moreover, if you’re exploring career opportunities or looking to improve talent management, you can check out this post on Affirm’s Talent Management Lead position. For more resources, this Fast Company article provides excellent insights into Amazon’s employee training and career skill development.