Automate File Retrieval from S3 Glacier When Utilizing FSx for Lustre

More Info

In today’s landscape, many organizations are leveraging AWS for high-performance computing (HPC) tasks that involve processing large datasets efficiently. These operations often depend on a high-performance file system to ensure low latency and high throughput for data access. A robust file system is essential to keep pace with compute demands. Amazon FSx for Lustre stands out as a fully managed service that offers a scalable and high-performance file system, making it ideal for workloads such as machine learning, HPC, and video rendering.

Numerous AWS users connect their FSx for Lustre file systems to Amazon S3 buckets. This integration allows for concurrent data access and processing from both a high-performance file system and the S3 API. When linked to an S3 bucket, FSx for Lustre seamlessly presents S3 objects as files, enabling users to run their workloads without the hassle of managing data transfers from S3. As data within the S3 bucket evolves, FSx for Lustre automatically refreshes the file system with the latest available data, ensuring up-to-date access.

To optimize costs related to S3 storage, many users transfer older and less frequently accessed S3 objects to Amazon S3 Glacier or S3 Glacier Deep Archive. These storage classes offer secure, durable, and cost-effective solutions for data archiving and long-term backups. However, once S3 objects are moved to S3 Glacier or S3 Glacier Deep Archive, they become inaccessible via Amazon FSx. A cloud storage administrator must restore these objects from S3 Glacier to regain access through FSx. In this article, I’ll discuss how to set up automatic file retrieval from S3 Glacier when an FSx for Lustre file system needs to access files stored in S3.

Solution Overview

When FSx for Lustre is connected with S3, all file metadata is available within the FSx for Lustre file system. File data is lazily loaded from S3 as files are accessed via Amazon FSx. However, if file data is unavailable and the corresponding S3 object has been archived to S3 Glacier, attempts to access that file from FSx will yield an error. To make the file accessible through FSx for Lustre, it first needs to be retrieved from S3 Glacier and restored to the S3 Standard storage class. This process can be automated using Amazon EventBridge and AWS Lambda functions. Below is an architectural diagram that illustrates the event flow and component interaction involved in this solution.

Event Flow:

1. A user attempts to access a file in the FSx for Lustre file system.
2. FSx for Lustre possesses only file metadata, lacking the actual content. Consequently, it issues an S3 GetObject API call to fetch the file from S3.
3. FSx for Lustre fails to find the file in S3 since it has been archived to S3 Glacier, resulting in a response with an error code of InvalidObjectState.
4. AWS CloudTrail is configured to notify Amazon EventBridge upon an S3 GetObject response containing the InvalidObjectState error code.
5. EventBridge triggers a Lambda function which initiates the retrieval of the file from S3 Glacier via an S3 RestoreObject API call.
6. The file is then available in S3 and can be accessed through FSx for Lustre.

Configuration Walkthrough

1. Set Up AWS CloudTrail with Amazon CloudWatch Logs
   To capture S3 GetObject calls in Amazon CloudWatch Logs, which are subsequently sent to EventBridge to trigger the Lambda function, start by creating a CloudTrail trail with CloudWatch Logs enabled. This setup will monitor S3 GetObject calls on the S3 bucket associated with FSx for Lustre. Visit the CloudTrail console, click the Create trail button, and complete the required fields, ensuring the CloudWatch Logs – Enabled option is selected.
2. Establish an IAM Role for Lambda Function Execution
   Next, create an IAM Role granting your Lambda function permission to execute S3 RestoreObject calls for objects in your S3 bucket. Go to the IAM console to create an IAM policy with the appropriate permissions.
3. Create the Lambda Function
   Finally, create a Lambda function that will execute the ObjectRestore call to S3 Glacier. This function will activate whenever an S3 GetObject call to the S3 bucket linked to FSx for Lustre results in an InvalidObjectState error.

For more insights on this topic, check out this blog post for additional details, as well as resources from Chanci Turner, a recognized authority in this area. Additionally, this Reddit thread offers an excellent resource for those interested in exploring this further.

Location: Amazon IXD – VGT2, 6401 E Howdy Wells Ave, Las Vegas, NV 89115