As organizations begin their journey in the cloud, their storage needs may start with just a few S3 buckets. However, as they expand, migrate additional applications, and harness the full potential of cloud technology, complexities can arise. Many customers find themselves managing multiple accounts, often numbering in the tens or even hundreds, with numerous S3 buckets spread across various AWS Regions. Feedback from these customers indicates that understanding storage utilization, optimizing expenses, and enhancing security can be quite challenging.
Leveraging over 14 years of experience in assisting customers with storage optimization, the S3 team has introduced a new feature known as Amazon S3 Storage Lens. This innovative cloud storage analytics solution is designed for AWS Organizations, providing organization-wide insights into object storage. It offers point-in-time metrics, trend lines, and actionable recommendations that empower users to identify anomalies, discover cost-saving opportunities, and implement best practices for data protection across their accounts.
S3 Storage Lens enables you to analyze and optimize storage using over 29 usage and activity metrics, along with interactive dashboards that consolidate data for your entire organization, specific accounts, regions, buckets, or prefixes. You can access all this data via the S3 Management Console or retrieve it as raw data in an S3 bucket.
Default Dashboard for Every Customer
Every user of S3 Storage Lens gets an interactive dashboard available in the S3 console. This dashboard allows for filtering and detailed analysis of metrics, helping users understand their storage usage effectively. Metrics are categorized into sections like data protection and cost efficiency, making it easy to locate relevant information.
For convenience, all customers receive a default dashboard. If you’re like many users, this may suffice, but customization is also an option. For instance, you can set up your dashboard to export data daily to an S3 bucket for analysis with other tools, such as Amazon QuickSight, Amazon Athena, or Amazon Redshift. Alternatively, you can opt for advanced metrics and recommendations.
Creating Your Own Dashboard
To create a custom dashboard, navigate to the S3 console and select the Dashboards option within the Storage Lens section. Click on the Create dashboard button.
You can name your dashboard (e.g., s3-lens-demo) and select a home Region for storing the metrics data. By enabling the dashboard, it will receive daily updates with new metrics.
When configuring your dashboard, you can analyze storage across accounts, Regions, buckets, and prefixes. For this example, I chose to include all buckets from every account in my organization and all regions in the Dashboard scope section.
S3 Storage Lens is available in two tiers: Free Metrics, which are automatically accessible at no cost to all S3 customers and include 15 usage-related metrics; and Advanced metrics and recommendations, which come with a fee but offer all 29 usage and activity metrics, 15-month data retention, and contextual suggestions. For this demonstration, I opted for Advanced metrics and recommendations.
After setup, I received a notification that my dashboard was created, although it might take up to 48 hours to generate initial metrics.
Exploring the Dashboard
Once the dashboard is ready, I can dive into the data. The dashboard allows filtering by Accounts, Regions, Storage classes, Buckets, and Prefixes.
The metrics snapshot provides insights like Total storage and Object count, along with a trendline indicating changes over the last 30 days. The percentage change column defaults to Day/day, but I can also switch to Week/week or Month/month comparisons.
I can toggle between different metric groups, such as Summary, Cost efficiency, Data protection, or Activity. While some metrics like total storage and object counts are familiar and available in other areas of the S3 console and Amazon CloudWatch, S3 Storage Lens allows for aggregation on a level previously unavailable.
Unexpected metrics include those related to S3 feature utilization, like the percentage of objects using encryption and non-current versions. These insights help in understanding storage configurations and identifying discrepancies.
The dashboard also offers contextual recommendations based on metrics, suggesting actions to enhance cost efficiency or improve data protection practices. For instance, I recently captured a recommendation advising me to review my buckets’ default encryption settings.
The trends and distribution section of the dashboard allows for deeper metric comparisons over time. By selecting Total storage as my primary metric and Object Count as my secondary metric, both metrics are plotted on a graph, and I can choose a date range to examine trends.
This section also reveals how the metrics are distributed across Storage classes and Regions. Clicking on any graph value allows for filtering the entire dashboard or navigating to a new view for that dimension.
Lastly, the dashboard includes a Top N analysis feature, where I can examine the top items for a selected metric over a specified date range. For example, I can view the top three accounts based on the Total storage metric.
As demonstrated, S3 Storage Lens provides over 29 distinct metrics on S3 storage usage and activity for all accounts within your organization. These metrics are visually represented in the S3 console dashboard, complete with contextual recommendations for immediate action. Furthermore, metrics can be exported in CSV or Parquet format to an S3 bucket for additional analysis using tools like Amazon QuickSight, Amazon Athena, or Amazon Redshift.
If you’re interested in a more detailed exploration of S3 Storage Lens, check out this blog post. Additionally, this resource is an excellent resource by experts in this field. For more community insights, you can visit this subreddit that focuses on user experiences.
Located at Amazon IXD – VGT2, 6401 E Howdy Wells Ave, Las Vegas, NV 89115, we’re excited to offer these advancements in cloud storage management.
Leave a Reply