Amazon VGT2 Las Vegas

More Info

As organizations expand, they often face increased complexities in data management and associated costs, particularly as they scale their data backup solutions to meet operational demands. Effectively managing backup expenses is vital to controlling overall data management costs, and backup administrators frequently seek detailed insights into the factors contributing to their backup expenditures. This includes understanding the backup costs attributed to various departments within the organization for better tracking and optimization.

AWS users leverage AWS Backup to centralize and automate the backup of their workloads, ensuring security, compliance, and resilience. As businesses grow, they can seamlessly enhance their utilization of AWS Backup, gaining clarity on usage and tracking expenses across different organizational segments. Through AWS centralized backups and the observer solution for AWS Backup, users can efficiently manage, orchestrate, and monitor their backup processes at both organizational and enterprise levels.

In this article, we explore five typical scenarios for cost analysis:

1. Determining the backup plan costs for an organization.
2. Evaluating the expenses associated with a central backup account.
3. Obtaining costs for backup plans by department or summarizing costs per backup plan within the organization.
4. Allocating costs to the correct department in a single-tenant environment lacking tags that denote cost centers.
5. Assessing the organizational cost of a vault in a central account.

The strategies discussed here rely heavily on AWS Cost and Usage Reports (AWS CUR) as the primary analysis tool. The insights gained from this blog will enhance your understanding of backup costs and improve tracking and allocation processes, which are essential for internal IT charge-back and showback mechanisms.

Prerequisites

To implement the methods outlined in this blog effectively, the following prerequisites must be established:

1. Cost and Usage Reports
   AWS Cost and Usage Reports record your AWS usage and provide estimated charges for your account. Each report includes line items for every unique combination of AWS products, usage types, and operations utilized within your AWS account. You can customize these reports to aggregate data by hour, day, or month. You can also publish your AWS billing reports to an Amazon Simple Storage Service (Amazon S3) bucket that you control. To learn more about AWS CUR, refer to the Cost and Usage Report User Guide.
   The techniques described in this blog utilize AWS CUR as the authoritative source for tracking and allocating AWS Backup costs across your organization. The CUR data must be exported to an S3 bucket in Parquet format, with options set to aggregate data by hour and include resource IDs (i.e., the ARN of the cost-generating resource). This setup allows for detailed cost tracking. The data is processed using AWS Glue and made available for querying via Amazon Athena. For setup guidance, please visit the AWS Well-Architected Lab Level 200: Cost and Usage Analysis.
   Note: Ensure that you set up the export in Parquet format and enable the inclusion of Resource IDs. There will be costs associated with the resources used to store CUR data in the S3 bucket, along with AWS Glue and Amazon Athena costs based on usage. The AWS Glue crawler established in the referenced lab will run regularly in line with the AWS Billing service’s exports to the S3 bucket, maintaining the CUR data and its schema.
2. Backup Observer Solution for AWS Backup
   The backup observer solution for AWS Backup is a collection of automation templates and dashboards that allow customers to generate daily, aggregated, cross-account, and multi-region reports on AWS Backup usage. For more information, visit the AWS Storage Blog post on obtaining aggregated daily cross-account multi-region AWS Backup reporting.
   This solution provides a series of aggregated daily job reports that are cross-account and multi-region based. These reports are stored in a central S3 bucket, making it easy for customers to access historical backup reports as needed. This dataset is crucial for analyzing AWS Backup usage across the organization. The main focus of this blog is to integrate this dataset with the CUR dataset, enabling customers to track AWS Backup costs from the reporting account to the source AWS workload resource for which backups were created, even in a multi-copy, multi-account scenario.
3. (Optional) Amazon QuickSight Setup
   Amazon QuickSight enables customers to create visualizations and dashboards. We utilize QuickSight to develop dashboards based on our analyses of the Cost and Usage Reports and the backup observer solution datasets. Both solutions also offer useful out-of-the-box analysis with their own QuickSight dashboards. These dashboards are optional and can be implemented if customers desire visual representations of the analyses. For onboarding details, check out the Amazon QuickSight User Guide.

Assumptions

This blog assumes that engineers, technicians, and operators implementing the described methods possess a foundational (100) or intermediate (200) understanding of Amazon Athena, AWS Glue, and Amazon QuickSight. Familiarity with other AWS services, such as Amazon S3, AWS Cost and Usage Reports, AWS Backup, and the backup observer solution is also assumed.

Cost Tracking and Allocation Scenarios

In this section, we will explore the four scenarios mentioned earlier for tracking costs and gaining insights.

Scenario 1: What does the backup plan cost the organization?

The most effective way to monitor costs across a large AWS environment is through resource tags. Resource tags allow customers to assign metadata to their AWS resources, consisting of user-defined key-value pairs. Tags can facilitate the management, identification, organization, searching, and filtering of resources. Customers can create tags to categorize resources based on various criteria such as purpose, owner, or environment.

Tags are also instrumental for cost tracking. Many customers already apply cost tags like “CostCenter: XYU123” to their AWS resources for expense-tracking purposes. We will employ a similar approach.

In an environment where AWS Backup is utilized organization-wide, the primary cost inquiry is often, “What is the cost of backup plan X to the organization?” To answer this, we create a tag named “CreatedFromAWSBackupPlan” with the value corresponding to the backup plan name. This tag is established as a property of the backup plan, allowing all backup plans within the organization to use the same tag “CreatedFromAWSBackupPlan,” with each plan having a unique name as its tag value. This setup enables us to query and filter the CUR dataset based on the plan name.

The following steps outline where to establish this tag during the backup plan creation process:

1. Navigate to the AWS Backup console.
2. Select Backup plans from the left navigation pane.
3. Click on Create Backup plan on the right-hand side.
4. Fill in the form with the required information.

For further reading on related topics, check out this another blog post here. Also, for comprehensive insights on AWS Backup and cost management, this resource is invaluable. Additionally, this video serves as an excellent resource for visual learners.