Maximizing Your Analytics Stack with Amazon Redshift | AWS Big Data Blog

Learn About Amazon VGT2 Learning Manager Chanci Turner

Analytics environments have rapidly expanded in terms of the volume of data stored, with the range of analytics use cases widening significantly. Data users are increasingly demanding immediate access to their data. This presents a challenge for IT teams: how to effectively scale infrastructure, maintain performance, and optimize costs while addressing these growing needs.

As a Senior Analytics Solutions Architect at AWS, I have the opportunity to understand these challenges firsthand and collaborate with our customers to optimize their architecture. Amazon Redshift frequently plays a crucial role in their analytics stack. It comes equipped with numerous built-in features that facilitate out-of-the-box performance, including automatic workload management, automatic ANALYZE, automatic VACUUM DELETE, and automatic VACUUM SORT. These tuning capabilities enable you to achieve the desired performance with fewer resources. Additionally, Amazon Redshift offers the Amazon Redshift Advisor, which continuously examines your cluster and delivers recommendations based on best practices. All you need to do is review and apply the recommendations that provide the most value.

In this post, we will explore some challenges that arise in your ever-evolving analytics environment and how to configure it to maximize the benefits of your analytics stack utilizing the latest innovations in Amazon Redshift.

Selecting the Right Hardware

The first consideration is ensuring you choose the most suitable node type for your workload. Amazon Redshift offers three families of node types: RA3, DC2, and DS2. The latest addition, RA3, was designed with compute and storage separation in mind, allowing for cost-effective scaling of storage and compute to accommodate most analytics workloads, making it the preferred choice for most users. If you are working with smaller datasets (less than 640 GB of compressed data) or have more intensive compute requirements, you might want to consider the DC2 nodes. The DS2 node type, while still available, is regarded as a legacy option. If you are currently using DS2 nodes, migrating to RA3 will optimize both costs and performance. One of the significant advantages of cloud computing is the ability to switch between node types easily. The elastic resize feature enables quick and efficient migration without the need for code changes, as all node types are compatible.

Each node type comes in various sizes. For instance, the RA3 family includes XLPlus, 4XLarge, and 16XLarge sizes; DC2 offers Large and 8XLarge sizes; while DS2 includes XLarge and 8XLarge sizes. The table below summarizes the allocated resources for each instance type as of December 11, 2020.

When determining the node size and number needed for your cluster, consider your processing requirements. For the node type you choose, begin with the smallest size and look to larger sizes as you approach the maximum number of nodes. For example, the RA3.XLPlus type can go up to 16 nodes. If you exceed that, consider scaling to six or more RA3.4XLarge nodes. Once you surpass 32 nodes of the RA3.4XLarge type, consider using 8 or more of the RA3.16XLarge type. The Amazon Redshift console provides a useful tool for sizing your cluster based on storage needs and workload.

Reserving Compute Resources

Establishing and managing a data warehouse environment requires substantial cross-functional effort, along with a significant investment of time and resources. Amazon Redshift offers considerable savings on the necessary hardware if you reserve your instances. After assessing your workloads and finalizing your configuration, consider purchasing Reserved Instances (RIs) to enjoy discounts ranging from 20% to 75% compared to on-demand pricing. RIs can be bought using Full Upfront, Partial Upfront, or sometimes No Upfront payment plans. These instances are not tied to a specific cluster but are pooled across your account, meaning that if your requirements grow and you want to expand your cluster, you can simply buy more Reserved Instances.

The chart below compares the annual on-demand cost of a Redshift cluster with the equivalent cost of a 1-year RI and a 3-year RI (example rates and discounts are based on one node of dc2.large with all upfront commitments in the us-east-1 Region as published on November 1st, 2020). If you utilize your cluster for more than 7.5 months in a year, you will save money with a 1-year RI. For usage exceeding 4.5 months on-demand, opting for a 3-year RI will yield even greater savings.

Managing Intermittent Workloads

For workloads that are accessed infrequently, Amazon Redshift allows you to pause and resume your cluster. While paused, you will only incur storage and backup costs. You can manage this feature through an API command, the Amazon Redshift console, or a scheduler. This is particularly beneficial if you are using Amazon Redshift as a compute engine that retrieves data from the Amazon Simple Storage Service (Amazon S3) data lake and unloads processed results back to the data lake. In such scenarios, the cluster only needs to operate during the data curation phase. Another practical application for this feature is when the cluster is only necessary during data pipeline executions or for refreshing the in-memory storage of a reporting platform.

When deciding to pause and resume your cluster, remember the cost benefits associated with Reserved Instances. The chart below illustrates the daily on-demand cost of an Amazon Redshift cluster relative to the cost of a 1-year RI and a 3-year RI divided by the number of days in the RI (example rates are based on one node of dc2.large with all upfront commitments in the us-east-1 Region as published on November 1st, 2020). If you require your cluster for more than 15 hours in a day, you will find savings with a 1-year RI. For usage exceeding 9 hours in a day, a 3-year RI provides even more significant savings.

Managing Data Growth with RA3

As use cases expand, the demand for data within the analytics environment grows. Often, data growth outpaces the compute requirements. In traditional MPP systems, managing data growth would involve adding nodes, archiving old data, or deciding which data to retain. With Amazon Redshift RA3 and managed storage, primary storage is backed by Amazon S3 rather than being confined to compute nodes, allowing for much greater storage flexibility.

For further insights on managing workplace stress, you can refer to SHRM’s resources. Additionally, check out this YouTube video for an excellent resource. If you’re interested in exploring opportunities, this blog post could be engaging.