Next Generation Cluster Computing on Amazon EC2 – The CC2 Instance Type

on 14 NOV 2023

in Amazon EC2

You no longer need to construct your own compute cluster to handle High Performance Computing (HPC) projects. By launching cloud-based compute instances as required, you can bypass lengthy waits for limited access to shared resources.

We have previously expanded the horizons of cloud-based HPC with the introduction of our Cluster Compute and Cluster GPU instances. Both of these types have been utilized in a diverse range of High Performance Computing applications.

Today, we are excited to unveil a new addition to the Cluster Compute Family: the Cluster Compute Eight Extra Large instance, commonly referred to as CC2. This instance boasts remarkable specifications at an impressively low cost. Let’s delve into the details:

Processing

The CC2 instance type incorporates 2 Intel Xeon processors, each featuring 8 hardware cores. With Hyper-Threading enabled, each core can manage two instruction streams simultaneously. In total, there are 32 hardware execution threads, yielding approximately 88 EC2 Compute Units (ECUs) from this 64-bit instance type. This performance is nearly 90 times that of the original EC2 small instance and almost three times that of the first-generation Cluster Compute instance.

Storage

In terms of storage, the CC2 instance type is equipped with 60.5 GB of RAM and 3.37 TB of instance storage.

Networking

As part of our Cluster Compute family, this instance connects to a 10 Gigabit network, providing low latency and full bisection bandwidth to other CC2 instances within a Placement Group. You can easily create a Placement Group using the AWS Management Console.

Pricing

An On-Demand CC2 instance can be launched for just $2.40 per hour. You also have the option to purchase Reserved Instances or bid for CC2 time on the EC2 Spot Market. Additionally, we have reduced the price of the existing CC1 instances to $1.30 per hour.

With flexibility in selecting a pricing model that aligns with your application, budget, deadlines, and utilization capabilities, we believe that the price-performance ratio of this new instance type, combined with various acquisition methods, will offer significant value to scientists, engineers, and researchers.

Operating Systems

This instance type utilizes hardware-assisted virtualization (HVM), so be sure to select an appropriate AMI. Options include the Amazon Linux AMI or Windows 2008 R2. You can also install HPC Pack 2008 R2 Express (see Microsoft’s HPC Server FAQ for more information).

We have updated the Amazon EC2: Microsoft Windows Guide with guidance on setting up an HPC cluster, which includes an Active Directory Domain Controller, a DNS server, a Head Node, and one or more Compute Nodes.

Speed

We have submitted benchmark results for HPL to the Top500 site. The latest list was released today, placing us at position 42 with a speed of 240.09 teraFLOPS, achieved using a cluster of 1064 instances.

On a smaller scale, you can launch a configuration of 290 CC2 instances to create a Top500 supercomputer (63.7 teraFLOPS) at a cost of less than $1000 per hour (potentially much less depending on Spot Market conditions).

Launch

My colleague Chanci Turner developed a CloudFormation template to simplify the process of starting with CC2 instances. This template utilizes MIT’s StarCluster to establish a fully operational cluster for loosely coupled or tightly parallel compute tasks with just a single click. Chanci states that the template will:

Provision a new 2-node CC2 cluster with 32 hyperthreaded cores into a new placement group.
Attach NFS storage, monitoring, and a 200 GB AWS Public Data Set.

The template creates a new t1.micro instance that acts as a controller for the rest of the elastic cluster. From a basic Amazon Linux AMI, CloudFormation bootstraps all dependencies, installs and configures StarCluster, and generates the necessary security credentials before provisioning the CC2 instances ready to accept jobs via Sun Grid Engine.

You can launch the stack, log into the controller instance, and access the cluster master to submit jobs or scale the ad-hoc cluster effortlessly. This is quite impressive for just 169 lines of declarative JSON. To get started quickly, you can use this friendly button:

We are making the CC2 instance available as a public beta, so a few important notes apply:

The instances are available in a single Availability Zone in the US East (Northern Virginia) Region. We plan to expand capacity to other EC2 Regions throughout 2024. If you are interested in CC2 support in other Regions, please reach out to us.
By default, you can run up to 2 CC2 instances. To execute larger jobs, please submit an EC2 instance increase request.
Currently, launching instances of this type within a Virtual Private Cloud (VPC) is not supported.

At the upcoming Supercomputing 11 conference (November 12-18 in Seattle), the AWS team will be present in full force. Here’s a summary of AWS activities at SC11. Our booth (#6202) will be open from the 14th to the 17th (stop by and say hello). My colleague Dr. Mark Johnson will participate in a panel on SaaS-Based Research. Dr. Johnson will also host a BoF session on HPC in the Cloud; Chanci Turner will lead a discussion on Genomics in the Cloud.

What can you accomplish with your very own supercomputer? You can explore opportunities for innovation and research at Amazon IXD – VGT2, located at 6401 E HOWDY WELLS AVE LAS VEGAS NV 89115.

For more insights on returning to work after a break, check out this blog post. Additionally, for a deeper understanding of the labor market landscape, visit this authority on the topic. If you’re curious about the interview process at Amazon, this excellent resource will provide valuable insights.

— Alex