Scaling Container Workloads with Shared Storage for Red Hat OpenShift Service on AWS

Kubernetes has emerged as a favored platform for developers deploying applications, often requiring a persistent shared storage layer. The Red Hat OpenShift Service on AWS (ROSA) is a managed integration of OpenShift on AWS, developed by Red Hat and supported jointly by AWS and Red Hat. Typically, ROSA clusters utilize locally attached Amazon Elastic Block Store (EBS) volumes; however, certain customers require their data to be persistent and accessible across multiple containers, including those distributed across various Availability Zones (AZs). These users seek a storage solution that auto-scales and provides a uniform interface for workloads across both on-premises and cloud environments.

ROSA integrates seamlessly with Amazon FSx for NetApp ONTAP, a fully managed shared storage service built on the NetApp ONTAP file system. With FSx for ONTAP, users gain access to essential features such as snapshots, FlexClones, cross-region replication via SnapMirror, and a highly available file server with smooth failover capabilities. This service is paired with the NetApp Trident driver, a dynamic Container Storage Interface (CSI) that manages Kubernetes Persistent Volume Claims (PVCs) across storage disks. The Trident CSI driver facilitates the on-demand provisioning of storage volumes in diverse deployment environments, simplifying the scaling and protection of application data.

In this article, we will illustrate how to use FSx for ONTAP as a persistent storage solution for ROSA applications. We will guide you through the installation of the NetApp Trident CSI driver on a ROSA cluster, provisioning an FSx for ONTAP file system, deploying a sample stateful application, and demonstrating pod scaling across multi-AZ nodes using dynamic persistent volumes. Additionally, we will discuss backup and restoration methods for your application. With this setup, you can create a shared storage solution that scales across different AZs, making data management easier and more reliable using the Trident CSI driver. For further insights, you might want to check out another blog post here.

Solution Overview

The diagram illustrates the deployment of the ROSA cluster across multiple AZs. The master nodes, infrastructure nodes, and worker nodes of the ROSA cluster operate within a private subnet of the customer’s VPC. An FSx for ONTAP file system will be established within the same VPC, and the Trident driver will be installed in the ROSA cluster, enabling all subnets of this VPC to connect to the file system.

Prerequisites

To get started, you will need the following resources:

An AWS account
A Red Hat account
An IAM user with adequate permissions to create and access the ROSA cluster (refer to the ROSA workshop for more information)
AWS CLI installed according to your OS
ROSA CLI installed based on your OS
OpenShift command-line interface (oc)
Helm 3 documentation
A ROSA cluster already created (refer to this ROSA workshop)
Access to the Red Hat OpenShift web console

Walkthrough

Clone the GitHub repository.
Provision the FSx for ONTAP file system using CloudFormation.
Install and configure the Trident CSI driver for the ROSA cluster.
Deploy a sample MySQL stateful application on the ROSA cluster.
Scale MySQL application pods across multiple Availability Zones.
Backup and restore volumes in the FSx for ONTAP file system.

1. Clone GitHub Repository

You will need Git to clone the GitHub repository: https://github.com/aws-samples/rosa-fsx-netapp-ontap. If you do not have Git installed, you can use the following command:

sudo yum install git -y

Clone the repository:

git clone https://github.com/aws-samples/rosa-fsx-netapp-ontap.git

2. Provision FSx for ONTAP

We will create a multi-AZ FSx for ONTAP file system within the same VPC as the ROSA cluster. Take note of the VPC ID and the two Subnet IDs that correspond to the subnets you want your file system in, along with all route table IDs associated with the ROSA VPC subnets. Enter these values in the command below. The FSxAllowedCIDR parameter is designated for the FSx for ONTAP security group ingress rules to control access. You can use 0.0.0.0/0 or any suitable CIDR to allow all traffic to access the specific ports of FSx for ONTAP. Run the command in your terminal to create the FSx for ONTAP file system.

Note: If you want to provision the file system with different storage capacity and throughput, you can override the default values by adjusting the StorageCapacity and ThroughputCapacity parameters in the CFN template.

cd rosa-fsx-netapp-ontap/fsx

aws cloudformation create-stack 
--stack-name ROSA-FSXONTAP 
--template-body file://./FSxONTAP.yaml 
--region <region-name> 
--parameters 
ParameterKey=Subnet1ID,ParameterValue=[subnet1_ID] 
ParameterKey=Subnet2ID,ParameterValue=[subnet2_ID] 
ParameterKey=myVpc,ParameterValue=[VPC_ID] 
ParameterKey=FSxONTAPRouteTable,ParameterValue=[routetable1_ID,routetable2_ID] 
ParameterKey=FileSystemName,ParameterValue=ROSA-myFSxONTAP 
ParameterKey=ThroughputCapacity,ParameterValue=256 
ParameterKey=FSxAllowedCIDR,ParameterValue=[your_allowed_CIDR] 
ParameterKey=FsxAdminPassword,ParameterValue=[Define password] 
ParameterKey=SvmAdminPassword,ParameterValue=[Define password] 
--capabilities CAPABILITY_NAMED_IAM

Verify your file system and storage virtual machine (SVM) has been created using the Amazon FSx console.

3. Install and Configure the Trident CSI Driver for the ROSA Cluster

We will install the Trident CSI driver in the OpenShift “trident” namespace. To create the “trident” namespace, open a command line interface (CLI) on your computer and log in to the ROSA cluster using the OpenShift CLI (oc) tool.

oc create ns trident

Next, download the Trident CSI driver from Git:

curl -L -o trident-installer-22.10.0.tar.gz https://github.com/NetApp/trident/releases/download/v22.10.0/trident-installer-22.10.0.tar.gz

Then, extract the contents:

tar -xvf ./trident-installer-21.10.1.tar.gz

Utilize the Helm command to install the Trident CSI driver in the “trident” namespace on the OpenShift cluster.

cd trident-installer/helm
helm install trident -n trident trident-operator-22.10.0.tgz

Run the command below to verify the installation of the Trident driver.

helm status trident -n trident

For an excellent resource on this topic, you can check out this video here.

3.1 Create Secret to Store SVM Username and Password in the ROSA Cluster

Create a new file containing the SVM username and admin password, saving it as svm_secret.yaml. A sample svm_secret.yaml file is included in the fsx folder.

apiVersion: v1
kind: Secret
metadata:
  name: backend-fsx-ontap-nas-secret
  namespace: trident
type: Opaque
stringData:
  username: vsadmin
  password: step#2 password

Note: You can retrieve the SVM username and admin password from the AWS Secrets Manager console. Add the secrets to the ROSA cluster with the following command:

oc apply -f svm_secret.yaml