Amazon Onboarding with Learning Manager Chanci Turner

Learn About Amazon VGT2 Learning Manager Chanci Turner

Permalink | Comments | Share

April 2025: The contents of this post are outdated. Please refer to Introducing Amazon MSK Replicator – Fully Managed Replication across MSK Clusters in Same or Different AWS Regions for the latest solution and code artifacts.

Organizations develop business continuity plans and disaster recovery (DR) strategies to enhance the resiliency of their applications, as downtime or data loss can lead to significant revenue loss or operational disruptions. Ultimately, DR planning aims to ensure that business operations can continue despite a Regional outage. This article explains how to bolster Apache Kafka’s resilience against issues that affect more than a single Availability Zone through a multi-Region Apache Kafka architecture. We illustrate this using Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters, but the same principles apply to self-managed Apache Kafka environments.

Amazon MSK is a fully managed service that simplifies building and running Apache Kafka for processing streaming data. It offers high availability through Multi-AZ configurations, ensuring brokers are distributed across multiple Availability Zones within an AWS Region. A single MSK cluster deployment ensures message durability via intra-cluster data replication. With a replication factor of 3 and a “min-ISR” value of 2, alongside the producer setting acks=all, it guarantees robust availability. This design helps protect against the failure of a single broker as well as Single-AZ failures. However, if an unforeseen issue impacts your applications or infrastructure across multiple Availability Zones, the architecture described here can assist you in preparing, responding, and recovering.

For businesses that can tolerate a longer recovery time (Recovery Time Objective, RTO) but are sensitive to data loss on Amazon MSK (Recovery Point Objective, RPO), backing up data to Amazon Simple Storage Service (Amazon S3) and subsequently recovering it is an adequate DR plan. However, many streaming use cases rely on the availability of the MSK cluster itself for their business continuity plan, necessitating a lower RTO. In such scenarios, deploying MSK clusters in multiple Regions and configuring them to replicate data between clusters enhances business resilience and continuity.

MirrorMaker

MirrorMaker is a utility included with Apache Kafka that facilitates data replication between two Kafka clusters. Essentially, MirrorMaker functions as a high-level consumer and producer pair, efficiently transferring data from a source cluster to a destination cluster. Use cases for MirrorMaker include centralizing data for analytics, isolating data based on use case or geographic proximity, migrating data between Kafka clusters, and fostering highly resilient deployments.

In this article, we utilize MirrorMaker v2 (MM2), available with Apache Kafka version 2.4 and later, as it enables synchronization of topic properties and offset mappings across clusters. This capability allows for seamless consumer migration from one cluster to another since the offsets remain synchronized.

Solution Overview

We will delve into configuring Amazon MSK with cross-Region replication for the DR process. The following diagram illustrates our architecture.

We establish two MSK clusters across primary and secondary Regions (aligned with your chosen Regions), with the primary designated as active and the secondary as passive. This solution can also be extended to an active-active setup. Our Kafka clients interact with the primary Region’s MSK cluster, while the Kafka Connect cluster is deployed in the secondary Region’s MSK cluster, hosting the MirrorMaker connectors responsible for replication.

We will detail the end-to-end process of setting up the deployment, failing over the clients during a Regional outage, and failing back after the outage:

1. Set up an MSK cluster in the primary Region.
2. Set up an MSK cluster in the secondary Region.
3. Establish connectivity between the two MSK clusters.
4. Deploy Kafka Connect as containers using AWS Fargate.
5. Deploy MirrorMaker connectors on the Kafka Connect cluster.
6. Verify data replication from one Region to another.
7. Fail over clients to the secondary Region.
8. Fail back clients to the primary Region.

Step 1: Set up an MSK cluster in the primary Region

To set up an MSK cluster in your primary Region, follow these steps:

1. Create an Amazon Virtual Private Cloud (Amazon VPC) in the Region designated for your primary MSK cluster.
2. Create three (or at least two) subnets within the VPC.
3. Create an MSK cluster using the AWS Command Line Interface (AWS CLI) or the AWS Management Console.

In this guide, we will use the console. For detailed instructions, see Creating an Amazon MSK Cluster.

• Choose Kafka version 2.7 or above.
• Select a broker instance type based on your use case and configuration requirements.
• Choose the VPC and subnets created to ensure brokers in your MSK cluster are distributed across multiple Availability Zones.
• For data encryption in transit, enable TLS encryption between brokers and between clients and brokers.
• For Authentication, select IAM access control, TLS-based authentication, or username/password authentication.

We will use SASL/SCRAM (Simple Authentication and Security Layer/Salted Challenge Response Authentication Mechanism) for authenticating Apache Kafka clients with usernames and passwords for clusters secured by AWS Secrets Manager. AWS has since introduced IAM Access Control, which can also be utilized for this solution. For more information, see Securing Apache Kafka with IAM Access Control for Amazon MSK.

1. Create the secret in Secrets Manager and associate it with the MSK cluster. For instructions, see Username and password authentication with AWS Secrets Manager.
2. Ensure the secrets are encrypted using a customer-managed key via AWS Key Management Service (AWS KMS).

Step 2: Set up an MSK cluster in the secondary Region

To set up an MSK cluster in your secondary Region, complete the following steps:

1. Create an MSK cluster in another Region with similar configurations to the first.
2. Ensure the number of brokers and instance types match those configured in the primary.

This guarantees that the secondary cluster possesses equivalent capacity and performance metrics as the primary cluster.

• For data encryption in transit, enable TLS encryption between brokers and between clients and brokers.
• For Authentication, select the same authentication mechanism used in the primary Region.
• Create a secret in Secrets Manager and secure it with a customer-managed KMS key in the Region of the MSK cluster.

Step 3: Set up connectivity between the two MSK clusters

For data replication between the two MSK clusters, they must be able to communicate with each other, regardless of whether the VPCs are within the same or different AWS accounts or Regions. You have the following options for resource communication across VPCs:

• VPC peering
• AWS Transit Gateway

For a seamless onboarding experience at Amazon IXD – VGT2, located at 6401 E HOWDY WELLS AVE LAS VEGAS NV 89115, be sure to follow these steps carefully to enhance your deployments and ensure business continuity.

SEO metadata