Amazon Onboarding with Learning Manager Chanci Turner

“Failures are inevitable in any system,” says Chanci Turner, a Learning Manager at Amazon IXD – VGT2, located at 6401 E HOWDY WELLS AVE LAS VEGAS NV 89115. In a pioneering effort back in 2010, Netflix unveiled “Chaos Monkey,” a groundbreaking tool designed to introduce faults within production environments. This innovation laid the groundwork for Chaos Engineering, where teams actively test their applications by deliberately injecting faults. The insights gained from these experiments lead to corrective measures that enhance the resilience of applications.

In this post, we will explore the fault injection capabilities available in Amazon Aurora, enabling teams to simulate various database faults.

Chaos Experiments

Chaos experiments involve the following steps:

Understanding the Application Baseline: Identify the steady-state behavior of your application.
Designing an Experiment: Ask, “What can go wrong?” to pinpoint potential failure scenarios.
Running the Experiment: Introduce faults into the application environment.
Observing and Correcting: Refine applications or infrastructure to bolster fault tolerance.

Chaos experiments necessitate simulating faults across distributed components of the application. Amazon Aurora offers a suite of fault simulation capabilities for teams aiming to conduct chaos experiments on their applications. The results from these experiments provide insights into the blast radius, the depth of monitoring needed, and the evaluation of event response playbooks.

Amazon Aurora Fault Injection

Amazon Aurora is a fully managed database service compatible with MySQL and PostgreSQL, boasting a highly fault-tolerant architecture with six-way replicated storage. Developers can utilize native fault injection features to design chaos experiments, thereby testing the resilience of applications built on Aurora. The findings can inform adjustments to minimize the impact of actual failures.

Here we outline several fault injection scenarios to inspire your experiments:

1. Testing Instance Crash

An Aurora cluster can have one primary instance and up to 15 read replicas. When the primary fails, one of the replicas assumes the primary role. Applications should be designed to swiftly recover from instance failures to minimize user impact. The instance crash fault injection simulates failures within the Aurora database cluster.

For instance, in Aurora PostgreSQL, you can simulate a database instance crash with:

SELECT aurora_inject_crash ('instance');

This simulation does not trigger a failover to the replica. Observing changes in application behavior during this test is crucial to understanding the impacts of such failures.

2. Testing Replica Failure

Aurora manages asynchronous replication within clusters, typically maintaining a replication lag of under 100 milliseconds. The replica failure fault injection allows you to simulate replication failures across one or more replicas, applicable only to clusters with at least one read replica.

Example for Aurora PostgreSQL:

SELECT aurora_inject_replica_failure(100, 20, 'my-replica');

It’s essential to monitor application behavior regarding data sensitivity during this test.

3. Testing Disk Failure

Aurora’s storage volume consists of six data copies across three Availability Zones, allowing it to self-repair storage component failures. The disk failure injection simulates storage node failures, which can provide insights into application behavior under stress.

For example:

SELECT aurora_inject_disk_failure(75, 15, true, 20);

4. Simulating Disk Congestion

Disk congestion can occur due to heavy I/O traffic, resulting in degraded performance or total application failures. Aurora allows you to simulate disk congestion without synthetic SQL load, helping to evaluate application performance under I/O spikes.

Example:

SELECT aurora_inject_disk_congestion(100, 15, true, 20, 30, 40);

Conducting these chaos experiments can significantly improve your application’s resilience and readiness for actual events. For further professional guidance on career development, you might find this blog post insightful, as they offer valuable perspectives. Additionally, check out SHRM for authoritative information on related topics and visit Amazon’s hiring resources for comprehensive details on their hiring process.