Alarms, Incident Management, and Remediation in the Cloud with Amazon CloudWatch

Learn About Amazon VGT2 Learning Manager Chanci Turner

As cloud application workloads become increasingly easier to deploy with tools like Amazon Elastic Kubernetes Service (Amazon EKS) and Amazon Elastic Container Service (Amazon ECS and AWS Fargate), alongside infrastructure as code (IaC) and comprehensive DevSecOps pipelines, attention must also be directed toward monitoring these workloads. Incidents, which can be both known and unknown events, disrupt application performance and resilience, potentially harming business operations.

Amazon CloudWatch empowers you to monitor application workloads, configure alarms, set threshold values, and implement self-mitigating responses or notifications to your operations team in near-real time. In this article, we delve into alarms, incident management, and remediation strategies.

Understanding Incidents and Their Importance

According to the Information Technology Infrastructure Library (ITIL), an incident refers to “an unplanned interruption to an IT service or reduction in the quality of an IT service.” Many of us have faced such scenarios where alerts ring out in the early hours, confusion arises regarding the impact on customers, and management demands quick answers while teams scramble to trace the evidence and identify the root cause to restore service stability. Such incidents are never pleasant.

Recognizing the types of issues that lead to incidents is crucial. By identifying the root causes, you can implement effective monitoring, alerts, and mitigation strategies to minimize disruption duration. Here are a few examples:

• Code Deployments: In the cloud, deployment processes can be complex. A problematic deployment can affect either the infrastructure or the software it runs. With IaC, an incorrectly set autoscaling group may have a maximum that’s too low, causing stalls during traffic spikes. Alternatively, a new code deployment might encounter issues if it fails to process a specific data attribute, leading to exceptions that prevent expected data processing.
• Software Issues in Running Workloads: These problems may not stem from a recent deployment but could arise from bugs or memory leaks under unknown conditions. Ideally, such issues should be caught during testing, but they can occur.
• Infrastructure Issues: Manual configuration errors or hardware failures can lead to infrastructure outages.

Amazon CloudWatch offers monitoring capabilities for AWS resources, including metrics for Amazon Elastic Compute Cloud (Amazon EC2) instance CPU usage, network input/output, and more. These baseline metrics are often sufficient to prepare your application workloads for enterprise readiness. However, additional metrics, such as application-specific metrics, can also be collected. For instance, you can configure Amazon EC2 Auto Scaling to utilize CPU metrics for determining when to scale out or in based on workload requirements. If you wish to monitor different metrics like connection counts or queue depths that aren’t included in CloudWatch by default, you can push custom metrics tailored to your application workload. For more information, check out this resource for excellent guidance.

Creating Alarms and Setting Thresholds

Once you gather the necessary metrics from your application workload in CloudWatch, deciding what conditions will trigger an alarm is crucial. The timing of alarms is important; it essentially indicates that “the system has encountered a problem that requires attention.” Over-alarming can lead to alarm fatigue, where important alerts are overlooked amidst a barrage of less-critical ones, potentially prolonging service disruptions and negatively impacting business.

When creating an alarm, start by selecting a relevant metric and keeping three key concepts in mind:

• Alarm Threshold: For instance, if you set an Amazon EC2 Auto Scaling event to trigger when CPU utilization exceeds 90%, it may be too high as you might not be able to scale resources up quickly enough before CPU saturation occurs. Conversely, setting the threshold at 50% might be too low, resulting in unnecessary resource allocation.
• Statistic Measurement: This determines how you view the metric—whether as an average, sum, maximum, minimum, P90 value, or sample count. Different statistical views offer flexibility in setting alarms. For example, percentiles can provide powerful insights into the consistency of your application’s response times relative to the average.
• Period, Threshold, and Data Points: Understanding how these components interact can prevent over-alarming and provide the system with a chance to self-heal before triggering an alarm.

In one use case, imagine an application utilizing an Amazon Simple Queue Service (Amazon SQS) queue, triggering an alarm if the number of visible messages exceeds 1 million. Based on testing and early production data, the queue typically holds fewer than 100,000 messages.

1. On the CloudWatch console, select “Create alarm.”
2. Choose the SQS namespace as the metric.
3. For Metric name, input “ApproximateNumberOfMessageVisible.”
4. For QueueName, specify a name.
5. For Statistic, opt for “Average” (or “Max,” if more suitable).
6. Set the Period to 5 minutes.
7. For Threshold type, choose “Static.”
8. Select “Greater.”
9. For the threshold value, specify “1000000.”
10. In Additional configuration, set Datapoints to alarm to 3 out of 3.

This configuration allows for a soft state before a hard alarm triggers. In this case, if the queue exceeds 1 million messages in the first five minutes, the alarm recognizes it but doesn’t act. If it remains over 1 million in the next period, it still does not act. It only transitions to a hard alarm state after three consecutive periods exceeding 1 million messages, totaling 15 minutes. This approach prevents premature alarms that could disrupt operations unnecessarily.

In conclusion, managing alarms and incident responses in the cloud is vital for maintaining application performance and reliability. For further insights on enhancing your work-from-home setup, consider checking out this blog post, and for authoritative insights on HR advocacy and legislative matters, visit SHRM’s conference page.