As many of you might be aware, AWS CloudTrail tracks API activities within your AWS account and sends log files of these activities to a specified S3 bucket (for more details, refer to my post, AWS CloudTrail – Capture AWS API Activity). Earlier this year, we introduced CloudWatch Logs, enabling you to store and monitor log files from your operating system and applications. As mentioned previously, CloudWatch Logs provides capabilities to monitor specific phrases, values, or patterns.
Introducing CloudTrail and CloudWatch Integration
Today, we are excited to unveil the integration of CloudTrail with CloudWatch Logs. This enhancement allows you to receive SNS notifications from CloudWatch, activated by specific API activities logged by CloudTrail.
With SNS notifications, you can swiftly respond when a significant pattern is detected. This could involve contacting the impacted user for further information, automatically generating a trouble ticket, or commencing other troubleshooting steps. For instance, you can create a CloudWatch Logs metric filter that monitors API calls related to a VPC, establish a CloudWatch metric and alarm, and receive SNS notifications whenever the metric count surpasses your designated threshold.
Once you enable this integration via the CloudTrail console, CloudTrail will begin sending log files containing API activities to the specified CloudWatch log group.
Just like any AWS feature, you can activate this integration using the AWS Command Line Interface (AWS CLI) or one of the AWS SDKs. Even after enabling this new integration, CloudTrail will continue to place log files in the designated S3 bucket.
Setting Up Metrics, Filters, and Alarms
After activating the integration, the next step is to create CloudWatch metric filters, metrics, and alarms for receiving SNS notifications and taking immediate action. For example, if I want to be alerted whenever an authorization failure occurs within my AWS account, I can set this up in three simple steps!
Since CloudTrail provides an error code in its payload whenever an API call fails due to inadequate or improper permissions, I can search through CloudTrail events for the terms “AccessDenied” and “UnauthorizedOperation” using a Metric Filter.
Next, I will configure the filter to generate a CloudWatch metric named “AuthorizationFailureCount” in the “LogMetrics” namespace. Each occurrence of “AccessDenied” or “UnauthorizedOperation” will increment the metric value by 1.
Following that, I can create a CloudWatch alarm and set a threshold. Since I want to be informed of every authorization failure, I will configure the alarm threshold so that it will trigger if one or more failures occur per minute. Of course, this can be customized as needed.
To test this, I can create an email subscription to my SNS topic and initiate an operation that will result in a failed authorization. The notification will read as follows:
“You are receiving this email because your Amazon CloudWatch Alarm ‘AuthorizationFailureCount’ in the US – N. Virginia region has entered the ALARM state, because ‘Threshold Crossed: 1 datapoint (3.0) was greater than the threshold (1.0).’ at ‘Wednesday 05 November, 2014 19:12:58 UTC.”
As you can see, the process of connecting CloudTrail to CloudWatch and configuring SNS notifications can be accomplished in just a few minutes. The CloudTrail team is eager to hear your feedback on this new feature, particularly regarding which APIs and activities you’d like to monitor. Feel free to visit the CloudTrail Forum and share your thoughts!
Pricing and Availability
This integration is currently available in the regions where CloudWatch Logs are supported: US East (N. Virginia), US West (Oregon), and Europe (Ireland). Standard charges for CloudWatch Logs and CloudWatch will apply.
Support from Partners
AWS Partners, CloudNexa and Graylog2, have introduced tools designed to analyze CloudTrail log files. CloudNexa, a premier consulting partner and AWS reseller, is offering their new CloudTrail functionality at no extra cost as part of their vNOC Platform. This tool allows you to view CloudTrail events by region and filter out irrelevant events, enabling quick access to those of interest. For more details, check out this blog post or their informative one-minute video.
Graylog2, on the other hand, is an open-source solution that permits you to combine CloudTrail log files with your operating system or application logs. Once these events are ingested, Graylog2 facilitates rapid searches across large datasets, allowing for event correlation from multiple sources and customizable dashboards. For additional insights, you can visit their site or watch this excellent resource.
If you want to understand more about this topic, check out this authority on the subject.
— Alex;
Modified 1/26/2021 – To enhance your experience, expired links in this post have been updated or removed from the original content.
Leave a Reply