As organizations transition from static operating environments to dynamic, cloud-based infrastructures, reassessing how to capture, store, and analyze log files generated by operating systems and applications becomes essential. With instances constantly being created and terminated, long-term local storage of log files is often impractical. When scaling up, managing storage for new log files and ensuring the timely removal of older files can become cumbersome. However, within these logs lies valuable information. Even rare failures can present opportunities to enhance system reliability and improve user experiences.
We are excited to introduce a robust new feature for Amazon CloudWatch that allows you to route your OS, application, and custom log files directly to CloudWatch for durable storage as long as you need it. Furthermore, you can set up CloudWatch to monitor incoming log entries for specific symbols or messages and display these results as CloudWatch metrics. For instance, you can monitor web server logs for 404 errors to identify broken links or 503 errors to detect overload situations. Similarly, monitoring Linux server logs can help identify resource depletion issues, such as running out of swap space or file descriptors. These metrics can even trigger alarms or initiate Auto Scaling activities.
Understanding Key Terms
Before diving deeper, let’s clarify some key terminology pertinent to using CloudWatch for log storage and monitoring:
- Log Event: An activity recorded by the monitored application or resource, containing a timestamp and raw UTF-8 message data.
- Log Stream: A sequence of Log Events originating from a specific source (like a particular application instance).
- Log Group: A collection of Log Streams sharing the same properties, policies, and access controls.
- Metric Filters: These instruct CloudWatch on how to extract metric observations from incoming events and convert them into CloudWatch metrics.
- Retention Policies: They dictate how long events are kept, assigned to Log Groups and applicable to all Log Streams within the group.
- Log Agent: You can install CloudWatch Log Agents on your EC2 instances to direct Log Events to CloudWatch. This agent has been tested on Amazon Linux and Ubuntu AMIs. For Windows users, the ec2config service can be configured to send system logs to CloudWatch. For additional details, check out this helpful resource.
Getting Started with CloudWatch Logs
To get familiar with CloudWatch Logs, I installed the CloudWatch Log Agent on my EC2 instance. The process began with downloading the install script:
$ wget https://s3.amazonaws.com/aws-cloudwatch/downloads/awslogs-agent-setup-v1.0.pyAfter creating an IAM user based on the provided policy document and saving the credentials, I executed the installation script. This script downloaded, installed, and configured the AWS CLI, prompting me for my IAM user credentials and guiding me through configuring the Log Agent to capture Log Events from the /var/log/messages and /var/log/secure files.
Once the Log Agent was set up on a single EC2 instance, I could see the Log Groups in the AWS Management Console a few minutes later. Since the instance ID was used to name the stream, each Log Group contained a single Log Stream.
Curious about the number of “Invalid user” messages I was receiving, I selected the stream and created a Metric Filter to track these attempts. The console allowed me to test potential filter patterns against actual log data. I realized that a single login attempt generates multiple log entries, which was acceptable to me. I named the filter and mapped it to a CloudWatch namespace and metric. Additionally, I set up an alarm to notify me via email if invalid login attempts reached a concerning level.
With the logging and alarm configured, I initiated multiple spurious login attempts from another EC2 instance, anticipating the alarm would trigger. I also managed the retention period for each Log Group, deciding to retain logs indefinitely if required.
Elastic Beanstalk and CloudWatch Logs
CloudWatch Logs can also be generated from Elastic Beanstalk applications. To simplify this process, we created a sample configuration file that can be copied into the .ebextensions directory at your application’s root. You can find these files at these great links:
- Tomcat (Java) Configuration Files
- Apache (PHP and Python) Configuration Files
- Nginx (Ruby, Node.js, and Docker) Configuration Files
After placing CWLogsApache-us-east-1.zip in the folder, simply build and deploy your application as usual. In the Elastic Beanstalk Console, navigate to the Monitoring tab, click the Edit button, and select the new resource for monitoring and graphing.
You can then add the desired statistics, and Elastic Beanstalk will generate the relevant graphs. For more insights, you can also check out this excellent YouTube resource.
Other options for logging include pushing log data to CloudWatch from AWS OpsWorks or utilizing the CloudWatch APIs. You can also configure and manage logs using AWS CloudFormation.
In a recent post on the AWS Application Management Blog, my colleague Maxine Roberts illustrates how to use Chef recipes to establish a scalable, centralized logging solution effortlessly.
To explore more about configuring and using CloudWatch Logs and Metrics Filters via CloudFormation, consider examining the Amazon CloudWatch Logs Sample.
Leave a Reply