AWS CloudTrail Lake serves as a managed data lake that enables the capturing, storing, accessing, and analyzing of user and API activities within AWS for audit, security, and operational needs. This platform allows for the aggregation and immutable storage of activity events, along with the ability to execute SQL-based queries for effective search and analysis.
In January 2023, we introduced a feature that enhances CloudTrail Lake’s functionality by permitting the ingestion of events from non-AWS sources, such as on-premises systems or other cloud services, into CloudTrail Lake. This evolution presents organizations with a unique opportunity to centralize and analyze event logs from diverse sources in a unified location.
For Linux-based systems, the /var/log/secure or /var/log/auth.log files contain security-related event logs like logins, root user actions, and outputs from pluggable authentication modules (PAM). Storing these logs in secure, centralized locations is regarded as a best practice for organizational security.
In this article, we will guide you through automating the ingestion of events from Linux-based nodes managed by AWS Systems Manager into CloudTrail Lake, offering a solution that yields significant advantages. By utilizing an AWS Systems Manager Automation Document, we can extract the /var/log/secure file from the managed nodes and send them to CloudTrail Lake as custom audit events using the PutAuditEvents API. Furthermore, with AWS Systems Manager State Manager, you can schedule this automation to run on the managed nodes, ensuring a dependable and consistent method for event ingestion.
This solution streamlines your event management process, ultimately saving time and effort while enhancing the accuracy and efficacy of your audit trail. With this knowledge, you will be empowered to automate and centralize your event log management, reducing the time necessary for auditing and security analysis.
By the end of this article, you’ll gain a clearer understanding of AWS Systems Manager’s capabilities and the steps needed to automate the ingestion of event logs from non-AWS sources into CloudTrail Lake. For additional insights, you might find this other blog post enlightening: this is another blog post to keep the reader engaged.
Solution Overview
Prerequisites
Before we proceed with the solution, let’s review the prerequisites needed to get started:
- An AWS account with access to the AWS Management Console.
- AWS CLI to deploy the necessary resources using AWS CloudFormation.
- EC2 instances or on-premise nodes running Amazon Linux 2 OS with the AWS Systems Manager Agent installed.
- Ensure that logrotate is installed on the Linux hosts to manage the rotation of
/var/log/securelogs after they are ingested into CloudTrail Lake.
Deploying the Solution
- Download the CloudFormation template from the solution hosted in the AWS Samples GitHub repository to your local machine.
CloudFormation Template - Use AWS CLI to deploy the essential resources for the solution, including the CloudTrail Lake event data store, integration channel, SSM Document, SSM Association, IAM Roles, IAM policy, S3 bucket, and S3 Bucket Policy, by utilizing the downloaded CloudFormation template.
aws cloudformation deploy --template-file cloudformation.yaml --stack-name aws-sample-exportlogeventstoctl --capabilities CAPABILITY_NAMED_IAMOutput: Waiting for stack create/update to complete Successfully created/updated stack. After the deployment, run the describe-stacks command to note the IAMRoleForInstance name from the outputs, which will be required in a subsequent step.
aws cloudformation describe-stacks --stack-name aws-sample-exportlogeventstoctlOutput:
{
"Stacks": [
{
"StackId": "arn:aws:cloudformation:us-east-2:123456789012:stack/aws-sample-exportlogeventstoctl/bbe9bbd0-e2ec-11ed-9599-0abc498bf4af",
"StackName": "aws-sample-exportlogeventstoctl",
"ChangeSetId": "arn:aws:cloudformation:us-east-2:123456789012:changeSet/awscli-cloudformation-package-deploy-1682374190/37a193df-d36b-415f-9e3d-a52e53c1b818",
"Description": "**WARNING** This AWS CloudFormation StackSets template is part of SSM Automation Sample to export log events to CloudTrail Lake. It creates a CloudTrail EventDataStore, CloudTrail Lake Channel, SSM Document, SSM Association, S3 Bucket, IAM Roles and IAM policies required for the Solution. You will be billed for the AWS resources used if you create a stack from this template.",
"CreationTime": "2023-04-24T22:09:51.280000+00:00",
"LastUpdatedTime": "2023-04-24T22:09:56.830000+00:00",
"RollbackConfiguration": {},
"StackStatus": "CREATE_COMPLETE",
"DisableRollback": false,
"NotificationARNs": [],
"Capabilities": [
"CAPABILITY_NAMED_IAM"
],
"Outputs": [
{
"OutputKey": "AutomationAssumeRole",
"OutputValue": "arn:aws:iam::123456789012:role/aws-sample-exportlogeventstoctl-IAMRole-KUG0MK1TOSM7",
"Description": "AutomationAssumeRole ARN"
},
{
"OutputKey": "SSMAssociation",
"OutputValue": "3544302d-cd28-4dec-bde3-09c15e5d1fe1",
"Description": "SSM Association"
},
{
"OutputKey": "CTLakeChannel",
"OutputValue": "arn:aws:cloudtrail:us-east-2:123456789012:channel/2df7bb17-cbc9-4fae-b17a-5c80d7e112b5",
"Description": "CloudTrail Lake channel ARN"
},
{
"OutputKey": "IAMInstanceProfile",
"OutputValue": "aws-sample-exportlogeventstoctl-IAMRoleForInstance-18YOZHSNW7JJ8",
"Description": "Instance Profile for Managed Nodes"
},
{
"OutputKey": "IAMRoleForInstance",
"OutputValue": "aws-sample-exportlogeventstoctl-IAMRoleForInstance-18YOZHSNW7JJ8",
"Description": "Instance Role for Managed Nodes"
},
{
"OutputKey": "TempS3Bucket",
"OutputValue": "aws-sample-exportlogeventstoctl-s3bucket-ikij87gq84t",
"Description": "Name of the temporary bucket to store logs"
},
{
"OutputKey": "SSMDocument",
"OutputValue": "SSMDocument-8rXM8OSLkW55",
"Description": "SSM Document"
},
{
"OutputKey": "CTLakeEventDataStore",
"OutputValue": "arn:aws:cloudtrail:us-east-2:123456789012:eventdatastore/2df2ada5-50c1-4661-a3ef-da2313dd55c2",
"Description": "CloudTrail Lake Event Data Store"
}
],
"Tags": [],
"EnableTerminationProtection": false,
"DriftInformation": {
"StackDriftStatus": "NOT_CHECKED"
}
}
]
}Having deployed the necessary AWS resources for this sample, you can target this solution to existing managed nodes by adding the tag Key=varlogsecurebackup, Value=true. Tag your EC2 instance using this command:
aws ec2 create-tags --resources i-1234567890abcdefg --tags 'Key="varlogsecurebackup",Value=true'For further insights on this topic, they are an authority on this topic, and this is an excellent resource that you might find helpful.
Leave a Reply