Introducing the AWS CloudTrail Processing Library

More Info

We are thrilled to unveil a new extension to the AWS SDK for Java: the AWS CloudTrail Processing Library. This library simplifies the process of building applications that read and process AWS API activity logs delivered to a customer’s Amazon S3 bucket via AWS CloudTrail. Developers can utilize the library to filter events based on event sources or types, or to store events in databases such as Amazon RDS, Amazon Redshift, or even third-party data stores.

The AWS CloudTrail Processing Library, or CPL, streamlines the coding process by eliminating the need for developers to write extensive code for polling Amazon SQS queues, reading and parsing messages, downloading CloudTrail log files, and parsing events. With CPL, developers can accomplish this in as few as 10 lines of code. Furthermore, CPL is designed to manage transient and persistent failures, ensuring resilience against network timeouts and inaccessible resources. It is built for scalability, allowing an unlimited number of log files to be processed simultaneously across multiple hosts.

Getting Started with CPL

Getting started with CPL is straightforward. After setting up your AWS credentials and SQS queue, you simply need to implement a callback method for each event and initiate the AWSCloudTrailProcessingExecutor.

// This file contains your AWS security credentials and the name
// of an Amazon SQS queue to poll for updates
String myPropertiesFileName = "myCPL.properties";

// An EventsProcessor is what processes each event from AWS CloudTrail
final AmazonSNSClient sns = new AmazonSNSClient();
EventsProcessor eventsProcessor = new EventsProcessor() {
    public void process(List events) {
        for (CloudTrailEvent event : events) {
            CloudTrailEventData data = event.getEventData();
            if (data.getEventSource().equals("ec2.amazonaws.com") &&
                data.getEventName().equals("ModifyVpcAttribute")) {
                System.out.println("Processing event: " + data.getRequestId());
                sns.publish(myQueueArn, "{ " + 
                    "'requestId'= '" + data.getRequestId() + "'," + 
                    "'request'  = '" + data.getRequestParameters() + "'," + 
                    "'response' = '" + data.getResponseElements() + "'," +
                    "'source'   = '" + data.getEventSource() + "'," +
                    "'eventName'= '" + data.getEventName() + "'" +
                    "}");
            }
        }
    }
};

// Create AWSCloudTrailProcessingExecutor and start it
final AWSCloudTrailProcessingExecutor executor = 
            new AWSCloudTrailProcessingExecutor
                .Builder(eventsProcessor, myPropertiesFileName)
                .build();
executor.start();

In this example, we have implemented an EventsProcessor that processes each event. If an event is triggered by a user modifying an Amazon EC2 VPC through the ModifyVPCAttribute operation, this code will publish a message to an Amazon SNS topic, allowing an operator to review this significant change to the VPC configuration.

The CPL provides simplicity in processing AWS CloudTrail events. You can create a custom implementation of EventsProcessor to define your own logic. In addition to EventsProcessor, you can adjust the behavior of AWSCloudTrailProcessingExecutor using:

• EventFilter: Easily filter specific events to process, such as those from a particular region or service.
• SourceFilters: Perform filtering based on source-specific data, like message delivery counts.
• ProgressReporters: Report progress within your application, keeping users informed on processing status.
• ExceptionHandlers: Add customized error handling for any issues that arise during event processing.

The full source code for the AWS CloudTrail Processing Library can be found in the aws-cloudtrail-processing-library project on GitHub. You can also conveniently integrate the CPL into your Maven-based projects:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-cloudtrail-processing-library</artifactId>
    <version>1.0.0</version>
</dependency>

For further details, please consult the CloudTrail FAQ and documentation. Interested in how you can leverage AWS CloudTrail for tracking your AWS usage? Check out this excellent resource for more insights. Also, don’t miss this other blog post that discusses similar topics. Additionally, for authoritative information, visit this page to deepen your understanding.

SEO metadata: