Visualize AWS Network Firewall Logs Using Amazon QuickSight Dashboards

More Info

As businesses increasingly adopt cloud-based and digital-native applications, leveraging various cloud services is essential for innovation and enhancing customer experiences. While these applications are vital for operational success, it is equally important to monitor network traffic across diverse segments, particularly for the protection of sensitive information. An analytics dashboard is crucial for extracting insights from network events in multi-account environments within Amazon Web Services (AWS). Such a dashboard can help prioritize and respond to security threats, fostering a proactive approach to network management and security. The use of analytics and visualizations can significantly enhance security for cloud applications.

This article will guide you through creating interactive dashboards with Amazon QuickSight to visualize your AWS Network Firewall logs. It will also discuss tracking mechanisms that enhance visibility into your Amazon Virtual Private Cloud (Amazon VPC) network traffic, which is essential for identifying security threats and networking issues. This integration allows for effective visual interpretation of network traffic flows and analysis of firewall logs, thus enabling thorough insights and agile decision-making.

Solution Overview:

The architecture diagram below outlines the log collection and visualization solution for Network Firewall logs using QuickSight. Network traffic originating from an Amazon Elastic Compute Cloud (Amazon EC2) instance and heading toward the internet routes through the Network Firewall endpoint established within an inspection VPC. The Network Firewall follows its configured policies to scrutinize the traffic, generating and storing logs in JSON format within an Amazon Simple Storage Service (Amazon S3) bucket.

Subsequently, an AWS Glue crawler accesses the S3 bucket to catalog the logs and create table definitions in the AWS Glue Data Catalog. Finally, Amazon Athena is employed to create views that can be visualized as dashboards in QuickSight.

Prerequisites:

1. Set up AWS Network Firewall.
2. Alternatively, to follow along with our solution, deploy a Network Firewall and related resources using this AWS CloudFormation template. The stack will deploy an Amazon VPC, a Network Firewall, and an EC2 instance which may incur costs during usage. Please refer to the AWS Network Firewall and Amazon EC2 Pricing.
3. Create an S3 bucket to store the Network Firewall logs.
4. Enable and configure Athena.
5. Sign up for QuickSight.

Step 1: Configure Network Firewall Logs Destination as Your S3 Bucket

1. Log in to the AWS Management Console and navigate to Amazon VPC.
2. Go to your VPC dashboard, then to Network Firewall and Firewalls.
3. Select the firewall you wish to configure to view its details.
4. In the Firewall details tab, locate the Logging section and click Edit.
5. For Log type, check the boxes for Alert and Flow to see further options.
6. For both the Alert log and Flow log destinations, select S3, and enter your S3 bucket name.
7. Click Save to finalize your firewall logging configuration.

Step 2: Generate the AWS Network Firewall Logs

Logs are produced whenever network traffic interacts with the Network Firewall. If you have a network firewall set up, it is already generating logs and sending them to the S3 bucket specified in the previous step, allowing you to move on to step 3. Alternatively, if you used the sample CloudFormation template from the prerequisites, connect to your EC2 instance {Stack Name}-test-instance via AWS Systems Manager Session Manager, and utilize the Ncat command to generate network traffic. This will result in firewall logs being stored in your S3 bucket.

Now that you have configured your network firewall to send logs to an Amazon S3 bucket, it’s time to prepare for log analysis. You will create the necessary analytics components, starting with the Data Catalog database and an AWS Glue crawler.

Step 3: Create a Data Catalog Database and an AWS Glue Crawler

A database in the Data Catalog serves as a container for tables. Create a database within the Data Catalog and name it anflogsdb. An AWS Glue crawler examines the raw data in the S3 bucket to identify its format, structure, and other attributes. It organizes the data into tables and maintains the catalog.

To create a crawler that reads the log files stored on Amazon S3:

1. In the AWS Glue console, select Data Catalog in the navigation pane, then choose Crawlers and Create crawler.
2. Name the Crawler network_firewall_log_data, optionally add a description and tags, then click Next.
3. On the Choose data sources and classifiers page, select Add a data source and choose S3 as the data source.
4. For Location of S3 data, select In this account, and enter the path of the S3 bucket where the firewall logs are stored. You can browse S3 to select the folder containing both logs (alert and flow logs). Click Add an S3 data source and then Next.

This blog focuses on both flow and alert firewall logs.

The crawler will need permissions to access the data store and create objects in the Data Catalog. Select Create an IAM role. The IAM role name starting with AWSGlueServiceRole- is pre-filled. Append network_firewall_logs to the field. Click Create and Next. For this demo, refer to this guide that discusses security best practices in IAM to configure the IAM role following the principle of least privilege.

Crawlers create tables in your Data Catalog database. Choose anflogsdb (created at the beginning of this step) as your target database. Optionally, you can add a table name prefix, like nfwlogs. Expand Advanced options and select Update all new and existing partitions with metadata from the table. This allows partitions to inherit metadata properties such as classification, input and output format, SerDe information, and schema from their parent table.

Under Crawler schedule, select your preferred frequency for the crawler to check for new or modified logs in S3. After the initial full run, it will add new partitions identified since the last run. Click Next and review the crawler configuration before clicking Create crawler. For more details, see Adding an AWS Glue crawler for excellent resources.

On the details page of the newly created crawler, select Run crawler. Once completed, you should see two tables in the Table changes column—one for flow logs and one for alert logs—to query your network firewall logs.

Step 4: Create Athena Views

To develop custom analytics, you can establish views in Athena. These are logical tables that allow you to query a subset of data. Views simplify complexity, perform basic transformations and aggregations on columns, and reduce maintenance when querying large tables. For instance, you will flatten the JSON data by extracting fields from the event JSON field and converting Unix time format data to timestamp format.

In this step, you will query the Data Catalog database tables created in the previous step and set up views in Athena to be utilized in your dashboards.