Amazon Onboarding with Learning Manager Chanci Turner

Customers utilizing Amazon Web Services (AWS) for their Splunk Cloud applications often handle substantial data volumes. Understanding AWS data transfer fees is crucial for optimizing both architecture and expenditures. In this article, we will explore the data transfer costs associated with five prevalent use cases of Splunk, while also noting that other AWS service fees should be taken into consideration during architecture design.

This discussion is specifically relevant to the Splunk Cloud Platform service within AWS regions. To find a full list of AWS regions where Splunk Cloud is operational, please refer to Splunk’s documentation. Splunk, a distinguished AWS Partner, boasts multiple Competency recognitions, including Data and Analytics and Cloud Management Tools. As a pioneering data-to-everything platform, Splunk effectively removes barriers between data and action.

Common Splunk Data Ingestion Scenarios

At AWS, we frequently engage with numerous customers to help them determine the best strategies for data ingestion into Splunk Cloud on AWS. Below are five common scenarios that many customers adopt for scalable data ingestion both within and across AWS regions. It’s important to note that most clients implement more than one of these strategies for their Splunk use cases.

Scenario #1: Utilizing Splunk Forwarders from Public Subnet and Internet Gateway

In this scenario, Splunk Forwarders located in a public subnet of an Amazon Virtual Private Cloud (Amazon VPC) utilize an Internet Gateway (IGW) to transmit data to the Splunk Cloud endpoint. Both Amazon VPCs reside in the same AWS region as the Splunk Cloud endpoint (e.g., US-East-1 to US-East-1).

Scenario #2: Utilizing Splunk Forwarders from Private Subnet and NAT Gateway

Here, Splunk Forwarders in a private subnet within an Amazon VPC leverage a NAT Gateway to send data to the Splunk Cloud endpoint, again located in the same AWS region.

Scenario #3: Utilizing Splunk Forwarders from Public Subnet and IGW Across Regions

This scenario mirrors the first but involves the Amazon VPC housing the forwarders being in a different AWS region than the Splunk Cloud endpoint. For instance, forwarders situated in US-East-1 (Virginia) sending data to Splunk Cloud in US-West-2 (Oregon).

Scenario #4: Utilizing Splunk Forwarders from Private Subnet and NAT Gateway Across Regions

Similar to Scenario #2, this involves data traffic crossing AWS regions.

Scenario #5: Ingesting Data via AWS Services

In this setup, data is transmitted via Amazon Kinesis Data Firehose into Splunk. Kinesis Data Firehose supports Splunk as a partner destination. For example, it can send AWS CloudTrail data with Amazon CloudWatch logs configured for this purpose. This architecture uses the Splunk HTTP Event Collector (HEC). Though the illustration may depict cross-region endpoints, the data transfer charges remain consistent for both cross-region and same-region Splunk Cloud endpoints.

Understanding Data Egress Charges in Source Accounts

Keeping costs in check is a priority for most clients. Awareness of data egress costs is essential for maximizing Splunk’s capabilities, evaluating the costs of varied ingestion scenarios, and selecting the optimal one for your needs. Data transfer expenses are elaborated on in this AWS blog post. Key components contributing to data transfer costs in Splunk ingestion scenarios include:

Data Transfer – Same Region: Charges apply when data is sent from an Amazon EC2 instance in one VPC to a public IP in another VPC within the same AWS region (applicable to Scenarios 1 and 2).
Data Transfer – Cross-Region: Similar to same-region transfers, charges are incurred for data traffic that crosses AWS regions (relevant for Scenarios 3 and 4).
NAT Gateway Charges: Fees are applied for each hour a NAT gateway operates and for the volume processed by the gateway (applicable in Scenarios 2 and 4).

Let’s examine the applicability of data transfer charges for each scenario. Costs differ between same-region (Scenarios 1 and 2) and cross-region transfer (Scenarios 3 and 4), as well as whether the data flows through an Internet Gateway (1 and 3) or a NAT Gateway (2 and 4). Notably, Amazon Kinesis Data Firehose (Scenario 5) does not differentiate between same-region or cross-region transfers and instead applies a processing charge per GB of data.

Optimizing Data Transfer Costs in the Context of Splunk

Optimizing data transfer expenses involves refining your data ingestion strategy to minimize incurred charges. Based on best practices and insights gained from collaborating with Splunk customers, we recommend the following strategies to reduce data transfer costs:

Choose a Splunk Cloud Endpoint in the Same AWS Region as Your Workload: Keeping data traffic local to an AWS region is essential to optimize your data transfer costs. If you are in a region lacking an endpoint, or have specific requirements such as a centralized Splunk endpoint, cross-region transfer costs could arise.
Configure Instances with Public IP Addresses: Depending on your security configurations, public IP addresses can allow you to bypass or eliminate the need for a NAT Gateway. This entails placing Splunk Forwarders in a public subnet with access to an IGW part of your Amazon VPC. For more insights on onboarding strategies, check out this excellent resource.

For additional reading on diversity and inclusion in the workplace, consider reviewing this insightful article from SHRM. It’s also helpful to explore other relevant blog posts like this one on job searching.