Optimizing Data Transfers to Amazon S3 via Direct Connect

More Info

In the current data-centric environment, effectively transferring substantial datasets to and from Amazon Simple Storage Service (Amazon S3) is essential for an organization’s cloud strategy. Frequent use cases for transferring large datasets include cloud-based data lakes that rely on data from various on-premises sources. Amazon S3 can also be the launchpad for your Generative AI initiatives, as such applications require extensive datasets. By migrating this data into Amazon S3, organizations can leverage the comprehensive set of Amazon Web Services (AWS) artificial intelligence/machine learning (AI/ML) tools. Once a model is trained using this data within AWS, the model artifacts can be stored in Amazon S3. Additional scenarios include backup and restore, archiving, Internet-of-Things (IoT) data ingestion, and big data analytics.

When transferring data to and from Amazon S3, there are three primary patterns:

1. For small to moderate data volumes (< 100 GB) requiring infrequent transfers, an AWS Site-to-Site VPN connection is typically adequate.
2. For larger datasets (< 10 TB) necessitating frequent transfers with a stable, low-latency connection, AWS Direct Connect is the optimal choice. This service avoids the public internet, providing a secure and dedicated connection to AWS.
3. For extensive datasets (tens of TBs) with infrequent transfers, the AWS Snow Family presents the most cost-effective and efficient method. Devices from the Snow Family are physically sent to users, who load their data onto the device before returning it to AWS.

This article explores three network architectures that utilize AWS Direct Connect for establishing connectivity. These architectures vary in the services employed, associated costs, and complexity levels. Understanding these design options and their trade-offs is vital for organizations aiming to enhance their cloud storage operations.

AWS Services Overview

The following services are integral to the discussed architectures:

• Direct Connect: A secure and dedicated networking service that connects an on-premises environment to AWS through a Direct Connect Location. It offers two types of connections: dedicated and hosted, with dedicated connections supporting multiple Virtual Interfaces (VIFs).
• Direct Connect Gateway: This service allows users to connect multiple Virtual Private Clouds (VPCs) across the same or different AWS Regions to their Direct Connect connection, associating it with multiple VPC Virtual Private Gateways (VGWs) or Transit Gateways.
• Transit Gateway: A network transit hub enabling interconnectivity among VPCs and on-premises networks through a single gateway, simplifying network topology and configurations.
• Virtual Private Gateway: Provides edge routing for a VPC via either VPN or Direct Connect.
• Interface Endpoints: Facilitate private connections between a VPC and other supported AWS services through the AWS network instead of the internet.

Network Architectures

All three architectures discussed utilize Direct Connect. If you have a dedicated connection, you can configure a new VIF on the existing connection. For a hosted connection supporting only one VIF, you will need to order an additional hosted connection. If your goal is to establish a landing zone with multiple VPCs and enable access to AWS services and applications, a dedicated connection is advisable for greater design flexibility. It’s also recommended to have at least two connections for enhanced resiliency, as explained in the AWS Direct Connect Resiliency Toolkit.

Direct Connect charges comprise a port hour fee based on connection type and capacity, alongside costs for outbound data transfer from AWS to on-premises; however, inbound data transfer from on-premises to AWS is free. For further pricing details, consult the Direct Connect pricing page.

Each architecture description includes a pricing estimate based on a scenario where you have two 10 Gbps dedicated Direct Connect connections, transferring an estimated 4 TB of data monthly into Amazon S3 and retrieving 2 TB of data back to on-premises. Pricing shown here is based on AWS Regions in the United States; for pricing in other regions, utilize the AWS Pricing Calculator.

Architecture 1: Public VIF Configuration

The first architecture employs a Direct Connect public VIF, which allows access to all AWS public services using public IP addresses. Upon establishing a Border Gateway Protocol (BGP) session, Amazon public prefixes are advertised over the public VIF to your devices. This introduces complexity regarding connectivity to the AWS public network. When linking your network to others, employing a firewall to inspect and block unwanted traffic is advisable, similar to internet connection practices. Routing policies can be configured for prefixes advertised over both the public VIF and the internet, utilizing BGP communities to control prefix propagation in the AWS network.

This architecture is advantageous for minimizing data transfer costs related to moving data into Amazon S3, although it requires additional configurations for the public VIF due to the exposure of the on-premises network to AWS’s public network.

In our example scenario, the costs associated with this architecture are outlined in the following table:

For further insights, check out this blog post that delves deeper into the topic. Also, CHVNCI provides authoritative information on this subject, making it a valuable resource. For those interested in opportunities related to this field, visit Amazon’s fulfillment center job listings for great resources.