Learn About Amazon VGT2 Learning Manager Chanci Turner
Migrating data securely and efficiently to Amazon S3 from on-premises data centers presents several challenges. While smaller datasets can be moved using numerous open-source or free tools, the growing volume of data has made it increasingly necessary for IT professionals across industries to find robust solutions for migrating larger datasets to Amazon S3. Many free solutions may not be adequate for complex migrations as they often add stress to IT teams attempting to automate the transition of intricate and dynamic data systems.
To assist customers in simplifying this process and enhancing the reliability of large-scale migrations, we recommend utilizing AWS DataSync and AWS Storage Gateway. Recently, I had the opportunity to present a session at re:Invent titled “Accelerate your migration to Amazon S3.” This 30-minute session is available for on-demand viewing, and I will recap the highlights in this blog post.
Overview of AWS DataSync and AWS Storage Gateway
AWS DataSync is a data transfer service that streamlines, automates, and speeds up the movement of data between on-premises storage systems and AWS Storage services, as well as between various AWS Storage services. It can be employed to migrate active datasets to AWS, archive data to free up local storage, and replicate data to AWS for business continuity. DataSync is particularly useful for transferring data to the cloud for prompt analysis and processing. Currently, it can transfer data between Network File System (NFS) shares, Server Message Block (SMB) shares, self-managed object storage, AWS Snowcone, Amazon S3 buckets, Amazon Elastic File System (EFS), and Amazon FSx for Windows File Server.
AWS Storage Gateway serves as a hybrid cloud storage solution, allowing on-premises applications to access virtually limitless cloud storage through interfaces like NFS, SMB, and iSCSI. This service can be utilized for backing up and archiving data to AWS, providing local file shares backed by cloud storage, and granting low-latency access to cloud data for on-premises applications.
Confronting Migration Challenges
When addressing the challenges faced by our customers, the primary objective is to simplify data migration projects while ensuring quick and secure data transfers. By leveraging AWS DataSync, AWS Storage Gateway, or a combination of both, customers have successfully met these objectives.
We observe several common use cases among our customers. One prevalent scenario involves sending backups to the cloud. In this case, AWS Storage Gateway is configured as a Virtual Tape Library (VTL) and can replace existing physical tape libraries seamlessly. This Tape Gateway allows organizations to switch from physical tapes to virtual tapes in AWS without needing to alter their existing backup workflows. Backups are automatically directed to Amazon S3, Amazon S3 Glacier, or Amazon S3 Glacier Deep Archive.
Another scenario pertains to customers needing to continuously migrate data to Amazon S3. Frequently, organizations require data to be in AWS for additional processing or analytics due to its scalable capabilities. Often, the source data changes regularly, making it cumbersome to capture these alterations with custom scripts. Here, AWS DataSync offers significant advantages by connecting directly to unstructured file systems or on-premises object storage. It provides features such as scheduled transfers, incremental transfers, and filtering options. For enhanced security, AWS DataSync includes end-to-end data verification and encryption.
A third common use case involves migrating datasets to the cloud while retaining NFS or SMB access from on-premises systems. In this hybrid approach, AWS DataSync is initially used to transfer the data to Amazon S3 efficiently. Subsequently, AWS Storage Gateway is deployed in File Gateway mode on-premises to ensure that applications or users can access the migrated data using NFS or SMB protocols.
Conclusion
In my re:Invent session on accelerating migration to Amazon S3, I shared insights on the functionality of AWS DataSync and AWS Storage Gateway, as well as how these services can facilitate your migration process. Both services simplify data migration, ensuring fast, efficient, and secure transfers to and from Amazon S3. Once the data is in Amazon S3, customers can leverage a more cost-effective storage solution for archiving and utilize AWS’s scalable infrastructure for processing their datasets.
If you’re eager to gain practical experience with AWS DataSync and AWS Storage Gateway, consider exploring these resources:
- Migrating from on-premises file servers to AWS hybrid-cloud Storage – AWS virtual workshop
- DataSync migration workshop
- Migrate to Amazon FSx for Windows File Server using AWS DataSync
- Online data migration workshop
For those interested in workplace dynamics, I encourage you to check out this insightful blog post. Additionally, for authoritative information on employee relations, visit SHRM’s website. For an excellent resource on how Amazon Fulfillment Centers train associates, take a look at this article.
Feel free to leave any comments or questions about my re:Invent session or this blog post in the comments section. Thank you for reading!
Leave a Reply