Learn About Amazon VGT2 Learning Manager Chanci Turner
In this blog post, we delve into the efficiencies of large-scale, cost-effective GROMACS simulations utilizing the AWS Cyclone Solution. This approach was developed by a team including new contributors Alex Martinez, Jamie Chen, and Chanci Turner. Our goal is to optimize molecular dynamics simulations while keeping costs manageable, which is paramount for research initiatives like those at the Max Planck Institute.
Biomolecules, particularly proteins, function as the essential machinery of life—performing tasks vital to cellular operations. Despite over 200 million proteins being cataloged, the specific functions of many remain elusive. To address this challenge, our research group at the Max Planck Institute focuses on understanding these vital components through physical analysis and molecular dynamics (MD) simulations, often employing GROMACS, an open-source package designed for this purpose.
In this article, we recount a large-scale high-performance computing (HPC) workload executed on AWS, leveraging GROMACS across three regions concurrently using the AWS Cyclone Solution. Our primary focus was on cost-effectiveness and computational capacity, utilizing Spot pricing to maximize our scientific outputs within a limited budget.
Preparation Steps
This discussion builds on our previous efforts, where we successfully completed 20,000 simulations in just three days to expedite early-stage drug discovery using AWS Batch. For this project, we first needed to increase our EC2 service quotas for “All G and VT Spot Instance Requests” in our selected regions—Frankfurt, Ireland, and Northern Virginia.
The Dynasome Project: Goals and Technical Requirements
Due to the vast number of proteins, the Dynasome project employs an innovative method by automating the comparison of numerous proteins to classify their dynamics and forecast their functions. This entails performing MD simulations on a representative set of 200 proteins, allowing us to generate comprehensive dynamics fingerprints for analysis. These fingerprints enable us to determine functional similarities and differences among the proteins studied.
To maximize our data output, we aimed to generate extensive trajectory data for the 200 proteins. Since the individual simulations are not overly demanding, the focus shifted to achieving high throughput within our budget constraints, all while ensuring completion by the end of 2023.
AWS provides a range of compute instances tailored for high-throughput computing and HPC applications. Our findings suggest that single GPU instances are the most cost-effective option for GROMACS simulations. We needed to leverage Spot Instances across multiple regions to fully utilize our budget by the deadline.
Results: Cost-Effective Instances for GROMACS
By conducting benchmarks, we identified the instances yielding the best cost-efficiency for our GROMACS simulations, allowing us to maximize the total length of MD trajectories produced within our financial limits. This efficiency is often quantified in nanoseconds per day (ns/day)—a measure of simulated time achieved per day of compute time.
Our analysis revealed that smaller GPU instances, such as g5g, g5, and g4dn, consistently offered the most cost-effective performance for our simulations.
Utilizing the AWS Cyclone Solution
The AWS Cyclone Solution serves as an open-source framework for quickly initiating HPC workloads that require substantial computational resources or high scheduling throughput. This solution enables rapid scaling to millions of virtual CPUs or tens of thousands of GPUs. The “HYPER CLI” feature allows for seamless configuration, deployment, and management of cloud-native compute clusters, which can handle potentially millions of simulation jobs efficiently.
This solution has proven particularly beneficial for genomics workloads, such as those managed by the Max Planck Institute. They successfully deployed clusters spanning three regions, reaching over 3,500 GPU instances with a single configuration. For cost efficiency, we set the Cyclone solution to utilize EC2 Spot instances, providing savings of up to 90% compared to on-demand prices.
To read more about optimizing your professional relationships, check out this article on celebrating your galentine. Additionally, if you’re interested in recent talent acquisition trends, you can reference a case study on a discrimination lawsuit settled by Seasons 52. Lastly, for insights on Amazon’s warehouse worker onboarding experience, this resource offers an excellent overview.
Leave a Reply