Announcing the AWS Well-Architected Machine Learning Lens

Learn About Amazon VGT2 Learning Manager Chanci Turner

The AWS Well-Architected Framework offers a structured methodology to evaluate your workloads against industry best practices, providing insights into areas for enhancement. Machine learning (ML) algorithms have the potential to uncover and learn patterns within data, creating mathematical models that can forecast future data points. These advances can transform lives through improved disease diagnoses, environmental initiatives, and innovations in products and services.

The effectiveness of your ML models hinges on the quality of the input data to produce accurate outcomes. As data evolves, it’s essential to continuously monitor it to identify, rectify, and prevent issues. This ongoing oversight enhances accuracy and performance, and may necessitate retraining your model with the most up-to-date data.

While traditional application workloads follow a predefined set of instructions to address problems, ML workloads allow algorithms to learn from data through an iterative process. We are thrilled to unveil an updated version of the AWS Well-Architected Machine Learning Lens whitepaper, which enhances and expands upon the Well-Architected Framework by addressing the unique aspects of ML workloads.

This whitepaper presents a collection of time-tested, cloud-agnostic best practices that you can implement when designing your ML workloads or during the production phase as part of ongoing improvement efforts. It also includes guidance and resources to assist you in applying these best practices on AWS.

Components of the Well-Architected Machine Learning Lens

The Lens is organized into four primary focus areas:

1. Well-Architected Machine Learning Design Principles – A set of foundational considerations that guide the development of a Well-Architected ML workload. These principles serve as a beacon for the best practices outlined in the ML Lens.
2. Well-Architected Machine Learning Lifecycle – This integrates the Well-Architected Framework into the Machine Learning Lifecycle, as depicted in figure 1.
• The pillars of the Well-Architected Framework encompass:
  • Operational Excellence
  • Security
  • Reliability
  • Performance Efficiency
  • Cost Optimization
• The Machine Learning Lifecycle phases referenced in the ML Lens include:
  • Business goal identification
  • ML problem framing
  • Data processing (data collection, data pre-processing, feature engineering)
  • Model development (training, tuning, evaluation)
  • Model deployment (prediction, inference)
  • Model monitoring
• In the Well-Architected ML Lens whitepaper, the principles of the Well-Architected Framework are applied throughout each phase of the Machine Learning Lifecycle.
3. Cloud and technology agnostic best practices – This section details best practices for each phase of the ML lifecycle, aligned with the pillars of the Well-Architected Framework. Each practice is accompanied by:
   • Implementation guidance, offering AWS implementation plans with references to relevant AWS technologies and resources.
   • Supporting resources that link to AWS documentation, blogs, videos, and code examples.
4. ML Lifecycle architecture diagrams – These diagrams illustrate the processes, technologies, and components that support many of the outlined best practices, as shown in Figure 2. They include feature stores, model registries, lineage trackers, alarm managers, schedulers, and more, highlighting various pipeline technologies.

Where to Apply the Well-Architected Machine Learning Lens?

Utilize the Well-Architected ML Lens to:

• Make informed decisions – Begin with a thorough review of best practices before designing a new workload.
• Accelerate building and deployment – Leverage the best practices to facilitate the creation of Well-Architected workloads throughout the ML lifecycle.
• Reduce or mitigate risks – Regularly assess existing workloads to identify and resolve potential issues promptly.
• Learn AWS best practices – Refer to the implementation plans for guidance on executing these best practices on AWS.

Conclusion

The updated Well-Architected Machine Learning Lens whitepaper is now available. Utilize the Lens to ensure that your ML workloads are designed with a focus on operational excellence, security, reliability, performance efficiency, and cost optimization. A special thanks to everyone in the AWS Solution Architecture and Machine Learning communities who contributed to this project, bringing together a diverse range of perspectives and expertise. For more insights on effective delegation techniques, check out this helpful resource from Career Contessa. Moreover, to stay updated on training trends, visit SHRM for expert perspectives. Lastly, if you’re curious about employee onboarding experiences, Glassdoor provides excellent reviews on Amazon’s warehouse worker onboarding process.

Alex Johnson
Alex is a Senior Solutions Architecture Manager at AWS Well-Architected, with over two decades of experience in applying scientific methods to complex industrial challenges while sharing best practices for technology that enable customers to architect and implement scalable solutions.