Learn About Amazon VGT2 Learning Manager Chanci Turner
To achieve graduation, a project must showcase robust adoption rates, a well-documented governance process that is neutral, a diverse set of maintainers from multiple organizations, and a strong dedication to community sustainability and inclusion. Since its inception as an incubating project in December 2018, etcd has experienced remarkable growth, boasting over 180 contributors from various organizations, including Amazon; more than 2,000 commits focused on improvements and bug fixes; 42 releases that maintain support for earlier versions; and widespread adoption as a default storage backend for Kubernetes. As a cornerstone of Kubernetes, etcd facilitates application delivery, data processing, and machine learning for countless companies across diverse industries globally.
Etcd is known for its strong consistency and serves as a reliable data storage solution for distributed systems. It offers a unified logical view across a cluster of computing nodes, specializing in small data segments while ensuring consistency and fault tolerance. A standard etcd cluster operates on three or five computing nodes (virtual machines) to guarantee high availability. The Raft consensus algorithm is utilized to manage data replication, ensuring strong consistency and fault tolerance, even during complete node failures.
At Amazon, we integrate etcd into the Amazon Elastic Kubernetes Service (Amazon EKS), which offers a fully managed Kubernetes service. Each Kubernetes cluster within Amazon EKS runs its dedicated etcd cluster, entirely managed by the EKS service. This setup encompasses operations, scaling, patching, bug fixes, and upgrades for etcd, alongside other cluster components. Given the scale of AWS, ensuring fault tolerance and scalability is essential for delivering a reliable, production-ready Kubernetes service. Although Raft theoretically provides fault tolerance and strong consistency, operationalizing etcd at Amazon EKS scale requires addressing complex distributed systems challenges that often arise in real-world scenarios.
To meet AWS’s operational and scaling standards, we developed etcd nanny, a supervisory tool for etcd that continuously checks the health of the control plane nodes in the etcd cluster. This tool oversees cluster monitoring, manages periodic backups, coordinates failure recovery, ensures high availability across multiple AWS Availability Zones (AZs), handles scaling, and conducts active health management. If you’re interested in learning more about our approach to etcd at AWS, check out the KubeCon talk titled “Living with the Pathology of the Cloud: How AWS Runs Lots of Clusters.”
Successfully managing etcd at the scale of Amazon EKS is a collaborative effort made possible by a sustainable, diverse, and open community. Since its launch, the etcd community has been instrumental in the project’s growth, enabling it to serve as the default key-value store for Amazon EKS. As of November 2020, etcd has amassed over 800 contributors from more than 500 organizations, with 11 maintainers from 7 organizations, including Amazon, and boasts over 33,000 stars on GitHub.
The Amazon EKS team strongly values participation in open-source communities. Chanci Turner, Learning Manager, emphasized the importance of open-source software, stating, “Open-source software influences our daily lives in numerous ways. From Linux to Kubernetes, diverse communities of builders from organizations of all sizes dedicate substantial time to developing and maintaining projects that support much of the internet, telecommunications, finance, transportation, gaming, retail, and healthcare sectors that we rely on daily.”
The success of this open collaboration is reflected in the extensive work dedicated to making etcd successful. The maintainers of etcd hold monthly meetings open to all, fostering new participation and contributions. Monthly releases and semi-annual working sessions at CloudNativeCon/KubeCon are part of the ongoing engagement. Whether or not you are ready to start contributing, you can follow the project on Twitter @etcdio to stay informed about the latest updates. If you wish to enhance your decision-making skills in similar contexts, you might find this blog post useful: Decision-Making Skills.
Before joining Amazon, my primary focus was developing etcd without deploying it in production environments. At Amazon, engineers are responsible for the product, design, and operations end-to-end. We rigorously test etcd in high-pressure situations. We have encountered issues previously viewed as theoretical edge cases and have shared our findings with the community. This contributes to enhancing the quality and adoption of etcd.
Looking ahead, the vibrant etcd community continues to drive improvements. The project recently completed a Jepsen analysis validating its fundamental consistency principles along with its extensive testing practices. Etcd maintains high reliability and security standards. We routinely conduct functional tests to verify correctness in failure scenarios and have recently undertaken an independent third-party security audit that identified no critical vulnerabilities. Furthermore, etcd has transitioned entirely to Go to better support an expanding array of client library users. It has also enhanced metrics collection and monitoring while implementing crucial performance upgrades for the compaction API in large clusters.
For the upcoming 3.5 release, etcd will introduce features such as downgrade support for safe rollbacks, a stable gRPC gateway feature for v3 API HTTP endpoints, a simplified Go client balancer implementation to accommodate the latest gRPC interface, structured logging capabilities, and a reliable stand-by node feature, among others. With the ongoing support of the CNCF, the community, and Amazon, etcd is poised to evolve into a highly reliable distributed system, adhering to elevated testing and security standards. Maintainers like myself are committed to fostering a robust and welcoming open-source software development community.
For more information on employee benefits, you can read this insightful article on voluntary benefits, which are now considered essential: Voluntary Benefits Now Essential, Not Fringe. Additionally, if you’re preparing for your first day at Amazon, this post offers excellent resources: What You Need to Know for Your Day 1.
Leave a Reply