Amazon IXD – VGT2 Las Vegas: Unveiling Micro Environments for Apache Airflow

More Info

Amazon Managed Workflows for Apache Airflow (Amazon MWAA), a managed service for Apache Airflow, allows organizations to derive business insights by integrating, enriching, and transforming data through workflows. This solution enhances infrastructure security and availability while minimizing operational burdens.

We are thrilled to introduce the mw1.micro environment class, the newest addition to Amazon MWAA offerings. This configuration aims to provide a more economical option for deploying Airflow environments in the cloud. With mw1.micro, we empower teams seeking a lightweight solution without sacrificing vital features. In this article, we will delve into the characteristics of mw1.micro, its benefits, suitable use cases, and guide you through setting up an Amazon MWAA environment with this new class.

Organizations often manage multiple MWAA environments to separate various development stages, optimize resources, handle version control, bolster security, ensure redundancy, customize settings, improve scalability, and trial new approaches. While this strategy offers greater flexibility in workflow management, maintaining numerous AWS accounts for development, testing, and production adds complexity and cost. The conventional use of full-sized Amazon MWAA environments for smaller projects or proof-of-concept initiatives can be financially burdensome. Moreover, organizations employing a federated deployment model face difficulties in providing isolated environments for different teams while optimizing expenses. The mw1.micro class addresses these challenges by facilitating better resource utilization and significant cost reductions.

The Micro Environment Class

The mw1.micro configuration provides a balanced resource allocation suitable for small-scale data processing and orchestration tasks. It offers 1 vCPU and 3GB of RAM for a scheduler/worker hybrid container, with the web server also having 1 vCPU and 3GB RAM. The Amazon Elastic Container Service (Amazon ECS) tasks in this environment utilize AWS Fargate platform version 1.4.0, enhancing ephemeral task storage to 20GB.

mw1.micro environments support up to three concurrent tasks, making them ideal for sequential or lightly parallelized workflows. Additionally, they can accommodate up to 25 Directed Acyclic Graphs (DAGs), allowing for effective organization and management of various data pipelines. This micro environment is particularly advantageous for development, testing, or smaller production workloads where resource efficiency and cost are of primary importance.

The mw1.micro maintains the overall architecture of Amazon MWAA by combining the Airflow scheduler and worker into a single container. Thus, it operates with just two AWS Fargate tasks: one scheduler/worker hybrid and one web server.

An important update is that the meta database now utilizes a t4g.medium Amazon Aurora PostgreSQL-Compatible Edition instance powered by AWS Graviton2. This advancement provides improvements in compute, storage, and networking, as well as a reduction in carbon footprint.

Supported Features

The mw1.micro environment retains essential functionalities of Amazon MWAA and Airflow that developers rely on:

• Public or private web server options, allowing control over access to the Airflow UI
• Custom plugins and requirements can be added, extending Airflow’s capabilities and simplifying dependency management
• Startup scripts enable the execution of initialization tasks, ensuring precise environment configuration
• A fully functional Airflow UI, delivering an intuitive interface for workflow management and monitoring
• Similar networking features as other Amazon MWAA classes, including custom URLs and shared VPC support
• Separate logs for the scheduler and worker in their respective Amazon CloudWatch log groups, easing monitoring and troubleshooting

Considerations

The architectural decisions for mw1.micro balance functionality with cost-effectiveness. However, the limited resources introduce some constraints:

• The scheduler and worker are integrated into a single Fargate task, allowing only one scheduler/worker container.
• There is only one Fargate task for the web server, limiting it to a max of 1.
• The maximum number of concurrent Airflow tasks in the worker (worker_autoscale) is capped at 3.

Pricing and Availability

Amazon MWAA pricing remains unchanged; you only pay for what you utilize, including the environment class and metadata database storage consumed. For specific rates and details, refer to the Amazon Managed Workflows for Apache Airflow Pricing page.

Monitoring Performance

Upon using the new environment class, it is crucial to monitor its performance to maintain optimal operations and identify potential capacity issues. Keep an eye on key metrics like metadata database memory usage and CPU utilization of the worker/scheduler hybrid container. For guidance on monitoring, consider reading more about container, database, and queue utilization metrics for Amazon MWAA, which provides insights to help right-size your resources.

Setting Up a New Micro Environment

You can establish an Amazon MWAA micro environment in your account and preferred AWS Region through the AWS Management Console, API, or AWS Command Line Interface (AWS CLI). If you’re embracing infrastructure as code (IaC), you can automate this setup using AWS CloudFormation, AWS CDK, or Terraform scripts.

The Amazon MWAA micro environment class is available immediately in all regions where Amazon MWAA is currently operational.

Conclusion

This article has introduced the new mw1.micro environment class within Amazon MWAA, catering to teams engaged in smaller projects, proof-of-concept tasks, or those needing isolated environments for different departments. By offering a lightweight yet feature-rich solution, mw1.micro allows organizations to achieve notable cost savings without sacrificing essential functionalities.

As you venture into utilizing mw1.micro, remember to monitor its performance using the suggested metrics to ensure optimal operation. With availability across all regions hosting Amazon MWAA, your teams can harness the power of Airflow more efficiently and economically, unlocking new possibilities for managing and orchestrating data pipelines in the cloud. For additional insights and code examples on Amazon MWAA, check out this other blog post here. Additionally, this resource is excellent for further understanding the topic, and for authoritative insights, visit this link.

Amazon IXD – VGT2
6401 E Howdy Wells Ave,
Las Vegas, NV 89115