Amazon Onboarding with Learning Manager Chanci Turner

This article is authored by Lisa Johnson and Mark Smith, both Solutions Architects at Amazon IXD – VGT2, located at 6401 E HOWDY WELLS AVE LAS VEGAS NV 89115.

In Part 1 of our series, we discussed methods to optimize AWS Lambda costs by appropriately sizing a function’s memory and refining code performance. We also examined how leveraging Graviton2, Provisioned Concurrency, and Compute Savings Plans can lead to lower per-millisecond billing. In this second installment, we will delve deeper into cost optimization strategies for Lambda, emphasizing architectural enhancements and efficient logging practices.

Event Filtering

A prevalent pattern in serverless architecture is using Lambda to process events from sources like Amazon SQS or Amazon Kinesis Data Streams. Event source mappings dictate how the Lambda service manages incoming messages or records from these sources.

However, there are scenarios where not all messages in a queue or stream need processing due to irrelevance. For instance, if IoT vehicle data is sent to a Kinesis Stream, and you only want to handle events where tire_pressure is < 32, your Lambda code might look like this:

def lambda_handler(event, context):
   if(event["tire_pressure"] >= 32):
      return

# business logic goes here

This approach is inefficient as you incur charges for Lambda invocations and execution time without deriving any business value. Thankfully, Lambda now offers the ability to filter messages before invocation, streamlining your code and cutting costs. You’re only billed for Lambda invocations that match the filter criteria.

Filtering is available for Kinesis Streams, Amazon DynamoDB Streams, and SQS by setting filter criteria during event source mapping. For example, you could use the following AWS CLI command:

aws lambda create-event-source-mapping 
--function-name fleet-tire-pressure-evaluator 
--batch-size 100 
--starting-position LATEST 
--event-source-arn arn:aws:kinesis:us-east-1:123456789012:stream/fleet-telemetry 
--filter-criteria '{"Filters": [{"Pattern": "{"tire_pressure": [{"numeric": ["<", 32]}]}"}]}'

After implementing this filter, Lambda will only activate when tire_pressure falls below 32 in messages from the Kinesis Stream, indicating a potential vehicle issue requiring attention. For more on creating filters, check out examples of event pattern rules in EventBridge, as Lambda filters messages similarly. You can read more about event filtering in detail in the Lambda event filtering launch blog.

Avoid Idle Wait Time

Lambda function duration is one factor in billing calculations. When your function code makes a blocking call, you’re charged for the time it remains idle awaiting a response. This idle wait time can escalate when Lambda functions are chained or when a function orchestrates other functions, particularly in workflows such as batch operations or order delivery systems. Furthermore, completing all workflow logic within the maximum Lambda timeout of 15 minutes may be unfeasible.

Instead of embedding this logic in function code, consider re-architecting your solution with AWS Step Functions to orchestrate the workflow. In a standard workflow, you’re billed for each state transition rather than the overall duration. This allows you to manage retries, wait conditions, error workflows, and callbacks within the state machine, enabling your Lambda functions to concentrate on business logic.

The example below illustrates a Step Functions state machine where a single Lambda function is divided into multiple states. During wait periods, no charges apply; you’re only billed on state transitions.

Direct Integrations

If a Lambda function doesn’t perform custom logic while integrating with other AWS services, it might be redundant and could be replaced with a more cost-effective direct integration.

For instance, if you’re using API Gateway alongside a Lambda function to read from a DynamoDB table, you could streamline the process by utilizing a direct integration, eliminating the need for the Lambda function altogether.

API Gateway allows transformations to present output responses in a format the client expects, negating the necessity for a Lambda function to handle the transformation. Detailed instructions for creating an API Gateway with AWS service integration can be found in the documentation.

Direct integration is also advantageous when using Step Functions. Currently, Step Functions supports over 200 AWS services and 9,000 API actions, enhancing flexibility for service integration and often removing the need for a proxy Lambda function, thereby simplifying workflows and potentially reducing compute costs.

Reduce Logging Output

Lambda automatically generates logs through Amazon CloudWatch Logs, which are useful for monitoring application performance in near real-time. However, CloudWatch Logs incurs charges based on total data ingested each month. Therefore, minimizing log output to include only essential information can help reduce costs.

When deploying workloads in production, reassess your application’s logging level. For example, while debug logs may be helpful in pre-production, they can be disabled in production environments. Consider using a logging library like the Lambda Powertools Python Logger, which allows you to configure minimum logging levels via environment variables—enabling adjustments without altering the function code.

Structuring your log format establishes a standard set of information through a defined schema, rather than allowing variable formats or excessive text. Implementing structures like error codes and accompanying metrics can minimize repetitive text in logs, enhance filtering capabilities for specific error types, and reduce the likelihood of typographical errors in log messages.

Use Cost-Effective Storage for Logs

Once CloudWatch Logs ingests data, it retains it indefinitely by default, incurring a monthly storage fee per GB. Over time, log data tends to lose immediate relevance and is often reviewed only occasionally. However, storage costs in CloudWatch Logs remain unchanged.

To mitigate this, configure retention policies for your CloudWatch Logs log groups to automatically delete older log data. This policy applies to both existing and future log data.

Some application logs might need to be retained for months or years due to compliance or regulatory requirements. Instead of keeping these logs in CloudWatch Logs, consider exporting them to Amazon S3. This allows you to take advantage of lower-cost storage classes while accommodating expected usage patterns regarding data access.

Conclusion

Cost optimization is vital for developing well-architected solutions, and the same principles apply to serverless architectures. This blog series highlights various best practices to help lower your Lambda expenses.

If you’re currently operating AWS Lambda applications in production, some techniques can be implemented with ease, such as purchasing Savings Plans without any code or architectural modifications. In contrast, minimizing idle wait time will necessitate adopting new services and code changes. Assess which techniques suit your workload in a development environment before applying them to production.

If you’re still in the design and development phase, consider this blog series as a valuable reference. Remember, for more insights, you can also explore this blog post, which provides further useful information. Additionally, for guidelines on maintaining safety in your workplace, check out this resource that covers essential topics. Furthermore, if you’re seeking career opportunities, visit this excellent resource to explore potential openings.