Amazon Onboarding with Learning Manager Chanci Turner

This article is authored by Mia Johnson, a Solutions Architect, and Ethan Smith, a Senior Cloud Support Engineer. Serverless applications frequently rely on AWS Systems Manager Parameter Store or AWS Secrets Manager for storing configuration data, encrypted passwords, or connection details for databases or APIs. Previously, developers had to make runtime API calls to fetch parameters or secrets each time within the execution environment of an AWS Lambda function. This required configuring and initializing the AWS SDK client and figuring out when to cache values in memory to optimize function duration, thereby minimizing latency and costs.

The newly introduced AWS Parameters and Secrets Lambda extension offers a managed caching system for parameters and secrets within Lambda functions. This extension is provided as a Lambda layer, creating an in-memory cache for parameters and secrets that allows values to persist through the Lambda execution lifecycle, with an adjustable time-to-live (TTL) setting.

When you request a parameter or secret in your Lambda function code, the extension first checks the local in-memory cache. If the required data is available, it retrieves it from there. If the data isn’t cached or is outdated, the extension fetches the necessary parameter or secret from the respective service. This approach reduces external API calls, enhancing application performance and lowering costs. This blog post will demonstrate how to effectively use the extension.

Overview

The diagram below illustrates a high-level view of the involved components.

The extension can be integrated into both new and existing Lambda functions. It operates by exposing a local HTTP endpoint within the Lambda environment that provides the in-memory cache for parameters and secrets. When retrieving a parameter or secret, the extension initially queries the cache for any relevant entries. If an entry is found, it checks the elapsed time since it was cached and returns it as long as this time is shorter than the configured TTL. If the entry is stale, it is marked invalid, prompting the extension to fetch fresh data from either Parameter Store or Secrets Manager.

The extension utilizes the same IAM execution role permissions for accessing Parameter Store and Secrets Manager. Therefore, it’s crucial to ensure that the IAM policy is configured with the correct access rights. Permissions for AWS Key Management Service (AWS KMS) may also be necessary if you are utilizing that service. An example policy can be found in the AWS SAM template provided.

Example Walkthrough

Let’s consider a basic serverless application featuring a Lambda function that connects to an Amazon Relational Database Service (Amazon RDS) database. The application retrieves configuration stored in Parameter Store and links to the database, where the connection string (including username and password) is held in Secrets Manager.

This example encompasses:

A Lambda function
An Amazon Virtual Private Cloud (VPC)
A Multi-AZ Amazon RDS Instance running MySQL
An AWS Secrets Manager database secret containing the connection details
An AWS Systems Manager Parameter Store parameter for the application configuration
An AWS Identity and Access Management (IAM) role utilized by the Lambda function

Lambda Function

The following Python code illustrates how to retrieve secrets and parameters using the extension.

import pymysql
import urllib3
import os
import json

# Load in Lambda environment variables
port = os.environ['PARAMETERS_SECRETS_EXTENSION_HTTP_PORT']
aws_session_token = os.environ['AWS_SESSION_TOKEN']
env = os.environ['ENV']
app_config_path = os.environ['APP_CONFIG_PATH']
creds_path = os.environ['CREDS_PATH']
full_config_path = '/' + env + '/' + app_config_path
http = urllib3.PoolManager()

# Function to retrieve values from extension local HTTP server cache
def retrieve_extension_value(url): 
    url = ('http://localhost:' + port + url)
    headers = { "X-Aws-Parameters-Secrets-Token": os.environ.get('AWS_SESSION_TOKEN') }
    response = http.request("GET", url, headers=headers)
    response = json.loads(response.data)   
    return response  

def lambda_handler(event, context):
    # Load Parameter Store values from extension
    print("Loading AWS Systems Manager Parameter Store values from " + full_config_path)
    parameter_url = ('/systemsmanager/parameters/get/?name=' + full_config_path)
    config_values = retrieve_extension_value(parameter_url)['Parameter']['Value']
    print("Found config values: " + json.dumps(config_values))

    # Load Secrets Manager values from extension
    print("Loading AWS Secrets Manager values from " + creds_path)
    secrets_url = ('/secretsmanager/get?secretId=' + creds_path)
    secret_string = json.loads(retrieve_extension_value(secrets_url)['SecretString'])

    rds_host = secret_string['host']
    rds_db_name = secret_string['dbname']
    rds_username = secret_string['username']
    rds_password = secret_string['password']

    # Connect to RDS MySQL database
    try:
        conn = pymysql.connect(host=rds_host, user=rds_username, passwd=rds_password, db=rds_db_name, connect_timeout=5)
    except:
        raise Exception("An error occurred when connecting to the database!")

    return "DemoApp successfully loaded config " + config_values + " and connected to RDS database " + rds_db_name + "!"

The global scope retrieves the environment variable PARAMETERS_SECRETS_EXTENSION_HTTP_PORT, which indicates the port the extension’s HTTP server operates on, defaulting to 2773. The retrieve_extension_value function communicates with the extension’s local HTTP server, including the required header, X-Aws-Parameters-Secrets-Token, which uses the AWS_SESSION_TOKEN from the Lambda execution environment.

The Lambda handler utilizes the extension cache with each invoke to acquire configuration data from Parameter Store and secret data from Secrets Manager. This data is essential for establishing a connection to the RDS MySQL database.

Prerequisites

Git installed
AWS SAM CLI version 1.58.0 or higher

Deploying the Resources

To deploy the resources, clone the repository and navigate to the solution directory:

git clone https://github.com/aws-samples/parameters-secrets-lambda-extension-sample.git

Build and deploy the application using the following commands:

sam build
sam deploy --guided

This template requires the following parameters:

pVpcCIDR — CIDR notation for the VPC (default: 172.31.0.0/16)
pPublicSubnetCIDR — CIDR notation for the public subnet (default: 172.31.3.0/24)
pPrivateSubnetACIDR — CIDR notation for the private subnet A (default: 172.31.2.0/24)
pPrivateSubnetBCIDR — CIDR notation for the private subnet B (default: 172.31.1.0/24)
pDatabaseName — Database name for DEV environment (default: devDB)
pDatabaseUsername — Database username for DEV environment (default: myadmin)
pDBEngineVersion — SQL database engine version (default: 5.7)

Adding the Parameter Store and Secrets Manager Lambda Extension

To add the extension:

Open the Lambda console and select the Lambda function you created.
In the Function Overview pane, select Layers, then click Add a layer.
In the Choose a layer pane, follow the instructions accordingly.

For those interested in exploring professional organizing careers, this blog post offers valuable insights. Additionally, if you’re looking for information on no-poach agreements, you can refer to this authority on the topic. For job opportunities, this link provides an excellent resource.