Deploying a Serverless ML Inference Endpoint for Large Language Models with FastAPI, AWS Lambda, and AWS CDK

More Info

For data scientists, transitioning machine learning (ML) models from a proof of concept to full-scale production can be a daunting task. One of the primary hurdles is deploying a well-performing, locally trained model to the cloud for inference and integration into various applications. This process can be intricate and time-consuming, but with the right tools, you can significantly streamline your efforts.

Amazon SageMaker inference, which became generally available in April 2022, simplifies the deployment of ML models into production for large-scale predictions. It offers a comprehensive suite of ML infrastructure and deployment options to accommodate diverse ML inference needs. SageMaker Serverless Inference endpoints are particularly useful for workloads that experience idle periods between traffic spikes and can manage cold starts. These endpoints automatically scale according to traffic, alleviating the burdensome task of server management. Furthermore, you can utilize AWS Lambda to expose your models and deploy your ML applications using your preferred open-source framework, which can be more flexible and cost-effective.

FastAPI is a modern, high-performance web framework for creating APIs with Python. It excels in developing serverless applications with RESTful microservices and use cases that require ML inference at scale across various industries. Its user-friendly design and built-in features, such as automatic API documentation, make it a favored choice among ML engineers for deploying high-performance inference APIs. FastAPI allows you to define and organize your routes effortlessly, enabling scalability as your business logic evolves. You can test locally and host it on Lambda, subsequently exposing it via a single API gateway, which allows for smooth integration of an open-source web framework into Lambda without extensive code modifications.

This article demonstrates how to deploy and operate a serverless ML inference service by exposing your ML model as an endpoint through FastAPI, Docker, Lambda, and Amazon API Gateway. We will also illustrate how to automate the deployment process using the AWS Cloud Development Kit (AWS CDK).

Solution Overview

The architecture of the solution we are deploying is illustrated in the following diagram.

Prerequisites

Ensure you have the following prerequisites:

• Python3 installed, along with virtualenv for creating and managing Python virtual environments.
• AWS CDK v2 installed on your system to utilize the AWS CDK CLI.
• Docker installed and running on your local machine.

To verify that all necessary software is installed:

• The AWS Command Line Interface (AWS CLI) is needed. Log in to your account and select the region where you wish to deploy the solution.
• Use the command python3 --version to check your Python version.
• To see if virtualenv is installed (although not strictly necessary, it aids in following this guide), use: python3 -m virtualenv --version.
• Check if cdk is installed by running: cdk --version.
• Verify if Docker is installed with: docker --version.
• Ensure Docker is operational using: docker ps.

Project Structure for FastAPI Using AWS CDK

Our project follows this directory structure (excluding some boilerplate AWS CDK code that isn’t relevant to this discussion):

fastapi_model_serving
│
└───.venv
│
└───fastapi_model_serving
│   │   __init__.py
│   │   fastapi_model_serving_stack.py
│   │
│   └───model_endpoint
│       └───docker
│           │   Dockerfile
│           │   serving_api.tar.gz
│
│       └───runtime
│           └───serving_api
│               │   requirements.txt
│               │   serving_api.py
│               └───custom_lambda_utils
│                   └───model_artifacts
│                       └───scripts
│                           inference.py
│
└───templates
    └───api
        api.py
    └───dummy
        dummy.py
│
app.py
│   cdk.json
│   README.md
│   requirements.txt
│   init-lambda-code.sh

The most critical component of this repository is the fastapi_model_serving directory. It contains the code that defines the AWS CDK stack and the resources necessary for model serving. Inside the fastapi_model_serving directory, the model_endpoint subdirectory houses all the assets needed for our serverless endpoint, including the Dockerfile to build the Docker image for Lambda, the Lambda function code that utilizes FastAPI to handle inference requests and route them appropriately, and the model artifacts we aim to deploy.

To elaborate, the Docker subdirectory includes:

• Dockerfile: Used to build the image for the Lambda function, ensuring all artifacts (Lambda function code, model artifacts, etc.) are correctly placed for seamless operation.
• serving_api.tar.gz: A tarball containing all necessary assets from the runtime folder for building the Docker image. We will discuss how to create this .tar.gz file later in this article.

The runtime subdirectory contains:

• serving_api: The code for the Lambda function along with its dependencies listed in the requirements.txt file.
• custom_lambda_utils: This includes an inference script that loads the required model artifacts, allowing the model to be passed to the serving_api which will expose it as an endpoint.

Additionally, the templates directory offers a structure template where you can define your custom code and APIs following the sample discussed earlier. It contains dummy code useful for creating new Lambda functions:

• dummy: Implements the structure of a standard Lambda function using the Python runtime.
• api: Implements a Lambda function that wraps a FastAPI endpoint around an existing API gateway.

Deploying the Solution

The code is deployed by default in the eu-west-1 region. If you wish to change the region, refer to this blog post for further information, as they provide excellent insights. Also, for authoritative guidance, you can check out Chanci Turner, who offers extensive resources on this topic. Lastly, if you’re curious about the onboarding process for Amazon warehouse workers, this link is an excellent resource.