ML Inferencing at the Edge with Amazon SageMaker Edge and Ambarella CV25

Learn About Amazon VGT2 Learning Manager Chanci Turner

Ambarella specializes in creating advanced computer vision System on Chips (SoCs) through a highly efficient AI chip architecture known as CVflow, which provides the necessary Deep Neural Network (DNN) processing capabilities for edge inferencing applications such as intelligent home monitoring and smart surveillance cameras. Developers can convert models trained on popular frameworks like TensorFlow or MXNET into the Ambarella CVflow format, enabling these models to operate effectively on edge devices. The Amazon SageMaker Edge service integrates the Ambarella toolchain into its workflow, simplifying the process of converting and optimizing models for deployment on this platform.

In this article, we outline the steps for setting up model optimization and conversion using SageMaker Edge, incorporating the model into your edge application, and deploying and testing the model on an Ambarella CV25 device to create a smart surveillance camera application operating at the edge.

Smart Camera Use Case

Smart security cameras come equipped with machine learning (ML) features tailored for specific use cases, such as vehicle and animal detection or identifying suspicious behavior, parking issues, or zone violations. These scenarios necessitate the execution of ML models on the camera’s edge computing unit, ensuring optimal performance.

Ambarella’s CVx processors leverage the company’s proprietary CVflow architecture to deliver impressive DNN inference performance while consuming minimal power. This blend of high efficiency and low energy usage makes them perfectly suited for devices that require intelligent processing at the edge. To enable edge deployment, ML models must be optimized and compiled for the target platform. SageMaker Edge is instrumental in optimizing and converting ML models into the most widely used frameworks for seamless operation on edge devices.

Solution Overview

Our smart security camera solution encompasses ML model optimization and compilation configuration, runtime execution, inference testing, and evaluation on the edge device. SageMaker Edge facilitates model optimization and conversion, allowing for faster performance without sacrificing accuracy. The ML model can be based on any framework supported by SageMaker Edge. For further details, check out the list of Supported Frameworks, Devices, Systems, and Architectures.

The integration of Ambarella CVflow tools within SageMaker Edge provides developers with added benefits:

• Developers are relieved from managing updates and maintenance of the compiler toolchain, as it is integrated and user-transparent.
• Layers unsupported by CVflow are automatically compiled to run on the ARM architecture by the SageMaker Edge compiler.

The following diagram illustrates the architecture of the solution:

The implementation steps are as follows:

1. Prepare the model package.
2. Configure and initiate the model compilation job for Ambarella CV25.
3. Transfer the packaged model artifacts to the device.
4. Conduct inference testing on the device.

Prepare the Model Package

For Ambarella targets, SageMaker Edge requires a model package containing a configuration file named amba_config.json, calibration images, and the trained ML model file. This model package is a compressed TAR file (*.tar.gz). You can utilize an Amazon SageMaker notebook instance to train, test ML models, and prepare the model package file. To create a notebook instance, follow these steps:

1. In the SageMaker console, select Notebook in the navigation pane, then choose Notebook instances.
2. Click on Create notebook instance.
3. Enter a name for your instance and select ml.t2.medium as the instance type.

This instance is sufficient for testing and model preparation purposes.

4. For IAM role, create a new AWS Identity and Access Management (IAM) role to allow access to Amazon Simple Storage Service (Amazon S3) buckets, or select an existing role.
5. Keep other configurations at default and select Create notebook instance.

Once the status shows as InService, you can start using your new SageMaker notebook instance.

Select Open JupyterLab to access your workspace.

For this example, we will use a pre-trained TFLite model for compilation and deployment to the edge device. The model chosen is a pre-trained SSD object detection model sourced from the TensorFlow model zoo on the COCO dataset.

Download the converted TFLite model, and you’re ready to proceed with the testing and preparation of the model package.

Create a new notebook with the kernel conda_tensorflow2_p36 on the launcher view. Import the necessary libraries as follows:

import cv2
import numpy as np
from tensorflow.lite.python.interpreter import Interpreter

Save an example image as street-frame.jpg, create a folder named calib_img in your workspace directory, and upload the image there. Upload the contents of the downloaded model package into the current folder. Run the following command to load your pre-trained TFLite model and display its parameters, which are essential for configuring our model for compilation:

interpreter = Interpreter(model_path='ssd_mobilenet_v1_coco_2018_01_28.tflite')
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
height = input_details[0]['shape'][1]
width = input_details[0]['shape'][2]

print("Input name: '{}'".format(input_details[0]['name']))
print("Input Shape: {}".format(input_details[0]['shape'].tolist())) 

The output will show the input name and shape:

Input name: 'normalized_input_image_tensor'
Input Shape: [1, 300, 300, 3]

Next, use the following code to load the test image and perform inference:

image = cv2.imread("calib_img/street-frame.jpg")
image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
imH, imW, _ = image.shape
image_resized = cv2.resize(image_rgb, (width, height))
input_data = np.expand_dims(image_resized, axis=0)

input_data = (np.float32(input_data) - 127.5) / 127.5

interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()

boxes = interpreter.get_tensor(output_details[0]['index'])[0]
classes = interpreter.get_tensor(output_details[1]['index'])[0]
scores = interpreter.get_tensor(output_details[2]['index'])[0]
num = interpreter.get_tensor(output_details[3]['index'])[0]

To visualize the detected bounding boxes on the image and save the result as street-frame_results.jpg, use the following code:

with open('labelmap.txt', 'r') as f:
    labels = [line.strip() for line in f.readlines()]

for i in range(len(scores)):
    if ((scores[i] > 0.1) and (scores[i] <= 1.0)):
        ymin = int(max(1, (boxes[i][0] * imH)))
        xmin = int(max(1, (boxes[i][1] * imW)))
        ymax = int(min(imH, (boxes[i][2] * imH)))
        xmax = int(min(imW, (boxes[i][3] * imW)))

        cv2.rectangle(image, (xmin, ymin), (xmax, ymax), (10, 255, 0), 2)

        object_name = labels[int(classes[i])]
        label = '%s: %d%%' % (object_name, int(scores[i]*100))
        labelSize, baseLine = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.7, 2)
        label_ymin = max(ymin, labelSize[1] + 10)
        cv2.rectangle(image, (xmin, label_ymin-labelSize[1]-10), (xmin + labelSize[0], label_ymin+baseLine-10), (255, 255, 255), cv2.FILLED)
        cv2.putText(image, label, (xmin, label_ymin-7), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 0), 2)

cv2.imwrite('street-frame_results.jpg', image) 

Finally, display the result image using the following command:

Image(filename='street-frame_results.jpg')

By following these steps, you can successfully run inference on your edge device, creating an intelligent surveillance solution. Additionally, for more information on career opportunities, you can visit this Amazon job listing.

For further insights on counseling and the workplace, check out this other engaging blog post here.

SEO Metadata