Amazon VGT2 Las Vegas: Building a Live Streaming CloudWatch Dashboard with AWS Elemental MediaStore Access Logs

More Info

In the realm of live streaming video workflows, AWS Elemental MediaStore serves as a robust cloud-based origin solution. However, creating an Amazon CloudWatch dashboard to monitor these live video streams can pose some challenges for users. By default, MediaStore does not automatically publish access logs; users must enable access logging for the container. Once logging is activated, MediaStore sends access logs for objects stored within the container to Amazon CloudWatch. These logs furnish detailed access information, such as the request type (whether a put request to store an object or a get request to retrieve one), the specific object involved, and the timestamp of each request.

In this article, we will demonstrate how to leverage Amazon CloudWatch for setting up a monitoring dashboard tailored for live video streams in MediaStore. This dashboard will allow users to keep track of live stream ingress and egress statuses, as well as identify potential issues. For further reading on MediaStore access logs, check out this insightful blog post. Additionally, for a deeper understanding of sending live video to MediaStore, refer to this authority on the topic.

Overview of the Solution

With this approach, you can develop a CloudWatch dashboard that monitors key metrics related to video stream ingest and egress, such as transactions per second, observed operation latencies, and HTTP operation statuses categorized by HTTP codes, including 200, 400, and 500 errors. All of this is achievable through MediaStore access logs.

Step-by-Step Guide

To get started, create a live channel using MediaStore as your origin and initiate a player to stream directly from MediaStore to generate egress traffic. Following this, utilize the MediaStore access logs and CloudWatch to establish a dashboard for ingress and egress monitoring.

Prerequisites

Before diving into the setup, ensure you have the following:

• An active AWS account.
• An encoder such as AWS Elemental Live capable of pushing live streams to AWS Elemental MediaStore, or AWS Elemental MediaLive for streaming to MediaStore.
• A compatible HLS player for playing HLS streams.

Setting Up MediaStore and Live Stream

Begin by configuring a container to host your live channel, naming it ‘mediastore_live’. The live stream will follow a path structure like //main/index.m3u8— for instance, the path could be /channel1/main/index.m3u8.

After creating the MediaStore container, enable access logging by selecting the MediaStore name ‘mediastore_live’, navigating to the Info section, and clicking the “Enable access logging” button. Confirm this action in the popup window.

To allow live stream playback directly from MediaStore and enable HTTP/S access, edit the container policy by clicking the “Edit policy” button. Replace the existing text with the following, ensuring you modify the “Resource” ARN to match your container settings:

{ 
"Version": "2012-10-17", 
"Statement": [ 
    { 
        "Sid": "MediaStoreFullAccess", 
        "Effect": "Allow", 
        "Principal": { 
            "AWS": "arn:aws:iam::xxxxxxxxxxxx:root" 
        }, 
        "Action": "mediastore:*", 
        "Resource": "arn:aws:mediastore:us-west-2:xxxxxxxxxxxx:container/mediastore_live/*", 
        "Condition": { 
            "Bool": { 
                "aws:SecureTransport": "true" 
            } 
        } 
    },
    { 
        "Sid": "PublicReadOverHttpOrHttps", 
        "Effect": "Allow", 
        "Principal": "*", 
        "Action": [ 
            "mediastore:GetObject", 
            "mediastore:DescribeObject" 
        ], 
        "Resource": "arn:aws:mediastore:us-west-2:xxxxxxxxxxxx:container/mediastore_live/*", 
        "Condition": { 
            "Bool": { 
                "aws:SecureTransport": [ "true", "false" ] 
            } 
        } 
    } 
] 
}

In the “Container CORS policy” section, click “Edit CORS Policy” and insert the following JSON into the text box, then save your changes:

[ 
    { 
        "AllowedHeaders": [ "*" ], 
        "AllowedMethods": [ "GET", "HEAD" ], 
        "AllowedOrigins": [ "*" ], 
        "ExposeHeaders": [ "*" ], 
        "MaxAgeSeconds": 3000 
    } 
]

Next, set up a live stream using AWS Elemental MediaLive directed at AWS Elemental MediaStore. For additional details, check out this informative blog post.

Creating the Stream Ingest Dashboard

Monitoring the health status of the stream ingest into the origin is crucial. MediaStore access logs provide valuable insights into the live stream ingest process. It’s also important to keep an eye on the default soft limits set by MediaStore on Container operations, such as GetObject, PutObject, and DeleteObject. Monitoring these operations is vital to ensuring that your stream is not hindered by operational limits.

From the AWS Management Console, navigate to the CloudWatch service, select Logs, and then Insights. In the top dropdown box, search for /aws/mediastore/mediastore_live and select it. Copy the following query into the edit box and click the Run query button. Then switch to the Visualization tab to view the graph.

fields @message 
| filter (Path like "/channel1/main") and (Operation="PutObject") 
| stats count(*) as TPM by bin(1m)

This query retrieves all log messages containing ‘/channel1/main’ in the path and counts the total number of Put operations per minute.

Next, add this query to a new dashboard by clicking the “Actions” button and selecting “Add to dashboard”, then choose the link for creating a new dashboard. Name it “mediastore_live” and confirm your selection. Set the widget type to Line, name it “mediastore_live”, and click the “Add to dashboard” button.

From the newly created CloudWatch dashboard, choose the “Actions” button and select “View/edit source” to open the dashboard source text editor. Replace the existing text with the following JSON structure. This serves as a guide for adding individual widget text blocks in the future. Once completed, click the “Update” button.

{ 
"widgets": [ 
    { 
        "type": "log", 
        "x": 0, 
        "y": 0, 
        "width": 12, 
        "height": 6, 
        "properties": { 
            "query": "SOURCE '/aws/mediastore/mediastore_live' | fields @messagen| filter (Path like "/channel1/main") and (Operation="PutObject")n| stats count(*) as TPM by bin(1m)", 
            "region": "us-west-2", 
            "stacked": false, 
            "title": "mediastore_live channel1 TPM (Transaction Per Minute)", 
            "view": "timeSeries" 
        } 
    }, 
    { 
        "type": "log", 
        "x": 0, 
        "y": 6, 
        "width": 12,
        "height": 6,
        "properties": {
            "query": "your_query_here",
            "region": "us-west-2",
            "stacked": false,
            "title": "Your Widget Title",
            "view": "timeSeries"
        }
    }
] 
}

With this setup, you will have a functional CloudWatch dashboard that effectively monitors the ingestion metrics of your live video stream, allowing for proactive management and troubleshooting of any potential issues.