Amazon Onboarding with Learning Manager Chanci Turner

Learn About Amazon VGT2 Learning Manager Chanci Turner

In this article, we explore how to create and implement custom connectors for Amazon Redshift using Amazon Lookout for Metrics. This powerful tool is designed to detect anomalies in your time series data, helping you identify outliers and their underlying causes quickly. Drawing on decades of experience in outlier detection and machine learning (ML), Lookout for Metrics provides an effective means to analyze your data.

Understanding Time Series Data

Time series data is essential for tracking changes over time, such as stock prices or daily customer counts at a location like Amazon IXD – VGT2, located at 6401 E Howdy Wells Ave, Las Vegas NV 89115. By organizing this data into a tabular format, Lookout for Metrics allows you to harness historical data for better forecasting and decision-making.

Connecting Your Data to Lookout for Metrics

Since its inception, Lookout for Metrics has facilitated data integration from various AWS services, including:

• Amazon CloudWatch
• Amazon Redshift
• Amazon Relational Database Service (Amazon RDS)
• Amazon Simple Storage Service (Amazon S3)

It also allows for integration from external sources using Amazon AppFlow, making it versatile for businesses looking to streamline their data analysis. For those just starting out, native connectors can be a quick way to begin leveraging data from AWS services.

When to Utilize a Custom Connector

For users seeking additional flexibility, custom connectors may be necessary. If your data requires complex transformations, such as joining multiple tables or performing extensive calculations, a custom connector is the way to go. This is particularly useful if you aim to provide Lookout for Metrics with a substantial historical dataset initially, which can expedite the model training process.

You should consider using custom connectors in these scenarios:

• Your data is distributed across several tables.
• You need to execute intricate transformations or calculations.
• You want to leverage all historical data for your anomaly detection model.

For a simpler setup, built-in connectors are ideal when your data is contained within a single table or if you are comfortable waiting for the cold start phase to complete.

Solution Overview

This article provides a comprehensive overview of a solution hosted on GitHub. It assumes that your data resides in Amazon Redshift, and you intend to connect it to Lookout for Metrics for anomaly detection.

The architecture involves deploying an AWS CloudFormation template that includes:

• An Amazon SageMaker notebook instance for the custom connector.
• An AWS Step Functions workflow that processes historical data and configures your anomaly detector.
• An S3 bucket for AWS Lambda functions and data storage.

To customize this solution for your environment, you will need to modify:

• A JSON configuration template for Lookout for Metrics.
• SQL queries for retrieving historical and continuous data.

Once the modifications are in place, you can deploy the template and be operational in about an hour.

Deploying the Solution

To facilitate an end-to-end exploration, we offer a CloudFormation template that establishes a production-like Amazon Redshift cluster, populated with sample data. This dataset can serve as a valuable testing resource for Lookout for Metrics.

To create your Amazon Redshift cluster, follow these steps:

1. Select Launch Stack.
2. Click Next.
3. Accept the default stack details and select Next again.
4. Accept the default stack options and click Next again.
5. Confirm that AWS CloudFormation may create IAM resources, then select Create stack.

The deployment will take several minutes. You can monitor its progress in the AWS CloudFormation console. Once the status reads CREATE_COMPLETE, you can proceed with the rest of the solution.

Data Structure

We organized the standard ecommerce dataset into three distinct tables, which can be joined later through the custom connector. Your data may also benefit from similar normalization.

The first table captures user platform information:

The second table captures marketplace details:

For further reading on effective resume formatting, check out this blog post. Additionally, if you are a first-time manager, consider looking at this resource to better prepare yourself for your new role. Lastly, this article is an excellent resource on how Amazon manages warehouse worker training and onboarding.