Amazon Onboarding with Learning Manager Chanci Turner

Amazon’s Managed Service for Prometheus is a serverless, Prometheus-compatible monitoring solution designed to efficiently track metrics in containerized environments at scale. This service allows users to leverage the open-source Prometheus query language (PromQL) for monitoring workload performance without the burden of managing the underlying infrastructure necessary for data ingestion, storage, alerting, and querying of operational metrics. With automatic scaling capabilities, Amazon Managed Service for Prometheus integrates seamlessly with AWS security services, ensuring secure and rapid access to data. Users can gather Prometheus metrics from Amazon Elastic Kubernetes Service (Amazon EKS) and Amazon Elastic Container Service (Amazon ECS) environments by utilizing AWS Distro for OpenTelemetry or Prometheus servers as collection agents.

Slack serves as a robust business communication platform that offers persistent chat rooms (channels) organized by topics, private groups, and direct messaging. A significant number of customers express the desire to receive immediate alerts in a Slack channel when Amazon Managed Service for Prometheus identifies performance issues impacting customers with critical applications. This article outlines a step-by-step process for configuring the Amazon Managed Service for Prometheus Alert Manager to send alerts to a Slack channel via Amazon Simple Notification Service (SNS) and AWS Lambda.

Solution Overview

The illustration below provides a high-level overview of the solution.

For this guide, you will need the following:

AWS account
AWS IAM user or role with appropriate permissions
Amazon Managed Service for Prometheus
Amazon Simple Notification Service (SNS)
AWS Lambda
Slack channel

Walk-through

The following steps summarize the overall process:

Create an Alert Manager definition in Amazon Managed Service for Prometheus to publish alerts to an SNS topic.
Develop a Lambda function that decodes the SNS message and forwards it to the Slack webhook API.
Configure the Lambda function as a target for messages sent to the SNS topic.

Create SNS Topic

An Amazon SNS topic functions as a logical access point serving as a communication channel. Topics enable you to group multiple endpoints (such as AWS Lambda, Amazon SQS, HTTP/S, or email). The initial task with Amazon SNS is creating a topic. You can use the AWS Management Console or AWS SDK for this. Follow the steps outlined in the AWS Well-Architected lab to create a topic. Ensure you have the SNS topic ARN ready to proceed.

Create Amazon Managed Service for Prometheus Alert Manager Definition

The Alert Manager is responsible for handling alerts generated by alerting rules executed by Amazon Managed Service for Prometheus. It manages deduplication, grouping, and routing of alerts to downstream receivers, such as Amazon SNS. You can upload an Alert Manager definition using the AWS CLI or AWS Management Console. Below is a sample configuration for the SNS receiver that integrates with Slack.

alertmanager_config: |
  global:
  templates:
  route:
    receiver: example-sns
  receivers:
    - name: example-sns
      sns_configs:
        - topic_arn: arn:aws:sns:us-east-2:123456789012:sns-receiver-2
          send_resolved: true
          sigv4:
            region: us-east-2
          message: |
            channel: 'general'
            text: >-
              {{ range .Alerts -}}
                *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
                *Description:* {{ .Annotations.description }}
                *Details:*
                  {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
                  {{ end }}
              {{ end }}
          attributes:
            key: severity
            value: SEV

You can upload the Alert Manager configuration to a workspace using the following AWS CLI command:

aws amp create-alert-manager-definition --data file://<path to base64-encoded file> --workspace-id <workspace_id> --region <region>

Alternatively, you can update an Alert Manager definition via the AWS Management Console. Make sure to grant Amazon Managed Service for Prometheus permission to send messages to your Amazon SNS topic. Refer to this document for guidance on creating the access policy.

Create Slack Webhook

Incoming Webhooks are a straightforward method to post messages into Slack from applications. By creating an Incoming Webhook, you obtain a unique URL to which you can send a JSON payload that includes the message text and any additional options. Leverage the usual formatting and layout blocks to make your messages visually appealing. Follow the instructions available to create your Slack webhook, but treat the generated webhook as sensitive information, similar to credentials, and avoid sharing it publicly.

Create Lambda Function

Next, we will create a Lambda function that performs the following actions:

Unwraps the YAML body of the SNS message and converts it to JSON.
Sends the JSON data from the SNS message to the Slack webhook API.

Lambda Function Execution Role

A Lambda function’s execution role is an AWS Identity and Access Management (IAM) role that grants the function permission to access AWS services and resources. Assign this role when creating a function, as Lambda assumes the role when the function is executed. It’s advisable to enable logging for your serverless Lambda functions. The AWS managed role AWSLambdaBasicExecutionRole allows permission to upload logs to CloudWatch.

This function will utilize the PyYAML library, so you need to create a deployment package that includes the necessary dependencies. Using the Lambda console, create the following Lambda function:

#!/usr/bin/python3.6
import urllib3
import json
import yaml
http = urllib3.PoolManager()
def lambda_handler(event, context):
    url = "<webhook_url>"
    msg = yaml.safe_load(event['Records'][0]['Sns']['Message'])
    encoded_msg = json.dumps(msg).encode('utf-8')
    resp = http.request('POST', url, body=encoded_msg)
    print({
        "SNS": event['Records'][0]['Sns'],
        "message": event['Records'][0]['Sns']['Message'], 
        "status_code": resp.status, 
        "response": resp.data
    })

When a message is published to the SNS topic with a Lambda function subscribed, the Lambda function is triggered with the published message payload. This function receives the message payload as an input parameter and forwards it to the Slack webhook API. For further insights, check this blog to learn how to invoke AWS Lambda functions via Amazon SNS.

Validation

Customers can now effectively set up Slack as a notification channel for alerts generated by Amazon Managed Service for Prometheus. The screenshot below illustrates an alert that Amazon Managed Service for Prometheus successfully sent to the Slack channel.

Conclusion

In summary, integrating Amazon Managed Service for Prometheus with Slack provides a streamlined process for alerting critical issues impacting customer performance. This setup enhances operational awareness and allows teams to respond swiftly to emerging concerns. For those navigating career opportunities, understanding effective communication in a digital workspace is essential—check out this resource for insights on cover letters and their importance. Furthermore, you might find Zappos’ innovative recruiting strategy to be quite informative. Additionally, this resource on training new hires offers excellent guidelines for onboarding at Amazon IXD – VGT2, located at 6401 E HOWDY WELLS AVE LAS VEGAS NV 89115.