Learn About Amazon VGT2 Learning Manager Chanci Turner
For organizations utilizing containerized workloads on their own Kubernetes clusters, Amazon EKS Anywhere (Amazon EKS-A) offers a robust solution. Users often seek clear guidance on how to effectively monitor their modern applications running on EKS-A. Leveraging AWS-managed open-source services like AWS Distro for OpenTelemetry (ADOT), Amazon Managed Service for Prometheus, and Amazon Managed Grafana enables users to alleviate the operational demands associated with observability infrastructure.
Amazon EKS-A curated packages are reliable, current, and compatible software supported by Amazon, enhancing your EKS-A cluster’s capabilities while minimizing the necessity for multiple vendor support agreements. The ADOT now includes support for EKS-A curated packages, acting as an OpenTelemetry collector that provides a vendor-neutral framework for receiving, processing, and exporting telemetry data. This functionality eliminates the need to deploy and manage various agents or collectors, as the ADOT Collector is an AWS-supported version of the OpenTelemetry Collector.
The Grafana Operator is a Kubernetes operator designed to simplify the management of Grafana instances within Kubernetes. With this operator, you can declaratively manage and create Grafana dashboards, data sources, and more across multiple instances in a scalable manner. The Grafana Operator now also supports the management of resources such as dashboards and data sources hosted on external platforms like Amazon Managed Grafana. By employing GitOps practices, Flux serves as a GitOps tool that automates the application deployment process on Kubernetes, continuously monitoring the state of a Git repository and implementing any changes to the cluster. Consequently, the Grafana Operator allows us to utilize GitOps methodologies alongside CNCF projects like Flux to create and oversee the lifecycle of resources within Amazon Managed Grafana from an Amazon EKS-A cluster.
In this article, we will guide you on utilizing the ADOT EKS-A curated package, AWS-managed open-source services, and Grafana Operator to monitor your on-premises Kubernetes cluster.
Solution Overview
Solution Walkthrough
This solution begins by employing the ADOT EKS-A curated package to remotely write Prometheus-compatible metrics from your EKS-A cluster to Amazon Managed Service for Prometheus (AMP). Following that, we will implement GitOps techniques with Flux and the Grafana Operator to create and manage Grafana resources, such as dashboards and data sources, that are hosted on external environments like Amazon Managed Grafana. This setup will allow you to visualize metrics from your on-premises Kubernetes cluster.
Prerequisites
Ensure the following prerequisites are fulfilled:
- A Linux-based host machine, using either an Amazon EC2 instance, a Cloud9 instance, or a local machine that has access to your AWS account.
- Verify that your AWS account has access to EKS Anywhere curated packages. If not, refer to EKS Anywhere curated package management for subscription details.
- Admin access to the EKS Anywhere cluster must be configured from the host machine.
- IAM Roles for Service Account (IRSA) should be set up on the EKS Anywhere cluster.
- An existing Amazon Managed Grafana Workspace in your AWS account.
- Install the following tools on your host machine:
- AWS CLI version 2 for interacting with AWS services using command-line operations.
- Helm for deploying and managing Kubernetes applications.
- kubectl for communication with the Kubernetes API server.
- eksctl and eksctl anywhere for creating and managing EKS Anywhere clusters.
- Git for cloning the necessary source repository from GitHub.
- curl for making HTTP requests.
- envsubst for substituting environment variables in the shell.
Setup Environment
Set the following environment variables:
export EKSA_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
export EKSA_REGION="us-east-2"
export EKSA_CLUSTER_NAME="sample-cluster" # EKS Anywhere cluster name
export KUBECONFIG="./sample-cluster/sample-cluster-eks-a-cluster.kubeconfig" # absolute path of EKS Anywhere cluster kubeconfig file
export EKSA_OIDC_PROVIDER=<value of $ISSUER_HOSTPATH as configured in IRSA setup>
export EKSA_ADOT_NAMESPACE="observability"
export EKSA_ADOT_SERVICE_ACCOUNT="curated-adot-sa"
export EKSA_ES_SERVICE_ACCOUNT="external-secrets-sa"
export EKSA_AMP_WORKSPACE_ALIAS=${EKSA_CLUSTER_NAME}-AMP-workspace
export EKSA_AMG_WORKSPACE_NAME="amp-adot-grafana" # replace with name of your desired AMG workspace
export GO_API_KEY_SECRET_NAME="/eksa/amg-api-key"
Make sure the pod-identity-webhook is deployed in the observability namespace where ADOT will be implemented. If not, follow the IAM Roles for Service Accounts configuration steps to deploy it.
Setting Up Amazon Managed Service for Prometheus
In this section, we will deploy a curated ADOT package configured to write metrics to Amazon Managed Service for Prometheus (AMP). Start by creating an Amazon Managed Service for Prometheus workspace using the command:
aws amp create-workspace
--region ${EKSA_REGION}
--alias ${EKSA_AMP_WORKSPACE_ALIAS}
Next, set the following environment variables with the values from the Amazon Managed Service for Prometheus workspace you just created:
export EKSA_AMP_WORKSPACE_ID=$(aws amp list-workspaces
--region=${EKSA_REGION}
--alias ${EKSA_AMP_WORKSPACE_ALIAS}
--query 'workspaces[0].[workspaceId]'
--output text)
export EKSA_AMP_WORKSPACE_ARN=$(aws amp list-workspaces
--region=${EKSA_REGION}
--alias ${EKSA_AMP_WORKSPACE_ALIAS}
--region=${EKSA_REGION}
--query 'workspaces[0].[arn]'
--output text)
export EKSA_AMP_ENDPOINT_URL=$(aws amp describe-workspace
--region=${EKSA_REGION}
--workspace-id ${EKSA_AMP_WORKSPACE_ID}
--query workspace.prometheusEndpoint
--output text)
export EKSA_AMP_REMOTEWRITE_URL=${EKSA_AMP_ENDPOINT_URL}api/v1/remote_write
Subsequently, execute the steps to create an IAM role that provides fine-grained permissions to the AMP workspace, with the OIDC provider set as a trusted entity to assume this role.
# create a trust policy for the IAM role
curl -s https://raw.githubusercontent.com/aws-samples/containers-blog-maelstrom/main/eksa-adot-cp/templates/irsa-trust-policy-template.json
| envsubst > ./irsa-trust-policy.json
# create a permission policy for the IAM role
curl -s https://raw.githubusercontent.com/aws-samples/containers-blog-maelstrom/main/eksa-adot-cp/templates/amp-irsa-perm-policy-template.json
| envsubst > ./amp-irsa-perm-policy.json
# create an IAM role
existingRole=$(aws iam list-roles --query "Roles[?RoleName=='${EKSA_ADOT_SERVICE_ACCOUNT}-role'].RoleName" --output text)
if [ -z ${existingRole} ]; then
aws iam create-role --role-name ${EKSA_ADOT_SERVICE_ACCOUNT}-role
--assume-role-policy-document file://irsa-trust-policy.json
--query
To learn more about managing your career journey, check out this webinar by Maxie McCoy. Additionally, for insights on whistleblowing regulations, you can read about the Financial Services Authority in Dubai. Another excellent resource is this guide on Amazon’s new hire orientation.
Leave a Reply