Amazon Onboarding with Learning Manager Chanci Turner: Part 3 – Distributed Tracing Utilizing AWS X-Ray

Traces play a crucial role in capturing requests as they navigate through various services and components, which is vital for comprehending application workflows. They are instrumental in identifying bottlenecks and hotspots. In the context of microservices, traces illustrate the interactions within modern applications, assisting developers and operators in visualizing request paths for efficient issue resolution.

In the first installment of this series, we instrumented an ASP.NET application for metrics. The second installment concentrated on log implementation via Fluent Bit and Amazon OpenSearch Service. In this final segment, we will instrument an ASP.NET application deployed on a managed container service utilizing OpenTelemetry and AWS X-Ray.

While this discussion is centered on Amazon Elastic Container Service (Amazon ECS), the same principles can be applied to Amazon Elastic Kubernetes Service (Amazon EKS).

Solution Architecture Overview

This post leverages a sample ASP.NET application housed in the aws-samples GitHub repository. The application will operate within an Amazon ECS cluster using the Amazon EC2 launch type.

To gather and transmit traces, the AWS Distro for OpenTelemetry (ADOT) collector is set up as a service on AWS Fargate. The AWS Distro for OpenTelemetry (ADOT) is a secure, production-ready, AWS-supported distribution of the Cloud Native Computing Foundation (CNCF) OpenTelemetry project. OpenTelemetry (OTel) offers open-source APIs, libraries, and agents for collecting logs, metrics, and traces.

With ADOT, you can instrument your applications once and forward correlated logs, metrics, and traces to one or more observability backends such as Amazon Managed Service for Prometheus, Amazon CloudWatch, AWS X-Ray, Amazon OpenSearch, or any OpenTelemetry Protocol (OTLP) compliant backend. You can automatically instrument using available libraries, or you can manually instrument your application. The architecture diagram illustrates the solution’s framework, where a user interacts with the application, which then sends traces to AWS X-Ray.

Prerequisites

To test and deploy this solution, ensure that your system meets the following prerequisites:

.NET 8.0 Software Development Kit (SDK)
AWS Command Line Interface (AWS CLI)
AWS CDK v2
Docker
Visual Studio (or your preferred IDE)
AWS credentials for your AWS account configured locally
Git
AWS Organizations configured through AWS IAM Identity Center (the successor to AWS Single Sign-On).

Walkthrough

Once you have met the prerequisites, follow these steps to deploy the application and an ADOT Collector ECS service to collect and export traces.

Download the aws-dotnet-ecs-contains-observability sample, which features an instrumented sample application that generates traffic by making a simple HTTP GET call to aws.amazon.com. It also includes the AWS CloudFormation template to deploy the required resources into your account, detailed in the following section.

Prepare Your Environment

You will utilize the CloudFormation template located in the project files to deploy the following resources in your account:

Amazon ECS cluster to run ECS Services.
Elastic Load Balancer (ELB) configured as an Application Load Balancer (ALB) with a target group to distribute traffic to the application.
Amazon Managed Service for Prometheus workspace to store metrics from the ADOT collector.
Amazon OpenSearch Service to retain logs from the AWS Fluent Bit sidecar.
Amazon Managed Grafana workspace to aggregate and visualize metrics.
AWS Identity and Access Management (IAM) roles to grant Amazon ECS tasks write permissions to AWS X-Ray, Amazon Managed Service for Prometheus, and Amazon OpenSearch Service.

Navigate to the directory where you downloaded the sample code. In a command or terminal window, go to the BlogSample-ASPDotNetApp/BlogResources folder and run the following command to deploy the CloudFormation template. We advise deploying this in a test environment to avoid disrupting production workloads.

aws cloudformation deploy --template-file ./blog-cf-template.yml --stack-name blog-solution-stack --capabilities CAPABILITY_NAMED_IAM

Head over to the CloudFormation service in the AWS management console and check your new stack to view the outputs that will be utilized in the subsequent steps. The CloudFormation stack provisioning may take up to 15 minutes to complete.

Inspecting Our .NET Application

OpenTelemetry will be used to generate and collect traces from your application. This section focuses on examining OpenTelemetry integration within the sample .NET application.

The application is accessible as a container image on the Amazon Elastic Container Registry (Amazon ECR) public gallery for deployment.

Inspect the NuGet Packages

Navigate to the folder where you downloaded the sample application. Open the BlogSample-ASPDotNet.sln solution file using Visual Studio or any preferred IDE. The solution file can be found in the main directory of the sample application. Within the solution, locate the Startup.cs file, which contains essential packages and code for transmitting data to the ADOT collector. Should you wish to create your own solution, review the packages specified in the project file for reference.

Inspect the Code Used to Instrument the Application with OpenTelemetry

Merely adding the packages does not instrument your application. OpenTelemetry services must be initialized in your project during startup. In the sample application, this is accomplished in Startup.cs; however, ensure to implement this code where you set up your IServiceCollection.

The provided code establishes OpenTelemetry tracing for the .NET application. It specifies the service name and version, creates a resource with these attributes, and configures tracing using AddOpenTelemetry().WithTracing(). The tracing setup includes a console exporter for debugging, an OTLP exporter to send traces to a designated endpoint, a trace source, and linking the resource with the traces.

Furthermore, the code facilitates automatic instrumentation for AWS SDK operations, outgoing HTTP requests via HttpClient, and incoming HTTP requests and middleware within ASP.NET Core. As of October 2023, AWS X-Ray supports W3C trace IDs generated by OpenTelemetry and no longer requires the addition of a separate X-Ray-compatible trace ID when utilizing the ADOT collector version 0.34.0 or newer.

var serviceName = "sample-app";
var serviceVersion = "1.0";

var appResourceBuilder = ResourceBuilder.CreateDefault()
        .AddService(serviceName: serviceName, serviceVersion: serviceVersion);

// Configure essential OpenTelemetry settings, the console exporter, and instrumentation library
…

services.AddOpenTelemetry()
.WithTracing(tracerProviderBuilder =>
    
    tracerProviderBuilder
        .AddSource(serviceName)
        .SetResourceBuilder(appResourceBuilder.AddTelemetrySdk())
        .AddAWSInstrumentation()
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddConsoleExporter()
        .AddOtlpExporter(options =>
        {
            options.Protocol = OtlpExportProtocol.Grpc;
            options.Endpoint = new Uri(Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT"));
        }));

Deploy the ADOT Collector to Amazon ECS Fargate

This section elaborates on configuring the ADOT collector in a serious tone, maintaining a similar overall length.

As you embark on this journey to enhance your skills, consider exploring resources such as this blog post to learn from successful black women entrepreneurs. Also, for matters related to equal pay transparency, you can refer to this authoritative source. Lastly, if you’re interested in becoming a learning trainer, check out this excellent resource.

6401 E HOWDY WELLS AVE LAS VEGAS NV 89115, Amazon IXD – VGT2