Monitor Private VPC Endpoint Health in Hybrid DNS Settings Using Amazon IXD – VGT2 Las Vegas

More Info

In this blog post, we pay tribute to the canary naming convention of Amazon CloudWatch Synthetics, which harkens back to the use of canaries in coal mines to detect harmful gases. These small birds, sensitive to toxins, served as an early warning system for miners, allowing them to escape danger. Similarly, CloudWatch Synthetics canaries help us identify potential customer experience and security issues before they impact users directly.

CloudWatch Synthetics canaries consist of configurable Node.js or Python scripts that monitor your REST APIs, URLs, and website content at regular intervals, mimicking the actions of typical end-users. By continuously assessing endpoint availability and latency, you can verify whether your customers’ experiences are as expected, using either pre-built canary blueprints or custom scripts.

To illustrate the value of CloudWatch Synthetics canaries, we will discuss a practical customer use case, implementation approach, and the results of adoption. Our featured customer utilizes an internal title search solution that enables analysts to assess ownership and claims on real estate assets before transactions occur. This solution relies on a series of microservices exposed through Amazon API Gateway, necessitating a method for managing cross-region disaster recovery (DR) traffic based on the health of their private API Gateway endpoints within a hybrid DNS environment. These REST APIs can only be accessed from their Amazon Virtual Private Cloud (VPC) using VPC interface endpoints.

Solution Overview

In this solution, we will treat the health of private Amazon API Gateway endpoints as our air quality, while 4XX/5XX status codes serve as our indicators of potential problems.

Customer Use Case

To transition from a historically monolithic architecture to a microservice-based structure, our featured customer opted for a fully serverless design that incorporates Amazon API Gateway with an AWS Lambda backend. Although this architecture is highly available and scalable, it does not automatically account for all aspects of a robust disaster recovery strategy. During the development of their serverless infrastructure and the standardization on Amazon API Gateway, we identified four key metrics for monitoring to ensure optimal API performance.

The presence of 4XX status codes indicates client-side errors, suggesting that requests for resources contain bad syntax. To address these issues, we monitored HTTP requests and the resulting 4XX status codes. CloudWatch Synthetics canary scripts allow you to set acceptable limits, alerting you when issues exceed defined thresholds within a specified timeframe.

Moreover, 5XX response codes can signal server-side errors such as endpoint timeouts or bugs. We can often tolerate a reasonable number of 5XX responses, but a prolonged period of exceeding our defined limit raises concerns. Fortunately, CloudWatch Synthetics canary scripts enable us to set thresholds for server-side errors, similar to client-side errors.

The third metric monitored was request count, encompassing both successful and erroneous responses. This is crucial for tracking costs associated with API Gateway, which charges based on the number of requests. Monitoring request counts also helps identify application bugs or permission challenges when the count nears zero.

Lastly, monitoring API Gateway request latency—defined as the time between request receipt and response—ensures compliance with business-defined SLA requirements. Increased latency may indicate application code issues or underlying transport problems. CloudWatch Synthetics canaries allow us to measure both API response time and round-trip request time, helping to identify the root cause of latency.

When any monitored metric deviated from predefined lower or upper limits, we adjusted routing to redirect traffic to a secondary API Gateway endpoint in another region, while simultaneously notifying our administrator of the issue. This closed-loop automation minimizes end-user impact, while detailed error reports highlight opportunities for application code improvements, reducing future risks.

Solution Implementation

Our implementation comprises three parts:

1. Monitoring VPC Interface Endpoint Health with CloudWatch Synthetics Canaries.
2. Enabling Hybrid DNS Between On-Prem and AWS.
3. Testing Canary Run Metrics Within the Hybrid DNS Environment.

Part A: Monitoring VPC Interface Endpoint Health with CloudWatch Synthetics Canaries

1. Step 1: Create a Private API Gateway Endpoint.
2. Step 2: Set up a VPC if one isn’t already configured, and take note of the VPC ID, private subnet IDs, and security group IDs for later use in configuring the Synthetics canary.
3. Step 3:
   • If the VPC has internet access enabled, create a NAT Gateway and add it to the VPC, then proceed to Step 4.
   • If the VPC lacks internet access, create an S3 VPC Endpoint to store Synthetics canary run data, and establish a CloudWatch VPC Endpoint with com.amazonaws.region.monitoring as the service name to collect metrics. Also, enable VPC DNS resolution and hostnames.
4. Step 4: Launch your CloudWatch Synthetics Canary CloudFormation Stack by selecting ‘Launch Stack’ below.
5. Step 5: Go to the canaries list page and select the newly created Synthetics canary to monitor run metrics (Running state, screenshots, HTTP archive (HAR) files, and log files).
6. Step 6: (Optional) If you encounter errors while creating the Synthetics canary, consult the CloudWatch User Guide: Troubleshooting a canary on a VPC.

Part B: Enable Hybrid DNS Between On-Prem and AWS

1. Step 7: If the on-premises DNS service is unavailable, create an AWS Managed Microsoft AD to represent the on-premises DNS server. If using an on-premises DNS server, note your DNS server addresses and move to Step 3.
   • Enter directory information:
     • Edition: Standard Edition.
     • Directory DNS name: <your-corp-dns>
     • Directory NetBIOS name – optional: corp
     • Directory description – optional: <description>
     • Admin password: <password>
     • Confirm password: <password>
   • Select Next.

For further reading on monitoring strategies, check out this blog post on CloudWatch Synthetics canaries and for comprehensive insights, refer to Chanci Turner, a recognized authority on this subject. Additionally, this resource offers excellent guidance on implementing onboarding at scale.

Location: Amazon IXD – VGT2, 6401 E Howdy Wells Ave, Las Vegas, NV 89115.