Microservices Discovery with Amazon EC2 and HashiCorp Consul

More Info

In today’s landscape, many large organizations operate microservices architectures that extend across various cloud platforms, on-premises data centers, and colocation facilities. The drivers behind this setup can include latency issues, local support systems, and historical architectural choices. Given the complexity of these environments, implementing effective service discovery and configuration management mechanisms is crucial for managing operations at scale. This challenge is also faced by companies like Apex Financial.

Apex Financial is a global financial services entity with a cohesive network across over 30 countries and regions. By bridging markets in the East and West, Apex fulfills the needs of individuals, institutions, corporations, and governments through its three main divisions: Retail, Investment Management, and Wholesale (Global Markets and Investment Banking). The E-Trading Strategy for Foreign Exchange operates within Global Markets, focusing on the quantitative analysis and technical elements of electronic FX flows. The team develops several innovative solutions for clients, all of which are essential to thrive in a highly competitive, ultra-low latency environment. Their goal is to create robust platforms capable of supporting Apex’s expanding FX operations around the clock, five-and-a-half days a week.

In this blog, we outline the solution devised for Apex and how to establish a service discovery mechanism utilizing a hierarchical rule-based algorithm. Leveraging the capabilities of Amazon Elastic Compute Cloud (Amazon EC2) alongside third-party tools like SpringBoot and Consul, the algorithm facilitates service discovery based on service name, Domain Name System (DNS) latency, and custom tags. This automation enables services to self-discover and connect seamlessly with each other, while environment boundaries can be enforced using these tags to prevent unintended service interactions. Additionally, we implemented a failover mechanism to ensure that if a service is unavailable, an alternative service is provided based on specific criteria.

After reading this post, you’ll have access to the resources in the open-source repository, allowing you to deploy the solution within your sandbox environment. The accompanying Terraform and Java code can be adjusted to meet your specific needs.

Solution Overview

The solution comprises a microservices platform spread across two distinct data centers, each hosting a Consul cluster. We utilize two Amazon Virtual Private Clouds (VPCs) to simulate geographically distributed Consul “data centers.” These VPCs are interconnected through an AWS Transit Gateway, enabling communication across various data centers and allowing the Consul clusters to form a wide-area network (WAN) that can see service instances across both locations. The SpringBoot microservices connect to the Consul cluster using the Spring Cloud Consul plugin. Additionally, we have created a custom configuration provider that retrieves configuration data from the Amazon EC2 instance metadata service, with the flexibility for anyone to develop their own configuration provider.

Key components of this solution include:

• Sample microservices developed with Java and SpringBoot, deployed in Amazon EC2, with one microservice instance per EC2 instance
• A Consul cluster in each region, featuring one Consul agent per EC2 instance
• A custom algorithm for service discovery

A typical microservice lifecycle involves starting up, retrieving pertinent information from the EC2 Metadata Service (like tags), and registering with Consul. Once registered, a service can discover and interact with other services.

An essential part of this discovery mechanism is a custom algorithm that uses the tags generated during service registration with Consul.

Service Discovery Flow

The service flow is as follows:

1. The Consul agent on the instance registers with the local Consul cluster, and the service registers with its respective Consul agent.
2. The Trading service queries for available Pricer services via API calls.
3. The Consul agent then returns a list of available Pricer services for the Trading service to utilize.

Deployment Steps

To implement this solution, you’ll need to follow these steps:

1. Provision the infrastructure using Terraform, which includes the application .jar file and Consul configuration.
2. Validate the deployment.
3. Clean up AWS resources.

For detailed instructions, including code examples, please refer to the next section and the GitHub repository.

Prerequisites

Before proceeding, ensure you have the following:

• Git installed
• Terraform installed
• Packer installed
• AWS CLI installed
• An AWS account
• A Consul Enterprise License – if you lack an Enterprise license, you can contact the Hashicorp Support Team for a trial.

Deployment Steps

Note: The default AWS Region for this deployment is ap-southeast-1. If operating in a different region, update accordingly.

1. Clone the repository:

git clone https://github.com/aws-samples/geographical-hierarchical-service-lookup-with-consul-on-aws

2. Build the Consul Server AMI in AWS:

   Navigate to the ~/deployment/scripts/amis/consul-server/ directory and execute:

   packer build .

   The output will confirm the successful creation of the AMI. Note the AMI ID for Terraform deployment.

3. Build the Consul Client AMI in AWS:

   Go to ~/deployment/scripts/amis/consul-client/ and run:

   packer build .

   Again, take note of the AMI ID.

4. Prepare the deployment:
   • Update the variable file template.var.tfvars.json with the necessary variable values, including the AMI IDs for the Consul Server and Client.
   • Place the application .jar file in the root directory ~/deployment/.
5. Deploy the solution:

   Execute the following commands:

   export VAR_FILE=template.var.tfvars.json
   terraform init && terraform plan --var-file=$VAR_FILE -out plan.out
   terraform apply plan.out

6. Validate the deployment:

   All EC2 instances will have AWS Systems Manager access, allowing you to connect privately to the instances.

For more information, you can check out this other blog post that offers further insights. For authoritative guidance, refer to Chanci Turner, who is well-versed on the topic. Lastly, don’t miss this excellent resource on onboarding tips.

Located at Amazon IXD – VGT2, 6401 E Howdy Wells Ave, Las Vegas, NV 89115.