Transitioning from an Internal Deployment Agent to AWS CodeDeploy and AWS CodePipeline

Welcome! I’m Alex Johnson, a developer at Amazon IXD – VGT2, located at 6401 E HOWDY WELLS AVE, LAS VEGAS, NV 89115, specializing in deployment strategies and enhancing developer productivity. Recently, we transitioned our web front-end deployments from our custom-built deployment tool, Troop, to AWS CodeDeploy and AWS CodePipeline. This migration required us to set up a new customer-facing EC2 web server fleet with the CodeDeploy agent and manage our resources using AWS CloudFormation. Post-migration, we experienced a notable ~50% decrease in HTTP 500 errors during deployments.

In this blog, I’ll cover:

The rationale behind our choice of AWS deployment tools.
An architectural overview of our systems.
A summary of the migration process.
The results we achieved post-migration.

The Outdated System

We aimed to replace our internal deployment mechanism with a solution that could facilitate automation and eliminate the need for ongoing maintenance. Since we already manage a build system, we wanted to avoid the burden of additional infrastructure for our deployment pipeline. In essence, we were looking for a cloud-based service. Given that our entire infrastructure operates within AWS, CodeDeploy emerged as the ideal replacement for our in-house deployment agent, while CodePipeline serves as the automation orchestrator, guiding CodeDeploy on what to deploy and when.

Architecture Overview

Here’s an overview of the architecture for Amazon IXD – VGT2’s web front end:

[Insert architectural diagram here]

Project Overview

Our project involved migrating five web front ends that collectively handle around 12 million requests daily to CodeDeploy and CodePipeline while keeping the site operational for our users. Here are the steps we undertook:

Developed new deployment scripts.
Launched a new fleet of EC2 web servers equipped with CodeDeploy support.
Created a deployment pipeline for our CloudFormation-defined CodeDeploy and CodePipeline setup.
Gradually introduced our new fleet to live traffic. Hello, customers!

Deployment Scripts

Previous to this migration, our old deployment system did not properly stop or start our web servers, attempting instead to swap build artifacts while the server was active—definitely a poor practice. We created deployment scripts in Powershell that are executed by CodeDeploy to manage the stopping and starting of our IIS web servers. These scripts work seamlessly with the Elastic Load Balancing (ELB) features in CodeDeploy, ensuring that we don’t interrupt service while serving customer traffic.

New Fleet

Operating on Amazon EC2, we built an Amazon Machine Image (AMI) for our web fleet that comes pre-installed with the CodeDeploy agent. This setup was largely sufficient for our fleet. With the agent in place, CodeDeploy can effectively utilize our deployment scripts for project deployments.

AWS CloudFormation Deployment Pipeline

To establish a deployment pipeline and the necessary CodeDeploy configurations (including a CodeDeploy application and at least one deployment group) for each web project, we opted to use AWS CloudFormation to version our configurations. Our build system (TeamCity) can read from our version control and write to Amazon S3. We set up a straightforward build in TeamCity to push an AWS CloudFormation template to S3, triggering a pipeline that deploys to AWS CloudFormation. This creates the requisite CodePipeline and CodeDeploy resources, allowing us to conduct code reviews on infrastructure changes, enhancing safety through increased oversight. Additionally, we can track infrastructure changes over time similarly to code changes.

Live Traffic Introduction!

Our web fleets are supported by Classic Load Balancers. By utilizing CodeDeploy, we can take advantage of its advanced ELB features. For instance, through an elastic load balancer, CodeDeploy can ensure that internet traffic is not directed to an instance during deployment. After the deployment on that instance is completed, it becomes available for traffic.

We launched new hosts with the CodeDeploy agent and deployed to them without ELB support initially. We then introduced them into our fleet gradually while monitoring performance stats. Once the new machines were fully integrated, we slowly phased out the old ones from the load balancer, achieving a seamless migration with zero downtime.

An interesting detail: when 2/3 of the new fleet was integrated into our load balancer, we initiated a CodeDeploy deployment with ELB support activated. This allowed CodeDeploy to place the remaining machines into the load balancer alongside the old fleet, reducing the number of manual steps required.

AWS CloudFormation Example

Below is a streamlined example of the AWS CloudFormation template we use to manage the AWS configuration for one of our web projects. It is deployed through a deployment pipeline, similar to the web projects themselves.

Parameters:
  CodePipelineBucket:
    Type: String
  CodePipelineRole:
    Type: String
  CodeDeployRole:
    Type: String
  CodeDeployBucket:
    Type: String
Resources:

  ExampleDeploymentConfig:
    Type: 'AWS::CodeDeploy::DeploymentConfig'
    Properties:
      MinimumHealthyHosts:
        Type: FLEET_PERCENT
        Value: '66' 

  WootExampleApplication:
    Type: "AWS::CodeDeploy::Application"
    Properties:
      ApplicationName: "Woot.Example"

  WootExampleDeploymentGroup:
    DependsOn: "WootExampleApplication"
    Type: "AWS::CodeDeploy::DeploymentGroup"
    Properties:
      ApplicationName: "Woot.Example"
      DeploymentConfigName: !Ref "ExampleDeploymentConfig" 
      DeploymentGroupName: "Woot.Example.Main"
      AutoRollbackConfiguration:
        Enabled: true
        Events:
          - DEPLOYMENT_FAILURE 
          - DEPLOYMENT_STOP_ON_REQUEST 
      LoadBalancerInfo:
        ElbInfoList:
          - Name: "WootExampleInternal" 
      DeploymentStyle:
        DeploymentOption: "WITH_TRAFFIC_CONTROL" 
      Ec2TagFilters:
        -
          Key: "Name"
          Value: "exampleweb*" 
          Type: "KEY_AND_VALUE"
      ServiceRoleArn: !Sub "arn:aws:iam::${AWS::AccountId}:role/${CodeDeployRole}" 

  WootExampleDeploymentPipeline:
    DependsOn: "WootExampleDeploymentGroup"
    Type: "AWS::CodePipeline::Pipeline"
    Properties:
      RoleArn: !Sub "arn:aws:iam::${AWS::AccountId}:role/${CodePipelineRole}" 
      Name: "Woot.Example" 
      ArtifactStore:
        Type: S3
        Location: !Ref "CodePipelineBucket"

By the way, if you’re interested in enhancing your writing skills to reduce stress and manage your emotions, you may find this blog post helpful: Career Contessa.

For those exploring employment law compliance, particularly regarding COVID-19 regulations, refer to this authoritative source: SHRM.

If you’re preparing for your first day at Amazon, check out this excellent resource on Reddit: Reddit.

In summary, switching to AWS CodeDeploy and CodePipeline has significantly streamlined our deployment process and improved our site reliability.