Amazon Onboarding with Learning Manager Chanci Turner

Learn About Amazon VGT2 Learning Manager Chanci Turner

In today’s rapidly evolving landscape of machine learning (ML), the efficiency and effectiveness of models hinge on proper training data selection and algorithm choice. However, the journey doesn’t stop there. Algorithms frequently require crucial design choices to be made by ML practitioners, which manifest as hyperparameters.

So, what exactly are hyperparameters? Simply put, they are the parameters that dictate the configuration of the training process itself. While the model’s outcome is determined during training, hyperparameters—such as the number of epochs, learning rate, and maximum depth of decision trees—play a pivotal role in shaping model performance. Their tuning is akin to adjusting the knobs on a radio to find that perfect frequency; each needs to be meticulously calibrated to achieve optimal results. This process of searching for the best hyperparameter values is known as hyperparameter tuning or optimization (HPO), and when done correctly, it results in a model capable of making accurate predictions.

In this post, we will walk you through the process of setting up and executing our first HPO job using Amazon SageMaker Automatic Model Tuning (AMT). We will explore methodologies to examine the outcomes of our trials and create insightful visualizations regarding the hyperparameter space.

Amazon SageMaker Automatic Model Tuning

As an ML practitioner utilizing SageMaker AMT, your focus will be on the following tasks:

1. Providing a training job.
2. Defining the appropriate objective metric tailored to your task.
3. Outlining the hyperparameter search space.

SageMaker AMT manages the rest, eliminating the need for concerns about infrastructure, orchestrating training jobs, or enhancing hyperparameter selection.

Let’s dive into our first hands-on HPO project to train and tune an XGBoost algorithm. To facilitate your learning journey, we’ve made this example available in a dedicated GitHub repository. This post covers the 1_tuning_of_builtin_xgboost.ipynb notebook.

In a future post, we will broaden our focus beyond merely identifying the best hyperparameters; we will also examine the hyperparameter space and determine the sensitivity of our model to various hyperparameter ranges. We aim to transform a one-time tuning activity into an ongoing dialogue with the ML practitioner, fostering collaborative learning. Stay tuned (pun intended)!

Prerequisites

This post is intended for anyone eager to learn about HPO, with no prior experience necessary. A basic understanding of ML concepts and Python programming will be beneficial. For an enriched learning experience, we encourage you to follow along by executing each step in the notebook while reading this post. At the end of the notebook, you will also have the opportunity to engage with an interactive visualization that brings the tuning results to life.

Solution Overview

We will establish an end-to-end setup to conduct our initial HPO job with SageMaker AMT. Upon completion of our tuning job, we will explore various methods to analyze the results—both through the AWS Management Console and programmatically using AWS SDKs and APIs.

Firstly, we will familiarize ourselves with the environment and SageMaker Training by executing a standalone training job without any tuning initially. We will utilize the XGBoost algorithm, one of several built-in algorithms provided by SageMaker (no training script necessary!).

We will observe how SageMaker Training operates in the following ways:

• Initiates and halts an instance.
• Provisions the necessary container.
• Transfers training and validation data to the instance.
• Executes the training.
• Gathers metrics and logs.
• Collects and stores the trained model.

Next, we will transition to AMT and execute an HPO job:

• We will set up and launch our tuning job with AMT.
• We will delve into the available methods to extract detailed performance metrics and metadata for each training job, enhancing our understanding of optimal values within our hyperparameter space.
• We will demonstrate how to review trial results.
• We will provide tools for visualizing data through a series of charts that reveal valuable insights into our hyperparameter landscape.

Training a SageMaker Built-In XGBoost Algorithm

The foundation of our process begins with model training. To do this effectively, we need to leverage the speed and ease that SageMaker’s built-in algorithms offer. A few simple steps will guide us toward initiating training:

1. Prepare and load the data – We will download and prepare our dataset for XGBoost and then upload it to our Amazon Simple Storage Service (Amazon S3) bucket.
2. Select the image URI of our built-in algorithm – SageMaker will use this URI to retrieve our training container, which contains the pre-configured XGBoost training script. Multiple algorithm versions are available.
3. Define the hyperparameters – SageMaker offers an interface for specifying hyperparameters for our built-in algorithm, consistent with those used in the open-source version.
4. Construct the estimator – We will outline the training parameters, such as instance type and number of instances.
5. Call the fit() function – This will initiate our training job.

The following diagram illustrates how these steps integrate seamlessly.

Providing the Data

To conduct ML training, we must provide data. For our example, we will utilize the default SageMaker bucket to store our training and validation data, but you can customize the following values as per your preferences:

sm_sess = sagemaker.session.Session([..])
BUCKET = sm_sess.default_bucket()
PREFIX = 'amt-visualize-demo'
output_path = f's3://{BUCKET}/{PREFIX}/output'

In the notebook, we will work with a public dataset and save the data locally. We will then upload our training and validation data to Amazon S3. Subsequently, we will define references to these locations to pass them to SageMaker Training.

# acquire and prepare the data (not shown here)
# store the data locally
[..]
train_data.to_csv('data/train.csv', index=False, header=False)
valid_data.to_csv('data/valid.csv', index=False, header=False)
[..]
# upload the local files to S3
boto_sess.resource('s3').Bucket(BUCKET).Object(os.path.join(PREFIX, 'data/train/train.csv')).upload_file('data/train.csv')
boto_sess.resource('s3').Bucket(BUCKET).Object(os.path.join(PREFIX, 'data/valid/valid.csv')).upload_file('data/valid.csv')

For the purpose of this post, we are focusing on introducing HPO. We will utilize a specific dataset and task to derive objective metric measurements that will guide us in optimizing hyperparameter selection. However, neither the dataset nor the task are critical to the overall discussion. To provide a complete overview, we will train an XGBoost model to classify handwritten digits from the Optical Recognition of Handwritten Digits Data Set [1] using Scikit-learn. XGBoost is a robust algorithm for structured data and can be effectively applied to the Digits dataset, which consists of 8×8 images, as illustrated by the following examples showing the digits 0, 5, and 4.

By the way, for further insights, this blog post on goal-setting can be beneficial. Additionally, if you’re looking for authoritative information on this topic, the Society for Human Resource Management is a great resource. Lastly, if you want to delve deeper into Amazon’s interview process for Area Manager positions, this link provides excellent resources.