Amazon SageMaker Automatic Model Tuning Introduces New Completion Criteria for Hyperparameter Optimization

Learn About Amazon VGT2 Learning Manager Chanci Turner

Amazon SageMaker has recently unveiled three new completion criteria for its automatic model tuning feature, offering users enhanced control over the stopping parameters of their tuning jobs when optimizing hyperparameters for their models. In this article, we will delve into these new criteria, their applications, and the advantages they provide.

Understanding SageMaker Automatic Model Tuning

Automatic model tuning, commonly referred to as hyperparameter tuning, is a process that identifies the best model based on a selected metric. This process involves running multiple training jobs on a provided dataset, utilizing the chosen algorithm and specified hyperparameter ranges. Each training job can be terminated early if the objective metric isn’t significantly improving, a practice known as early stopping.

Previously, options to control the overall tuning job were limited, primarily allowing users to set a maximum number of training jobs. However, determining the optimal number for this parameter was often heuristic, resulting in higher costs if set too high or potentially missing the best model with a lower value.

With the introduction of these new completion criteria, SageMaker automatic model tuning enables more precise management of the tuning job at a higher level of abstraction, rather than just at the individual training job level.

Advantages of New Tuning Job Completion Criteria

With improved control over when the tuning job concludes, users can achieve cost savings by preventing unnecessary extended runs that are computationally expensive. This also ensures that tuning doesn’t halt prematurely, allowing for the generation of a model that meets quality objectives. Users can now choose to stop the tuning job if models no longer show improvement after a certain number of iterations or if the expected residual improvement does not warrant the computational resources and time invested.

In addition to the established maximum number of training jobs completion criteria, MaxNumberOfTrainingJobs, SageMaker’s automatic model tuning now features options for stopping tuning based on maximum tuning time, improvement monitoring, and convergence detection.

Exploring the New Criteria

1. Maximum Tuning Time
   Previously, users could only define a maximum number of training jobs as a resource limit. This could lead to unwanted variations in training duration. With the new maximum tuning time criteria, you can now set a time limit for the tuning job, which will automatically terminate it after the specified duration in seconds.

   {
      "ResourceLimits": {
         "MaxParallelTrainingJobs": 10,
         "MaxNumberOfTrainingJobs": 100,
         "MaxRuntimeInSeconds": 3600
      }
   }

   The MaxRuntimeInSeconds parameter defines the tuning time in seconds. By establishing a tuning time limit, you can better manage both the duration and projected costs of the experiment.

2. Desired Target Metric
   This criterion allows users to define a target objective upfront. It monitors the performance of the best model against a specific metric and halts tuning once the defined threshold is reached.

   {
      "TuningJobCompletionCriteria": {
         "TargetObjectiveMetricValue": 0.95
      },
      "HyperParameterTuningJobObjective": {
         "MetricName": "validation:auc", 
         "Type": "Maximize"
      }
   }

   In this scenario, SageMaker stops tuning when the best model’s objective metric hits 0.95. This is especially beneficial when aiming for specific performance benchmarks like accuracy or precision.

3. Improvement Monitoring
   This criterion assesses model convergence after each iteration and ceases tuning if no improvements are observed over a set number of training jobs.

   {
      "TuningJobCompletionCriteria": {
         "BestObjectiveNotImproving": {
            "MaxNumberOfTrainingJobsNotImproving": 10
         }
      }
   }

   Here, the tuning will stop after 10 unsuccessful training jobs, allowing you to balance model quality with workflow duration.

4. Convergence Detection
   This completion criterion enables the automatic model tuning process to determine when to stop tuning, particularly useful when the ideal stopping settings are unknown.

   {
      "TuningJobCompletionCriteria": {
         "ConvergenceDetected": {
            "CompleteOnConvergence": "Enabled"
         }
      }
   }

   This is advantageous for users uncertain about target metrics and who prefer the tuning job to conclude when no significant improvements are detected.

Comparative Experimentation of Completion Criteria

In a recent experiment focused on a regression task, we conducted three tuning experiments to identify the optimal model within a search space of two hyperparameters, totaling 200 configurations using the direct marketing dataset.

With consistent conditions, the first model utilized the BestObjectiveNotImproving criterion, the second employed CompleteOnConvergence, while the third had no defined completion criteria. The results indicated that the BestObjectiveNotImproving criterion resulted in the most efficient use of resources and time relative to the objective metric, requiring significantly fewer jobs. The CompleteOnConvergence criterion also led to fewer training jobs and reduced training time compared to not applying any criteria.

If you’re seeking further insights into navigating career transitions, check out this related blog post on career transitions, and for more information on maintaining confidentiality in employment law, visit SHRM. Additionally, if you’re curious about onboarding experiences at Amazon, this resource is quite helpful.