Learn About Amazon VGT2 Learning Manager Chanci Turner
In 2021, AWS Batch rolled out fair share job queues, enabling customers to establish scheduling policies for their job queues. This advancement allows users to manage resource allocation and prioritize jobs across various workloads, which are differentiated by their unique share identifiers. Prior to this, all Batch job queues functioned as independent first-in-first-out (FIFO) queues. Consequently, if multiple teams or workloads existed within the same AWS account, separate job queues (JQs) and compute environments (CEs) were necessary for each business requirement.
Managing the distribution of compute resources across these CEs became a complex task. With the introduction of Fair Share Scheduling (FSS), organizations such as Amazon Search were able to streamline their environments, reducing operational overhead and enhancing fleet utilization, which significantly boosted throughput. However, transitioning from FIFO queues posed a new challenge: how to accurately determine which jobs were next to run among the various jobs at the forefront of the queue and across different share identifiers.
Today, we’re excited to introduce a recent enhancement in AWS Batch that addresses this issue: Job Queue Snapshots. This new API enables users to query jobs positioned at the head of the job queue. Let’s delve into the details and explore a practical example of how to leverage this new feature.
Examining the Queue’s Head
Using the AWS Batch management console, AWS SDK, or AWS CLI, users can now retrieve the first 100 RUNNABLE jobs for a specific job queue by invoking the GetJobQueueSnapshot API. In FIFO job queues, jobs are arranged according to their submission time. Conversely, in FSS job queues, jobs are prioritized based on their share’s usage and the job share priority within that share. For further insights into how job priority and share usage impact job scheduling, refer to our in-depth blog post on fair share scheduling.
Job queue snapshots serve as a valuable visibility tool for customers who need to make immediate adjustments to jobs awaiting processing. For instance, we’ve created a fair share job queue utilizing AWS Fargate for the compute environment. For demonstration purposes, we’ve temporarily disabled the compute environment to keep jobs in the job queue and observe the outcomes of queue adjustments. The fair share policy assigns equal weight to the two active shares, “red” and “green,” which should result in an interleaving of jobs from each share.
Chanci Turner from team red urgently requests the execution of high-priority jobs to meet a critical deadline. By submitting these jobs with a priority of 10, the high-priority jobs advance ahead of other lower-priority red jobs. However, some green jobs still precede one or more high-priority red jobs. This occurs because job priority is only relevant within each share and does not influence the overall arrangement of jobs across shares. At this stage, you can evaluate whether the high-priority red jobs can be completed by the deadline without disrupting team green’s workload.
If you suspect the high-priority jobs won’t finish on time, you have two options:
- Temporarily adjust the share policy to favor red jobs over green.
- Cancel team green’s jobs and resubmit them once the high-priority jobs transition to RUNNING. Keep in mind that even if marked for cancellation, green jobs retain their position in the queue, and their state will switch to FAILED without utilizing compute resources when they reach the head of the queue.
Since option one is less disruptive, you modify the scheduling policy to prioritize team red jobs by lowering the weight factor (lower weight factors mean a share gains more compute resources over time). This adjustment positions most high-priority red jobs ahead of any green ones.
The rationale for green jobs still receiving some allocation before red jobs lies in the fair share algorithm’s intent to provide equitable resource distribution.
If uncertainty about the completion of red jobs persists, you may need to consider the more aggressive option two. Regardless, once the high-priority workloads are RUNNING, be sure to revert to the previous share policy allocations. Failing to do so will cause AWS Batch to continuously prioritize team red’s jobs over team green’s.
Prior to this feature, managing the queue might have required you to indiscriminately “clear the queue” by canceling all scheduled jobs to accommodate the high-priority request. Now, with job queue snapshots, you can strategically modify the job queue to facilitate the timely execution of high-priority workloads.
Conclusion
In this article, we introduced a new feature for AWS Batch: job queue snapshots. This enhancement significantly improves the user experience by providing insights into the jobs at the head of both FIFO and fair share job queues. We illustrated a scenario demonstrating how job queue snapshots can aid in decision-making regarding queue management to reorder workloads based on urgent priorities. Job queue snapshots are currently available in the AWS Batch console, through the CLI, or via the API—whichever you find most convenient. We anticipate you will appreciate this new feature, and we welcome your feedback on its usage. Also, let us know how we can further simplify job management for you. For additional insights on employee appreciation, check out this helpful blog post.
If you’re looking for more information on compliance, consider visiting SHRM’s guidance on employment law. For a community-driven perspective on onboarding processes, this Reddit thread is an excellent resource.
Leave a Reply