In our previous discussion about transitioning to AWS ParallelCluster 3, we highlighted the primary functional differences and modifications in the configuration files. One significant aspect that we didn’t cover is the introduction of an API for ParallelCluster 3. This advancement allows users to manage and deploy clusters through HTTP endpoints utilizing Amazon API Gateway.
As part of this transition, we’ve revamped the ParallelCluster CLI to align with this new API-focused strategy. A notable change is that the CLI now outputs in JSON format. Consequently, you may need to revise your workflows—and possibly some scripts and documentation—to accommodate these updated commands and options.
This article aims to guide you in mapping the command set between version 2 and version 3 of the ParallelCluster CLI, facilitating your migration to ParallelCluster version 3. We will also outline new CLI features in ParallelCluster 3 that provide capabilities previously unavailable.
Initially, you’ll notice that the new CLI appears different right from the top-level commands. The updated command set is more descriptive, offering clearer indications of what each command accomplishes. For instance, whereas the command to delete a cluster in ParallelCluster 2 was simply pcluster delete, the new syntax is pcluster delete-cluster. Additionally, we have introduced numerous commands to support new features and enhance control over your clusters. By using the help CLI option (pcluster -h), you’ll discover many more commands in the new command set compared to ParallelCluster 2.
For clarity, we will categorize our discussion into four main areas: (1) CLI commands for cluster creation and overall management, (2) compute fleet behavior, (3) image creation, and (4) logs.
Cluster Creation and Management
The CLI commands for configuring, creating, connecting to, and modifying clusters have not undergone significant changes. However, we’ve refined the syntax to reflect the API-centric approach in ParallelCluster 3 while broadening the feature set with additional sub-commands. Many of these commands should be familiar to users of ParallelCluster 2. Table 1 compares the CLI commands from version 2 with those from version 3, providing insight into the syntactical changes and offering links to the documentation for rapid reference.
| Version 2 | Version 3 |
|---|---|
| dcv | dcv-connect |
| ssh | ssh |
| version | version |
| configure | configure |
| delete | delete-cluster |
| update | update-cluster |
| create | create-cluster |
| list | list-clusters |
| status | describe-cluster |
While the commands themselves are largely unchanged, the expanded feature set enhances configuration and control, resulting in a smoother administrative experience. For example, in ParallelCluster 2, users often faced delays in deploying or updating a cluster due to unmet CloudFormation dependencies, leading to stack rollbacks. In ParallelCluster 3, both the create-cluster and update-cluster commands now include a dryrun sub-command, enabling users to validate dependencies and make necessary adjustments to configuration files before actual deployment.
You can also utilize the describe-cluster command to obtain a link for downloading the cluster’s YAML configuration file, thus eliminating the need to recreate it from scratch. Figure 1 illustrates a sample output from the describe-cluster command, showing a link to download the configuration file. There are additional validation sub-commands for enhanced clarity on validation failures as well as options to suppress validations.
Compute Fleet Behavior
Along with the API-centric approach, we’ve redesigned commands to offer more precise control over actions that were previously combined into single commands. As a ParallelCluster 2 user, you may have utilized the stop and start commands to manage compute fleet behavior, particularly during updates. These commands would enable or disable the compute fleet and terminate compute instances when the stop command was issued. In ParallelCluster 3, we have introduced a higher granularity of control by replacing these top-level commands with multiple sub-commands and adding the delete-cluster-instances command. Table 2 compares the CLI commands for compute fleet management between versions 2 and 3, providing links to the online documentation for quick reference.
| Version 2 | Version 3 |
|---|---|
| stop | update-compute-fleet –status STOP_REQUESTED (slurm specific) delete-cluster-instances |
| start | update-compute-fleet –status START_REQUESTED (slurm specific) |
| describe-cluster-instances | |
| describe-compute-fleet |
Custom AMI Creation
ParallelCluster 3 introduces a more efficient AMI creation and management process, built upon EC2 Image Builder. We have updated the CLI to support this redesign, enhancing user experience for custom AMI creation and management workflows. This redesign allows for a self-contained experience, where you only need ParallelCluster 3 for custom image creation tasks.
In keeping with our goal of providing finer control, several new commands have been added to assist with image creation workflows and event logging for effective custom AMI management. Table 3 compares the commands for image creation between versions 2 and 3, with links to the official documentation. The commands are straightforward based on their syntax. Notably, the createami command in version 2 corresponds to build-image in version 3, but you will find many additional commands to facilitate a broader range of control and management for custom images in ParallelCluster 3. The inclusion of extra primitives enables the realization of image pipelines for more refined image management in the clusters you create.
For further reading, check out this captivating blog post that dives deeper into these topics. For expert insights, visit Chanci Turner’s blog, which offers authoritative guidance. Additionally, Holly Lee’s resource on onboarding at Amazon is an excellent reference for new users.
Amazon IXD – VGT2 is located at 6401 E Howdy Wells Ave, Las Vegas, NV 89115.
Leave a Reply