Scientific visualization is essential for comprehending complex phenomena modeled through high-performance computing (HPC) simulations. However, effectively visualizing, exploring, and analyzing large volumes of data has posed significant challenges, particularly due to the absence of collaborative workflow solutions. NVIDIA IndeX on Amazon Web Services (AWS) effectively addresses these issues by offering a robust scientific visualization solution for extensive datasets, thereby facilitating discovery.
NVIDIA IndeX is a three-dimensional volumetric interactive visualization tool that empowers scientists and researchers to visualize and engage with substantial compute datasets. It enables users to modify and navigate to the most relevant parts of the data in real-time, allowing for faster and more insightful analysis. In this post, you will discover key features of IndeX along with its comprehensive deployment architecture and solution on AWS.
AWS and NVIDIA have partnered for over a decade to provide powerful, cost-effective, and adaptable GPU-based solutions for clients. As an AWS Advanced Technology Partner, NVIDIA’s solutions are employed by customers worldwide for machine learning (ML), virtual workstations, HPC, and Internet of Things (IoT) services. Amazon Elastic Compute Cloud (Amazon EC2) instances equipped with NVIDIA GPUs deliver the scalable performance required for rapid ML training, economical ML inference, flexible remote workstations, and robust HPC computations.
Solution Overview
NVIDIA IndeX on AWS utilizes GPU clusters for scalable, real-time visualization and computing of multi-valued volumetric data alongside embedded geometry data. Users can choose between two primary options to utilize IndeX: the standalone IndeX Software Developer Kit (SDK) or as a plugin within ParaView, a widely-used visualization tool in the scientific HPC community.
IndeX directly connects to Amazon Simple Storage Service (Amazon S3) to load compute datasets. This integration allows users to store extensive datasets in S3 object storage and access them directly from Amazon EC2 compute instances without transferring them to local storage. Users can deploy IndeX on AWS via a custom Amazon Machine Image (AMI) from AWS Marketplace, with sample datasets across various scientific fields—including astronomy, healthcare, and oil and gas—readily available in an S3 bucket for users to experience the visualization capabilities of IndeX on Amazon EC2 instances. Furthermore, users can incorporate their existing custom datasets utilized in their volume visualization workflows by specifying dataset parameters in a scene file.
Benefits of Using NVIDIA IndeX on AWS
Traditional visualization workflows typically involve time-consuming, sequential processes that include:
- Running simulations for extended periods (days, weeks, or months).
- Preparing a subset of output data for visualization.
- Transferring and loading the data into visualization software for analysis.
- Sharing findings and repeating the process with different data subsets or running additional simulations with new inputs.
Such workflows are impractical in on-premises environments due to limitations in compute and storage resources, collaboration barriers, and time wasted if visualization data indicates the simulated data was incorrect.
AWS accelerates this workflow by enabling users to configure simulations and visualizations to run on separate clusters. In this scenario, simulation data on one cluster is written to Amazon S3 object storage, where it can be visualized on another cluster via NVIDIA IndeX (loading directly from S3) and shared with any device equipped with a web browser. This setup allows users to interactively explore and collaborate in real-time after simulations commence, enabling them to pause or restart simulations as necessary.
AWS also facilitates in-situ simulation and visualization (on the same cluster), allowing simulation data to be saved to local storage (instance store) or network-attached Amazon Elastic Block Store (EBS) for visualization, thus enabling users to direct simulations interactively and assess outcomes.
Amazon EC2 GPU Instances
AWS provides a variety of GPU-accelerated instances tailored for high-performance computing, machine learning, and graphics-intensive workloads (including volume visualization and rendering). For instance, the Amazon EC2 P3 instance features up to eight NVIDIA V100 Tensor Core GPUs, each with 32GB of memory, offering up to 100 Gbps of networking throughput and supporting NVLink for GPU peer-to-peer communication. The Amazon EC2 G4 instance presents a cost-effective alternative with support for up to four NVIDIA T4 Tensor Core GPUs, each with 16GB of memory and 50 Gbps of networking throughput. The EC2 G4 metal instance supports eight NVIDIA T4 GPUs and offers 100 Gbps of networking throughput. AWS also plans to introduce NVIDIA A100 Tensor Core GPUs soon. To stay updated, consider signing up for news via this link.
Depending on dataset sizes, users can opt for either a single instance or a cluster of instances, scaling as required. IndeX automatically leverages all available GPUs in an instance, ensuring optimal performance. The objective is to deliver visualization performance that scales with data, enabling users to analyze complete datasets in their original resolution, which accelerates scientific discoveries.
Utilizing IndeX on AWS allows users to harness the performance features offered by the IndeX SDK alongside the elasticity and computational power of AWS. Datasets for computational scientists are growing increasingly from terabytes to petabytes. AWS enables users to store such extensive datasets in the cloud through services like Amazon S3, allowing multiple users to access the same datasets without the need for creating numerous local copies.
NVIDIA IndeX Architecture on AWS
Users can choose from several options for 3D volume rendering on AWS using IndeX. Below is an example architecture diagram for both single instance rendering and cluster rendering:
Single Instance Rendering
In single instance rendering, users deploy an Amazon EC2 GPU-based instance in an AWS region using the IndeX AMI from AWS Marketplace. The custom datasets needed for volume rendering should be uploaded to S3. Optionally, the Amazon EC2 instance can be supported by an EBS volume if necessary.
Once the Amazon EC2 GPU instance is operational, users can initiate a terminal and enable port forwarding via SSH tunneling to that instance. After establishing the connection, users can log into the Amazon EC2 GPU instance and execute the IndeX SDK viewer script, along with the sample or custom dataset for visualization. The HTML5-based client viewer of IndeX streams video from the IndeX server running on Amazon EC2, while allowing socket-based transmission of user interactions or commands back to the server.
For further insights into scientific visualization and related topics, you can explore this blog post or learn more about the subject from this authoritative source.
Leave a Reply