The National Environmental Satellite, Data, and Information Service (NESDIS), part of the National Oceanic and Atmospheric Administration (NOAA), is dedicated to providing secure and timely access to essential global environmental data from satellites and other sources. This effort aims to bolster the United States’ security, environment, economy, and quality of life. In a significant development, NOAA has made the Himawari-8 satellite dataset available on AWS.
As part of the Amazon Sustainability Data Initiative, we invited Alex Thompson, technology architect and cloud director for NESDIS, and Sarah Lee, communications lead and co-technical data lead for NOAA’s Big Data Program (BDP), to discuss how NOAA leverages Amazon Web Services (AWS) to enhance access to crucial weather data.
Improving Access to Essential Weather Information
In December, AWS expanded its partnership with NOAA to streamline access to environmental data through the Big Data Program (BDP). Users can now obtain authoritative NOAA data on AWS without needing to download or store the datasets themselves. This allows researchers and entrepreneurs to utilize on-demand computing resources in the cloud, facilitating rapid analysis and reducing costs, thereby enabling more experimentation.
The Himawari-8 dataset, managed by NESDIS, is now a key resource within this collaboration, accessible via the Registry of Open Data on AWS. This marks NESDIS as the sole source of the Himawari-8 data in the United States.
Harnessing Cloud Technology for NESDIS Objectives
As part of a broader cloud initiative, NESDIS has developed the NESDIS Common Cloud Framework (NCCF). A pivotal service operationalized within this framework is the Consolidated Ingest Service, which securely imports foreign and commercial data. This service retrieves Himawari-8 data from the Japan Meteorological Agency (JMA), ensuring that the data is free from viruses and other malicious threats. The sanitized data is then utilized to generate various NOAA products for users, including the National Weather Service (NWS) and other operational partners, while simultaneously being made available to BDP partners like AWS for wider accessibility.
Traditionally, foreign and commercial datasets were disseminated through on-premises hardware, which faced bandwidth and resource constraints. The collaboration with AWS through the BDP provides a more efficient, cost-effective, and reliable data dissemination model. This scalable approach allows NESDIS and NOAA to offer a greater variety of datasets to the public than was previously feasible by leveraging cloud capabilities for data staging and analysis.
Transforming Data Distribution Models
Through the BDP, NESDIS can assess datasets like Himawari-8 for security risks before they enter mission-critical systems. Utilizing the AWS Cloud has enabled NESDIS to establish an enterprise-level capability for secure and reliable inspection of foreign and commercial data. This cloud-based approach eliminates the need for additional on-premises hardware, allowing them to scale computing resources according to demand. The redundant and highly available cloud services ensure robust and fault-tolerant operations, enabling quicker implementations compared to traditional methods. Currently, the NCCF v1.0 service processes around 41,000 files daily from JMA, delivering low latency and high availability.
Moreover, the NCCF benefits from a robust redundant network architecture, utilizing diverse network providers and AWS Direct Connect for reliable connections. The services offered by NCCF v1.0 are deployed using agile methodologies and blue-green deployment techniques, with performance monitored through dashboards powered by Kibana, an open-source visualization tool, providing alerts for NESDIS support teams.
Potential to Enhance Weather Forecasting
By making Himawari-8 data readily accessible on AWS, opportunities for improved forecasting applications emerge, fostering collaboration among research entities. The Advanced Himawari Imager (AHI) equipped on Himawari-8 features 16 channels, a significant upgrade compared to earlier geostationary satellites. Increased accessibility to this data can assist meteorologists in refining weather analysis and forecasting techniques.
One specific application of this dataset is in tropical cyclone tracking and forecasting. Himawari-8 imagery covers the western North Pacific, the globe’s most active tropical cyclone area, as well as parts of the southern Hemisphere and Indian Ocean. This makes the Himawari-8 data invaluable for National Meteorological Services and cyclone research. With GOES-16/17 data already available on AWS, the addition of Himawari-8 allows for more advanced satellite data applications, enhancing the understanding of phenomena like tropical cyclone intensity variations and integrating these insights into forecasting processes. For further engagement, you can read more about this topic in another blog post here.
To gain deeper insights, check out this resource as they are an authority on this topic. For those interested in opportunities within this field, this link serves as an excellent resource.
Leave a Reply