Double the Data: Putting Weather Observations in the Cloud Increases Access

From UCAR/NCAR Meteorologists and other users accessed more than twice as much U.S. weather radar data after the National Oceanic and Atmospheric Administration (NOAA) partnered with Amazon and the University Corporation for Atmospheric Research (UCAR) to make the data available in the cloud. The collaboration is part of the NOAA Big Data Project. Launched in 2015, the project aims to make NOAA's vast storehouse of environmental data easier to access with the hope that both the public and industry will find ways to capitalize on it and spur new economic growth. Partners in the project include Amazon Web Services, Google Cloud Platform, IBM, Microsoft, and the Open Commons Consortium. The first dataset chosen for the project was from the Next Generation Weather Radar (NEXRAD) Weather Surveillance Radar system, and the first collaborator to put that dataset in the cloud was Amazon. The entire archive of NEXRAD data, which stretches back to 1991, is now available via NEXRAD on AWS, and Amazon has partnered with UCAR's Unidata program to update the database in real time and to provide the tools users need to make sense of the data. "As a leading provider of geoscience data to universities across the country, it made perfect sense to partner with Amazon to explore how cloud computing can expand our reach and provide new capabilities to our users," said Unidata Director Mohan Ramamurthy. "The success of this project has given us a chance to be a part of what the future of data delivery will look like." [A visualization of the sequence of radar reflectivity scans capturing the deadly 2013 Moore, Oklahoma, tornado event. The vertical structure of the debris signature is derived from interpolating isosurfaces from the volume scan] In a new paper published recently in the Bulletin of the American Meteorological Society, collaborators describe the project's early accomplishments. Chief among them: users are now accessing 2.3 times more NEXRAD Level II data. This data, collected from 160 sites and updated approximately every 5 minutes, characterizes precipitation and winds from across the U.S. and is an important input for weather forecasts. Prior to the NOAA Big Data project, archived Level II NEXRAD data was stored only at NOAA's National Centers for Environmental Information (NCEI). While this archive represents a critical dataset for researchers interested in large-scale analysis of how weather patterns may have changed over time, in practice it was very difficult to use. In part that's because of its size — about 270 terabytes — and in part because of the time and cost to obtain the files. The authors of the new study estimate that it would have cost $203,310 and taken 540 days for NCEI to fulfill one researcher's request to obtain the entire NEXRAD Level II data archive. Now, scientists can access the data on the Amazon cloud at no charge. As a result, about 80 percent of the NEXRAD data accessed by users now comes from the cloud, and only about 20 percent comes from NCEI. Jeff Weber, who leads Unidata's part of the project, said that this kind of easy, open access to geoscience data "removes the friction" of doing the science. Moving forward, Weber envisions having satellite data and weather model output available alongside the radar data. [The Climate Corporation data flow prior to BDP] [The Climate Corporation data flow after the completion of the BDP transfer of NEXRAD Level II data.] "Once we're able to bring all these components together in the cloud, I think we're going to see a huge leap in the science," he said. "When you make it easier for scientists to just focus on the science — and not worry about accessing and storing huge amounts of data — breakthroughs are bound to happen." The collaboration with Amazon has allowed Unidata to test how scientists will respond to data stored in the cloud. Part of Unidata's long-term vision is to take data that it now pushes out to users across the country and move it instead to a cloud platform. Aside from the benefit of scientists being able to access the data from anywhere, moving data to the cloud also addresses the reality that increasingly complex and high-resolution models and observational instruments are straining the physical capacity to deliver data to individual research institutions. "The data volumes are growing exponentially — they are growing so fast that we can't just keep pushing all of the data out to our users," Ramamurthy said. "But just putting data out there in the cloud isn't enough either. We need to train our community on how to access that data and provide them the tools they need in the cloud to make it as easy as possible to use the data that's stored there." Edited for WeatherNation by Meteorologist Mace Michaels