A major mistake? NASA’s full migration of data to the AWS cloud ignores access costs

Last year, NASA chose Amazon Cloud Services (AWS) to host data from Earth Science and ESDIS information systems. The new database makes it easy to organize task information, and storage requirements are expected to grow from the current 32PB to nearly 250PB. In addition to the additional capacity requirements, however, the agency seems to have forgotten the cost of accessing and retrieving AWS, the so-called “export cost.”

A major mistake? NASA's full migration of data to the AWS cloud ignores access costs

Infographic (from: NASA)

When moving data from the cloud to another area, such as a local workstation for scientists to analyze, NASA will have to pay an additional fee on top of the monthly service charge.

In other words, the more data you retrieve, the older the overhead. The agency currently stores data locally in 12 distributed Activity Archive Centers (DAAC), but hopes to complete a full cloud migration in the next few years.

The initial transfer was planned in the first quarter of this year, but where did the data come from and how did NASA get to 215PB?

NASA is expected to estimate based on 15 of the agency’s upcoming visits and is expected to generate more than 100 terabytes of information per day.

This includes data from nasa-ISRO Synthetic Aperture Radar (NISAR) and Surface Water and Marine Terrain (SWOT) satellites, which will be first uploaded to Earthdata Cloud.

If the agency can afford to export this part of the data, it would be useful for NASA researchers to store all that data in the cloud, rather than being as dispersed as it is now.

A major mistake? NASA's full migration of data to the AWS cloud ignores access costs

2017 to 2025 Data Growth (Blue) / Extract speed (orange)

Unfortunately, according to an audit report issued in March this year by NASA’s Office of the Inspector General, the simulations do not accurately determine how high the export costs of ESDIS and EOSDIS will be.

If NASA limits data output for cost-controlled reasons, it will also pose a potential risk that scientific data will be significantly less accessible to end users.

Based on this, the report makes three recommendations:

First, once NISAR and SWOT are operational and provide sufficient data, NASA must complete a separate analysis to determine cloud migration and ensure the long-term financial sustainability of current DAAC operations.

Second, appropriately incorporate institutional guidance to coordinate ESDIS and OCI ostoins in the early stages of developing a data management plan.

Third, ensure that all applicable types of information are considered in the DAAC classification process, that appropriate prerequisites are taken into account in determining the level of impact, and that the processes are standardized.