NASA is upping its storage to 215 petabytes but forgot about retrieval costs AWS charge for taking data back from the Cloud
It was widely reported last year that NASA, after an extensive consultation period, had chosen AWS (Amazon Web Services) over other AWS competitors to run their Earthdata Cloud, a repository for data for the Earth Science Data and Information System (ESDIS) that collates all information collected from NASA’s missions.
NASA made the jump to AWS after realising that, by 2025, they’d have increased from around 32 petabytes of data to nearly 250 petabytes (which is a lot!)
Unfortunately, either no one told them or they forgot about the associated costs of retrieving the data they feed into AWS.
For anyone unsure, egress charges are a cost occurred on AWS when moving data from the Cloud to another area, in this instance we could imagine a local workstation for a NASA engineer or scientist.
Most AWS subscriptions will have them charged on top of a clients monthly Cloud bill and the more data that you retrieve… the more you’ll have to pay extra.
Previously NASA was storing all its data on prem, in 12 DAACs (Distributed Active Archive Centres) and was expecting to complete the transfer to AWS in the next couple of years.
If you’re wondering why NASA suddenly needs another 215 petabytes of information, it’s down to the fifteen upcoming missions they have planned, each of which is expected to produce around 100 terabytes of information… daily!
Those missions will include the launching of NISAR, the NASA-ISRO Synthetic Aperture Radar as well as the Surface Water and Ocean Topography (SWOT) satellites, with all data collected being directly uploaded to their Earthdata Cloud.
NASA’s main goal behind the migration was to have all their data available in the Cloud rather than siloed away in different geographical locations as it currently is.
This will give NASA’s scientists a huge boost… if they can afford the new costs!
The office of the Inspector General of NASA ordered a full audit of the project earlier this month, concluding that the Earth Observing System Data and Information System (EOSDIS), which makes the information from ESDIS available, hadn't properly taken in to account the additional costs egress charges would incur:
Specifically, the Agency faces the possibility of substantial cost increases for data egress (i.e., when end users download data from a network to an external location) from the cloud. Currently, when end-users access and egress data through a DAAC there is no additional cost to NASA other than maintaining the current infrastructure. However, when end users download data from Earthdata Cloud, the Agency, not the user, will be charged every time data is egressed. Ultimately, ESDIS will be responsible for both cloud costs, including egress charges, and the costs to operate the 12 DAACS.
In addition, ESDIS has not yet determined which data sets will transition to Earthdata Cloud, nor has it developed cost models based on operational experience and metrics for usage and egress. As a result, current cost projections may be lower than what will actually be necessary to cover future expenses and cloud adoption may become more expensive and difficult to manage.
Collectively, this presents potential risks that scientific data may become less available to end users if NASA imposes limitations on the amount of data egress for cost control reasons.
Perhaps even more worrying though was the fact that the report also highlighted the fact that the Evolution, Enhancement, and Efficiency (E&E) panel chosen to review the DAACs didn't even attempt to identify potential cost savings (after the Mission Support Council removed the need to do so), didn't follow the National Institute of Standards and Technology's (NIST's) data integrity standards and lacked independence, with six of the twelve panel members working on ESDIS.
Following that the Office of the Inspector General for Nasa has made three recommendations…
Migrating to the Cloud may seem like an easy task but as you can see... even organisations like NASA can make mistakes. If you'd like any help or advice then feel free to get in touch with cloudThing today