Opening virtual datasets with NASA dmrpp files

[ayushnag]

The idea is to speed up opening up netcdf4/hdf5 datasets with a NASA specific optimization. Load data like xr.open_mfdataset with kerchunk/zarr speeds by translating existing dmr++ metadata files to zarr metadata on the fly. Much more context and discussion here.

virtualizarr PR for the parser here

earthaccess PR here

earthaccess additions:

1. Open a virtual dataset (like a view of the data that contains dimensions, variables, attrs, chunks, but no actual data)

2. Concatenate virtual xr.Datasets
   1. Use xarrays concatenation logic to create virtual views of netcdf’s (more details in virtualizarr documentation)
   2. Save as json/parquet/in-memory dict

3. Read netcdf/hdf5 data
   1. Use the zarr engine in xarray to load a dataset (with indexes)

Questions/Suggestions:

Changes to the API?

• **xr.combine_nested kwargs might be confusing and could be reworked if needed

NASA datasets you want me to test?

• Many with unique considerations and the goal is to handle all NASA netcdf4/hdf5 datasets
• So far I’ve tested and had success MUR-JPL-L4-GLOB-v4.1 (netcdf), SWOT_SSH_2.0 (netcdf), ATL-03 ICE-SAT (hdf5)
• Feel free to test it out yourself or add a comment with a dataset and I can take a look

Take a look at the virtualizarr parser PR and leave suggestions

• Speed improvements?
• The current bottleneck is creating the xml object (dmrpp are xml files) with ET.fromstring(dmr_str) since the xml.ElementTree library needs to read the text, validate the xml, and create a parsable object. I am looking into a non-validating parser like xml.parsers.expat

[Mikejmnez]

This looks great @ayushnag ! I will play with this in the next couple of days