subsurface floats dataset

[philippemiron]

resolves #320

• quick implementation
• clean up variable names
• create variables: rowsize, ids
• variable attributes (add coordinates)
• docstring
• tests to at least validate that the url is valid (to identify update)
• wrap this in datasets
• fix output to netcdf
• references (?)

Comments?

[milancurcic]

Comments? Awesome. :)

[philippemiron]

@milancurcic, I don't remember exactly how this works, but do we always create a wrapper in the datasets module that just calls the to_xarray() function?

[philippemiron]

Someone asked me for the subsurface floats dataset recently, so I decided that instead of updating my code, I would do it here. I'm thinking of adding Andro (#319) or YoMaHa after this, based on Argo floats.

[milancurcic]

Adapter is needed whenever a dataset needs to be re-organized to be a CloudDrift conformable ragged-array dataset (obs and traj dims, rowsize variable; can have different names but they must be defined). For example, adapter is not needed for the upcoming Spotter open dataset because we coordinated with them to ensure it's a CloudDrift conformable ragged array.

Dataset function is merely a convenience accessor to an existing ragged-array dataset, so that the user doesn't need to find and manage data URLs. For example, in case of GDP, we simply xr.open_dataset(url) and return ds. In case of MOSAiC, the dataset is relatively small so we invoke the adapter under the hood.

So, I expect that every clouddrift.adapter.* will have a corresponding clouddrift.datasets.*, but not all clouddrift.datasets.* need to have an adapter.

From the end-user (as in consumer of data for analysis; not a contributor to the library) point of view, they should only ever have to access clouddrift.datasets.

[philippemiron]

Thanks, it makes sense.

[selipot]

Tiny bit more info here about the dataset perhaps?

[selipot]

rename to subsurfacefloat.py?

[milancurcic]

Does this dataset have a proper name? It's not clear to me from its website. If not, I suggest subsurface_floats.

[philippemiron]

That was also my hesitation. I don't see any real defined name for it.. They write subsurface float trajectories, that's why I didn't put the s. I don't really mind either way.

[philippemiron]

I added some details. Let me know if you think this should be further improved.

[selipot]

Looks good but it does not seem like the dataset can be used with subset for variables with dimension exp. Should not this be possible?

[milancurcic]

@philippemiron I will test this locally today. Can you also cast all float variables except time, lat, lon to float32?

[philippemiron]

@milancurcic Yes. However, doesn't this in the glad adapter converts all variables?

    # Cast double floats to singles
    for var in ds.variables:
        if ds[var].dtype == "float64":
            ds[var] = ds[var].astype("float32")

[milancurcic]

Yes. I special-cased GLAD because all longitudes are in the Gulf (rather than global) and the position error included in the dataset exceeds the precision of the single-precision ulp.

Since subsurface floats are global (as I understand it), it's safest to keep lat, lon as float64.

[philippemiron]

Ok good! I will add this for the 3 datasets I'm working on.

    # Cast double floats to singles
    double_vars = ["lat", "lon"]
    for var in [v for v in ds.variables if v not in double_vars]:
        if ds[var].dtype == "float64":
            ds[var] = ds[var].astype("float32")

[milancurcic]

This is great; thank you. Will merge.