starting surf trip EDS model notebook

[MathewBiddle]

closes #99

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[MathewBiddle]

@ocefpaf might need your help here. Check out the last cell.

I am trying to iterate through each model, extract a time-series for a specific variable (abbreviation='HTSGW'), and plot those timeseries on the same figure. Something isn't working quite right though.

First, only the last timeseries stays on the plot (probably because I'm overwriting ax each time?).

Second, it takes forever to run the cell (see below for %%time info). Are there areas for improvement? I do want to pull in other variables as well (wave period/direction, forecasted winds), so maybe I should be returning all the data I might need as one piece, then plotting as another?

CPU times: total: 1.5 s
Wall time: 12min 12s

[ocefpaf]

Can you run wvhgt_gfs.to_netcdf("wvhgt_gfs.nc") there and make that file available? Looks like that server doesn't like me, I'm getting tons of RuntimeError: NetCDF: DAP failure .

[ocefpaf]

First, only the last timeseries stays on the plot (probably because I'm overwriting ax each time?).

That is not the problem. What you are doing there is correct. I cannot confirm for all models b/c I cannot access them but, for the nwpsm the last time in the reftime=1 that you are selecting is 2022-12-04T06:00:00. That means your left xlim of datetime.datetime.today()-datetime.timedelta(days=1) is too restrictive. You'll need at east 4 to see some data.

[MathewBiddle]

I'm getting errors during the write step. See the updated notebook and broken netCDF file included in the most recent commit, and below:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File :39

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\dataset.py:1903, in Dataset.to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
   1900     encoding = {}
   1901 from ..backends.api import to_netcdf
-> 1903 return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(
   1904     self,
   1905     path,
   1906     mode=mode,
   1907     format=format,
   1908     group=group,
   1909     engine=engine,
   1910     encoding=encoding,
   1911     unlimited_dims=unlimited_dims,
   1912     compute=compute,
   1913     multifile=False,
   1914     invalid_netcdf=invalid_netcdf,
   1915 )

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\api.py:1230, in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
   1225 # TODO: figure out how to refactor this logic (here and in save_mfdataset)
   1226 # to avoid this mess of conditionals
   1227 try:
   1228     # TODO: allow this work (setting up the file for writing array data)
   1229     # to be parallelized with dask
-> 1230     dump_to_store(
   1231         dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
   1232     )
   1233     if autoclose:
   1234         store.close()

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\api.py:1277, in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
   1274 if encoder:
   1275     variables, attrs = encoder(variables, attrs)
-> 1277 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\common.py:266, in AbstractWritableDataStore.store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
    263 if writer is None:
    264     writer = ArrayWriter()
--> 266 variables, attributes = self.encode(variables, attributes)
    268 self.set_attributes(attributes)
    269 self.set_dimensions(variables, unlimited_dims=unlimited_dims)

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\common.py:355, in WritableCFDataStore.encode(self, variables, attributes)
    352 def encode(self, variables, attributes):
    353     # All NetCDF files get CF encoded by default, without this attempting
    354     # to write times, for example, would fail.
--> 355     variables, attributes = cf_encoder(variables, attributes)
    356     variables = {k: self.encode_variable(v) for k, v in variables.items()}
    357     attributes = {k: self.encode_attribute(v) for k, v in attributes.items()}

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\conventions.py:868, in cf_encoder(variables, attributes)
    865 # add encoding for time bounds variables if present.
    866 _update_bounds_encoding(variables)
--> 868 new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}
    870 # Remove attrs from bounds variables (issue #2921)
    871 for var in new_vars.values():

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\conventions.py:868, in (.0)
    865 # add encoding for time bounds variables if present.
    866 _update_bounds_encoding(variables)
--> 868 new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}
    870 # Remove attrs from bounds variables (issue #2921)
    871 for var in new_vars.values():

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\conventions.py:273, in encode_cf_variable(var, needs_copy, name)
    264 ensure_not_multiindex(var, name=name)
    266 for coder in [
    267     times.CFDatetimeCoder(),
    268     times.CFTimedeltaCoder(),
   (...)
    271     variables.UnsignedIntegerCoder(),
    272 ]:
--> 273     var = coder.encode(var, name=name)
    275 # TODO(shoyer): convert all of these to use coders, too:
    276 var = maybe_encode_nonstring_dtype(var, name=name)

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\coding\times.py:680, in CFDatetimeCoder.encode(self, variable, name)
    679 def encode(self, variable, name=None):
--> 680     dims, data, attrs, encoding = unpack_for_encoding(variable)
    681     if np.issubdtype(data.dtype, np.datetime64) or contains_cftime_datetimes(
    682         variable
    683     ):
    684         (data, units, calendar) = encode_cf_datetime(
    685             data, encoding.pop("units", None), encoding.pop("calendar", None)
    686         )

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\coding\variables.py:104, in unpack_for_encoding(var)
    103 def unpack_for_encoding(var):
--> 104     return var.dims, var.data, var.attrs.copy(), var.encoding.copy()

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\variable.py:435, in Variable.data(self)
    433     return self._data
    434 else:
--> 435     return self.values

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\variable.py:608, in Variable.values(self)
    605 @property
    606 def values(self):
    607     """The variable's data as a numpy.ndarray"""
--> 608     return _as_array_or_item(self._data)

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\variable.py:314, in _as_array_or_item(data)
    300 def _as_array_or_item(data):
    301     """Return the given values as a numpy array, or as an individual item if
    302     it's a 0d datetime64 or timedelta64 array.
    303 
   (...)
    312     TODO: remove this (replace with np.asarray) once these issues are fixed
    313     """
--> 314     data = np.asarray(data)
    315     if data.ndim == 0:
    316         if data.dtype.kind == "M":

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\indexing.py:653, in MemoryCachedArray.__array__(self, dtype)
    652 def __array__(self, dtype=None):
--> 653     self._ensure_cached()
    654     return np.asarray(self.array, dtype=dtype)

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\indexing.py:650, in MemoryCachedArray._ensure_cached(self)
    648 def _ensure_cached(self):
    649     if not isinstance(self.array, NumpyIndexingAdapter):
--> 650         self.array = NumpyIndexingAdapter(np.asarray(self.array))

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\indexing.py:623, in CopyOnWriteArray.__array__(self, dtype)
    622 def __array__(self, dtype=None):
--> 623     return np.asarray(self.array, dtype=dtype)

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\indexing.py:524, in LazilyIndexedArray.__array__(self, dtype)
    522 def __array__(self, dtype=None):
    523     array = as_indexable(self.array)
--> 524     return np.asarray(array[self.key], dtype=None)

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\netCDF4_.py:85, in NetCDF4ArrayWrapper.__getitem__(self, key)
     84 def __getitem__(self, key):
---> 85     return indexing.explicit_indexing_adapter(
     86         key, self.shape, indexing.IndexingSupport.OUTER, self._getitem
     87     )

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\indexing.py:815, in explicit_indexing_adapter(key, shape, indexing_support, raw_indexing_method)
    793 """Support explicit indexing by delegating to a raw indexing method.
    794 
    795 Outer and/or vectorized indexers are supported by indexing a second time
   (...)
    812 Indexing result, in the form of a duck numpy-array.
    813 """
    814 raw_key, numpy_indices = decompose_indexer(key, shape, indexing_support)
--> 815 result = raw_indexing_method(raw_key.tuple)
    816 if numpy_indices.tuple:
    817     # index the loaded np.ndarray
    818     result = NumpyIndexingAdapter(np.asarray(result))[numpy_indices]

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\netCDF4_.py:98, in NetCDF4ArrayWrapper._getitem(self, key)
     96     with self.datastore.lock:
     97         original_array = self.get_array(needs_lock=False)
---> 98         array = getitem(original_array, key)
     99 except IndexError:
    100     # Catch IndexError in netCDF4 and return a more informative
    101     # error message.  This is most often called when an unsorted
    102     # indexer is used before the data is loaded from disk.
    103     msg = (
    104         "The indexing operation you are attempting to perform "
    105         "is not valid on netCDF4.Variable object. Try loading "
    106         "your data into memory first by calling .load()."
    107     )

File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\common.py:69, in robust_getitem(array, key, catch, max_retries, initial_delay)
     67 for n in range(max_retries + 1):
     68     try:
---> 69         return array[key]
     70     except catch:
     71         if n == max_retries:

File src\netCDF4\_netCDF4.pyx:4973, in netCDF4._netCDF4.Variable.__getitem__()

File src\netCDF4\_netCDF4.pyx:5930, in netCDF4._netCDF4.Variable._get()

File src\netCDF4\_netCDF4.pyx:2028, in netCDF4._netCDF4._ensure_nc_success()

RuntimeError: NetCDF: DAP failure

[MathewBiddle]

Add Caribbean SWAN Height @ CARICOOS SWAN Wave Height: https://edsdata.oceansmap.com/thredds/catalog/EDS/CARIB_SWAN/catalog.html

[MathewBiddle]

@ocefpaf things are looking better in this notebook. I think TDS is behaving well and we have a good start. Marking this as ready for review, knowing that work needs to be done on the story telling side.

[ocefpaf]

I have some minor comments but it is good to go with or without them.

1. cell[1] → instead of using datetime and rounding to a date, you can use date directly with datetime.date.today()
2. cell[1] → there is a lot of printing there. I guess some of those are just to debug, like the lon and lon360.
3. cell[2] → f-string can make things more readable: title=f'lon:{lon} lat:{lat}'
4. cell[2] → my old eyes would like a slightly bigger markersize
5. cell[3] → remove commented out code #import cf_xarray
6. cell[6-8] → remove commented out cells
7. cell[12] → the plot is empty
8. cell[13] → maybe, for the sake of brevity, the combined plot can be the only plot for the time-series
9. cell[15] → b/c you repeat the same operation for another place we can probably create a function that takes a space-time dict as an entry and returns the data for plotting. If the goal is to show step-by-step then disregard this comment