New docker build for ravenpy birdy xclim #75

tlvu · 2021-05-27T16:31:13Z

Overview

New Jupyter env to get latest of everything, including ravenpy, birdy, xclim.

Jenkins build passing with updated Finch and Pavics-sdi notebooks: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-for-ravenpy/25/

Matching Finch notebook PR: bird-house/finch#188
Matching Pavics-sdi notebook PR: Ouranosinc/pavics-sdi#222
Matching PR to deploy this Jupyter env to PAVICS: bird-house/birdhouse-deploy#175

Matching Raven notebook failure found Ouranosinc/raven#392 (http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-for-ravenpy/14/console) and matching fix PR Ouranosinc/raven#393 (http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-for-ravenpy/24/).

Also added new libraries for upcoming Raven notebooks (fixes Ouranosinc/raven#394).

Pavics-landing homepage notebooks manually tested by @tlogan2000

Deployed to staging env for manual testing: https://medus.ouranos.ca/jupyter/

Also added a way to quickly test a new package without having to do a full rebuild.

Changes

<   - ravenpy=0.4.2=py37_1
>   - ravenpy=0.5.2=pyh7f9bfb9_0

# Renamed.
<   - raven=3.0.4.318=hc9bffa2_2
>   - raven-hydro=3.0.4.322=h516393e_0

<   - ostrich=21.03.16=h2bc3f7f_0
>   - ostrich=21.03.16=h4bd325d_1

<   - xclim=0.25.0=pyhd8ed1ab_0
>   - xclim=0.27.0=pyhd8ed1ab_0

# Old version was from pip.
<     - birdhouse-birdy==0.7.0
>   - birdy=v0.8.0=pyh6c4a22f_0

# Was previously included in another package, now it is standalone.
>   - pydantic=1.8.2=py37h5e8e339_0

# New libs for upcoming Raven notebooks
>   - gcsfs=2021.6.0=pyhd8ed1ab_0
>   - intake=0.6.2=pyhd8ed1ab_0
>   - intake-esm=2021.1.15=pyhd8ed1ab_0
>   - zarr=2.8.3=pyhd8ed1ab_0

<   - xarray=0.17.0=pyhd8ed1ab_0
>   - xarray=0.18.2=pyhd8ed1ab_0

<   - owslib=0.23.0=pyhd8ed1ab_0
>   - owslib=0.24.1=pyhd8ed1ab_0

<   - cf_xarray=0.5.1=pyh44b312d_0
>   - cf_xarray=0.5.2=pyh6c4a22f_0

<   - clisops=0.6.3=pyh44b312d_0
>   - clisops=0.6.5=pyh6c4a22f_0

<   - dask=2021.2.0=pyhd8ed1ab_0
>   - dask=2021.6.0=pyhd8ed1ab_0

# Downgrade !
<   - gdal=3.2.1=py37hc5bc4e4_7
>   - gdal=3.1.4=py37h2ec2946_8

# Downgrade !
<   - rasterio=1.2.2=py37hd5c4cce_0
>   - rasterio=1.2.1=py37ha549118_0

<   - hvplot=0.7.1=pyh44b312d_0
>   - hvplot=0.7.2=pyh6c4a22f_0

<   - rioxarray=0.3.1=pyhd8ed1ab_0
>   - rioxarray=0.4.1.post0=pyhd8ed1ab_0

# Downgrade !
<   - xskillscore=0.0.19=pyhd8ed1ab_0                                                                                                                 
>   - xskillscore=0.0.18=py_1

Full diff of conda env export:
210415-210527.1-update210615-conda-env-export.diff.txt

Full new conda env export:
210527.1-update210615-conda-env-export.yml.txt

Useful to avoid full rebuild just to test adding a new package or upgrading an existing package. Sort of a restore of commit cbfa5d5.

…nt build Previous build probably got pydantic via indirect dependency but somehow this has changed so now have to add pydantic explicitly, for Raven notebooks.

The fix did not make it to RavenPy 0.5.1. To fix error running Raven notebooks: ``` --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-1-021da278d129> in <module> 2 3 from birdy import WPSClient ----> 4 from ravenpy.utilities.testdata import get_file 5 import os 6 import xarray as xr /opt/conda/envs/birdy/lib/python3.7/site-packages/ravenpy/utilities/testdata.py in <module> 14 from xarray import Dataset 15 from xarray import open_dataset as _open_dataset ---> 16 from xarray.tutorial import file_md5_checksum 17 18 _default_cache_dir = Path.home() / ".raven_testing_data" ImportError: cannot import name 'file_md5_checksum' from 'xarray.tutorial' (/opt/conda/envs/birdy/lib/python3.7/site-packages/xarray/tutorial.py) ```

…onda-forge

See PR Ouranosinc/PAVICS-e2e-workflow-tests#75 for more details. Relevant changes: ```diff < - raven=3.0.4.318=hc9bffa2_2 > - raven-hydro=3.0.4.322=h516393e_0 < - ravenpy=0.4.2=py37_1 > - ravenpy=0.5.2=pyh7f9bfb9_0 < - ostrich=21.03.16=h2bc3f7f_0 > - ostrich=21.03.16=h4bd325d_1 < - xclim=0.25.0=pyhd8ed1ab_0 > - xclim=0.26.1=pyhd8ed1ab_0 < - birdhouse-birdy==0.7.0 > - birdy=v0.8.0=pyh6c4a22f_0 > - pydantic=1.8.2=py37h5e8e339_0 < - xarray=0.17.0=pyhd8ed1ab_0 > - xarray=0.18.2=pyhd8ed1ab_0 < - owslib=0.23.0=pyhd8ed1ab_0 > - owslib=0.24.1=pyhd8ed1ab_0 < - cf_xarray=0.5.1=pyh44b312d_0 > - cf_xarray=0.5.2=pyh6c4a22f_0 < - cftime=1.4.1=py37h902c9e0_0 > - cftime=1.5.0=py37h6f94858_0 < - clisops=0.6.3=pyh44b312d_0 > - clisops=0.6.4=pyh6c4a22f_0 < - dask=2021.2.0=pyhd8ed1ab_0 > - dask=2021.5.0=pyhd8ed1ab_0 < - gdal=3.2.1=py37hc5bc4e4_7 > - gdal=3.1.4=py37h2ec2946_8 < - hvplot=0.7.1=pyh44b312d_0 > - hvplot=0.7.2=pyh6c4a22f_0 < - rioxarray=0.3.1=pyhd8ed1ab_0 > - rioxarray=0.4.1.post0=pyhd8ed1ab_0 < - xskillscore=0.0.19=pyhd8ed1ab_0 > - xskillscore=0.0.18=py_1 ``` Full diff of `conda env export`: [210415-210527-conda-env-export.diff.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/6555480/210415-210527-conda-env-export.diff.txt) Full new `conda env export`: [210527-conda-env-export.yml.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/6555483/210527-conda-env-export.yml.txt)

aulemahal · 2021-05-27T16:49:25Z

The update to cftime==1.5.0 is supposed to break the notebooks of the PAVICS homepage, I gather they are not tested?

tlvu · 2021-05-27T16:58:28Z

The update to cftime==1.5.0 is supposed to break the notebooks of the PAVICS homepage, I gather they are not tested?

@aulemahal
Nice catch, notebooks of PAVICS homepage are not part of regular test yet. It's a private git repo so had trouble hooking it into the standard test.

What would be a possible work-around, pin cftime to the previous version?

@tlogan2000 We have to have a plan to ensure testing of the PAVICS homepage notebooks. Each release of Jupyter env, I ask you to manually test them? Make the pavics-landing git repo public so it's simple to plug into the existing testing mechanism? Or I completely re-write the testing and tuto notebooks deployment to be able to handle private git repo?

tlvu · 2021-05-27T17:00:55Z

This new Jupyter env is deployed to staging env for manual testing: https://medus.ouranos.ca/jupyter/

aulemahal · 2021-05-27T17:01:20Z

As sad as it is, I think pinning cftime is the current solution. In fact, the problem arises from nc-time-axis (see SciTools/nc-time-axis#58). That's the small plugin that allows us to plot with matplotlib when the time axis is made of cftime objects (instead of numpy objects).

tlogan2000 · 2021-05-27T18:40:07Z

@tlvu I think we can make probably make pavics-landing public. Testing is definitely good idea but there are a couple of "advanced" sections that calculate a lot of data and take a while to run etc... we might need some precalculated data in the repo and a method to somehow skip a cell or two in a given notebook in the tests? We can try set up a plan for sure

tlvu · 2021-05-28T02:37:37Z

New build with cftime pinning is on https://medus.ouranos.ca/jupyter/. Many build problems on DockerHub today.

…ershed_properties.ipynb

…:210415 to reproduce Raven nb prob

…ew Jupyter env Fixes #392. With the new Jupypter env for latest RavenPy `0.5.2` from PR Ouranosinc/PAVICS-e2e-workflow-tests#75 `Extract_geographical_watershed_properties.ipynb` fails with the following error below. ``` 23:00:58 _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 7 _ 23:00:58 Notebook cell execution failed 23:00:58 Cell 7: Cell execution caused an exception 23:00:58 23:00:58 Input: 23:00:58 # Note that geojson, and rasterio/rioxarray need to be installed for this to work. 23:00:58 # In the next birdy release, the tiff grid will be automatically converted to a DataArray. 23:00:58 # Here we need to manually convert it. 23:00:58 features, statistics, grid = stats_resp.get(asobj=True) 23:00:58 23:00:58 with rasterio.io.MemoryFile(grid[0]) as memfile: 23:00:58 with memfile.open() as dataset: 23:00:58 grid = rio.open_rasterio(dataset) 23:00:58 23:00:58 grid.plot() 23:00:58 23:00:58 Traceback: 23:00:58 23:00:58 --------------------------------------------------------------------------- 23:00:58 ValueError Traceback (most recent call last) 23:00:58 <ipython-input-8-8b167a3ecdc9> in <module> 23:00:58 4 features, statistics, grid = stats_resp.get(asobj=True) 23:00:58 5 23:00:58 ----> 6 with rasterio.io.MemoryFile(grid[0]) as memfile: 23:00:58 7 with memfile.open() as dataset: 23:00:58 8 grid = rio.open_rasterio(dataset) 23:00:58 23:00:58 /opt/conda/envs/birdy/lib/python3.7/site-packages/rasterio/io.py in __init__(self, file_or_bytes, dirname, filename, ext) 23:00:58 103 """ 23:00:58 104 super(MemoryFile, self).__init__( 23:00:58 --> 105 file_or_bytes=file_or_bytes, dirname=dirname, filename=filename, ext=ext) 23:00:58 106 23:00:58 107 @ensure_env 23:00:58 23:00:58 rasterio/_io.pyx in rasterio._io.MemoryFileBase.__init__() 23:00:58 23:00:58 /opt/conda/envs/birdy/lib/python3.7/site-packages/xarray/core/common.py in __bool__(self) 23:00:58 127 23:00:58 128 def __bool__(self: Any) -> bool: 23:00:58 --> 129 return bool(self.values) 23:00:58 130 23:00:58 131 def __float__(self: Any) -> float: 23:00:58 23:00:58 ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() 23:00:58 23:00:58 _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 10 _ 23:00:58 Notebook cell execution failed 23:00:58 Cell 10: Cell execution caused an exception 23:00:58 23:00:58 Input: 23:00:58 unique, counts = numpy.unique(grid.sel(band=1), return_counts=True) 23:00:58 display(unique, counts) 23:00:58 23:00:58 # Pixels values at '127' are NaN and can be ignored. 23:00:58 plt.bar(unique[:-1], counts[:-1]) 23:00:58 plt.show() 23:00:58 23:00:58 grid.where(grid != 127).sel(band=1).plot.imshow(cmap="tab20") 23:00:58 plt.show() 23:00:58 23:00:58 Traceback: 23:00:58 23:00:58 --------------------------------------------------------------------------- 23:00:58 AttributeError Traceback (most recent call last) 23:00:58 <ipython-input-11-a26c52126e9c> in <module> 23:00:58 ----> 1 unique, counts = numpy.unique(grid.sel(band=1), return_counts=True) 23:00:58 2 display(unique, counts) 23:00:58 3 23:00:58 4 # Pixels values at '127' are NaN and can be ignored. 23:00:58 5 plt.bar(unique[:-1], counts[:-1]) 23:00:58 23:00:58 AttributeError: 'list' object has no attribute 'sel' 23:00:58 23:00:58 _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 11 _ 23:00:58 Notebook cell execution failed 23:00:58 Cell 11: Cell execution caused an exception 23:00:58 23:00:58 Input: 23:00:58 import cartopy.crs as ccrs 23:00:58 23:00:58 # Set a CRS transformation: 23:00:58 crs = ccrs.LambertConformal( 23:00:58 central_latitude=49, central_longitude=-95, standard_parallels=(49, 77) 23:00:58 ) 23:00:58 23:00:58 ax = plt.subplot(projection=crs) 23:00:58 grid.name = "Land Use Categories" 23:00:58 grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20") 23:00:58 plt.show() 23:00:58 23:00:58 Traceback: 23:00:58 23:00:58 --------------------------------------------------------------------------- 23:00:58 AttributeError Traceback (most recent call last) 23:00:58 <ipython-input-12-f9fdb6f0e562> in <module> 23:00:58 7 23:00:58 8 ax = plt.subplot(projection=crs) 23:00:58 ----> 9 grid.name = "Land Use Categories" 23:00:58 10 grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20") 23:00:58 11 plt.show() 23:00:58 23:00:58 AttributeError: 'list' object has no attribute 'name' ```

…10615

Relevant changes: ```diff < - xclim=0.25.0=pyhd8ed1ab_0 > - xclim=0.27.0=pyhd8ed1ab_0 > - gcsfs=2021.6.0=pyhd8ed1ab_0 > - intake=0.6.2=pyhd8ed1ab_0 > - intake-esm=2021.1.15=pyhd8ed1ab_0 > - zarr=2.8.3=pyhd8ed1ab_0 ``` See PR Ouranosinc/PAVICS-e2e-workflow-tests#75 for more details.

…ut-for-jenkins finch-usage.ipynb: update output due to new birdy 0.8.0 New birdy removed `output_formats`. Clarification from DavidH: To clarify, the `output_formats` option still exists, but it's only appearing when the process has a complexoutput supporting more than one data format. bird-house/birdy@2bea46c#diff-ba0f245ac7a3747697d42961f7a19568e294dc5d2b4ad14b62a47d8c240ed12eR230-R238 Also removed `NBVAL_IGNORE_OUTPUT` since Jenkins is configured to by default ignore html changes, see PR Ouranosinc/PAVICS-e2e-workflow-tests#73. Jenkins passing build with upcoming Jupyter env: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-for-ravenpy/10/console New Jupyter env PR: Ouranosinc/PAVICS-e2e-workflow-tests#75 To fix this error: ``` _________ finch-master/docs/source/notebooks/finch-usage.ipynb::Cell 1 _________ Notebook cell execution failed Cell 1: Cell outputs differ Input: help(wps.frost_days) Traceback: mismatch 'stdout' assert reference_output == test_output failed: 'Help on meth...ut files.\n\n' == 'Help on meth...ut files.\n\n' Skipping 201 identical leading characters in diff, use -v to show - iable=None) method of birdy.client.base.WPSClient instance + iable=None, output_formats=None) method of birdy.client.base.WPSClient instance ? +++++++++++++++++++++ Number of days where daily minimum temperatures are below 0. Parameters ---------- tasmin : ComplexData:mimetype:`application/x-netcdf`, :mimetype:`application/x-ogc-dods` NetCDF Files or archive (tar/zip) containing netCDF files. thresh : string Freezing temperature. freq : {'YS', 'MS', 'QS-DEC', 'AS-JUL'}string Resampling frequency. check_missing : {'any', 'wmo', 'pct', 'at_least_n', 'skip', 'from_context'}string Method used to determine which aggregations should be considered missing. missing_options : ComplexData:mimetype:`application/json` JSON representation of dictionary of missing method parameters. cf_compliance : {'log', 'warn', 'raise'}string Whether to log, warn or raise when inputs have non-CF-compliant attributes. data_validation : {'log', 'warn', 'raise'}string Whether to log, warn or raise when inputs fail data validation checks. variable : string Name of the variable in the NetCDF file. Returns ------- output_netcdf : ComplexData:mimetype:`application/x-netcdf` The indicator values computed on the original input grid. output_log : ComplexData:mimetype:`text/plain` Collected logs during process run. ref : ComplexData:mimetype:`application/metalink+xml; version=4.0` Metalink file storing all references to output files. ```

…t-for-jenkins subsetting.ipynb: update output due to new birdy 0.8.0 New birdy removed `output_formats`. Clarification from DavidH: To clarify, the `output_formats` option still exists, but it's only appearing when the process has a complexoutput supporting more than one data format. bird-house/birdy@2bea46c#diff-ba0f245ac7a3747697d42961f7a19568e294dc5d2b4ad14b62a47d8c240ed12eR230-R238 Also removed `NBVAL_IGNORE_OUTPUT` since Jenkins is configured to by default ignore html changes, see PR Ouranosinc/PAVICS-e2e-workflow-tests#73. Jenkins passing build with upcoming Jupyter env: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-for-ravenpy/10/console New Jupyter env PR: Ouranosinc/PAVICS-e2e-workflow-tests#75 To fix this error: ``` _______ pavics-sdi-master/docs/source/notebooks/subsetting.ipynb::Cell 2 _______ Notebook cell execution failed Cell 2: Cell outputs differ Input: help(fp.subset_continents) Traceback: mismatch 'stdout' assert reference_output == test_output failed: 'Help on meth... outputs.\n\n' == 'Help on meth... outputs.\n\n' Skipping 101 identical leading characters in diff, use -v to show - n='Africa') method of birdy.client.base.WPSClient instance + n='Africa', output_formats=None) method of birdy.client.base.WPSClient instance ? +++++++++++++++++++++ Return the data whose grid cells intersect the selected continents for each input dataset. Parameters ---------- region : {'Africa', 'Asia', 'Australia', 'North America', 'Oceania', 'South America', 'Antarctica', 'Europe'}string Continent name. resource : ComplexData:mimetype:`application/x-netcdf`, :mimetype:`application/x-tar`, :mimetype:`application/zip` NetCDF Files or archive (tar/zip) containing netCDF files. Returns ------- output : ComplexData:mimetype:`application/x-netcdf` NetCDF output for first resource file. metalink : ComplexData:mimetype:`application/metalink+xml; version=4.0` Metalink file with links to all NetCDF outputs. ```

…nd-birdy jupyter: new version for updated ravenpy, birdy and xclim PR to deploy the new Jupyter env to PAVICS. See PR Ouranosinc/PAVICS-e2e-workflow-tests#75 for more details. Relevant changes: ```diff < - ravenpy=0.4.2=py37_1 > - ravenpy=0.5.2=pyh7f9bfb9_0 # Renamed. < - raven=3.0.4.318=hc9bffa2_2 > - raven-hydro=3.0.4.322=h516393e_0 < - ostrich=21.03.16=h2bc3f7f_0 > - ostrich=21.03.16=h4bd325d_1 < - xclim=0.25.0=pyhd8ed1ab_0 > - xclim=0.27.0=pyhd8ed1ab_0 # Old version was from pip. < - birdhouse-birdy==0.7.0 > - birdy=v0.8.0=pyh6c4a22f_0 # Was previously included in another package, now it is standalone. > - pydantic=1.8.2=py37h5e8e339_0 # New libs for upcoming Raven notebooks > - gcsfs=2021.6.0=pyhd8ed1ab_0 > - intake=0.6.2=pyhd8ed1ab_0 > - intake-esm=2021.1.15=pyhd8ed1ab_0 > - zarr=2.8.3=pyhd8ed1ab_0 < - xarray=0.17.0=pyhd8ed1ab_0 > - xarray=0.18.2=pyhd8ed1ab_0 < - owslib=0.23.0=pyhd8ed1ab_0 > - owslib=0.24.1=pyhd8ed1ab_0 < - cf_xarray=0.5.1=pyh44b312d_0 > - cf_xarray=0.5.2=pyh6c4a22f_0 < - clisops=0.6.3=pyh44b312d_0 > - clisops=0.6.5=pyh6c4a22f_0 < - dask=2021.2.0=pyhd8ed1ab_0 > - dask=2021.6.0=pyhd8ed1ab_0 # Downgrade ! < - gdal=3.2.1=py37hc5bc4e4_7 > - gdal=3.1.4=py37h2ec2946_8 # Downgrade ! < - rasterio=1.2.2=py37hd5c4cce_0 > - rasterio=1.2.1=py37ha549118_0 < - hvplot=0.7.1=pyh44b312d_0 > - hvplot=0.7.2=pyh6c4a22f_0 < - rioxarray=0.3.1=pyhd8ed1ab_0 > - rioxarray=0.4.1.post0=pyhd8ed1ab_0 # Downgrade ! < - xskillscore=0.0.19=pyhd8ed1ab_0 > - xskillscore=0.0.18=py_1 ``` Full diff of `conda env export`: [210415-210527.1-update210615-conda-env-export.diff.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/6658638/210415-210527.1-update210615-conda-env-export.diff.txt) Full new `conda env export`: [210527.1-update210615-conda-env-export.yml.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/6658646/210527.1-update210615-conda-env-export.yml.txt)

Extract_geographical_watershed_properties.ipynb: fix run failure in new Jupyter env Fixes #392 using hints from #388. Closes #388. Note I am not sure what I am doing as I do not understand the code. I simply try to get around the errors and produce similar graph. The old graph had `spatial_ref` in the title, the new one don't. Wonder if it's something critical. Clarification from Trevor: ========== Putting it simply, the object returned (`xarray.DataArray`) is different from what the following cell needed to handle for (`rasterio._io.RasterReader` → `numpy.ndarray`): ``` 23:00:58 Input: 23:00:58 # Note that geojson, and rasterio/rioxarray need to be installed for this to work. 23:00:58 # In the next birdy release, the tiff grid will be automatically converted to a DataArray. 23:00:58 # Here we need to manually convert it. 23:00:58 features, statistics, grid = stats_resp.get(asobj=True) 23:00:58 23:00:58 with rasterio.io.MemoryFile(grid[0]) as memfile: 23:00:58 with memfile.open() as dataset: 23:00:58 grid = rio.open_rasterio(dataset) 23:00:58 23:00:58 grid.plot() ``` The `spatial_ref`title is something that `rasterio` adds to the plot, and it isn't at all needed. The notebook should work perfectly so long as the client running it has `rioxarray`available in their ipython kernel. ========== With the new Jupypter env for latest RavenPy `0.5.2` from PR Ouranosinc/PAVICS-e2e-workflow-tests#75 `Extract_geographical_watershed_properties.ipynb` fails with the following error below. ``` 23:00:58 _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 7 _ 23:00:58 Notebook cell execution failed 23:00:58 Cell 7: Cell execution caused an exception 23:00:58 23:00:58 Input: 23:00:58 # Note that geojson, and rasterio/rioxarray need to be installed for this to work. 23:00:58 # In the next birdy release, the tiff grid will be automatically converted to a DataArray. 23:00:58 # Here we need to manually convert it. 23:00:58 features, statistics, grid = stats_resp.get(asobj=True) 23:00:58 23:00:58 with rasterio.io.MemoryFile(grid[0]) as memfile: 23:00:58 with memfile.open() as dataset: 23:00:58 grid = rio.open_rasterio(dataset) 23:00:58 23:00:58 grid.plot() 23:00:58 23:00:58 Traceback: 23:00:58 23:00:58 --------------------------------------------------------------------------- 23:00:58 ValueError Traceback (most recent call last) 23:00:58 <ipython-input-8-8b167a3ecdc9> in <module> 23:00:58 4 features, statistics, grid = stats_resp.get(asobj=True) 23:00:58 5 23:00:58 ----> 6 with rasterio.io.MemoryFile(grid[0]) as memfile: 23:00:58 7 with memfile.open() as dataset: 23:00:58 8 grid = rio.open_rasterio(dataset) 23:00:58 23:00:58 /opt/conda/envs/birdy/lib/python3.7/site-packages/rasterio/io.py in __init__(self, file_or_bytes, dirname, filename, ext) 23:00:58 103 """ 23:00:58 104 super(MemoryFile, self).__init__( 23:00:58 --> 105 file_or_bytes=file_or_bytes, dirname=dirname, filename=filename, ext=ext) 23:00:58 106 23:00:58 107 @ensure_env 23:00:58 23:00:58 rasterio/_io.pyx in rasterio._io.MemoryFileBase.__init__() 23:00:58 23:00:58 /opt/conda/envs/birdy/lib/python3.7/site-packages/xarray/core/common.py in __bool__(self) 23:00:58 127 23:00:58 128 def __bool__(self: Any) -> bool: 23:00:58 --> 129 return bool(self.values) 23:00:58 130 23:00:58 131 def __float__(self: Any) -> float: 23:00:58 23:00:58 ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() 23:00:58 23:00:58 _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 10 _ 23:00:58 Notebook cell execution failed 23:00:58 Cell 10: Cell execution caused an exception 23:00:58 23:00:58 Input: 23:00:58 unique, counts = numpy.unique(grid.sel(band=1), return_counts=True) 23:00:58 display(unique, counts) 23:00:58 23:00:58 # Pixels values at '127' are NaN and can be ignored. 23:00:58 plt.bar(unique[:-1], counts[:-1]) 23:00:58 plt.show() 23:00:58 23:00:58 grid.where(grid != 127).sel(band=1).plot.imshow(cmap="tab20") 23:00:58 plt.show() 23:00:58 23:00:58 Traceback: 23:00:58 23:00:58 --------------------------------------------------------------------------- 23:00:58 AttributeError Traceback (most recent call last) 23:00:58 <ipython-input-11-a26c52126e9c> in <module> 23:00:58 ----> 1 unique, counts = numpy.unique(grid.sel(band=1), return_counts=True) 23:00:58 2 display(unique, counts) 23:00:58 3 23:00:58 4 # Pixels values at '127' are NaN and can be ignored. 23:00:58 5 plt.bar(unique[:-1], counts[:-1]) 23:00:58 23:00:58 AttributeError: 'list' object has no attribute 'sel' 23:00:58 23:00:58 _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 11 _ 23:00:58 Notebook cell execution failed 23:00:58 Cell 11: Cell execution caused an exception 23:00:58 23:00:58 Input: 23:00:58 import cartopy.crs as ccrs 23:00:58 23:00:58 # Set a CRS transformation: 23:00:58 crs = ccrs.LambertConformal( 23:00:58 central_latitude=49, central_longitude=-95, standard_parallels=(49, 77) 23:00:58 ) 23:00:58 23:00:58 ax = plt.subplot(projection=crs) 23:00:58 grid.name = "Land Use Categories" 23:00:58 grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20") 23:00:58 plt.show() 23:00:58 23:00:58 Traceback: 23:00:58 23:00:58 --------------------------------------------------------------------------- 23:00:58 AttributeError Traceback (most recent call last) 23:00:58 <ipython-input-12-f9fdb6f0e562> in <module> 23:00:58 7 23:00:58 8 ax = plt.subplot(projection=crs) 23:00:58 ----> 9 grid.name = "Land Use Categories" 23:00:58 10 grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20") 23:00:58 11 plt.show() 23:00:58 23:00:58 AttributeError: 'list' object has no attribute 'name' ```

@huard

New docker build for raven notebooks Add `intake-xarray` (fixes Ouranosinc/pavics-sdi#223) @huard @tlogan2000 also proactively add `intake-geopandas` and `intake-thredds` as they look interesting. Tried to add `intake-stac` but got into conflicting dependencies during build so had to leave it out. This is an incremental build (not a full build) from the previous release (#75) so only those are changed (so regression risk is extremely low): ``` package | build ---------------------------|----------------- intake-geopandas-0.2.4 | pyhd8ed1ab_0 12 KB conda-forge intake-thredds-2021.6.16 | pyhd8ed1ab_0 13 KB conda-forge intake-xarray-0.5.0 | pyhd8ed1ab_0 1.4 MB conda-forge ------------------------------------------------------------ ``` Jenkins build with only known error: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-raven-notebooks/1/console Raven notebooks passing: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-raven-notebooks/2/console Manual tested Pavics homepage notebooks as well, using https://medus.ouranos.ca/jupyter since this image is deployed there. Matching PR to deploy to PAVICS bird-house/birdhouse-deploy#178

…ew Jupyter env Fixes #392. With the new Jupypter env for latest RavenPy `0.5.2` from PR Ouranosinc/PAVICS-e2e-workflow-tests#75 `Extract_geographical_watershed_properties.ipynb` fails with the following error below. ``` 23:00:58 _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 7 _ 23:00:58 Notebook cell execution failed 23:00:58 Cell 7: Cell execution caused an exception 23:00:58 23:00:58 Input: 23:00:58 # Note that geojson, and rasterio/rioxarray need to be installed for this to work. 23:00:58 # In the next birdy release, the tiff grid will be automatically converted to a DataArray. 23:00:58 # Here we need to manually convert it. 23:00:58 features, statistics, grid = stats_resp.get(asobj=True) 23:00:58 23:00:58 with rasterio.io.MemoryFile(grid[0]) as memfile: 23:00:58 with memfile.open() as dataset: 23:00:58 grid = rio.open_rasterio(dataset) 23:00:58 23:00:58 grid.plot() 23:00:58 23:00:58 Traceback: 23:00:58 23:00:58 --------------------------------------------------------------------------- 23:00:58 ValueError Traceback (most recent call last) 23:00:58 <ipython-input-8-8b167a3ecdc9> in <module> 23:00:58 4 features, statistics, grid = stats_resp.get(asobj=True) 23:00:58 5 23:00:58 ----> 6 with rasterio.io.MemoryFile(grid[0]) as memfile: 23:00:58 7 with memfile.open() as dataset: 23:00:58 8 grid = rio.open_rasterio(dataset) 23:00:58 23:00:58 /opt/conda/envs/birdy/lib/python3.7/site-packages/rasterio/io.py in __init__(self, file_or_bytes, dirname, filename, ext) 23:00:58 103 """ 23:00:58 104 super(MemoryFile, self).__init__( 23:00:58 --> 105 file_or_bytes=file_or_bytes, dirname=dirname, filename=filename, ext=ext) 23:00:58 106 23:00:58 107 @ensure_env 23:00:58 23:00:58 rasterio/_io.pyx in rasterio._io.MemoryFileBase.__init__() 23:00:58 23:00:58 /opt/conda/envs/birdy/lib/python3.7/site-packages/xarray/core/common.py in __bool__(self) 23:00:58 127 23:00:58 128 def __bool__(self: Any) -> bool: 23:00:58 --> 129 return bool(self.values) 23:00:58 130 23:00:58 131 def __float__(self: Any) -> float: 23:00:58 23:00:58 ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() 23:00:58 23:00:58 _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 10 _ 23:00:58 Notebook cell execution failed 23:00:58 Cell 10: Cell execution caused an exception 23:00:58 23:00:58 Input: 23:00:58 unique, counts = numpy.unique(grid.sel(band=1), return_counts=True) 23:00:58 display(unique, counts) 23:00:58 23:00:58 # Pixels values at '127' are NaN and can be ignored. 23:00:58 plt.bar(unique[:-1], counts[:-1]) 23:00:58 plt.show() 23:00:58 23:00:58 grid.where(grid != 127).sel(band=1).plot.imshow(cmap="tab20") 23:00:58 plt.show() 23:00:58 23:00:58 Traceback: 23:00:58 23:00:58 --------------------------------------------------------------------------- 23:00:58 AttributeError Traceback (most recent call last) 23:00:58 <ipython-input-11-a26c52126e9c> in <module> 23:00:58 ----> 1 unique, counts = numpy.unique(grid.sel(band=1), return_counts=True) 23:00:58 2 display(unique, counts) 23:00:58 3 23:00:58 4 # Pixels values at '127' are NaN and can be ignored. 23:00:58 5 plt.bar(unique[:-1], counts[:-1]) 23:00:58 23:00:58 AttributeError: 'list' object has no attribute 'sel' 23:00:58 23:00:58 _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 11 _ 23:00:58 Notebook cell execution failed 23:00:58 Cell 11: Cell execution caused an exception 23:00:58 23:00:58 Input: 23:00:58 import cartopy.crs as ccrs 23:00:58 23:00:58 # Set a CRS transformation: 23:00:58 crs = ccrs.LambertConformal( 23:00:58 central_latitude=49, central_longitude=-95, standard_parallels=(49, 77) 23:00:58 ) 23:00:58 23:00:58 ax = plt.subplot(projection=crs) 23:00:58 grid.name = "Land Use Categories" 23:00:58 grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20") 23:00:58 plt.show() 23:00:58 23:00:58 Traceback: 23:00:58 23:00:58 --------------------------------------------------------------------------- 23:00:58 AttributeError Traceback (most recent call last) 23:00:58 <ipython-input-12-f9fdb6f0e562> in <module> 23:00:58 7 23:00:58 8 ax = plt.subplot(projection=crs) 23:00:58 ----> 9 grid.name = "Land Use Categories" 23:00:58 10 grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20") 23:00:58 11 plt.show() 23:00:58 23:00:58 AttributeError: 'list' object has no attribute 'name' ```

Extract_geographical_watershed_properties.ipynb: fix run failure in new Jupyter env Fixes #392 using hints from #388. Closes #388. Note I am not sure what I am doing as I do not understand the code. I simply try to get around the errors and produce similar graph. The old graph had `spatial_ref` in the title, the new one don't. Wonder if it's something critical. Clarification from Trevor: ========== Putting it simply, the object returned (`xarray.DataArray`) is different from what the following cell needed to handle for (`rasterio._io.RasterReader` → `numpy.ndarray`): ``` 23:00:58 Input: 23:00:58 # Note that geojson, and rasterio/rioxarray need to be installed for this to work. 23:00:58 # In the next birdy release, the tiff grid will be automatically converted to a DataArray. 23:00:58 # Here we need to manually convert it. 23:00:58 features, statistics, grid = stats_resp.get(asobj=True) 23:00:58 23:00:58 with rasterio.io.MemoryFile(grid[0]) as memfile: 23:00:58 with memfile.open() as dataset: 23:00:58 grid = rio.open_rasterio(dataset) 23:00:58 23:00:58 grid.plot() ``` The `spatial_ref`title is something that `rasterio` adds to the plot, and it isn't at all needed. The notebook should work perfectly so long as the client running it has `rioxarray`available in their ipython kernel. ========== With the new Jupypter env for latest RavenPy `0.5.2` from PR Ouranosinc/PAVICS-e2e-workflow-tests#75 `Extract_geographical_watershed_properties.ipynb` fails with the following error below. ``` 23:00:58 _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 7 _ 23:00:58 Notebook cell execution failed 23:00:58 Cell 7: Cell execution caused an exception 23:00:58 23:00:58 Input: 23:00:58 # Note that geojson, and rasterio/rioxarray need to be installed for this to work. 23:00:58 # In the next birdy release, the tiff grid will be automatically converted to a DataArray. 23:00:58 # Here we need to manually convert it. 23:00:58 features, statistics, grid = stats_resp.get(asobj=True) 23:00:58 23:00:58 with rasterio.io.MemoryFile(grid[0]) as memfile: 23:00:58 with memfile.open() as dataset: 23:00:58 grid = rio.open_rasterio(dataset) 23:00:58 23:00:58 grid.plot() 23:00:58 23:00:58 Traceback: 23:00:58 23:00:58 --------------------------------------------------------------------------- 23:00:58 ValueError Traceback (most recent call last) 23:00:58 <ipython-input-8-8b167a3ecdc9> in <module> 23:00:58 4 features, statistics, grid = stats_resp.get(asobj=True) 23:00:58 5 23:00:58 ----> 6 with rasterio.io.MemoryFile(grid[0]) as memfile: 23:00:58 7 with memfile.open() as dataset: 23:00:58 8 grid = rio.open_rasterio(dataset) 23:00:58 23:00:58 /opt/conda/envs/birdy/lib/python3.7/site-packages/rasterio/io.py in __init__(self, file_or_bytes, dirname, filename, ext) 23:00:58 103 """ 23:00:58 104 super(MemoryFile, self).__init__( 23:00:58 --> 105 file_or_bytes=file_or_bytes, dirname=dirname, filename=filename, ext=ext) 23:00:58 106 23:00:58 107 @ensure_env 23:00:58 23:00:58 rasterio/_io.pyx in rasterio._io.MemoryFileBase.__init__() 23:00:58 23:00:58 /opt/conda/envs/birdy/lib/python3.7/site-packages/xarray/core/common.py in __bool__(self) 23:00:58 127 23:00:58 128 def __bool__(self: Any) -> bool: 23:00:58 --> 129 return bool(self.values) 23:00:58 130 23:00:58 131 def __float__(self: Any) -> float: 23:00:58 23:00:58 ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() 23:00:58 23:00:58 _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 10 _ 23:00:58 Notebook cell execution failed 23:00:58 Cell 10: Cell execution caused an exception 23:00:58 23:00:58 Input: 23:00:58 unique, counts = numpy.unique(grid.sel(band=1), return_counts=True) 23:00:58 display(unique, counts) 23:00:58 23:00:58 # Pixels values at '127' are NaN and can be ignored. 23:00:58 plt.bar(unique[:-1], counts[:-1]) 23:00:58 plt.show() 23:00:58 23:00:58 grid.where(grid != 127).sel(band=1).plot.imshow(cmap="tab20") 23:00:58 plt.show() 23:00:58 23:00:58 Traceback: 23:00:58 23:00:58 --------------------------------------------------------------------------- 23:00:58 AttributeError Traceback (most recent call last) 23:00:58 <ipython-input-11-a26c52126e9c> in <module> 23:00:58 ----> 1 unique, counts = numpy.unique(grid.sel(band=1), return_counts=True) 23:00:58 2 display(unique, counts) 23:00:58 3 23:00:58 4 # Pixels values at '127' are NaN and can be ignored. 23:00:58 5 plt.bar(unique[:-1], counts[:-1]) 23:00:58 23:00:58 AttributeError: 'list' object has no attribute 'sel' 23:00:58 23:00:58 _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 11 _ 23:00:58 Notebook cell execution failed 23:00:58 Cell 11: Cell execution caused an exception 23:00:58 23:00:58 Input: 23:00:58 import cartopy.crs as ccrs 23:00:58 23:00:58 # Set a CRS transformation: 23:00:58 crs = ccrs.LambertConformal( 23:00:58 central_latitude=49, central_longitude=-95, standard_parallels=(49, 77) 23:00:58 ) 23:00:58 23:00:58 ax = plt.subplot(projection=crs) 23:00:58 grid.name = "Land Use Categories" 23:00:58 grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20") 23:00:58 plt.show() 23:00:58 23:00:58 Traceback: 23:00:58 23:00:58 --------------------------------------------------------------------------- 23:00:58 AttributeError Traceback (most recent call last) 23:00:58 <ipython-input-12-f9fdb6f0e562> in <module> 23:00:58 7 23:00:58 8 ax = plt.subplot(projection=crs) 23:00:58 ----> 9 grid.name = "Land Use Categories" 23:00:58 10 grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20") 23:00:58 11 plt.show() 23:00:58 23:00:58 AttributeError: 'list' object has no attribute 'name' ```

tlvu added 7 commits May 25, 2021 09:43

release: update to use image pavics/workflow-tests:210525

c4d6da1

docker: add new Dockerfile for testing new package

ad6d514

Useful to avoid full rebuild just to test adding a new package or upgrading an existing package. Sort of a restore of commit cbfa5d5.

docker: add pydantic, was part of previous build but missing in curre…

8103b74

…nt build Previous build probably got pydantic via indirect dependency but somehow this has changed so now have to add pydantic explicitly, for Raven notebooks.

release: update to use image pavics/workflow-tests:210526

155ff69

docker: remove conda channel zeitsperre, ravenpy now available from c…

eed7904

…onda-forge

release: update to use image pavics/workflow-tests:210527

be6aa30

tlvu requested review from huard, cjauvin and aulemahal May 27, 2021 16:31

tlvu added 2 commits May 27, 2021 13:12

docker: pin previous version of cftime due to issue with nc-time-axis

6e84c79

release: update to use image pavics/workflow-tests:210527.1

2715a31

huard approved these changes May 27, 2021

View reviewed changes

tlvu mentioned this pull request Jun 1, 2021

Extract_geographical_watershed_properties.ipynb failure in new Jupyter env Ouranosinc/raven#392

Closed

tlvu added 2 commits June 1, 2021 18:34

docker: test pin rasterio=1.2.2 to fix Raven Extract_geographical_wat…

61a6ddb

…ershed_properties.ipynb

docker: test pin rasterio=1.2.1 from older base pavics/workflow-tests…

0c82a3e

…:210415 to reproduce Raven nb prob

tlvu mentioned this pull request Jun 10, 2021

Extract_geographical_watershed_properties.ipynb: fix run failure in new Jupyter env Ouranosinc/raven#393

Merged

tlvu added 2 commits June 15, 2021 12:15

docker: update latest xclim 0.27.0

e1f5087

docker: add intake intake-esm gcsfs zarr for Raven notebooks

08f652c

release: update to use image pavics/workflow-tests:210527.1-update202…

cfb10a2

…10615

cjauvin approved these changes Jun 15, 2021

View reviewed changes

tlvu merged commit fae2468 into master Jun 15, 2021

tlvu deleted the new-docker-build-for-ravenpy branch June 15, 2021 21:34

tlvu mentioned this pull request Jun 18, 2021

New docker build for raven notebooks #76

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New docker build for ravenpy birdy xclim #75

New docker build for ravenpy birdy xclim #75

tlvu commented May 27, 2021 •

edited

Loading

aulemahal commented May 27, 2021

tlvu commented May 27, 2021

tlvu commented May 27, 2021

aulemahal commented May 27, 2021

tlogan2000 commented May 27, 2021

tlvu commented May 28, 2021

New docker build for ravenpy birdy xclim #75

New docker build for ravenpy birdy xclim #75

Conversation

tlvu commented May 27, 2021 • edited Loading

Overview

Changes

aulemahal commented May 27, 2021

tlvu commented May 27, 2021

tlvu commented May 27, 2021

aulemahal commented May 27, 2021

tlogan2000 commented May 27, 2021

tlvu commented May 28, 2021

tlvu commented May 27, 2021 •

edited

Loading