Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update nbs, tutorial data and html versions #78

Merged
merged 8 commits into from
May 9, 2024
Merged

Conversation

tlogan2000
Copy link
Collaborator

@tlogan2000 tlogan2000 commented Mar 8, 2024

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@tlogan2000 tlogan2000 requested a review from tlvu March 8, 2024 20:32
tlvu added 3 commits March 26, 2024 10:56
```
  _ PAVICS-landing-fix_nbs_jupyter_alpha_refresh_output/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb::Cell 0 _
  Notebook cell execution failed
  Cell 0: Cell outputs differ

  Input:
  import warnings

  import xarray as xr
  from IPython.display import display  # Fancy representation of xarray objects
  from siphon.catalog import TDSCatalog

  warnings.simplefilter("ignore")
  url = "https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/catalog/datasets/simulations/bias_adjusted/cmip5/ouranos/cb-oura-1.0/catalog.xml"  # TEST_USE_PROD_DATA

  # Create Catalog
  cat = TDSCatalog(url)

  # DAP link for this demo
  ds_url = cat.datasets[0].access_urls["OPENDAP"]

  # xarray.Dataset
  ds = xr.open_dataset(ds_url, chunks=dict(time=256 * 2, lon=32, lat=32))
  display(ds)
  a = ds.tasmin.isel(time=0).plot(figsize=(10, 4))

  Traceback:
   mismatch 'text/plain'

   assert reference_output == test_output failed:

    '<xarray.Data...id:       NCC' == '<xarray.Data...id:       NCC'

    Skipping 957 identical leading characters in diff, use -v to show
    -   driving_experiment:         historical,rcp85
    +   driving_model:              NorESM1-M
          ...                         ...
          frequency:                  day
          modeling_realm:             atmos
          target_dataset:             CANADA : ANUSPLIN interpolated Canada daily 3...
          target_dataset_references:  CANADA : https://doi.org/10.1175/2011BAMS3132...
          driving_institution:        Norwegian Climate Centre
          driving_institute_id:       NCC

  _ PAVICS-landing-fix_nbs_jupyter_alpha_refresh_output/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb::Cell 1 _
  Notebook cell execution failed
  Cell 1: Cell outputs differ

  Input:
  from clisops.core import subset

  lon = [-75.4, -85, -65.5]  # Longitude
  lat = [46.67, 41, 55.3]  # Latitude

  ds_gridpoint = subset.subset_gridpoint(ds, lon=lon, lat=lat)
  display(ds_gridpoint)

  # Plot first year of tasmax data
  a = ds_gridpoint.tasmax.isel(time=slice(0, 365)).plot.line(x="time", figsize=(10, 4))

  Traceback:
   mismatch 'text/plain'

   assert reference_output == test_output failed:

    '<xarray.Data...id:       NCC' == '<xarray.Data...id:       NCC'

    Skipping 892 identical leading characters in diff, use -v to show
    -   driving_experiment:         historical,rcp85
    +   driving_model:              NorESM1-M
          ...                         ...
          frequency:                  day
          modeling_realm:             atmos
          target_dataset:             CANADA : ANUSPLIN interpolated Canada daily 3...
          target_dataset_references:  CANADA : https://doi.org/10.1175/2011BAMS3132...
          driving_institution:        Norwegian Climate Centre
          driving_institute_id:       NCC

  _ PAVICS-landing-fix_nbs_jupyter_alpha_refresh_output/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb::Cell 3 _
  Notebook cell execution failed
  Cell 3: Cell outputs differ

  Input:
  # Specify the longitude and latitude boundaries
  lon_bnds = [-80.5, -60.2]
  lat_bnds = [44, 55]

  ds1 = subset.subset_bbox(ds, lon_bnds=lon_bnds, lat_bnds=lat_bnds)
  display(ds1)

  # Plot a map of first timestep
  a = ds1.tasmax.isel(time=0).plot()

  Traceback:
   mismatch 'text/plain'

   assert reference_output == test_output failed:

    '<xarray.Data...id:       NCC' == '<xarray.Data...id:       NCC'

    Skipping 955 identical leading characters in diff, use -v to show
    -   driving_experiment:         historical,rcp85
    +   driving_model:              NorESM1-M
          ...                         ...
          frequency:                  day
          modeling_realm:             atmos
          target_dataset:             CANADA : ANUSPLIN interpolated Canada daily 3...
          target_dataset_references:  CANADA : https://doi.org/10.1175/2011BAMS3132...
          driving_institution:        Norwegian Climate Centre
          driving_institute_id:       NCC
```
```
  _ PAVICS-landing-fix_nbs_jupyter_alpha_refresh_output/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb::Cell 2 _
  Notebook cell execution failed
  Cell 2: Cell outputs differ

  Input:
  import xclim

  # Access some station data
  url = "https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/datasets/station_obs/ECCC_AHCCD_gen3_temperature.ncml"
  ds_stat = xr.open_dataset(url, chunks=dict(station=1))
  istat = 115
  # Compare use of differernt percent tolerance levels  (tolerance == 0.1)
  plt.figure(figsize=(15, 10))
  color = ["blue", "green", "orange"]

  for i, tol in enumerate([0.25, 0.1, 0.05]):
      plt.subplot(3, 1, i + 1)
      with xclim.set_options(
          check_missing="pct", missing_options={"pct": {"tolerance": tol}}
      ):
          # compute yearly max tasmax
          tx_mean = xclim.atmos.tx_mean(
              tasmax=ds_stat.isel(station=istat).tasmax, freq="YS"
          )

          tx_mean.plot(
              marker="o",
              color=color[i],
              label=f"{len(tx_mean.dropna('time').values)} \
                       valid years w/ <={tol*100}% missing data",
          )
          plt.title("")
          plt.xlabel("")
          if i == 0:
              name = str(ds_stat.isel(station=istat).station_name.values)
              plt.title(f"{name.capitalize().replace('_',' ')} Station")

          elif i == 2:
              plt.xlabel("time")

          plt.legend(loc="lower right")

  Traceback:
  Unexpected output fields from running code: {'stderr'}
```
Attempt to fix the following error:
```
  _ PAVICS-landing-fix_nbs_jupyter_alpha_refresh_output/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb::Cell 2 _
  Notebook cell execution failed
  Cell 2: Cell outputs differ

  Input:
  import xclim

  # Access some station data
  url = "https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/datasets/station_obs/ECCC_AHCCD_gen3_temperature.ncml"
  ds_stat = xr.open_dataset(url, chunks=dict(station=1))
  istat = 115
  # Compare use of differernt percent tolerance levels  (tolerance == 0.1)
  plt.figure(figsize=(15, 10))
  color = ["blue", "green", "orange"]

  for i, tol in enumerate([0.25, 0.1, 0.05]):
      plt.subplot(3, 1, i + 1)
      with xclim.set_options(
          check_missing="pct", missing_options={"pct": {"tolerance": tol}}
      ):
          # compute yearly max tasmax
          tx_mean = xclim.atmos.tx_mean(
              tasmax=ds_stat.isel(station=istat).tasmax, freq="YS"
          )

          tx_mean.plot(
              marker="o",
              color=color[i],
              label=f"{len(tx_mean.dropna('time').values)} \
                       valid years w/ <={tol*100}% missing data",
          )
          plt.title("")
          plt.xlabel("")
          if i == 0:
              name = str(ds_stat.isel(station=istat).station_name.values)
              plt.title(f"{name.capitalize().replace('_',' ')} Station")

          elif i == 2:
              plt.xlabel("time")

          plt.legend(loc="lower right")

  Traceback:
  Missing output fields from running code: {'stderr'}
```
@tlvu tlvu mentioned this pull request May 1, 2024
tlogan2000 and others added 2 commits May 8, 2024 08:44
…into fix_nbs_jupyter_alpha

� Conflicts:
�	content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb
�	content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb
�	content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.html
�	content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb
�	content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb
�	content/notebooks/climate_indicators/output.zip
�	src/assets/notebooks/PAVICStutorial_ClimateDataAnalysis-4Ensembles.html
tlvu added a commit to Ouranosinc/PAVICS-e2e-workflow-tests that referenced this pull request May 9, 2024
…t xclim and ravenpy to smooth transition (#121)

# Overview

This new full build has latest of almost everything except `xclim` and
`ravenpy` as intermediate step to smooth transition to `pandas` 2.2 freq
strings changes.

## Changes

- New: save conda env export, DockerHub build logs and Jenkins test
result in the repo to track changes much more easily between releases

- Jenkins: add `SAVE_RESULTING_NOTEBOOK_TIMEOUT` for slow notebooks or
slow machine

- Jupyter env changes:
- add `conda-pack` so we can export the conda env outside of the docker
image if need to run locally without docker
  - upgrade from Python 3.9 to 3.11
  - Relevant changes (alphabetical order):
```diff
-  - birdy=0.8.4=pyh1a96a4e_0
+      - birdhouse-birdy==0.8.7

# major upgrade from v2 to v3
-  - bokeh=2.4.3=pyhd8ed1ab_3
+  - bokeh=3.4.1=pyhd8ed1ab_0

-  - cartopy=0.21.1=py39h6e7ad6e_0
+  - cartopy=0.23.0=py311h320fe9a_0

-  - cf_xarray=0.8.0=pyhd8ed1ab_0
+  - cf_xarray=0.9.0=pyhd8ed1ab_0

-  - cfgrib=0.9.10.4=pyhd8ed1ab_0
+  - cfgrib=0.9.11.0=pyhd8ed1ab_0

-  - cftime=1.6.2=py39h2ae25f5_1
+  - cftime=1.6.3=py311h1f0f07a_0

-  - climpred=2.3.0=pyhd8ed1ab_0
+  - climpred=2.4.0=pyhd8ed1ab_0

-  - clisops=0.9.6=pyh1a96a4e_0
+  - clisops=0.13.0=pyhca7485f_0

-  - dask=2023.5.1=pyhd8ed1ab_0
+  - dask=2024.5.0=pyhd8ed1ab_0

-  - geopandas=0.13.0=pyhd8ed1ab_0
+  - geopandas=0.14.4=pyhd8ed1ab_0

-  - hvplot=0.8.3=pyhd8ed1ab_0
+  - hvplot=0.9.2=pyhd8ed1ab_0

-  - numpy=1.23.5=py39h3d75532_0
+  - numpy=1.24.4=py311h64a7726_0

-  - numba=0.57.0=py39hb75a051_1
+  - numba=0.59.1=py311h96b013e_0

# major upgrade from v1 to v2
-  - pandas=1.3.5=py39hde0f152_0
+  - pandas=2.1.4=py311h320fe9a_0

# major upgrade to v1
-  - panel=0.14.4=pyhd8ed1ab_0
+  - panel=1.4.2=pyhd8ed1ab_0

# major upgrade from v1 to v2
-  - pydantic=1.10.8=py39hd1e30aa_0
+  - pydantic=2.7.1=pyhd8ed1ab_0

# Python 3.9 to 3.11
-  - python=3.9.16=h2782a2a_0_cpython
+  - python=3.11.6=hab00c5b_0_cpython

-  - raven-hydro=0.2.1=py39h8e2dbb5_1
+  - raven-hydro=0.2.4=py311h64a4d7b_0

-  - ravenpy=0.12.1=py39hf3d152e_0
+      - ravenpy==0.13.1

-  - rioxarray=0.14.1=pyhd8ed1ab_0
+  - rioxarray=0.15.5=pyhd8ed1ab_0

-  - roocs-utils=0.6.4=pyh1a96a4e_0
+  - roocs-utils=0.6.8=pyhd8ed1ab_0

-  - scipy=1.9.1=py39h8ba3f38_0
+  - scipy=1.13.0=py311h517d4fd_1

-  - xarray=2023.1.0=pyhd8ed1ab_0
+  - xarray=2023.8.0=pyhd8ed1ab_0

-  - xclim=0.43.0=py39hf3d152e_1
+  - xclim=0.47.0=py311h38be061_0

-  - xesmf=0.7.1=pyhd8ed1ab_0
+  - xesmf=0.8.5=pyhd8ed1ab_0

-  - xskillscore=0.0.24=pyhd8ed1ab_0
+  - xskillscore=0.0.26=pyhd8ed1ab_0

+  - xscen=0.8.2=pyhd8ed1ab_0

+      - figanos==0.3.0

-      - xncml==0.2
+      - xncml==0.4.0

```


## Test

- Deployed as "beta" image in production for bokeh visualization
performance regression testing.
- Manual test notebook
https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb
for bokeh visualization performance and it looks fine.
- Jenkins build:
- Default notebooks, all passed:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/54792e6510adfcd1bb21e1bd31fdfa36c5c634e0/docker/saved_buildout/jenkins-buildlogs-default.txt
- Raven notebooks, only known `HydroShare_integration.ipynb` failing:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/jenkins-buildlogs-raven.txt


## Related Issue / Discussion

- Matching notebook fixes:
  - Pavics-sdi: PR Ouranosinc/pavics-sdi#321
  - Finch: PR url: None
- PAVICS-landing: PR
Ouranosinc/PAVICS-landing#78
  - RavenPy: PR CSHS-CWRA/RavenPy#356
  - Resolves Ouranosinc/PAVICS-landing#65
  - Resolves Ouranosinc/PAVICS-landing#66

- Deployment to PAVICS:
bird-house/birdhouse-deploy#453

- Jenkins-config changes for new notebooks: PR url: None

- Other issues found while working on this one
  - computationalmodelling/nbval#204
  - jupyterlab-contrib/jupyter-archive#132
  - CSHS-CWRA/RavenPy#357
  - CSHS-CWRA/RavenPy#361
  - CSHS-CWRA/RavenPy#362

- Previous release: PR
#134


## Additional Information

Full diff conda env export:

81deb99...931cfc9#diff-e8f2a6a53085ae29bb7cedc701c1d345a330651ae971555e85a5c005e94f4cd9


Full new conda env export:

https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/conda-env-export.yml


DockerHub build log

https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/docker-buildlogs.txt
@tlvu tlvu merged commit e69f2da into master May 9, 2024
1 check passed
@tlvu tlvu deleted the fix_nbs_jupyter_alpha branch May 9, 2024 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants