Skip to content

Commit

Permalink
Jupyter docker: new full build with latest of everything (#137)
Browse files Browse the repository at this point in the history
# Overview

New full build with latest of everything.


## Changes

- New: save output of `conda env export` and environment size directly
in the docker image for ease of tracking change between releases.
- Updated pull request template for new docker release.
- Jupyter env changes:
- Unpin `libnetcdf`
(Ouranosinc/PAVICS-landing#66 fixed).
  - Avoid `dask != 2024.11.0` due to bugs with Raven nb.
  - Relevant changes (alphabetical order):
```diff
-  - bokeh=3.4.1=pyhd8ed1ab_0    
+  - bokeh=3.5.2=pyhd8ed1ab_0          

-  - cartopy=0.23.0=py311h320fe9a_0
+  - cartopy=0.24.0=py311h7db5c69_0

-  - cf_xarray=0.9.0=pyhd8ed1ab_0
+  - cf_xarray=0.10.0=pyhd8ed1ab_0

-  - cfgrib=0.9.11.0=pyhd8ed1ab_0      
+  - cfgrib=0.9.14.1=pyhd8ed1ab_0

-  - cftime=1.6.3=py311h1f0f07a_0
+  - cftime=1.6.4=py311h9f3472d_1   
   
-  - climpred=2.4.0=pyhd8ed1ab_0 
+  - climpred=2.5.0=pyhd8ed1ab_0   

-  - clisops=0.13.0=pyhca7485f_0 
+  - clisops=0.14.1=pyhd8ed1ab_0  

-  - dask=2024.5.0=pyhd8ed1ab_0   
+  - dask=2024.10.0=pyhd8ed1ab_0     

-  - esmf=8.4.0=nompi_hdb2cfa9_4      
+  - esmf=8.6.1=nompi_h4441c20_3   

-  - fiona=1.9.1=py311h3f14cef_0      
+  - fiona=1.9.5=py311hf8e0aa6_2    

-  - gdal=3.6.2=py311hadb6153_6          
+  - gdal=3.8.5=py311hf92cf48_11  

-  - geopandas=0.14.4=pyhd8ed1ab_0
+  - geopandas=1.0.1=pyhd8ed1ab_1      

-  - hvplot=0.9.2=pyhd8ed1ab_0        
+  - hvplot=0.11.1=pyhd8ed1ab_0    

-  - libnetcdf=4.8.1=nompi_h261ec11_106
+  - libnetcdf=4.9.2=nompi_h135f659_114

-  - numba=0.59.1=py311h96b013e_0     
+  - numba=0.60.0=py311h4bc866e_0        

-  - numpy=1.24.4=py311h64a7726_0 
+  - numpy=1.26.4=py311h64a7726_0  

-  - owslib=0.28.1=pyhd8ed1ab_0      
+  - owslib=0.32.0=pyhd8ed1ab_0       

-  - pandas=2.1.4=py311h320fe9a_0  
+  - pandas=2.2.3=py311h7db5c69_1     

-  - panel=1.4.2=pyhd8ed1ab_0       
+  - panel=1.5.3=pyhd8ed1ab_0         

-  - pydantic=2.7.1=pyhd8ed1ab_0
+      - pydantic==2.7.4                                                       

-  - pyogrio=0.5.1=py311h3f14cef_0                                                    
+  - pyogrio=0.7.2=py311hf8e0aa6_1 

-  - python=3.11.6=hab00c5b_0_cpython               
+  - python=3.11.10=hc5c86c4_3_cpython              

-  - rasterio=1.3.6=py311h567e639_0                 
+  - rasterio=1.3.10=py311h239598e_2                

-  - raven-hydro=0.2.4=py311h64a4d7b_0              
+  - raven-hydro=0.3.2=py311h81cb690_1              

-      - ravenpy==0.13.1     
+  - ravenpy=0.16.0=pyhd8ed1ab_0                                                                         

-  - rioxarray=0.15.5=pyhd8ed1ab_0                  
+  - rioxarray=0.17.0=pyhd8ed1ab_0                         

-  - roocs-utils=0.6.8=pyhd8ed1ab_0                        
+  - roocs-utils=0.6.9=pyhd8ed1ab_0                        

-  - scipy=1.13.0=py311h517d4fd_1                          
+  - scipy=1.14.1=py311he9a78e4_1                          

-  - shapely=2.0.1=py311h0f577a2_0                         
+  - shapely=2.0.4=py311h0bed3d6_1                         

-  - xarray=2023.8.0=pyhd8ed1ab_0                          
+  - xarray=2024.9.0=pyhd8ed1ab_1                          

-  - xclim=0.47.0=py311h38be061_0                          
+  - xclim=0.53.2=pyhd8ed1ab_0                                                 

-  - xesmf=0.8.5=pyhd8ed1ab_0                                                                                          
+  - xesmf=0.8.8=pyhd8ed1ab_0                                                  

-  - xscen=0.8.2=pyhd8ed1ab_0                                                  
+  - xscen=0.10.1=pyhd8ed1ab_0                                                 

```


## Test

- Deployed as "beta" image in production for bokeh visualization
performance regression testing.
- Manual test notebook
https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb
for bokeh visualization performance and it looks fine.
- Jenkins build:
- Default notebooks, all passed:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/f8c1c585a4f1b17c0a3dc67453695deb2c30fb11/docker/saved_buildout/jenkins-buildlogs-default.txt
- Raven notebooks, only known `HydroShare_integration.ipynb`:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/0b3419bba4aa42ee5a1fa9a8de0c8a6b91bf8547/docker/saved_buildout/jenkins-buildlogs-raven.txt


## Related Issue / Discussion

- Matching notebook fixes:
  - Pavics-sdi: Ouranosinc/pavics-sdi#336
  - Finch: PR url
  - PAVICS-landing: Ouranosinc/PAVICS-landing#98
  - RavenPy: CSHS-CWRA/RavenPy#395
  - (...)

- Deployment to PAVICS:
bird-house/birdhouse-deploy#475

- Jenkins-config changes for new notebooks: PR url

- Other issues found while working on this one
  - Issue 1 URL
  - Issue 2 URL
  - (...)

- Previous release:
#121


## Additional Information

Full diff conda env export:

release-py311-240506-update240508...f8c1c58

Full new conda env export:

https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/0b3419bba4aa42ee5a1fa9a8de0c8a6b91bf8547/docker/saved_buildout/conda-env-export.yml

DockerHub build log

https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/0b3419bba4aa42ee5a1fa9a8de0c8a6b91bf8547/docker/saved_buildout/docker-buildlogs.txt
  • Loading branch information
tlvu authored Nov 14, 2024
2 parents 35d6272 + f8c1c58 commit 5d816cf
Show file tree
Hide file tree
Showing 11 changed files with 9,330 additions and 2,648 deletions.
106 changes: 8 additions & 98 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ Please include a summary of the changes and which issues is fixed. Please also i

This PR fixes [issue id](url)

<!-- NOTE: Remember to tag 'release-py###-######' on the commit of this merge. -->


## Changes

Expand All @@ -25,9 +27,9 @@ This PR fixes [issue id](url)

- Deployed as "beta" image in production for bokeh visualization performance regression testing.
- Manual test notebook https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb for bokeh visualization performance and it looks fine.
- Jenkins build, all passed:
- To upload `job-PAVICS-e2e-workflow-tests-new-docker-build-###-consoleText.txt`
- To upload `ravenpy-job-PAVICS-e2e-workflow-tests-new-docker-build-###-consoleText.txt`
- Jenkins build:
- Default notebooks https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/NEW-release-py###-######/docker/saved_buildout/jenkins-buildlogs-default.txt
- Raven notebooks https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/NEW-release-py###-######/docker/saved_buildout/jenkins-buildlogs-raven.txt


## Related Issue / Discussion
Expand All @@ -54,102 +56,10 @@ This PR fixes [issue id](url)
## Additional Information

Full diff conda env export:
<details>
<summary>OLD_TAG-NEW_TAG-conda-env-export.diff.txt</summary>
<!-- note: cannot use inline styles, Github removes them during parsing -->
<pre>
<code>

(...)

71c71
< - ca-certificates=2023.5.7=hbcca054_0
---
> - ca-certificates=2023.7.22=hbcca054_0
81c81
< - certifi=2023.5.7=pyhd8ed1ab_0
---
> - certifi=2023.7.22=pyhd8ed1ab_0
427c427
< - openssl=3.1.1=hd590300_1
---
> - openssl=3.1.4=hd590300_0
547a548
> - salib=1.4.7=pyhd8ed1ab_0

(...)

</code>
</pre>
</details>

https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/compare/PREVIOUS-release-py###-######...NEW-release-py###-######

Full new conda env export:
<details>
<summary>NEW_TAG-conda-env-export.yml.txt</summary>
<!-- note: cannot use inline styles, Github removes them during parsing -->
<pre>
<code>
name: birdy
channels:
- cdat
- conda-forge
- defaults
dependencies:
- (...)
- raven-hydro=0.2.1=py39h8e2dbb5_1
- ravenpy=0.12.1=py39hf3d152e_0
- xclim=0.43.0=py39hf3d152e_1
- (...)
- pip:
- (...)
- xncml==0.2
- (...)
prefix: /opt/conda/envs/birdy
</code>
</pre>
</details>

https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/NEW-release-py###-######/docker/saved_buildout/conda-env-export.yml

DockerHub build log
<details>
<summary>NEW_TAG-buildlogs.txt</summary>
<!-- note: cannot use inline styles, Github removes them during parsing -->
<pre>
<code>
2023-10-25T18:13:11Z Building in Docker Cloud's infrastructure...
2023-10-25T18:13:11Z Cloning into '.'...
2023-10-25T18:13:12Z Warning: Permanently added the RSA host key for IP address '140.82.114.4' to the list of known hosts.
2023-10-25T18:13:13Z Switched to a new branch 'dockerupdate-py39-230601-1-update231025'
2023-10-25T18:13:14Z KernelVersion: 5.4.0-1068-aws
2023-10-25T18:13:14Z Components: [{u'Version': u'20.10.15', u'Name': u'Engine', u'Details': {u'KernelVersion': u'5.4.0-1068-aws', u'Os': u'linux', u'BuildTime': u'2022-05-05T13:17:24.000000000+00:00', u'ApiVersion': u'1.41', u'MinAPIVersion': u'1.12', u'GitCommit': u'4433bf6', u'Arch': u'amd64', u'Experimental': u'false', u'GoVersion': u'go1.17.9'}}, {u'Version': u'1.6.21', u'Name': u'containerd', u'Details': {u'GitCommit': u'3dce8eb055cbb6872793272b4f20ed16117344f8'}}, {u'Version': u'1.1.7', u'Name': u'runc', u'Details': {u'GitCommit': u'v1.1.7-0-g860f061'}}, {u'Version': u'0.19.0', u'Name': u'docker-init', u'Details': {u'GitCommit': u'de40ad0'}}]
2023-10-25T18:13:14Z Arch: amd64
2023-10-25T18:13:14Z BuildTime: 2022-05-05T13:17:24.000000000+00:00
2023-10-25T18:13:14Z ApiVersion: 1.41
2023-10-25T18:13:14Z Platform: {u'Name': u'Docker Engine - Community'}
2023-10-25T18:13:14Z Version: 20.10.15
2023-10-25T18:13:14Z MinAPIVersion: 1.12
2023-10-25T18:13:14Z GitCommit: 4433bf6
2023-10-25T18:13:14Z Os: linux
2023-10-25T18:13:14Z GoVersion: go1.17.9
2023-10-25T18:13:14Z Buildkit: Starting build for index.docker.io/pavics/workflow-tests:py39-230601-1-update231025...

(...)

2023-10-25T18:22:32Z #5 [1/2] FROM docker.io/pavics/workflow-tests:py39-230601-1@sha256:1c00af199dc9248568f36ca03ba7f9ba0fa988733662fbe2d7751f036b508b8c
2023-10-25T18:22:33Z #5 ...
2023-10-25T18:22:33Z
2023-10-25T18:22:33Z #7 exporting to image
2023-10-25T18:22:33Z #7 exporting layers
2023-10-25T18:22:33Z #7 exporting layers 0.3s done
2023-10-25T18:22:33Z #7 writing image sha256:1702b01fb199f56e4cf5b3b3c6ce11f59f1d7d9ab3d0772c29d6797d38340c3a done
2023-10-25T18:22:33Z #7 naming to docker.io/pavics/workflow-tests:py39-230601-1-update231025 done
2023-10-25T18:22:33Z #7 DONE 0.3s
2023-10-25T18:22:33Z
2023-10-25T18:22:33Z #5 [1/2] FROM docker.io/pavics/workflow-tests:py39-230601-1@sha256:1c00af199dc9248568f36ca03ba7f9ba0fa988733662fbe2d7751f036b508b8c
2023-10-25T18:22:33Z Pushing index.docker.io/pavics/workflow-tests:py39-230601-1-update231025...
2023-10-25T18:22:38Z Done!
2023-10-25T18:22:38Z Build finished
</code>
</pre>
</details>
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/NEW-release-py###-######/docker/saved_buildout/docker-buildlogs.txt
2 changes: 1 addition & 1 deletion Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ pipeline {
// https://jenkins.io/doc/book/pipeline/syntax/
agent {
docker {
image "pavics/workflow-tests:py311-240506-update240508"
image "pavics/workflow-tests:py311-241111"
label 'linux && docker'
}
}
Expand Down
2 changes: 1 addition & 1 deletion binder/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM pavics/workflow-tests:py311-240506-update240508
FROM pavics/workflow-tests:py311-241111

USER root

Expand Down
9 changes: 7 additions & 2 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,9 @@ COPY environment.yml /environment.yml
# DEBUG conda.common._logic:_run_sat(607): Invoking SAT with clause count: 2500273
#
RUN umask 0000 && \
mamba create --name birdy --channel conda-forge --channel defaults xclim ravenpy python=3.11 --yes && \
mamba create --name birdy --channel conda-forge --channel defaults xclim ravenpy xscen python=3.11 --yes && \
(conda env export --name birdy 2>&1 | tee /conda-env-export-phase1.yml) && \
(du -sh /opt/conda 2>&1 | tee /conda-envs-size-phase1.txt) && \
mamba env update --name birdy --file /environment.yml && \
mamba clean --all --yes

Expand All @@ -53,7 +55,10 @@ ENV PATH="/opt/conda/envs/birdy/bin:$PATH"

# our notebooks are hardcoded to lookup for kernel named 'birdy'
# this python is from the birdy env above
RUN python -m ipykernel install --name birdy
RUN python -m ipykernel install --name birdy && \
(conda env export --name birdy 2>&1 | tee /conda-env-export-final.yml) && \
(diff --unified /conda-env-export-phase1.yml /conda-env-export-final.yml 2>&1 | tee /conda-env-export.diff) && \
(du -sh /opt/conda 2>&1 | tee /conda-envs-size-final.txt)

# install using same channel preferences as birdy original env to not downgrade
# anything accidentally
Expand Down
13 changes: 6 additions & 7 deletions docker/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,15 @@ dependencies:

# Pin latest xclim and ravenpy to avoid downgrading during the second installation phase.
# Mamba is quicker to solve dependencies than conda, but it is less precise so accidental downgrades can happen.
- xclim >= 0.47.0
- ravenpy >= 0.13.0
- xclim >= 0.53.2
- ravenpy >= 0.16.0

# https://anaconda.org/conda-forge/xscen
# A climate change scenario-building analysis framework, built with xclim/xarray.
# PIN to 0.8.2 for xclim 0.47.0 compat
- xscen == 0.8.2
- xscen >= 0.10.1

#- dask # from xclim and ravenpy
- dask != 2024.11.0 # bug in this specific version broke Raven nb
#- distributed

# flox: fast & furious GroupBy reductions for dask.array
Expand All @@ -43,8 +43,7 @@ dependencies:
- birdy >= 0.8.7
# - owslib>=0.23.0 # from ravenpy
# - netcdf4 # from ravenpy
# TODO: remove libnetcdf PIN because https://github.com/Ouranosinc/PAVICS-landing/issues/66
- libnetcdf <= 4.8.1
- libnetcdf
# https://github.com/ecmwf/cfgrib
# Python interface to map GRIB files to the Unidata's Common Data Model v4
# following the CF Conventions.
Expand Down Expand Up @@ -178,7 +177,7 @@ dependencies:
- jupyterlab-git >= 0.44.0
# Voilà turns Jupyter notebooks into standalone web applications
- voila
# PIN jupyter-archive due to
# TODO: UNPIN jupyter-archive due to
# https://github.com/jupyterlab-contrib/jupyter-archive/issues/132
- jupyter-archive <= 3.3.4
# https://github.com/jtpio/jupyterlab-topbar
Expand Down
Loading

0 comments on commit 5d816cf

Please sign in to comment.