Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using unified conda-lock file in postBuild for reproducible conda environment #14

Draft
wants to merge 31 commits into
base: main
Choose a base branch
from
Draft

Using unified conda-lock file in postBuild for reproducible conda environment #14

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
5857db0
Rename environment.yml file to environment-cryocloud.yml
weiji14 Nov 29, 2022
90fe9f6
Bump to cartopy 0.21.0, rasterio 1.3.3, hdf5 1.12.2, qgis 3.22.11
weiji14 Nov 29, 2022
1d23d26
Replace explicit conda-linux-64.lock file with unified conda-lock.yml
weiji14 Nov 29, 2022
95d8ec8
Temporarily rename environment-cryocloud.yml back to environment.yml
weiji14 Nov 29, 2022
09ecd55
Revert "Temporarily rename environment-cryocloud.yml back to environm…
weiji14 Nov 29, 2022
00cc297
Add postBuild script to install dependencies from unified conda-lock.yml
weiji14 Nov 29, 2022
d07eafb
Minimal conda-lock.yml with only Python 3.10
weiji14 Nov 29, 2022
ef9cc29
Fix conda-lock install command in postBuild
weiji14 Nov 29, 2022
e609d9c
Pin to conda-lock 1.0.5 to keep relative paths
weiji14 Nov 29, 2022
4ea512b
Document how to update conda-lock.yml file in the main README.md
weiji14 Nov 29, 2022
abb8710
Install conda-lock with pip_support extras
weiji14 Nov 29, 2022
33436bc
Run docker build and test workflow when conda-linux.yml is modified
weiji14 Nov 29, 2022
3e08468
Pin minimum jupyterlab to 3.5
weiji14 Nov 29, 2022
85b01cb
[condalock-command] autogenerated conda-lock files
actions-bot Nov 29, 2022
f82355e
Set notebook as conda environment name in docker image
weiji14 Nov 29, 2022
f0b86fe
Bump conda-lock from 1.0.5 to 1.2.1
weiji14 Nov 29, 2022
20ec34b
Update conda-lock's dependency file source
weiji14 Nov 29, 2022
3c6a826
[condalock-command] autogenerated conda-lock files
actions-bot Nov 29, 2022
70fb920
Improvements to postBuild script
weiji14 Nov 29, 2022
57e22d2
Merge branch 'main' into unified-lockfile
weiji14 Nov 30, 2022
9f36ec5
Merge branch 'main' into unified-lockfile
weiji14 Dec 4, 2022
7f9db0c
Fix incorrect merge conflict omission
weiji14 Dec 4, 2022
27f3a8b
Delete conda-linux-64.lock
weiji14 Dec 4, 2022
5486608
Merge branch 'main' into unified-lockfile
weiji14 May 28, 2023
18b1b25
Bump conda-lock to 2.0.0
weiji14 May 28, 2023
8353e4d
Use conda-lock 2.0.0 in postBuild
weiji14 May 28, 2023
4a21b81
Update conda-lock.yml to v1 schema
weiji14 May 28, 2023
71a2f4e
Fix typo conda-linux.yml to conda-lock.yml
weiji14 Jun 10, 2023
4cff62b
Merge branch 'main' into unified-lockfile
weiji14 Jun 10, 2023
bccefa2
Merge branch 'main' into unified-lockfile
weiji14 Jun 17, 2023
9c8e894
Bump conda-lock from 2.0.0 to 2.1.0
weiji14 Jun 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@ on:
- main
paths:
- 'apt.txt'
- 'conda-linux-64.lock'
- 'environment.yml'
- 'conda-linux.yml'
weiji14 marked this conversation as resolved.
Show resolved Hide resolved
- 'environment-cryocloud.yml'
- 'postBuild'
- 'start'

jobs:
build-and-push:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/conda-lock-command.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,12 @@ jobs:

# Install conda-lock library
- name: Install conda-lock
run: pip install conda-lock
run: pip install conda-lock[pip_support]==1.0.5
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using conda-lock 1.0.5 instead of the latest 1.2.1 to keep the relative file paths intact. See #15.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, might need to upgrade to conda-lock 1.2.1 actually, because it contains important bugfixes related to pip dependency hashes. E.g. see https://github.com/CryoInTheCloud/hub-image/actions/runs/3578432352/jobs/6018581680#step:5:4747

  ERROR:root:ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
  ERROR:root:    fastjsonschema@ https://files.pythonhosted.org/packages/e4/be/cf1b876348070a23cb0c3ebfee7a452ad3a91b07b456dade3bd514656009/fastjsonschema-2.16.2-py3-none-any.whl#md5=None from https://files.pythonhosted.org/packages/e4/be/cf1b876348070a23cb0c3ebfee7a452ad3a91b07b456dade3bd514656009/fastjsonschema-2.16.2-py3-none-any.whl#md5=None (from -r /tmp/tmpn_q_dp50 (line 1)):
  ERROR:root:        Expected md5 None
  ERROR:root:             Got        fff77553c8612b49c8782a5233541199
  ERROR:root:
  ERROR:root:
  ERROR:root:ERROR conda.cli.main_run:execute(49): `conda run pip install --no-deps -r /tmp/tmpn_q_dp50` failed. (See above for error)
  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/bin/conda-lock", line 8, in <module>
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1055, in main
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/conda_lock/conda_lock.py", line 1268, in install
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/conda_lock/conda_lock.py", line 219, in do_conda_install
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/conda_lock/invoke_conda.py", line 131, in _invoke_conda
  poetry.utils._compat.CalledProcessError: Command '['/srv/conda/bin/mamba', 'run', '--name', 'notebook', 'pip', 'install', '--no-deps', '-r', '/tmp/tmpn_q_dp50']' returned non-zero exit status 1.
  Removing intermediate container 4d439bce016f
  The command '/bin/sh -c ./postBuild' returned a non-zero code: 1


# Run "conda-lock" for linux-64 only
- name: Run conda-lock
run: |
conda-lock lock --mamba --kind explicit --file environment.yml --platform linux-64
conda-lock lock --mamba --file environment-cryocloud.yml --platform linux-64

# Commit the change to the PR branch if any changes
- name: Commit condalock files to PR
Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,10 @@ on:
pull_request:
paths:
- 'apt.txt'
- 'conda-linux-64.lock'
- 'environment.yml'
- 'conda-linux.yml'
weiji14 marked this conversation as resolved.
Show resolved Hide resolved
- 'environment-cryocloud.yml'
- 'postBuild'
- 'start'

jobs:
test-build:
Expand Down
26 changes: 17 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,22 @@
The JupyterHub docker image used for [hub.cryointhecloud.com](https://cryointhecloud.com),
hosted on https://quay.io/repository/cryointhecloud/cryo-hub-image

## Adding packages to this repository
## Updating packages in this repository

You can easily add more packages to the cryointhecloud hub by making pull requests to this
repository. Edit either the `environment.yml` file (for most packages) or `apt.txt` file
(for packages that need to be used in the Linux Desktop environment in the cloud), and make
a pull request - you can even do that just from the GitHub UI! A bot will comment with a link to
[mybinder.org](https://mybinder.org) where you can test your pull request to make sure it works
as you would expect.
You can add or update packages on the cryointhecloud hub by making pull requests to this
repository. Follow these steps:

Once your PR gets merged, you can follow the [2i2c docs](https://docs.2i2c.org/en/latest/admin/howto/configurator.html)
to have the hub use them!
1. Edit either the `environment-cryocloud.yml` file (for most packages) or `apt.txt` file
(for packages that need to be used in the Linux Desktop environment in the cloud)
2. Start a [pull request](https://github.com/CryoInTheCloud/hub-image/pulls) -
you can even do that just from the GitHub UI! A bot will comment with a link to
[mybinder.org](https://mybinder.org) where you can test your pull request to make sure it works
as you would expect.
3. In the Pull Request, write a comment with the slash command `/condalock`.
This will refresh the [`conda-lock.yml`](https://conda-incubator.github.io/conda-lock/output/#unified-lockfile)
file that contains a snapshot of the exact library versions that will be
used in the conda environment.

Once your Pull Request gets reviewed and merged, you can follow the
[2i2c docs](https://docs.2i2c.org/en/latest/admin/howto/configurator.html)
to have the JupyterHub use them!
518 changes: 0 additions & 518 deletions conda-linux-64.lock
View file
Open in desktop

This file was deleted.

Loading