Repo2docker image for Cryo cloud community

[weiji14]

Create an initial conda virtual environment specification for the NASA Cryo community! This will be used in the default docker image for the 2i2c Cryo deployment. From 2i2c-org/infrastructure#1702 (comment), the idea appears to be to combine the dependencies listed in:

• 🛰️ ICESat-2 2022 Hackweek - https://github.com/ICESAT-2HackWeek/website2022/tree/2022.04.14/conda
• ❄️ SnowEx 2022 Hackweek - https://github.com/snowex-hackweek/website2022/tree/main/conda
• 🌎 Jupyter Meets the Earth (JMTE) - https://github.com/pangeo-data/jupyter-earth/blob/master/hub.jupytearth.org-image/Dockerfile

Note that 2/3 of these are derived from https://github.com/uwhackweek/jupyterbook-template/tree/main/conda

List of example packages to include:

• icepyx
• xarray
• ➕ others, see e.g. https://github.com/awesome-cryosphere/cryosphere-links#isg-software

Template repository and instructions from 2i2c:

• https://docs.2i2c.org/en/latest/admin/howto/environment/index.html
• https://github.com/2i2c-org/hub-user-image-template

Question: Where to put the conda virtual environment specification?

1. here in github.com/CryoInTheCloud/CryoCloudWebsite ✅
2. a dedicated 'image' repo, e.g. github.com/CryoInTheCloud/cryo-image (or some other name)
3. under the 2i2c org, e.g. github.com/2i2c-org/cryo-image following https://github.com/2i2c-org?q=&type=all&language=dockerfile&sort= (not needed, see Repo2docker image for Cryo cloud community #1 (comment))

[sgibson91]

Heya, just to say there is no requirement from 2i2c that this image repo be under our org - although we can put it there if other locations are not appropriate. The only thing we need to know is the image name and tag so we can pull it from a container registry :)

[tsnow03]

Thanks @weiji14! It will be best to have it in the CryoCloud github for sure. I'll answer definitively on the rest tomorrow. Would you like to meet to discuss?

[weiji14]

Thanks @weiji14! It will be best to have it in the CryoCloud github for sure. I'll answer definitively on the rest tomorrow. Would you like to meet to discuss?

Sure, I should be free to chat after 12 noon (MDT) tomorrow. Would be good to discuss about things like names for the docker image, what are the core Python packages needed, etc.

@sgibson91 It would be great to use this template. I was mimicking the Hackweek GitHub because we want to build a JupyterBook for the group but I have no attachments to anything. I am a nube to this part of the work. We do want all of this in our github so it is all together in one place. I will look through the documents to figure this out.

Originally posted by @tsnow03 in 2i2c-org/infrastructure#1702 (comment)

After looking at the 2i2c template and the Hackweek GItHub template, the key difference seems to be in terms of dependency lockfiles (Hackweek uses conda-lock but 2i2c template doesn't). I have a slight preference for having lockfiles because it will be more reproducible long-term, though it might require a bit more work/expertise. No reason we can't combine the 2i2c and Hackweek styles though! Let's discuss this tomorrow perhaps.

[sgibson91]

If you're going to take anything from the 2i2c template, take the GitHub Actions workflows that automate the building and pushing of the image using repo2docker. Once you have that in place, everything else is just tweaking environment files.

[weiji14]

If you're going to take anything from the 2i2c template, take the GitHub Actions workflows that automate the building and pushing of the image using repo2docker. Once you have that in place, everything else is just tweaking environment files.

Yes, those GitHub Actions CI workflows will definitely be needed, just pushed a commit at c783191! The Hackweek repo does have a similar workflow with some extra customizations at https://github.com/uwhackweek/jupyterbook-template/blob/0d7ca36851ae399438433a8f3b48a8af90f40e30/.github/workflows/repo2docker.yaml, e.g. setting a CalVer tag for the built docker image. We might look at bringing in some of those extra customizations to align with the Cryo community's standards.

[tsnow03]

I have added the test and other workflows so you should have what you need now for actions.

[weiji14]

Thanks @tsnow03 for setting up the accounts on https://quay.io/user/cryointhecloud and https://hub.docker.com/u/cryointhecloud (and configuring the related QUAY_USERNAME, QUAY_PASSWORD, DOCKER_USERNAME and DOCKER_PASSWORD secrets)! The repo2docker action works now (see latest push at https://github.com/CryoInTheCloud/CryoCloudWebsite/actions/runs/3231362352, after #8), and there are now docker images published on these Docker container registries 🎉

• https://quay.io/repository/cryointhecloud/cryocloudwebsite?tab=tags
• https://hub.docker.com/r/cryointhecloud/cryocloudwebsite/tags

I haven't tested those docker images (tagged 2022.10.12) extensively though, will leave this issue open for a little longer in case some extra things are needed.

[weiji14]

Tracking some extra packages requested by @dfelikson and @tsnow03:

• vi/vim
• aws cli
• s3fs Add s3fs #34
• matplotlib widgets magic
• ipyleaflet
• rasterio and rioxarray
• xarray-datatree
• boto3
• pystac_client
• earthdata (will become earthaccess soon)
• geojson visualization (??)
• VNC/linux desktop
• SYNCthing

Likely upgrades on the horizon:

• Matlab kernel-license provided by user

[jmunroe]

While not essentially, did you want to consider splitting off your image into a separate repo from this website repo? I can see the possibility in the future that you will have GitHub actions set up that will need to respond only to changes in the image but you don't want/need the website to also be redeployed. Again, not a firm requirement if using a mono-repo works better but it is just a suggestion.

Relatedly, it looks are primarily thinking about building your image using repo2docker and having to define everything in a conda environment file. While this is fine for many groups, you can also manage your own Dockerfile to build your image. In particular, if you want to support a VNC/Linux desktop I think this is beyond what is possble with only a conda environment file. The jupyter meets the earth project has it's image repo at https://github.com/pangeo-data/jupyter-earth which may be another resource as you are customizing the image you want to use for your community.

[weiji14]

While not essentially, did you want to consider splitting off your image into a separate repo from this website repo? I can see the possibility in the future that you will have GitHub actions set up that will need to respond only to changes in the image but you don't want/need the website to also be redeployed. Again, not a firm requirement if using a mono-repo works better but it is just a suggestion.

Yes, we did consider a dedicated docker image repo (see #2 (comment)). The current mono-repo approach was meant to keep things simple since there's only a few maintainers, but if things get more complicated, then splitting the Jupyter Book content from the conda enviroment specification seems wise.

Relatedly, it looks are primarily thinking about building your image using repo2docker and having to define everything in a conda environment file. While this is fine for many groups, you can also manage your own Dockerfile to build your image. In particular, if you want to support a VNC/Linux desktop I think this is beyond what is possble with only a conda environment file. The jupyter meets the earth project has it's image repo at https://github.com/pangeo-data/jupyter-earth which may be another resource as you are customizing the image you want to use for your community.

True, using a Dockerfile would be necessary if VNC/Linux is needed. Maybe @tsnow03 can chime in on what the use case for VNC/Linux would be, since this would require some non-trivial changes to the current repo2docker setup using conda enviroment.yml files.

[tsnow03]

I had a meeting with @jmunroe about the CryoCloud. Between @jmunroe, @fperez, and Fernando, and others, it sounds like 2i2c will be helping to do two things:

• Add new packages and tools in relation to this comment on our github.
• Streamlining the workflow for deploying the CryoCloud docker image to follow the practices built in the Berkeley JMTE hub.

Both of these items are targeted to be complete before the first onboarding training session for CryoCloud, which has been pushed back to Dec 2 @ 9am PT.

[yuvipanda]

@tsnow03 I created a new repo based on a message from @fperez at https://github.com/2i2c-org/nasa-cryocloud-image. I'd love to move this repo into this github org :) Can you give me org-level permissions so I can do that?

Would also need to update the QUAY credentials on GitHub actions for it to work

[yuvipanda]

I've added everything in #1 (comment), except the linux desktop - which I'll add later today.

[yuvipanda]

Update: I didn't find time to add the desktop environment today. Let's do the following, and I'll add the desktop env?

• Move the repo into this org
• Setup quay.io pushing

[fperez]

Thanks @yuvipanda! I sent you an invite with access to both teams but member role - do you need owner role (I can change it if needed, of course)?

[fperez]

The quay credentials update will need @tsnow03's input, I don't have those.

[yuvipanda]

@fperez looks like I need owner perms.

[fperez]

Done @yuvipanda!

[yuvipanda]

@fperez great, I've it here now: https://github.com/CryoInTheCloud/hub-image.

Will need the quay.io creds next!

[fperez]

Awesome, thx! Should I add an apt.txt like the stat159 one??

And I'd want to pull in also all the packages we keep around in the stat159 and JMTE envs to form a "happy home in the clouds", though we'd probably want to do a bit of weed trimming for anything there that's gone stale, and run your version update script over those files.

[fperez]

Also, do we have a way to set up env variables without touching the dockerfile? There's a few useful ones in the JMTE setup but I think most of that shouldn't require touching the Dockerfile itself.

[yuvipanda]

@fperez I've added all the packages requested in #1 (comment), as well as all the apt packages from stat159 into the image. I've also combed through the stat159 image and added what I think of as baseline packages to the image.

However, at this point, I would strongly recommend against a wholesale copying of conda packages from stat159 to this image - additional packages always cause additional work, now or in the future :) I'd say add them as the need arises. If you look at the packages in https://github.com/CryoInTheCloud/hub-image/blob/main/environment.yml#L4, you'll see that I've added a comment to each on on why it is there. In our experience managing large images for berkeley, this is absolutely important as without that it becomes impossible to know why something is there, and what might break if it is bumped or removed. So my suggestion is to try run through your demo, and add packages as needed if it fails. If there's something that you think should be counted as 'baseline' that I didn't pull in, I think adding that with a comment on why it's important would be helpful.

Once the quay.io setup is done, it doesn't block on us either - @tsnow03 can add them as necessary! I've amended the README with information on how you can do this. This is actually much better than the setup you are used to with stat159 @fperez as you can test package additions here purely via the GitHub web UI + mybinder.

I think TODO still is:

• Setup quay.io (I'll do so once I get access)
• Document how to setup environment varialbes in the image (should be doable with a start file).

I'd also personally heavily recommend against PIP_USER, as it means everyone's local hub environment now will be different in subtle ways that make collaboration difficult. I understand having to reinstall pip packages each time your server restarts might be bothersome, but I would heavily recommend against setting PIP_USER since I guarantee it will bite you when you least expect it.

[yuvipanda]

I also updated the README in https://github.com/CryoInTheCloud/hub-image. The Linux desktop still seems broken, I'll look at that probably after thanksgiving.

[fperez]

Thx @yuvipanda! Done here, we can probably now continue working directly on that repo via PRs and issues.

[fperez]

Oh, and agreed on PIP_USER, actually. I keep forgetting that the reason it works for me is b/c I personally combine it with manually-configured PYTHONUSERBASE that lives inside the conda env in a subdir.

But that pattern is a bit unusual and not something I'm keen to explain to new users for now, so let's keep it simple :)

Still, setting env vars without touching the dockerfile could be useful (aside from PIP_USER), don't you think?

[weiji14]

Thanks Yuvi and Fernando for starting the new docker image repo!

Just wanted to throw out a question on reproducibility. Will there be lockfiles generated from the environment.yml file so that people can reliably re-create the conda environment in the future with the exact library versions? This is something we've done in the monorepo right from the start at #4 (with the conda-lock.yml file), but it doesn't look to be replicated yet on https://github.com/CryoInTheCloud/hub-image.

I realize that the packages in apt.txt will be a bit trickier to generate a manifest file for, and that this is a big ask in terms of maintainability from 2i2c. I'm more than happy to contribute to set up the infrastructure around the lockfile stuff, but wanted to know if there's an appetite for ensuring long-term reproducibility.

[fperez]

@weiji14 - absolutely!! My take on this issue is that we should take a "two-level" approach:

• Once we settle on a complete list of packages for the base environment.yml file, then we run a script @yuvipanda wrote that will write in the current conda-forge package versions in it. But that's not a conda lock file, b/c it remains platform independent as it won't have build hashes in it. This makes it easy for users to replicate the environment locally across OSes, which I think is a very important feature (not everything can be done locally, but it's very valuable for users to at least have a reproducible offline workflow that matches the cloud service as much as possible).
• Then, we generate a conda-lock.yml file to pin the docker images with exact builds. We tag the image with a version number at that point.

Regarding versioning, my suggestion would be to use an 2022.X.Y scheme, where the "major" number X changes when new packages are added, whereas the "minor" Y changes when any package version or even build is updated. This will let people say "This project needs the cryohub environment 2022.3.4" with that being a pretty unambiguous reference.

As for the apt packages, I think that's a bit less of a concern, since they change more slowly and we can ensure that a given YYYY series never changes the underlying Ubuntu base image, so only minor version fixes will go in. I don't know of the equivalent of a conda-lock for apt packages, but that's probably just my ignorance.

Speaking of apt - are there any desktop packages you'd really want to have there from day 1? Only the most basic stuff has been installed, mostly CLI tools. But I was thinking of adding at least qgis and grass that I imagine many people might want to use (I know Whyjay in our group uses qgis, and I'm a light user of it too).

Very happy to see this taking shape!

[fperez]

ps - @weiji14, I don't know if qgis and/or grass are in the default ubuntu repos, so I won't add them to the apt file unless you confirm they'd be useful, as they may require adding extra sources lists.

I won't have time to dive into using it during the Dec 2 demo, so unless you need it right away, we might just make an issue for other desktop tools needed by the users and sort that out after some users have a chance to kick the tires a bit.

[yuvipanda]

I hadn't planned on setting up lockfiles in the new image, mostly due to lack of capacity. The primary goal was to remove the requirement for folks needing a local docker setup or similar to be able to make changes to the repo. The pangeo-docker-stacks project does that with this action (https://github.com/pangeo-data/pangeo-docker-images/blob/master/.github/workflows/CondaLock.yml) that @scottyhq wrote I think. If we can generalize that, or even make a copy of that here to autogenerate lock files, I'd be happy to review! Unfortunately I won't have time to work on it myself this year.

I would also say it's probably ok to just get desktop apps (outsdie of the base env, like XFCE and X) from conda, as the apt versions can be quite outdated. qgis is definitely much newer in conda-forge than in apt, for example.

So @weiji14, if you wanna take a shot at the conda-lock automation, i'd love to see that :) I completely agree it is super important for long-term (or even medium term reproducibility), and could be generally useful in a lot of places.

[fperez]

np @yuvipanda - if @weiji14 has bandwidth to set up the conda-lock automation that would be awesome, else we can put it on the backlog for later.

And in a fun turn that shows my brain is mush, I just realized I ended up rewriting your script and had totally forgotten!

Oh, fun...

I've named the env and will try to build it locally, then will run my script over it and will thus finalize this PR with full versions of the packages.

Not quite the conda-lock that @weiji14 is asking for, but it will go a ways in that direction.

[fperez]

Ok, using my little script I added version numbers so at least now I think this part is in reasonable shape (and we can improve with conda-locks later, obviously).

[tsnow03]

It will be nice to add the conda-locks later. I'm happy to help with that as well.

I think the PIP-USER discussion is an important one. We have had several researchers with big complaints about needing to install packages on every re-opening of their servers in other hubs. If people are installing big packages, I can see how it could be a pain point. I can also see how it impacts reproducibility to not keep track of pip installed packages, but I wonder if there is an extra txt file people can create that either automatically pip installs those packages on startup or at least just keeps track of the pip installs occurring in a user's server so that the next person knows what they will need to (nearly) exactly reproduce. We will want to train on this in our onboarding either way.

[weiji14]

The pangeo-docker-stacks project does that with this action (https://github.com/pangeo-data/pangeo-docker-images/blob/master/.github/workflows/CondaLock.yml) that @scottyhq wrote I think. If we can generalize that, or even make a copy of that here to autogenerate lock files, I'd be happy to review!

Cool, that CondaLock.yml action is pretty much the automated GitHub Actions version of what we've been using at https://github.com/CryoInTheCloud/CryoCloudWebsite/blob/main/conda/lock-environment.sh. Let me see if I can port that over to the hub-image.

I would also say it's probably ok to just get desktop apps (outsdie of the base env, like XFCE and X) from conda, as the apt versions can be quite outdated. qgis is definitely much newer in conda-forge than in apt, for example.

A quick scan through the packages in apt.txt does show quite a few packages on conda-forge already. I'd definitely recommend QGIS from conda-forge, though that can wait until after Dec 2.

I wonder if there is an extra txt file people can create that either automatically pip installs those packages on startup or at least just keeps track of the pip installs occurring in a user's server so that the next person knows what they will need to (nearly) exactly reproduce.

This is steering towards the 'Binder for Everything' vision mentioned at https://discourse.pangeo.io/t/future-of-pangeo-cloud-i-binder-for-everything/1574 😄. I know 2i2c is looking at operating a BinderHub on AWS us-west2 for the general Pangeo community, but I'm not sure if we want something similar for CryoCloud. There's typically a tradeoff between having having too many packages in the base image (which leads to slow server startup times) and too few packages (users then need to install it themselves). But anyways, this can be discussed later.

[yuvipanda]

This is deployed in the hub now, and I've fixed the linux desktop environment :) Please test it out and let me know!

[weiji14]

This is deployed in the hub now, and I've fixed the linux desktop environment :) Please test it out and let me know!

Startup time seems a bit slower, but it works after a bit of light testing on JupyterLab! Not quite sure how to access the desktop apps (e.g. QGIS) though, but I'll find out on Dec 2 😄

Oh, and I think I've got the conda-lock automation working, see CryoInTheCloud/hub-image#5 (comment). So people will need to write /condalock in a Pull Request comment when they update any dependencies in the environment.yml file, and a bot will refresh the conda-lock-linux64.lock lockfile automatically. Will probably need to document this in a Pull Request template or something.

[tsnow03]

The hub is looking great so far! The desktop worked great for me and QGIS came up perfectly when I ran it from the command line. From my current pull request, here are the few errors I've found so far with the environment. I'll start working on those things once the open hub-image pull requests are settled.

[fperez]

@weiji14 is there anything else on this issue we should do, or can we close it now, and start opening new more targeted ones as new ideas/issues come up?

I just updated the image in this PR, and though I get an icepyx error in the tutorial, I think most packages are now in good shape and all the tutorial imports work.

[weiji14]

Yes, let's close it since we should have most if not all of the packages added. New discussions can continue over at https://github.com/CryoInTheCloud/hub-image/issues.