Objective JH-5: Dynamic, shareable environments

[batpad]

Motivation

As hubs get a wider user base, different users need different environments to do their work. The same user may need multiple environments at different times too. Long term, we can not rely on admins to continuously update the small set of images available, and for users to have to keep up with modifying their code to work with newer packages. We need a way for users to be able to specify environments (without having to really understand Docker), and share it with others in a standardized way. This should be done in a way that’s compatible with community standards as well, rather than re-inventing something new. As part of the previous PI, we have deployed jupyterhub-fancy-profiles to VEDA for an improved logged-in environment choosing experience. This builds on top of that, with the goal of making it easier for users to easily & reproducibly create and use whatever environments they want, and share it with others as well.

Proposal

We improve the Dynamic Image Building feature of jupyterhub-fancy-profiles (courtesy a prior grant from GESIS) until it can be put in front of production users. Once deployed, end users can easily create environments in the form of GitHub repositories (or Zenodo DOIs, or many other versioned sources) and share it with each other. These same environments will work on mybinder.org, our existing image building infrastructure and the ephemeral hub set up as part of JH-5 by using the same backend technology to build images (binderhub and repo2docker).

Owners

• @yuvipanda (2i2c)
• @batpad (Development Seed)
• @oliverroick (Development Seed)

Success Criteria

• Upstream refactoring of the BinderHub UI code is completed and merged, so it can be reused
• Dynamic image building features are worked into jupyterhub-fancy-profiles
• Ability to persistently ‘import’ an existing environment is implemented in jupyterhub-fancy-profiles
• Ability to share a URL that points to the environment to use is implemented in jupyterhub-fancy-profiles
• A binderhub-service installation is deployed next to the VEDA hub
• New functionality in jupyterhub-fancy-profiles is deployed to the VEDA hub

[batpad]

I created an issue for URL support for dynamic environments here: yuvipanda/jupyterhub-fancy-profiles#66

For this PI, the plan will be to:

• Enable the Binderhub backend for dynamic image building on the staging hub
• Enable URL handling for dynamic environments in fancy profiles as per Support URL parameters to populate Dynamic Image Building component yuvipanda/jupyterhub-fancy-profiles#66

This will allow us to have dynamic image building working on the VEDA hub and have the ability to share URLs with the environment details encapsulated.

Once we are able to prove this all works and get some feedback from users, what we probably plan for next PI:

• Enable a server-side redirect for "binder-style" URLs like https://staging.hub.openveda.cloud/service/fancy-profile/v2/gh/batpad/foo and come up with a URL scheme that we feel we can commit to supporting long-term
• Have an easy way for user to construct these URLs - either as part of a tool people already use to construct these URLs like https://nbgitpuller.readthedocs.io/en/latest/link.html OR integrated into the fancy-profiles UI (a button to say "Share Environment URL" or so
• Enable URL handling for all aspects of the profile form - machine size config, image selection, etc. - this will likely take a bunch of thinking through to spec, as we want these URLs to be portable across different hubs, so should not be too tied to specifics of profile options on a particular hub.

@yuvipanda let me know if this note and the ticket on fancy-profiles seems to accurately represent what we talked through and please feel free to add more context :-)

cc @oliverroick @freitagb

[batpad]

Done this quarter:

• Binder backend deployed to VEDA staging
• Changes to Fancy Profiles to enable Dynamic Image building - add fields for ref, validation for repository, etc.
• URL handling for URL parameters to populate dynamic image building fields to be able to share environment URLs

We need to deploy all the frontend changes to VEDA staging this week to test and take a call on moving this to production.