GESIS Collaboration - Dynamic image building in a JupyterHub

[choldgraf]

Context

We recently started a collaboration with GESIS with the goal of generalizing, improving, and sustaining the "persistent binderhub" deployment at notebooks.gesis.org.

Here are the major things we'd like to do as part of this collaboration:

• Generalize the "Persistent BinderHub" setup so that it is no longer unique to GESIS infrastructure
• Improve aspects of the design and implementation to be more sustainable, scalable, usable, etc
• Upstream as much as possible to parts of the JupyterHub / Binder ecosystem

Here's a link to the deliverables on the collaboration

Project roles

• Point of contact with @arnim and GESIS: @jmunroe
• Engineering project manager: @damianavila
• Engineering developer: @consideRatio (with @yuvipanda's help as of now Feb 1st 2023).

References

• Yuvi's write-up about unifying BinderHub/JupyterHub under one Helm Chart
• Ryan's idea of "Binder for everything"

We'll use this issue to track progress on the collaboration, and to list issues where we have discussion and iteration on more focused parts of the collaboration. Below is a rough list of things to do.

Updates

Preview Give feedback

1. Announce this collaboration publicly in some way
2. https://github.com/2i2c-org/meta/issues/254
3. Overview of technical work planned for the GESIS collaboration #2120
   +0
   
4. Upstreaming and deploying a persistent binderhub like experience #2119
   +0
   
5. Have a community meeting for feedback and brainstorms about design and end-product goals

Cloud cost reminders

Preview Give feedback

1. https://github.com/2i2c-org/meta/issues/547
2. https://github.com/2i2c-org/meta/issues/546

Dedicated board

Ref: https://github.com/orgs/2i2c-org/projects/33.

[choldgraf]

ping to @bitnik @arnim and @MridulS in case any of them are interested in following along, collaborating, or joining in discussions and meetings.

[arnim]

Thank you @choldgraf ;)

[sgibson91]

Just sharing some slides I presented to a Turing-based project around what I perceive the use cases/target audiences are for JupyterHub and BinderHub and how this dynamic image building in JupyterHub work might affect BinderHub as a separate project. This was presented in order for them to understand how work on internal infrastructure would affect their project and help them make decisions, and is only a reflection of my opinion after having conversations with different folks.

• https://docs.google.com/presentation/d/1ueejB2ZcYaZTve3CqVRE1d4fl0TCSMXF_vuD8hm23fU/edit?usp=sharing

[arnim]

@choldgraf should we do some kick-off?

[choldgraf]

Two quick updates:

Meeting / timing

@arnim yes I believe it's a good idea - we have been working to finish up deploying Pangeo's BinderHub before working on this, and I think we figured out a short-term path forward there last week so hopefully can then shift focus to this project. We will ping here once it's time.

Kernels as a service

In a recent conversation with @jlperla we noted that this work might be related to another feature that many have requested in the Jupyter ecosystem, which is something like "Binder / Jupyter kernels as a service for scalable computation". The idea is that you could define an environment via a Binder-like repository, build it into an image with repo2docker, and then scale computation using that repository's environment via some cloud mechanism. This wouldn't be as rapidly scalable as something like Dask Gateway, but could be a useful way to rapidly / interactively parallelize something in the cloud. Just noting that here in case we find a way to connect it with this work.

[arnim]

The "kernels as a service for scalable computation" idea sounds interesting ;) All the best for the deployment of Pangeo's Hub. A while ago @rabernat raised also an extremely interesting question. Maybe we can indeed move one step closer to a generic backplane for all kinds of scientific infrastructure. Computational replicability is key in economics, geoscience, and the social sciences ;)

[yuvipanda]

Here's a document I wrote almost a year ago now that might help https://hackmd.io/OXhKs4xyQra0KgBglegSBQ

[choldgraf]

@sgibson91 and I discussed this one a little bit today as well, and we agree that it might be helpful to spend some of her time acting as community strategic lead w/ JupyterHub to help steward discussions and feedback amongst stakeholders for this work (with technical vision / ideas / etc coming from others in the JupyterHub ecosystem).

This might be a way to help make progress on this issue and also experiment with ways that the Community Strategic Lead can start improving team processes for discussion and feedback-gathering.

[choldgraf]

Update: Damian serving in a PM role

I spoke with @damianavila today and we agreed that this project will make more rapid progress if we can assign somebody to serve in a project manager style role to help us keep track of plans, conversations, timelines, and deliverables.

@damianavila said he would be willing to serve in this role and he will reduce his time on the PyData theme in order to grow the capacity to work here.

So our next steps are to:

• Assign @damianavila in a project manager role on this issue
• Create a place to track and plan this effort (maybe this issue + a project board for deliverables)
• Plan initial meetings to align / plan
• Plan our strategy around Engineering capacity (a mix of growing capacity in the 2i2c team, or bringing in capacity via sub-contracts with others

[arnim]

This is great news. Thank you @damianavila for helping with this 🎉

[damianavila]

@arnim, how does your calendar looks-like for a meeting next week? I can send a when2meet link for next week if you are available. 2i2c eng resources should be available to meet next week as well.

[arnim]

how does your calendar looks-like for a meeting next week?

Do you have a preferred day? Usually around this time would work for me well :)

[arnim]

@damianavila Typically somewhere between 12:00 UTC and 23:59 UTC should work

[damianavila]

OK, I have create a when2meet event so we can find a slot that works for everyone involved in this conversation: https://www.when2meet.com/?16135851-mYuiB

@consideRatio @yuvipanda @sgibson91, I would appreciate it if you can join this meeting. Please check the link and drop your availability. Thanks!

[arnim]

@damianavila Should we say Thursday, 21th July, 2022 at 17:15 UTC (or your timezone)?

Video - Jitsi: https://meet.jit.si/DynamicImageBuilding

[damianavila]

@armin, I was finalizing the survey of availability and I was going to propose Wed 20th at 17 UTC, instead.
Can you make it?
Btw, I can offer a 2i2c zoom room in the invite I am going to send if Zoom is OK for you.

[arnim]

That's fine as well :)

[damianavila]

Can you tell me the email to use in the invite? I will post the details here as well in case you do not want to share your email here.

[arnim]

arnim dot bleier at gmail ...

[damianavila]

Thank you (some people use dedicated calendars for meeting invites, this is why I explicitly asked).
Invitation sent! cc @2i2c-org/tech-team

[damianavila]

Details.
When: Wed 20th 17 UTC
Where: 2i2c zoom room
Agenda (dropping some points that we can change if we want/need to):

1. Quick presentation
2. State of the problem
3. Exploration/design phase
4. Implementation
5. Provision
6. Status updates

[arnim]

Can we send @MridulS an invite in case he wants to join?

[damianavila]

@arnim, I sent you an invite to our Slack space.

[arnim]

THX - just sent my hello message :)

[damianavila]

Some updates:

• Created a dedicated team (@consideRatio, @yuvipanda, and @sgibson91) who is assigned to push forward this project.
• Created a dedicated Slack channel to have focused discussions about the project.
• Created a dedicated GH project board to collect issues related to this project.
• Coordinated and discuss next steps with the team.

We are starting with an exploratory phase consisting of:

1. Series of technical meetings
2. MVP to validate the ideas coming from those meetings
3. MVP presentation in front of stakeholders (feedback loop)
4. Iteration

Next step:

• A series of 3 technical meetings (one per week) to discuss specific technical items involving @yuvipanda's proposal, UX/UI needs, underlying technology, pre-existing experiences, etc.
  • This series will be tracked here: A series of technical-focused meetings to discuss about the BinderHub-JupyterHub project #1577

[damianavila]

After the first meeting, we have some next steps briefly described here: #1577 (comment)

[arnim]

Hi @damianavila, I thought that you would ping me first when you have some time to chat. I'm not sure we agreed to build on tljh in our first meeting.

[sgibson91]

We are only deploying tljh-repo2docker as a means to perform user experience research to inform what we should build (but won't be the sole source of guidance) for, most likely, the z2jh helm chart.

[arnim]

I'm also certainly interested in trying tljh-r2d and have to confess to my shame that I haven't done so yet :) Already subscribed to /issues/1596

[damianavila]

Hi @damianavila, I thought that you would ping me first when you have some time to chat. I'm not sure we agreed to build on tljh in our first meeting.

For future readers, there was a conversation with @arnim in the specific binderhub-jupyterhub 2i2c Slack channel that further clarifies the context and the explanation @sgibson91 shared above: https://2i2c.slack.com/archives/C03RLNFM43F/p1659687870338259.

[arnim]

Hi 👋 I'm back to the office this month. Should we have a small meeting, maybe next week? Let me know if there is anything I can do.

[arnim]

We should also consider if the proposed new architecture allows for things to be used such as dynamic repository credentials (jupyterhub/binderhub/pull/1169) should they land in BHub or would this then better be integrated on the JHub side.

[damianavila]

I think https://infrastructure.2i2c.org/en/latest/howto/features/github.html might help with this ⬆️.
There is a blog post from @yuvipanda at https://blog.jupyter.org/securely-pushing-to-github-from-a-jupyterhub-3ee42dfdc54f.

[arnim]

Added a short link to the Jupyter community forum post on the Persistent BinderHub. The link includes a brief description of the deliverables for the collaboration, so that readers can form an idea of what to expect.

[choldgraf]

Hey all - I've added a "project roles" section to the top comment here and tagged folks that I think are dedicating their time to the project. Please edit if I got any of that wrong!

[damianavila]

The last available update of the plan lives at 2i2c-org/binderhub-service#27.