[New Hub] Allen Institute Summer Workshop on the Dynamic Brain

[colliand]

Hub Description

The Allen Institute requests a hub for their upcoming Summer Workshop on the Dynamic Brain

Community Representative(s)

@mabuice
@fcollman

Important dates

Notes: target and required dates are outdated, we need to update them accordingly to new information and prioritization.

• Target start date: 2022-07-31
• Required start date: 2022-08-05
• Any important dates for usage: The event runs 2022-08-20 through 2022-09-04.

Hub Authentication Type

GitHub Authentication (e.g., @MyGitHubHandle)

Hub logo information

• URL to Hub Image: {{ URL HERE }}
• URL for Image Link: https://github.com/AllenInstitute/swdb_2022/blob/main/resources/cropped-SummerWorkshop_Header.png

Hub user image

• Repository for user image: https://github.com/AllenInstitute/swdb_2022
• User image registry: { REGISTRY IF ONE ALREADY EXISTS }
• User image tag and name: { NAME AND TAG IF IT EXISTS }

Extra features you'd like to enable

• Specific cloud provider or datacenter: AWS
• Dedicated Kubernetes cluster
• Scalable Dask Cluster

Other relevant information

No response

Hub URL

allen-swdb.2i2c.cloud

Hub Type

daskhub

Tasks to deploy the hub

• Engineer who will deploy the hub is assigned
• Deploy information filled in above
• Initial Hub deployment PR: Add allen-swdb hub #1585
• Administrators able to log on
• Community Representative satisfied with hub environment
• Hub now in steady-state

[damianavila]

@colliand request ack. I have added the issue to our backlog board and we will prioritize the hub deployment accordingly to available eng resources so we can deploy the hub on a timely manner.

@mabuice, @fcollman, we will ping you soon with some questions about the specific of the hub deployment.

[fcollman]

Where can i read instructions about the requirements for the course docker image? (i.e. what should be running on what ports/etc?)

Will plan on putting materials and Dockerfile in this repo (just a stub for now)
https://github.com/AllenInstitute/swdb_2022

[colliand]

Hi Forrest! This and related pages in 2i2c's docs may help. Our engineers will likely have better pointers soon. https://docs.2i2c.org/en/latest/admin/howto/environment/index.html

[sgibson91]

@fcollman The best thing to do is to fork this repo: https://github.com/2i2c-org/hub-user-image-template and follow the instructions Jim linked to setup your environment. There are docs on using nbgitpuller for materials https://docs.2i2c.org/en/latest/admin/howto/content.html

[colliand]

Following a suggestion from @mabuice, I am adding Saskia de Vries @saskiad to this issue thread and related email chains. Welcome Saskia!

[mabuice]

Adding @aamster and @morriscb, representatives from the Allen Institute Technology team to this thread.

[mabuice]

Hub repo is here: https://github.com/AllenInstitute/swdb_2022_hub_image

(I have not yet gone through the steps in the instructions other than creating this repo.)

Content repo is here: https://github.com/alleninstitute/swdb_2022
The content repo contains the necessary splash logo:
https://github.com/AllenInstitute/swdb_2022/blob/main/resources/cropped-SummerWorkshop_Header.png

If I understand correctly, the above hub_image is all we need to get something off the ground, and we can continue to configure and modify it as we go forward.

We have an AWS account with an associated domain name that we'll be using for this. We will serve data to students via an S3 bucket through this account, so whatever is necessary on the configuration end to facilitate this will need to be done (hence connecting @aamster and @morriscb, who will be preparing the data on our side).

From reading through the docs I understand (more or less) how to configure the hub image and how to prepare content (although we'll likely have questions as we go through this process). I'm unclear on how persistent storage for the students works and how we configure that (or if that's just part of the magic that happens in the background).

[mabuice]

Alright, I've followed the instructions on this page: https://docs.2i2c.org/en/latest/admin/howto/environment/hub-user-image-template-guide.html#hub-user-image-template-guide-how-to up to step 6, which unless I've misunderstood something is as far as I can go.

[mabuice]

@damianavila Are there any further steps you are waiting on from us? When is a reasonable timeline to expect an initial hub to be up and running?

[damianavila]

@mabuice, this new hub request is currently in the pipeline of new hubs to be deployed.
I am trying to secure resources to make this happen as soon as possible.

Accordingly to the initial request, the event is happening by the end of August, can you confirm?
Can you also clarify how much time you might need to interact with the hub and get ready? Is a few days before the event enough?

Btw, I will update the dates at the top of the issue to reflect reality as soon as I get more clarity about your hub availability needs. Thanks!!

[mabuice]

The event itself starts August 20th. We were hoping to have the initial hub up and running by the end of July so that we can iterate on the environment and test the various parts we need to incorporate (data from AWS, MySQL server for one dataset, etc.) as well as making sure we can get TAs and staff comfortable with administering the environment and fix our course materials beforehand. We would like as much time for that as you can give us. This is why I threw together the basic hub linked above.

[colliand]

I updated the Required start date in the top entry of the issue to 2021-07-31.

[damianavila]

@mabuice, thanks for the additional information!
I have assigned the task to @yuvipanda who is the engineer who is going to deploy the new hub.

I updated the Required start date in the top entry of the issue to 2021-07-31.

I moved that date to be the target date and I add a few more days to the required date.
Realistically, I think we are going to be closed to the target date but I do not want to promise something if there is a chance of delays, and given the fact that today is July 27th, I think adding a few more buffer days set the right expectations.

We have an AWS account with an associated domain name that we'll be using for this.

@mabuice, can you provide access to @yuvipanda? Thanks!!
Ref: https://infrastructure.2i2c.org/en/latest/topic/cloud-auth.html#access-individual-aws-accounts.

[mabuice]

For hooking up the right AWS account I’m going to connect @yuvipanda with @aamster and @morriscb.

[yuvipanda]

ty, @mabuice! I'm at yuvipanda@2i2c.org for an invite.

[aamster]

@yuvipanda I just granted you access. You should have received an email. Let me know if you have what you need.

[yuvipanda]

@aamster can confirm I have access! \o/

I'll also grant access to the other 2i2c engineers shortly. Hub should be up in a day or two. Thanks!

[yuvipanda]

I'm working on this now!

[yuvipanda]

I'm setting this up on us-west-2 (Oregon) now. I had to delete the existing empty VPC (swdb-2020-vpc) to make room for this, as we were up to the limit of VPCs (5)

[mabuice]

might need cudatoolkit==11 on the dot. The K80 might not be compatibility with later versions.

But let me try explicitly specifying pytorch-gpu...

[yuvipanda]

ok, the GPU profile should now only be available to people with the GPU team on the staging hub. Can you get someone else to test it too? if it works I'll deploy it to production.

[mabuice]

will do.

[mabuice]

pytorch -> pytorch-gpu fixes the issue, even with cudatoolkit==11.

Solved.

[mabuice]

@yuvipanda: @fcollman can see the gpu instance on the staging hub.

[yuvipanda]

@mabuice I see the GPU profile! but same as before with the image - I can see nvidia-smi show the GPU, but pytorch doesn't see it. I am using the latest tag available in https://quay.io/repository/mabuice/swdb2022?tab=tags which is a5fb5c42540b. I see that you have admin now, so you should be able to set the image for staging too at https://staging.allen-swdb.2i2c.cloud/services/configurator/.

[mabuice]

The current image with explicitly installed pytorch-gpu has torch.cuda.is_available()==True. We're good on that end. Tested on staging.

@yuvipanda Looks to me like we can update the main Hub.

[yuvipanda]

@mabuice done!

[mabuice]

@yuvipanda Is the user of a given instance on the sudoer list? Is there a sudo password?

[yuvipanda]

@mabuice there's no sudo enabled in the container. What is the use case you're trying to solve?

[mabuice]

The current use case is that one of our more technically minded TAs asked me if we had sudo access, so I said I didn’t think so, but I’d ask. :).

[yuvipanda]

@mabuice aaah, cool :) Almost everything they can do with sudo access, they can by modifying the imgae via PRs!

How's the event going?

[mabuice]

That’s kinda what I figured. 😀

Things are going pretty well. We’ll see how it goes when things ramp up later in the week.

[yuvipanda]

@mabuice just wanted to check-in - how is it going? :)

[fcollman]

It's going well! we have had no major issues in the introduction portion of the dataset, and we even did a couple of within course live nggitpuller links with relatively few issues.

we do seem to sporadically run into edge cases that I don't totally understand where people aren't able to do the nbgitpuller, and we revert to renaming their folders and pulling a fresh one. But that has been the minority of the experience.

[yuvipanda]

@fcollman if you can report the errors they get in https://github.com/jupyterhub/nbgitpuller/, that would be helpful!

[yuvipanda]

@fcollman @mabuice how did this go?

[colliand]

This hub can be decommissioned at the end of September. FYI @damianavila .

[mabuice]

@yuvipanda It went very well! Thank you for your help!

[damianavila]

Thanks @mabuice for your feedback!

Keeping this issue open until we merge: #1585.

This hub can be decommissioned at the end of September. FYI @damianavila .

Btw, I have created a decommission issue over here: #1722.

[choldgraf]

I'm removing the "due date" for this issue because we've now deployed this hub. I believe that we can also close this issue (especially once #1722 is complete) but will leave that to others to decide since you have more context than I do.

[yuvipanda]

This is done!