[EVENT] OceanHackWeek 2022

[GeorgianaElena]

Summary

Event Info

• Community Representative: @abkfenris
• Event begin: August 15
• Event end: August 19
• Active times: todo
• Number of attendees: around 100 participants or so, and 30ish organizers/instructors/helpers
• Notes: The undergraduate program starts a week earlier and goes a week later, so the event is longer (August 8-26), though that is going to be about a tenth of the users as we will have from 15-19 for the HackWeek itself.
• Hub Events Calendar

Hub info

• Hub URL: https://oceanhackweek.2i2c.cloud/user
• Hub decommisioned after event?: todo

Task List

Before the event

• Dates confirmed with the community representative and added to Hub Events Calendar.
• Quotas from the cloud provider are high-enough to handle expected usage.
• One week before event Hub is running.
• Confirm with Community Representative that their workflows function as expected.
  • 👉Template message to send to community representative

    Hey {{ COMMUNITY REPRESENTATIVE }}, the date of your event is getting close!
    
    Could you please confirm that your hub environment is ready-to-go, and matches your hub's infrastructure setup, by ensuring the following things:
    - [ ] Confirm that the "Event Info" above is correct
    - [ ] On your hub: log-in and authentication works as-expected
    - [ ] `nbgitpuller` links you intend to use resolve properly
    - [ ] Your notebooks and content run as-expected
    

• 1 day before event, either a separate nodegroup is provisioned for the event or the cluster is scaled up.

During and after event

• Confirm event is finished.
• Nodegroup created for the hub is decommissioned / cluster is scaled down.
• bump-ohw-image-tags.yaml from https://github.com/2i2c-org/infrastructure/tree/master/.github/workflows is deleted
• Hub decommissioned (if needed).
• Debrief with community representative.
  • 👉Template debrief to send to community representative

    Hey {{ COMMUNITY REPRESENTATIVE }}, your event appears to be over 🎉
    
    We hope that your hub worked out well for you! We are trying to understand where we can improve our hub infrastructure and setup around events, and would love any feedback that you're willing to give. Would you mind answering the following questions? If not, just let us know and that is no problem!
    
    - Did the infrastructure behave as expected?
    - Anything that was confusing or could be improved?
    - Any extra functionality you wish you would have had?
    - Could you share a story about how you used the hub?
    
    - Any other feedback that you'd like to share?
    
    

[damianavila]

@abkfenris, one thing we forgot to ask you in #1515 is if there is an issue with pre-existing users (from the previous incantation of the hub) being able to still have access to this new hub.

cc @GeorgianaElena who raised this point in previous conversations.

[abkfenris]

@damianavila i don't think there is an issue if they can or can't access it. I gave them a warning a week or so ago that they needed to take care of anything they cared about. At some point we should figure out how to archive or delete all of them.

[abkfenris]

Is there a way to add a shutdown or log out option to RStudio? Or is the best option going to be loudly advertising https://oceanhackweek.2i2c.cloud/hub/home anytime participants may be using R?

[sgibson91]

Is there a way to add a shutdown or log out option to RStudio? Or is the best option going to be loudly advertising oceanhackweek.2i2c.cloud/hub/home anytime participants may be using R?

I think this is the way to go unfortunately. It is really difficult to get in and edit the RStudio UI from the hub (probably due to it being company-operated)

[abkfenris]

Thanks, I thought that was probably the case.

[damianavila]

Number of attendees: around 100 participants or so, and 30ish organizers/instructors/helpers

@abkfenris can you confirm the active times? Looking here: https://oceanhackweek.github.io/ohw22/schedule.html, it seems it will be a US Pacific time event, am I correct?

@sgibson91 @GeorgianaElena do you think a pre-warm step would be needed on Monday morning?

[abkfenris]

Number of attendees: around 100 participants or so, and 30ish organizers/instructors/helpers

@abkfenris can you confirm the active times? Looking here: https://oceanhackweek.github.io/ohw22/schedule.html, it seems it will be a US Pacific time event, am I correct?

@sgibson91 @GeorgianaElena do you think a pre-warm step would be needed on Monday morning?

Ehh, kind of. That's when we will have the most people active for tutorials, but various in person events will be logging in earlier.

For instance for us in Maine, we're planning on getting folks logged in either sometime that morning, or just after lunch so we can make sure everyone is ready when the tutorials start up (so then we can also have the bandwidth to help virtual folks).

[sgibson91]

I don't believe there's anything to do in terms of scale up?

• A separate node pool has already been provisioned here:

  infrastructure/terraform/gcp/projects/pilot-hubs.tfvars

  Lines 40 to 52 in dd74fe1

  	"ohw": {
  	min: 0,
  	max: 100,
  	machine_type: "n1-highmem-8",
  	labels: {
  	"2i2c.org/community": "ohw"
  	},
  	gpu: {
  	enabled: false,
  	type: "",
  	count: 0
  	}
  	}

• And the appropriate nodeSelector has already been configured here and in other places in the config:

  infrastructure/config/clusters/2i2c/ohw.values.yaml

  Lines 6 to 7 in dd74fe1

  	nodeSelector:
  	2i2c.org/community: ohw

I've never scaled up a hub before an event so I might be missing something. We also don't have any docs on this process so I opened #1628

I'm considering this as "No action needed" from me unless I'm told otherwise

[abkfenris]

@ocefpaf and I have been bashing our heads against an error that we're seeing in both the Python and R profiles.

It appears that both Jupyter Notebooks/console and RStudio are removing some environment variables that various packages expect to see.

The most common issue is that the PROJ_LIB environment variable seems to be getting removed but there are others that go missing too.

The easiest way to see the difference is to compare env | grep PROJ in a terminal and !env | grep PROJ in a Python console. os.environ also doesn't include PROJ_LIB and neither does an R Sys.getenv().

oceanhackweek/ohw-tutorials#94
oceanhackweek/jupyter-image#57

Any thoughts?

[sgibson91]

I remember @damianavila went through an arduous process in trying to get RStudio to load Keras + Tensorflow correctly from a conda environment and did something with environment variables at some point.

• His full write-up is here: https://discourse.jupyter.org/t/openssl-mismatch-between-rstudio-and-conda-environments/14123
• And here is a relevant PR: A different approach trying to make RStudio lo load the proper libssl stuff utoronto-image#29

It may involve adding an Rprofile.site file to set environment variables but I'm not sure.

---

On a related note, we recommend using the support channel for bugs like this because that gets greater visibility from the engineering team and helps us isolate urgent or critical bugs from the "noise" of hub issues.

[yuvipanda]

You definitely need an Rprofile.site for setting env vars in RStudio.

See this line in the utoronto Rprofille.site: https://github.com/2i2c-org/utoronto-image/blob/main/Rprofile.site#L11

And it is copied over in the Dockerfile here: https://github.com/2i2c-org/utoronto-image/blob/d422076c3695e44f09a959b14cb10f89b6e5f538/Dockerfile#L178

[yuvipanda]

If I launch a server with the R image, I do see PROJ_LIB in notebook:

and in terminal too. Not in RStudio but that does need Rprofile.site.

Is the python image lagging behind maybe? Does it not have any specific change the R one does?

[ocefpaf]

Does it not have any specific change the R one does?

Yep. I hardcoded those env vars for the R image in oceanhackweek/jupyter-image#57 but I did not port those to the Python image yet. I was hoping that we would not need them.

Thanks for the RPorfile tip! We are not R folks trying to make it work for an R tutorial.

@abkfenris I'm getting ready for my tutorial. Unless you beat me to it I'll create the RProfile when I'm done.

[abkfenris]

I'll take a swing at setting up a RProfile.site.

I'm still confused why env variables that show up in the terminal from the conda env aren't making it into Notebooks or console sessions.

[ocefpaf]

I'm still confused why env variables that show up in the terminal from the conda env aren't making it into Notebooks or console sessions.

Same. Looks like the activation is working but is getting clobbered by something.

[abkfenris]

Making progress. Still have the GEOS linking version error, but I can at least run things locally without setting the env variables manually now.

Building an image now oceanhackweek/jupyter-image#59

[abkfenris]

Ok, PR made #1632

[abkfenris]

Thank you @sgibson91 and @yuvipanda R is playing nicer (or at least our R folks say it is)!

[yuvipanda]

So I know why this is happening - it looks like the PROJ env var is set by conda, and runs when conda is activated. conda is activated when you open a terminal, but not when you start the notebook itself. repo2docker 'fixes' this with https://github.com/jupyterhub/repo2docker/blob/main/repo2docker/buildpacks/python3-login as the entrypoint.

So if you add an entrypoint script to your image that is just:

#!/bin/bash -l
exec "$@"

and set it as ENTRYPOINT in your Dockerfile, it will solve the issue!

You still need the Rprofile thing for R tho

[abkfenris]

Awesome, thank you @yuvipanda! I'm working on getting that in now.

We need to take a look at repo2docker for building our environment again. We may have a nice addition from some of our R users for a more reliable/predictable R install script.

[yuvipanda]

@abkfenris hope it worked ok!

Lots of nice R goodies in repo2docker ;)

[GeorgianaElena]

Hey @abkfenris , the event appears to be over, is that right?

[abkfenris]

Yep, we've wrapped up the scheduled weeks. We would like to keep the hub active for as long as reasonable, but we can scale down the dedicated node pools and similar at this point.

[GeorgianaElena]

Makes sense! Thanks for letting us know @abkfenris.

We hope that your hub worked out well for you! We are trying to understand where we can improve our hub infrastructure and setup around events, and would love any feedback that you're willing to give. Would you mind answering the following questions? If not, just let us know and that is no problem!

- Did the infrastructure behave as expected?
- Anything that was confusing or could be improved?
- Any extra functionality you wish you would have had?
- Could you share a story about how you used the hub?

- Any other feedback that you'd like to share?

[abkfenris]

We're debriefing the event tomorrow, so I'll get back to you after that.

Probably sometime this weekend as I'm road tripping across the country and will have a bunch of time in the passenger seat with low bandwidth to force me to focus.

[GeorgianaElena]

We're debriefing the event tomorrow, so I'll get back to you after that.

There's no hurry

Probably sometime this weekend as I'm road tripping across the country and will have a bunch of time in the passenger seat with low bandwidth to force me to focus.

Have fun road tripping 🚗

[abkfenris]

Other than the hiccups with the proxy/spawner due to Google hiccups, I think the hub worked largely as expected. We were lucky that those didn't occur during the main part of OHW, let alone when I was commuting to/from the lab. It's good to see that you are working on solving the bugs upstream and have a plan for migrating the infrastructure to regional infrastructure.

It did seem like the admin may not have accurately reflected who was active during the event. It would have been nice to have access to Grafana to be sure and debug issues. Maybe via an admin se

It would be really nice to have a faster way to test and update the images (for when I get told about packages just before they are used for a tutorial), that didn't require 2i2c's intervention.

One thought is if we had a test hub without profiles, so that we could use configurator to test images in a live hub.

My other thought is if you had a bot or github action that could merge PRs from community representatives to image fields.

It might be cool to have something like cdsdashboards setup, so that participants could make and share some lightweight apps, without having to deal with trying to set up their own servers.

For cool things that happened on the hub, you can check out the project presentations.

Thanks again for all your work!

[GeorgianaElena]

Thanks for sharing this extremely valuable feedback @abkfenris ✨ We will open up new issues based on your suggestions 🚀 (cc @damianavila)