[EVENT] OceanHackWeek 2023

[abkfenris]

Summary

OceanHackWeek 2023 is coming up August 7-11th.

Event Info

• Community Representative: Alex Kerney @abkfenris
• Event begin: 8/7 8 AM Pacific
  • In your timezone: https://arewemeetingyet.com/Los%20Angeles/2023-08-07/08:00/OHW%2023
• Event end: 8/12 8 PM Australia
  • In your timezone: https://arewemeetingyet.com/Perth/2023-08-12/20:00/OHW%2023%20-%20Australia%20edition
• Active times: I'm expecting the most activity 8 am - 8 pm Pacific, with another group in an Australian timezone
  • In your timezone: https://arewemeetingyet.com/Los%20Angeles/2023-08-07/08:00/OHW%2023
• Number of attendees: ~100
• Hub Events Calendar

Hub info

• Hub URL: https://oceanhackweek.2i2c.cloud
• Hub decommisioned after event?: Eventually, @colliand has the exact date, I've lost track of the email where we set it.

Task List

Before the event

• Dates confirmed with the community representative and added to Hub Events Calendar.
• Quotas from the cloud provider are high-enough to handle expected usage.
• One week before event Hub is running.
• Confirm with Community Representative that their workflows function as expected.
  • 👉Template message to send to community representative

    Hey {{ COMMUNITY REPRESENTATIVE }}, the date of your event is getting close!
    
    Could you please confirm that your hub environment is ready-to-go, and matches your hub's infrastructure setup, by ensuring the following things:
    - [ ] Confirm that the "Event Info" above is correct
    - [ ] On your hub: log-in and authentication works as-expected
    - [ ] `nbgitpuller` links you intend to use resolve properly
    - [ ] Your notebooks and content run as-expected
    

• 1 day before event, either a separate nodegroup is provisioned for the event or the cluster is scaled up.

During and after event

• Confirm event is finished.
• Nodegroup created for the hub is decommissioned / cluster is scaled down.
• Hub decommissioned (if needed).
• Debrief with community representative.
  • 👉Template debrief to send to community representative

    Hey {{ COMMUNITY REPRESENTATIVE }}, your event appears to be over 🎉
    
    We hope that your hub worked out well for you! We are trying to understand where we can improve our hub infrastructure and setup around events, and would love any feedback that you're willing to give. Would you mind answering the following questions? If not, just let us know and that is no problem!
    
    - Did the infrastructure behave as expected?
    - Anything that was confusing or could be improved?
    - Any extra functionality you wish you would have had?
    - Could you share a story about how you used the hub?
    
    - Any other feedback that you'd like to share?
    
    

[colliand]

Hi @abkfenris! Thanks for creating this event issue.

The note I see in the service agreement reads:

Hub is already set up at https://oceanhackweek.2i2c.cloud/.
Special service offering: $1500 operations/month X 2 months.
$3,000 total cost for hub to operate continuously through the end of August 2023.
Cloud costs will be billed separately.

[colliand]

FYI @abkfenris. The splash page for this hub is rendered in Spanish. It was set up in that way for the AECID funded event in March. Should this be updated for the upcoming event?

[abkfenris]

FYI @abkfenris. The splash page for this hub is rendered in Spanish. It was set up in that way for the AECID funded event in March. Should this be updated for the upcoming event?

It should. In theory I had a tab open with the PR that changed it to Spanish, so I knew where to start reverting, but I appear to have been buried by a tab-avalanche in the last few days and I can't dig it up now.

[abkfenris]

I thought there was a custom image built for the splash page, but maybe it's not as big of a change as I thought. #2233

[abkfenris]

Hmm, I was hoping that PR would change things back, but is there somewhere else that I need to make changes?

[yuvipanda]

@abkfenris I think #2909 should do the trick

[abkfenris]

Ah, I thought there was another repo involved, but I didn't know how it was referenced. Thanks!

In the past (looking at #1576 ) we've had a separate node pool for the event, and maybe had image pre-pulling configured. Is that still needed for our usage, or is the shared cluster up to the task without additional tweaks?

[sgibson91]

It's probably better to spin up a dedicated nodepool and enable pre-pulling for the sake of an enjoyable event experience. Without these things, users may suffer from slow startup times which can be frustrating during live events. I can set this up at the end of this week or in the European AM on Monday.

[abkfenris]

Great! Either of those times work for us.

[damianavila]

From a recent PR: #2907 (comment)

In the updated Python image, I also added the github-scoped-creds package. Does that require any additional setup from the 2i2c side?

Ref: https://infrastructure.2i2c.org/howto/features/github/

cc @sgibson91 (in case you did not see that comment).

[sgibson91]

Thanks for the ping @damianavila - I can make sure to add that in too

[sgibson91]

• The dedicated nodepool is being provisioned in Create a dedicated nodepool for OHW event #2921
• gh-scoped-creds is being configured in Configure gh-scoped-creds for OHW hub #2922

@abkfenris once #2922 is merged, you'll need to make sure this GitHub App is installed on the repository you want users to be able to push to https://github.com/apps/ohw-gh-scoped-creds-app

Also, I looked into enabling the image pre-puller, but it seems you have two images, a Python one and an R one. I didn't know which you wanted to be pre-pulled by default, and I also wasn't sure if this would cause issues if someone requested the image that wasn't specified to be pre-pulled.

[abkfenris]

Great! I just authorized the app.

The Python image will get the most use, so that is more likely to need to be pre-pulled, though if I am reading the spec right, it should pull images for both profiles?

[sgibson91]

Hmmm, yes perhaps so!

[sgibson91]

• Enabling prepuller: Enable pre-puller in ohw hub for event #2924

[sgibson91]

I think we are ready to go. I will add some nodes to the pool in the European AM on Monday to help with start up times.

[sgibson91]

I have (attempted to) scale up the nodepool in #2935, but the new nodes haven't come up and I'm not sure why.

[abkfenris]

I was just able to hop on and it spun up super quick. We are hopefully having everyone log in today, but we aren't doing a lot of compute or tutorials till tomorrow.

[sgibson91]

I ended up forcing the nodes to come up via the console, so you have 4 which I calculated would fit a maximum of 32 users each provided the lower resource limits of 0.5 CPU and 4 GB RAM each. If when you start doing heavy compute tomorrow it seems slow, let me know, and I'll bump up a couple of extra nodes.

[abkfenris]

Great, thank you.

[abkfenris]

We have at least some participants who can see the shared-readwrite/ directory, which they can browse, but not write to.

[sgibson91]

Ok @abkfenris can you open a support ticket so we can track it properly please? And I'll look into it

[sgibson91]

The shared-readwrite folder is an admin only feature, so only users who are admins should be permitted to write to the folder (or even see it, IIUC). All other users should be reading from the shared folder

• https://infrastructure.2i2c.org/topic/infrastructure/storage-layer/#hub-user-mount

[abkfenris]

I just submitted a ticket, but it's that our non-admin users are seeing it, which they haven't been able to in the past.

I haven't gotten to work with more than a few users, so I don't know if it's everyone yet, or just a small group, or try to look at what permissions they are seeing. The user who I was helping when I first noticed that they were seeing ~/shared-readwrite/ didn't have permission to write to it.

[sgibson91]

Ok, thanks for the ticket, I've made sure the engineering team are aware

[abkfenris]

Pencils are down, projects have been presented, and we're done. Thank you!

[damianavila]

@sgibson91, can you take care of the after-event tasks? Thanks!

[sgibson91]

• Disable pre-puller in ohw hub after event #2982
• Remove dedicated nodepool for OHW after event #2983

[sgibson91]

Thank @abkfenris! I'm leaving your config changes for gh-scoped-creds in place for now if you're using the same image, but have removed your dedicated nodepool and disabled the prepuller. I'll close this issue now.