Temporary pause of 2i2c Hub until Cloud Credits renewed

[jules32]

Hi everyone, unfortunately, we need to temporarily pause the 2i2c Hub until we get more cloud credits. We anticipate this pause will be very brief and will be resolved by tomorrow; we submitted a request to AWS/NASA for more credits on April 26 and they are aware that we have just ~$400 left and pausing until we have more credits. I am really sorry that this will disrupt your work in the meantime, and we’ll have it back as soon as possible. We’ll learn from the evolving credits process going forward to make sure we don’t have these kinds of disruptions.

In next 10 mins, I plan to press “Shutdown Hub”. We have already messaged active users to save what they’re doing and will update you when we are able to continue.

[jules32]

Today @yuvipanda was able to shut down the hub temporarily while we were together on Zoom with @betolink @ebolch @danielfromearth Eli Holmes @ateucher . The motivation to “shut down the hub”, meant to prevent anyone from using compute any more until we have more credits. If we use up credits, then Openscapes’s credit card gets billed. Why did this happen? seems that this quicker-than-expected use of credits was because of two coincident things: 1) May 30 workshop - 2024-SBG workshop. 60 participants, SinglePassword access for first time. (see screenshot below of compute costs) 2) our cloud credits being too low following the request for more credits Apr 26. In summary: High compute from a 60-person EMIT-ECOSTRESS workshop & followup. We would remove them after a week; wouldn’t be a problem except coincident with Cloud credits being too low.

Background from today and last weeks

• 2024-SBG Workshop:

  • git basics and then searching and using EMIT + ECOSTRESS data together an working with those file formats (point person: Eric Bolch, LP DAAC)

  • One large day (May 30) appropriate usage - but then followed by 2 more days of higher usage because when one user did not log out the node did not scale down - was this a bug, or was a user still logged in there? No crypto activity. But this contributed to this.

    • More background from Eli Holmes: We I taught a workshop and it was scaling down, the last user who was on was on a particular node type. Since that user was the last on, that node stayed running.

• Cloud credits were too low; submitted request for $15,000 more credits on April 26, still pending. Have alerted AWS/NASA contacts that we’ve temporarily shut down the hub.

• Today, Yuvi "shut down the hub" and brought down to 1 node from 3 nodes.

Follow-ups to do

• Investigate why the servers don’t scale down. Next steps prometheus data on Yuvi’s side (Grafana can’t be accessed while Hub is shut down)

• SinglePassword - change password after a week - how to automate?

• Learn: requesting credits from AWS/NASA

  • how long in advance should we request credits?

  • Is $15,000 too much, too little? Currently we use $10,000 in ~9 months. We’re reducing unnecessary storage costs and teaching users so we can get better estimate

• Failsafe - what is the other failsafe we need to prevent overspend, and so that there is not this gap again?

  • Could part of the grant/contract - “up to $5K will be reimbursed from contract to Julie if her credit card “ ?

  • Or is there a way to set up in “2i2c, I have $5,000 I don’t want to go above it”. (Better for 2i2c to shut it down based on budgets vs AWS shutting it down (bc we’d lose the home directory)) - Yuvi & 2i2c are working on quotas - each user can use XX compute, could this be part of that?

  • **Or alerts from grafana: “this node has been idle for 3 hours” (**What is important to control the costs - what is “idle”? See below

  • A node not being used - have an alert for this behavior - something 2i2c could look into 

• What does “idling” mean? Andy Teucher to investigate - 

  • What’s the current default behavior for idlying for an hour?> If you close the tab, the timeout is an hour. But if you leave the tab open, then there is still communication/socket with the front and back end. 

  • And what if someone has a longer-than-hour process running and close the tab? Will the process be killed in an hour?

[ateucher]

We have restored the 2i2c Hub and it is back up and running. @jules32 worked quickly to get the previously requested credits in place, and @yuvipanda restored the Hub to operation at ~14:00 PDT today.