Create a process for tracking and invoicing cloud infrastructure costs

[choldgraf]

Context

There are several communities for which we are paying their cloud costs on their behalf. We are hand wavy about calculating cloud costs and do not have a system in place for rigorously tracking their cloud usage, conveying this information to CS&S, and invoicing them for it. This means that we are paying for cloud infrastructure without recovering the cost of doing so.

Proposal

We need to define a process for tracking and invoicing cloud infrastructure. This is going to require input from all of our teams, and I'll try to break this down by the issues that I think are team-specific. There are three things that we need to do:

Define how we calculate "usage" for a JupyterHub in order to calculate cloud costs. This is broken down by "shared clusters" and "dedicated clusters".

shared clusters

• Calculate cloud costs for shared hubs on a cluster on Google Cloud #499
• Formula for calculating hub-specific cloud costs on a shared cluster infrastructure#730

dedicated clusters will be much simpler

• Define Monthly process to Calculate cloud costs for dedicated clusters on Google Cloud #498

Then we must automate this process:

• Build a script / tool that calculates this statistic for each hub on a cluster, and turns that into a $ amount
• Figure out what kind of form CS&S wants the data in

And send it to CSS to try out:

• Run this script, give CS&S the data, and see how it goes

Updates and actions

We have an Engineering related companion issue at 2i2c-org/infrastructure#1853.

[damianavila]

This is super important (and why I prioritize it as critical), otherwise, we are effectively losing revenue that will impact our sustainability.

[choldgraf]

Considering how important this one is, I think that we should find somebody that is responsible for championing this and make progress on it relatively quickly. @damianavila can you find somebody on engineering that can put some cycles into this? Or can you champion this and set a checkpoint time + goal to accomplish so that we can make progress?

[choldgraf]

Update: James will spearhead this one

In a slack message, @jmunroe mentioned that he will spearhead this issue so that we can make progress. I've assigned him and added it to our backlog. @jmunroe maybe we should discuss this in our community content meeting tomorrow?