Decomission the org-ops/ repository

[yuvipanda]

When doing #2 we realized that managing auth access via terraform is complex and often doesn't work, since cloud providers want access to be manually done. We should just archive this repo, and remove references to it from our documentation.

TODO:

• Update our access control docs to remove references to org-ops repo
• Figure out where our terraform state will be stored, and how it will be managed
• Remove the documentation section about "setting up new google cloud projects for all new hubs" https://pilot-hubs.2i2c.org/en/latest/howto/operate/setup-new-project.html
• Archive the repository (or make a 2i2c-attic org and move it there)

[choldgraf]

Looking through our docs, I don't see anything about org-ops in our onboarding documentation, or in the pilot hubs documentation, so I think that the first item has already been finished.

So I think that the only things to do next here are to:

• Remove this page: https://github.com/2i2c-org/pilot-hubs/blob/master/docs/howto/operate/setup-new-project.md
• Confirm that indeed our terraform state now lives here: https://github.com/2i2c-org/pilot-hubs/tree/master/terraform and mentioned here: https://pilot-hubs.2i2c.org/en/latest/topic/terraform.html (can somebody confirm this is right?)
• Archive the org-ops repository

That's it.

Does anybody think there's something else to do here? If not, I'll quickly do all of this so we can remove this extra bit of complexity from our repos :-)

[damianavila]

I think (I might be wrong about it) there is still a migration needed:

Figure out where our terraform state will be stored, and how it will be managed

Pretty sure @yuvipanda and @sgibson91 can add more info 😉

[sgibson91]

I think there's a misunderstanding around the word state here. Our terraform code lives here: https://github.com/2i2c-org/pilot-hubs/tree/master/terraform When you run terraform code, it outputs a terraform.tfstate file (and an ephemeral terraform.tfstate.lock file), this is how terraform knows what is already deployed and whether new changes will be destructive (tear down and redeploy) or non-destructive (change in place). Our state (.tfstate files) actually live in GCP storage buckets here: https://console.cloud.google.com/storage/browser/2i2c-terraform-state;tab=objects?forceOnBucketsSortingFiltering=false&authuser=1&project=two-eye-two-see&prefix=&forceOnObjectsSortingFiltering=false When you run terraform init -backend-config=/path/to/config, we are actually telling which bucket terraform should look at to get the state and update it as it performs operations.

The reason this is important is because terraform usually outputs the state locally, but that's not helpful when a distributed team is interacting with the same infrastructure as code 😉

Does this make sense @choldgraf?

[sgibson91]

I think the buckets for the org-ops repo are actually in the two-eye-two-see-org project: https://console.cloud.google.com/storage/browser?authuser=1&project=two-eye-two-see-org&prefix=

Also, the terraform code that is in this repo is not in pilot-hubs. The terraform code here creates a whole new project in GCP, whereas the code in pilot-hubs only deploys a Kubernetes cluster given a project already exists.

If the terraform code here moves to pilot-hubs I don't think the GCP buckets also need to move, we just need to make sure that that piece of terraform code knows where to look when it's run.

[choldgraf]

Thanks for this explanation, it helps a lot! I will try to incorporate this into PR in the pilot-hubs/ repo and perhaps you can set me straight there with whatever I get wrong, and we can use that as a step to close this issue.

Is it correct, then, that the terraform in this repository is not relevant at all for our current infrastructure, because it primarily concerns itself with setting up new projects, and this is not something that we plan on doing? In that case, I feel like we should just:

• Document the terraform state that's described in the two-eye... project
• Archive this repository since we won't use this terraform code anyway.

[sgibson91]

it primarily concerns itself with setting up new projects, and this is not something that we plan on doing?

I'm not sure. I think we need to define a few scenarios:

1. A community just wants a hub and they're fine being on 2i2c-pilot cluster (or equivalent on Azure/AWS) (e.g. Grenoble)
2. A community wants a hub in a separate project, and they want us to create/manage that project
3. A community wants a hub and they've brought their own project for us to deploy into (e.g. Pangeo)

Are we saying we're explicitly no longer supporting scenario 2?

[sgibson91]

Reflecting and reading the linked issue, I think you are right @choldgraf. I don't think anything needs to be migrated to pilot-hubs from here, instead we grant access manually as per the onboarding issue template. I think the backend or state is documented in the terraform docs you linked to as well, though I'm happy to review any clarifications you'd like to make.

[damianavila]

Are we saying we're explicitly no longer supporting scenario 2?

I think that is still a relevant question for this repo.

Btw, looking at https://pilot-hubs.2i2c.org/en/latest/howto/operate/setup-new-project.html#client-organization-provides-billing-account, how do manage billing without this repo in that scenario? Do we do it manually somehow?

[sgibson91]

how do manage billing without this repo in that scenario? Do we do it manually somehow?

It's certainly possible to do it non-programmatically https://cloud.google.com/billing/docs/how-to/modify-project I think the question is who needs what permissions on what resource to make the connection

[choldgraf]

Thanks for this discussion - I'm definitely not saying we shouldn't support option 2...but I'm wondering if we can track this as an ongoing issue. I worry that the existence of this repository gives us false confidence that we have a solution here, even though we know that this repo isn't actually up-to-date with our practices. So my thinking would be:

1. Archive this repository
2. Add docs to pilot-hubs/ that says "here's when to open a new project for people" that mostly contains manual steps
3. Open an issue that tracks our practices around creating new projects for people if they need them in the medium term

I think we definitely do need a story for deploying into projects we create for other communities (I don't know how else we can cap the cloud spend for a specific community, or easily keep track of exactly what their costs are). I just think that's a longer-term question, and this issue makes me think it is dis-entangled from this repository which is outdated. Does that sound right?

[choldgraf]

I got pulled into doing some more work for the Columbia sub-award, so I am going to remove this one from the Sprint and add the sub-award bit