Refactor GCP terraform code + add MOEM-IGE hub #429
Conversation
|
I think I am confused here... or I am missing some context. There is a Grenoble hub already in some of the existing cluster: https://github.com/2i2c-org/pilot-hubs/blob/master/config/hubs/2i2c.cluster.yaml#L305 Why we are deploying a new cluster with the meom name and a new hub there? |
|
I just added a little more context in #207 (comment). Basically they got their own project and we're moving the cluster there. |
|
Thanks for the clarification and the additional info @yuvipanda! |
Mmm... that is weird... maybe the setup was slightly different in comparison with the previous NFS server? |
Yeah, I spent many hours trying to figure out what could possibly be different but couldn't :( |
yuvipanda
changed the title
|
I did a major refactor, added inline docs, and updated the PR description :) I've deployed this too. |
|
I like what I saw here... the splitting helps to understand the structure and should be easier to maintain in the future. |
yuvipanda
force-pushed
the
grenoble-1
branch
2 times, most recently
from
83f5a65
to
d447f9a
Compare
Calls terraform-docs dynamically from sphinx
- Remove terraform-specific dir, not enough docs there. Autogenerated reference docs will be moved to a different PR as I can't seem to get conda to work
Can't seem to get conda to work on RTD for now
Leftover from environment.yml conversion
|
Ok, I added docs as well and addressed your comments, @damianavila! |
docs/topic/cluster-design.md
Outdated
| Long-term, (2) is the appropriate way to do this for everyone. However, it affects the size | ||
| of the core node pool, since it runs some components in the cluster. For now, we use (1) for | ||
| single-tenant clusters, and (2) for multi-tenant clusters. If nobody wants a scratch GCS bucket, | ||
| neither option is required. |
There was a problem hiding this comment.
This last sentence suggests, IMHO, that we can disable the config connector (2) and the matadata concealment (1), but AFAIK, the option (1) is always available even if we do not want a scratch bucket, right?
There was a problem hiding this comment.
OK, this is getting big, and what we currently have is in good shape.
I think we should merge this one ASAP and, eventually iterate on leftovers.
Thanks for all the efforts in the docs and explanatory comments, @yuvipanda.
|
yay, thank you, @damianavila :) |
|
w00t, tyvm @damianavila and @sgibson91! |
This wasn't working earlier because of a typo in .readthedocs.yml that I didn't spot This reverts commit c77ef3e.
Since conda seems to now work with RTD This reverts commit 70b39e8.
--output requires a newer version of terraform-docs on conda-forge (conda-forge/go-terraform-docs-feedstock#25).
|
Thanks to @sgibson91's spot I managed to get conda working on RTD, but need a newer version of terraform-docs. I opened conda-forge/go-terraform-docs-feedstock#25, and we can autogenerate terraform reference docs after that is merged! |
It was already merged! |
of the higher level gke module. The code gets simpler, and
makes more terraform features (like for_each) accessible more easily.
research hubs want 2-3 options of notebook sizes to optimize for
spend. I attempted to use gke node autoprovisioning instead of
requiring manual nodepool provisioning, but it consistently
provisioned nodes bigger than required. We should re-evaluate it
later.
restricted SA is used for this, to prevent damage as much as possible.
Users can then make requests to GCS buckets in other
projects on behalf of this project.
nodepools. Can be overriden if necessary.
into multiple files for easier maintenance
already bumped in newer z2jh versions
insecureflag - even though other hubs are setup the same way and didn't
need this. NFS situation needs to be sorted.
Ref #207
TODO:
[ ] Add a template for new clusters(another PR)