Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minimize base cost of our clusters #235

Closed
yuvipanda opened this issue Feb 17, 2021 · 2 comments
Closed

Minimize base cost of our clusters #235

yuvipanda opened this issue Feb 17, 2021 · 2 comments

Comments

@yuvipanda
Copy link
Member

I want our clusters to be as cheap as possible when nobody is using them. This means:

  1. Just one core node kept up at all times, and this should be as small as possible.
  2. Autoscale up only when users start it - we could either have one user node running, or 0 as we see fit. We can actually have enough room in the core node for just a placeholder pod, and then have a new node come up when the first user shows up.
  3. Figure out what to do with the NFS server - probably Run NFS servers in-cluster #50
  4. Run zonal clusters, not regional ones.
@yuvipanda
Copy link
Member Author

The smallest cluster you can run needs at least 4 cores, which is atrociously high. Almost all this is kube-system pods, and should really be counted as the GKE management fee. https://cloud.google.com/kubernetes-engine/docs/how-to/small-cluster-tuning has some suggestions, but isn't good enough. https://www.eficode.com/blog/custom-dns-on-gke also has useful information.

n1-custom-4-3840 is the smallest core node we can get. n1 is preferable over e2 since these nodes will be on all the time. It'll be 99% idle most of the time, which is infuriating, but hey that's the tradeoff for having a managed service!

I also performed some 'bring down' actions:

  • Make NFS VM smallest possible
  • Move the home directories away from their own external disk to the NFS VM's base disk.

NFS should definitely go in-cluster - see #50

@yuvipanda yuvipanda removed their assignment Nov 9, 2022
@yuvipanda
Copy link
Member Author

#2199 is a better version of this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant