Understand reasons for clusters having more than one core node running

[consideRatio]

Without some active care, we end up with more than 1 core node even though a single core node could handle the CPU/Memory requirements of the core workloads to run on it.

I'd like to go through the current clusters with multiple core nodes to see if we can do some quick and simple actions to reduce a few.

Known reasons for multiple core nodes

• A horisontal pod autoscaler adds multiple replicas of a pod that has anti-affinity to itself, making it cause the cluster-autoscaler to start additional core nodes to fit the pod (user nodes are tainted and not tolerated)
  • Example: GCP, konnectivity-agent - a simple fix has been to tweak a configmap in kube-system declaring how it should scale with the total number of nodes in the cluster so that it doesn't scale up so many konnectivity-agent pods - which is acceptable for this pod.
• The amount of allocatable pods per node is filled up
  On GKE, we can schedule a bit more than 100 pods per node, but on EKS a 2 CPU node can only fit ~27 pods or so, and a 4 CPU pod can only fit ~57 or so. This is the cause for 2i2c-aws-us having 2 core nodes atm for example.

Current core node count

# clusters with multiple core nodes
Azure, utoronto, 3
AWS, 2i2c-aws-us, 2
AWS, catalystproject-africa, 2
GCP, cloudbank, 2
GCP, leap, 3
GCP, pangeo-hubs, 4

# clusters with 1 core node
AWS, bican, 1
AWS, dandi, 1
AWS, earthscope, 1
AWS, gridsst, 1
AWS, jupyter-health, 1
AWS, jupyter-meets-the-earth, 1
AWS, kitware, 1
AWS, linc, 1
AWS, nasa-cryo, 1
AWS, nasa-esdis, 1
AWS, nasa-ghg, 1
AWS, nasa-veda, 1
AWS, openscapes, 1
AWS, opensci, 1
AWS, smithsonian, 1
AWS, ubc-eoas, 1
AWS, victor, 1
GCP, 2i2c-uk, 1
GCP, 2i2c, 1
GCP, awi-ciroh, 1
GCP, catalystproject-latam, 1
GCP, hhmi, 1
GCP, linked-earth, 1
GCP, meom-ige, 1
GCP, qcl, 1

Related

• Pod reductions
  • basehub: rely on a single user-scheduler replica #3869
  • basehub: allow shared-volume-reporter to be disabled, and disable it on 2i2c-aws-us and catalystproject-africa #4000
• Pod with multiple replicas and anti-affinity of some kind
  • AKS / UToronto: calico-typha scaled to three replicas, forcing three core nodes #3592
  • Decide and document on pod scaling of GKE's clusters calico-typha pods, and konnectivity-agent pods #2490

Action points

• Investigate the reasons for multiple core node in
  • Azure, utoronto, 3
    AKS / UToronto: calico-typha scaled to three replicas, forcing three core nodes #3592 (comment)
  • AWS, 2i2c-aws-us, 2
    Core node not out of CPU/Memory, but out of allocatable pods.
  • AWS, catalystproject-africa, 2
    Core node not out of CPU/Memory, but out of allocatable pods.
  • GCP, cloudbank, 2
    OK - out of CPU/memory/pods if using only 1 node
  • GCP, leap, 3 (now 1)
    Failure to scale down after scaling up due to konnectivity-agent and calico-typha horizontal pod autoscaler scaling up as for example ~100 dask worker nodes get started.
    I manually drained the two extra core nodes, but ideally we could get the cluster-autoscaler to scale down these nodes being almost empty by itself when the massive dask-worker node count has decreased.
  • GCP, pangeo-hubs, 4 (now 2)
    Same as leap, but could not go to a single core node since prometheus-server had massive memory requests, so reduced from 4 to 2.
• Close this and open a new issue for another action point if that makes sense
  I think the followup action I'd like to take can be tracked by Reduce cost of 2i2c-aws-us (and catalystproject-africa) by reducing core nodes from 2 to 1 #4003.