[BUG] - AWS EKS user and worker node groups fail to scale to 0 nodes

[Adam-D-Lewis]

Description

AWS EKS user and worker node groups fail to scale to 0 nodes

[Adam-D-Lewis]

I think the cluster autoscaler won't scale down if a pod from a deployment is scheduled on it. AWS EKS includes a few add-ons by default (specifically the coredns and ebs-csi-controller deployments). Pods from those deployments were scheduled on the worker and user pods. This may be part of the reason the worker and user node groups didn't scale down. I saw a similar issue with kbatch and user-scheduler in jupyterhub but those can be disabled if needed.

You can't disable these default add-ons as far as I can tell. You can patch the deployment after they are deployed which may be a solution.

As a workaround, adding the following to the problematic deployment manifests can resolve the problem after deployment.

    nodeAffinity:    
      requiredDuringSchedulingIgnoredDuringExecution:    
        nodeSelectorTerms:    
        - matchExpressions:    
          - key: eks.amazonaws.com/nodegroup    
            operator: In    
            values:    
            - general    

[Adam-D-Lewis]

We could patch the problematic add-on deployments in the nebari/template/stages/03-kubernetes-initialize module

[iameskild]

@Adam-D-Lewis is this a priority for you? Should we assign someone to work on this?

[Adam-D-Lewis]

Not a priority because of the workaround

[viniciusdc]

Hi @Adam-D-Lewis, I assume #2353 fixed this issue right? can we mark this as completed?