Set min nodes to 0 for worker and user.

[pt247]

Reference Issues or PRs

#2154

What does this implement/fix?

Reduces cost of default deployment.

Change

An AWS Node group can be associated with several Auto Scaling groups, but in our case, it's just one. To scale from zero in AWS, you need to follow these steps:

1. Define the service node selector, such as dedicated=user.
2. Ensure that the nodes groups have a corresponding label like dedicated=user. This will help the service decide which node group to place the pod on.
3. ASG is required to give cluster-autoscaler a hint like k8s.io/cluster-autoscaler/node-template/label/dedicated=user.

Put a x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features not to work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):
  As a part of this change, the user scheduler is moved to the General node. This is needed to ensure the user node scales down to zero when the user is not running notebooks or desk clusters.

Testing

• Did you test the pull request locally?
• Did you add new tests?

Any other comments?

[pt247]

This needs further testing. As scaling up from 0 nodes required more than just setting min nodes to 0.

[pt247]

I have now tested scaling up and down of user and worker nodes from and to 0. I did this by creating the following:

1. Creating a JupyterHub cluster for the user.
2. Creating a Dask-cluster for the user.
3. Deleting Dask-cluster.
4. Release the node from the Jupyter node hub.

I had to move the user scheduler to general, though. This was needed because the user scheduler on the user node will always trigger the user node scale-up.

[pt247]

Just a couple of changes are needed.

1. Moving tagging of ASG to a later stage. As it needs to exist before it can be tagged.
2. Revert changes to retain original node selectors.

[pt247]

I need to figure out why Local Integration Tests are falling:

[terraform]: │ Error: failed to refresh cached credentials, no EC2 IMDS role found,
[terraform]: │ operation error ec2imds: GetMetadata, http response error StatusCode: 404,
[terraform]: │ request to EC2 IMDS failed

I'm still trying to figure out what the root cause is. If anyone has suggestions, please let me know.
I will work on this later today.

[pt247]

Update

After moving the tagging logic to the post_deploy stage of Phase 2, we were able to successfully get past the Deploy Nebari phase in the test-local-integration. However, we are now encountering a failure at the Cypress Run stage. Specifically, the failure is occurring during the First Test -- Check Nebari login and start JupyterLab. Upon reviewing the saved video, it appears that the login is functioning properly, but the JupyterLab server is not starting.

Next steps

I have set up an EC2 instance and installed the local cluster. I will attempt to log in to it by exploring the display through SSH.

Please let me know if there is a better way.

[pt247]

I have modified the code to make sure that only changes for AWS don't affect other deployments. With this, the tests are working as expected except for the Playwright Tests, which is failing in development, as far as I know.

[costrouc]

Looks great to me!

[pt247]

All checks pass(ed). If you are happy with the changes, please feel free to merge them @costrouc