cortex get command unresponsive

[WasiqMalik]

Version

0.18.1

Description

If the AWS cluster is manually deleted from AWS EKS console, the 'cortex cluster info' command and 'cortex get' command to display information stops responding. Even with the cluster deleted from EKS, cortex API still remains deployed on the virtual machines but 'cortex get' isn't able to return this.

Configuration

The issue is Irrelevant to cortex configuration.

Steps to reproduce

1. Delete the EKS cluster from AWS EKS console
2. run command 'cortex cluster info'
3. run command 'cortex get'

Expected behavior

There should be an error response from these commands in such a case to help the user understand that they have to 'cortex cluster down' and then delete cluster properly from AWS console to fix the issue.

Actual behavior

There was no error or information/issue displayed for the user.

Suggested solution

Just an error message to the user in this case, and guide the user to relevant solution when the cluster isn't pinned down properly.

[vishalbollu]

Thanks for bringing this to our attention. The cortex get command may take a while because it tries to get deployed APIs from all environments listed in cortex env list. cortex cluster down deletes the environment automatically so this shouldn't happen typically, however, because you've encountered issues while spinning down the cluster and deleted the cluster manually, the environment to your cluster may not have been deleted. Therefore when you run cortex get, it will try to get APIs from a deleted cluster and hang.

You can verify that this is your scenario by using cortex env list. If you still find the environment for your cluster (it is usually named aws be default), you can delete the environment with cortex env delete. Let us know if this works for you.

Following up, do you have more information on the error you've encountered while spinning down the cluster?

As for your suggestions, I have filed a ticket #1319 to add documentation for cortex cluster down cleanup.

Furthermore, maybe a timeout should be added to CLI requests to prevent the CLI from hanging #1320.

[deliahu]

@WasiqMalik We have added these docs for troubleshooting any issues that arise when running cortex cluster down. I'll go ahead and close this issue, feel free to reach out if you have run into any other problems.