Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to delete the clusters which failed before calling cluster_driver.create_cluster #126

Closed
okozachenko1203 opened this issue Jun 20, 2023 · 0 comments · Fixed by #127, #120 or #338

Comments

@okozachenko1203
Copy link
Member

okozachenko1203 commented Jun 20, 2023

The cluster creation has been failed at cert_manager.generate_certificates_to_cluster(). i.e. cluster_driver.create_cluster was not called and it means cluster.stack_id is none. Cluster-autoscaler helmrelease deletion is failed because its name is None.

Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: Command: helm delete --namespace magnum-system None
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: Exit code: 1
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: Stdout: ''
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: Stderr: 'Error: uninstall: Release name is invalid: None\n'
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/osprofiler/profiler.py", line 159, in wrapper
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server   File "/opt/stack/magnum/magnum/conductor/handlers/cluster_conductor.py", line 191, in cluster_delete
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server     cluster_driver.delete_cluster(context, cluster)
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server   File "/home/ubuntu/magnum-cluster-api/magnum_cluster_api/driver.py", line 266, in delete_cluster
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server     resources.ClusterAutoscalerHelmRelease(self.k8s_api, cluster).delete()
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server   File "/home/ubuntu/magnum-cluster-api/magnum_cluster_api/helm.py", line 92, in __call__
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server     return super().__call__()
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server   File "/home/ubuntu/magnum-cluster-api/magnum_cluster_api/helm.py", line 36, in __call__
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server     return super().__call__(
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server   File "/home/ubuntu/magnum-cluster-api/magnum_cluster_api/helm.py", line 23, in __call__
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server     return processutils.execute("helm", *self.COMMAND, *args, **kwargs)
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 438, in execute
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server     raise ProcessExecutionError(exit_code=_returncode,
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server Command: helm delete --namespace magnum-system None
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server Exit code: 1
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server Stdout: ''
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server Stderr: 'Error: uninstall: Release name is invalid: None\n'
Jun 20 12:32:47 mcapi-large magnum-conductor[1983841]: ERROR oslo_messaging.rpc.server 
@okozachenko1203 okozachenko1203 changed the title Failed to delete the cluster because stack_id is None Failed to delete the clusters which failed before calling cluster_driver.create_cluster Jun 20, 2023
mnaser pushed a commit that referenced this issue Jun 22, 2023
This was referenced Jul 31, 2023
This was referenced Dec 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment