Manual Removal - Namespace Stuck at Terminating Status

[woodwardmatt]

Describe the bug:
I was trying a fresh install with the manifest file, but have a number of errors appear to do with the webhook. I following the troubleshooting but couldn't get a resolution. I decided to manually remove all objects and retry with a helm install instead. All objects within the namespace "cert-manager" are deleted, but I am unable to delete the "cert-manager" namespace itself - it's stuck at the "Terminating" status.

Expected behaviour:
Deleting the empty namespace should remove the namespace from the cluster

Steps to reproduce the bug:
Run a fresh install with the manifest file. Delete all objects within the namespace manually after install is "complete". Then try to delete the namespace.

Anything else we need to know?:

Command message:

# What I see when trying to delete the namespace
kubectl delete namespace cert-manager
Error from server (Conflict): Operation cannot be fulfilled on namespaces "cert-manager": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

# What I see when listing objects in the namespace
kubectl get all --namespace cert-manager
No resources found.

# What I see when inspecting the namespace object
kubectl get namespace cert-manager -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"labels":{"certmanager.k8s.io/disable-validation":"true"},"name":"cert-manager","namespace":""}}
  creationTimestamp: 2019-02-20T10:09:06Z
  deletionTimestamp: 2019-02-22T09:02:27Z
  labels:
    certmanager.k8s.io/disable-validation: "true"
  name: cert-manager
  resourceVersion: "13191191"
  selfLink: /api/v1/namespaces/cert-manager
  uid: 8ea8e65e-34f7-11e9-9519-d2c2f38b7628
spec:
  finalizers:
  - kubernetes
status:
  phase: Terminating

Environment details::

• Kubernetes version (e.g. v1.11.2): v1.11.3
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): Azure
• cert-manager version (e.g. v0.4.0): 0.6
• Install method (e.g. helm or static manifests): Static Manifests

/kind bug

[elbaro]

Experienced the same. There is an upstream issue about zombie namespace. kubectl get all --namespace=cert-manager does not list all objects, so try delete -f ~~.yaml. For me kubectl delete apiservice [cert-manager-related-apiservices] and reboot fixed it.

[woodwardmatt]

Thanks @elbaro that makes sense, so would I just use the original manifest file in that delete command? :)

[woodwardmatt]

Just to resolve this issue. I followed @elbaro's advice above and ran the delete command with the original manifest file. This cleared up any left over objects from the original install and a few moments later the namespace was removed by the K8s.

Now to try the Helm install! :)

[kopaygorodsky]

Status of webhook apiservice is false.
v1beta1.webhook.cert-manager.io cert-manager/cert-manager-webhook False (ServiceNotFound)

This is the fix: kubectl delete apiservice v1beta1.webhook.cert-manager.io

[seunroava]

Even that command didn't work for me @kopaygorodsky

nonroot@cloudshell:~/spinnaker$ kubectl delete apiservice v1beta1.webhook.cert-manager.io
Error from server (NotFound): apiservices.apiregistration.k8s.io "v1beta1.webhook.cert-manager.io" not found

[kopaygorodsky]

are there any other hung apiservices from cert-manager ?

[seunroava]

As far as I knew, nothing. But here is a solution that eventually worked for me:

NAMESPACE=cert-manager

kubectl get namespace $NAMESPACE -o json > $NAMESPACE.json
sed -i -e 's/"kubernetes"//' $NAMESPACE.json
kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f ./$NAMESPACE.json

I had implemented the first two steps earlier which didn't work. The third command finished it off for me.

Credit to Stackoverflow. Can't remember the link right now.

[darrenli6]

As far as I knew, nothing. But here is a solution that eventually worked for me:

NAMESPACE=cert-manager

kubectl get namespace $NAMESPACE -o json > $NAMESPACE.json
sed -i -e 's/"kubernetes"//' $NAMESPACE.json
kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f ./$NAMESPACE.json

I had implemented the first two steps earlier which didn't work. The third command finished it off for me.

Credit to Stackoverflow. Can't remember the link right now.

Thanks ,I solve my problem to use this way.