PV not deleting #202

tylerpotts · 2020-12-10T22:14:40Z

tylerpotts
Dec 10, 2020

After running these commands with your persistent volume names in them:

kubectl patch pvc db-pv-claim -p '{"metadata":{"finalizers":null}}'
kubectl patch pod db-74755f6698-8td72 -p '{"metadata":{"finalizers":null}}'

Did you run the commands to delete the persistent volumes again? Conda-store won't work if the persistent volumes are deleted.

Once the volumes are deleted, redeploying in your github repo (with any kind of change to qhub-config) will create the volumes again

Answered by tylerpotts

Dec 11, 2020

@jkellndorfer The first step is to get those pvcs to delete. We'll worry about the pods later (they'll never work while the pvc is still in a terminating state)

I'd try manually deleting the PVC volumes for conda store (.... NOT SHARED VOLUME...) in the AWS console, and then following the same steps for deleting the pvc in kubernetes.

I found some other solutions in stack overflow that all involve a similar patching of that "finalizer" metadata. https://stackoverflow.com/questions/51358856/kubernetes-cant-delete-persistentvolumeclaim-pvc

I'm deploying a new qhub tonight, so I'll try to replicate the problem and see if I can get some more concrete solutions

View full answer

jkellndorfer · 2020-12-10T22:18:04Z

jkellndorfer
Dec 10, 2020

Here is what I get:

(seppo) Josefs-MBP:infrastructure josefk$  kubectl patch pvc conda-store-dev-share -p '{"metadata":{"finalizers":null}}' -n dev
persistentvolumeclaim/conda-store-dev-share patched
(seppo) Josefs-MBP:infrastructure josefk$  kubectl patch pod conda-store-conda-store-59cccb446-xl6cd -p '{"metadata":{"finalizers":null}}' -n dev
pod/conda-store-conda-store-59cccb446-xl6cd patched (no change)

However, still can't seem to delete the conda store PVC:

(seppo) Josefs-MBP:infrastructure josefk$   kubectl delete pv conda-store-dev-share
persistentvolume "conda-store-dev-share" deleted

hangs and I have to CRTL-C it. And the pvc is still terminating:

(seppo) Josefs-MBP:infrastructure josefk$ kubectl get pv --all-namespaces
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                                 STORAGECLASS            REASON   AGE
conda-store-dev-share                      50Gi       RWX            Delete           Terminating   dev/conda-store-dev-share             conda-store-dev-share            15d
nfs-mount-dev-share                        100Gi      RWX            Retain           Bound         dev/nfs-mount-dev-share               nfs-mount-dev-share              27d
pvc-1f62f788-8a35-4860-9e87-113c9f1ecc99   1Gi        RWO            Delete           Bound         dev/hub-db-dir                        gp2                              27d
pvc-29798cc8-35c0-4dee-8756-334d94e3f782   50Gi       RWO            Delete           Terminating   dev/conda-store-conda-store-storage   gp2                              15d

2 replies

jkellndorfer Dec 10, 2020

I am showing my pods again, in case that helps to trouble shoot:

(seppo) Josefs-MBP:infrastructure josefk$  kubectl get pod --all-namespaces
NAMESPACE     NAME                                                         READY   STATUS    RESTARTS   AGE
dev           cluster-autoscaler-aws-cluster-autoscaler-6597bf744f-gbrpp   1/1     Running   0          112m
dev           conda-store-conda-store-59cccb446-xl6cd                      0/2     Pending   0          137m
dev           continuous-image-puller-nzd7j                                1/1     Running   0          8h
dev           gateway-dask-gateway-7549d87c9c-d9677                        1/1     Running   0          70m
dev           hub-684964cf8c-t5nb2                                         1/1     Running   2          5h39m
dev           ingress-cert-manager-5b66678c67-tmh4n                        1/1     Running   0          28d
dev           ingress-cert-manager-cainjector-7f66cbff8f-tcsxx             1/1     Running   1          28d
dev           ingress-cert-manager-webhook-66fcf5dfc8-xrjhn                1/1     Running   0          28d
dev           ingress-nginx-ingress-controller-665cd75d5b-mqzns            1/1     Running   0          28d
dev           ingress-nginx-ingress-default-backend-86bf989c9c-dfnhv       1/1     Running   0          28d
dev           proxy-5fd7f647b8-4hgl6                                       1/1     Running   0          27d
dev           scheduler-proxy-dask-gateway-68bdd99bdf-ddv5r                1/1     Running   0          5h39m
dev           user-scheduler-7c4d995596-fmw6l                              1/1     Running   1          27d
dev           user-scheduler-7c4d995596-k6bcc                              1/1     Running   2          27d
dev           web-proxy-dask-gateway-85577cf5f-clgt6                       1/1     Running   0          5h39m
kube-system   aws-node-8rv6l                                               1/1     Running   0          28d
kube-system   aws-node-bjsnn                                               1/1     Running   0          8h
kube-system   aws-node-hb7jc                                               1/1     Running   0          6h48m
kube-system   coredns-75b44cb5b4-q95qb                                     1/1     Running   0          16d
kube-system   coredns-75b44cb5b4-qbjr8                                     1/1     Running   0          5h39m
kube-system   kube-proxy-5fpwm                                             1/1     Running   0          6h48m
kube-system   kube-proxy-cqtd9                                             1/1     Running   0          28d
kube-system   kube-proxy-qmtw6                                             1/1     Running   0          8h

jkellndorfer Dec 11, 2020

@tylerpotts @rsignell-usgs
I had a couple more attempts to this, to no avail. This is where I am at:

(seppo) Josefs-MBP:infrastructure josefk$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                                         READY   STATUS    RESTARTS   AGE
dev           cluster-autoscaler-aws-cluster-autoscaler-6597bf744f-gbrpp   1/1     Running   0          6h17m
dev           conda-store-conda-store-59cccb446-dd966                      0/2     Pending   0          42m
dev           continuous-image-puller-gcjn7                                1/1     Running   0          45m
dev           gateway-dask-gateway-7549d87c9c-8ns9l                        1/1     Running   0          16m
dev           hub-684964cf8c-wxj2t                                         1/1     Running   0          46m
dev           ingress-cert-manager-5b66678c67-tmh4n                        1/1     Running   0          28d
dev           ingress-cert-manager-cainjector-7f66cbff8f-tcsxx             1/1     Running   1          28d
dev           ingress-cert-manager-webhook-66fcf5dfc8-xrjhn                1/1     Running   0          28d
dev           ingress-nginx-ingress-controller-665cd75d5b-mqzns            1/1     Running   0          28d
dev           ingress-nginx-ingress-default-backend-86bf989c9c-dfnhv       1/1     Running   0          28d
dev           proxy-5fd7f647b8-4hgl6                                       1/1     Running   0          27d
dev           scheduler-proxy-dask-gateway-68bdd99bdf-x2xrn                1/1     Running   0          48m
dev           user-scheduler-7c4d995596-fmw6l                              1/1     Running   1          27d
dev           user-scheduler-7c4d995596-k6bcc                              1/1     Running   2          27d
dev           web-proxy-dask-gateway-85577cf5f-g6vxz                       1/1     Running   0          46m
kube-system   aws-node-8rv6l                                               1/1     Running   0          28d
kube-system   aws-node-bjsnn                                               1/1     Running   0          12h
kube-system   aws-node-hb7jc                                               1/1     Running   0          11h
kube-system   coredns-75b44cb5b4-q95qb                                     1/1     Running   0          16d
kube-system   coredns-75b44cb5b4-qbjr8                                     1/1     Running   0          10h
kube-system   kube-proxy-5fpwm                                             1/1     Running   0          11h
kube-system   kube-proxy-cqtd9                                             1/1     Running   0          28d
kube-system   kube-proxy-qmtw6                                             1/1     Running   0          12h
(seppo) Josefs-MBP:infrastructure josefk$ kubectl get pv --all-namespaces
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                                 STORAGECLASS          REASON   AGE
nfs-mount-dev-share                        100Gi      RWX            Retain           Bound         dev/nfs-mount-dev-share               nfs-mount-dev-share            27d
pvc-1f62f788-8a35-4860-9e87-113c9f1ecc99   1Gi        RWO            Delete           Bound         dev/hub-db-dir                        gp2                            27d
pvc-29798cc8-35c0-4dee-8756-334d94e3f782   50Gi       RWO            Delete           Terminating   dev/conda-store-conda-store-storage   gp2                            15d
(seppo) Josefs-MBP:infrastructure josefk$ kubectl get volumeattachment -n dev
No resources found in dev namespace.
(seppo) Josefs-MBP:infrastructure josefk$ kubectl delete pv pvc-29798cc8-35c0-4dee-8756-334d94e3f782 --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
persistentvolume "pvc-29798cc8-35c0-4dee-8756-334d94e3f782" force deleted
^C
(seppo) Josefs-MBP:infrastructure josefk$

As you can see from kubectl get volumeattachment -n dev, no more volumes are attached, which I also confirmed in the AWS console. The volume still shows there, but is "available". A terraform apply triggered with GITHUB ACTIONS FAILS.

I tried several times to delete the conda-store pod, only to see it come back with a different ID immediately.

Do you have any more tricks up your sleeves? Why do you think this is even happening?

tylerpotts · 2020-12-11T04:17:18Z

tylerpotts
Dec 11, 2020
Author

@jkellndorfer The first step is to get those pvcs to delete. We'll worry about the pods later (they'll never work while the pvc is still in a terminating state)

I'd try manually deleting the PVC volumes for conda store (.... NOT SHARED VOLUME...) in the AWS console, and then following the same steps for deleting the pvc in kubernetes.

I found some other solutions in stack overflow that all involve a similar patching of that "finalizer" metadata. https://stackoverflow.com/questions/51358856/kubernetes-cant-delete-persistentvolumeclaim-pvc

I'm deploying a new qhub tonight, so I'll try to replicate the problem and see if I can get some more concrete solutions

0 replies

tylerpotts · 2020-12-11T04:21:35Z

tylerpotts
Dec 11, 2020
Author

@jkellndorfer Could you also list your persistent volume claims with k get pvc -n dev and paste them here? It looks like you changed the persistent volume metadata but then deleted the "persistent volume", not the "persistent volume claim"

To delete the claim, run kubectl delete pvc <PVC_NAME>, and then run kubectl delete pv conda-store-dev-share

6 replies

tylerpotts Dec 11, 2020
Author

@jkellndorfer I'd suggest pulling the latest master branch locally, and then from the repo directory running:

cd infrastructure
terraform apply --auto-approve

This will manually have terraform check the state of your infrastructure and re-create any missing pieces. It's possible that the change you pushed in github actions didn't trigger terraform to apply any changes.

If this doesn't work I'll ping some more senior devs to give some suggestions

jkellndorfer Dec 11, 2020

Tried that and am running into the following error, which might be related to me using a new laptop that was not used for the original deploy. Any thoughts on how ti fix that?


Error: services "conda-store-conda-store" is forbidden: User "system:anonymous" cannot get resource "services" in API group "" in the namespace "dev"



Error: configmaps "conda-environments" is forbidden: User "system:anonymous" cannot get resource "configmaps" in API group "" in the namespace "dev"



Error: persistentvolumeclaims "conda-store-conda-store-storage" is forbidden: User "system:anonymous" cannot get resource "persistentvolumeclaims" in API group "" in the namespace "dev"



Error: storageclasses.storage.k8s.io "conda-store-dev-share" is forbidden: User "system:anonymous" cannot get resource "storageclasses" in API group "storage.k8s.io" at the cluster scope



Error: namespaces "dev" is forbidden: User "system:anonymous" cannot get resource "namespaces" in API group "" in the namespace "dev"



Error: query: failed to query with labels: secrets is forbidden: User "system:anonymous" cannot list resource "secrets" in API group "" in the namespace "dev"

tylerpotts Dec 11, 2020
Author

With AWS you have to explicitly allow new IAM users access to the system for kubectl. Here's some docs on your ARN: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html

You can add your arn by running: kubectl edit -n kube-system configmap/aws-auth and adding your current IAM ARN that is attached to the public/private AWS keypair you are currently using. You might have to do this with the original key pair that was used to create the cluster. Here's the docs for editing the config map for kubectl: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

If all of that fails, I'd suggest deploying a new cluster and restoring all of your user data to it. If you haven't got it fixed over the weekend I'll spend some time writing up docs explaining the backup option.

jkellndorfer Dec 14, 2020

@tylerpotts Thanks for all the suggestions. I tried the first step to authorize, but the errors remain, i.e. I probably am missing some steps in the authorization process. Before I start to become a kubectl master, it's probably best to redeploy. So, your approach on backing up the EFS share and redeploying the data would be good to have. Ideally, it would be interesting to just reuse the existing EFS share, but I guess with a terraform destroy it will vanish, right?

tylerpotts Jan 19, 2021
Author

@jkellndorfer Sorry this got lost in the ether.

Yes, terraform destroy will annihilate the PVC volume unless otherwise backed up. We have an issue to track the backup plan in #99. I also posted some thoughts on how to redeploy a cluster with the same data.

Let me know if you have any other questions!

Adam-D-Lewis · 2024-09-17T17:07:00Z

Adam-D-Lewis
Sep 17, 2024
Maintainer

I'm closing old discussions in a repo cleanup effort. Feel free to re-open if needed.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nebari-dev

PV not deleting #202

{{title}}

Replies: 4 comments 8 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

nebari-dev

PV not deleting #202

tylerpotts Dec 10, 2020

Replies: 4 comments · 8 replies

jkellndorfer Dec 10, 2020

jkellndorfer Dec 10, 2020

jkellndorfer Dec 11, 2020

tylerpotts Dec 11, 2020 Author

tylerpotts Dec 11, 2020 Author

tylerpotts Dec 11, 2020 Author

jkellndorfer Dec 11, 2020

tylerpotts Dec 11, 2020 Author

jkellndorfer Dec 14, 2020

tylerpotts Jan 19, 2021 Author

Adam-D-Lewis Sep 17, 2024 Maintainer

tylerpotts
Dec 10, 2020

Replies: 4 comments 8 replies

jkellndorfer
Dec 10, 2020

tylerpotts
Dec 11, 2020
Author

tylerpotts
Dec 11, 2020
Author

tylerpotts Dec 11, 2020
Author

tylerpotts Dec 11, 2020
Author

tylerpotts Jan 19, 2021
Author

Adam-D-Lewis
Sep 17, 2024
Maintainer