Pods not deleted after its Rancher App is deleted #2048
Comments
|
Are you perhaps deleting the deployment manually instead of deleting the helm chart custom resource that the deployment comes from? Can you provide any more information on the actual charts/deployments/pods that exist on your cluster before and after you try to delete things? Showing what resources exist, not just the commands you're running to delete them, would be helpful. |
|
Thanks for replying.
Here an example of the workaround in action, after instigating the deletion of the Helm Chart of the Rancher UI, to delete the orphan objects: This is what I need to do after supposedly deleting the Helm Chart from the "Apps" section in the Rancher UI. If I don't do that, they run invisibly to Rancher, forever. |
I still don't have any info to show what state things are in before you force-delete them. Is the helm chart actually gone? Are other resources deleting but stuck on a finalizer? |
|
The objects in the given namespace, after deleting the corresponding Helm Chart in the Rancher UI: As you can see, they are running invisibly, as already described in the previous post. |
|
Can you also |
|
|
|
Okay, so can you show the full replicaset output ( |
|
Taken from one ReplicaSet I picked, I assume you are seeking for this: ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: Deployment
name: app-proxy
uid: d0e36770-8c2c-46d8-abfc-cb53b390ea94
resourceVersion: "5724742"
selfLink: /apis/apps/v1/namespaces/app-230-rc15-1/replicasets/app-proxy-6666ddb967
uid: 9f6c74c7-6c5a-468a-ab40-c2e8b6c6cb83Is |
|
Right, so with that ownerReference on there, with blockOwnerDeletion, it should not be possible delete the parent Deployment until the child ReplicaSet has been deleted. The child Pods will have similar ownerReferences back to the ReplicaSet, which is supposed to be how everything gets cleaned up automatically when you delete the Deployment. This is all just core Kubernetes functionality, nothing unique to Helm Charts or k3s. Remind me how are you removing the deployments? Are you setting |
Then why do the exact same Charts get deleted properly on RKE clusters? Why is the behaviour different? There has to be a difference or uniqueness to the matter or else the deletion would behave the same, too.
Is this part of the core Kubernetes functionality? Why is this set? See above question on why the behaviour is different with RKE.
All I'm doing is to delete the Helm Chart from the Apps section, when being in a cluster's project, in the Rancher UI. Which is also the same place where the Charts are launched from. The CLI is only used, if necessary. Doing this on RKE works, without resulting orphaned objects. Reading through your comment again, I am not sure, if I interpreted it correctly. I thought, you were saying that the deletion not being successful here is just normal Kubernetes behaviour. However, perhaps you just explained each part of the core functionality without deeming it "normal" or "expected" in such a situation. |
|
Yes, I was saying that the cascading deletion via ownerReference is core Kubernetes behavior. Which version of Rancher are you using? Can you replicate this same behavior on k3s 1.18.6? |
Independent of this issue, the cluster was upgraded to Testing setup and removal of After deletion: After first iteration of the workaround: After second iteration of the workaround: |
|
Hey, just out of curiosity, can you try something?
This should force the kube-controller-manager to run on another node; I'm curious if something is going on with the current node that's causing it to not garbage collect properly. |
|
Would gladly try it out, but this cluster is mainly intended for testing newly developed stuff, so it only has 1 node, as more aren't necessary. |
|
I believe I'm seeing the same behavior. I'm on a single master, 6 agent cluster running I've got a helm chart for an app that I've been deploying via helm 3.X for a few months now and helm upgrades and deletes have worked as expected but they are not now. I think this coincided with my upgrade to 1.18.6. My Continuous Integration builds/tests run I've seen issues with both the upgrades and deletes not removing the old replicasets. The replicaset and pod ownerReferences look correct to me. After an upgrade where I've got the Any suggestions on what additional information or logs I could provide to help diagnose the issue? I tried the k3s service restart mentioned above but since I'm single-master and the controller manager was running on that master, it didn't get failed over to a different node. on the replicaset: ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: Deployment
name: my-web-thing
uid: 0ccba6b8-b66e-4b84-bf93-b281b5593c94on the pod: ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: my-web-thing-6bb9f99cfb
uid: 3d0dfca2-d062-406c-86ed-4074af1c5849kubectl get rs |
|
@jonstelly the deployment referred to by the ReplicaSet is gone? |
|
For Deletes: No, the deployment gets deleted, so ReplicaSets and Pods are remaining |
|
Without the full info on the pods its hard to tell what's supposed to belong to what. We'd want to see things hanging around with an ownerReferences entry pointing at an object that doesn't exist. |
|
An example of lingering pod+replicaset after a I just noticed that helm 3.3 was released yesterday and that's what my CI process is picking up (latest stable) but I'm running 3.2.4 locally. I'll upgrade to 3.3 just to check that out, This issue doesn't happen for me in my Azure AKS environments (running k8s 1.17 instead of 1.18) Command being run in my Azure Devops Pipeline is: and the output is simply: The Deployment that the ReplicaSet Owner refers to does not exist. ReplicaSet kind: ReplicaSet
apiVersion: apps/v1
metadata:
name: foo-alpha-acceptance-0812-203256-web-57b4bccd
namespace: default
selfLink: >-
/apis/apps/v1/namespaces/default/replicasets/foo-alpha-acceptance-0812-203256-web-57b4bccd
uid: f88b7ef6-307a-4116-91c5-d3859d17f8d7
resourceVersion: '28861001'
generation: 1
creationTimestamp: '2020-08-12T20:33:45Z'
labels:
app.kubernetes.io/instance: foo-alpha-acceptance-0812-203256
app.kubernetes.io/name: foo-web
pod-template-hash: 57b4bccd
annotations:
deployment.kubernetes.io/desired-replicas: '1'
deployment.kubernetes.io/max-replicas: '2'
deployment.kubernetes.io/revision: '1'
meta.helm.sh/release-name: foo-alpha-acceptance-0812-203256
meta.helm.sh/release-namespace: default
ownerReferences:
- apiVersion: apps/v1
kind: Deployment
name: foo-alpha-acceptance-0812-203256-web
uid: 77e2b9b2-dcf6-44ce-9ce4-e35365129772
controller: true
blockOwnerDeletion: true
managedFields:
- manager: k3s
operation: Update
apiVersion: apps/v1
time: '2020-08-12T20:34:32Z'
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:annotations':
.: {}
'f:deployment.kubernetes.io/desired-replicas': {}
'f:deployment.kubernetes.io/max-replicas': {}
'f:deployment.kubernetes.io/revision': {}
'f:meta.helm.sh/release-name': {}
'f:meta.helm.sh/release-namespace': {}
'f:labels':
.: {}
'f:app.kubernetes.io/instance': {}
'f:app.kubernetes.io/name': {}
'f:pod-template-hash': {}
'f:ownerReferences':
.: {}
'k:{"uid":"77e2b9b2-dcf6-44ce-9ce4-e35365129772"}':
.: {}
'f:apiVersion': {}
'f:blockOwnerDeletion': {}
'f:controller': {}
'f:kind': {}
'f:name': {}
'f:uid': {}
'f:spec':
'f:replicas': {}
'f:selector':
'f:matchLabels':
.: {}
'f:app.kubernetes.io/instance': {}
'f:app.kubernetes.io/name': {}
'f:pod-template-hash': {}
'f:template':
'f:metadata':
'f:labels':
.: {}
'f:app.kubernetes.io/instance': {}
'f:app.kubernetes.io/name': {}
'f:pod-template-hash': {}
'f:spec':
'f:containers':
'k:{"name":"foo-web"}':
.: {}
'f:env':
.: {}
'k:{"name":"DOTNET_ENVIRONMENT"}':
.: {}
'f:name': {}
'f:value': {}
'k:{"name":"COMPANY__INSTANCE"}':
.: {}
'f:name': {}
'f:value': {}
'f:image': {}
'f:imagePullPolicy': {}
'f:livenessProbe':
.: {}
'f:failureThreshold': {}
'f:httpGet':
.: {}
'f:path': {}
'f:port': {}
'f:scheme': {}
'f:periodSeconds': {}
'f:successThreshold': {}
'f:timeoutSeconds': {}
'f:name': {}
'f:ports':
.: {}
'k:{"containerPort":80,"protocol":"TCP"}':
.: {}
'f:containerPort': {}
'f:name': {}
'f:protocol': {}
'f:readinessProbe':
.: {}
'f:failureThreshold': {}
'f:httpGet':
.: {}
'f:path': {}
'f:port': {}
'f:scheme': {}
'f:periodSeconds': {}
'f:successThreshold': {}
'f:timeoutSeconds': {}
'f:resources': {}
'f:terminationMessagePath': {}
'f:terminationMessagePolicy': {}
'f:volumeMounts':
.: {}
'k:{"mountPath":"/root/.docker"}':
.: {}
'f:mountPath': {}
'f:name': {}
'f:dnsPolicy': {}
'f:restartPolicy': {}
'f:schedulerName': {}
'f:securityContext': {}
'f:serviceAccount': {}
'f:serviceAccountName': {}
'f:terminationGracePeriodSeconds': {}
'f:tolerations': {}
'f:topologySpreadConstraints':
.: {}
'k:{"topologyKey":"node","whenUnsatisfiable":"DoNotSchedule"}':
.: {}
'f:maxSkew': {}
'f:topologyKey': {}
'f:whenUnsatisfiable': {}
'f:volumes': {}
'f:status':
'f:availableReplicas': {}
'f:fullyLabeledReplicas': {}
'f:observedGeneration': {}
'f:readyReplicas': {}
'f:replicas': {}
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/instance: foo-alpha-acceptance-0812-203256
app.kubernetes.io/name: foo-web
pod-template-hash: 57b4bccd
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/instance: foo-alpha-acceptance-0812-203256
app.kubernetes.io/name: foo-web
pod-template-hash: 57b4bccd
spec:
volumes:
containers:
- name: foo-web
image: 'hub.foo.us/foo-web:0.7.10-alpha.921'
ports:
- name: http
containerPort: 80
protocol: TCP
env:
- name: COMPANY__INSTANCE
value: alpha-acceptance-0812-203256
- name: DOTNET_ENVIRONMENT
value: alpha
resources: {}
volumeMounts:
livenessProbe:
httpGet:
path: /
port: http
scheme: HTTP
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /
port: http
scheme: HTTP
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: foo-alpha-acceptance-0812-203256
serviceAccount: foo-alpha-acceptance-0812-203256
securityContext: {}
schedulerName: default-scheduler
tolerations:
- key: company/ephemeral
operator: Exists
topologySpreadConstraints:
- maxSkew: 1
topologyKey: node
whenUnsatisfiable: DoNotSchedule
status:
replicas: 1
fullyLabeledReplicas: 1
readyReplicas: 1
availableReplicas: 1
observedGeneration: 1Pod kind: Pod
apiVersion: v1
metadata:
name: foo-alpha-acceptance-0812-203256-web-57b4bccd-wzljt
generateName: foo-alpha-acceptance-0812-203256-web-57b4bccd-
namespace: default
selfLink: >-
/api/v1/namespaces/default/pods/foo-alpha-acceptance-0812-203256-web-57b4bccd-wzljt
uid: 889408bd-3e3b-4549-9eb7-e064b2d74dbf
resourceVersion: '28860998'
creationTimestamp: '2020-08-12T20:33:45Z'
labels:
app.kubernetes.io/instance: foo-alpha-acceptance-0812-203256
app.kubernetes.io/name: foo-web
pod-template-hash: 57b4bccd
ownerReferences:
- apiVersion: apps/v1
kind: ReplicaSet
name: foo-alpha-acceptance-0812-203256-web-57b4bccd
uid: f88b7ef6-307a-4116-91c5-d3859d17f8d7
controller: true
blockOwnerDeletion: true
managedFields:
- manager: k3s
operation: Update
apiVersion: v1
time: '2020-08-12T20:34:32Z'
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:generateName': {}
'f:labels':
.: {}
'f:app.kubernetes.io/instance': {}
'f:app.kubernetes.io/name': {}
'f:pod-template-hash': {}
'f:ownerReferences':
.: {}
'k:{"uid":"f88b7ef6-307a-4116-91c5-d3859d17f8d7"}':
.: {}
'f:apiVersion': {}
'f:blockOwnerDeletion': {}
'f:controller': {}
'f:kind': {}
'f:name': {}
'f:uid': {}
'f:spec':
'f:containers':
'k:{"name":"foo-web"}':
.: {}
'f:env':
.: {}
'k:{"name":"DOTNET_ENVIRONMENT"}':
.: {}
'f:name': {}
'f:value': {}
'k:{"name":"COMPANY__INSTANCE"}':
.: {}
'f:name': {}
'f:value': {}
'f:image': {}
'f:imagePullPolicy': {}
'f:livenessProbe':
.: {}
'f:failureThreshold': {}
'f:httpGet':
.: {}
'f:path': {}
'f:port': {}
'f:scheme': {}
'f:periodSeconds': {}
'f:successThreshold': {}
'f:timeoutSeconds': {}
'f:name': {}
'f:ports':
.: {}
'k:{"containerPort":80,"protocol":"TCP"}':
.: {}
'f:containerPort': {}
'f:name': {}
'f:protocol': {}
'f:readinessProbe':
.: {}
'f:failureThreshold': {}
'f:httpGet':
.: {}
'f:path': {}
'f:port': {}
'f:scheme': {}
'f:periodSeconds': {}
'f:successThreshold': {}
'f:timeoutSeconds': {}
'f:resources': {}
'f:terminationMessagePath': {}
'f:terminationMessagePolicy': {}
'f:volumeMounts':
.: {}
'k:{"mountPath":"/root/.docker"}':
.: {}
'f:mountPath': {}
'f:name': {}
'f:dnsPolicy': {}
'f:enableServiceLinks': {}
'f:restartPolicy': {}
'f:schedulerName': {}
'f:securityContext': {}
'f:serviceAccount': {}
'f:serviceAccountName': {}
'f:terminationGracePeriodSeconds': {}
'f:tolerations': {}
'f:topologySpreadConstraints':
.: {}
'k:{"topologyKey":"node","whenUnsatisfiable":"DoNotSchedule"}':
.: {}
'f:maxSkew': {}
'f:topologyKey': {}
'f:whenUnsatisfiable': {}
'f:volumes': {}
'f:status':
'f:conditions':
'k:{"type":"ContainersReady"}':
.: {}
'f:lastProbeTime': {}
'f:lastTransitionTime': {}
'f:status': {}
'f:type': {}
'k:{"type":"Initialized"}':
.: {}
'f:lastProbeTime': {}
'f:lastTransitionTime': {}
'f:status': {}
'f:type': {}
'k:{"type":"Ready"}':
.: {}
'f:lastProbeTime': {}
'f:lastTransitionTime': {}
'f:status': {}
'f:type': {}
'f:containerStatuses': {}
'f:hostIP': {}
'f:phase': {}
'f:podIP': {}
'f:podIPs':
.: {}
'k:{"ip":"10.42.2.174"}':
.: {}
'f:ip': {}
'f:startTime': {}
spec:
volumes:
containers:
- name: foo-web
image: 'hub.foo.us/foo-web:0.7.10-alpha.921'
ports:
- name: http
containerPort: 80
protocol: TCP
env:
- name: COMPANY__INSTANCE
value: alpha-acceptance-0812-203256
- name: DOTNET_ENVIRONMENT
value: alpha
resources: {}
volumeMounts:
livenessProbe:
httpGet:
path: /
port: http
scheme: HTTP
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /
port: http
scheme: HTTP
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: foo-alpha-acceptance-0812-203256
serviceAccount: foo-alpha-acceptance-0812-203256
nodeName: pi4b
securityContext: {}
schedulerName: default-scheduler
tolerations:
- key: company/ephemeral
operator: Exists
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 300
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 300
priority: 0
enableServiceLinks: true
topologySpreadConstraints:
- maxSkew: 1
topologyKey: node
whenUnsatisfiable: DoNotSchedule
status:
phase: Running
conditions:
- type: Initialized
status: 'True'
lastProbeTime: null
lastTransitionTime: '2020-08-12T20:33:45Z'
- type: Ready
status: 'True'
lastProbeTime: null
lastTransitionTime: '2020-08-12T20:34:32Z'
- type: ContainersReady
status: 'True'
lastProbeTime: null
lastTransitionTime: '2020-08-12T20:34:32Z'
- type: PodScheduled
status: 'True'
lastProbeTime: null
lastTransitionTime: '2020-08-12T20:33:45Z'
hostIP: 192.168.8.21
podIP: 10.42.2.174
podIPs:
- ip: 10.42.2.174
startTime: '2020-08-12T20:33:45Z'
containerStatuses:
- name: foo-web
state:
running:
startedAt: '2020-08-12T20:34:12Z'
lastState: {}
ready: true
restartCount: 0
image: 'hub.foo.us/foo-web:0.7.10-alpha.921'
imageID: >-
hub.foo.us/foo-web@sha256:086c977aa8149abeff094e59bb8af3ae6ae1f0ed8d15c3de5c382c579b82cf60
containerID: >-
containerd://c35e5bda27c7914e21317991081cbbc80241246f567ed948ffc0f24502158fb3
started: true
qosClass: BestEffort |
|
Below is the output of Tomorrow I'll try and create an isolated repro for this issue using k3d. At this point it could be a bug in the upstream kubernetes code, something in k3s and the patch-sets, something in helm, or something strange about my chart. I'll report back with findings from the repro with whatever they show. |
|
Hmm iirc purge will force deletion, even if there are things like dangling owner reference that should block it. I wonder why it's doing that. |
|
Hi, I was faced with a relevant issue. Out-of-box my k3s operate normally but after dozen deployments/deletions something went wrong. All resources was deleted exclude replicaset. Pod recreating and recreating until I execute |
|
Given all this information, this issue should have some |
Yeah, the purge seems to be to clean up the helm releases and history on uninstall like the issue you linked, but it doesn't appear to be the same thing as I've been testing with the following: Commands
$iterations=3;
1..$iterations | % {
$time = [DateTime]::Now.ToString("MMdd-HHmmss");
helm install "app-$time" ./orphan-testing --wait;
helm upgrade "app-$time" ./orphan-testing --set replicaCount=2 --wait;
helm delete "app-$time";
}Findings
An additional interesting note: If I delete the orphaned ReplicaSet, that doesn't clean up the Pods either. I have to manually delete both the ReplicaSets and the Pods. And if I delete the Pods before the ReplicaSets, the pods get recreated. But from this the problem seems to be something cluster-specific. I'm going to try rebooting all nodes on my cluster, then running the above test and if that doesn't fix it, I'll do OS updates and trying again. I'll dig into the k3s logs too. To capture node version/kernel info before I change anything, here are the nodes from my cluster: EDIT - After Reboot of master, no improvement, same behavior with lingering RS and Pods. No updates for Ubuntu, I'm current there. Findings from my cluster are below. This is the state after running the above. It's worth noting that the upgrade to reduce the replicaCount worked: ReplicaSets Pods Here are the events from my existing cluster: EDITAnd here's the event log from a successful cleanup on my k3d instance for comparison. It seems interesting that the above log reports a |
|
If some contributor wants to see that behavior, I can provide access to the affected cluster for some experiments. |
|
I can't give direct access but am happy to run any commands or one-off builds to diagnose. This cluster has also been through a few upgrades. I had to do a cluster rebuild at some point but I think this current cluster started off as a 1.17 k3s release that got updated to 1.18. |
|
I'm running latest freshly installed K3OS 0.11.0 single node. After installing a chart, I deleted the HelmChart resource - it didn't remove the pods. Currently I have another HelmChart stuck that I am not able to delete - not with --now not with grace and force. |
|
#2074 may be related. Linking the two. |
|
I haven't been able to reproduce this with newly created clusters or upgraded clusters from v1.17.x to v1.18.x. I am using rancher v2.5.1. Has anyone else who has seen this issue been able to recreate it on a fresh k3s setup? Either when doing steps through Rancher UI or directly through Helm? If so, please provide your steps and configs used. It looks like this only happens on older setups, which leads me to believe something else has caused the cluster to degrade and this issue probably isn't strictly for rancher apps but would happen for other deployments and replicasets, possibly when created via helm. Most recently I attempted the following steps:
All resources were deleted successfully. See commands below. |
|
I'd like to be able to reproduce this and get the issue fixed. With that said... @cubic3d @theAkito @jonstelly Do any of you know a reliable way to reproduce this on a fresh setup? |
|
Hi @cubic3d @theAkito @jonstelly as Max mentioned we're having a hard time reproducing this. Might we be missing something? Max has tried hard to reproduce but we're not quite sure what else to try. Any help you might be able to offer would be very much appreciated. We don't want to just close this out as unable to reproduce if we can help it. |
|
Hey you two, thank you for your work and help. Unfortunately I don't have access to the environment anymore. I haven't been able to reproduce it myself, it appeared randomly. |
|
I have a few notes. |
The certbot chart has some nested resources that had problems with previous versions of helm and kubectl which led to freezes of the apply and delete actions. Can't look for the issue now, but sounds that may be the cause for undeleted resources after crds are initially nstalled. |
|
My cluster seems to have blown up...
Not sure if I'll be able to get it back up and running or need to rebuild. Will let you know if I get it back up and running or if I see the issue after recreating. |
|
@jonstelly That error looks like #1403 -- do you think that could be the case for you here? |
|
Yeah, that looks like it. I'll rebuild the cluster and report back here if I see the same pod lingering behavior. Might take me a couple days to get it built back up to that point. |
|
We will stand by a while longer for input such as from @jonstelly |
|
Hi. As per my prior comment, it seems like we are having a tough time reproducing this now especially with newer versions. For any comments indicating issues not directly related to what this issue captured in it's summary, please open separate GitHub issues. I'd like to close this out. If there are any serious objections please comment. We'll need to then work together to try to reproduce this and identify a root cause. |
|
Since custom resources are involved, this may be related: kubernetes/kubernetes#92743 |
|
I've had my cluster up for a couple weeks now and haven't seen the issue in the new cluster. Thanks for the work trying to track this one down. |
Environmental Info:
K3s Version:
k3s version v1.18.2+k3s1 (698e444a)Node(s) CPU architecture, OS, and Version:
4.15.0-101-generic Ubuntu x86_64Cluster Configuration:
1 master imported within Rancher.
Describe the bug:
Currently, I have a Rancher deployment with 2 clusters. 1 being a generic RKE cluster and the other one being a k3s one.
When deleting a Rancher App on the first one, all goes fine with deploying and deleting Rancher Apps.
However, when deploying a Rancher App and deleting it afterwards on the k3s cluster, the App's pods don't get deleted and still run under the radar, not detected by Rancher. Even worse, they obviously still suck up all the resources that I attempted to gain when deleting the App. So of course the cluster fills up over time and cannot accept new deployments, because all other ones are still running behind the scenes.
Original Bug Description
I just discovered that this issue is much worse than originally assumed.
To delete a pod from a deployment, you have to delete the deployment, or else the pod resurrects, automatically. However, in this situation there is no deployment to be deleted. The pods are still left. If you now try to delete the pods, they revive, again. So there is no obvious way to delete the pods, since you'd need to delete their deployments which are already gone, though.
For testing purposes, I set up a chart and deleted its deployments with
The pods remained and hat to be deleted with below workaround.
Steps To Reproduce:
kubectl get pods --all-namespacesRunningstate and remain like that.Expected behavior:
Pods of the deleted Rancher App also get deleted.
Actual behavior:
Pods of the deleted Rancher App do not get deleted.
Additional context / logs:
Cluster was imported quite a while ago. This issue wasn't discovered too early, because there are more than enough resources on the server and it is not used that frequently.
Workaround
Currently, the only way to delete the orphaned pods is this:
This command has to be run twice per namespace.
The text was updated successfully, but these errors were encountered: