k3s rolling upgrade Job stuck in Removing state

[theAkito]

Environmental Info:
K3s Version: v1.18.6+k3s1

Node(s) CPU architecture, OS, and Version:

4.15.0-101-generic Ubuntu x86_64
Cluster Configuration:

1 master imported within Rancher.

Rancher	v2.4.5
User Interface	v2.4.28
Helm	v2.16.8-rancher1
Machine	v0.15.0-rancher43

Describe the bug:

After successfully performing a rolling upgrade from k3s version v1.18.2+k3s1 (698e444a) to v1.18.6+k3s1, the upgrade job is Waiting on object cleanup. And it is in a Removing state in the Rancher UI.

Steps To Reproduce:

• Upgrade from k3s version v1.18.2+k3s1 (698e444a) to v1.18.6+k3s1.

Expected behavior:

Upgrade job should not hang in removing state.

Actual behavior:

Upgrade job should hangs in removing state.

Additional context / logs:

All logs are fine and successful.

[rancher-max]

I did not see this happen using Rancher v2.4.5 and upgrading through the rancher ui. The upgrade pod and app do remain on the cluster after it runs. I am able to manually remove this app and subsequently the pod successfully.

@theAkito Are there any other details about your cluster that could be causing this or have you recreated in a fresh setup?

[theAkito]

@rancher-max

Thank you for the response.

The upgrade pod and app do remain on the cluster after it runs. I am able to manually remove this app and subsequently the pod successfully.

So it is normal behaviour? It was expected that it would be removed automatically on successful run. It's especially confusing, since the Removing state causes it to be highlighted in red in the Rancher UI, which is counter-intuitive.

If it is not normal behaviour, there is another issue on the same cluster that has issues removing stuff: #2048

However, I don't see how those issues are related, except that both are about the cluster having trouble removing something.

If it is normal behaviour after all, it would be probably nice if the Job would be removed automatically, to avoid confusion.

[brandond]

See: rancher/rancher#25929 (comment)

[theAkito]

@brandond

Well, it says the job was created 4 days ago. Since then, it was not deleted.

[davidnuzik]

To me it seems like #2048 is related or somewhat so. #2048 has been triaged and we plan to look into. Out of that we should see what our findings are and can revisit this issue later if the issue persists.

If anyone objects - please voice your opinions.