use rate limiting queue with delaying for pod deletion events

[zhangzujian]

• Make sure you have followed Kube-OVN Code Style.

What type of this PR

Examples of user facing changes:

• Features
• Bug fixes
• Docs
• Tests

Which issue(s) this PR fixes:

Fixes #(issue-number)

WHAT

🤖 Generated by Copilot at 4becd66

Changed the pod controller to delay pod deletion events based on the pod's termination grace period or other criteria. This is to prevent network disruption caused by premature IP address deallocation.

🤖 Generated by Copilot at 4becd66

Oh we are the Controller crew, we manage the pods and IPs
We don't want to cause a mess, we delay the deletes with ease
Heave away, heave away, heave away on the deletePodQueue
We respect the grace and state, we don't delete before it's due

HOW

🤖 Generated by Copilot at 4becd66

• Change the type of deletePodQueue field of the Controller struct to support delayed enqueuing of pod deletion events (link)
• Modify the enqueueUpdatePod function to calculate a delay based on the pod's termination grace period and deletion timestamp and use it to enqueue the pod deletion event to the deletePodQueue with the AddAfter method (link)
• Apply the same delay logic to stateful pods and virtual machine pods in the enqueueUpdatePod function by checking the pod's owner references and labels (link, link)

[github-actions]

• The code changes in controller.go and pod.go should be reviewed for potential bugs, formatting errors, and performance issues.
• In controller.go, the addition of a new rate limiting queue with delaying interface may improve performance, but it should be tested thoroughly to ensure that it does not introduce any new bugs or issues.
• In pod.go, the use of a delay variable to determine when to add a pod to the deletePodQueue may be prone to errors if the delay is not calculated correctly. This should be reviewed carefully to ensure that it works as intended.
• The use of go routines in pod.go to enqueue pods for deletion may cause performance issues if there are too many go routines running at once. This should be monitored and optimized if necessary.
• The logging statements in pod.go should be reviewed to ensure that they provide enough information to debug any issues that may arise.