Failing to drain/cordon causes CrashLoopBackOff

[liorfranko]

Describe the bug
When running in IMDS mode and aws-node-termination-handler can't drain a node due to PDB the pod crash
Why when running in IMDS mode the pod exit?
https://github.com/aws/aws-node-termination-handler/blob/main/cmd/node-termination-handler.go#L401

Application Logs
2021/12/07 08:02:20 ERR There was a problem while trying to cordon and drain the node error="[error when evicting pods/\"audience-consumer-64cb9bc5d6-h866g\" -n \"delivery-apps\": global timeout reached: 2m0s, error when evicting pods/\"audience-consumer-64cb9bc5d6-x9nq4\" -n \"delivery-apps\": global timeout reached: 2m0s, error when evicting pods/\"audience-consumer-64cb9bc5d6-999ft\" -n \"delivery-apps\": global timeout reached: 2m0s, error when evicting pods/\"events-restore-consumer-86c75c6759-l926p\" -n \"delivery-apps\": global timeout reached: 2m0s, error when evicting pods/\"audience-consumer-64cb9bc5d6-xp4ph\" -n \"delivery-apps\": global timeout reached: 2m0s, error when evicting pods/\"audience-consumer-64cb9bc5d6-ccjkr\" -n \"delivery-apps\": global timeout reached: 2m0s, error when evicting pods/\"audience-consumer-64cb9bc5d6-g79dn\" -n \"delivery-apps\": global timeout reached: 2m0s]"

Environment

• NTH App Version:
  1.14.0
• NTH Mode (IMDS/Queue processor):
  IMDS
• Kubernetes version:
  Client Version: v1.18.9-eks-d1db3c
  Server Version: v1.20.7-eks-d88609
• Installation method:
  Helm

[snay2]

I agree that we should modify this behavior. In queue processor mode, we already prevent the NTH pod from exiting, since we don't want to suspend processing for the whole cluster just because one node encountered an issue. Especially in the PDB situation you describe, it would be better for NTH to keep retrying to drain the pods, since it may ultimately be able to satisfy the PDB as other pods from the node get rescheduled.

Here's a solution:

1. If we remove these lines from cordonAndDrainNode(), that should solve the cordon-and-drain case and let NTH retry draining the pods again until it times out or the node gets reclaimed by EC2.
2. We may also want to remove the exit statement in cordonNode() for a similar reason--if the node can't be cordoned right away, we may want to keep retrying, but at the very least we don't want NTH to die if it's running in queue processor mode when this happens.

Feel free to open the PR for this if you have the bandwidth!

[liorfranko]

Thank you @snay2
#552
I tested it and it just tried to drain the node and fail without crashing.