Fix graceful recovery tests

[kate-osborn]

Fix graceful recovery tests

Problem: Graceful recovery tests were failing in the pipeline

Solutions:

• Increase timeout when testing for passing/failing traffic after a container or pod restart. The timeout was set to 20 seconds, with a polling interval of 500ms, but the per-request timeout was 10 seconds. This means we were only retrying the request twice when a request timed out because it couldn't connect. Increasing the overall timeout to 60 seconds increased the number of retries and eliminated the errors.
• Remove the kubectl drain flag --delete-local-data since it no longer exists and replace it with --delete-emptydir-data. See: Update deprecated kubectl flag delete-local-data in kubeadm docs kubernetes/website#29058.

Fixes #2429

Testing:

• Ran locally multiple times without error
• Ran in pipeline multiple times without error

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.84%. Comparing base (0075167) to head (22c88b9).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2440   +/-   ##
=======================================
  Coverage   88.84%   88.84%           
=======================================
  Files         100      100           
  Lines        7527     7527           
  Branches       50       50           
=======================================
  Hits         6687     6687           
  Misses        784      784           
  Partials       56       56           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.