Run graceful recovery tests in pipeline

[salonichf5]

Proposed changes

Problem: As a user, I want to automate graceful recovery tests

Solution: Added a job to run graceful recovery tests in pipeline

Testing: Running it as part of pipeline

Please focus on (optional): If you any specific areas where you would like reviewers to focus their attention or provide
specific feedback, add them here.

Closes #1910

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

[bjee19]

Looking in the pipeline that was run for the functional tests, looks like because of some label the test is getting skipped:

Will run 2 of 17 specs
------------------------------
[BeforeSuite] 
/home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/tests/suite/system_suite_test.go:248
  [SKIPPED] in [BeforeSuite] - /home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/tests/suite/system_suite_test.go:130 @ 05/29/24 00:22:[45](https://github.com/nginxinc/nginx-gateway-fabric/actions/runs/9278039607/job/25528584310?pr=2045#step:14:46).584
S [SKIPPED] [0.015 seconds]
[BeforeSuite] 
/home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/tests/suite/system_suite_test.go:248

  [SKIPPED] NFR tests can only run in GKE
  In [BeforeSuite] at: /home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/tests/suite/system_suite_test.go:130

[sjberman]

Be sure to name your branches appropriately so the PR gets tagged the right way. These are test changes, not a feature or enhancement.

[bjee19]

Why did the bot add the enhancement label and remove the test label 🤔 ? Is it because of the naming of the branch?

[sjberman]

@bjee19 Because the branch is named with feat instead of test.

[salonichf5]

Be sure to name your branches appropriately so the PR gets tagged the right way. These are test changes, not a feature or enhancement.

I'll ensure to update the tag before merge! @bjee19 @sjberman