[BACKPORT][v1.6.3][BUG] kubectl drain node is blocked by unexpected orphan engine processes

[github-actions]

backport #6552

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps are at: [BUG] kubectl drain node is blocked by unexpected orphan engine processes #6552 (comment).
  The test steps are at: fix: attempt to delete processes from instance-managers in unknown state longhorn-manager#3127 (comment).

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is:
  Verify that ONLY extra processes are running in the instance-manager, then directly delete its pod.

• Does the PR include the explanation for the fix or the feature?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at: fix: attempt to delete processes from instance-managers in unknown state (backport #3127) longhorn-manager#3154.

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at: test: skeleton for don't orphan processes when node not ready longhorn-tests#2081.
  The issue of automation test case implementation is at: [TEST][BUG] kubectl drain node is blocked by unexpected orphan engine processes #9375.

[chriscchien]

Verified pass on longhorn v1.6.x(longhorn/longhorn-manager@275f7ce) with test steps

• I was able to reproduce the issue of an extra engine running in instance-manager CR in v1.6.2 with test steps, however, in v1.6.x with the same steps, did not observe extra engine process running in instance-manager CR.

> kl get instancemanager -o yaml | grep instanceEngines
    instanceEngines:
>
> kl get instancemanager -o yaml | grep instanceEngines | wc -l
1

• For the Rerun RWX fast failover tests from [FEATURE] Share manager HA - Experimental #6205 (comment) to ensure no regression mentioned in test steps, because v1.6.x did not support setting rwx-volume-fast-failover, so the step not be tested.

[roger-ryao]

Remove the require/qa-review-coverage label since we have a related PR: longhorn/longhorn-tests#2081.