fix: attempt to delete processes from instance-managers in unknown state

[ejweber]

Which issue(s) this PR fixes:

longhorn/longhorn#6552

What this PR does / why we need it:

See longhorn/longhorn#6552 (comment) for context. It may be possible to delete engine and replica processes from an instance-manager even when the instance-manager's state is unknown. Doing so prevents the processes from becoming orphans if the instance-manager eventually recovers from the unknown state.

This PR is looking good in some local testing, but I want to put it through a few more paces before review.

[derekbit]

CI failures:

controller/engine_controller.go:2430:16: S1039: unnecessary use of fmt.Sprintf (gosimple)
		return true, fmt.Sprintf("the RWX volume is delinquent")
		             ^
controller/replica_controller.go:910:16: S1039: unnecessary use of fmt.Sprintf (gosimple)
		return true, fmt.Sprintf("the RWX volume is delinquent")

[ejweber]

To test (in contrast with longhorn/longhorn#6552 (comment)):

1. Deploy Longhorn with these changes.
2. Create and attach a volume that has the same number of replicas as nodes (e.g. with the UI).
3. SSH to the node the volume is attached to.
4. Stop the kubelet.
5. Wait ~30s for Kubernetes to notice and for the instance-manager state to transition to unknown.
6. Detach the volume.
7. Verify that no related engine or replica processes exist in any instance-manager CR.
8. Check longhorn-manager logs for some of the new messages (e.g. "Communicating with instance manager <>, state unknown, IP <>").

Rerun RWX fast failover tests from longhorn/longhorn#6205 (comment) to ensure no regression.

[james-munson]

Looks good. The re-factor makes the logic clearer, too.

[ejweber]

@james-munson raised a concern about the effect this might have on RWX fast failover. I will test it a bit before merging (and modify the test plan so QA also tests it eventually).

[ejweber]

@james-munson, I ran case 1 from longhorn/longhorn#6205 (comment) a few times. The results were:

• ShareManager up in 10 seconds, workload pods writing in 52 seconds and 57 seconds respectively.
• ShareManager up in 9 seconds, workload pods writing in 53 and 58 seconds respectively.
• ShareManager up in 7 seconds, workload pods writing in 52 seconds and 52 seconds respectively.

I think RWX fast failover is still working as expected.

[PhanLe1010]

LGTM. Leave a styling comment but not strong about it. Feel free to ignore it if you prefer the current implementation

Thank you for the investigation and the fix

[ejweber]

@mergify backport v1.6.x v1.5.x

[mergify]

backport v1.6.x v1.5.x

✅ Backports have been created

• #3154 fix: attempt to delete processes from instance-managers in unknown state (backport #3127) has been created for branch v1.6.x but encountered conflicts
• #3155 fix: attempt to delete processes from instance-managers in unknown state (backport #3127) has been created for branch v1.5.x but encountered conflicts