Bug Report: vtorc goes from a DeadPrimary -> ClusterHasNoPrimary; promotes final replica

[olyazavr]

Overview of the Issue

When you have a primary and a replica, and then you kill that primary, vtorc will first detect DeadPrimary, but then not intervene with ERS because it cannot find a suitable candidate that would still satisfy the semi_sync durability policy. However, it soon transitions to detecting ClusterHasNoPrimary (likely when the primary server completely terminates) and then performs PRS on that final replica, meaning you have one primary and no replicas, which is not a functional situation when running with semi_sync. After this, it detects LockedSemiSyncPrimary and cannot get out of it.

One workaround I have found is to introduce a patch to block PRS from choosing a new primary similar to this check, where we check if len(validTablets) == 1 && !currentlyHasAPrimary and if so, return nothing.

Reproduction Steps

1. Set up a keyspaceshard with one primary and one replica
2. Kill that primary (eg. mysql -- pkill -9 mysqld and then shut down the server abruptly)
3. Watch as vtorc first finds DeadPrimary, then ClusterHasNoPrimary, and promotes the replica to primary

Binary Version

Vitess 16

Operating System and Environment details

Centos8, Linux 5.4.141-hs22.el8.x86_64

Log Fragments

No response

[GuptaManan100]

Could you share the VTOrc and vttablet logs? I couldn't reproduce the problem locally following the steps listed.

[olyazavr]

vtorc logs: https://gist.github.com/olyazavr/a42483e8755add6a6ccee3aee153682a (omitted slack hook content for brevity)
vttablet logs: https://gist.github.com/olyazavr/57a450705eb6c7afcf3e9856745e3a06

[mattlord]

cc @vitessio/vtorc

[GuptaManan100]

@olyazavr I don't have time right now to be able to fix this issue, but if you want to take it up, I'd be happy to review the PR. I can review any pseudocode/ideas that you want to talk about too.

[GuptaManan100]

@olyazavr Never mind, I started working on this today. Draft PR - #13587