Use LOCK TABLES during SwitchWrites to prevent lost writes

[mattlord]

Description

ℹ️ This is not an issue for Reshard workflows because open transactions are closed (either by waiting for a grace period or finally cancelled) as part of the primary tablet shutting down since the source shards stop serving in SwitchWrites.

Issuing a LOCK TABLES statement on all of the tables that we are doing a SwitchWrites operation for as part of a MoveTables workflow ensures that no lost writes occur due to long running transactions (including long running single statement auto-commit transactions) on the source tablets committing after writes have been switched. I say this because:

1. During SwitchWrites we first we update the DenyList on the source tablets to prevent any new writes against the tables at the tablet level in the query planner/executor.
2. We then wait on a LOCK TABLES call to return on the source mysqld's which will in turn wait for any open/running RW statements/transactions on those tables to complete or time out.
3. Af this point, after steps 1 and 2, we know that any existing writes against the source tables have completed and no new ones can come in.
4. We then collect the vreplication metadata snapshot, put the reverse vreplication streams in place, and update the routing rules.

Without doing this (before this PR is merged) you can not only break reverse vreplication -- which would have to be fixed manually -- but if you don't notice the errant writes and then manually reconcile the tables across keyspaces (old and new) then those writes are lost.

With this patch, when you go through the manual test here and get to the point where you attempt another insert or commit in the previously opened transaction started against the source before we switched writes, you get an error like this:

ERROR 1317 (70100): vttablet: rpc error: code = Aborted desc = transaction 1641661850058067114: ended at 2022-01-08 17:12:17.090 UTC (exceeded timeout: 30s) (CallerID: userData1)

Related Issue(s)

Fixes: #9400

Checklist

• Should this PR be backported? At least to 12.0, I think...
• Tests were added
• Documentation: I don't think we need any

[DeathBorn]

Before this implementation we found this manual workaround
 LOCK TABLES <all_tables_that_move_to_other_shard> READ;

[harshit-gangal]

looks good overall.
some nits.

[rohit-nayak-ps]

ltgm

[mattlord]

Thank you for the review and comments, @rohit-nayak-ps and @harshit-gangal ! I just pushed another commit that addresses them. ❤️

[rohit-nayak-ps]

lgtm