Feature Request: ReverseTraffic for only replica and rdonly tablet types

[iheanyi]

Feature Description

Sometimes we want to cancel a workflow when writes have not been switched. As things exist today, we are unable to do a ReverseTraffic --tablet-types "rdonly,replica" because the reverse stream is only created once the primary traffic is switched. If you were to reverse traffic of replicas only and then try to cancel, you'd encounter an error of

E1008 18:52:57.525791   19246 main.go:56] rpc error: code = Unknown desc = no streams found in keyspace commerce for commerce2customer_reverse

E1008 18:52:45.022103   19116 main.go:56] rpc error: code = Unknown desc = cannot cancel workflow because you have already switched some or all read and write traffic

Use Case(s)

Switching read traffic and then reversing and cancelling a workflow if something is wrong

[mattlord]

Related to this, you cannot reverse traffic for replica and/or rdonly tablet types (AKA switch reads) once switched. If you do this:

vtctldclient MoveTables SwitchTraffic --tablet-types replica

Then you cannot cancel the workflow, but you also cannot switch that traffic back using ReverseTraffic as that command requires that the _reverse workflow exist in the original source Keyspace, but that is only created when switching writes with --tablet-types primary.

To demonstrate:

git checkout main && make build

alias vtctldclient='command vtctldclient --server=localhost:15999'
cd examples/local

./101_initial_cluster.sh; mysql < ../common/insert_commerce_data.sql; ./201_customer_tablets.sh; ./202_move_tables.sh; ./203_switch_reads.sh

vtctldclient MoveTables --workflow commerce2customer --target-keyspace customer cancel

vtctldclient MoveTables --workflow commerce2customer --target-keyspace customer reversetraffic --tablet-types "rdonly,replica"

Results being:

❯ vtctldclient MoveTables --workflow commerce2customer --target-keyspace customer cancel
E1008 18:52:45.022103   19116 main.go:56] rpc error: code = Unknown desc = cannot cancel workflow because you have already switched some or all read and write traffic

❯ vtctldclient MoveTables --workflow commerce2customer --target-keyspace customer reversetraffic --tablet-types "rdonly,replica"
E1008 18:52:57.525791   19246 main.go:56] rpc error: code = Unknown desc = no streams found in keyspace commerce for commerce2customer_reverse

[iheanyi]

I've gone ahead and changed this issue as a tracking issue for the reverse traffic problem, as that will give the same desired behavior even though it has an extra step.

[mattlord]

Just noting that this is actually a bug related to the client transition. This works in vtctlclient today but not vtctldclient. So we'll close that gap.

❯ vtctldclient MoveTables --workflow commerce2customer --target-keyspace customer reversetraffic --tablet-types "rdonly,replica"
E1008 18:52:57.525791   19246 main.go:56] rpc error: code = Unknown desc = no streams found in keyspace commerce for commerce2customer_reverse

❯ vtctlclient MoveTables -- reversetraffic customer.commerce2customer --tablet_types "rdonly,replica"
W1008 19:45:47.734875   67717 log.go:39] Failed to read in config : Config File "vtconfig" Not Found in "[/Users/matt/git/vitess/examples/local]". This is optional, and can be ignored if you are not using config files. For a detailed explanation, see https://github.com/vitessio/vitess/blob/main/doc/viper/viper.md#config-files.
reversetraffic was successful for workflow customer.commerce2customer
Start State: All Reads Switched. Writes Not Switched
Current State: Reads Not Switched. Writes Not Switched