VDiff: vttablet panics for vdiffs with multiple tables under heavy load

[rohit-nayak-ps]

Overview of the Issue

There is a race condition in VDiff when a workflow has multiple tables. Consider a workflow with two tables.

1. The diff of the first table is deemed done, because too many rows have errors, for example. The corresponding shard streamers are waiting on cancellation of the context passed into vdiff (which times out on action_timeout (1 hour) or on the controller's context cancellation which happens when the entire vdiff (for all tables) is done.
2. The second table's diff starts. This does creates a new channel source.result = make(chan *sqltypes.Result, 1) for the shard streamers to communicate their rows on.
3. The second table's diff ends. The corresponding shard streamers for the second table are also waiting on the same conditions as the first table.
4. Since all tables have completed vdiff the controller's context is done and both sets of shard streamers try to close(source.result), which is pointing to the same channel due to the race.
5. This results in a vttablet panic for the second streamer.

The race which I encountered happened under conditions of load where there was a load simulator running DMLs at ~1K QPS.

Reproduction Steps

Happened while working on #14345

Binary Version

v19