RFC: rewrite vtgate's healthcheck

[deepthi]

Problem Statement

vtgate uses "healthcheck" to decide which tablets to route queries to. The "health" of each vttablet is stored by vtgate using a TabletStatsCache.
Choosing a tablet to execute a query requires going through the vtgate/resolver -> srvtopo/resolver which returns a list of ResolvedShard structs. But a ResolvedShard doesn't actually tell us which tablet the query should go to. That decision is only made at execution time.

In order to choose a tablet, the current implementation requires 3 distinct structures: a healthcheck, a TopologyWatcher and a TabletStatsCache like here. This seems like unnecessary complexity.

Proposed Solution

Rewrite the code that is used to choose a tablet for execution to be simpler. To start with, instead of the complex mechanism we currently have for keeping track of tablet health, it can be simplified as follows:

• A HealthCheck struct keeps track of all the tablets' health.
• A tabletHealthCheck struct is created when a new tablet is created in topo. It is removed when a tablet is deleted from topo.
• a goroutine which updates tabletHealthCheck periodically based on the results of the StreamHealth RPC
• vtgate's gateway gets the healthy tablets from HealthCheck to make routing decisions.

According to @sougou there are some special failure modes that we need to worry about and handle. To start with, we can build an alternate implementation for tablet health, gated by a flag. Once it reaches parity with the current implementation, it can be made available for general use.