VTGate HealthCheckUpdate failing to update vttablet serving status

[gedgar]

Overview of the Issue

Despite receiving SIGKILL, VTTablet still marked as SERVING by VTGate.

Reproduction Steps

1. Run ./101_initial_cluster.sh
2. Kill rdonly vttablet

kill -9 `pgrep vttablet | tail -n 1`

From the vtgate logs, HealthCheckUpdate does appear to update the rdonly's serving status to false:

==> vtgate.INFO <==
I0209 00:38:32.330542   10475 tablet_health_check.go:111] HealthCheckUpdate(Serving State): tablet: zone1-102 (localhost) serving true => false for commerce/0 (RDONLY) reason: vttablet: rpc error: code = Unavailable desc = transport is closing

However, vtgate still shows the rdonly vttablet as SERVING:

mysql --host 127.0.0.1 --port 15306 -e "show vitess_tablets"

+-------+----------+-------+------------+---------+------------------+-----------+----------------------+
| Cell  | Keyspace | Shard | TabletType | State   | Alias            | Hostname  | MasterTermStartTime  |
+-------+----------+-------+------------+---------+------------------+-----------+----------------------+
| zone1 | commerce | 0     | MASTER     | SERVING | zone1-0000000100 | localhost | 2021-02-09T08:37:04Z |
| zone1 | commerce | 0     | REPLICA    | SERVING | zone1-0000000101 | localhost |                      |
| zone1 | commerce | 0     | RDONLY     | SERVING | zone1-0000000102 | localhost |                      |
+-------+----------+-------+------------+---------+------------------+-----------+----------------------+

And will still route queries to it:

mysql --host 127.0.0.1 --port 15306 @rdonly -e "show tables"
ERROR 1105 (HY000) at line 1: vtgate: http://localhost:15001/: target: commerce.0.rdonly, used tablet: zone1-102 (localhost): vttablet: Connection Closed

Once VTGate is restarted, rdonly tablet is properly marked as NOT_SERVING:

mysql --host 127.0.0.1 --port 15306 -e "show vitess_tablets"
+-------+----------+-------+------------+-------------+------------------+-----------+----------------------+
| Cell  | Keyspace | Shard | TabletType | State       | Alias            | Hostname  | MasterTermStartTime  |
+-------+----------+-------+------------+-------------+------------------+-----------+----------------------+
| zone1 | commerce | 0     | MASTER     | SERVING     | zone1-0000000100 | localhost | 2021-02-09T08:37:04Z |
| zone1 | commerce | 0     | REPLICA    | SERVING     | zone1-0000000101 | localhost |                      |
| zone1 | commerce | 0     | RDONLY     | NOT_SERVING | zone1-0000000102 | localhost |                      |
+-------+----------+-------+------------+-------------+------------------+-----------+----------------------+

Binary version

Version: 10.0.0-SNAPSHOT (Git revision 676911a branch 'master') built on Mon Feb  8 22:37:54 PST 2021 by gary@planetscale using go1.15.6 linux/amd64

Log Fragments