kv: operations failing when encountering decommissioned nodes

[erikgrinaker]

On 21.1.1, operations tend to fail when encountering decommissioned nodes. Users have reported that after doing a full cycle of spinning up new nodes and decommissioning all old nodes, CHANGEFEED and BACKUP operations would fail:

CREATE CHANGEFEED FOR TABLE t INTO 'experimental-s3://bucket/changefeed/?AUTH=implicit' WITH compression = 'gzip', diff, format = 'json', initial_scan, resolved = '5m', schema_change_events = 'column_changes', schema_change_policy = 'backfill', updated;
ERROR: failed to connect to n4 at 172.31.91.238:26257: initial connection heartbeat failed: rpc error: code = PermissionDenied desc = n4 was permanently removed from the cluster at 2021-05-27 21:08:12.282567682 +0000 UTC; it is not allowed to rejoin the cluster

Restarting the affected nodes fixed the problem, so it seems related to some transitory state like range descriptor/leaseholder caches or gossip state.

This hasn't been reproducible, but other operations such as SET CLUSTER SETTING and SELECT * FROM t have occasionally been seen to fail this way.

There seems to be multiple reasons for this:

• Any RPC ping involving a decommissioned node (both to and from) returned PermissionDenied, which is treated as a permanent error. However, this was too symmetrical: when e.g. the DistSender tries to contact a range and encounters a decommissioned node, it shouldn't give up, but rather refresh its caches and retry against a different replica. Any KV operation could fail due to this. Fixed by server: return FailedPrecondition when talking to decom node #66199.

• Operations that explicitly choose nodes to operate on (in particular, DistSQL jobs) often only consider node liveness, but not its decommissioned state. Fixed by *: check node decommissioned/draining state for DistSQL/consistency  #66632.

• changefeedccl: Rangefeeds might fail due to stale range cache #66636 Range feeds appear to not not flush range descriptor caches or lease caches when encountering authentication errors, this was found to be due to a bug in range feed error handling. Fixed by kvcoord: fix rangefeed retries on transport errors #66910.

/cc @cockroachdb/kv

[erikgrinaker]

I believe all of the known bugs should be fixed now.

[erikgrinaker]

To verify, I wrote up the following script which readily failed before these fixes with:

rpc error: code = PermissionDenied desc = n1 was permanently removed from the cluster at 2021-06-29 12:41:50.019428021 +0000 UTC; it is not allowed to rejoin the cluster

After these fixes, I did dozens of runs without seeing a single failure.

roachprod destroy local

set -euxo pipefail

roachprod create local -n 9
roachprod start local:1-3
roachprod sql local:1 -- -e 'CREATE TABLE t (id int primary key, value int);'
roachprod sql local:1 -- -e 'INSERT INTO t VALUES (1, 1), (2, 2), (3, 3);'
sleep 3

roachprod start local:4-9
for N in 4 5 6 7 8 9; do
  roachprod sql local:$N -- -e 'SELECT * FROM t;'
done

cockroach node decommission --insecure 1 2 3

roachprod sql local:5 -- -e 'SET CLUSTER SETTING kv.rangefeed.enabled = true;'
roachprod sql local:7 -- -e "CREATE CHANGEFEED FOR TABLE t INTO 'experimental-nodelocal://6/tmp/changefeed';"
roachprod sql local:8 -- -e "BACKUP TABLE t INTO 'nodelocal://9/tmp/backup/t';"
for N in 4 5 6 7 8 9; do
  roachprod sql local:$N -- -e 'SELECT * FROM t;'
done

[erikgrinaker]

Relevant fixes have been backported to 20.2 and 21.1, will be included in the next patch releases (20.2.13 and 21.1.5).