rpc: verify node ID in handshake

[petermattis]

Outside of gossip, when a cockroach node opens an RPC connection it knows the ID of the node it expects to be talking to. Due to the ephemerality and reuse of hostnames under Kubernetes, the recipient might be a different node. For sanity, and reduction of log spam, we should extend the handshake request PingRequest to take an optional node ID. If present the request should fail if the remote node ID differs from the requested ID.

See #34120 (comment).

[knz]

Currently looking at this.

[tbg]

This is a little less straightforward than it seems because the NodeDialer isn't the only use of the rpcContext (or so I think). An alternative would be to establish the connection, but to only hand it out to callers that have a NodeID if the NodeID attached to this connection matches.

PS not all nodes have a NodeID (they only do once they're fully part of the cluster). Connecting to a node that doesn't have one via a NodeID should fail.

[petermattis]

This is a little less straightforward than it seems because the NodeDialer isn't the only use of the rpcContext (or so I think). An alternative would be to establish the connection, but to only hand it out to callers that have a NodeID if the NodeID attached to this connection matches.

I think the users of rpc.Context are NodeDialer and gossip. (I could be mistaken about this). Your proposal to only hand out NodeDialer connections if the remote NodeID matches sounds reasonable. There is still the complexity of how to pass along the remote NodeID for connections that were initiated by gossip or before the remote side had a NodeID.

[knz]

My PR has a solution to that. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.