kv: DistSender for SQL tenants

[tbg]

Background

DistSender needs changes to work on SQL tenants. The two approaches we've discussed are either giving it an endpoint to do range lookups (and changing a bunch of stuff on how it uses gossip and other systems that won't be available) OR pushing it into the KV layer for these deployments.

Abstractly the second option seems cleaner and easier, but it's also new territory. Either way, we need to pick one and do it.

[tbg]

@nvanbenschoten and I discussed this a bit yesterday.

Nathan made the good point that by moving DistSender into the KV layer, there is an extra (AZ-local) hop in every request. This doesn't matter much for throughput (tenants are going to be rate limited anyway) but it does matter for latency. Roughly, with DistSender in the tenant, we get latency(client,tenant)+latency(tenant,leaseholder). With DistSender in the KV layer, we get latency(client,tenant)+latency(tenant,nodeA)+latency(tenant,leaseholder) which is expected to be approximately latency(client,tenant)+2*latency(tenant,leaseholder). It's possibly not a huge deal now, but it's not pretty and could become important down the road.

There were two reasons to originally want to move DistSender into KV:

One was that we won't initially have a very good upgrade story around multi-tenancy (see #47919), and one argument for moving DistSender into KV (in multi-tenant deploys) was making this story simpler.

The other argument was that DistSender has some pretty KV-ey dependencies: it wants the Gossip network (to translate NodeID->NodeAddress and Meta1Descriptor) and reads from the meta ranges, which are global cluster data (i.e. not per-tenant).

Neither of these seems particularly compelling on its own. I have looked into the upgrade story a bit and while there may be a snag or two for some migrations, overall assuming we establish clear APIs between the tenant and KV (which we'll absolutely do) we should manage, given that we are for now the only operator of multi-tenant deploys.

The dependencies are also dealt with relatively straightforwardly:

• add a TenantRangeLookup RPC to the KV tenant service which allows looking up range descriptors, but only those in the purview of the tenant
• add a TenantNodeInfo endpoint which streams (NodeID,NodeAddress) pairs back to the tenant, essentially providing a RangeFeed of node addresses (with an update whenever an IP changes).
• the tenant does not need meta1, since it can hit the TenantRangeLookup endpoint on any KV node, and we'll simply point it at a (AZ-preferring) round-robin.

There are variations on this idea that create more homogeneity between single- and multi-tenant deployments. We can (and should) ensure that we use essentially the same mechanisms in both, i.e. have DistSender always pull range descriptors from the specialized endpoint (which abstracts over the internal scan of the meta ranges that DistSender itself does today), and similarly for the id->ip mapping. This amounts to moving only the gossip and meta dependency into KV RPC.

Lastly on top of the latency improvement there is another straightforward argument for leaving DistSender on the tenant, namely caching and resource consumption. On the tenant, the DistSender cache is going to cache exclusively the tenant's range descriptors, whereas in the KV layer they would all get mixed up. Besides, DistSender is a complex piece of code with many moving parts. It is better if any bug or resource exhaustion occurs on the SQL tenant.

And, for another important one: DistSQL relies on leaseholder locations and more generally knowledge of ranges for setting up flows, which we eventually want it to do across multiple SQL containers for the same tenant. Moving DistSender into KV will make this harder.

All things considered, it now seems preferable to leave DistSender in the tenant and abstract away its KV dependencies behind RPCs instead.

I expect @nvanbenschoten to look into the details soon at which point we'll have a more concrete plan of action.

[tbg]

Something I forgot to add: we were also considering whether we should stop using gossip to translate id to address altogether (in the comment above, the TenantNodeInfo endpoint implementation would essentially just relay gossip callbacks to the tenant).

The idea here would be to have a "sign-in table" in which nodes post their address when they start (this is already in the store status, I think, but with a lot of junk we don't want to stream out, so it may have to be a new table). Every node would then have a RangeFeed on that table.

My feeling is that the id->ip mapping is one of the few very legitimate uses of Gossip, so I'm not really interested in making that change. However, it is a future change we could make if we wanted to reduce our dependency on Gossip. On the other hand, @andreimatei is looking into improved DistSender range descriptor/leaseholder cache coherence by broadcasting updates after splits/lease transfers, for which the rangefeed solution falls flat. So I think there's an additional reason to stick for Gossip for now (and probably ever).