Skip to content

Commit

Permalink
kv/kvclient: introduce new tenant Proxy
Browse files Browse the repository at this point in the history
Fixes cockroachdb#47909.

This commit starts by adding two RPCs to the Internal service:
```
service Internal {
	...
	rpc RangeLookup (RangeLookupRequest) returns (RangeLookupResponse)     {}
	rpc NodeInfo    (NodeInfoRequest)    returns (stream NodeInfoResponse) {}
}

// RangeLookupRequest is a request to proxy a RangeLookup through a Tenant
// service. Its fields correspond to a subset of the args of kv.RangeLookup.
message RangeLookupRequest {
    ...
}

// NodeInfoRequest is a request to establish an indefinite stream on a Tenant
// service that provides an initial NodeInfoResponse and a NodeInfoResponse
// whenever the collection of KV nodes in a cluster changes. It effectively
// proxies any updates to NodeDescriptors in the KV gossip network back to the
// client of the request.
message NodeInfoRequest {}
```

The commit then introduces new `kvtenant.Proxy` object. Proxy mediates
the communication of cluster-wide state to sandboxed SQL-only tenant
processes through a restricted interface. A Proxy is seeded with a set
of one or more network addresses that reference existing KV nodes in the
cluster (or a load-balancer which fans out to some/all KV nodes). On
startup, it establishes contact with one of these nodes to learn about
the topology of the cluster and bootstrap the rest of SQL <-> KV network
communication.

Proxy has two main roles:

First, Proxy is capable of providing information on each of the KV nodes
in the cluster in the form of NodeDescriptors. This obviates the need
for SQL-only tenant processes to join the cluster-wide gossip network.
In doing so, it satisfies the `NodeDescStore` interface and can be used
as an `AddressResolver` with a small adapter.

Second, Proxy is capable of providing Range addressing information in
the form of RangeDescriptors through delegated RangeLookup requests.
This is necessary because SQL-only tenants are restricted from reading
Range Metadata keys directly. Instead, the RangeLookup requests are
proxied through existing KV nodes while being subject to additional
validation (e.g. is the Range being requested owned by the requesting
tenant?). In doing so, it satisfies the `RangeDescriptorDB` interface
and can be used to delegate all DistSender/RangeCache descriptor lookups
to KV nodes.

With this commit, we can mostly run a SQL-only tenant process without
joining the KV cluster's gossip network. This works if I comment out a
few of the uses of gossip due to cockroachdb#49692 and cockroachdb#47150 in SQL. Notably,
with the call to `DeprecatedRegisterSystemConfigChannel` in `sql.Server.Start`
removed, I can remove `Gossip` from `makeSQLServerArgs` entirely and
things "just work".
  • Loading branch information
nvanbenschoten committed Jul 2, 2020
1 parent 65270c1 commit f3c7547
Show file tree
Hide file tree
Showing 15 changed files with 4,897 additions and 705 deletions.
1,174 changes: 1,174 additions & 0 deletions c-deps/libroach/protos/roachpb/api.pb.cc

Large diffs are not rendered by default.

1,042 changes: 1,036 additions & 6 deletions c-deps/libroach/protos/roachpb/api.pb.h

Large diffs are not rendered by default.

7 changes: 3 additions & 4 deletions pkg/gossip/gossip.go
Original file line number Diff line number Diff line change
Expand Up @@ -1151,19 +1151,18 @@ func (g *Gossip) GetSystemConfig() *config.SystemConfig {
// system config. It is notified after registration (if a system config is
// already set), and whenever a new system config is successfully unmarshaled.
func (g *Gossip) RegisterSystemConfigChannel() <-chan struct{} {
g.systemConfigMu.Lock()
defer g.systemConfigMu.Unlock()

// Create channel that receives new system config notifications.
// The channel has a size of 1 to prevent gossip from blocking on it.
c := make(chan struct{}, 1)
g.systemConfigChannels = append(g.systemConfigChannels, c)

// Notify the channel right away if we have a config.
if g.systemConfig != nil {
c <- struct{}{}
}

g.systemConfigMu.Lock()
g.systemConfigChannels = append(g.systemConfigChannels, c)
g.systemConfigMu.Unlock()
return c
}

Expand Down
1 change: 1 addition & 0 deletions pkg/gossip/infostore.go
Original file line number Diff line number Diff line change
Expand Up @@ -310,6 +310,7 @@ func (is *infoStore) registerCallback(
if targetCB == cb {
numCBs := len(is.callbacks)
is.callbacks[i] = is.callbacks[numCBs-1]
is.callbacks[numCBs-1] = nil // for GC
is.callbacks = is.callbacks[:numCBs-1]
break
}
Expand Down
12 changes: 12 additions & 0 deletions pkg/kv/kvclient/kvcoord/send_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,22 @@ func (n Node) Batch(
return &roachpb.BatchResponse{}, nil
}

func (n Node) RangeLookup(
_ context.Context, _ *roachpb.RangeLookupRequest,
) (*roachpb.RangeLookupResponse, error) {
panic("unimplemented")
}

func (n Node) RangeFeed(_ *roachpb.RangeFeedRequest, _ roachpb.Internal_RangeFeedServer) error {
panic("unimplemented")
}

func (n Node) GossipSubscription(
_ *roachpb.GossipSubscriptionRequest, _ roachpb.Internal_GossipSubscriptionServer,
) error {
panic("unimplemented")
}

// TestSendToOneClient verifies that Send correctly sends a request
// to one server using the heartbeat RPC.
func TestSendToOneClient(t *testing.T) {
Expand Down
14 changes: 14 additions & 0 deletions pkg/kv/kvclient/kvcoord/transport_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -162,9 +162,23 @@ func (m *mockInternalClient) Batch(
return br, nil
}

// RangeLookup implements the roachpb.InternalClient interface.
func (m *mockInternalClient) RangeLookup(
ctx context.Context, rl *roachpb.RangeLookupRequest, _ ...grpc.CallOption,
) (*roachpb.RangeLookupResponse, error) {
return nil, fmt.Errorf("unsupported RangeLookup call")
}

// RangeFeed is part of the roachpb.InternalClient interface.
func (m *mockInternalClient) RangeFeed(
ctx context.Context, in *roachpb.RangeFeedRequest, opts ...grpc.CallOption,
) (roachpb.Internal_RangeFeedClient, error) {
return nil, fmt.Errorf("unsupported RangeFeed call")
}

// GossipSubscription is part of the roachpb.InternalClient interface.
func (m *mockInternalClient) GossipSubscription(
ctx context.Context, args *roachpb.GossipSubscriptionRequest, _ ...grpc.CallOption,
) (roachpb.Internal_GossipSubscriptionClient, error) {
return nil, fmt.Errorf("unsupported GossipSubscripion call")
}
Loading

0 comments on commit f3c7547

Please sign in to comment.