keyspace: keyspace meta accounts for 65.28% of in-use memory of PD/API leader in dev env

[binshi-bing]

Enhancement Task

There are more >20K keyspaces in dev. Is this much memory usage by keyspace meta expected? Do we need to keep all keyspace meta in memory?

[lhy1024]

every client call WatchKeyspaces to maintain a watch-loop, but tidb only need to get all keyspace.

func (s *KeyspaceServer) WatchKeyspaces(request *keyspacepb.WatchKeyspacesRequest, stream keyspacepb.Keyspace_WatchKeyspacesServer) error {
    .....
    go watcher.StartWatchLoop()
    if err := watcher.WaitLoad(); err != nil {
        cancel() // cancel context to stop watcher
        return err
    }
    <-ctx.Done() // wait for context done
    return nil
}

func (w *GCWorker) getAllKeyspace(ctx context.Context) []*keyspacepb.KeyspaceMeta {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    watchChan, err := w.pdClient.WatchKeyspaces(ctx)
    if err != nil {
        logutil.Logger(ctx).Error("WatchKeyspaces error")
    }
    initialLoaded := <-watchChan
    return initialLoaded
}

I think we only provide GetAllKeyspace in client is enough and maintain only one watch-loop in server
And we also can LoadKeyspace by this server watch-loop and avoid to unnecessary load by etcd.

[lhy1024]

After discussion, @ystaticy will fix it