Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: Add a new way to store metadata of regions #1237

Merged
merged 21 commits into from
Oct 10, 2018

Conversation

nolouch
Copy link
Contributor

@nolouch nolouch commented Sep 6, 2018

What problem does this PR solve?

This Version is used for testing first, not completed.

What is changed and how it works?

Check List

Tests

  • Unit test
  • Manual test (add detailed scripts or steps below)

@nolouch nolouch added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 6, 2018
@siddontang
Copy link
Contributor

em, why introduce LevelDB, do we really need it?

@nolouch nolouch removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 11, 2018
@nolouch nolouch force-pushed the save-goleveldb branch 2 times, most recently from 92240fe to 4eaa00c Compare September 12, 2018 08:25
@nolouch
Copy link
Contributor Author

nolouch commented Sep 12, 2018

@siddontang we need load regions from regions to speed up recover router of the regions.
PTAL @disksing @rleungx @liukun4515

@nolouch nolouch changed the title *: save metadata of regions to leveldb *: Add a new way to store metata of regions Sep 12, 2018
@nolouch nolouch changed the title *: Add a new way to store metata of regions *: Add a new way to store metadata of regions Sep 12, 2018
@nolouch
Copy link
Contributor Author

nolouch commented Sep 13, 2018

/rebuild

return err
}

cc, err := grpc.Dial(u.Host, grpc.WithInsecure(), grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(8*1024*1024)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to use a constant?

s.Unlock()
}

func (s *regionSyncer) stopSyncerWithLeader() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe stopSyncWithLeader?

s.wg.Wait()
}

func (s *regionSyncer) statSyncerWithLeader(addr string) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean start*?

}
u, err := url.Parse(addr)
if err != nil {
return err
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about errors.WithStack(err)?

count++
}
iter.Release()
//iter.Error()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this comment mean?

@@ -27,6 +28,13 @@ type KVBase interface {
Delete(key string) error
}

// RegionKV used to save region metadata.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is used

server/config.go Outdated
@@ -633,6 +635,12 @@ func (s SecurityConfig) ToTLSConfig() (*tls.Config, error) {
return tlsConfig, nil
}

// PDServerConfig is the configuration for pd server.
type PDServerConfig struct {
// EnableRegionStorage enable the independent region storage.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enables and remove the extra space

DefaultBatchSize = 100
)

// WithLevelDBKV store the regions information in levelDB.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stores

}()
}

// FlushRegion save the cache region to region kv storage.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

saves

db *leveldb.DB
}

// NewLeveldbKV to store regions information.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is used to store

}
kv.cacheSize = 0
kv.batchRegions = make(map[string]*metapb.Region, kv.batchSize)
return nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicated

kv.cacheSize = 0
kv.batchRegions = make(map[string]*metapb.Region, kv.batchSize)
return nil
//return saveProto(kv.regionKV, kv.regionPath(region.GetId()), region)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be removed?

server/server.go Outdated
@@ -388,7 +393,8 @@ func (s *Server) bootstrapCluster(req *pdpb.BootstrapRequest) (*pdpb.BootstrapRe
}

log.Infof("bootstrap cluster %d ok", clusterID)

s.kv.SaveRegion(req.GetRegion())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to check error?

server/server.go Outdated
@@ -208,7 +209,8 @@ func (s *Server) startServer() error {

s.idAlloc = &idAllocator{s: s}
kvBase := newEtcdKVBase(s)
s.kv = core.NewKV(kvBase)
path := filepath.Join(s.cfg.DataDir, "leveldb")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about making it something like "region-meta"?

return status.Errorf(codes.FailedPrecondition, "mismatch cluster id, need %d but got %d", s.clusterID, request.GetHeader().GetClusterId())
}
switch request.GetTp() {
case 1:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the Tp used for? I think better to define it as enum in proto file.

if !isFlush {
continue
}
kv.FlushRegion()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check the error?

running: false,
clusterID: clusterID,
clusterRoot: s.getClusterRootPath(),
clients: make(map[string]pdpb.PDClient),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unused?

Header: &pdpb.ResponseHeader{ClusterId: c.s.clusterID},
Data: data,
}
for _, stream := range c.regionSyncer.streams {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read streams without lock?


type regionSyncer struct {
sync.Mutex
streams map[string]pdpb.PD_SyncRegionsServer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think streams can be managed in Server directly. As it is used for leader to sync regions to followers, while other fields are for sync regions with leader.

server/leader.go Outdated
log.Error("reload config failed:", err)
return
}
err = s.cluster.regionSyncer.statSyncerWithLeader(leader.GetClientUrls()[0])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not the first address. Maybe part of addrs can be accessed and the valid address is not in the first place.

return nil
}

func (kv *KV) doGC() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think GC is a suitable name here.

}

// FlushRegion save the cache region to region kv storage.
func (kv *KV) FlushRegion() error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it really necessary to save regions to levelDB in batches?

cancel()
return nil, err
}
err = client.Send(&pdpb.SyncRegionRequest{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need client streaming if it only sends one request?

Copy link
Contributor Author

@nolouch nolouch Sep 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we also use it to receive the response of newly updated regions from leader.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess server streaming will be enough.
rpc SyncRegions(stream SyncRegionRequest) returns (stream SyncRegionResponse) {}

@nolouch
Copy link
Contributor Author

nolouch commented Sep 26, 2018

PTAL @disksing @rleungx @liukun4515

func NewKVProxy(defaultKV KVBase, regionKV *RegionKV) *KVProxy {
kv := &KVProxy{
defaultKV: defaultKV,
regionKV: regionKV,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seem we don't need to initialize regionKV with nil here

}
// ensure flush to region kv
time.Sleep(3 * time.Second)
leaderServer.Stop()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check error?

regions = append(regions, core.NewRegionInfo(r, r.Peers[0]))
}
for _, region := range regions {
rc.HandleRegionHeartbeat(region)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check error?

@@ -633,6 +635,12 @@ func (s SecurityConfig) ToTLSConfig() (*tls.Config, error) {
return tlsConfig, nil
}

// PDServerConfig is the configuration for pd server.
type PDServerConfig struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this? Can we put EnableRegionStorage into Config directly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use it to persist the config of PD-server.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by persist to PD-server?

for _, sender := range s.streams {
err := sender.Send(regions)
if err != nil {
log.Error("region syncer send data meet error:", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to remove the stream from the map?

@@ -26,7 +26,6 @@ type KVBase interface {
Save(key, value string) error
Delete(key string) error
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove this line?

server/leader.go Outdated
if isSync {
s.cluster.regionSyncer.stopSyncWithLeader()
}
}()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if sync {
    startSyncWithLeader
    defer stopSyncWithLeader
}

isDefault atomic.Value
}

// NewKVProxy return a proxy of KV storage.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

returns

continue
}
if err = kv.FlushRegion(); err != nil {
log.Info("flush regions error: ", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log.error

// Close closes the kv.
func (kv *RegionKV) Close() error {
kv.cancel()
return kv.db.Close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we flush the cached region before closing the leveldb?

c.Assert(leaderServer, NotNil)
loadRegions := leaderServer.server.GetRaftCluster().GetRegions()
c.Assert(len(loadRegions), Equals, regionLen)
cluster.Destroy()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe use defer here?

}
}
}
// Flush flush the dirty region to storage.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flushes

defaultBatchSize = 100
)

// NewRegionKV return a kv storage that is used to save regions.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

returns

rangeLimit := maxKVRangeLimit
for {
key := regionPath(nextID)
res, err := kv.LoadRange(key, endKey, rangeLimit)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to use startKey?

server/server.go Outdated

err = s.kv.SaveRegion(req.GetRegion())
if err != nil {
log.Warnf("save the bootstrap region faild: %s", err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/faild/failed/g

func (kv *RegionKV) SaveRegion(region *metapb.Region) error {
kv.mu.Lock()
defer kv.mu.Unlock()
if kv.cacheSize < kv.batchSize {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kv.cacheSize < kv.batchSize -1

kv.cacheSize < kv.batchSize means that the flush will be triggered until the size of batchRegions reach to batchSize+1

Copy link
Contributor

@liukun4515 liukun4515 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

func newLeveldbKV(path string) (*leveldbKV, error) {
db, err := leveldb.OpenFile(path, nil)
if err != nil {
return nil, err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to wrap the leveldb errors with call stack.

@disksing
Copy link
Contributor

disksing commented Oct 9, 2018

PTAL @rleungx

Copy link
Member

@rleungx rleungx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rest LGTM.

@@ -35,11 +37,14 @@ const (
const (
maxKVRangeLimit = 10000
minKVRangeLimit = 100
dirtyFlushTick = time.Second
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to move it into region_kv.go?

}
}
}
// Flush flushs the dirty region to storage.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flushes

@nolouch
Copy link
Contributor Author

nolouch commented Oct 10, 2018

/rebuild

@nolouch
Copy link
Contributor Author

nolouch commented Oct 10, 2018

/run-all-tests

@nolouch nolouch merged commit ef2bdf2 into tikv:master Oct 10, 2018
@nolouch nolouch deleted the save-goleveldb branch October 10, 2018 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants