optimize copyset creation #1211
Conversation
cw123
force-pushed
the
optimize_copyset_creation
branch
3 times, most recently
from
23f28da to
137bb4e
Compare
| int randomValue = GetOneRandomNumber(0, copysetList.size() - 1); | ||
| auto iter = copysetList.begin(); | ||
| std::advance(iter, randomValue); | ||
| auto it = copySetMap_.find(*iter); |
There was a problem hiding this comment.
auto it = copySetMap_.find(*iter) is needn't ?
if (iter != copySetMap_.end()) {
}
There was a problem hiding this comment.
iter is copysetKey, it is copysetInfo.
it select key first, then use the key find the info
|
|
||
| // 5. generate result | ||
| if (candidateMap.size() == 0) { | ||
| LOG(WARNING) << "can not find available for copyset."; |
There was a problem hiding this comment.
"can not find available metaservers for creating copyset." ?
| return TopoStatusCode::TOPO_OK; | ||
| } | ||
|
|
||
| TopoStatusCode ret = TopoStatusCode::TOPO_OK; |
There was a problem hiding this comment.
You can add some notes here to explain the logic.
| TopoStatusCode CreateEnoughCopyset(); | ||
| TopoStatusCode CreateCopysetRandom(int createNum); | ||
| TopoStatusCode CreateCopysetByResourceUsage(int createNum); | ||
| TopoStatusCode CreateNewCopyset(const CopysetCreateInfo& copyset); |
There was a problem hiding this comment.
CreateNewCopyset -> CreateCopyset ?
cw123
force-pushed
the
optimize_copyset_creation
branch
from
137bb4e to
25925c2
Compare
| @@ -1257,8 +1281,152 @@ TopoStatusCode TopologyImpl::ChooseNewMetaServerForCopyset( | |||
| return TopoStatusCode::TOPO_OK; | |||
| } | |||
|
|
|||
| TopoStatusCode TopologyImpl::ChooseAvailableMetaServers( | |||
| std::set<MetaServerIdType> *metaServers, PoolIdType *poolId) { | |||
| TopoStatusCode TopologyImpl::GenCopysetAddrRandomBatch( | |||
There was a problem hiding this comment.
This function is too long, and it does different things, you can consider splitting it.It's also easier to test
| std::vector<const MetaServer *> vec; | ||
| for (const auto &it : metaServerMap_) { | ||
| if (it.second.GetOnlineState() == OnlineState::ONLINE | ||
| && it.second.GetMetaServerSpace().IsMetaserverResourceAvailable()) { |
There was a problem hiding this comment.
In addition to considering memory and disk utilization, the number of copies also needs to be considered.
If there are too many replication groups, no matter how the number of balanced leaders on this metaserver is large, it should become a bottleneck.
cw123
force-pushed
the
optimize_copyset_creation
branch
from
25925c2 to
cd2c8c9
Compare
What problem does this PR solve?
Issue Number: #1196
Problem Summary: Initially created copies data is not balanced
What is changed and how it works?
What's Changed: optimize copyset creation
1、优化了copyset的创建算法，copyset的创建分为两个场景：
场景一：集群第一次创建copyset，为每个copyset随机选择metaserver；
场景二：集群已有copyset，继续创建copyset的场景，根据metaserver的资源使用率，选择使用率低的metaserver创建copyset。
2、优化了partition选择copyset的算法，为每个partition都选择一次copyset。