Replica usage should be limited in database level to reduce the fe memory pressure and the amount of tablet in be

[caiconghui]

Not only data but also replica size should be limited. Too much replica would occupy too much memory in fe and cause poor performance for tablet manage task in be, we need to know the replica usage and set the quota in db level, like the following implementation.
mysql> alter database test set replica quota 1000;
Query OK, 0 rows affected (0.06 sec)

mysql> show data;
+-------------+------------+-------------+
| TableName | DataSize | ReplicaSize |
+-------------+------------+-------------+
| table_hash | 1.771 KB | 32 |
| table_hash1 | .000 | 60 |
| table_hash2 | .000 | 60 |
| Total | 1.771 KB | 152 |
| Quota | 500.000 GB | 1000 |
| Left | 500.000 GB | 848 |
+-------------+------------+-------------+
6 rows in set (0.01 sec)

mysql>

[chaoyli]

you also need to set this options

[imay]

@chaoyli
Only the committers has the privilege to change this. You can do it for others next time.

[imay]

@caiconghui

This is a good feature. I have some question about it.

1. ReplicaSize -> Number of Replica
2. How will this statistics be collected, and when the limit will be checked? You can explain it more carefully.
3. If number of replica reach the limit, is there any method leave for user to handle this situation? Besides giving them more replica limit, Doris should give them more options to handle this situation. For example, merge multiple partition to one partition, merge multiple tablet in one partition etc. And this can be following issues for this issue

[caiconghui]

en,

1. Because there is replication_num property for table or partition create setting, which mean one tablet has "replication_num" replica, I use replicaSize instead of replica_num to separate replica_num from replication_num clearly, and maybe replica_num is more accurate for the original meaning
2. the replica statistics is collected by db->table -> partition -> MaterializedIndex -> tablet->replica
   hierarchy，include all replicas with all replicaState in table, and we check it when create table, alter table, if current num of replica in db not exceed the replica quota in db，we doesn't prohibit the operation.
3. at the moment ,we just limit the quota, I think it is a good idea to give user more options to optimize the usage of the replica, such as resize tablets of one partition when user doesn't need to reload the data, merge multiple partition to one partition, and we can also monitor the replica data size in every table, if serval replica data size are under low water level value for long time，then should remind user of optimizing the setting for partition

[imay]

en,

1. Because there is replication_num property for table or partition create setting, which mean one tablet has "replication_num" replica, I use replicaSize instead of replica_num to separate replica_num from replication_num clearly, and maybe replica_num is more accurate for the original meaning

Is ReplicaCount better?

2. the replica statistics is collected by db->table -> partition -> MaterializedIndex -> tablet->replica
   hierarchy，include all replicas with all replicaState in table

How often do you want to collect it? Or it will be collected through event?

[caiconghui]

@imay
at the moment, replica quota will be checked when create table or alter table then the replica count would be collected , or when we execute "show data" command or other show proc command can trigger the replica statistics collect operation.
ReplicaCount is more clear, I will modify it

[imay]

at the moment, replica quota will be checked when create table or alter table then the replica count would be collected , or when we execute "show data" command or other show proc command can trigger the replica statistics collect operation.

This will traverse all the metadata of a database, I don't think this is a good idea. It may hold the database lock for a long time.

[caiconghui]

the time cost is almost same with the getDataSize, we now use the db level lock to get the meta info, about 0.26s cost for about 2500 tables for getDataSize on our production environment, maybe table level lock would be better?

[imay]

Why not maintain a counter in Database object like data size of the database?

[caiconghui]

en， if the traverse cost is really bad, it will be a better implementation, let me make a experiment test first

[caiconghui]

I test show data with the total 800 tables in one db with 448852 replicas which cost about only 20ms