[Proposal] Making doris stream load more robust with higher concurrent performance

[caiconghui]

The main purpose of this proposal is to make doris stream load more robust with higher concurrent peformance.
Now, in our production environment, we mainly use stream load to load business data into doris.

And we found the following problem:

1. the transaction processing in GlobalTransactionMgr performance is poor, because the transaction lock is global without any isolation, for some operation we need to iterate the whole map.
2. Fe thrift server use Executors.newFixedThreadPool and Executors.newCachedThreadPool to construct thread pool, which may cause oom or too many thread to be created and then crash
3. the rpc cache client num is not limited in be and when rpc client timeout, the request task is still blocked in fe, the client encounter rpc_timeout and retry, it may cause disaster for fe sometimes.
4. the tablet writer open cost too much time, because when user doesn't specify the partition, the all tablet writer would be opened.

Implementation
So the following work we need to implement:

1. Support db level isolation for transaction processing and use ArrayDeque to stored the finished transaction
2. Remove all code that used newFixedThreadPool or newCachedThreadPool, construct the thread pool explicitly, support thread pool metric monitor
3. use client pool to manage rpc client, set the core and max cache client num, use try lock instead of lock to support server task be can be canceled in time due to timeout
4. open the tablet writer when needed not open all when partitions of table are not specified.

[kangkaisen]

@caiconghui Great job!
Could we support table level isolation for transaction processing？ If we only support DB level isolation and one DB has hundreds of tables, the transaction performance will still poor!

[caiconghui]

@kangkaisen I think if we want to support table level isolation, there is still many work to do, because now the meta design for doris is on db level isolation, and this PR not only support db level isolation, but also do some optimization for txn manage task, the transaction performance may not be poor although one DB has hundreds of tables, we still need some performance data to decide to if we need to support table level isolation, we just do it now.

[kangkaisen]

@kangkaisen I think if we want to support table level isolation, there is still many work to do, because now the meta design for doris is on db level isolation, and this PR not only support db level isolation, but also do some optimization for txn manage task, the transaction performance may not be poor although one DB has hundreds of tables, we still need some performance data to decide to if we need to support table level isolation, we just do it now.

OK, When you do all work. would better give a benchmark like #2377. Thanks you.

[kangkaisen]

@morningman Hi, Why do you close this issue? I think there are many work to do about this issue.

[morningman]

@morningman Hi, Why do you close this issue? I think there are many work to do about this issue.

My bad, I linked this issue the the PR #3369, and it automatically closed this issue after the PR
merged. I reopen it.

Aha, @chaoyli dit that, not me. LoL~