-
Notifications
You must be signed in to change notification settings - Fork 720
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize the performance of GetTS #1847
Comments
@nolouch
on different terminal:
I tried with 2 different go-version I.e. 1.13.3 & 1.12.1 Can you specify the RAM and number of CPUs used for the setup to run? I will try to it in container with limits to reproduce. |
Hi @mgkanani, Thank you for your attention. |
@nolouch , I tried few options. The issue stopped occurring when GC is turned off and invoked only per Second. The reasoning behind so:
Let's assume below steps:
So it requires 2 GC() invocations to clear unused pool memory. Changes I made to disable GC() and trigger only per second:
command to run:
PS: |
Amazing! @mgkanani that's an awesome work and your point is fantastic, PTAL @disksing @tiancaiamao
the numbers in the last few lines of this details expose the problem. The main problem we want to solve: When
It's very easy to reproduce:
Do you have any ideas? |
|
@nolouch @tiancaiamao , In my changes, I have chosen the default Array size based on metrics data. During my investigations, I found that the gRPC-server who handles TSO request never reported latency concerns(partially due to batching at client side before requesting to pd-server).
Based on these data, we can scale/shard pd-cluster and tidb cluster whichever could be bottleneck. |
thanks @mgkanani. |
TL;DR As per current implementation:Long running routines:
Bottlenecks:
Proposal:tsLoop being a consumer, should never wait for lock as long as data is available.
I am thinking of trying above approach and comparing it with channels via benchmarks. Tradeoffs:
|
A kind remind: the benchmark program may not reflect the real case somehow. In the benchmark, the batch size can be quite large, producers call And another factor is the goroutine count on the scheduler. |
I agree with your remarks. With sysbench, I didn't see data which tells Golang follows cooperative and fair scheduling principles. Even if, we able schedule tsLoop when we desired, it will not solve the latency problem. It is better to create separate Issue for TiDB-repo and close this. In TiDB, we should look for reducing unnecessary cpu-intensive go-routines. |
Hi @mgkanani |
Description
Obtaining TSO is a key step in the transaction. Improving the performance of
GetTS
can improve the performance of the entire cluster. One of the more obvious problems is that when the client is busy, getting TSO has a significant long tail problem. We have a benchmark clientbench-tso
in PD project, and you can run it with 10000 concurrency. Example output:The requests greater than 30ms occur periodically.
Goals:
Difficulty
Score
Mentor(s)
Recommended Skills
The text was updated successfully, but these errors were encountered: