A performance bottleneck when running TPCC workload the ARM servers

[mahjonp]

Performance Questions

• What version of TiDB are you using?

v3.0.7

• What's the observed and your expected performance respectively?

We deploy 1 tidb-server and 3 tikv-server on each arm machine, and have 3 arm machines for test.

Then we use one TPCC client to perform workload on a fixed tidb-server, set 1000 warehouse and add terminals step by step to test the cluster's performance bottleneck. when the terminals was added to 1200 terminals, we found the qps was smaller than 1000 terminals, and what's more strange is that kv cmd OPS became smaller but kv cmd duration was greater than 1000 terminals.

[SunRunAway]

[SunRunAway]

We can see that it is stuck by autoid

----------------------------------------------------------+-------------
      flat  flat%   sum%        cum   cum%   calls calls% + context 	 	 
----------------------------------------------------------+-------------
                                       245263.77ms 41.50% |   github.com/pingcap/tidb/meta/autoid.(*allocator).Alloc /build/tidb/meta/autoid/autoid.go:276
                                       177427.83ms 30.02% |   github.com/pingcap/tidb/statistics/handle.(*Handle).DumpFeedbackToKV /build/tidb/statistics/handle/update.go:405
                                        49682.63ms  8.41% |   github.com/sirupsen/logrus.(*MutexWrap).Unlock /root/go/pkg/mod/github.com/sirupsen/logrus@v1.2.0/logger.go:58
                                        27629.81ms  4.68% |   google.golang.org/grpc/internal/transport.(*controlBuffer).executeAndPut /root/go/pkg/mod/google.golang.org/grpc@v1.17.0/internal/transport/controlbuf.go:274
                                        21699.20ms  3.67% |   sync.(*Map).Store /usr/local/go/src/sync/map.go:162
                                        13526.19ms  2.29% |   github.com/pingcap/goleveldb/leveldb/memdb.(*lockedSource).Int63 /root/go/pkg/mod/github.com/pingcap/goleveldb@v0.0.0-20171020122428-b9ff6c35079e/leveldb/memdb/memdb.go:28
                                        11315.29ms  1.91% |   google.golang.org/grpc/internal/transport.(*recvBuffer).put /root/go/pkg/mod/google.golang.org/grpc@v1.17.0/internal/transport/transport.go:89
                                        10632.77ms  1.80% |   context.(*cancelCtx).Done /usr/local/go/src/context/context.go:330
                                        10063.36ms  1.70% |   context.propagateCancel /usr/local/go/src/context/context.go:256
                                         5788.11ms  0.98% |   google.golang.org/grpc/internal/transport.(*recvBuffer).put /root/go/pkg/mod/google.golang.org/grpc@v1.17.0/internal/transport/transport.go:83
                                         3520.66ms   0.6% |   sync.(*Pool).pinSlow /usr/local/go/src/sync/pool.go:220
                                         2973.78ms   0.5% |   github.com/pingcap/tidb/session.(*domainMap).Get /build/tidb/session/tidb.go:65
                                         2542.88ms  0.43% |   github.com/pingcap/tidb/statistics/handle.(*Handle).dumpTableStatCountToKV /build/tidb/statistics/handle/update.go:345
                                         2088.89ms  0.35% |   math/rand.(*lockedSource).Int63 /usr/local/go/src/math/rand/rand.go:382
                                         1938.28ms  0.33% |   sync.(*Map).Load /usr/local/go/src/sync/map.go:119
                                         1450.65ms  0.25% |   context.removeChild /usr/local/go/src/context/context.go:296
                                         1322.14ms  0.22% |   google.golang.org/grpc/internal/transport.(*controlBuffer).get /root/go/pkg/mod/google.golang.org/grpc@v1.17.0/internal/transport/controlbuf.go:308
                                          789.88ms  0.13% |   github.com/pingcap/tidb/sessionctx/stmtctx.(*StatementContext).MergeExecDetails /build/tidb/sessionctx/stmtctx/stmtctx.go:439
                                          628.35ms  0.11% |   sync.(*Pool).pinSlow /usr/local/go/src/sync/pool.go:230
                                          474.78ms  0.08% |   sync.(*RWMutex).Unlock /usr/local/go/src/sync/rwmutex.go:136
                                          161.47ms 0.027% |   sync.(*Map).Delete /usr/local/go/src/sync/map.go:277
                                           16.70ms 0.0028% |   google.golang.org/grpc.(*clientStream).withRetry /root/go/pkg/mod/google.golang.org/grpc@v1.17.0/stream.go:566
                                           16.46ms 0.0028% |   github.com/pingcap/tidb/statistics/handle.(*Handle).Update /build/tidb/statistics/handle/handle.go:177
                                           10.60ms 0.0018% |   context.(*cancelCtx).cancel /usr/local/go/src/context/context.go:378
                                            9.62ms 0.0016% |   google.golang.org/grpc/internal/transport.(*recvBuffer).load /root/go/pkg/mod/google.golang.org/grpc@v1.17.0/internal/transport/transport.go:102
                                            8.79ms 0.0015% |   github.com/pingcap/tidb/statistics/handle.(*Handle).columnStatsFromStorage /build/tidb/statistics/handle/handle.go:417
                                            5.91ms 0.001% |   github.com/pingcap/tidb/store/tikv/latch.(*Latches).releaseSlot /build/tidb/store/tikv/latch/latch.go:196
                                            3.89ms 0.00066% |   google.golang.org/grpc/internal/transport.(*controlBuffer).get /root/go/pkg/mod/google.golang.org/grpc@v1.17.0/internal/transport/controlbuf.go:316
                                            1.89ms 0.00032% |   google.golang.org/grpc/internal/transport.(*controlBuffer).get /root/go/pkg/mod/google.golang.org/grpc@v1.17.0/internal/transport/controlbuf.go:312
                                            1.48ms 0.00025% |   github.com/pingcap/tidb/store/tikv/latch.(*Latches).acquireSlot /build/tidb/store/tikv/latch/latch.go:264
                                            0.46ms 7.8e-05% |   google.golang.org/grpc/internal/transport.(*controlBuffer).finish /root/go/pkg/mod/google.golang.org/grpc@v1.17.0/internal/transport/controlbuf.go:345
                                            0.44ms 7.4e-05% |   log.(*Logger).Output /usr/local/go/src/log/log.go:173
                                            0.39ms 6.5e-05% |   github.com/pingcap/tidb/statistics/handle.(*Handle).UpdateStatsByLocalFeedback /build/tidb/statistics/handle/update.go:423
                                            0.04ms 6.5e-06% |   github.com/pingcap/tidb/store/tikv/latch.(*Latches).acquireSlot /build/tidb/store/tikv/latch/latch.go:259
                                            0.03ms 4.8e-06% |   google.golang.org/grpc.(*addrConn).createTransport /root/go/pkg/mod/google.golang.org/grpc@v1.17.0/clientconn.go:1247
                                            0.02ms 4.1e-06% |   github.com/pingcap/tidb/store/tikv/latch.(*Latches).acquireSlot /build/tidb/store/tikv/latch/latch.go:253
                                            0.02ms 3.3e-06% |   github.com/pingcap/tidb/statistics/handle.(*Handle).indexStatsFromStorage /build/tidb/statistics/handle/handle.go:374
590997.44ms 99.91% 99.91% 590997.44ms 99.91%                | sync.(*Mutex).Unlock /usr/local/go/src/sync/mutex.go:190

[Deepak275]

@SunRunAway Forgive me for the little knowledge, can you please tell me which version of tidb supports arm64 architecture?
Thanks in advance :)

[SunRunAway]

@SunRunAway Forgive me for the little knowledge, can you please tell me which version of tidb supports arm64 architecture?
Thanks in advance :)

@alex-quan-001 Cloud you answer this question?

[martin-g]

I experience tests failures on Linux ARM64 even on latest master: #23552

[alex-quan-001]

@SunRunAway Forgive me for the little knowledge, can you please tell me which version of tidb supports arm64 architecture?
Thanks in advance :)

@alex-quan-001 Cloud you answer this question?

The official website announces that it is supported, but we are currently only running it once before the release of the release branch. master branch is not yet integrated into ci cross-platform support. We are planning and requesting resources for master ci cross-platform support.

[martin-g]

@alex-quan-001 What CI system do you use ?
I work for OpenLab Testing and we could provide you a Linux ARM64 VM that you could use as a build agent for your CI.
If you are interested then please contact me at mgrigorov @ apache . org.

[Deepak275]

oh okay, thanks @alex-quan-001 @SunRunAway for info..

[jebter]

You can ask new questions at https://ask.pingcap.com/ or https://asktug.com/ (Chinese). I will close this issue. If there are any updates, you can reopen it.