30s or 1min slow query on low load cluster

[tiancaiamao]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Found in our customer's environment

In the tidb log, "recycle idle connection" can be found.
The cluster is not busy, but sometime it get slow query even on point get operation.
Typically, the slow query takes 30s ... (which is the timeout retry duration)

Maybe in some cases, the latency of the load schema is also high from grafana metrics.

2. What did you expect to see? (Required)

No unexpected slow query when the cluster is not busy.

3. What did you see instead (Required)

"recycle idle connection" log in TiDB, and also some unexpected slow query...

4. What is your TiDB version? (Required)

release-5.1

[tiancaiamao]

Fixed in 5.2.0
tikv/client-go#292

[tiancaiamao]

There are currently two known cases the bug may trigger:

• In a low load cluster, where "recycle idle connection" can be observed
• In a cluster with TiFlash, and during the TiFlash reboot, the bug may cause TiDB qps drop to 0

[windtalker]

@tiancaiamao if it is already fixed since 5.2.0, can we close this issue?