Configurable KV Timeout Tracking Issue

[cfzjywxk]

Background

• As described in the proposal, to mitigate the impact of latency jitter encountered by the tikv nodes.
• The related feature request issue.

Tasks

TiDB

• Add new session variable tidb_kv_read_timeout. kv request: configurable KV Timeout  #45601
• Support hint processing for tidb_kv_read_timeout. When parsing and processing user hints in the ExtractTableHintsFromStmtNode and handleStmtHints functions, convert the input hint value into the corresponding field of StmtContext so it can be used later. kv request: configurable KV Timeout  #45601
• Add the KVReadTimeout field in the StatementContext, it could be set from the session variable or the hint value. kv request: configurable KV Timeout  #45601
• Support timeout processing logic in the pointGetExecutor and batchPointGetExecutor, setting the timeout value to the underlying KVSnapshot which is used by the executors.kv request: configurable KV Timeout  #45601
• Support timeout processing logic in the copTask, setting the timeout value to the Context.max_execution_duration_ms field when building coprocessor kv-reuqests. kv request: configurable KV Timeout  #45601

KV-Client

• Add set interface SetKVReadTimeout and get interface GetKVReadTimeout for KVSnapshot. So the executors in TiDB could use them to get or set timeout values. Setting the tidb_kv_read_timeout in the Context.max_execution_duration_ms when get an batchGet are used. use tidb_kv_read_timeout as first kv request timeout tikv/client-go#919
• Support specific timeout retry strategy in the KV-Client for the tidb_kv_read_timeout. When a timeout happens, retry the next available peer for the read requests like stale read and non-leader-only snapshot read requests. The strategy is:
  • Create a specific error type for this kind of timeout, for example, ConfigurableReadTimeoutErr. It should not be mixed with the existing RPCError. use tidb_kv_read_timeout as first kv request timeout tikv/client-go#919
  • Try the next peer if the first request times out immediately. use tidb_kv_read_timeout as first kv request timeout tikv/client-go#919
  • If all the responses from all the peers are timeout errors, return timeout error to the upper layer, Backoff with the timeout error, and retry the requests with the original default ReadShortTimeout and ReadMediumTimeout values. Continue to process like before. use tidb_kv_read_timeout as first kv request timeout tikv/client-go#919
  • The timeout retry details should be recorded and handled properly in the request tracker, and so are the metrics.
  • Add related retry and statement management logic for KVSnapshot and ClientHelper.

TiKV

• Support timeout check during the request handling in TiKV. When there's new point get and batch point get
  requests are received, the kv_read_timeout value should be read from the GetReuqest and BatchGetRequest
  requests and passed to the pointGetter. By now there's no canceling mechanism when the task is scheduled to the
  read pool in TiKV, a simpler way is to check the deadline duration before the next processing and try to return
  directly if the deadline is exceeded, but once the reading task is spawned to the read pool it could not be
  canceled anymore. storage: add deadline check in get and batch_get request handler tikv/tikv#15307
• Support Timeout Configuration For Coprocessor Tasks. Use the kv_read_timeout value passed in to calculate the deadline result in parse_and_handle_unary_request instead of using the default one.
  • Coprocessor task already support kv_read_timeout.

[cfzjywxk]

/cc @Tema @hihihuhu @crazycs520

[hihihuhu]

i could first work on Add new session variable tidb_kv_read_timeout

[Tema]

@cfzjywxk can you create a dev branch for it where we can collaborate?

[cfzjywxk]

@Tema
The tidb development branch: https://github.com/pingcap/tidb/tree/dev_tidb_kv_read_timeout
The client-go development branch: https://github.com/tikv/client-go/tree/dev_tidb_kv_read_timeout
/cc @hihihuhu @crazycs520

[Tema]

@cfzjywxk could you please take a look at tikv/client-go#917. This will help us to work independently on tidb and client-go repos as we discussed.

[cfzjywxk]

@hihihuhu @Tema
I previously named the branch in a way that doesn't comply with CI standards, which caused CI triggers to malfunction. I have now renamed the branch to feature/dev_tidb_kv_read_timeout. Please take a look.