Support timestamp bounded read-only transactions on data replicas Tasklist

[NingLin-P]

Development Task

Timestamp bounded read-only transactions return stale data within a user-provided staleness bound, and can provide better performance than normal read-only transactions:

• Can perform at replicas that don't have the most up-to-date data, so the ReadIndex request can be avoided and reduce the network latency cost, especially in cross-region deployment.
• With the specified staleness, write-read conflict can be greatly reduced

Issue: #19817
Propoasl: Follower to serve local(stale) read (In Chinese)

New feature

Basic work

• Support timestamp bounded syntax Add Cloud Spanner style read only transaction parser#610
• Get the historical timestamp issues/21522 @Andrewmatilde
• Choose the closest replicas by labels issues/21540 @Yisaer
• Add timestamp bounded option to read requests at kvproto

Timestamp bounded read-only transactions

• Support exact staleness read-only transactions

  • Begin a staleness read-only transaction Begin a staleness read-only transaction #21821 @Yisaer
  • Allocate start timestamps for exact staleness read-only transactions Allocate start timestamps for exact staleness read-only transactions #21717 @Yisaer
  • Validate statements for read-only transactions Validate statements for read-only transactions #21824 @Yisaer
  • Replace the WITH Staleness to AS OF clause Replace the WITH Staleness to AS OF clause #24291 @JmPotato

• Support bounded staleness read-only transactions

  • Maintain a zone-level resolved ts @Yisaer
  • Get start ts from resolved ts and specified timestamp *: implement tidb_bounded_staleness built-in function #24328 @JmPotato
  • Choose replicas by distance @Yisaer

• Compatibility with other features

  • Keep compatible with DDL Make stale read and history read compatible with DDL #22427 P1 @xhebox
  • Option: Keep compatible with prepare Make AsOf compatible with Prepare #24932
  • Keep compatible with plan cache Make plan cache compatible with stale read and history read #22470
  • Keep compatible with coprocessor cache Make stale read and history read compatible with coprocessor cache #22510
  • Keep compatible with SQL binding and SPM
  • Keep compatible with history read (can replace)
  • Keep compatible with hot read region schedule @Yisaer

• Errors handling @JmPotato

  • retry the leader or next peer on error
  • Stale read request may send to a not initailized peer #23271 Stale read request may send to a not initialized peer

Supports SET TRANSACTION READ ONLY AS OF #22765

• Support syntax SET TRANSACTION READ ONLY AS OF @Yisaer
• Implement SET TRANSACTION READ ONLY AS OF executor: implement set transaction read only as of transaction #24766 @Yisaer

Supports Select statement with ASOF #24557

• Support syntax SELECT ... AS OF .. Support syntax SET TRANSACTION READ ONLY AS OF ... and SELECT ... AS OF for Stale read #22765 P1 @nolouch
• Implement Select .... AS OF syntax to support Stale Read planner, executor: supports select statement with AS OF #24613 @nolouch
• Support resolving schema of different tables in different time @xhebox
• Use the matching schema for the select statement @xhebox
• Make sure the special PointGet optimize with usemaxts do not work in stale read
• Option: Support the multiple staled TS in the one statements

TiKV

• Maintain replica's Resolved TS @NingLin-P
  • Decouple Resolved TS from component/cdc @5kbpers
  • Forward replica's Resolved TS when the region is inactive raftstore: forward the replica's safe timestamp tikv/tikv#9384
  • (optional) Forward Resolved TS when there are ongoing large transactions
  • (optional) Forward Resolved TS when there are aborted transactions
• (optional) Maintain replica's Applied TS
  • Forward replica's Applied TS by applying transactional raft logs
  • Forward replica's Applied TS when the region is inactive
  • Keep compatible with Async Commit
• Support handling stale read request on replica
  • Change the replica read process *: support handling stale read request on replica tikv/tikv#9346
  • Refine the handling of EpochNotMatch error
• (optional) "1PC" optimization

TODO:

• Add metric

[5kbpers]

For decoupling Resolved TS, may include the following changes:

• Move the CmdObserver implementation of CdcObserver to resolved ts package
• Start a new thread for capturing changes from apply worker, tracking the lock, and redirecting changes to cdc thread
• Provide an interface (through an atomic variable) for getting the current resolved ts

[nolouch]

Hi, @sunxiaoguang , Is anyone interested in #21522 and #21540？

[Yisaer]

assign Supports SET TRANSACTION READ ONLY AS OF

[JmPotato]

assign Errors handling.

[tisonkun]

Hi @NingLin-P ! I'd like to know that is stale read included in v5.1.0?

[xhebox]

@tisonkun It is in release-5.1 branch, fixing bugs and testing. But you could try if you like.

[tisonkun]

@xhebox thanks!

[yiwu-arbug]

filed tikv/tikv#11015 for adding metrics for stale read