server: enhancement of ts validity check for all types of requests

[cfzjywxk]

Enhancement

Currently, some KV request-related and timestamp-related interfaces are directly exposed to the upper-layer TiDB. This can lead to issues due to inappropriate usages, including:

1. Incomplete KV Request Error Handling:
   For example, inadequate handling of various KV error types, such as backoff, retries, and others.

2. Incorrect Manually Specified Timestamps:
   Providing timestamps that exceed the maximum pre-allocated by PD can break the linearizability of the entire cluster.

Similar issues have occurred, such as:

• [Issue #57769](server: using of timestamps not allocated by PD for reads can break the cluster's linearizability constraints #57769)
• [Issue #56809](SELECT ... AS OF TIMESTAMP statement with explicit time may break linearizability when TSO is drifting #56809)

These problems highlight the need to refine these interfaces and enforce stricter controls.

For internal task execution, internal sessions and internal transactions should be used to operate, avoiding directly assembling and sending KV requests.

Possible enhancements

1. Restrict the interfaces of distsql and its executorBuilder. For exmaple, a executor builder offer interface to set the startTS without any reminding and checking https://github.com/pingcap/tidb/blob/master/pkg/distsql/request_builder.go#L251.
2. Restrict the usages of Storage interfaces. For example, a snapshot could be generated by any input timestampes https://github.com/pingcap/tidb/blob/master/pkg/kv/kv.go#L695.
3. Adding ts validity check in both tidb and tikv. For example, checking if the being used timestamp is within the gap of some configured allowed or tolerated clockdifft, return error if it's a problematic ts.

There are mainly 3 interfaces provided for the usages in tidb from the persipective of kv-client and storage

• The Storage interface, it exposes the Snapshot(TS) interface
• The DistSQL interface, it exposes the SetStartTS interface
• The Transaction interface

Detailed tasks

TiDB

• Deprecate the exposed [SetStartTS](https://github.com/pingcap/tidb/blob/master/pkg/distsql/request_builder.go#L251) interface. Internal tasks should no longer set any timestamp value directly. Instead, provide options such as AllocatedTS or MaxUint64TS. For DAG builders within a transaction, introduce a builderWithTxn interface to enforce using the activated transaction's startTS.
• Add constraint checks to ensure only PD-allocated timestamps or MaxUint64 are used when fetching snapshots through the Storage.Snapshot interface.
• Unify timestamp validation logic for tidb_snapshot and AS OF TIMESTAMP features related to stale reads.
• (Long term) Abstract a TimeStamp type similar to TiKV's implementation, adding a field to indicate the source of the timestamp (e.g., whether it is allocated by the PD server or calculated).

Client-go

• Try to validate read ts for all RPC requests tikv/client-go#1513 Implement a timestamp validation mechanism to ensure the read timestamp is valid for all KV read request types(actually write request types are needed too). If the condition is violated, make the TiDB server panic (or use log.Fatal) and provide detailed source request information.
  • For new releases, consider adding a configuration option or system variable to set the allowable gap.
  • For stable releases, log errors with detailed debug information instead of triggering a panic.

TiKV

• Enhancement: Add timestamp safety boundary to prevent unreasonable max_ts updates tikv/tikv#17916
  Add a timestamp validation mechanism to ensure that updates to the max_ts value remain within an allowable gap from the PD-allocated timestamp. If the gap is exceeded, make the TiKV server panic and log the source request details.
  • For stable releases, log detailed error information rather than panicking.
• (Long term) Extend the TimeStamp type to include a field indicating its source. This field can be used to validate whether the timestamp is eligible for updating the max_ts.

[cfzjywxk]

The bug label is used to pick these check enhancements to LTS release versions.