Local reads via Stale Reads do not work

[Tema]

Bug Report

What version of TiDB Operator are you using?
v1.3.5
TidbCluster kind v6.1.0

What did you do?
I try to perform local reads using Stale Reads described in https://docs.pingcap.com/tidb/stable/three-dc-local-read#perform-local-read-using-stale-read, but TIDB sends commands to all tikv across all zones.

What did you expect to see?
Per documentation it should send requests to TiKV in the same zone

I use the following PD config:

      [replication]
        max-replicas = 3
        location-labels = ["topology.kubernetes.io/zone", "kubernetes.io/hostname"]

I also see that corresponding labels are assigned to TiKV nodes:

[2022/07/14 00:45:20.651 +00:00] [INFO] [replication_mode.rs:55] ["associated store labels"] [labels="[key: \"kubernetes.io/hostname\" value: \"gke-non-mesh-tidb-pe-basic-tikv-n2d-8-ab73507c-7tqz\", key: \"topology.kubernetes.io/zone\" value: \"us-east4-b\"]"] [store_id=1]
[2022/07/14 00:45:20.651 +00:00] [INFO] [replication_mode.rs:55] ["associated store labels"] [labels="[key: \"kubernetes.io/hostname\" value: \"gke-non-mesh-tidb-pe-basic-tikv-n2d-8-ac1d383d-09th\", key: \"topology.kubernetes.io/zone\" value: \"us-east4-c\"]"] [store_id=4]
[2022/07/14 00:45:20.651 +00:00] [INFO] [replication_mode.rs:55] ["associated store labels"] [labels="[key: \"kubernetes.io/hostname\" value: \"gke-non-mesh-tidb-pe-basic-tikv-n2d-8-753c179c-tm45\", key: \"topology.kubernetes.io/zone\" value: \"us-east4-a\"]"] [store_id=5]

However, I don't see same labels assigned to TiDB nodes, hence the feature does not work.

[Ishiihara]

@DanielZhangQD Can you help take a look?

[Ishiihara]

@Tema What are the labels for TiDB nodes?

[Tema]

@Tema What are the labels for TiDB nodes?

I don't see any. This is a problem. I expect operator to assign the same labels to TiDB as it does for TiKV nodes based on k8s "topology.kubernetes.io/zone" label (see log captured in the issue description) , so that we can leverage stale reads.

[zimulala]

@Tema What are the labels for TiDB nodes?

There is no documentation at present, please refer to the example in the test: https://github.com/pingcap/tidb/pull/19212/files#diff-373f5578b8f362a364b50b0d01919a57da64c128547ca10d49df8ae422237c82R215

In addition, how to configure the TiKV labels, refer to: https://docs.pingcap.com/tidb/stable/schedule-replicas-by-topology-labels#configure-labels-based-on-the-cluster-topology

[DanielZhangQD]

Adding label for TiDB Pods is not supported yet.

[zhangjinpeng87]

After the following PR landed and released, this issue can be fixed.
pingcap/tidb#36845
#4663