diff --git a/schedule-replicas-by-topology-labels.md b/schedule-replicas-by-topology-labels.md index 9604dd74f371d..36dfab793fa12 100644 --- a/schedule-replicas-by-topology-labels.md +++ b/schedule-replicas-by-topology-labels.md @@ -14,9 +14,156 @@ To improve the high availability and disaster recovery capability of TiDB cluste To make this mechanism effective, you need to properly configure TiKV and PD so that the topology information of the cluster, especially the TiKV location information, is reported to PD during deployment. Before you begin, see [Deploy TiDB Using TiUP](/production-deployment-using-tiup.md) first. -## Configure `labels` based on the cluster topology +## Configure `labels` for TiKV, TiFlash, and TiDB -### Configure `labels` for TiKV and TiFlash +You can configure `labels` for TiKV, TiFlash, and TiDB based on the cluster topology. + +### Configure a cluster using TiUP (recommended) + +When using TiUP to deploy a cluster, you can configure the TiKV location in the [initialization configuration file](/production-deployment-using-tiup.md#step-3-initialize-cluster-topology-file). TiUP will generate the corresponding configuration files for TiDB, TiKV, PD, and TiFlash during deployment. + +In the following example, a two-layer topology of `zone/host` is defined. The TiDB nodes, TiKV nodes, and TiFlash nodes of the cluster are distributed among three zones, z1, z2, and z3. + +- In each zone, there are two hosts that have TiDB instances deployed, and each host has a separate TiDB instance deployed. +- In each zone, there are two hosts that have TiKV instances deployed. In z1, each host has two TiKV instances deployed. In z2 and z3, each host has a separate TiKV instance deployed. +- In each zone, there are two hosts that have TiFlash instances deployed, and each host has a separate TiFlash instance deployed. + +In the following example, `tidb-host-machine-n` represents the IP address of the `n`th TiDB node, `tikv-host-machine-n` represents the IP address of the `n`th TiKV node, and `tiflash-host-machine-n` represents the IP address of the `n`th TiFlash node. + +``` +server_configs: + pd: + replication.location-labels: ["zone", "host"] +tidb_servers: +# z1 + - host: tidb-host-machine-1 + config: + labels: + zone: z1 + host: tidb-host-machine-1 + - host: tidb-host-machine-2 + config: + labels: + zone: z1 + host: tidb-host-machine-2 +# z2 + - host: tidb-host-machine-3 + config: + labels: + zone: z2 + host: tidb-host-machine-3 + - host: tikv-host-machine-4 + config: + labels: + zone: z2 + host: tidb-host-machine-4 +# z3 + - host: tidb-host-machine-5 + config: + labels: + zone: z3 + host: tidb-host-machine-5 + - host: tidb-host-machine-6 + config: + labels: + zone: z3 + host: tidb-host-machine-6 +tikv_servers: +# z1 + # machine-1 on z1 + - host: tikv-host-machine-1 + port：20160 + config: + server.labels: + zone: z1 + host: tikv-host-machine-1 + - host: tikv-host-machine-1 + port：20161 + config: + server.labels: + zone: z1 + host: tikv-host-machine-1 + # machine-2 on z1 + - host: tikv-host-machine-2 + port：20160 + config: + server.labels: + zone: z1 + host: tikv-host-machine-2 + - host: tikv-host-machine-2 + port：20161 + config: + server.labels: + zone: z1 + host: tikv-host-machine-2 +# z2 + - host: tikv-host-machine-3 + config: + server.labels: + zone: z2 + host: tikv-host-machine-3 + - host: tikv-host-machine-4 + config: + server.labels: + zone: z2 + host: tikv-host-machine-4 +# z3 + - host: tikv-host-machine-5 + config: + server.labels: + zone: z3 + host: tikv-host-machine-5 + - host: tikv-host-machine-6 + config: + server.labels: + zone: z3 + host: tikv-host-machine-6 + +tiflash_servers: +# z1 + - host: tiflash-host-machine-1 + learner_config: + server.labels: + zone: z1 + host: tiflash-host-machine-1 + - host: tiflash-host-machine-2 + learner_config: + server.labels: + zone: z1 + host: tiflash-host-machine-2 +# z2 + - host: tiflash-host-machine-3 + learner_config: + server.labels: + zone: z2 + host: tiflash-host-machine-3 + - host: tiflash-host-machine-4 + learner_config: + server.labels: + zone: z2 + host: tiflash-host-machine-4 +# z3 + - host: tiflash-host-machine-5 + learner_config: + server.labels: + zone: z3 + host: tiflash-host-machine-5 + - host: tiflash-host-machine-6 + learner_config: + server.labels: + zone: z3 + host: tiflash-host-machine-6 +``` + +For details, see [Geo-distributed Deployment topology](/geo-distributed-deployment-topology.md). + +> **Note:** +> +> If you have not configured `replication.location-labels` in the configuration file, when you deploy a cluster using this topology file, an error might occur. It is recommended that you confirm `replication.location-labels` is configured in the configuration file before deploying a cluster. + +### Configure a cluster using command lines or configuration files + +#### Configure `labels` for TiKV and TiFlash You can use the command-line flag or set the TiKV or TiFlash configuration file to bind some attributes in the form of key-value pairs. These attributes are called `labels`. After TiKV and TiFlash are started, they report their `labels` to PD so users can identify the location of TiKV and TiFlash nodes. @@ -50,7 +197,7 @@ rack = "" host = "" ``` -### (Optional) Configure `labels` for TiDB +#### (Optional) Configure `labels` for TiDB When [Follower read](/follower-read.md) is enabled, if you want TiDB to prefer to read data from the same region, you need to configure `labels` for TiDB nodes. @@ -68,7 +215,7 @@ host = "" > > Currently, TiDB depends on the `zone` label to match and select replicas that are in the same region. To use this feature, you need to include `zone` when [configuring `location-labels` for PD](#configure-location-labels-for-pd), and configure `zone` when configuring `labels` for TiDB, TiKV, and TiFlash. For more details, see [Configure `labels` for TiKV and TiFlash](#configure-labels-for-tikv-and-tiflash). -### Configure `location-labels` for PD +## Configure `location-labels` for PD According to the description above, the label can be any key-value pair used to describe TiKV attributes. But PD cannot identify the location-related labels and the layer relationship of these labels. Therefore, you need to make the following configuration for PD to understand the TiKV node topology. @@ -100,7 +247,7 @@ To configure `location-labels`, choose one of the following methods according to pd-ctl config set location-labels zone,rack,host ``` -### Configure `isolation-level` for PD +## Configure `isolation-level` for PD If `location-labels` has been configured, you can further enhance the topological isolation requirements on TiKV clusters by configuring `isolation-level` in the PD configuration file. @@ -127,115 +274,6 @@ The `location-level` configuration is an array of strings, which needs to corres > > `isolation-level` is empty by default, which means there is no mandatory restriction on the isolation level. To set it, you need to configure `location-labels` for PD and ensure that the value of `isolation-level` is one of `location-labels` names. -### Configure a cluster using TiUP (recommended) - -When using TiUP to deploy a cluster, you can configure the TiKV location in the [initialization configuration file](/production-deployment-using-tiup.md#step-3-initialize-cluster-topology-file). TiUP will generate the corresponding configuration files for TiKV, PD, and TiFlash during deployment. - -In the following example, a two-layer topology of `zone/host` is defined. The TiKV nodes and TiFlash nodes of the cluster are distributed among three zones, z1, z2, and z3. - -- In each zone, there are two hosts that have TiKV instances deployed. In z1, each host has two TiKV instances deployed. In z2 and z3, each host has a separate TiKV instance deployed. -- In each zone, there are two hosts that have TiFlash instances deployed, and each host has a separate TiFlash instance deployed. - -In the following example, `tikv-host-machine-n` represents the IP address of the `n`th TiKV node, and `tiflash-host-machine-n` represents the IP address of the `n`th TiFlash node. - -``` -server_configs: - pd: - replication.location-labels: ["zone", "host"] - -tikv_servers: -# z1 - # machine-1 on z1 - - host: tikv-host-machine-1 - port：20160 - config: - server.labels: - zone: z1 - host: tikv-host-machine-1 - - host: tikv-host-machine-1 - port：20161 - config: - server.labels: - zone: z1 - host: tikv-host-machine-1 - # machine-2 on z1 - - host: tikv-host-machine-2 - port：20160 - config: - server.labels: - zone: z1 - host: tikv-host-machine-2 - - host: tikv-host-machine-2 - port：20161 - config: - server.labels: - zone: z1 - host: tikv-host-machine-2 -# z2 - - host: tikv-host-machine-3 - config: - server.labels: - zone: z2 - host: tikv-host-machine-3 - - host: tikv-host-machine-4 - config: - server.labels: - zone: z2 - host: tikv-host-machine-4 -# z3 - - host: tikv-host-machine-5 - config: - server.labels: - zone: z3 - host: tikv-host-machine-5 - - host: tikv-host-machine-6 - config: - server.labels: - zone: z3 - host: tikv-host-machine-6 - -tiflash_servers: -# z1 - - host: tiflash-host-machine-1 - learner_config: - server.labels: - zone: z1 - host: tiflash-host-machine-1 - - host: tiflash-host-machine-2 - learner_config: - server.labels: - zone: z1 - host: tiflash-host-machine-2 -# z2 - - host: tiflash-host-machine-3 - learner_config: - server.labels: - zone: z2 - host: tiflash-host-machine-3 - - host: tiflash-host-machine-4 - learner_config: - server.labels: - zone: z2 - host: tiflash-host-machine-4 -# z3 - - host: tiflash-host-machine-5 - learner_config: - server.labels: - zone: z3 - host: tiflash-host-machine-5 - - host: tiflash-host-machine-6 - learner_config: - server.labels: - zone: z3 - host: tiflash-host-machine-6 -``` - -For details, see [Geo-distributed Deployment topology](/geo-distributed-deployment-topology.md). - -> **Note:** -> -> If you have not configured `replication.location-labels` in the configuration file, when you deploy a cluster using this topology file, an error might occur. It is recommended that you confirm `replication.location-labels` is configured in the configuration file before deploying a cluster. - ## PD schedules based on topology label PD schedules replicas according to the label layer to make sure that different replicas of the same data are scattered as much as possible.