tools: add two commands in tikv-ctl

tools/tikv-control.md

@@ -6,7 +6,9 @@ category: tools
 
 # TiKV Control User Guide
 
-TiKV Control (`tikv-ctl`) is a command line tool of TiKV, used to manage the cluster. When you compile TiKV, the `tikv-ctl` command is also compiled at the same time. If the cluster is deployed using Ansible, the `tikv-ctl` binary file exists in the corresponding `tidb-ansible/resources/bin` directory. If the cluster is deployed using the binary, the `tikv-ctl` file is in the `bin` directory together with other files such as `tidb-server`, `pd-server`, `tikv-server`, etc.
+TiKV Control (`tikv-ctl`) is a command line tool of TiKV, used to manage the cluster.
+
+When you compile TiKV, the `tikv-ctl` command is also compiled at the same time. If the cluster is deployed using Ansible, the `tikv-ctl` binary file exists in the corresponding `tidb-ansible/resources/bin` directory. If the cluster is deployed using the binary, the `tikv-ctl` file is in the `bin` directory together with other files such as `tidb-server`, `pd-server`, `tikv-server`, etc.
 
 ## General options
 
@@ -157,17 +159,6 @@ success!
 > - This command only supports the local mode.
 > - The argument of the `--pd/-p` option specifies the PD endpoints without the `http` prefix. Specifying the PD endpoints is to query whether PD can securely switch to Tombstone. Therefore, before setting a PD instance to Tombstone, you need to take off the corresponding Peer of this Region on the machine in `pd-ctl`.
 
-### Force Region to recover the service from multiple replicas failure
-
-Use the `unsafe-recover remove-fail-stores` command to remove the failed machines from the peers list of all Regions. Then after you restart TiKV, these Regions can continue to provide services using the other healthy replicas. This command is usually used in circumstances where multiple TiKV stores are damaged or deleted.
-
-```bash
-$ tikv-ctl --db /path/to/tikv/db unsafe-recover remove-fail-stores 3,4,5
-success!
-```
-
-> **Note:** This command only supports the local mode. It prints `success!` when successfully run.
-
 ### Send a `consistency-check` request to TiKV
 
 Use the `consistency-check` command to execute a consistency check among replicas in the corresponding Raft of a specific Region. If the check fails, TiKV itself panics. If the TiKV instance specified by `--host` is not the Region leader, an error is reported.
@@ -231,3 +222,36 @@ success!
 $ tikv-ctl modify-tikv-config -m raftdb -n default.disable_auto_compactions -v true
 success!
 ```
+
+### Force Region to recover the service from failure of multiple replicas
+
+Use the `unsafe-recover remove-fail-stores` command to remove the failed machines from the peer list of Regions. Then after you restart TiKV, these Regions can continue to provide services using the other healthy replicas. This command is usually used in circumstances where multiple TiKV stores are damaged or deleted.
+
+The `--stores` option accepts multiple `store_id` separated by comma and uses the `--regions` flag to specify involved Regions. Otherwise, all Regions' peers located on these stores will be removed by default.
+
+```bash
+$ tikv-ctl --db /path/to/tikv/db unsafe-recover remove-fail-stores --stores 3 --regions 1001,1002
+success!
+```
+
+> **Note:**
+> 
+> - This command only supports the local mode. It prints `success!` when successfully run.
+> - You must run this command for all stores that specify Regions' peers. If `--regions` is not set,all Regions are involved, and you need to run this command for all stores.
+
+### Recover from MVCC data corruption
+
+Use the `recover-mvcc` command in circumstances where TiKV cannot run normally caused by MVCC data corruption. It cross-checks 3 CFs ("default", "write", "lock") to recover from various kinds of inconsistency.
+
+Use the `--regions` option to specify involved Regions by `region_id`. Use the `--pd`option to specify PD endpoints.
+
+```bash
+$ tikv-ctl --db /path/to/tikv/db recover-mvcc --regions 1001,1002 --pd 127.0.0.1:2379
+success!
+```
+
+> **Note**:
+> 
+> - This command only supports the local mode. It prints `success!` when successfully run.
+> - The argument of the `--pd/-p` option specifies the PD endpoints without the `http` prefix. Specifying the PD endpoints is to query whether the specified `region_id` is validated or not.
+> - You need to run this command for all stores that specify Regions' peers.