Vreplication docs: MoveTables and Resharding

content/en/docs/concepts/move-tables.md

@@ -0,0 +1,26 @@
+---
+title: MoveTables
+---
+
+MoveTables is a new workflow based on VReplication. It enables you to relocate tables between keyspaces without downtime.
+
+## Identifying Candidate Tables
+
+It is recommended to keep tables that need to join on eachother in the same keyspace, so typical candidates for a MoveTables operation are a set of tables which logically group together or are otherwise isolated.
+
+If you have multiple groups of tables as candidates, which makes the most sense to move may depend on the specifics of your environment. For example, a larger table will take more time to move, but in doing so you might also be able to migrate to newer hardware which has more headroom before you need to perform additional operations such as sharding.
+
+Similarly, tables that are updated at a more frequent rate could increase the move time.
+
+### Impact to Production Traffic
+
+Internally, a MoveTables operation is comprised of both a table copy and a subscription to all changes made to the table. Vitess uses batching to improve the performance of both table copying and applying subscription changes, but you should expect that tables with lighter modification rates to move faster.
+
+During the active move process, data is copied from replicas instead of the master server. This helps ensure minimal production traffic impact.
+
+During the `SwitchWrites` phase of the MoveTables operation, Vitess may be briefly unavailable. This unavailability is usually a few seconds, but will be higher in the event that your system has a high replication delay.
+
+
+**Related Vitess Documentation**
+
+* [MoveTables User Guide](../../user-guides/move-tables)

content/en/docs/concepts/shard.md

@@ -8,9 +8,39 @@ Each MySQL instance within a shard has the same data (excepting some replication
 
 An unsharded keyspace has effectively one shard. Vitess names the shard `0` by convention. When sharded, a keyspace has `N` shards with non-overlapping data.
 
+## Shard Naming
+
+Shard names have the following characteristics:
+
+* They represent a range, where the left number is included, but the right is not.
+* Their notation is hexadecimal.
+* They are left justified.
+* A `-` prefix means: anything less than the right value.
+* A `-` postfix means: anything greater than or equal to the LHS value.
+* A plain `-` denotes the full keyrange.
+
+Thus: `-80` == `00-80` == `0000-8000` == `000000-800000`
+
+`80-` is not the same as `80-FF`. This is why:
+
+`80-FF` == `8000-FF00`. Therefore `FFFF` will be out of the `80-FF` range.
+
+`80-` means: ‘anything greater than or equal to `0x80`
+
+A `hash` vindex produces an 8-byte number. This means that all numbers less than `0x8000000000000000` will fall in shard `-80`. Any number with the highest bit set will be >= `0x8000000000000000`, and will therefore belong to shard `80-`.
+
+This left-justified approach allows you to have keyspace ids of arbitrary length. However, the most significant bits are the ones on the left.
+
+For example an `md5` hash produces 16 bytes. That can also be used as a keyspace id.
+
+A `varbinary` of arbitrary length can also be mapped as is to a keyspace id. This is what the `binary` vindex does.
+
 ## Resharding
 
-Vitess supports [dynamic resharding](../../reference/sharding#resharding), in which the number of shards is changed on a live cluster. This can be either splitting one or more shards into smaller pieces, or merging neighboring shards into bigger pieces.
+Vitess supports [resharding](../../user-guides/resharding), in which the number of shards is changed on a live cluster. This can be either splitting one or more shards into smaller pieces, or merging neighboring shards into bigger pieces.
+
+During resharding, the data in the source shards is copied into the destination shards, allowed to catch up on replication, and then compared against the original to ensure data integrity. Then the live serving infrastructure is shifted to the destination shards, and the source shards are deleted.
 
-During dynamic resharding, the data in the source shards is copied into the destination shards, allowed to catch up on replication, and then compared against the original to ensure data integrity. Then the live serving infrastructure is shifted to the destination shards, and the source shards are deleted.
+**Related Vitess Documentation**
 
+* [Resharding User Guide](../../user-guides/resharding)

content/en/docs/get-started/local.md

@@ -56,7 +56,7 @@ sudo setenforce 0
 
 ## Install Vitess
 
-Download the [latest binary release](https://github.com/vitessio/vitess/releases) for Vitess on Linux. For example with Vitess 5.0:
+Download the [latest binary release](https://github.com/vitessio/vitess/releases) for Vitess on Linux. For example with Vitess 5:
 
 ```
 tar -xzf vitess-5.20+20200204-17a806ae5.tar.gz
@@ -87,43 +87,51 @@ You should see output similar to the following:
 
 ```
 ~/my-vitess-example> ./101_initial_cluster.sh
-enter etcd2 env
+$ ./101_initial_cluster.sh 
 add /vitess/global
 add /vitess/zone1
 add zone1 CellInfo
 etcd start done...
-enter etcd2 env
 Starting vtctld...
-Access vtctld web UI at http://morgox1:15000
-Send commands with: vtctlclient -server morgox1:15999 ...
-enter etcd2 env
 Starting MySQL for tablet zone1-0000000100...
-Starting MySQL for tablet zone1-0000000101...
-Starting MySQL for tablet zone1-0000000102...
 Starting vttablet for zone1-0000000100...
-Access tablet zone1-0000000100 at http://morgox1:15100/debug/status
+HTTP/1.1 200 OK
+Date: Wed, 25 Mar 2020 17:32:45 GMT
+Content-Type: text/html; charset=utf-8
+
+Starting MySQL for tablet zone1-0000000101...
 Starting vttablet for zone1-0000000101...
-Access tablet zone1-0000000101 at http://morgox1:15101/debug/status
+HTTP/1.1 200 OK
+Date: Wed, 25 Mar 2020 17:32:53 GMT
+Content-Type: text/html; charset=utf-8
+
+Starting MySQL for tablet zone1-0000000102...
 Starting vttablet for zone1-0000000102...
+HTTP/1.1 200 OK
+Date: Wed, 25 Mar 2020 17:33:01 GMT
+Content-Type: text/html; charset=utf-8
+
+W0325 11:33:01.932674   16036 main.go:64] W0325 17:33:01.930970 reparent.go:185] master-elect tablet zone1-0000000100 is not the shard master, proceeding anyway as -force was used
+W0325 11:33:01.933188   16036 main.go:64] W0325 17:33:01.931580 reparent.go:191] master-elect tablet zone1-0000000100 is not a master in the shard, proceeding anyway as -force was used
 ..
 ```
 
 You can also verify that the processes have started with `pgrep`:
 
 ```
 ~/my-vitess-example> pgrep -fl vtdataroot
-26563 etcd
-26626 vtctld
-26770 mysqld_safe
-26771 mysqld_safe
-26890 mysqld_safe
-29910 mysqld
-29925 mysqld
-29945 mysqld
-30035 vttablet
-30036 vttablet
-30037 vttablet
-30218 vtgate
+14119 etcd
+14176 vtctld
+14251 mysqld_safe
+14720 mysqld
+14787 vttablet
+14885 mysqld_safe
+15352 mysqld
+15396 vttablet
+15492 mysqld_safe
+15959 mysqld
+16006 vttablet
+16112 vtgate
 ```
 
 _The exact list of processes will vary. For example, you may not see `mysqld_safe` listed._
@@ -143,7 +151,7 @@ You should now be able to connect to the VTGate server that was started in `101_
 ~/my-vitess-example> mysql -h 127.0.0.1 -P 15306
 Welcome to the MySQL monitor.  Commands end with ; or \g.
 Your MySQL connection id is 1
-Server version: 5.5.10-Vitess (Ubuntu)
+Server version: 5.7.9-Vitess (Ubuntu)
 
 Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
 
@@ -179,8 +187,8 @@ Repeating the previous step, you should now be able to use the `mysql` client wi
 ```
 ~/my-vitess-example> mysql
 Welcome to the MySQL monitor.  Commands end with ; or \g.
-Your MySQL connection id is 1
-Server version: 5.5.10-Vitess (Ubuntu)
+Your MySQL connection id is 2
+Server version: 5.7.9-Vitess (Ubuntu)
 
 Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
 

content/en/docs/user-guides/horizontal-sharding.md

@@ -1,10 +1,14 @@
 ---
 title: Horizontal Sharding
-weight: 7
+weight: 99
 ---
 
+{{< warning >}}
+In Vitess 6, Horizontal Sharding became obsolete with the introduction of Resharding! It is recommended to skip this guide, and continue on with the [resharding user guide](../resharding) instead.
+{{< /warning >}}
+
 {{< info >}}
-This guide follows on from [Vertical Split](../../user-guides/vertical-split) and [Get Started with a Local deployment](../../get-started/local). It assumes that several scripts have been executed, and you have a running Vitess cluster.
+This guide follows on from [Vertical Split](../../user-guides/vertical-split) and [Get Started with a Local deployment](../../get-started/local). It assumes that several scripts have been executed, and that you have a running Vitess cluster.
 {{< /info >}}
 
 The DBAs you hired with massive troves of hipster cash are pinging you on Slack and are freaking out. With the amount of data that you’re loading up in your keyspaces, MySQL performance is starting to tank - it’s okay, you’re prepared for this! Although the query guardrails and connection pooling are cool features that Vitess can offer to a single unsharded keyspace, the real value comes into play with horizontal sharding.

content/en/docs/user-guides/move-tables.md

@@ -0,0 +1,185 @@
+---
+title: MoveTables
+weight: 6
+---
+
+{{< info >}}
+This guide follows on from [get started with a local deployment](../../get-started/local). It assumes that the `./101_initial_cluster.sh` script has been executed, and that you have a running Vitess cluster.
+{{< /info >}}
+
+[MoveTables](../../concepts/move-tables) is a new VReplication workflow in Vitess 6, and obsoletes Vertical Split from earlier releases.
+
+This feature enables you to move a subset of tables between keyspaces without downtime. For example, after [Initially deploying Vitess](../../get-started/local), your single commerce schema may grow so large that it needs to be split into multiple keyspaces.
+
+As a stepping stone towards splitting a single table across multiple servers (sharding), it usually makes sense to first split from having a single monolithic keyspace (`commerce`) to having multiple keyspaces (`commerce` and `customer`). For example, in our ecommerce system we know that `customer` and `corder` tables are closely related and growing at a high rate just by themselves.
+
+Let's start by simulating this situation by loading sample data:
+
+```sql
+mysql < ../common/insert_commerce_data.sql
+```
+
+We can look at what we just inserted:
+
+```sh
+mysql --table < ../common/select_commerce_data.sql
+Using commerce/0
+Customer
++-------------+--------------------+
+| customer_id | email              |
++-------------+--------------------+
+|           1 | alice@domain.com   |
+|           2 | bob@domain.com     |
+|           3 | charlie@domain.com |
+|           4 | dan@domain.com     |
+|           5 | eve@domain.com     |
++-------------+--------------------+
+Product
++----------+-------------+-------+
+| sku      | description | price |
++----------+-------------+-------+
+| SKU-1001 | Monitor     |   100 |
+| SKU-1002 | Keyboard    |    30 |
++----------+-------------+-------+
+COrder
++----------+-------------+----------+-------+
+| order_id | customer_id | sku      | price |
++----------+-------------+----------+-------+
+|        1 |           1 | SKU-1001 |   100 |
+|        2 |           2 | SKU-1002 |    30 |
+|        3 |           3 | SKU-1002 |    30 |
+|        4 |           4 | SKU-1002 |    30 |
+|        5 |           5 | SKU-1002 |    30 |
++----------+-------------+----------+-------+
+```
+
+Notice that we are using keyspace `commerce/0` to select data from our tables.
+
+## Planning to Move Tables
+
+In this scenario, we are going to split the `commerce` keyspace into `commerce` and `customer` keyspaces. The tables `Customer` and `COrder` will be moved into the newly created keyspace, and the `Product` table will remain in the `commerce` keyspace. This operation is online, which means that it does not block either read or write operations to the tables, __except__ for a small window during the final cut-over.
+
+## Create new tablets
+
+The first step in our MoveTables operation is to deploy new tablets for our `customer` keyspace. By convention, we are going to use the UIDs 200-202 as the `commerce` keyspace previously used `100-102`. Once the tablets have started, we can force the first tablet to be the master using the `-force` flag:
+
+```sh
+# Example 201_customer_tablets.sh
+
+source ./env.sh
+
+for i in 200 201 202; do
+ CELL=zone1 TABLET_UID=$i ./scripts/mysqlctl-up.sh
+ CELL=zone1 KEYSPACE=customer TABLET_UID=$i ./scripts/vttablet-up.sh
+done
+
+vtctlclient -server localhost:15999 InitShardMaster -force customer/0 zone1-200
+vtctlclient -server localhost:15999 ApplyVSchema -vschema '{ "tables": { "product": {} } }' commerce
+vtctlclient -server localhost:15999 ApplyVSchema -vschema '{ "tables": { "customer": {}, "corder": {} } }' customer
+```
+
+The last two commands here set the VSchema. In our case, we need to tell VTGate that the `customer` keyspace will now contain the tables `customer` and `corder`. The `commerce` keyspace will continue to hold just the `Product` table.
+
+__Note:__ This change does not change the actual routing yet. We will use a _switch_ directive to achieve that shortly.
+
+## Start the Move
+
+In this step we will initiate the MoveTables, which copies tables from the commerce keyspace into customer. This operation does not block any database activity; the MoveTables operation is performed online:
+
+```sh
+# Example 202_move_tables.sh
+
+vtctlclient \
+    -server localhost:15999 \
+    -log_dir "$VTDATAROOT"/tmp \
+    -alsologtostderr \
+    MoveTables \
+    -workflow=commerce2customer \
+    commerce customer customer,corder
+
+sleep 2
+```
+
+## Phase 1: Switch Reads
+
+Once the MoveTables operation is complete, the first step in making the changes live is to _switch_ `SELECT` statements to read from the new keyspace. Other statements will continue to route to the `commerce` keyspace. By staging this as two operations, Vitess allows you to test the changes and reduce the associated risks. For example, you may have a different configuration of hardware or software on the new keyspace.
+
+```sh
+# Example 203_switch_reads.sh
+
+vtctlclient \
+ -server localhost:15999 \
+ -log_dir "$VTDATAROOT"/tmp \
+ -alsologtostderr \
+ SwitchReads \
+ -tablet_type=rdonly \
+ customer.commerce2customer
+
+vtctlclient \
+ -server localhost:15999 \
+ -log_dir "$VTDATAROOT"/tmp \
+ -alsologtostderr \
+ SwitchReads \
+ -tablet_type=replica \
+ customer.commerce2customer
+
+```
+
+## Phase 2: Switch Writes
+
+After the reads have been _switched_, and you have verified that the system is operating as expected, it is time to _switch_ the write operations. The command to execute the switch is very similar to switching reads:
+
+```sh
+# Example 204_switch_writes.sh
+
+vtctlclient \
+ -server localhost:15999 \
+ -log_dir "$VTDATAROOT"/tmp \
+ -alsologtostderr \
+ SwitchWrites \
+ customer.commerce2customer
+
+```
+
+We can then verify that both reads and writes go to the new keyspace:
+
+```sh
+# Works
+mysql --table < ../common/select_customer0_data.sql
+
+# Expected to Fail!
+mysql --table < ../common/select_commerce_data.sql
+```
+
+## Cleanup
+
+The final step is to remove the data from the original keyspace. As well as freeing space on the original tablets, this is an important step to eliminate potential future confusions. If you have a misconfiguration down the line and accidentally route queries for the  `customer` and `corder` tables to `commerce`, it is much better to return a "table not found" error, rather than return stale data:
+
+```sh
+# Example 205_clean_commerce.sh
+
+vtctlclient -server localhost:15999 ApplySchema -sql-file drop_commerce_tables.sql commerce
+vtctlclient -server localhost:15999 SetShardTabletControl -blacklisted_tables=customer,corder -remove commerce/0 rdonly
+vtctlclient -server localhost:15999 SetShardTabletControl -blacklisted_tables=customer,corder -remove commerce/0 replica
+vtctlclient -server localhost:15999 SetShardTabletControl -blacklisted_tables=customer,corder -remove commerce/0 master
+
+```
+
+After this step is complete, you should see the following error:
+
+```sh
+# Expected to fail!
+mysql -h 127.0.0.1 -P 15306 --table < ../common/select_commerce_data.sql
+```
+
+This confirms that the data has been correctly cleaned up.
+
+## Next Steps
+
+Congratulations! You've sucessfully moved tables between keyspaces. The next step to try out is to shard one of your keyspaces in [Resharding](../resharding).
+
+Alternatively, if you would like to teardown your example:
+
+``` bash
+./401_teardown.sh
+```