|
| 1 | +--- |
| 2 | +title: TiProxy Traffic Replay |
| 3 | +summary: Introduce the use cases and steps for the TiProxy traffic replay feature. |
| 4 | +--- |
| 5 | + |
| 6 | +# TiProxy Traffic Replay |
| 7 | + |
| 8 | +> **Warning:** |
| 9 | +> |
| 10 | +> Currently, the TiProxy traffic replay feature is experimental. It is not recommended that you use it in production environments. This feature might be changed or removed without prior notice. If you find a bug, you can report an [issue](https://github.com/pingcap/tiproxy/issues) on GitHub. |
| 11 | +
|
| 12 | +Starting from TiProxy v1.3.0, you can use TiProxy to capture access traffic in a TiDB production cluster and replay it in a test cluster at a specified rate. This feature enables you to reproduce actual workloads from the production cluster in a test environment, verifying SQL statement execution results and performance. |
| 13 | + |
| 14 | +<img src="https://docs-download.pingcap.com/media/images/docs/tiproxy/tiproxy-traffic-replay.png" alt="TiProxy traffic replay" width="800" /> |
| 15 | + |
| 16 | +## Use cases |
| 17 | + |
| 18 | +Traffic replay is suitable for the following scenarios: |
| 19 | + |
| 20 | +- **Verify TiDB version upgrades**: Replay production traffic on a test cluster with a new TiDB version to verify that the new TiDB version can successfully execute all SQL statements. |
| 21 | +- **Assess change impact**: Simulate production traffic on a test cluster to verify the impact of changes on the cluster. For example, verify the effects before modifying configuration items or system variables, altering table schemas, or enabling new TiDB features. |
| 22 | +- **Validate performance before TiDB scaling**: Replay traffic at corresponding rates on a test cluster with a new scale to validate whether the performance meets requirements. For example, to plan a 50% cluster downscale for cost savings, replay traffic at half speed to validate if SQL latency meets requirements after scaling. |
| 23 | +- **Test performance limits**: Replay traffic multiple times on a test cluster of the same scale, increasing the replay rate each time to test the throughput limit of that scale and assess whether performance meets future business growth needs. |
| 24 | + |
| 25 | +Traffic replay is not suitable for the following scenarios: |
| 26 | + |
| 27 | +- Verify SQL compatibility between TiDB and MySQL: TiProxy only supports reading traffic files it generates and cannot capture traffic from MySQL for replay on TiDB. |
| 28 | +- Compare SQL execution results between TiDB versions: TiProxy only verifies if SQL statements execute successfully but does not compare results. |
| 29 | + |
| 30 | +## Usage |
| 31 | + |
| 32 | +1. Prepare the test environment: |
| 33 | + |
| 34 | + 1. Create a test cluster. For more information, see [Deploy a TiDB Cluster Using TiUP](/production-deployment-using-tiup.md). |
| 35 | + 2. Install `tiproxyctl` and ensure the host with `tiproxyctl` can connect to TiProxy instances in both production and test clusters. For more information, see [Install TiProxy Control](/tiproxy/tiproxy-command-line-flags.md#install-tiproxy-control). |
| 36 | + 3. Replicate data from the production cluster to the test cluster. For more information, see [Data Migration Overview](/migration-overview.md). |
| 37 | + 4. Run the [`ANALYZE`](/sql-statements/sql-statement-analyze-table.md) statement in the test cluster to update statistics. |
| 38 | + |
| 39 | +2. Use the [`tiproxyctl traffic capture`](/tiproxy/tiproxy-command-line-flags.md#traffic-capture) command to connect to the production cluster's TiProxy instance and start capturing traffic. |
| 40 | + |
| 41 | + > **Note:** |
| 42 | + > |
| 43 | + > - TiProxy captures traffic on all connections, including existing and newly created ones. |
| 44 | + > - In TiProxy primary-secondary mode, connect to the primary TiProxy instance. |
| 45 | + > - If TiProxy is configured with a virtual IP, it is recommended to connect to the virtual IP address. |
| 46 | + > - The higher the CPU usage of TiProxy, the greater the impact of traffic capture on QPS. To reduce the impact on the production cluster, it is recommended to reserve at least 30% of CPU capacity, which results in an approximately 3% decrease in average QPS. For detailed performance data, see [Traffic capture test](/tiproxy/tiproxy-performance-test.md#traffic-capture-test). |
| 47 | + > - TiProxy does not automatically delete previous capture files when capturing traffic again. You need to manually delete them. |
| 48 | +
|
| 49 | + For example, the following command connects to the TiProxy instance at `10.0.1.10:3080`, captures traffic for one hour, and saves it to the `/tmp/traffic` directory on the TiProxy instance: |
| 50 | + |
| 51 | + ```shell |
| 52 | + tiproxyctl traffic capture --host 10.0.1.10 --port 3080 --output="/tmp/traffic" --duration=1h |
| 53 | + ``` |
| 54 | + |
| 55 | + Traffic files are automatically rotated and compressed. Example files in the `/tmp/traffic` directory: |
| 56 | + |
| 57 | + ```shell |
| 58 | + ls /tmp/traffic |
| 59 | + # meta traffic-2024-08-29T17-37-12.477.log.gz traffic-2024-08-29T17-43-11.166.log.gz traffic.log |
| 60 | + ``` |
| 61 | + |
| 62 | + For more information, see [`tiproxyctl traffic capture`](/tiproxy/tiproxy-command-line-flags.md#traffic-capture). |
| 63 | + |
| 64 | +3. Copy the traffic file directory to the test cluster's TiProxy instance. |
| 65 | +4. Use [`tiproxyctl traffic replay`](/tiproxy/tiproxy-command-line-flags.md#traffic-replay) to connect to the test cluster's TiProxy instance and start replaying traffic. |
| 66 | + |
| 67 | + By default, SQL statements are executed at the same rate as in the production cluster, and each database connection corresponds to a connection in the production cluster to simulate the production load and ensure consistent transaction execution order. |
| 68 | + |
| 69 | + For example, the following command connects to the TiProxy instance at `10.0.1.10:3080` using username `u1` and password `123456`, reads traffic files from the `/tmp/traffic` directory on the TiProxy instance, and replays the traffic: |
| 70 | + |
| 71 | + ```shell |
| 72 | + tiproxyctl traffic replay --host 10.0.1.10 --port 3080 --username="u1" --password="123456" --input="/tmp/traffic" |
| 73 | + ``` |
| 74 | + |
| 75 | + Because all traffic runs under user `u1`, ensure `u1` can access all databases and tables. If no such user exists, create one. |
| 76 | + |
| 77 | + For more information, see [`tiproxyctl traffic replay`](/tiproxy/tiproxy-command-line-flags.md#traffic-replay). |
| 78 | + |
| 79 | +5. View the replay report. |
| 80 | + |
| 81 | + After replay completion, the report is stored in the `tiproxy_traffic_replay` database on the test cluster. This database contains two tables: `fail` and `other_errors`. |
| 82 | + |
| 83 | + The `fail` table stores failed SQL statements, with the following fields: |
| 84 | + |
| 85 | + - `cmd_type`: the type of a failed command, such as `Query` (execute an ordinary statement), `Prepare` (prepare a statement), and `Execute` (execute a prepared statement). |
| 86 | + - `digest`: the digest of the failed SQL statement. |
| 87 | + - `sample_stmt`: the SQL text when the statement first failed. |
| 88 | + - `sample_err_msg`: the error message when the SQL statement failed. |
| 89 | + - `sample_conn_id`: the connection ID recorded in the traffic file for the SQL statement. You can use this to view the execution context in the traffic file. |
| 90 | + - `sample_capture_time`: the execution time recorded in the traffic file for the SQL statement. You can use this to view the execution context in the traffic file. |
| 91 | + - `sample_replay_time`: the time when the SQL statement failed during replay. You can use this to view error information in the TiDB log file. |
| 92 | + - `count`: the number of times the SQL statement failed. |
| 93 | + |
| 94 | + The following is an example output of the `fail` table: |
| 95 | + |
| 96 | + ```sql |
| 97 | + SELECT * FROM tiproxy_traffic_replay.fail LIMIT 1\G |
| 98 | + ``` |
| 99 | + |
| 100 | + ``` |
| 101 | + *************************** 1. row *************************** |
| 102 | + cmd_type: StmtExecute |
| 103 | + digest: 89c5c505772b8b7e8d5d1eb49f4d47ed914daa2663ed24a85f762daa3cdff43c |
| 104 | + sample_stmt: INSERT INTO new_order (no_o_id, no_d_id, no_w_id) VALUES (?, ?, ?) params=[3077 6 1] |
| 105 | + sample_err_msg: ERROR 1062 (23000): Duplicate entry '1-6-3077' for key 'new_order.PRIMARY' |
| 106 | + sample_conn_id: 1356 |
| 107 | + sample_capture_time: 2024-10-17 12:59:15 |
| 108 | + sample_replay_time: 2024-10-17 13:05:05 |
| 109 | + count: 4 |
| 110 | + ``` |
| 111 | + |
| 112 | + The `other_errors` table stores unexpected errors, such as network errors or database connection errors, with the following fields: |
| 113 | + |
| 114 | + - `err_type`: the type of error, presented as a brief error message. For example, `i/o timeout`. |
| 115 | + - `sample_err_msg`: the complete error message when the error first occurred. |
| 116 | + - `sample_replay_time`: the time when the error occurred during replay. You can use this to view error information in the TiDB log file. |
| 117 | + - `count`: the number of occurrences for this error. |
| 118 | + |
| 119 | + The following is an example output of the `other_errors` table: |
| 120 | + |
| 121 | + ```sql |
| 122 | + SELECT * FROM tiproxy_traffic_replay.other_errors LIMIT 1\G |
| 123 | + ``` |
| 124 | + |
| 125 | + ``` |
| 126 | + *************************** 1. row *************************** |
| 127 | + err_type: failed to read the connection: EOF |
| 128 | + sample_err_msg: this is an error from the backend connection: failed to read the connection: EOF |
| 129 | + sample_replay_time: 2024-10-17 12:57:39 |
| 130 | + count: 1 |
| 131 | + ``` |
| 132 | + |
| 133 | + > **Note:** |
| 134 | + > |
| 135 | + > - The table schema of `tiproxy_traffic_replay` might change in future versions. It is not recommended to directly read data from `tiproxy_traffic_replay` in your application or tool development. |
| 136 | + > - Replay does not guarantee that the transaction execution order between connections exactly matches the capture sequence. This might lead to incorrect error reports. |
| 137 | + > - TiProxy does not automatically delete the previous replay report when replaying traffic. You need to manually delete it. |
| 138 | + |
| 139 | +## Test throughput |
| 140 | + |
| 141 | +To test cluster throughput, use the `--speed` option to adjust the replay rate. |
| 142 | + |
| 143 | +For example, `--speed=2` executes SQL statements at twice the rate, reducing the total replay time by half: |
| 144 | + |
| 145 | +```shell |
| 146 | +tiproxyctl traffic replay --host 10.0.1.10 --port 3080 --username="u1" --password="123456" --input="/tmp/traffic" --speed=2 |
| 147 | +``` |
| 148 | + |
| 149 | +Increasing the replay rate only reduces idle time between SQL statements and does not increase the number of connections. When session idle time is already short, increasing the speed might not effectively improve throughput. In such cases, you can deploy multiple TiProxy instances to replay the same traffic files simultaneously, increasing concurrency to improve throughput. |
| 150 | + |
| 151 | +## View and manage tasks |
| 152 | + |
| 153 | +During capture and replay, tasks automatically stop if unknown errors occur. To view the current task progress or error information from the last task, use the [`tiproxyctl traffic show`](/tiproxy/tiproxy-command-line-flags.md#traffic-show) command: |
| 154 | + |
| 155 | +```shell |
| 156 | +tiproxyctl traffic show --host 10.0.1.10 --port 3080 |
| 157 | +``` |
| 158 | + |
| 159 | +For example, the following output indicates a running capture task: |
| 160 | + |
| 161 | +```json |
| 162 | +[ |
| 163 | + { |
| 164 | + "type": "capture", |
| 165 | + "start_time": "2024-09-03T09:10:58.220644+08:00", |
| 166 | + "duration": "2h", |
| 167 | + "output": "/tmp/traffic", |
| 168 | + "progress": "45%", |
| 169 | + "status": "running" |
| 170 | + } |
| 171 | +] |
| 172 | +``` |
| 173 | + |
| 174 | +For more information, see [`tiproxyctl traffic show`](/tiproxy/tiproxy-command-line-flags.md#traffic-show). |
| 175 | + |
| 176 | +To cancel the current capture or replay task, use the [`tiproxyctl traffic cancel`](/tiproxy/tiproxy-command-line-flags.md#traffic-cancel) command: |
| 177 | + |
| 178 | +```shell |
| 179 | +tiproxyctl traffic cancel --host 10.0.1.10 --port 3080 |
| 180 | +``` |
| 181 | + |
| 182 | +For more information, see [`tiproxyctl traffic cancel`](/tiproxy/tiproxy-command-line-flags.md#traffic-cancel). |
| 183 | + |
| 184 | +## Limitations |
| 185 | + |
| 186 | +- TiProxy only supports replaying traffic files captured by TiProxy and does not support other file formats. Therefore, make sure to capture traffic from the production cluster using TiProxy first. |
| 187 | +- TiProxy traffic replay does not support filtering SQL types and DML and DDL statements are replayed. Therefore, you need to restore the cluster data to its pre-replay state before replaying again. |
| 188 | +- TiProxy traffic replay does not support testing [Resource Control](/tidb-resource-control-ru-groups.md) and [privilege management](/privilege-management.md) because TiProxy uses the same username to replay traffic. |
| 189 | +- TiProxy does not support replaying [`LOAD DATA`](/sql-statements/sql-statement-load-data.md) statements. |
| 190 | + |
| 191 | +## More resources |
| 192 | + |
| 193 | +For more information about the traffic replay of TiProxy, see the [design document](https://github.com/pingcap/tiproxy/blob/main/docs/design/2024-08-27-traffic-replay.md). |
0 commit comments