ddl: introduce a new system variable to control the store-write-bwlimit when ingesting

[CbcWestwolf]

What problem does this PR solve?

Issue Number: close #57156

Problem Summary:

See the issue description

What changed and how does it work?

Introduce a new system variable to control the store-write-bwlimit when ingesting

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Manual test:

1. bootstrap a cluster
2. Turn on tidb_ddl_enable_fast_reorg, and with tidb_enable_dist_task on or off, do the following test:
   1. set the tidb_ddl_reorg_max_write_speed to 1mb
   2. add index for a table, the speed is slow
   3. after setting the tidb_ddl_reorg_max_write_speed to 0(no limitation), the speed of adding index increases

For a table containing 200,000 rows:

TiDB root@127.0.0.1:test> select count(*) from t1;
+----------+
| count(*) |
+----------+
| 200000   |
+----------+
1 row in set
Time: 0.272s
TiDB root@127.0.0.1:test> select * from t1 limit 3;
+----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | data                                                                                                                                                                                                                                                            |
+----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 0  | fIhixzEuzMNSXLflhCwAktUGYYhXHfgqVHhmognLsnbaEQhXXvzJOehnEXunFcaeVoZSgLNXEFWTAWjfcPYCQRYNKRlHFGJCstAarZvqyeqwmJoHquIfGHFAUIUovzCeyisLSCrIeypScZdsubrPCcbiyBggiehBqnplnbRoVgNogIVdymQwXdUoyhXNzUzLdBhuqdYcznbnCbkZoFdPQiDYBhECaxNtGwbIjHqGsziqIZTHSxzxlmQVZrsUaRb |
| 1  | UdcuwmGBTvKnZbvFUISeLxhzCYmmGqBkTBodTGxKlPIheJffnZUezUAKzVSOCcIFyJxAGmWrqtJhIblTwOLuSPEbzyxGjStWbCrKQxndouhfvnbgeqzCWdNGcSPCjWOyKGixbEjHejDrCXOuhTVdMqouiEKhWPlqBGydoAVUSBHAVqSJdjIhgASkbzIWiFbrtgOMYuPqSqnoMnMaJdatMUoWofkVhRdxOorjyOyWtfaMKTdWdXxQvIjRmZUezRl |
| 2  | UJttwvfdAXHvQxCITRMoAvLNVElknAuGssAOciBtXzqtoiYISIiicCelyiWGNqjHKfXgqxUJWbbtlhFuEdhvlVPGrcwcArMwcmLaIxklvCDqOJnBNgoPMGTSQaMesuugXAuHHmgjgoIpkQGUdsCcAiDOhTmGBYhZfzqcdCXZNDOFiqOCsYQUCiFrMzEOOcZeJPTHGaZCsMHSQhXxcFqThkWUULEIQRlgBaQBgexlXOFqASzhtBGaFtRMcMlbIGF |
+----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
3 rows in set
Time: 0.019s
TiDB root@127.0.0.1:test> show create table t1;
+-------+-------------------------------------------------------------+
| Table | Create Table                                                |
+-------+-------------------------------------------------------------+
| t1    | CREATE TABLE `t1` (                                         |
|       |   `id` int NOT NULL,                                        |
|       |   `data` varchar(255) DEFAULT NULL,                         |
|       |   PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */     |
|       | ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin |
+-------+-------------------------------------------------------------+
1 row in set
Time: 0.013s

Setting tidb_ddl_reorg_max_write_speed from 4mb to unlimited would result in different speeds of adding index:

TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '4mb';
Query OK, 0 rows affected
Time: 0.010s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 63.228s

TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.107s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '8mb';
Query OK, 0 rows affected
Time: 0.031s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 33.176s

TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.105s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '16mb';
Query OK, 0 rows affected
Time: 0.059s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 15.432s

TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.220s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '32mb';
Query OK, 0 rows affected
Time: 0.026s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 7.717s

TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.139s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '64mb';
Query OK, 0 rows affected
Time: 0.018s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 4.387s

TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.140s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '128mb';
Query OK, 0 rows affected
Time: 0.019s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 4.375s

TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.096s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = 0;
Query OK, 0 rows affected
Time: 0.038s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 4.229s

With the tidb_ddl_reorg_max_write_speed increased, the metric shows that the number of write operations increased as expected

To verify that decreasing the tidb_ddl_reorg_max_write_speed helps to lower the impact of ADD INDEX on online service, we observe that with a further speed limitation, the QPS of sysbench increases.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Add a system variable tidb_ddl_reorg_max_write_speed to limit the speed of ingesting when adding index
增加系统变量 tidb_ddl_reorg_max_write_speed 来限制加索引时 ingesting 的速度上限

[codecov]

Codecov Report

Attention: Patch coverage is 68.96552% with 9 lines in your changes missing coverage. Please review.

Project coverage is 74.8591%. Comparing base (3651427) to head (91b644b).
Report is 81 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57145        +/-   ##
================================================
+ Coverage   73.2148%   74.8591%   +1.6442%     
================================================
  Files          1652       1717        +65     
  Lines        456167     485802     +29635     
================================================
+ Hits         333982     363667     +29685     
+ Misses       101669      99640      -2029     
- Partials      20516      22495      +1979     

Flag	Coverage Δ
integration	49.3224% <37.9310%> (?)
unit	72.4685% <68.9655%> (-0.0263%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.7673% <ø> (-0.1806%)	⬇️
parser	∅ <ø> (∅)
br	59.7891% <ø> (+13.8076%)	⬆️

[CbcWestwolf]

/retest

[CbcWestwolf]

/hold

[CbcWestwolf]

/run-check-issue-triage-complete

[CbcWestwolf]

/unhold

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-6.5: #57374.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.1: #57375.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.5: #57376.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.1: #57377.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.5: #57378.