pitr: add ingest recorder to repair indexes

[Leavrth]

What problem does this PR solve?

Issue Number: close #41668 ref #38045

Problem Summary:
PiTR is not compatible with accelerated indexing

What is changed and how it works?

1. record the ingest indexes from history ddl
2. drop the indexes
3. re-add the indexes

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

upstream SQL

CREATE TABLE test.pairs(x int auto_increment primary key, y int DEFAULT RAND(), z int DEFAULT RAND());
INSERT INTO test.pairs VALUES (), ... (skip 500), ();
ALTER TABLE test.pairs ADD INDEX i1(y, z) USING HASH COMMENT "edelw;fe?fewfe\nefwe" INVISIBLE;
ALTER TABLE test.pairs ADD UNIQUE KEY u1(x, y) USING RTREE VISIBLE;
ALTER TABLE test.pairs ADD INDEX i2(y, (z + 1)) USING BTREE COMMENT "123";
ALTER TABLE test.pairs ADD UNIQUE KEY u2(x, (y+1)) USING HASH COMMENT "243";

CREATE TABLE test.pairs2(x int, y int, z varchar(20));
INSERT INTO test.pairs2 VALUES (1,1,"1"), ... (skip 500), (500,500,"500");
ALTER TABLE test.pairs2 ADD INDEX i1(y, z(10));
ALTER TABLE test.pairs2 ADD UNIQUE KEY u1(y, z(10), (y * 2)) USING RTREE VISIBLE;
ALTER TABLE test.pairs2 ADD PRIMARY KEY (x) USING HASH;

CREATE TABLE test.pairs3(id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, name CHAR(10), custinfo JSON);
INSERT INTO test.pairs3 VALUES (1, 'pingcap', '{"zipcode": [1,2]}');
ALTER TABLE test.pairs3 ADD INDEX zips2((CAST(custinfo->'$.zipcode' AS UNSIGNED ARRAY)));

CREATE TABLE test.pairs4(x int auto_increment primary key, y int DEFAULT RAND(), z int DEFAULT RAND());
INSERT INTO test.pairs4 VALUES (), ... (skip 500), ();
ALTER TABLE test.pairs4 ADD INDEX i1(y, z) USING HASH COMMENT "edelw;fe?fewfe\nefwe" INVISIBLE;
ALTER TABLE test.pairs4 ADD UNIQUE KEY u1(x, y) USING RTREE VISIBLE;
ALTER TABLE test.pairs4 ADD INDEX i2(y, (z + 1)) USING BTREE COMMENT "123";
ALTER TABLE test.pairs4 ADD UNIQUE KEY u2(x, (y+1)) USING HASH COMMENT "243";
ALTER TABLE test.pairs4 DROP INDEX i1;
ALTER TABLE test.pairs4 DROP INDEX u1;
ALTER TABLE test.pairs4 DROP INDEX i2;
ALTER TABLE test.pairs4 DROP INDEX u2;

CREATE TABLE test.pairs5(x int, y int, z varchar(20));
INSERT INTO test.pairs5 VALUES (1,1,"1"), ... (skip 500), (500,500,"500");
ALTER TABLE test.pairs5 ADD INDEX i1(y, z(10));
ALTER TABLE test.pairs5 ADD UNIQUE KEY u1(y, z(10), (y * 2)) USING RTREE VISIBLE;
ALTER TABLE test.pairs5 ADD PRIMARY KEY (x) USING HASH;
ALTER TABLE test.pairs5 DROP INDEX i1;
ALTER TABLE test.pairs5 DROP INDEX u1;
ALTER TABLE test.pairs5 DROP INDEX `PRIMARY`;

CREATE TABLE test.pairs6(id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, `nam``;e` int, `cust``;info` JSON);
INSERT INTO test.pairs6 VALUES (1, 1, '{"zipcode": [1,2]}');
ALTER TABLE test.pairs6 ADD INDEX zips2((CAST(`cust``;info`->'$.zipcode' AS UNSIGNED ARRAY)));
ALTER TABLE test.pairs6 ADD INDEX i1(`nam``;e`, (`nam``;e` * 2));
RENAME TABLE test.pairs6 TO test.pairs7;
ALTER TABLE test.pairs7 RENAME INDEX i1 to i2;

restore repair ingest index SQL from BR log

ALTER TABLE `test`.`pairs` DROP INDEX `i1`
ALTER TABLE `test`.`pairs` ADD INDEX `i1`(`y`,`z`) USING HASH COMMENT 'edelw;fe?fewfe\\nefwe' INVISIBLE
ALTER TABLE `test`.`pairs` DROP INDEX `u1`
ALTER TABLE `test`.`pairs` ADD UNIQUE KEY `u1`(`x`,`y`) USING RTREE VISIBLE
ALTER TABLE `test`.`pairs` DROP INDEX `i2`
ALTER TABLE `test`.`pairs` ADD INDEX `i2`(`y`,(`z` + 1)) USING BTREE COMMENT '123' VISIBLE
ALTER TABLE `test`.`pairs` DROP INDEX `u2`
ALTER TABLE `test`.`pairs` ADD UNIQUE KEY `u2`(`x`,(`y` + 1)) USING HASH COMMENT '243' VISIBLE
ALTER TABLE `test`.`pairs2` DROP INDEX `PRIMARY`
ALTER TABLE `test`.`pairs2` ADD PRIMARY KEY (`x`) NONCLUSTERED USING HASH VISIBLE
ALTER TABLE `test`.`pairs2` DROP INDEX `i1`
ALTER TABLE `test`.`pairs2` ADD INDEX `i1`(`y`,`z`(10)) USING BTREE VISIBLE
ALTER TABLE `test`.`pairs2` DROP INDEX `u1`
ALTER TABLE `test`.`pairs2` ADD UNIQUE KEY `u1`(`y`,`z`(10),(`y` * 2)) USING RTREE VISIBLE
ALTER TABLE `test`.`pairs3` DROP INDEX `zips2`
ALTER TABLE `test`.`pairs3` ADD INDEX `zips2`((cast(json_extract(`custinfo`, _utf8'$.zipcode') as unsigned array))) USING BTREE VISIBLE
ALTER TABLE `test`.`pairs7` DROP INDEX `zips2`
ALTER TABLE `test`.`pairs7` ADD INDEX `zips2`((cast(json_extract(`cust``;info`, _utf8'$.zipcode') as unsigned array))) USING BTREE VISIBLE
ALTER TABLE `test`.`pairs7` DROP INDEX `i2`
ALTER TABLE `test`.`pairs7` ADD INDEX `i2`(`nam``;e`,(`nam``;e` * 2)) USING BTREE VISIBLE

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• YuJuncen
• tangenta

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[Benjamin2037]

here, currently, do not support clusterd PK online changed. also should care able add unique index.

[Benjamin2037]

Add generate uk statement

[tangenta]

Is the debug log left here on purpose? If not, please remove them.

Why does dropSQL use Execute() and addSQL() use ExecuteInternal?

[Leavrth]

Because the Comment has some escape characters such as " and \n, so here use ExecuteInter to send with args instead of plaintext.
The function pattern is as follow:
Execute(context, sql string)
ExecuteInternal(context, sql string, args...)

[Leavrth]

done, both use ExecuteInternal.

[YuJuncen]

Should we escape here? Perhaps some guys like to name their indices like(How absurd!):

CREATE INDEX `idx``; DROP DATABASE business; --`(some_row);

[Leavrth]

😂

[Leavrth]

done. test updated

[tangenta]

What is the filter used for?

[Leavrth]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 1b57703

[Leavrth]

/hold

[Leavrth]

add progress for repair ingest index

[Leavrth]

/retest

[Leavrth]

/retest

[YuJuncen]

LGTM for the PB part

[Leavrth]

/retest

[Leavrth]

/retest

[Leavrth]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 739bef9

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-6.3: #42217.

[BornChanger]

/cherry-pick release-6.5

[ti-chi-bot]

@BornChanger: new pull request created to branch release-6.5: #46418.

In response to this:

/cherry-pick release-6.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.