tikv 8141 assertion failed while reorganize partition

[aytrack]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. create table like

CREATE TABLE `sbtest1` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `k` int(11) NOT NULL DEFAULT '0',
  `c` char(120) NOT NULL DEFAULT '',
  `pad` char(60) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */,
  KEY `k_1` (`k`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin AUTO_INCREMENT=7650001
PARTITION BY RANGE (`id`)
(PARTITION `p0` VALUES LESS THAN (2000000),
 PARTITION `p1` VALUES LESS THAN (4000000),
 PARTITION `p2` VALUES LESS THAN (6000000),
 PARTITION `p3` VALUES LESS THAN (8000000),
 PARTITION `p4` VALUES LESS THAN (10000000),
 PARTITION `p5` VALUES LESS THAN (MAXVALUE))

2. use sysbench prepare data
3. run sysbench oltp_read_write
4. reorganize table

alter table sbtest1 reorganize partition p0, p1, p2, p3, p4 into( partition pnew values less than (10000000));

2. What did you expect to see? (Required)

No Error

3. What did you see instead (Required)

[2023/03/21 15:29:50.440 +08:00] [INFO] [ddl_worker.go:960] ["[ddl] run DDL job"] [worker="worker 4, tp general"] [job="ID:4363, Type:alter table reorganize partition, State:running, SchemaState:write reorganization, SchemaID:324, TableID:3881, RowCount:922624, ArgLen:0, start time: 2023-03-21 15:29:34.22 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:440240380371271683, UniqueWarnings:0"]
[2023/03/21 15:29:51.271 +08:00] [ERROR] [session.go:780] ["assertion failed"] [conn=120565848032150137] [message="[tikv:8141]assertion failed: key: 748000000000000f2b5f728000000000356eef, assertion: NotExist, start_ts: 440240384513147069, existing start ts: 0, existing commit ts: 0"] ["mvcc history"="{\"decoded\":{\"432008707192651894\":{\"c\":\"54325927229-92551291829-21672538259-06996857468-43046795368-71396980678-74291024006-37010947921-55837102926-23484006538\",\"k\":\"5019042\",\"pad\":\"85289104575-03306411387-47035023324-49699784330-43468025430\"}},\"key\":\"748000000000000F2B5F728000000000356EEF\",\"mvcc\":{\"info\":{\"lock\":{\"start_ts\":440240384513147069,\"primary\":\"dIAAAAAAABAbX3KAAAAAADVECg==\",\"short_value\":\"gAADAAAAAgMEBAB7ALYAEGI1ADE2Mjg0NjM0OTI1LTYzOTEyMzMxODM4LTg0MjE1MzU3MDIwLTQ1ODM5Mjk2MjI5LTU3MjE0MTc0OTQ0LTg0MjAyODIyNzQ4LTc2ODc2Njk3MDcyLTcxNzkzNDM3OTg0LTQyNzExOTUxOTM0LTgyNzk5NjgxODMxMjI5MzY5NjI2NzEtNjE4NTE4ODY2MzQtMTM0OTExMzI5OTQtNTQyNTg0ODE4OTMtODg2NDkyODIzMTM=\"},\"writes\":[{\"start_ts\":432008707192651894,\"commit_ts\":432008707192651896,\"short_value\":\"gAADAAAAAgMEBAB7ALYAopVMADU0MzI1OTI3MjI5LTkyNTUxMjkxODI5LTIxNjcyNTM4MjU5LTA2OTk2ODU3NDY4LTQzMDQ2Nzk1MzY4LTcxMzk2OTgwNjc4LTc0MjkxMDI0MDA2LTM3MDEwOTQ3OTIxLTU1ODM3MTAyOTI2LTIzNDg0MDA2NTM4ODUyODkxMDQ1NzUtMDMzMDY0MTEzODctNDcwMzUwMjMzMjQtNDk2OTk3ODQzMzAtNDM0NjgwMjU0MzA=\"}]}},\"regionID\":740}"]
[2023/03/21 15:29:51.271 +08:00] [WARN] [session.go:953] ["can not retry txn"] [conn=120565848032150137] [label=general] [error="[tikv:8141]assertion failed: key: 748000000000000f2b5f728000000000356eef, assertion: NotExist, start_ts: 440240384513147069, existing start ts: 0, existing commit ts: 0"] [IsBatchInsert=false] [IsPessimistic=true] [InRestrictedSQL=false] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2023/03/21 15:29:51.271 +08:00] [WARN] [session.go:969] ["commit failed"] [conn=120565848032150137] ["finished txn"="Txn{state=invalid}"] [error="[tikv:8141]assertion failed: key: 748000000000000f2b5f728000000000356eef, assertion: NotExist, start_ts: 440240384513147069, existing start ts: 0, existing commit ts: 0"]
[2023/03/21 15:29:51.271 +08:00] [WARN] [session.go:2202] ["run statement failed"] [conn=120565848032150137] [schemaVersion=258] [error="[tikv:8141]assertion failed: key: 748000000000000f2b5f728000000000356eef, assertion: NotExist, start_ts: 440240384513147069, existing start ts: 0, existing commit ts: 0"] [session="{\n  \"currDBName\": \"partitioninsert_1647963896275\",\n  \"id\": 120565848032150137,\n  \"preparedStmtCount\": 290,\n  \"status\": 2,\n  \"strictMode\": true,\n  \"user\": {\n    \"Username\": \"root\",\n    \"Hostname\": \"172.16.6.116\",\n    \"CurrentUser\": false,\n    \"AuthUsername\": \"root\",\n    \"AuthHostname\": \"%\",\n    \"AuthPlugin\": \"mysql_native_password\"\n  }\n}"]
[2023/03/21 15:29:51.271 +08:00] [ERROR] [session.go:780] ["assertion failed"] [conn=120565848032150201] [message="[tikv:8141]assertion failed: key: 748000000000000f2b5f72800000000035e24b, assertion: NotExist, start_ts: 440240384513147102, existing start ts: 0, existing commit ts: 0"] ["mvcc history"="{\"decoded\":{\"432008751494988180\":{\"c\":\"49664764444-25677505682-75501990543-29764275559-96261665213-89998408870-03334445911-20651303803-06313580566-88703206325\",\"k\":\"5038578\",\"pad\":\"56552417136-43730050216-74669325131-67653162703-62475506457\"}},\"key\":\"748000000000000F2B5F72800000000035E24B\",\"mvcc\":{\"info\":{\"lock\":{\"start_ts\":440240384513147102,\"primary\":\"dIAAAAAAABATX3KAAAAAADVFlQ==\",\"short_value\":\"gAADAAAAAgMEBAB7ALYAqMs1ADYzNzcyODgyODIwLTA4ODk4NzM1NTExLTMyOTQyNzMzNTEyLTQyOTEyMjQ5MjUzLTk4MDk3MjM3NzA0LTI4OTc5MDEyMTg4LTQ2ODg5NDk1Njg1LTIyNTA4NzIxNjQ1LTM2NDUxMjkzMDczLTgzMzI1NzI1MTY4MjExMTU0MDQzNTEtOTQyMDE5MjI5NTMtNDYwOTA1MTk3MzMtMjU5MjI4ODg2NzUtMDA4MzIzODMzMDM=\"},\"writes\":[{\"start_ts\":432008751494988180,\"commit_ts\":432008751494988190,\"short_value\":\"gAADAAAAAgMEBAB7ALYA8uFMADQ5NjY0NzY0NDQ0LTI1Njc3NTA1NjgyLTc1NTAxOTkwNTQzLTI5NzY0Mjc1NTU5LTk2MjYxNjY1MjEzLTg5OTk4NDA4ODcwLTAzMzM0NDQ1OTExLTIwNjUxMzAzODAzLTA2MzEzNTgwNTY2LTg4NzAzMjA2MzI1NTY1NTI0MTcxMzYtNDM3MzAwNTAyMTYtNzQ2NjkzMjUxMzEtNjc2NTMxNjI3MDMtNjI0NzU1MDY0NTc=\"}]}},\"regionID\":742}"]
[2023/03/21 15:29:51.271 +08:00] [WARN] [session.go:953] ["can not retry txn"] [conn=120565848032150201] [label=general] [error="[tikv:8141]assertion failed: key: 748000000000000f2b5f72800000000035e24b, assertion: NotExist, start_ts: 440240384513147102, existing start ts: 0, existing commit ts: 0"] [IsBatchInsert=false] [IsPessimistic=true] [InRestrictedSQL=false] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2023/03/21 15:29:51.271 +08:00] [WARN] [session.go:969] ["commit failed"] [conn=120565848032150201] ["finished txn"="Txn{state=invalid}"] [error="[tikv:8141]assertion failed: key: 748000000000000f2b5f72800000000035e24b, assertion: NotExist, start_ts: 440240384513147102, existing start ts: 0, existing commit ts: 0"]

4. What is your TiDB version? (Required)

partitioninsert_1647963896275> select tidb_version();
+-----------------------------------------------------------+
| tidb_version()                                            |
+-----------------------------------------------------------+
| Release Version: v7.0.0                                   |
| Edition: Community                                        |
| Git Commit Hash: fc166ffae8576ac3efb2f5abac848d5edbaf2657 |
| Git Branch: heads/refs/tags/v7.0.0                        |
| UTC Build Time: 2023-03-20 07:08:59                       |
| GoVersion: go1.20.2                                       |
| Race Enabled: false                                       |
| TiKV Min Version: 6.2.0-alpha                             |
| Check Table Before Drop: false                            |
| Store: tikv                                               |
+-----------------------------------------------------------+

[tiancaiamao]

A minimal reproduce, no need to alter table reorganize partition ...

CREATE TABLE `sbtest1` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `k` int(11) NOT NULL DEFAULT '0',
  `c` char(120) NOT NULL DEFAULT '',
  `pad` char(60) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */,
  KEY `k_1` (`k`)
) PARTITION BY RANGE (`id`)
(PARTITION `pnew` VALUES LESS THAN (10000000),
 PARTITION `p5` VALUES LESS THAN (MAXVALUE));

insert into sbtest1 (id, k, c, pad) VALUES (502571, 499449, "init", "val");

begin;
DELETE FROM sbtest1 WHERE id=502571;
insert into sbtest1 (id, k, c, pad) VALUES (502571, 499449, "abc", "def");
commit;

[tiancaiamao]

It's here

tidb/table/tables/tables.go

Line 1348 in 6897c1a

err = txn.SetAssertion(key, kv.SetAssertExist)

Before #42000, shouldAssert return true for the statement, and after that it returns false:

DELETE FROM sbtest1 WHERE id=502571;

[tiancaiamao]

@mjonss You can set it to kv.SetAssertUnknown if you want to skip the check here.

The assertion should be kv.SetAssertExist, but shouldAssert() returns false, so it's skipped here.
And later, another assertion (which is kv.SetAssertNotExist) is set for that key (id=502571 row key).

Note, the first Assertion take effect, so the wrong assertion is set finally!