bug: (Primary/Secondary)The StoneDB's insert delay lead to data on the primary database and that on the secondary database are inconsistent.

[haitaoguan]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

Please confirm if bug report does NOT exists already ?

• I confirm there is no existing issue for this

Describe the problem

###MASTER
create table ttt(id int primary key,name varchar(5));
insert into ttt values(1,'AAA');
delete from ttt where id=1;
insert into ttt values(1,'aaa');
mysql> select * from ttt;
+----+------+
| id | name |
+----+------+
|  1 | aaa  |
+----+------+
1 row in set (0.00 sec)

###SLAVE
mysql> select * from ttt;
+----+------+
| id | name |
+----+------+
|  1 | AAA  |
+----+------+
1 row in set (0.00 sec)

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.30.40
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mybinlog.000010
          Read_Master_Log_Pos: 1152
               Relay_Log_File: HAST05-relay-bin.000028
                Relay_Log_Pos: 849
        Relay_Master_Log_File: mybinlog.000010
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 6
                   Last_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 1 failed executing transaction 'f21bb118-3e57-11ed-b5a2-46a07fb8ee2c:18727' at master log mybinlog.000010, end_log_pos 864. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 638
              Relay_Log_Space: 2356
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 6
               Last_SQL_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 1 failed executing transaction 'f21bb118-3e57-11ed-b5a2-46a07fb8ee2c:18727' at master log mybinlog.000010, end_log_pos 864. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 40
                  Master_UUID: f21bb118-3e57-11ed-b5a2-46a07fb8ee2c
             Master_Info_File: /stonedb57/install/data/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: 
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 221013 16:54:42
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: f21bb118-3e57-11ed-b5a2-46a07fb8ee2c:1-18728
            Executed_Gtid_Set: 0b863da5-49f4-11ed-acd0-ee399c8dd823:1-2473,
f21bb118-3e57-11ed-b5a2-46a07fb8ee2c:1-18726
                Auto_Position: 1
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
1 row in set (0.00 sec)

mysql> select * from performance_schema.replication_applier_status_by_worker\G
*************************** 1. row ***************************
         CHANNEL_NAME: 
            WORKER_ID: 1
            THREAD_ID: NULL
        SERVICE_STATE: OFF
LAST_SEEN_TRANSACTION: f21bb118-3e57-11ed-b5a2-46a07fb8ee2c:18727
    LAST_ERROR_NUMBER: 6
   LAST_ERROR_MESSAGE: Worker 1 failed executing transaction 'f21bb118-3e57-11ed-b5a2-46a07fb8ee2c:18727' at master log mybinlog.000010, end_log_pos 864; Could not execute Delete_rows event on table zz.ttt; assert failed on i < m_idx.size() at rc_attr.h:332, msg: [bad dpn index 0/0], Error_code: 6; Can't find record in 'ttt', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log FIRST, end_log_pos 864
 LAST_ERROR_TIMESTAMP: 2022-10-13 16:54:42

Expected behavior

###SLAVE
mysql> select * from ttt;
+----+------+
| id | name |
+----+------+
|  1 | aaa  |
+----+------+
1 row in set (0.00 sec)

How To Reproduce

This problem is caused by StoneDB‘s insert delay.If you set tianmu_insert_delayed is off, there is no such exception.

Environment

###StoneDB slave
Ver 14.14 Distrib 5.7.36_v1.0.1_beta_2-StoneDB, for Linux (x86_64) using EditLine wrapper

Are you interested in submitting a PR to solve the problem?

• Yes, I will!

[konghaiya]

ACK，This problem can be fixed after the development of the delta layer is completed