=====================================================================…

…==========

This is the initial patch to show how gap lock is inherited during
purge. The tests are for debugging.

The code can be used to remove gap lock inherit code from purge process.

===============================================================================
Extend gap locks in row_search_mvcc().

This is preliminary code without a good testing. The general logic is
the following:

1) Use two directions to extend gap locks - FORWARD and BACKWARD only if
"direction" argument of row_search_mvcc() is 0, otherwise use only
FORWARD.

2) FORWARD and BACKWARD does not really mean forward or backward
iteration through B-tree leafs, FORWARD corresponds to the same
direction which was choosen in row_search_mvcc() when "moves_up"
variables is set, while BACKWARD means opposite direction.

3) If "direction" argument of row_search_mvcc() is 0 then cursor
position is stored in local cursor object before going to the next record in
BACKWARD direction.

4) When the first non-delete-marked record is reached in BACKWARD scan,
mini-transaction is committed, the cursor position is restored from local
cursor object and scan direction is changed.

Currently all innodb tests are passed.

Things to do:

1) Copy changes in row_sel(),
2) Remove gap lock inheritance from purge process,
3) Add more cases in mtr test:
  a) spatial indexes
  b) tests for row_sel()
  c) ...
  d) PROFIT!!!

Notes for code reviewer:
I count on the preliminary review to be sure I am moving in the correct
direction and did not make some obvoius errors, so please please don't pay
attention on code format and non-full testing.

===============================================================================
row_sel() changes

===============================================================================
do not inherit gap locks on purge

===============================================================================
Test for row_search_mvcc

===============================================================================
Foreign keys constraints check fix.

The problem of the current fix is that it's complexity is n*m. Because
there will be one pass of parent gap for each record in childs gap.

Duplicates check is not implemented.

===============================================================================
This is a try to implement the way of row_sel() testing.

The idea is to have special debug variable innobase_debug_que_eval_sql,
when this variable is set, the inernal innodb query parser is invoked,
and the result is sent to user.

===============================================================================
Add forward scan and insert intention locking for insert operation.

Removed backward scan from foreign key contraints check.

Removed backward scan from row_search_mvcc(), leave it only for the case
of ROW_SEL_EXACT_PREFIX(i.e. for ORDER BY ... DESC).

Removed backward scan from rol_sel() (except ORDER BY ... DESC).

Added forward scan for secondary indexes duplicates check.

===============================================================================
Add debug check.

If purgeable  record has gap, the next record must has gap too.

===============================================================================
Some debug tests. Can be useful for research.

===============================================================================
Code cleanup.

===============================================================================
Revert "This is a try to implement the way of row_sel() testing."

This reverts commit dc33c72c3ba69989e11f377aa902ed9b32f8854a.

===============================================================================
Do not set LOCK_REC_NOT_GAP for delete-marked records in row_search_mvcc()

===============================================================================
Check if lock_update_delete() is invoked from purge process, the deleted
record must be delete-marked.

===============================================================================
Do not take into account insert intention locks on debug check.

===============================================================================
The current commit solves the following issues:

-------------------------------------------------------------------------------
I. If some record is deleted by rollback, it's lock is inherited as gap
lock to the next record. And if the next record is then purged while the
lock is still held, the debug check will fail.

The scenario is the following:

1) Some thread executes "INSERT" and checks clustered index for
duplicates, it sets shared lock for checked record(let's call it record
A) converting implicit lock to explicit one. Note that the record's
transaction id is the same as the current transaction id:
-------------------
0x000055f1e65c6bd6 in lock_rec_create_low (c_lock=0x0, thr=0x0, type_mode=1059, space=11, page_no=3, page=0x2e167079c000 "l\206", <incomplete sequence \372\221>, heap_no=9, index=0x149444393cd0,
    trx=0x55522d18d188, holds_trx_mutex=true) at ./storage/innobase/lock/lock0lock.cc:1466
1466            lock->type_mode = (type_mode & ~LOCK_TYPE_MASK) | LOCK_REC;
(rr) bt
\#0  0x000055f1e65c6bd6 in lock_rec_create_low (c_lock=0x0, thr=0x0, type_mode=1059, space=11, page_no=3, page=0x2e167079c000 "l\206", <incomplete sequence \372\221>, heap_no=9, index=0x149444393cd0,
    trx=0x55522d18d188, holds_trx_mutex=true) at ./storage/innobase/lock/lock0lock.cc:1466
\#1  0x000055f1e65c2b29 in lock_rec_create (c_lock=0x0, thr=0x0, type_mode=1059, block=0x2e1670080560, heap_no=9, index=0x149444393cd0, trx=0x55522d18d188, caller_owns_trx_mutex=true)
    at ./storage/innobase/include/lock0lock.ic:133
\#2  0x000055f1e65c83fe in lock_rec_add_to_queue (type_mode=1059, block=0x2e1670080560, heap_no=9, index=0x149444393cd0, trx=0x55522d18d188, caller_owns_trx_mutex=true)
    at ./storage/innobase/lock/lock0lock.cc:1941
\#3  0x000055f1e65d228f in lock_rec_convert_impl_to_expl_for_trx (block=0x2e1670080560, rec=0x2e167079c299 "\200", index=0x149444393cd0, trx=0x55522d18d188, heap_no=9)
    at ./storage/innobase/lock/lock0lock.cc:5832
\#4  0x000055f1e65d2537 in lock_rec_convert_impl_to_expl (block=0x2e1670080560, rec=0x2e167079c299 "\200", index=0x149444393cd0, offsets=0x663f4cace6d0)
    at ./storage/innobase/lock/lock0lock.cc:5886
\#5  0x000055f1e65d32d9 in lock_clust_rec_read_check_and_lock (flags=0, block=0x2e1670080560, rec=0x2e167079c299 "\200", index=0x149444393cd0, offsets=0x663f4cace6d0, mode=LOCK_S, gap_mode=1024,
    thr=0x611370025b88) at ./storage/innobase/lock/lock0lock.cc:6194
\#6  0x000055f1e666b03c in row_ins_set_shared_rec_lock (type=1024, block=0x2e1670080560, rec=0x2e167079c299 "\200", index=0x149444393cd0, offsets=0x663f4cace6d0, thr=0x611370025b88)
    at ./storage/innobase/row/row0ins.cc:1427
\#7  0x000055f1e666cf4f in row_ins_duplicate_error_in_clust (flags=0, cursor=0x663f4cace9e0, entry=0x61137c044900, thr=0x611370025b88)
    at ./storage/innobase/row/row0ins.cc:2360
\#8  0x000055f1e666db48 in row_ins_clust_index_entry_low (flags=0, mode=2, index=0x149444393cd0, n_uniq=1, entry=0x61137c044900, n_ext=0, thr=0x611370025b88)
    at ./storage/innobase/row/row0ins.cc:2658
\#9  0x000055f1e666f0cb in row_ins_clust_index_entry (index=0x149444393cd0, entry=0x61137c044900, thr=0x611370025b88, n_ext=0)
    at ./storage/innobase/row/row0ins.cc:3146
\#10 0x000055f1e666f4a8 in row_ins_index_entry (index=0x149444393cd0, entry=0x61137c044900, thr=0x611370025b88) at ./storage/innobase/row/row0ins.cc:3265
\#11 0x000055f1e666f9a4 in row_ins_index_entry_step (node=0x611370025658, thr=0x611370025b88) at ./storage/innobase/row/row0ins.cc:3416
\#12 0x000055f1e666fd2a in row_ins (node=0x611370025658, thr=0x611370025b88) at ./storage/innobase/row/row0ins.cc:3553
\#13 0x000055f1e66700c6 in row_ins_step (thr=0x611370025b88) at ./storage/innobase/row/row0ins.cc:3677
\#14 0x000055f1e668c7e0 in row_insert_for_mysql (mysql_rec=0x61137005c460 "\377", prebuilt=0x611370024d50) at ./storage/innobase/row/row0mysql.cc:1408
\#15 0x000055f1e6556c1f in ha_innobase::write_row (this=0x611370033a00, record=0x61137005c460 "\377") at ./storage/innobase/handler/ha_innodb.cc:8284
\#16 0x000055f1e6378051 in handler::ha_write_row (this=0x611370033a00, buf=0x61137005c460 "\377") at ./sql/handler.cc:6118
\#17 0x000055f1e60f21c4 in write_record (thd=0xa48640010a8, table=0x61137005b8c8, info=0x663f4cacfa00) at ./sql/sql_insert.cc:1939
\#18 0x000055f1e60f00f4 in mysql_insert (thd=0xa48640010a8, table_list=0xa4864010d40, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false)
    at ./sql/sql_insert.cc:1066
\#19 0x000055f1e61149d7 in mysql_execute_command (thd=0xa48640010a8) at ./sql/sql_parse.cc:4220
\#20 0x000055f1e611faf6 in mysql_parse (thd=0xa48640010a8,
    rawbuf=0xa48640109e0 "INSERT INTO t6 (col1,col2, col_int, col_string, col_text) VALUES /* NULL */ (NULL,NULL,NULL,REPEAT(SUBSTR(CAST( NULL AS CHAR),1,1), 10),REPEAT(SUBSTR(CAST( NULL AS CHAR),1,1), @fill_amount) ), (NULL,N"..., length=347, parser_state=0x663f4cad0670, is_com_multi=false, is_next_command=false) at ./sql/sql_parse.cc:7796
-------------------

2) Then duplicate key is found in row_ins_duplicate_error_in_clust(),
and the transaction is rolled back. When it's rolled back, the lock is
inherited to the next record(let's call it record B) as a gap lock:
-------------------
\#0  lock_rec_create_low (c_lock=0x0, thr=0x0, type_mode=547, space=11, page_no=3, page=0x2e167079c000 "l\206", <incomplete sequence \372\221>, heap_no=33, index=0x149444393cd0, trx=0x55522d18d188,
    holds_trx_mutex=false) at ./storage/innobase/lock/lock0lock.cc:1467
\#1  0x000055f1e65c2b29 in lock_rec_create (c_lock=0x0, thr=0x0, type_mode=547, block=0x2e1670080560, heap_no=33, index=0x149444393cd0, trx=0x55522d18d188, caller_owns_trx_mutex=false)
    at ./storage/innobase/include/lock0lock.ic:133
\#2  0x000055f1e65c83fe in lock_rec_add_to_queue (type_mode=547, block=0x2e1670080560, heap_no=33, index=0x149444393cd0, trx=0x55522d18d188, caller_owns_trx_mutex=false)
    at ./storage/innobase/lock/lock0lock.cc:1941
\#3  0x000055f1e65c9fee in lock_rec_inherit_to_gap (heir_block=0x2e1670080560, block=0x2e1670080560, heir_heap_no=33, heap_no=9)
    at ./storage/innobase/lock/lock0lock.cc:2580
\#4  0x000055f1e65cc057 in lock_update_delete (block=0x2e1670080560, rec=0x2e167079c299 "\200", from_purge=false)
    at ./storage/innobase/lock/lock0lock.cc:3559
\#5  0x000055f1e6788e57 in btr_cur_optimistic_delete (cursor=0xa486405aff0, flags=0, mtr=0x663f4cacf250, from_purge=false)
    at ./storage/innobase/btr/btr0cur.cc:5252
\#6  0x000055f1e68af6aa in row_undo_ins_remove_clust_rec (node=0xa486405af80) at ./storage/innobase/row/row0uins.cc:141
\#7  0x000055f1e68b05b5 in row_undo_ins (node=0xa486405af80, thr=0xa4864042d78) at ./storage/innobase/row/row0uins.cc:518
\#8  0x000055f1e66d7d80 in row_undo (node=0xa486405af80, thr=0xa4864042d78) at ./storage/innobase/row/row0undo.cc:298
\#9  0x000055f1e66d7f2d in row_undo_step (thr=0xa4864042d78) at ./storage/innobase/row/row0undo.cc:351
\#10 0x000055f1e663e6a0 in que_thr_step (thr=0xa4864042d78) at ./storage/innobase/que/que0que.cc:1039
\#11 0x000055f1e663e8c1 in que_run_threads_low (thr=0xa4864042d78) at ./storage/innobase/que/que0que.cc:1103
\#12 0x000055f1e663ea73 in que_run_threads (thr=0xa4864042d78) at ./storage/innobase/que/que0que.cc:1143
\#13 0x000055f1e6733cb9 in trx_rollback_to_savepoint_low (trx=0x55522d18d188, savept=0x55522d18e198) at ./storage/innobase/trx/trx0roll.cc:107
\#14 0x000055f1e6733f5f in trx_rollback_to_savepoint (trx=0x55522d18d188, savept=0x55522d18e198) at ./storage/innobase/trx/trx0roll.cc:148
\#15 0x000055f1e6734756 in trx_rollback_last_sql_stat_for_mysql (trx=0x55522d18d188) at ./storage/innobase/trx/trx0roll.cc:281
\#16 0x000055f1e654fb65 in innobase_rollback (hton=0x55f1e8c17968, thd=0xa48640010a8, rollback_trx=false) at ./storage/innobase/handler/ha_innodb.cc:4875
\#17 0x000055f1e636dfb4 in ha_rollback_trans (thd=0xa48640010a8, all=false) at ./sql/handler.cc:1708
\#18 0x000055f1e6262a1b in trans_rollback_stmt (thd=0xa48640010a8) at ./sql/transaction.cc:565
\#19 0x000055f1e611b5e4 in mysql_execute_command (thd=0xa48640010a8) at ./sql/sql_parse.cc:6067
\#20 0x000055f1e611faf6 in mysql_parse (thd=0xa48640010a8,
    rawbuf=0xa48640109e0 "INSERT INTO t6 (col1,col2, col_int, col_string, col_text) VALUES /* NULL */ (NULL,NULL,NULL,REPEAT(SUBSTR(CAST( NULL AS CHAR),1,1), 10),REPEAT(SUBSTR(CAST( NULL AS CHAR),1,1), @fill_amount) ), (NULL,N"..., length=347, parser_state=0x663f4cad0670, is_com_multi=false, is_next_command=false) at ./sql/sql_parse.cc:7796
-------------------

3) purge is invoked, it tries to purge record B, record B has gap
lock, but the record next to the record B does not have gap lock, the
debug check is failed.

But initially on step 1 the acquired lock is not gap lock:
-----------------
dberr_t
row_ins_duplicate_error_in_clust(...)
{
...
        if (cursor->low_match >= n_unique) {
...
                        if (flags & BTR_NO_LOCKING_FLAG) {
                                /* Do nothing if no-locking is set */
                                err = DB_SUCCESS;
                        } else if (trx->duplicates) {
                                /* If the SQL-query will update or replace
                                duplicate key we will take X-lock for
                                duplicates ( REPLACE, LOAD DATAFILE REPLACE,
                                INSERT ON DUPLICATE KEY UPDATE). */
                                err = row_ins_set_exclusive_rec_lock(
                                        LOCK_REC_NOT_GAP,
                                        btr_cur_get_block(cursor),
                                        rec, cursor->index, offsets, thr);
                        } else {
                                err = row_ins_set_shared_rec_lock(
                                        LOCK_REC_NOT_GAP,
                                        btr_cur_get_block(cursor), rec,
                                        cursor->index, offsets, thr);
                        }
        }
...
}
-----------------

Then lock_rec_inherit_to_gap() copies this non-gap lock to gap lock
to the next record when transaction is rolled back and the record is
being deleted with btr_cur_optimistic_delete().

So, rollback converted that into a gap lock, what is wrong, the lock should
simply be deleted.

For this purpose convert_lock_to_gap flag is added to
lock_rec_inherit_to_gap() function arguments. When this flag is not set,
lock_rec_inherit_to_gap() ignores non-gap locks, and this flag is set
when lock_rec_inherit_to_gap() is invoked from rollback.

-------------------------------------------------------------------------------
II. When locking read is in progress, and requested ordinary-lock can not be
granted for delete-marked record due to conflicting lock, mtr is
committed, page latch is released, and purge thread can try to purge the
record. The debug check will fail as the record next to delete-marked ordinary-
locked one is not ordinary-locked.

To solve this issue hash-table of scanned record ids(page_id, heap_no
pairs) is stored in trx_t. After locking read is finished at the end of
row_search_mvcc() and rol_sel(), the hash-table is cleaned-up.

When permanent cursor is restored after the lock is granted and the
transaction thread is woken up, and the record stored in the cursor is
purged, then the position will be set to the previous or next record
dependin on the direction of scanning.

-------------------------------------------------------------------------------
Warning: the current implementation for row_sel() is wrong, because the
behaviour when there is conflicting lock is not the same as in
row_search_mvcc(), i.e. when transaction is suspended/woken up, the execution
does not leave row_search_mvcc(), while for row_sel() all necessary
steps to suspend/wake-up the thread are executed outside on row_sel(),
at the higher layer.

===============================================================================
The new system variable is added to test row_sel().

Some initial test is also added.

mysql-test/suite/innodb/r/gap_lock_dup_check.result

@@ -0,0 +1,20 @@
+CREATE TABLE t1 (a int not null, b int not null, primary key(a), unique key(b)) engine = innodb;
+insert into t1 values (10,10), (20,20), (30,30), (40,40), (50,50), (60,60),
+(70,70), (80,80), (90,90);
+connect prevent_purge,localhost,root,,;
+start transaction with consistent snapshot;
+connection default;
+delete from t1 where a between 30 and 80;
+connect con1, localhost, root;
+begin;
+insert into t1 values (65,60);
+connection default;
+SET session innodb_lock_wait_timeout=1;
+insert into t1 values (45,45);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+insert into t1 values (75,75);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+disconnect con1;
+disconnect prevent_purge;
+connection default;
+drop table t1;

mysql-test/suite/innodb/r/gap_lock_fk_check.result

@@ -0,0 +1,76 @@
+SET @saved_frequency = @@GLOBAL.innodb_purge_rseg_truncate_frequency;
+SET GLOBAL innodb_purge_rseg_truncate_frequency = 1;
+#
+# Create parent table
+#
+create table parent(a int unsigned primary key) engine=innodb;
+connect  prevent_purge,localhost,root,,;
+start transaction with consistent snapshot;
+connection default;
+begin;
+#
+# Fill parent table
+#
+insert into parent select seq*10 from seq_1_to_16384;
+#
+# Create child table
+#
+CREATE TABLE child (
+id INT unsigned NOT NULL PRIMARY KEY,
+fl0_id INT unsigned,
+CONSTRAINT `fkl0`
+    FOREIGN KEY (fl0_id) REFERENCES parent (a)
+ON DELETE CASCADE
+ON UPDATE RESTRICT
+) ENGINE = InnoDB;
+#
+# Insert into child table
+#
+insert into child select seq*10, seq*10 from seq_1_to_16384;
+SET @pg = (4096 * 10);
+SET @pg_gap = ((4096 * 10)*6);
+SET @pg_not_gap = ((4096 * 10)*2);
+#
+# Insert some row in the parent table to have ability to insert
+# row with the same foreign key in the child table
+#
+insert into parent values(@pg + (@pg_gap/2) + 5);
+insert into parent values(55);
+insert into parent values((16384 * 10) - 55);;
+#
+# delete some range from child table
+#
+delete from child where id between @pg and @pg + @pg_gap;
+delete from child where id < 100;
+delete from child where id > ((16384 * 10) - 100);
+commit;
+begin;
+#
+# Delete some row from the parent table to lock gap in the child table
+#
+delete from parent where a = @pg + (@pg_gap/2);
+delete from parent where a = 60;
+delete from parent where a = ((16384*10) - 40);;
+connect  con1,localhost,root,,;
+set transaction isolation level serializable;
+begin;
+SET @pg = (4096 * 10);
+SET @pg_gap = ((4096 * 10)*6);
+SET @pg_not_gap = ((4096 * 10)*2);
+SET session innodb_lock_wait_timeout=1;
+#
+# Insert into locked gap
+#
+insert into child values(55, 55);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+insert into child values((16384 * 10) - 55, (16384 * 10) - 55);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+insert into child values(@pg + (@pg_gap/2) + 5, @pg + (@pg_gap/2) + 5);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+disconnect con1;
+disconnect prevent_purge;
+connection default;
+rollback;
+drop table child;
+drop table parent;
+SET GLOBAL innodb_purge_rseg_truncate_frequency = @saved_frequency;

mysql-test/suite/innodb/r/gap_lock_rollback.result

@@ -0,0 +1,57 @@
+create table t(k tinyint unsigned primary key) engine=innodb;
+insert into t values (60), (70), (80), (90), (100);
+insert into t values (160), (170), (180), (190), (200);
+begin;
+insert into t values (10);
+insert into t values (20);
+insert into t values (30);
+insert into t values (40);
+insert into t values (50);
+insert into t values (110);
+insert into t values (120);
+insert into t values (130);
+insert into t values (140);
+insert into t values (150);
+insert into t values (210);
+insert into t values (220);
+insert into t values (230);
+insert into t values (240);
+insert into t values (250);
+connect  prevent_purge,localhost,root,,;
+start transaction with consistent snapshot;
+connection default;
+rollback;
+connect  con1,localhost,root,,;
+set transaction isolation level serializable;
+begin;
+select * from t where k = 40;
+k
+select * from t where k = 140;
+k
+select * from t where k = 240;
+k
+connect  con2,localhost,root,,;
+SET session innodb_lock_wait_timeout=1;
+insert into t values (5);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+insert into t values (45);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+insert into t values (55);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+insert into t values (105);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+insert into t values (145);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+insert into t values (155);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+insert into t values (205);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+insert into t values (245);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+insert into t values (255);
+ERROR HY000: Lock wait timeout exceeded; try restarting transaction
+disconnect con2;
+disconnect con1;
+disconnect prevent_purge;
+connection default;
+drop table t;