FB8-98, FB8-126, FB8-127: Introducing high prpiority DDLs #1011

dutow · 2019-04-02T11:59:12Z

Jira ticket: https://jira.percona.com/browse/FB8-98

Reference commit: 2b59a06
Reference commit: f816fee
Reference commit: 397fa65
Reference commit: 00323f7
Reference commit: bffb8d7
Reference commit: e9cb575
Reference commit: b145f26
Reference commit: 7e41852
Reference commit: 434d496

-------- 2b59a06 --------

This diff introduces high priority DDLs. If enabled, DDLs will kill the
blocking connections that hold shared metadata locks. Below are the details:

Added high_priority_ddl system variable that will allow DDL to kill blocking
connections which hold shared locks. High priority DDLs are only available to
admin users. By default the functionality is turned off.
Only the shared locks (read/write) will be affected, which are DML queries
such as select, insert, update, etc. Any lock higher than upgradable MDL
(i.e. from conflicting DDL) will not be affected.
When killing the blocking connections, their transactions will automatically
be rolled back. This is the same behavior as kill processlist_id command.
The DDL will kill blocking connections at the end of the timeout instead of
instantly. Admin can control the waiting period by setting the session
lock_wait_timeout before issuing the DDL.

-------- 397fa65 --------

Adding high_priority syntax in DDL commands

This diff introduces HIGH_PRIORITY syntax in some DDL commands and optimize
command. The purpose is the same as the system variable high_priority_ddl
which is described in D4784076.

The list of commands that support HIGH_PRIORITY syntax is:

create/drop/alter/truncate table
create/drop index
create/drop trigger
optimize table

With this diff, there are two ways of issuing high priority DDLs:
e.g. CREATE HIGH_PRIORITY INDEX ...

Recap of the high_priority ddl feature:

Only the shared locks (read/write) will be affected, which are DML queries
such as select, insert, update, etc. Any lock higher than upgradable MDL
(i.e. from conflicting DDL) will not be affected.
When killing the blocking connections, their transactions will automatically
be rolled back. This is the same behavior as kill processlist_id command.
The DDL will kill blocking connections at the end of the timeout instead of
instantly. Admin can control the waiting period by setting the session
lock_wait_timeout before issuing the DDL.

-------- b145f26 --------

Add LOCK TABLES as a supported command for high priority ddl

Although LOCK TABLES is not a ddl command, it is useful to have it support taking mdl locks with high priority, via the high_priority_ddl session variable.

-------- 7e41852 --------

Add high_priority_lock_wait_timeout system variable

The original lock_wait_timeout is used for all queries and also the long
running slave threads. It'll be better that high_priority commands (DDLs) use
a separate system variable for timeout, which can be used across the
master/slaves without messing with lock_wait_timeout.

This diff introduces a new system variable high_priority_lock_wait_timeout that
can be set for high_priority command's lock wait timeout, after which the
command's thread will kill shared locks that are blocking the command.

The new variable doesn't apply to regular queries.

percona-ysorokin · 2019-04-02T12:30:14Z

mysql-test/include/rpl_ddl_high_priority.inc

+
+# create t1
+--disable_warnings
+drop table if exists t1;


percona-ysorokin · 2019-04-02T12:31:06Z

mysql-test/include/rpl_ddl_high_priority.inc

+#set @@global.slave_transaction_retries=2;
+#--source include/restart_slave_sql.inc
+
+connection master;


Suggested change

connection master;

--source include/rpl_connection_master.inc

percona-ysorokin · 2019-04-02T12:32:40Z

mysql-test/include/rpl_ddl_high_priority.inc

+connection slave;
+select * from t1;
+
+--echo connection slave_block


Suggested change

--echo connection slave_block

--let $rpl_connection_name= slave_block

--source include/rpl_connect.inc

percona-ysorokin · 2019-04-02T12:33:19Z

mysql-test/include/rpl_ddl_high_priority.inc

+show create table t1;
+insert into t1 values (1), (2), (3);
+
+sync_slave_with_master;


Suggested change

sync_slave_with_master;

--source include/sync_slave_sql_with_master.inc

percona-ysorokin · 2019-04-02T12:34:15Z

mysql-test/include/rpl_ddl_high_priority.inc

+
+sync_slave_with_master;
+
+--echo connection slave


Suggested change

--echo connection slave

--source include/rpl_connection_slave.inc

percona-ysorokin · 2019-04-02T12:50:51Z

sql/sql_class.h

+    @retval  TRUE   if the blocking thread is killed
+    @retval  FALSE  No thread is killed
+   */
+  virtual bool kill_shared_locks(MDL_context_owner *in_use);


Suggested change

virtual bool kill_shared_locks(MDL_context_owner *in_use);

virtual bool kill_shared_locks(MDL_context_owner *in_use) override;

percona-ysorokin · 2019-04-02T12:51:14Z

sql/sql_lex.h

@@ -3612,6 +3612,10 @@ struct LEX : public Query_tables_list {
  /* Partition info structure filled in by PARTITION BY parse part */
  partition_info *part_info;

+
+    /* High priority DDL */


percona-ysorokin · 2019-04-02T12:51:46Z

sql/sql_parse.cc

@@ -1051,6 +1051,29 @@ bool is_explainable_query(enum enum_sql_command command) {
  return (sql_command_flags[command] & CF_CAN_BE_EXPLAINED) != 0;
 }

+bool support_high_priority(enum enum_sql_command command)


Suggested change

bool support_high_priority(enum enum_sql_command command)

bool support_high_priority(enum enum_sql_command command) noexcept

percona-ysorokin · 2019-04-02T12:52:16Z

sql/sql_parse.h

@@ -93,6 +93,7 @@ bool merge_sp_var_charset_and_collation(const CHARSET_INFO **to,
 bool check_host_name(const LEX_CSTRING &str);
 bool mysql_test_parse_for_slave(THD *thd);
 bool is_update_query(enum enum_sql_command command);
+bool support_high_priority(enum enum_sql_command command);


Suggested change

bool support_high_priority(enum enum_sql_command command);

bool support_high_priority(enum enum_sql_command command) noexcept;

percona-ysorokin · 2019-04-02T12:54:18Z

unittest/gunit/test_mdl_context_owner.h

@@ -53,6 +53,8 @@ class Test_MDL_context_owner : public MDL_context_owner {
  virtual void notify_hton_post_release_exclusive(const MDL_key *) {}

  virtual uint get_rand_seed() { return 0; }
+
+     virtual bool kill_shared_locks(MDL_context_owner * /* unused */) { return false; }


Formatting

Suggested change

virtual bool kill_shared_locks(MDL_context_owner * /* unused */) { return false; }

virtual bool kill_shared_locks(MDL_context_owner * /* unused */) override { return false; }

laurynas-biveinis · 2019-04-03T06:17:03Z

mysql-test/include/ddl_high_priority.inc

+
+# Ensure con2 is closed
+--source include/wait_until_count_sessions.inc
+# Secondary check to ensure con2 is closed


The "primary" check checks the code path which is already past the "secondary" check point. Consider removing the secondary check

laurynas-biveinis · 2019-04-03T06:18:33Z

mysql-test/include/ddl_high_priority.inc

+
+drop user test_user1@localhost;
+drop user test_user2@localhost;
+--disable_warnings


No need for warnings and if exists

dutow · 2019-04-10T08:03:06Z

Updated.

percona-ysorokin

LGTM

facebook-github-bot

@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

hermanlee

I don't think PR is working as intended. A simple test, even with regular DML conflicting with DDL, does not pass. It looks like there are some changes in MDL_context::acquire_lock_nsec which is causing the high priority ddl request to loop around notify_conflicting_locks(), and never getting to issuing the kill.

In my test case:

connection 1:
create table t1 (a int, b int, primary key (a)) engine=innodb;
insert into t1 values (1,1);
begin;
select * from t1;

--connection 2:
 create high_priority index idx1 on t1 (b); /* Blocks */

Here's the output from performance_schema.metadata_locks:

mysql> select * from performance_schema.metadata_locks;
+-------------------+--------------------+----------------+-------------+-----------------------+---------------------+---------------+-------------+--------------------+-----------------+----------------+
| OBJECT_TYPE       | OBJECT_SCHEMA      | OBJECT_NAME    | COLUMN_NAME | OBJECT_INSTANCE_BEGIN | LOCK_TYPE           | LOCK_DURATION | LOCK_STATUS | SOURCE             | OWNER_THREAD_ID | OWNER_EVENT_ID |
+-------------------+--------------------+----------------+-------------+-----------------------+---------------------+---------------+-------------+--------------------+-----------------+----------------+
| TABLE             | test               | t1             | NULL        |       140736473807280 | SHARED_READ_ONLY    | TRANSACTION   | GRANTED     | sql_parse.cc:6125  |              42 |          37794 |
| SCHEMA            | test               | NULL           | NULL        |       140736473813200 | INTENTION_EXCLUSIVE | TRANSACTION   | GRANTED     | dd_schema.cc:108   |              42 |          37797 |
| COLUMN STATISTICS | test               | t1             | a           |       140736473814240 | SHARED_READ         | STATEMENT     | GRANTED     | sql_base.cc:573    |              42 |          38549 |
| COLUMN STATISTICS | test               | t1             | b           |       140736473815680 | SHARED_READ         | STATEMENT     | GRANTED     | sql_base.cc:573    |              42 |          38549 |
| GLOBAL            | NULL               | NULL           | NULL        |       140736461202128 | INTENTION_EXCLUSIVE | STATEMENT     | GRANTED     | sql_base.cc:5412   |              46 |           1588 |
| SCHEMA            | test               | NULL           | NULL        |       140736461202688 | INTENTION_EXCLUSIVE | TRANSACTION   | GRANTED     | sql_base.cc:5399   |              46 |           1588 |
| TABLE             | test               | t1             | NULL        |       140736461201888 | SHARED_UPGRADABLE   | TRANSACTION   | GRANTED     | sql_parse.cc:6125  |              46 |           1590 |
| BACKUP LOCK       | NULL               | NULL           | NULL        |       140736461202288 | INTENTION_EXCLUSIVE | TRANSACTION   | GRANTED     | sql_base.cc:5419   |              46 |           1591 |
| TABLESPACE        | NULL               | test/t1        | NULL        |       140736461202528 | INTENTION_EXCLUSIVE | TRANSACTION   | GRANTED     | lock.cc:806        |              46 |           1697 |
| TABLE             | test               | #sql-339fa7_c  | NULL        |       140736461201968 | EXCLUSIVE           | STATEMENT     | GRANTED     | sql_table.cc:14687 |              46 |           1722 |
| TABLE             | test               | t1             | NULL        |       140736461203088 | EXCLUSIVE           | TRANSACTION   | PENDING     | mdl.cc:3720        |              46 |           1732 |
| TABLE             | performance_schema | metadata_locks | NULL        |       140736473816240 | SHARED_READ         | TRANSACTION   | GRANTED     | sql_parse.cc:6125  |              42 |          38921 |
+-------------------+--------------------+----------------+-------------+-----------------------+---------------------+---------------+-------------+--------------------+-----------------+----------------+

hermanlee · 2019-04-23T22:52:29Z

mysql-test/suite/rpl/t/rpl_ddl_high_priority.test

@@ -0,0 +1 @@
+--source include/rpl_ddl_high_priority.inc


Is there a .result file for this test?

There should be, as I remember creating it, and also that it was working. This and the blocking issue is probably related to a bad rebasing/amending or recreating the entire commit from a diff later, as I did that multiple times during this. I'll look into this and fix it.

hermanlee · 2019-04-24T17:35:41Z

mysql-test/include/rpl_ddl_high_priority.inc

+select * from t1;
+
+--let $rpl_connection_name= slave_block
+--source include/rpl_connect.inc


I think this should be include/rpl_connection.inc

hermanlee · 2019-04-24T18:22:48Z

sql/mdl.cc

+
+  while ((conflicting_ticket = it++)) {
+    if (conflicting_ticket->get_ctx() != ctx &&
+        conflicting_ticket->get_type() <= MDL_SHARED_NO_WRITE) {


This seems to have changed from MDL_SHARED_UPGRADEABLE in the original diff to MDL_SHARED_NO_WRITE.

In looking through the code and testing, it looks like lock tables <table> read; has change the enumeration of the lock acquired to MDL_SHARED__READ_ONLY, which is higher than MDL_SHARED_UPGRADEABLE. I assume that is why this was done?

Yes - and I think the issue was more generic than lock tables, but it's been 4 weeks since I debugged this issue.

dutow · 2019-06-04T19:53:02Z

@hermanlee Fixed - I'm still running a full test suite to see if there's any issues, but the high priority part is working again. I left out a part of one of the commits when I rebased on the separate lock wait timeout commit, and that caused these issues.

facebook-github-bot · 2019-06-04T19:54:11Z

@dutow has updated the pull request. Re-import the pull request

facebook-github-bot · 2019-06-05T05:37:16Z

@dutow has updated the pull request. Re-import the pull request

facebook-github-bot

@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

hermanlee · 2019-06-18T18:39:41Z

sql/sql_yacc.yy

          '(' key_list_with_expression ')' opt_fulltext_index_options opt_index_lock_and_algorithm
          {
            $$= NEW_PTN PT_create_index_stmt(YYMEM_ROOT, KEYTYPE_FULLTEXT, $4,
                                             NULL, $6, $8, $10,
                                             $11.algo.get_or_default(),
                                             $11.lock.get_or_default());
          }
-        | CREATE SPATIAL_SYM INDEX_SYM ident ON_SYM table_ident
+        | create_opt SPATIAL_SYM INDEX_SYM ident ON_SYM table_ident


I'm seeing bison register another conflict:

/home/herman/rocks-mysql/8.0/sql/sql_yacc.yy: error: shift/reduce conflicts: 110 found, 109 expected

The conflict comes from the above rule and rule for CREATE SPATIAL_SYM REFERENCE_SYM". Changing "CREATE SPATIAL_SYM REFERENCE_SYM" to "create_opt SPATIAL_SYM REFERENCE_SYM" for the create_srs_stmt production resolves this, but also changing the create_srs_stmt -> CREATE OR_SYM REPLACE_SYM SPATIAL_SYM REFERENCE_SYM causes another conflict.

What is the best way to resolve this, if any?

I would approach this from a different angle:

do you want to add the high priority option to other commands?

if yes, do you actually want those to be logical? (e.g. if one variant gets it, others get it too)

I'm not sure if it can be correctly resolved if one of the answers is no. (and the conflict was there in 5.6 too)

Yes. We should add high priority ddl to those statements too and having them be logical would be fine.

To clarify: this statement is the CREATE SPATIAL REFERENCE SYSTEM, not the CREATE SPATIAL INDEX, and I don't think high priority ddl will do anything for it?

But if I do add the high priority option to it, the warning is fixable, so I updated my PR with that.

If I do not add the option to it, the conflict is only resolvable with a larger refactoring, which I don't think we should do.

Yes, adding high priority DDL to those statements to fix the warning without a major refactoring is fine. That's the temporary fix I'm using at the moment to remove the warning. Since high priority ddl is basically a noop for it, I think it would be okay.

hermanlee · 2019-06-19T00:34:32Z

sql/system_variables.h

@@ -228,6 +228,7 @@ struct System_variables {
  ulonglong histogram_generation_max_mem_size;
  ulong join_buff_size;
  ulonglong lock_wait_timeout_nsec;
+  ulonglong high_priority_lock_wait_timeout_nsec;


I noticed the original patch wasn't updated this initializing this correctly, resulting in it being 0 unless it's explicitly set. The fix is to initialize it similar to lock_wait_timeout_nsec in mysqld.cc:get_options().

hermanlee · 2019-06-20T18:14:35Z

mysql-test/include/rpl_ddl_high_priority.inc

+--source include/rpl_connection_slave.inc
+select * from t1;
+
+--source include/rpl_connection_slave.inc


This needs to be another connection to the slave. It gets killed by high priority ddl and the subsequent sync_slave_with_master call on it will fail.

hermanlee · 2019-06-20T18:14:55Z

mysql-test/include/rpl_ddl_high_priority.inc

+
+--source include/rpl_connection_slave.inc
+
+select * from performance_schema.metadata_locks;


It's not clear to me this output is deterministic..

facebook-github-bot · 2019-06-24T05:48:10Z

@dutow has updated the pull request. Re-import the pull request

facebook-github-bot · 2019-06-25T07:38:06Z

@dutow has updated the pull request. Re-import the pull request

facebook-github-bot

@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

hermanlee · 2019-07-09T22:48:03Z

mysql-test/include/ddl_high_priority.inc

+##
+
+#--let $blocking_sql = lock tables t1 read;
+#--let $cmd = create trigger ins_sum before insert on t1 for each row set @sum = @sum + new.i;


I just noticed this test was disabled and it looks like create trigger always runs as high priority ddl even when high priority ddl is not specified. Have you had a chance to debug this and determine what is causing it?

After discussion with the team, I think we're going to remove the high_priority syntax from the create statements and rely on other means to support this functionality over replication. So no need to investigate this further. Thanks!

So I should remove the syntax related parts of this commit, and keep only the session variable?

Yes, that would be great. It'll simplify the PR. We will rely on the session variable to enable hi priority ddl.

Jira ticket: https://jira.percona.com/browse/FB8-98 Reference commit: 2b59a06 Reference commit: f816fee Reference commit: 397fa65 Reference commit: 00323f7 Reference commit: bffb8d7 Reference commit: e9cb575 Reference commit: b145f26c86e Reference commit: 7e41852 Reference commit: 434d496 -------- 2b59a06 -------- This diff introduces high priority DDLs. If enabled, DDLs will kill the blocking connections that hold shared metadata locks. Below are the details: - Added high_priority_ddl system variable that will allow DDL to kill blocking connections which hold shared locks. High priority DDLs are only available to admin users. By default the functionality is turned off. - Only the shared locks (read/write) will be affected, which are DML queries such as select, insert, update, etc. Any lock higher than upgradable MDL (i.e. from conflicting DDL) will not be affected. - When killing the blocking connections, their transactions will automatically be rolled back. This is the same behavior as `kill processlist_id` command. - The DDL will kill blocking connections at the end of the timeout instead of instantly. Admin can control the waiting period by setting the session lock_wait_timeout before issuing the DDL. -------- 397fa65 -------- Adding high_priority syntax in DDL commands This diff introduces `HIGH_PRIORITY` syntax in some DDL commands and optimize command. The purpose is the same as the system variable `high_priority_ddl` which is described in D4784076. The list of commands that support `HIGH_PRIORITY` syntax is: - create/drop/alter/truncate table - create/drop index - create/drop trigger - optimize table With this diff, there are two ways of issuing high priority DDLs: e.g. `CREATE HIGH_PRIORITY INDEX ...` Recap of the high_priority ddl feature: - Only the shared locks (read/write) will be affected, which are DML queries such as select, insert, update, etc. Any lock higher than upgradable MDL (i.e. from conflicting DDL) will not be affected. - When killing the blocking connections, their transactions will automatically be rolled back. This is the same behavior as `kill processlist_id` command. - The DDL will kill blocking connections at the end of the timeout instead of instantly. Admin can control the waiting period by setting the session lock_wait_timeout before issuing the DDL. -------- b145f26 -------- Add LOCK TABLES as a supported command for high priority ddl Although LOCK TABLES is not a ddl command, it is useful to have it support taking mdl locks with high priority, via the high_priority_ddl session variable. -------- 7e41852 -------- Add high_priority_lock_wait_timeout system variable The original lock_wait_timeout is used for all queries and also the long running slave threads. It'll be better that high_priority commands (DDLs) use a separate system variable for timeout, which can be used across the master/slaves without messing with lock_wait_timeout. This diff introduces a new system variable high_priority_lock_wait_timeout that can be set for high_priority command's lock wait timeout, after which the command's thread will kill shared locks that are blocking the command. The new variable doesn't apply to regular queries.

facebook-github-bot · 2019-07-17T13:11:54Z

@dutow has updated the pull request. Re-import the pull request

facebook-github-bot

@luqun has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Summary: Jira ticket: https://jira.percona.com/browse/FB8-98 Reference commit: 2b59a06 Reference commit: f816fee Reference commit: 397fa65 Reference commit: 00323f7 Reference commit: bffb8d7 Reference commit: e9cb575 Reference commit: b145f26 Reference commit: 7e41852 Reference commit: 434d496 -------- 2b59a06 -------- This diff introduces high priority DDLs. If enabled, DDLs will kill the blocking connections that hold shared metadata locks. Below are the details: - Added high_priority_ddl system variable that will allow DDL to kill blocking connections which hold shared locks. High priority DDLs are only available to admin users. By default the functionality is turned off. - Only the shared locks (read/write) will be affected, which are DML queries such as select, insert, update, etc. Any lock higher than upgradable MDL (i.e. from conflicting DDL) will not be affected. - When killing the blocking connections, their transactions will automatically be rolled back. This is the same behavior as `kill processlist_id` command. - The DDL will kill blocking connections at the end of the timeout instead of instantly. Admin can control the waiting period by setting the session lock_wait_timeout before issuing the DDL. -------- 397fa65 -------- This feature has been dropped. Adding high_priority syntax in DDL commands This diff introduces `HIGH_PRIORITY` syntax in some DDL commands and optimize command. The purpose is the same as the system variable `high_priority_ddl` which is described in D4784076. The list of commands that support `HIGH_PRIORITY` syntax is: - create/drop/alter/truncate table - create/drop index - create/drop trigger - optimize table With this diff, there are two ways of issuing high priority DDLs: e.g. `CREATE HIGH_PRIORITY INDEX ...` Recap of the high_priority ddl feature: - Only the shared locks (read/write) will be affected, which are DML queries such as select, insert, update, etc. Any lock higher than upgradable MDL (i.e. from conflicting DDL) will not be affected. - When killing the blocking connections, their transactions will automatically be rolled back. This is the same behavior as `kill processlist_id` command. - The DDL will kill blocking connections at the end of the timeout instead of instantly. Admin can control the waiting period by setting the session lock_wait_timeout before issuing the DDL. -------- b145f26 -------- Add LOCK TABLES as a supported command for high priority ddl Although LOCK TABLES is not a ddl command, it is useful to have it support taking mdl locks with high priority, via the high_priority_ddl session variable. -------- 7e41852 -------- Add high_priority_lock_wait_timeout system variable The original lock_wait_timeout is used for all queries and also the long running slave threads. It'll be better that high_priority commands (DDLs) use a separate system variable for timeout, which can be used across the master/slaves without messing with lock_wait_timeout. This diff introduces a new system variable high_priority_lock_wait_timeout that can be set for high_priority command's lock wait timeout, after which the command's thread will kill shared locks that are blocking the command. The new variable doesn't apply to regular queries. Pull Request resolved: #1011 Reviewed By: hermanlee Differential Revision: D14884073 Pulled By: hermanlee fbshipit-source-id: 6f3eae1

…) (facebook#1011) Summary: Jira ticket: https://jira.percona.com/browse/FB8-98 Reference commit: 2b59a06 Reference commit: f816fee Reference commit: 397fa65 Reference commit: 00323f7 Reference commit: bffb8d7 Reference commit: e9cb575 Reference commit: b145f26 Reference commit: 7e41852 Reference commit: 434d496 Reference commit: facebook@ee63260f74b -------- 2b59a06 -------- This diff introduces high priority DDLs. If enabled, DDLs will kill the blocking connections that hold shared metadata locks. Below are the details: - Added high_priority_ddl system variable that will allow DDL to kill blocking connections which hold shared locks. High priority DDLs are only available to admin users. By default the functionality is turned off. - Only the shared locks (read/write) will be affected, which are DML queries such as select, insert, update, etc. Any lock higher than upgradable MDL (i.e. from conflicting DDL) will not be affected. - When killing the blocking connections, their transactions will automatically be rolled back. This is the same behavior as `kill processlist_id` command. - The DDL will kill blocking connections at the end of the timeout instead of instantly. Admin can control the waiting period by setting the session lock_wait_timeout before issuing the DDL. -------- 397fa65 -------- This feature has been dropped. Adding high_priority syntax in DDL commands This diff introduces `HIGH_PRIORITY` syntax in some DDL commands and optimize command. The purpose is the same as the system variable `high_priority_ddl` which is described in D4784076 (facebook@2b59a06). The list of commands that support `HIGH_PRIORITY` syntax is: - create/drop/alter/truncate table - create/drop index - create/drop trigger - optimize table With this diff, there are two ways of issuing high priority DDLs: e.g. `CREATE HIGH_PRIORITY INDEX ...` Recap of the high_priority ddl feature: - Only the shared locks (read/write) will be affected, which are DML queries such as select, insert, update, etc. Any lock higher than upgradable MDL (i.e. from conflicting DDL) will not be affected. - When killing the blocking connections, their transactions will automatically be rolled back. This is the same behavior as `kill processlist_id` command. - The DDL will kill blocking connections at the end of the timeout instead of instantly. Admin can control the waiting period by setting the session lock_wait_timeout before issuing the DDL. -------- b145f26 -------- Add LOCK TABLES as a supported command for high priority ddl Although LOCK TABLES is not a ddl command, it is useful to have it support taking mdl locks with high priority, via the high_priority_ddl session variable. -------- 7e41852 -------- Add high_priority_lock_wait_timeout system variable The original lock_wait_timeout is used for all queries and also the long running slave threads. It'll be better that high_priority commands (DDLs) use a separate system variable for timeout, which can be used across the master/slaves without messing with lock_wait_timeout. This diff introduces a new system variable high_priority_lock_wait_timeout that can be set for high_priority command's lock wait timeout, after which the command's thread will kill shared locks that are blocking the command. The new variable doesn't apply to regular queries. Pull Request resolved: facebook#1011 Differential Revision: D14884073 Pulled By: hermanlee

… (percona#1011) Summary: Jira ticket: https://jira.percona.com/browse/FB8-98 Reference commit: 2b59a06 Reference commit: f816fee Reference commit: 397fa65 Reference commit: 00323f7 Reference commit: bffb8d7 Reference commit: e9cb575 Reference commit: b145f26 Reference commit: 7e41852 Reference commit: 434d496 Reference commit: facebook/mysql-5.6@ee63260f74b -------- 2b59a06 -------- This diff introduces high priority DDLs. If enabled, DDLs will kill the blocking connections that hold shared metadata locks. Below are the details: - Added high_priority_ddl system variable that will allow DDL to kill blocking connections which hold shared locks. High priority DDLs are only available to admin users. By default the functionality is turned off. - Only the shared locks (read/write) will be affected, which are DML queries such as select, insert, update, etc. Any lock higher than upgradable MDL (i.e. from conflicting DDL) will not be affected. - When killing the blocking connections, their transactions will automatically be rolled back. This is the same behavior as `kill processlist_id` command. - The DDL will kill blocking connections at the end of the timeout instead of instantly. Admin can control the waiting period by setting the session lock_wait_timeout before issuing the DDL. -------- 397fa65 -------- This feature has been dropped. Adding high_priority syntax in DDL commands This diff introduces `HIGH_PRIORITY` syntax in some DDL commands and optimize command. The purpose is the same as the system variable `high_priority_ddl` which is described in D4784076 (facebook/mysql-5.6@2b59a06). The list of commands that support `HIGH_PRIORITY` syntax is: - create/drop/alter/truncate table - create/drop index - create/drop trigger - optimize table With this diff, there are two ways of issuing high priority DDLs: e.g. `CREATE HIGH_PRIORITY INDEX ...` Recap of the high_priority ddl feature: - Only the shared locks (read/write) will be affected, which are DML queries such as select, insert, update, etc. Any lock higher than upgradable MDL (i.e. from conflicting DDL) will not be affected. - When killing the blocking connections, their transactions will automatically be rolled back. This is the same behavior as `kill processlist_id` command. - The DDL will kill blocking connections at the end of the timeout instead of instantly. Admin can control the waiting period by setting the session lock_wait_timeout before issuing the DDL. -------- b145f26 -------- Add LOCK TABLES as a supported command for high priority ddl Although LOCK TABLES is not a ddl command, it is useful to have it support taking mdl locks with high priority, via the high_priority_ddl session variable. -------- 7e41852 -------- Add high_priority_lock_wait_timeout system variable The original lock_wait_timeout is used for all queries and also the long running slave threads. It'll be better that high_priority commands (DDLs) use a separate system variable for timeout, which can be used across the master/slaves without messing with lock_wait_timeout. This diff introduces a new system variable high_priority_lock_wait_timeout that can be set for high_priority command's lock wait timeout, after which the command's thread will kill shared locks that are blocking the command. The new variable doesn't apply to regular queries. Pull Request resolved: facebook/mysql-5.6#1011 Differential Revision: D14884073 Pulled By: hermanlee

…) (facebook#1011) Summary: Jira ticket: https://jira.percona.com/browse/FB8-98 Reference commit: 2b59a06 Reference commit: f816fee Reference commit: 397fa65 Reference commit: 00323f7 Reference commit: bffb8d7 Reference commit: e9cb575 Reference commit: b145f26 Reference commit: 7e41852 Reference commit: 434d496 Reference commit: facebook@ee63260f74b -------- 2b59a06 -------- This diff introduces high priority DDLs. If enabled, DDLs will kill the blocking connections that hold shared metadata locks. Below are the details: - Added high_priority_ddl system variable that will allow DDL to kill blocking connections which hold shared locks. High priority DDLs are only available to admin users. By default the functionality is turned off. - Only the shared locks (read/write) will be affected, which are DML queries such as select, insert, update, etc. Any lock higher than upgradable MDL (i.e. from conflicting DDL) will not be affected. - When killing the blocking connections, their transactions will automatically be rolled back. This is the same behavior as `kill processlist_id` command. - The DDL will kill blocking connections at the end of the timeout instead of instantly. Admin can control the waiting period by setting the session lock_wait_timeout before issuing the DDL. -------- 397fa65 -------- This feature has been dropped. Adding high_priority syntax in DDL commands This diff introduces `HIGH_PRIORITY` syntax in some DDL commands and optimize command. The purpose is the same as the system variable `high_priority_ddl` which is described in D4784076 (facebook@2b59a06). The list of commands that support `HIGH_PRIORITY` syntax is: - create/drop/alter/truncate table - create/drop index - create/drop trigger - optimize table With this diff, there are two ways of issuing high priority DDLs: e.g. `CREATE HIGH_PRIORITY INDEX ...` Recap of the high_priority ddl feature: - Only the shared locks (read/write) will be affected, which are DML queries such as select, insert, update, etc. Any lock higher than upgradable MDL (i.e. from conflicting DDL) will not be affected. - When killing the blocking connections, their transactions will automatically be rolled back. This is the same behavior as `kill processlist_id` command. - The DDL will kill blocking connections at the end of the timeout instead of instantly. Admin can control the waiting period by setting the session lock_wait_timeout before issuing the DDL. -------- b145f26 -------- Add LOCK TABLES as a supported command for high priority ddl Although LOCK TABLES is not a ddl command, it is useful to have it support taking mdl locks with high priority, via the high_priority_ddl session variable. -------- 7e41852 -------- Add high_priority_lock_wait_timeout system variable The original lock_wait_timeout is used for all queries and also the long running slave threads. It'll be better that high_priority commands (DDLs) use a separate system variable for timeout, which can be used across the master/slaves without messing with lock_wait_timeout. This diff introduces a new system variable high_priority_lock_wait_timeout that can be set for high_priority command's lock wait timeout, after which the command's thread will kill shared locks that are blocking the command. The new variable doesn't apply to regular queries. Pull Request resolved: facebook#1011 Differential Revision: D14884073 Pulled By: hermanlee

facebook-github-bot added the CLA Signed label Apr 2, 2019

percona-ysorokin suggested changes Apr 2, 2019

View reviewed changes

laurynas-biveinis suggested changes Apr 3, 2019

View reviewed changes

percona-ysorokin approved these changes Apr 10, 2019

View reviewed changes

dutow changed the title ~~WIP FB8-98, FB8-126, FB8-127: Introducing high prpiority DDLs~~ FB8-98, FB8-126, FB8-127: Introducing high prpiority DDLs Apr 10, 2019

facebook-github-bot reviewed Apr 10, 2019

View reviewed changes

hermanlee requested changes Apr 24, 2019

View reviewed changes

facebook-github-bot reviewed Jun 11, 2019

View reviewed changes

facebook-github-bot reviewed Jun 12, 2019

View reviewed changes

hermanlee reviewed Jun 18, 2019

View reviewed changes

hermanlee reviewed Jun 19, 2019

View reviewed changes

hermanlee reviewed Jun 20, 2019

View reviewed changes

facebook-github-bot reviewed Jun 27, 2019

View reviewed changes

hermanlee reviewed Jul 9, 2019

View reviewed changes

facebook-github-bot reviewed Jul 17, 2019

View reviewed changes

hermanlee closed this Aug 19, 2019

	connection master;
	--source include/rpl_connection_master.inc

	--echo connection slave_block
	--let $rpl_connection_name= slave_block
	--source include/rpl_connect.inc

	sync_slave_with_master;
	--source include/sync_slave_sql_with_master.inc

	--echo connection slave
	--source include/rpl_connection_slave.inc

	virtual bool kill_shared_locks(MDL_context_owner *in_use);
	virtual bool kill_shared_locks(MDL_context_owner *in_use) override;

	bool support_high_priority(enum enum_sql_command command)
	bool support_high_priority(enum enum_sql_command command) noexcept


		--source include/rpl_connection_slave.inc

		select * from performance_schema.metadata_locks;

FB8-98, FB8-126, FB8-127: Introducing high prpiority DDLs #1011

FB8-98, FB8-126, FB8-127: Introducing high prpiority DDLs #1011

Conversation

dutow commented Apr 2, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dutow commented Apr 10, 2019

percona-ysorokin left a comment

Choose a reason for hiding this comment

facebook-github-bot left a comment

Choose a reason for hiding this comment

hermanlee left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dutow commented Jun 4, 2019

facebook-github-bot commented Jun 4, 2019

facebook-github-bot commented Jun 5, 2019

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Jun 24, 2019

facebook-github-bot commented Jun 25, 2019

facebook-github-bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Jul 17, 2019

facebook-github-bot left a comment

Choose a reason for hiding this comment

dutow commented Apr 2, 2019 •

edited

Loading