FB8-87, FB8-119: Supporting dynamic scheduling in MTS #999

dutow · 2019-03-22T14:48:10Z

Jira ticket: https://jira.percona.com/browse/FB8-87
Jira ticket: https://jira.percona.com/browse/FB8-119

Reference commit: 5846968
Reference commit: d6217c9

Statically assigning shards to slave workers can cause imbalance if a few shards are hotter than others. We should be able to check the imbalance among slave workers and dynamically reassign shards.

percona-ysorokin · 2019-03-24T13:16:44Z

mysql-test/suite/rpl/t/rpl_slave_db_load_table.test

+# This test verifies the functionality of slave_db_load table.
+
+source include/master-slave.inc;
+source include/have_debug.inc;


swap these 2 includes

percona-ysorokin · 2019-03-24T13:19:17Z

mysql-test/suite/rpl/t/rpl_slave_db_load_table.test

+source include/have_debug.inc;
+
+let rpl_server_number=2;
+source include/rpl_restart_server.inc;


Probably,
source include/rpl_stop_server.inc; is we are starting in at line 15?

percona-ysorokin · 2019-03-24T13:19:55Z

mysql-test/suite/rpl/t/rpl_slave_db_load_table.test

+let rpl_server_number=2;
+source include/rpl_restart_server.inc;
+
+connection slave;


Suggested change

connection slave;

source include/rpl_connection_slave.inc;

percona-ysorokin · 2019-03-24T13:20:38Z

mysql-test/suite/rpl/t/rpl_slave_db_load_table.test

+set global mts_dynamic_rebalance=TRUE;
+set global slave_parallel_workers=2;
+set global debug="+d,skip_checkpoint_load_reset";
+start slave;


Suggested change

start slave;

source include/start_slave.inc

percona-ysorokin · 2019-03-24T13:21:13Z

mysql-test/suite/rpl/t/rpl_slave_db_load_table.test

+# Load should be empty
+select * from information_schema.slave_db_load;
+
+connection master;


Suggested change

connection master;

source include/rpl_connection_master.inc;

percona-ysorokin · 2019-03-24T15:30:26Z

sql/sql_parse_com_rpc.cc

+
+void srv_session_end_statement(Srv_session* session) {}
+
+#endif // #ifndef EMBEDDED_LIBRARY


No EMBEDDED_LIBRARY in 8.0

percona-ysorokin · 2019-03-24T15:31:44Z

sql/sql_parse_com_rpc.h

@@ -0,0 +1,11 @@
+#pragma once


#pragma once is a non-standard extension. Use standard #define header guards.

percona-ysorokin · 2019-03-24T15:32:57Z

sql/rpl_rli_pdb.h

+
+  worker_load(Slave_worker *worker) : worker_load(worker, 0) {}
+
+  bool operator>(const worker_load &other) const {


Suggested change

bool operator>(const worker_load &other) const {

bool operator>(const worker_load &other) const noexcept {

percona-ysorokin · 2019-03-24T15:34:15Z

sql/sql_show.cc

+		    continue;
+	    }
+
+  mysql_mutex_lock(&rli->slave_worker_hash_lock);


Formatting (indentation)

percona-ysorokin · 2019-03-24T15:40:29Z

sql/rpl_rli_pdb.cc

+    if (new_max_wrk_load < wrk_load.load) new_max_wrk_load = wrk_load.load;
+    total_wrk_load += entry.second->load;
+
+    heap.push(wrk_load);


Suggested change

heap.push(wrk_load);

heap.emplace(std::move(wrk_load));

dutow · 2019-04-03T17:46:01Z

Updated.

percona-ysorokin

LGTM

facebook-github-bot

@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

hermanlee

I think a number of tests like main.information_schema_* need to be re-recorded since SLAVE_DB_LOAD was added.

main.information_schema_db
main.information_schema_cs

There might be some ci ones too?

hermanlee · 2019-04-12T00:19:25Z

sql/rpl_rli.cc

@@ -2599,7 +2599,9 @@ void Relay_log_info::clear_relay_log_truncated() {
 }

 bool Relay_log_info::is_time_for_mts_checkpoint() {
-  if (is_parallel_exec() && opt_mts_checkpoint_period != 0) {
+  bool period_check = opt_mts_checkpoint_period != 0 &&
+                      !curr_group_seen_begin && !curr_group_seen_gtid;


I seem to be seeing a ubsan runtime issue here:

../../sql/rpl_rli.cc:2603:50: runtime error: load of value 86, which is not a valid value for type 'bool' #0 0x6ac5253 in Relay_log_info::is_time_for_mts_checkpoint() sql/rpl_rli.cc:2603:50 #1 0x6a66fa0 in exec_relay_log_event(THD*, Relay_log_info*, Rpl_applier_reader*) sql/rpl_slave.cc:4696:23 #2 0x6a3b883 in handle_slave_sql sql/rpl_slave.cc:6826:9 #3 0x81047da in pfs_spawn_thread(void*) storage/perfschema/pfs.cc:2836:3 #4 0x7f63b9ff966d in start_thread (/usr/local/fbcode/platform007/lib/libpthread.so.0+0x766d) #5 0x7f63bb06fe2e in __GI___clone (/usr/local/fbcode/platform007/lib/libc.so.6+0x11ae2e)

I see curr_group_seen_begin initialized in the constructor, but not curr_group_seen_gtid. But it's not clear how we are skipping the initialization of these two variables in slave_start_workers(), but it looks like initializing curr_group_seen_gtid in the constructor removes the problem.

Jira ticket: https://jira.percona.com/browse/FB8-87 Jira ticket: https://jira.percona.com/browse/FB8-119 Reference commit: 5846968 Reference commit: d6217c9 -------- 5846968 -------- Statically assigning shards to slave workers can cause imbalance if a few shards are hotter than others. We should be able to check the imbalance among slave workers and dynamically reassign shards.

facebook-github-bot · 2019-04-29T05:04:03Z

@dutow has updated the pull request. Re-import the pull request

dutow · 2019-04-30T08:12:35Z

updated.

facebook-github-bot

@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Summary: Jira ticket: https://jira.percona.com/browse/FB8-87 Jira ticket: https://jira.percona.com/browse/FB8-119 Reference commit: 5846968 Reference commit: d6217c9 Statically assigning shards to slave workers can cause imbalance if a few shards are hotter than others. We should be able to check the imbalance among slave workers and dynamically reassign shards. Pull Request resolved: #999 Differential Revision: D14883857 fbshipit-source-id: efc8ddf

Summary: Jira ticket: https://jira.percona.com/browse/FB8-87 Jira ticket: https://jira.percona.com/browse/FB8-119 Reference commit: 5846968 Reference commit: d6217c9 Statically assigning shards to slave workers can cause imbalance if a few shards are hotter than others. We should be able to check the imbalance among slave workers and dynamically reassign shards. Pull Request resolved: #999 Differential Revision: D14883857 fbshipit-source-id: 7c13d7f

Summary: Jira ticket: https://jira.percona.com/browse/FB8-87 Jira ticket: https://jira.percona.com/browse/FB8-119 Reference commit: facebook@5846968 Reference commit: facebook@d6217c9 Statically assigning shards to slave workers can cause imbalance if a few shards are hotter than others. We should be able to check the imbalance among slave workers and dynamically reassign shards. Pull Request resolved: facebook#999 Differential Revision: D14883857 fbshipit-source-id: 7c13d7f

…acebook#999) Summary: Jira ticket: https://jira.percona.com/browse/FB8-87 Jira ticket: https://jira.percona.com/browse/FB8-119 Reference commit: facebook@5846968 Reference commit: facebook@d6217c9 Statically assigning shards to slave workers can cause imbalance if a few shards are hotter than others. We should be able to check the imbalance among slave workers and dynamically reassign shards. Pull Request resolved: facebook#999 Differential Revision: D14883857

…ercona#999) Summary: Jira ticket: https://jira.percona.com/browse/FB8-87 Jira ticket: https://jira.percona.com/browse/FB8-119 Reference commit: facebook/mysql-5.6@5846968 Reference commit: facebook/mysql-5.6@d6217c9 Statically assigning shards to slave workers can cause imbalance if a few shards are hotter than others. We should be able to check the imbalance among slave workers and dynamically reassign shards. Pull Request resolved: facebook/mysql-5.6#999 Differential Revision: D14883857

…acebook#999) Summary: Jira ticket: https://jira.percona.com/browse/FB8-87 Jira ticket: https://jira.percona.com/browse/FB8-119 Reference commit: facebook@5846968 Reference commit: facebook@d6217c9 Statically assigning shards to slave workers can cause imbalance if a few shards are hotter than others. We should be able to check the imbalance among slave workers and dynamically reassign shards. Pull Request resolved: facebook#999 Differential Revision: D14883857

facebook-github-bot added the CLA Signed label Mar 22, 2019

percona-ysorokin suggested changes Mar 24, 2019

View reviewed changes

percona-ysorokin approved these changes Apr 5, 2019

View reviewed changes

dutow changed the title ~~WIP FB8-87, FB8-119: Supporting dynamic scheduling in MTS~~ FB8-87, FB8-119: Supporting dynamic scheduling in MTS Apr 8, 2019

facebook-github-bot reviewed Apr 10, 2019

View reviewed changes

hermanlee requested changes Apr 12, 2019

View reviewed changes

facebook-github-bot reviewed Apr 30, 2019

View reviewed changes

hermanlee approved these changes May 24, 2019

View reviewed changes

hermanlee closed this Jun 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FB8-87, FB8-119: Supporting dynamic scheduling in MTS #999

FB8-87, FB8-119: Supporting dynamic scheduling in MTS #999

dutow commented Mar 22, 2019 •

edited by hermanlee

Loading

percona-ysorokin Mar 24, 2019

percona-ysorokin Mar 24, 2019

percona-ysorokin Mar 24, 2019

percona-ysorokin Mar 24, 2019

percona-ysorokin Mar 24, 2019

percona-ysorokin Mar 24, 2019

percona-ysorokin Mar 24, 2019

percona-ysorokin Mar 24, 2019

percona-ysorokin Mar 24, 2019

percona-ysorokin Mar 24, 2019

dutow commented Apr 3, 2019

percona-ysorokin left a comment

facebook-github-bot left a comment

hermanlee left a comment

hermanlee Apr 12, 2019

facebook-github-bot commented Apr 29, 2019

dutow commented Apr 30, 2019

facebook-github-bot left a comment


		void srv_session_end_statement(Srv_session* session) {}

		#endif // #ifndef EMBEDDED_LIBRARY


		worker_load(Slave_worker *worker) : worker_load(worker, 0) {}

		bool operator>(const worker_load &other) const {

FB8-87, FB8-119: Supporting dynamic scheduling in MTS #999

FB8-87, FB8-119: Supporting dynamic scheduling in MTS #999

Conversation

dutow commented Mar 22, 2019 • edited by hermanlee Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dutow commented Apr 3, 2019

percona-ysorokin left a comment

Choose a reason for hiding this comment

facebook-github-bot left a comment

Choose a reason for hiding this comment

hermanlee left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Apr 29, 2019

dutow commented Apr 30, 2019

facebook-github-bot left a comment

Choose a reason for hiding this comment

dutow commented Mar 22, 2019 •

edited by hermanlee

Loading