FB8-132: NUMA support #987

dutow · 2019-03-18T09:03:26Z

Jira ticket: https://jira.percona.com/browse/FB8-132

Reference commit: e50c754
Reference commit: 286e975

-------- e50c754 --------

WebScaleSQL Feature: NUMA Support

Credits for research and implementation: Jeremy Cole and Davi Arnaut

This patch provides startup options:

flush-caches: Flush and purge buffers/caches
numa-interleave: Run mysqld with its memory interleaved on all CPUs

It also provides a config option:

innodb_buffer_pool_populate: pre-allocation of buffer pool
memory at start up:
-- Use MAP_POPULATE if supported (Linux 2.6.23 and higher)
-- Forced pre-allocation using memset

-------- 286e975 --------

Automation will pass in the right value to this option if
it needs to bind mysqld to a particular socket. That will happen
via systemd scripts.

This is only adding the capability on the mysqld_safe side to be able to
pass in numactl to the server. The turning on of this feature will need more
testing and slow roll, and that will happen in different diffs on the ops repo.

percona-ysorokin

LGTM

dutow · 2019-03-18T15:20:55Z

Note: I didn't do much testing - I have no numa capable hardware.

laurynas-biveinis · 2019-03-19T06:43:00Z

scripts/mysqld_safe.sh

@@ -17,6 +17,9 @@ MYSQLD=
 niceness=0
 mysqld_ld_preload=
 mysqld_ld_library_path=
+flush_caches=0
+numa_interleave=0


Isn't mysqld_safe.sh numa_interleave option redundant due to 5.7 core server --innodb-numa-interleave?

laurynas-biveinis · 2019-03-19T06:43:45Z

storage/innobase/handler/ha_innodb.cc

+    PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY,
+    "Preallocate (pre-fault) the page frames required for the mapping "
+    "established by the buffer pool memory region. Disabled by default.",
+    NULL, NULL, FALSE);


Suggested change

NULL, NULL, FALSE);

nullptr, nullptr, false);

laurynas-biveinis · 2019-03-19T06:44:05Z

storage/innobase/os/os0proc.cc

@@ -70,10 +83,25 @@ ulint os_proc_get_number(void) {
 #endif
 }

+/****************************************************************/ /**


New style header comment please

laurynas-biveinis · 2019-03-19T06:44:20Z

storage/innobase/os/os0proc.cc

@@ -70,10 +83,25 @@ ulint os_proc_get_number(void) {
 #endif
 }

+/****************************************************************/ /**
+ Retrieve and compare operating system release.
+ @return	TRUE if the OS release is equal to, or later than release. */


Suggested change

@return TRUE if the OS release is equal to, or later than release. */

@return true if the OS release is equal to, or later than release. */

laurynas-biveinis · 2019-03-19T06:44:49Z

storage/innobase/os/os0proc.cc

+  struct utsname name;
+  return (uname(&name) == 0 && strverscmp(name.release, release) >= 0);
+#else
+  return (FALSE);


Suggested change

return (FALSE);

return (false);

laurynas-biveinis · 2019-03-19T06:45:11Z

storage/innobase/os/os0proc.cc

+  populate = populate && !os_compare_release("2.6.23");
+
+  if (ptr && populate) {
+    fprintf(stderr,


hermanlee · 2019-03-22T17:03:59Z

Can this be rebased onto the latest 8.0.13? Thanks!

Jira ticket: https://jira.percona.com/browse/FB8-132 Reference commit: e50c754 Reference commit: 286e975 -------- e50c754 -------- WebScaleSQL Feature: NUMA Support Credits for research and implementation: Jeremy Cole and Davi Arnaut This patch provides startup options: * flush-caches: Flush and purge buffers/caches * numa-interleave: Run mysqld with its memory interleaved on all CPUs It also provides a config option: * innodb_buffer_pool_populate: pre-allocation of buffer pool memory at start up: -- Use MAP_POPULATE if supported (Linux 2.6.23 and higher) -- Forced pre-allocation using memset -------- 286e975 -------- Automation will pass in the right value to this option if it needs to bind mysqld to a particular socket. That will happen via systemd scripts. This is only adding the capability on the mysqld_safe side to be able to pass in numactl to the server. The turning on of this feature will need more testing and slow roll, and that will happen in different diffs on the ops repo.

facebook-github-bot

@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

hermanlee · 2019-04-15T20:25:06Z

storage/innobase/srv/srv0start.cc

@@ -1816,7 +1816,8 @@ dberr_t srv_start(bool create_new_db, const std::string &scan_directories) {
  ib::info(ER_IB_MSG_1130, size, unit, srv_buf_pool_instances, chunk_size,
           chunk_unit);

-  err = buf_pool_init(srv_buf_pool_size, srv_buf_pool_instances);
+  err = buf_pool_init(srv_buf_pool_size, srv_buf_pool_populate,


I'll fix this, but populate and instances need to be swapped.

Summary: Jira ticket: https://jira.percona.com/browse/FB8-132 Reference commit: e50c754 Reference commit: 286e975 WebScaleSQL Feature: NUMA Support Credits for research and implementation: Jeremy Cole and Davi Arnaut This patch provides startup options: * flush-caches: Flush and purge buffers/caches * numa-nodebind: Run mysqld with CPU affinity Note, the original patch provided numa-interleave, but that is supported natively by 8.0 now. It also provides a config option: * innodb_buffer_pool_populate: pre-allocation of buffer pool memory at start up: -- Use MAP_POPULATE if supported (Linux 2.6.23 and higher) -- Forced pre-allocation using memset Automation will pass in the right value to this option if it needs to bind mysqld to a particular socket. That will happen via systemd scripts. This is only adding the capability on the mysqld_safe side to be able to pass in numactl to the server. The turning on of this feature will need more testing and slow roll, and that will happen in different diffs on the ops repo. Pull Request resolved: #987 Reviewed By: lth Differential Revision: D14652411 Pulled By: lth fbshipit-source-id: c51a5a3

Summary: Jira ticket: https://jira.percona.com/browse/FB8-132 Reference commit: e50c754 Reference commit: 286e975 WebScaleSQL Feature: NUMA Support Credits for research and implementation: Jeremy Cole and Davi Arnaut This patch provides startup options: * flush-caches: Flush and purge buffers/caches * numa-nodebind: Run mysqld with CPU affinity Note, the original patch provided numa-interleave, but that is supported natively by 8.0 now. It also provides a config option: * innodb_buffer_pool_populate: pre-allocation of buffer pool memory at start up: -- Use MAP_POPULATE if supported (Linux 2.6.23 and higher) -- Forced pre-allocation using memset Automation will pass in the right value to this option if it needs to bind mysqld to a particular socket. That will happen via systemd scripts. This is only adding the capability on the mysqld_safe side to be able to pass in numactl to the server. The turning on of this feature will need more testing and slow roll, and that will happen in different diffs on the ops repo. Pull Request resolved: #987 Reviewed By: lth Differential Revision: D14652411 Pulled By: lth fbshipit-source-id: 4d8d1f9

Summary: Jira ticket: https://jira.percona.com/browse/FB8-132 Reference commit: facebook@e50c754 Reference commit: facebook@286e975 WebScaleSQL Feature: NUMA Support Credits for research and implementation: Jeremy Cole and Davi Arnaut This patch provides startup options: * flush-caches: Flush and purge buffers/caches * numa-nodebind: Run mysqld with CPU affinity Note, the original patch provided numa-interleave, but that is supported natively by 8.0 now. It also provides a config option: * innodb_buffer_pool_populate: pre-allocation of buffer pool memory at start up: -- Use MAP_POPULATE if supported (Linux 2.6.23 and higher) -- Forced pre-allocation using memset Automation will pass in the right value to this option if it needs to bind mysqld to a particular socket. That will happen via systemd scripts. This is only adding the capability on the mysqld_safe side to be able to pass in numactl to the server. The turning on of this feature will need more testing and slow roll, and that will happen in different diffs on the ops repo. Pull Request resolved: facebook#987 Reviewed By: lth Differential Revision: D14652411 Pulled By: lth fbshipit-source-id: 4d8d1f9

Summary: Jira ticket: https://jira.percona.com/browse/FB8-132 Reference commit: facebook@e50c754 Reference commit: facebook@286e975 WebScaleSQL Feature: NUMA Support Credits for research and implementation: Jeremy Cole and Davi Arnaut This patch provides startup options: * flush-caches: Flush and purge buffers/caches * numa-nodebind: Run mysqld with CPU affinity Note, the original patch provided numa-interleave, but that is supported natively by 8.0 now. It also provides a config option: * innodb_buffer_pool_populate: pre-allocation of buffer pool memory at start up: -- Use MAP_POPULATE if supported (Linux 2.6.23 and higher) -- Forced pre-allocation using memset Automation will pass in the right value to this option if it needs to bind mysqld to a particular socket. That will happen via systemd scripts. This is only adding the capability on the mysqld_safe side to be able to pass in numactl to the server. The turning on of this feature will need more testing and slow roll, and that will happen in different diffs on the ops repo. Pull Request resolved: facebook#987 Reviewed By: lloyd Differential Revision: D14652411 (facebook@b5d4a67) Pulled By: lth fbshipit-source-id: 7b1dd922612

Summary: Jira ticket: https://jira.percona.com/browse/FB8-132 Reference commit: facebook@e50c754 Reference commit: facebook@286e975 WebScaleSQL Feature: NUMA Support Credits for research and implementation: Jeremy Cole and Davi Arnaut This patch provides startup options: * flush-caches: Flush and purge buffers/caches * numa-nodebind: Run mysqld with CPU affinity Note, the original patch provided numa-interleave, but that is supported natively by 8.0 now. It also provides a config option: * innodb_buffer_pool_populate: pre-allocation of buffer pool memory at start up: -- Use MAP_POPULATE if supported (Linux 2.6.23 and higher) -- Forced pre-allocation using memset Automation will pass in the right value to this option if it needs to bind mysqld to a particular socket. That will happen via systemd scripts. This is only adding the capability on the mysqld_safe side to be able to pass in numactl to the server. The turning on of this feature will need more testing and slow roll, and that will happen in different diffs on the ops repo. Pull Request resolved: facebook#987 Reviewed By: lloyd Differential Revision: D14652411 Pulled By: lth ----------------------------------------------------------------------------- Fix InnoDB large_page_aligned_alloc function signature on macOS (facebook#1207) Summary: On Linux, it takes an extra bool parameter for populating mmap'ed memory. This is not available on macOS, so just add an unnamed argument. Pull Request resolved: facebook#1207 Reviewed By: luqun Differential Revision: D38956552 Pulled By: hermanlee fbshipit-source-id: 26b710e

Summary: Jira ticket: https://jira.percona.com/browse/FB8-132 Reference commit: facebook@e50c754 Reference commit: facebook@286e975 WebScaleSQL Feature: NUMA Support Credits for research and implementation: Jeremy Cole and Davi Arnaut This patch provides startup options: * flush-caches: Flush and purge buffers/caches * numa-nodebind: Run mysqld with CPU affinity Note, the original patch provided numa-interleave, but that is supported natively by 8.0 now. It also provides a config option: * innodb_buffer_pool_populate: pre-allocation of buffer pool memory at start up: -- Use MAP_POPULATE if supported (Linux 2.6.23 and higher) -- Forced pre-allocation using memset Automation will pass in the right value to this option if it needs to bind mysqld to a particular socket. That will happen via systemd scripts. This is only adding the capability on the mysqld_safe side to be able to pass in numactl to the server. The turning on of this feature will need more testing and slow roll, and that will happen in different diffs on the ops repo. Pull Request resolved: facebook#987 Reviewed By: lloyd Differential Revision: D14652411 Pulled By: lth ----------------------------------------------------------------------------- Fix InnoDB large_page_aligned_alloc function signature on macOS (facebook#1207) Summary: On Linux, it takes an extra bool parameter for populating mmap'ed memory. This is not available on macOS, so just add an unnamed argument. Pull Request resolved: facebook#1207 Reviewed By: luqun Differential Revision: D38956552 Pulled By: hermanlee

Summary: Jira ticket: https://jira.percona.com/browse/FB8-132 Reference commit: facebook/mysql-5.6@e50c754 Reference commit: facebook/mysql-5.6@286e975 WebScaleSQL Feature: NUMA Support Credits for research and implementation: Jeremy Cole and Davi Arnaut This patch provides startup options: * flush-caches: Flush and purge buffers/caches * numa-nodebind: Run mysqld with CPU affinity Note, the original patch provided numa-interleave, but that is supported natively by 8.0 now. It also provides a config option: * innodb_buffer_pool_populate: pre-allocation of buffer pool memory at start up: -- Use MAP_POPULATE if supported (Linux 2.6.23 and higher) -- Forced pre-allocation using memset Automation will pass in the right value to this option if it needs to bind mysqld to a particular socket. That will happen via systemd scripts. This is only adding the capability on the mysqld_safe side to be able to pass in numactl to the server. The turning on of this feature will need more testing and slow roll, and that will happen in different diffs on the ops repo. Pull Request resolved: facebook/mysql-5.6#987 Reviewed By: lloyd Differential Revision: D14652411 Pulled By: lth ----------------------------------------------------------------------------- Fix InnoDB large_page_aligned_alloc function signature on macOS (percona#1207) Summary: On Linux, it takes an extra bool parameter for populating mmap'ed memory. This is not available on macOS, so just add an unnamed argument. Pull Request resolved: facebook/mysql-5.6#1207 Reviewed By: luqun Differential Revision: D38956552 Pulled By: hermanlee

Summary: Jira ticket: https://jira.percona.com/browse/FB8-132 Reference commit: facebook@e50c754 Reference commit: facebook@286e975 WebScaleSQL Feature: NUMA Support Credits for research and implementation: Jeremy Cole and Davi Arnaut This patch provides startup options: * flush-caches: Flush and purge buffers/caches * numa-nodebind: Run mysqld with CPU affinity Note, the original patch provided numa-interleave, but that is supported natively by 8.0 now. It also provides a config option: * innodb_buffer_pool_populate: pre-allocation of buffer pool memory at start up: -- Use MAP_POPULATE if supported (Linux 2.6.23 and higher) -- Forced pre-allocation using memset Automation will pass in the right value to this option if it needs to bind mysqld to a particular socket. That will happen via systemd scripts. This is only adding the capability on the mysqld_safe side to be able to pass in numactl to the server. The turning on of this feature will need more testing and slow roll, and that will happen in different diffs on the ops repo. Pull Request resolved: facebook#987 Reviewed By: lloyd Differential Revision: D14652411 Pulled By: lth ----------------------------------------------------------------------------- Fix InnoDB large_page_aligned_alloc function signature on macOS (facebook#1207) Summary: On Linux, it takes an extra bool parameter for populating mmap'ed memory. This is not available on macOS, so just add an unnamed argument. Pull Request resolved: facebook#1207 Reviewed By: luqun Differential Revision: D38956552 Pulled By: hermanlee

facebook-github-bot added the CLA Signed label Mar 18, 2019

percona-ysorokin approved these changes Mar 18, 2019

View reviewed changes

dutow changed the title ~~WIP FB8-132: NUMA support~~ FB8-132: NUMA support Mar 18, 2019

laurynas-biveinis suggested changes Mar 19, 2019

View reviewed changes

hermanlee force-pushed the fb-mysql-8.0.13 branch from e9926c3 to c3bd1c6 Compare March 21, 2019 21:53

facebook-github-bot reviewed Mar 27, 2019

View reviewed changes

hermanlee approved these changes Apr 15, 2019

View reviewed changes

hermanlee closed this May 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FB8-132: NUMA support #987

FB8-132: NUMA support #987

dutow commented Mar 18, 2019

percona-ysorokin left a comment

dutow commented Mar 18, 2019

laurynas-biveinis Mar 19, 2019

laurynas-biveinis Mar 19, 2019

laurynas-biveinis Mar 19, 2019

laurynas-biveinis Mar 19, 2019

laurynas-biveinis Mar 19, 2019

laurynas-biveinis Mar 19, 2019

hermanlee commented Mar 22, 2019

facebook-github-bot left a comment

hermanlee Apr 15, 2019

	@return TRUE if the OS release is equal to, or later than release. */
	@return true if the OS release is equal to, or later than release. */

FB8-132: NUMA support #987

FB8-132: NUMA support #987

Conversation

dutow commented Mar 18, 2019

percona-ysorokin left a comment

Choose a reason for hiding this comment

dutow commented Mar 18, 2019

laurynas-biveinis Mar 19, 2019

Choose a reason for hiding this comment

laurynas-biveinis Mar 19, 2019

Choose a reason for hiding this comment

laurynas-biveinis Mar 19, 2019

Choose a reason for hiding this comment

laurynas-biveinis Mar 19, 2019

Choose a reason for hiding this comment

laurynas-biveinis Mar 19, 2019

Choose a reason for hiding this comment

laurynas-biveinis Mar 19, 2019

Choose a reason for hiding this comment

hermanlee commented Mar 22, 2019

facebook-github-bot left a comment

Choose a reason for hiding this comment

hermanlee Apr 15, 2019

Choose a reason for hiding this comment