Skip to content

Commit

Permalink
MDEV-31953 madvise(..., MADV_FREE) is causing a performance regression
Browse files Browse the repository at this point in the history
buf_page_t::set_os_unused(): Remove the system call that had been added in
commit 16c9718 and revised in
commit c1fd082 for Microsoft Windows.

buf_pool_t::garbage_collect(): A new function to collect any garbage
from the InnoDB buffer pool that can be removed without writing any
log or data files. This will also invoke madvise() for all of buf_pool.free.

To trigger this the following MDEV is implemented:
MDEV-24670 avoid OOM by linux kernel co-operative memory management

To avoid frequent triggers that caused the MDEV-31953 regression, while
still preserving the 10.11 functionality of non-greedy kernel memory
usage, memory triggers are used.

On the triggering of memory pressure, if supported in the Linux kernel,
trigger the garbage collection of the innodb buffer pool.

The hard coded triggers occur where there is:
* some memory pressure in 5 of the last 10 seconds
* a full stall on memory pressure for 10ms in the last 2 seconds

The kernel will trigger only one in each of these time windows. To avoid
mariadb being in a constant state of memory garbage collection, this has
been limited to once per minute.

For a small set of kernels in 2023 (6.5, 6.6), there was a limit requiring
CAP_SYS_RESOURCE that was lifted[1] to support the use case of user
memory pressure. It not currently possible to set CAP_SYS_RESOURCES in
a systemd service as its setting a capability inside a usernamespace.

Running under systemd v254+ requires the default MemoryPressureWatch=auto
(or alternately "on").

Functionality was tested in a 6.4 kernel Fedora successfully under a
systemd service.

Running in a container requires that (unmask=)/sys/fs/cgroup be writable
by the mariadbd process.

To aid testing, the buf_pool_resize was a convient trigger point on
which to trigger garbage collection.

ref [1]: https://lore.kernel.org/all/CAMw=ZnQ56cm4Txgy5EhGYvR+Jt4s-KVgoA9_65HKWVMOXp7a9A@mail.gmail.com/T/#m3bd2a73c5ee49965cb73a830b1ccaa37ccf4e427

Co-Author: Daniel Black (on memory pressure trigger)

Reviewed by: Marko Mäkelä, Vladislav Vaintroub, Vladislav Lesin,
   Thirunarayanan Balathandayuthapani

Tested by: Matthias Leich
  • Loading branch information
dr-m authored and grooverdan committed Nov 6, 2023
1 parent 818a9f3 commit 90c1e91
Show file tree
Hide file tree
Showing 4 changed files with 372 additions and 6 deletions.
19 changes: 19 additions & 0 deletions mysql-test/suite/innodb/r/mem_pressure.result
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#
# MDEV-24670 avoid OOM by linux kernel co-operative memory management
#
set @save_dbug=@@debug_dbug;
CREATE TABLE t1 (t TEXT) ENGINE=InnoDB;
INSERT INTO t1 SELECT CONCAT(REPEAT('junk text', 500), CAST(seq AS CHAR)) FROM seq_1_to_1000;
SELECT CAST(VARIABLE_VALUE AS INTEGER) INTO @dirty_prev FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='Innodb_buffer_pool_pages_dirty';
SELECT CAST(VARIABLE_VALUE AS INTEGER) INTO @flush_prev FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='Innodb_buffer_pool_pages_flushed';
set debug_dbug="d,trigger_garbage_collection";
SET GLOBAL innodb_buffer_pool_size=@@innodb_buffer_pool_size;
SELECT VARIABLE_VALUE < @dirty_prev AS LESS_DIRTY_IS_GOOD FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='Innodb_buffer_pool_pages_dirty';
LESS_DIRTY_IS_GOOD
1
DROP TABLE t1;
FOUND 1 /InnoDB: Memory pressure event freed/ in mysqld.1.err
set debug_dbug=@save_dbug;
#
# End of 10.11 tests
#
33 changes: 33 additions & 0 deletions mysql-test/suite/innodb/t/mem_pressure.test
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
--source include/have_debug.inc
--source include/linux.inc
--source include/not_embedded.inc
--source include/have_innodb.inc
--source include/have_sequence.inc

--echo #
--echo # MDEV-24670 avoid OOM by linux kernel co-operative memory management
--echo #

set @save_dbug=@@debug_dbug;

CREATE TABLE t1 (t TEXT) ENGINE=InnoDB;
INSERT INTO t1 SELECT CONCAT(REPEAT('junk text', 500), CAST(seq AS CHAR)) FROM seq_1_to_1000;

SELECT CAST(VARIABLE_VALUE AS INTEGER) INTO @dirty_prev FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='Innodb_buffer_pool_pages_dirty';
SELECT CAST(VARIABLE_VALUE AS INTEGER) INTO @flush_prev FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='Innodb_buffer_pool_pages_flushed';
set debug_dbug="d,trigger_garbage_collection";
SET GLOBAL innodb_buffer_pool_size=@@innodb_buffer_pool_size;

SELECT VARIABLE_VALUE < @dirty_prev AS LESS_DIRTY_IS_GOOD FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='Innodb_buffer_pool_pages_dirty';

DROP TABLE t1;

let SEARCH_FILE= $MYSQLTEST_VARDIR/log/mysqld.1.err;
let SEARCH_PATTERN= InnoDB: Memory pressure event freed;
--source include/search_pattern_in_file.inc

set debug_dbug=@save_dbug;

--echo #
--echo # End of 10.11 tests
--echo #
Loading

0 comments on commit 90c1e91

Please sign in to comment.