FB8-59: Database admission control #969
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
satya-bodapati
changed the title
laurynas-biveinis
suggested changes
Feb 25, 2019
percona-ysorokin
suggested changes
Feb 25, 2019
satya-bodapati
force-pushed
the
fb8-59
branch
2 times, most recently
from
55f2ad6 to
5258dc9
Compare
|
Ready for internal review again @percona-ysorokin @laurynas-biveinis. Testcase improvements will be done later. |
|
Testcase comments addressed too. Ready for internal review @laurynas-biveinis @percona-ysorokin |
satya-bodapati
changed the title
|
Thanks @percona-ysorokin Ready for FB review |
There was a problem hiding this comment.
@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
hermanlee
reviewed
Mar 22, 2019
| @@ -18724,6 +18724,9 @@ ER_BINLOG_CANT_READ_INDEX | |||
| ER_RPL_FAILED_IN_RLI_INIT_INFO | |||
| eng "Failed in %s called from Relay_log_info::rli_init_info()." | |||
|
|
|||
| ER_DB_ADMISSION_CONTROL | |||
There was a problem hiding this comment.
This should replace ER_PLACEHOLDER_50028. In the original 5.6 code, it came after ER_DB_READ_ONLY.
hermanlee
requested changes
Mar 22, 2019
| case SQLCOM_CREATE_INDEX: | ||
| case SQLCOM_CREATE_DB: | ||
| case SQLCOM_CREATE_FUNCTION: | ||
| // case SQLCOM_CREATE_NPROCEDURE: |
There was a problem hiding this comment.
The only one we'd likely add back is SQLCOM_SHOW_CONNECTION_ATTRIBUTES. But the rest will be dropped.
| // Currently connection and query limits are two resource types. More will be | ||
| // supported in the future. | ||
| enum class enum_multi_tenancy_resource_type : int32_t { | ||
| MULTI_TENANCY_RESOURCE_CONNECTION, |
There was a problem hiding this comment.
Would it be too onerous to remove the references to the multi_tenancy_* enums and use most of the original code which did not depend on these enums? Or do the subsequent fixes added after the refactoring into sql_multi_tenancy.cc make this difficult?
There was a problem hiding this comment.
I delayed the removals, renaming (and refactoring) because I want this patch to be tested for correctness first. Once it is good, all these things can be handled with new proposed JIRA that moves this entire code to a audit plugin type.
How did the internal testing go? Any issues found?
hermanlee
requested changes
Apr 2, 2019
There was a problem hiding this comment.
Looks like the following tests need to be fixed or re-recorded:
sys_vars.all_vars main.mysqld--help-notwin main.admission_control_multi_query perfschema.dml_setup_instruments main.information_schema_cs
Should mysql_rm_db() call db_ac->remove(db) somewhere, or has it been intentionally removed?
The unit tests, other than the ones listed, look to be passing.
I'm going to setup some more comprehensive testing with this patch.
| friend class AC; | ||
| #ifdef HAVE_PSI_INTERFACE | ||
| PSI_mutex_key key_lock; | ||
| PSI_mutex_info key_lock_info[1] = {{&key_lock, "Ac_info::lock", 0}}; |
There was a problem hiding this comment.
compiler error: need to add PSI_FLAG_SINGLETON and PSI_DOCUMENT_ME.
There was a problem hiding this comment.
How did you get this error? I cannot see this.
Some part I actually converted to new interface but old was working fine too. Let me know the steps to get compiler error
There was a problem hiding this comment.
Here's the error:
In file included from /home/herman/rocks-mysql/8.0/sql/sql_admission_control.cc:17:
/home/herman/rocks-mysql/8.0/sql/sql_admission_control.h:182:37: error: missing field 'm_volatility' initializer [-Werror,-Wmissing-field-initializers]
{&key_lock, "Ac_info::lock", 0}};
So missing-field-initializers?
| #ifdef HAVE_PSI_INTERFACE | ||
| PSI_rwlock_key key_rwlock_LOCK_ac; | ||
| PSI_rwlock_info key_rwlock_LOCK_ac_info[1] = { | ||
| {&key_rwlock_LOCK_ac, "AC::rwlock", 0}}; |
There was a problem hiding this comment.
compiler error: need to add PSI_FLAG_SINGLETON and PSI_DOCUMENT_ME.
| &old_stage); | ||
| // Spurious wake-ups are rare and fine in this design. | ||
| mysql_cond_wait(&ac_node->cond, &ac_node->lock); | ||
| mysql_mutex_unlock(&ac_node->lock); |
There was a problem hiding this comment.
Hm.. the original code doesn't have this unlock call. Seems like the feature would be broken without it! I'm not sure where the unlock() for the waiting case.
There was a problem hiding this comment.
Ah, okay, 5.6 has exit_cond unlock the mutex, but this is different in 8.0.
There was a problem hiding this comment.
Right! spot on. In 8.0, it is different.
what is the failure with main.admission_control_multi_query test? Locally it passes for me.
Good catch. Don't know how it got missed.
Don't know the "list" you are referring here. Hopefully nothing related to admission control are failing.
Please do and let me know the results. |
JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Summary: Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Test Plan: Added mtr tests. Ran performance tests. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set @@global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Summary: Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. Test Plan: mtr Original Reviewers: jkedgar Originally Reviewed By: jkedgar ----- facebook@cf7ce48 ------ Summary: In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set @@global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. Test Plan: mtr. The test timeouts because of mysqld hang without this diff. Original Reviewers: jkedgar, pengt Originally Reviewed By: pengt ---- facebook@bcbd77a ------ Summary: Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. Test Plan: mtr Original Reviewers: tianx, jkedgar Originally Reviewed By: jkedgar ----- facebook@928c2a5 -------- Get accurate status variables Summary: Hold gobal admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures Test Plan: mtr Original Reviewers: tianx Originally Reviewed By: tianx ------- facebook@86587af ----------- Summary: Added SHOW command filter in admission control. Test Plan: mtr Original Reviewers: santoshb, jkedgar Originally Reviewed By: jkedgar ------- facebook@7870828 ------ Change the ac_map to be a map of shared_ptr instead of unique_ptr Summary: The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. Test Plan: MTR Original Reviewers: santoshb Oringally Reviewed By: santoshb ---- facebook@75de821 ------ Summary: Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. Originally Reviewed By: lth --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx Summary: - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Oringally Reviewed By: jkedgar
|
@satya-bodapati has updated the pull request. Re-import the pull request |
|
The main.admission_control_multi_query test problem was due to an internal configuration difference. I'll update the test locally. The list of tests was referrring to: "sys_vars.all_vars main.mysqld--help-notwin main.admission_control_multi_query perfschema.dml_setup_instruments main.information_schema_cs" The testing is still on my TODO. I'm hoping to get to it soon. |
There was a problem hiding this comment.
@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
hermanlee
approved these changes
May 8, 2019
facebook-github-bot
pushed a commit
that referenced
this pull request
May 28, 2019
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: 5cbfab0 Reference patch: 0ebaf75 Reference patch: cf7ce48 Reference patch: 3842a8f Reference patch: bcbd77a Reference patch: e6c2b87 Reference patch: 928c2a5 Reference patch: 86587af Reference patch: 7870828 Reference patch: 0151b45 Reference patch: 75de821 Reference patch: 08d4174 Reference patch: c1e5b4f Reference patch: 08368d4 Reference patch: a2198e0 Reference patch: 0730342 Reference patch: d868c1b Reference patch: 645fc5e <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- 5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set @global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- 0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set @global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- 928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- 86587af ----------- Added SHOW command filter in admission control. ------- 7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- 75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: #969 Reviewed By: lth Differential Revision: D14567747 Pulled By: lth fbshipit-source-id: 8cb9fa0
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Jun 19, 2023
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Jun 23, 2023
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
hermanlee
pushed a commit
to hermanlee/mysql-5.6
that referenced
this pull request
Oct 3, 2023
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
hermanlee
pushed a commit
to hermanlee/mysql-5.6
that referenced
this pull request
Oct 18, 2023
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Apr 23, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Apr 23, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Apr 25, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
May 7, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
May 8, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
May 9, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
May 10, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
May 13, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
May 15, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
May 16, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
May 17, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
May 17, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
May 21, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
May 21, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
May 30, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Jun 14, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Jun 19, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Jun 20, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Jun 21, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Jun 25, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Jul 2, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Jul 19, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Jul 19, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Jul 31, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Aug 2, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
inikep
pushed a commit
to inikep/mysql-5.6
that referenced
this pull request
Aug 6, 2024
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 Pulled By: lth
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA: https://jira.percona.com/browse/FB8-59
Reference patch: 5cbfab0
Reference patch: 0ebaf75
Reference patch: cf7ce48
Reference patch: 3842a8f
Reference patch: bcbd77a
Reference patch: e6c2b87
Reference patch: 928c2a5
Reference patch: 86587af
Reference patch: 7870828
Reference patch: 0151b45
Reference patch: 75de821
Reference patch: 08d4174
Reference patch: c1e5b4f
Reference patch: 08368d4
Reference patch: a2198e0
Reference patch: 0730342
Reference patch: d868c1b
Reference patch: 645fc5e
This patch ports Database admission control. The multi-tenancy plugin
changes are not ported. This code will be eventually moved to new plugin
of audit type.
There are possible improvements with code. Below are the deferred items
(makes it easier for review. Can compare with 5.6 code)
thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is
verified in 5.6 and it works well. (5.6 equivalent is:
thd->transaction.all.ha_list == NULL;)
"!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of
thd->is_real_trans.
The above check seems sufficient but after porting thd->is_real_trans, it should be
possible to replace the above check with thd->is_real_trans
------- 5cbfab0 --------
Add two global sys vars max_running_queries and max_waiting_queries.
They control maximum number of running queries on a database and maximum
waiting queries when max_running_queries limit is crossed on that database.
A value of 0 implies no limits are applied for the queries. If max_waiting_queries
limit is crossed, then new queries will simply fail.
For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'.
Only dagtabase set at the session level is considered in these checks. Admission checks
are by-passed in the following cases
Performance testing
Set innodb_flush_log_at_trx_commit=2.
mysql> set @@global.innodb_flush_log_at_trx_commit=2
Check rows inserted using
== Innodb_rows_inserted with concurrency=100 ==
38106 (max_running_queries=0, max_waiting_queries=0)
38733 (max_running_queries=1000, max_waiting_queries=0)
38365 (max_running_queries=100, max_waiting_queries=0)
40308 (max_running_queries=70, max_waiting_queries=0)
43790 (max_running_queries=50, max_waiting_queries=0)
47321 (max_running_queries=20, max_waiting_queries=0)
49473 (max_running_queries=10, max_waiting_queries=0)
54851 (max_running_queries=5, max_waiting_queries=0)
== Innodb_rows_inserted with concurrency=100 ==
26516 (max_running_queries=0, max_waiting_queries=0)
44494 (max_running_queries=10, max_waiting_queries=0)
51193 (max_running_queries=5, max_waiting_queries=0)
The above numbers show the improvement in quality of service.
== Innodb_rows_inserted with concurrency=20 ==
47228 (max_running_queries=0, max_waiting_queries=0)
47107 (max_running_queries=100, max_waiting_queries=0)
47082 (max_running_queries=20, max_waiting_queries=0)
47515 (max_running_queries=10, max_waiting_queries=0)
There isn't much difference under low concurrency workloads.
There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value.
Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar
----- 0ebaf75 -----
Added the following status variables:
----- cf7ce48 ------
In order to avoid possible mysqld hangs, we need an option to blacklist
specific commands in admission control. Example usage is
set @@global.admission_control_filter='BEGIN,COMMIT,SET';
An empty value for admission_control_filter implies no commands are blacklisted
in the admission control.
---- bcbd77a ------
Initial version of admission control applied limits on each query statement in multi
query packet. The behavior is modified to the following:
A thread enters admission control when executing the first non-filtered command
in the multi query packet and exits after executing all the query statements in
the multi query packet.
This diff also fixes a bug where admission control limits are not initialized during
mysqld startup.
----- 928c2a5 --------
Get accurate status variables
Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were
not accurate causing test failures
------- 86587af -----------
Added SHOW command filter in admission control.
------- 7870828 ------
The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically.
---- 75de821 ------
Fixing the following two issues in admission control when changing database is
in a multi-statement query:
number of multi-query packet to run, even the packet starts with
use [db_name].the limit of the default database, even if we change database in the
multi-query packet.
This diff fixes the above issues in the following way:
usecommand isexecuted. So in the multi-statement query, the subsequent sub-queries are not
bypassed.
thd->dbdirectly instead of attribute map, sothe attribute map doesn't need to store the session database.
USE [db_name]statement is added toadmission_control_filter.In addition, I also changed the multitenancy plugin interface to use a wrapper
object to pass down the connection_attrs_map, query_attrs_map, and session
database info. Attribute maps will not be modified.
--------- c1e5b4f --------
Fix resource leaks by admission_control_by_trx