From 68bb35b1a5b997cd099adc352047ce298ba33dfe Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Tue, 3 Aug 2021 15:17:23 +0800 Subject: [PATCH 1/8] https://github.com/pingcap/docs-cn/pull/6695 --- statement-summary-tables.md | 74 +++++++++++++++++++++++++++++++++---- 1 file changed, 67 insertions(+), 7 deletions(-) diff --git a/statement-summary-tables.md b/statement-summary-tables.md index 740869a12425f..494b55393100b 100644 --- a/statement-summary-tables.md +++ b/statement-summary-tables.md @@ -89,11 +89,17 @@ The table schema of `statements_summary_history` is identical to that of `statem The fields `SUMMARY_BEGIN_TIME` and `SUMMARY_END_TIME` represent the start time and the end time of the historical time range. -## `cluster_statements_summary` and `cluster_statements_summary_history` +## `statements_summary_evicted` -`statements_summary` and `statements_summary_history` display the statement summary data of only a single TiDB server. To query the data of the entire cluster, you need to query `cluster_statements_summary` and `cluster_statements_summary_history`. +The `tidb_stmt_summary_max_stmt_count` variable controls the maximum number of statements that the `statement summary` table stores in memory. Using the LRU algorithm, once the types of SQL statements exceed the `tidb_stmt_summary_max_stmt_count` value, the longest missed record is evicted from the table. The types and number of evicted SQL statements during each period are recorded in the `statements_summary_evicted` table. -`cluster_statements_summary` displays the `statements_summary` data of each TiDB server, and `cluster_statements_summary_history` displays the `statements_summary_history` data of each TiDB server. These two tables use the `INSTANCE` field to represent the address of the TiDB server. The other fields are the same as those in `statements_summary`. +The `statements_summary_evicted` table is updated only when a SQL record is evicted from the `statement summary` table. The `statements_summary_evicted` only records the period during which the eviction occurs. + +## The `cluster` tables for statement summary + +The `statements_summary`, `statements_summary_history`, and `statements_summary_evicted` tables only show the statement summary of a single TiDB server. To query the data of the entire cluster, you need to query the `cluster_statements_summary`, `cluster_statements_summary_history`, or `cluster_statements_summary_evicted` tables. + +`cluster_statements_summary` displays the `statements_summary` data of each TiDB server. `cluster_statements_summary_history` displays the `statements_summary_history` data of each TiDB server. `cluster_statements_summary_evicted` displays the `statements_summary_evicted` data of each TiDB server. These tables use the `INSTANCE` field to represent the address of the TiDB server. The other fields are the same as those in `statements_summary`. ## Parameter configuration @@ -101,14 +107,14 @@ The following system variables are used to control the statement summary: - `tidb_enable_stmt_summary`: Determines whether to enable the statement summary feature. `1` represents `enable`, and `0` means `disable`. The feature is enabled by default. The statistics in the system table are cleared if this feature is disabled. The statistics are re-calculated next time this feature is enabled. Tests have shown that enabling this feature has little impact on performance. - `tidb_stmt_summary_refresh_interval`: The interval at which the `statements_summary` table is refreshed. The time unit is second (s). The default value is `1800`. -- `tidb_stmt_summary_history_size`: The size of each SQL statement category stored in the `statements_summary_history` table. The default value is `24`. -- `tidb_stmt_summary_max_stmt_count`: Limits the number of SQL statements that can be stored in statement summary tables. The default value is `3000`. If the limit is exceeded, those SQL statements that recently remain unused are cleared. +- `tidb_stmt_summary_history_size`: The size of each SQL statement category stored in the `statements_summary_history` table, which is also the maximum number of records in the `statement_summary_evicted` table. The default value is `24`. +- `tidb_stmt_summary_max_stmt_count`: Limits the number of SQL statements that can be stored in statement summary tables. The default value is `3000`. If the limit is exceeded, those SQL statements that recently remain unused are cleared. These cleared SQL statements are recorded in the `statement_summary_evicted` table. - `tidb_stmt_summary_max_sql_length`: Specifies the longest display length of `DIGEST_TEXT` and `QUERY_SAMPLE_TEXT`. The default value is `4096`. - `tidb_stmt_summary_internal_query`: Determines whether to count the TiDB SQL statements. `1` means to count, and `0` means not to count. The default value is `0`. > **Note:** > -> When a category of SQL statement needs to be removed because the `tidb_stmt_summary_max_stmt_count` limit is exceeded, TiDB removes the data of that SQL statement category of all time ranges from the `statement summary history` table. Therefore, even if the number of SQL statement categories in a certain time range does not reach the limit, the number of SQL statements stored in the `statement summary history` table is less than the actual number of SQL statements. If this situation occurs, you are recommended to increase the value of `tidb_stmt_summary_max_stmt_count`. +> When a category of SQL statement needs to be removed because the `tidb_stmt_summary_max_stmt_count` limit is exceeded, TiDB removes the data of that SQL statement category of all time ranges from the `statement summary history` table. Therefore, even if the number of SQL statement categories in a certain time range does not reach the limit, the number of SQL statements stored in the `statement summary history` table is less than the actual number of SQL statements. If this situation occurs and affects performance, you are recommended to increase the value of `tidb_stmt_summary_max_stmt_count`. An example of the statement summary configuration is shown as follows: @@ -122,6 +128,8 @@ set global tidb_stmt_summary_history_size = 24; After the configuration above takes effect, every 30 minutes the `statements_summary` table is cleared. The `statements_summary_history` table stores data generated over the recent 12 hours. +The `statements_summary_evicted` table records the recent 24 periods during which SQL statements are evicted from the statement summary. The `statements_summary_evicted` table is updated every 30 minutes. + The system variables above have two scopes: global and session. These scopes work differently from other system variables: - After setting the global variable, your setting applies to the whole cluster immediately. @@ -133,6 +141,50 @@ The system variables above have two scopes: global and session. These scopes wor > > The `tidb_stmt_summary_history_size`, `tidb_stmt_summary_max_stmt_count`, and `tidb_stmt_summary_max_sql_length` configuration items affect memory usage. It is recommended that you adjust these configurations based on your needs. It is not recommended to set them too large values. +### Set a proper size for statement summary + +After the system has run for a period of time, you can check the `statement summary` table to see whether SQL eviction has occurred. For example: + +```sql +select @@global.tidb_stmt_summary_max_stmt_count; +select count(*) from information_schema.statements_summary; +``` + +```sql ++-------------------------------------------+ +| @@global.tidb_stmt_summary_max_stmt_count | ++-------------------------------------------+ +| 3000 | ++-------------------------------------------+ +1 row in set (0.001 sec) + ++----------+ +| count(*) | ++----------+ +| 3001 | ++----------+ +1 row in set (0.001 sec) +``` + +You can see that the `statement summary` table is full of records. Then check the evicted data from the `statements_summary_evicted` table: + +```sql +select * from information_schema.statements_summary_evicted; +``` + +```sql ++---------------------+---------------------+---------------+ +| BEGIN_TIME | END_TIME | EVICTED_COUNT | ++---------------------+---------------------+---------------+ +| 2020-01-02 16:30:00 | 2020-01-02 17:00:00 | 59 | ++---------------------+---------------------+---------------+ +| 2020-01-02 16:00:00 | 2020-01-02 16:30:00 | 45 | ++---------------------+---------------------+---------------+ +2 row in set (0.001 sec) +``` + +From the result above, you can see that a maximum of 59 SQL categories are evicted, which indicates that the proper size of the statement summary is 59 records. + ## Limitation The statement summary tables have the following limitation: @@ -197,7 +249,9 @@ The result shows that the following three categories of SQL statements consume t 3 rows in set (0.00 sec) ``` -### Fields description +## Fields description + +### `statements_summary` fields description The following are descriptions of fields in the `statements_summary` table. @@ -285,3 +339,9 @@ Transaction-related fields: - `BACKOFF_TYPES`: All types of errors that require retries and the number of retries for each type. The format of the field is `type:number`. If there is more than one error type, each is separated by a comma, like `txnLock:2,pdRPC:1`. - `AVG_AFFECTED_ROWS`: The average number of rows affected. - `PREV_SAMPLE_TEXT`: When the current SQL statement is `COMMIT`, `PREV_SAMPLE_TEXT` is the previous statement to `COMMIT`. In this case, SQL statements are grouped by the digest and `prev_sample_text`. This means that `COMMIT` statements with different `prev_sample_text` are grouped to different rows. When the current SQL statement is not `COMMIT`, the `PREV_SAMPLE_TEXT` field is an empty string. + +### `statements_summary_evicted` fields description + +- `BEGIN_TIME`: Records the starting time. +- `END_TIME`: Records the ending time. +- `EVICTED_COUNT`: The number of SQL categories that are evicted during the record period. From 2cc96f45428307f11fb4279b7c1fe5f65915eaa2 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Tue, 3 Aug 2021 15:21:41 +0800 Subject: [PATCH 2/8] Update statement-summary-tables.md --- statement-summary-tables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/statement-summary-tables.md b/statement-summary-tables.md index 494b55393100b..49bd7270fa8b5 100644 --- a/statement-summary-tables.md +++ b/statement-summary-tables.md @@ -91,7 +91,7 @@ The fields `SUMMARY_BEGIN_TIME` and `SUMMARY_END_TIME` represent the start time ## `statements_summary_evicted` -The `tidb_stmt_summary_max_stmt_count` variable controls the maximum number of statements that the `statement summary` table stores in memory. Using the LRU algorithm, once the types of SQL statements exceed the `tidb_stmt_summary_max_stmt_count` value, the longest missed record is evicted from the table. The types and number of evicted SQL statements during each period are recorded in the `statements_summary_evicted` table. +The `tidb_stmt_summary_max_stmt_count` variable controls the maximum number of statements that the `statement summary` table stores in memory. The `statement summary` table uses the LRU algorithm. Once the types of SQL statements exceed the `tidb_stmt_summary_max_stmt_count` value, the longest unused record is evicted from the table. The types and number of evicted SQL statements during each period are recorded in the `statements_summary_evicted` table. The `statements_summary_evicted` table is updated only when a SQL record is evicted from the `statement summary` table. The `statements_summary_evicted` only records the period during which the eviction occurs. From ee8699d62ba0c8a6c6c2caf3c9d831ca53e3d92f Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Tue, 3 Aug 2021 15:29:14 +0800 Subject: [PATCH 3/8] Update statement-summary-tables.md --- statement-summary-tables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/statement-summary-tables.md b/statement-summary-tables.md index 49bd7270fa8b5..1698feff79b4e 100644 --- a/statement-summary-tables.md +++ b/statement-summary-tables.md @@ -13,7 +13,7 @@ Therefore, starting from v4.0.0-rc.1, TiDB provides system tables in `informatio - [`statements_summary`](#statements_summary) - [`statements_summary_history`](#statements_summary_history) - [`cluster_statements_summary`](#cluster_statements_summary-and-cluster_statements_summary_history) -- [`cluster_statements_summary_history`](#cluster_statements_summary-and-cluster_statements_summary_history) +- [`statements_summary_evicted`](#statements_summary_evicted) This document details these tables and introduces how to use them to troubleshoot SQL performance issues. From ebba8e063db4cbfccea8835df1da13631f512225 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Tue, 3 Aug 2021 15:46:27 +0800 Subject: [PATCH 4/8] Apply suggestions from code review --- statement-summary-tables.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/statement-summary-tables.md b/statement-summary-tables.md index 1698feff79b4e..6adf48af9ccb5 100644 --- a/statement-summary-tables.md +++ b/statement-summary-tables.md @@ -91,9 +91,9 @@ The fields `SUMMARY_BEGIN_TIME` and `SUMMARY_END_TIME` represent the start time ## `statements_summary_evicted` -The `tidb_stmt_summary_max_stmt_count` variable controls the maximum number of statements that the `statement summary` table stores in memory. The `statement summary` table uses the LRU algorithm. Once the types of SQL statements exceed the `tidb_stmt_summary_max_stmt_count` value, the longest unused record is evicted from the table. The types and number of evicted SQL statements during each period are recorded in the `statements_summary_evicted` table. +The `tidb_stmt_summary_max_stmt_count` variable controls the maximum number of statements that the `statement summary` table stores in memory. The `statement summary` table uses the LRU algorithm. Once the types of SQL statements exceed the `tidb_stmt_summary_max_stmt_count` value, the longest unused record is evicted from the table. The number of evicted SQL statements during each period are recorded in the `statements_summary_evicted` table. -The `statements_summary_evicted` table is updated only when a SQL record is evicted from the `statement summary` table. The `statements_summary_evicted` only records the period during which the eviction occurs. +The `statements_summary_evicted` table is updated only when a SQL record is evicted from the `statement summary` table. The `statements_summary_evicted` only records the period during which the eviction occurs and the number of evicted SQL statements. ## The `cluster` tables for statement summary From b7506dc54ec93da0704057cee0a2b411acbd22f9 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Tue, 3 Aug 2021 16:18:42 +0800 Subject: [PATCH 5/8] Update statement-summary-tables.md Co-authored-by: Liuxiaozhen12 <82579298+Liuxiaozhen12@users.noreply.github.com> --- statement-summary-tables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/statement-summary-tables.md b/statement-summary-tables.md index 6adf48af9ccb5..a790b5762acb7 100644 --- a/statement-summary-tables.md +++ b/statement-summary-tables.md @@ -91,7 +91,7 @@ The fields `SUMMARY_BEGIN_TIME` and `SUMMARY_END_TIME` represent the start time ## `statements_summary_evicted` -The `tidb_stmt_summary_max_stmt_count` variable controls the maximum number of statements that the `statement summary` table stores in memory. The `statement summary` table uses the LRU algorithm. Once the types of SQL statements exceed the `tidb_stmt_summary_max_stmt_count` value, the longest unused record is evicted from the table. The number of evicted SQL statements during each period are recorded in the `statements_summary_evicted` table. +The `tidb_stmt_summary_max_stmt_count` variable controls the maximum number of statements that the `statement summary` table stores in memory. The `statement summary` table uses the LRU algorithm. Once the number of SQL statements exceed the `tidb_stmt_summary_max_stmt_count` value, the longest unused record is evicted from the table. The number of evicted SQL statements during each period are recorded in the `statements_summary_evicted` table. The `statements_summary_evicted` table is updated only when a SQL record is evicted from the `statement summary` table. The `statements_summary_evicted` only records the period during which the eviction occurs and the number of evicted SQL statements. From b9da8cd8666f68522c6377d124f4a7339cd8f807 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 4 Aug 2021 16:04:25 +0800 Subject: [PATCH 6/8] Update statement-summary-tables.md --- statement-summary-tables.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/statement-summary-tables.md b/statement-summary-tables.md index a790b5762acb7..8abb94e84cae6 100644 --- a/statement-summary-tables.md +++ b/statement-summary-tables.md @@ -12,7 +12,9 @@ Therefore, starting from v4.0.0-rc.1, TiDB provides system tables in `informatio - [`statements_summary`](#statements_summary) - [`statements_summary_history`](#statements_summary_history) -- [`cluster_statements_summary`](#cluster_statements_summary-and-cluster_statements_summary_history) +- [`cluster_statements_summary`](#statements_summary_evicted) +- [`cluster_statements_summary_history`](#statements_summary_evicted) +- [`statements_summary_evicted`](#statements_summary_evicted) - [`statements_summary_evicted`](#statements_summary_evicted) This document details these tables and introduces how to use them to troubleshoot SQL performance issues. From ff00e9fb385545ad41ce6047738a349c31d05d56 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 4 Aug 2021 16:05:13 +0800 Subject: [PATCH 7/8] Update statement-summary-tables.md --- statement-summary-tables.md | 1 - 1 file changed, 1 deletion(-) diff --git a/statement-summary-tables.md b/statement-summary-tables.md index 8abb94e84cae6..f7a96ae850b57 100644 --- a/statement-summary-tables.md +++ b/statement-summary-tables.md @@ -15,7 +15,6 @@ Therefore, starting from v4.0.0-rc.1, TiDB provides system tables in `informatio - [`cluster_statements_summary`](#statements_summary_evicted) - [`cluster_statements_summary_history`](#statements_summary_evicted) - [`statements_summary_evicted`](#statements_summary_evicted) -- [`statements_summary_evicted`](#statements_summary_evicted) This document details these tables and introduces how to use them to troubleshoot SQL performance issues. From acc6c860b483c993bcc7a45610a7b007fb9c69d1 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 25 Aug 2021 13:26:33 +0800 Subject: [PATCH 8/8] Apply suggestions from code review --- statement-summary-tables.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/statement-summary-tables.md b/statement-summary-tables.md index f7a96ae850b57..fbc448e7a7e26 100644 --- a/statement-summary-tables.md +++ b/statement-summary-tables.md @@ -92,9 +92,9 @@ The fields `SUMMARY_BEGIN_TIME` and `SUMMARY_END_TIME` represent the start time ## `statements_summary_evicted` -The `tidb_stmt_summary_max_stmt_count` variable controls the maximum number of statements that the `statement summary` table stores in memory. The `statement summary` table uses the LRU algorithm. Once the number of SQL statements exceed the `tidb_stmt_summary_max_stmt_count` value, the longest unused record is evicted from the table. The number of evicted SQL statements during each period are recorded in the `statements_summary_evicted` table. +The `tidb_stmt_summary_max_stmt_count` variable controls the maximum number of statements that the `statement_summary` table stores in memory. The `statement_summary` table uses the LRU algorithm. Once the number of SQL statements exceeds the `tidb_stmt_summary_max_stmt_count` value, the longest unused record is evicted from the table. The number of evicted SQL statements during each period is recorded in the `statements_summary_evicted` table. -The `statements_summary_evicted` table is updated only when a SQL record is evicted from the `statement summary` table. The `statements_summary_evicted` only records the period during which the eviction occurs and the number of evicted SQL statements. +The `statements_summary_evicted` table is updated only when a SQL record is evicted from the `statement_summary` table. The `statements_summary_evicted` only records the period during which the eviction occurs and the number of evicted SQL statements. ## The `cluster` tables for statement summary @@ -115,7 +115,7 @@ The following system variables are used to control the statement summary: > **Note:** > -> When a category of SQL statement needs to be removed because the `tidb_stmt_summary_max_stmt_count` limit is exceeded, TiDB removes the data of that SQL statement category of all time ranges from the `statement summary history` table. Therefore, even if the number of SQL statement categories in a certain time range does not reach the limit, the number of SQL statements stored in the `statement summary history` table is less than the actual number of SQL statements. If this situation occurs and affects performance, you are recommended to increase the value of `tidb_stmt_summary_max_stmt_count`. +> When a category of SQL statement needs to be removed because the `tidb_stmt_summary_max_stmt_count` limit is exceeded, TiDB removes the data of that SQL statement category of all time ranges from the `statement_summary_history` table. Therefore, even if the number of SQL statement categories in a certain time range does not reach the limit, the number of SQL statements stored in the `statement_summary_history` table is less than the actual number of SQL statements. If this situation occurs and affects performance, you are recommended to increase the value of `tidb_stmt_summary_max_stmt_count`. An example of the statement summary configuration is shown as follows: @@ -144,7 +144,7 @@ The system variables above have two scopes: global and session. These scopes wor ### Set a proper size for statement summary -After the system has run for a period of time, you can check the `statement summary` table to see whether SQL eviction has occurred. For example: +After the system has run for a period of time, you can check the `statement_summary` table to see whether SQL eviction has occurred. For example: ```sql select @@global.tidb_stmt_summary_max_stmt_count; @@ -167,7 +167,7 @@ select count(*) from information_schema.statements_summary; 1 row in set (0.001 sec) ``` -You can see that the `statement summary` table is full of records. Then check the evicted data from the `statements_summary_evicted` table: +You can see that the `statements_summary` table is full of records. Then check the evicted data from the `statements_summary_evicted` table: ```sql select * from information_schema.statements_summary_evicted;