Skip to content

tikv: update auto compaction config #21656

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+83 −5
Open

tikv: update auto compaction config #21656

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions dynamic-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,10 +138,6 @@ The following TiKV configuration items can be modified dynamically:
| `raftstore.max-apply-unpersisted-log-limit` | The maximum number of committed but not persisted Raft logs that can be applied |
| `raftstore.split-region-check-tick-interval` | The time interval at which to check whether the Region split is needed |
| `raftstore.region-split-check-diff` | The maximum value by which the Region data is allowed to exceed before Region split |
| `raftstore.region-compact-check-interval` | The time interval at which to check whether it is necessary to manually trigger RocksDB compaction |
| `raftstore.region-compact-check-step` | The number of Regions checked at one time for each round of manual compaction |
| `raftstore.region-compact-min-tombstones` | The number of tombstones required to trigger RocksDB compaction |
| `raftstore.region-compact-tombstones-percent` | The proportion of tombstone required to trigger RocksDB compaction |
| `raftstore.pd-heartbeat-tick-interval` | The time interval at which a Region's heartbeat to PD is triggered |
| `raftstore.pd-store-heartbeat-tick-interval` | The time interval at which a store's heartbeat to PD is triggered |
| `raftstore.snap-mgr-gc-tick-interval` | The time interval at which the recycle of expired snapshot files is triggered |
Expand Down Expand Up @@ -195,6 +191,12 @@ The following TiKV configuration items can be modified dynamically:
| `gc.max-write-bytes-per-sec` | The maximum bytes that can be written into RocksDB per second |
| `gc.enable-compaction-filter` | Whether to enable compaction filter |
| `gc.compaction-filter-skip-version-check` | Whether to skip the cluster version check of compaction filter (not released) |
| `gc.auto-compaction.check-interval` | The time interval for checking whether to trigger TiKV automatic (RocksDB) compaction |
| `gc.auto-compaction.tombstone-num-threshold` | The number of RocksDB tombstones required to trigger TiKV automatic (RocksDB) compaction |
| `gc.auto-compaction.tombstone-percent-threshold` | The percentage of RocksDB tombstones required to trigger TiKV automatic (RocksDB) compaction |
| `gc.auto-compaction.redundant-rows-threshold` | The number of redundant MVCC rows required to trigger TiKV automatic (RocksDB) compaction |
| `gc.auto-compaction.redundant-rows-percent-threshold` | The percentage of redundant MVCC rows required to trigger TiKV automatic (RocksDB) compaction |
| `gc.auto-compaction.bottommost-level-force` | Whether to force the bottommost files in RocksDB to participate in compaction |
| `{db-name}.max-total-wal-size` | The maximum size of total WAL |
| `{db-name}.max-background-jobs` | The number of background threads in RocksDB |
| `{db-name}.max-background-flushes` | The maximum number of flush threads in RocksDB |
Expand Down
10 changes: 9 additions & 1 deletion garbage-collection-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,4 +107,12 @@ show config where type = 'tikv' and name like '%enable-compaction-filter%';
| tikv | 172.16.5.36:20163 | gc.enable-compaction-filter | true |
| tikv | 172.16.5.35:20163 | gc.enable-compaction-filter | true |
+------+-------------------+-----------------------------+-------+
```
```

> **Note:**
>
> When using the Compaction Filter mechanism, GC progress might be delayed, which can affect TiKV scan performance. If your workload contains a large number of coprocessor requests and you observe in the [**TiKV-Details > Coprocessor Detail**](/grafana-tikv-dashboard.md#coprocessor-detail) panel that the `next()` or `prev()` call count in **Total Ops Details** significantly exceeds three times the `processed_keys` calls, you can take the following actions:
>
> - For TiDB versions before v7.1.3, it is recommended to disable Compaction Filter to speed up GC.
> - For TiDB versions from v7.1.3 to v7.5.6, TiDB automatically triggers compaction based on the number of redundant versions in each Region [`region-compact-min-redundant-rows`](/tikv-configuration-file.md#region-compact-min-redundant-rows-new-in-v710) and the percentage of redundant versions [`region-compact-redundant-rows-percent`](/tikv-configuration-file.md#region-compact-redundant-rows-percent-new-in-v710) to improve Compaction Filter GC performance. In this case, adjust these configuration items instead of disabling Compaction Filter.
> - Starting from v7.5.7, [`region-compact-min-redundant-rows`](/tikv-configuration-file.md#region-compact-min-redundant-rows-new-in-v710) and [`region-compact-redundant-rows-percent`](/tikv-configuration-file.md#region-compact-redundant-rows-percent-new-in-v710) are deprecated. TiDB now automatically triggers compaction based on [`gc.auto-compaction.redundant-rows-threshold`](/tikv-configuration-file.md#redundant-rows-threshold-new-in-v710-and-v900) and [`gc.auto-compaction.redundant-rows-percent-threshold`](/tikv-configuration-file.md#redundant-rows-percent-threshold-new-in-v710-and-v900). In this case, adjust these configuration items instead of disabling Compaction Filter.
68 changes: 68 additions & 0 deletions tikv-configuration-file.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -817,12 +817,20 @@ Configuration items related to Raftstore.

### `region-compact-check-interval`

> **Warning:**
>
> Starting from v7.5.7 and v9.0.0, this configuration item is deprecated and replaced by [`gc.auto-compaction.check-interval`](#check-interval-new-in-v757-and-v900).

+ The time interval at which to check whether it is necessary to manually trigger RocksDB compaction. `0` means that this feature is disabled.
+ Default value: `"5m"`
+ Minimum value: `0`

### `region-compact-check-step`

> **Warning:**
>
> Starting from v7.5.7 and v9.0.0, this configuration item is deprecated.

+ The number of Regions checked at one time for each round of manual compaction
+ Default value:

Expand All @@ -832,25 +840,41 @@ Configuration items related to Raftstore.

### `region-compact-min-tombstones`

> **Warning:**
>
> Starting from v7.5.7 and v9.0.0, this configuration item is deprecated and replaced by [`gc.auto-compaction.tombstone-num-threshold`](#tombstone-num-threshold-new-in-v757-and-v900).

+ The number of tombstones required to trigger RocksDB compaction
+ Default value: `10000`
+ Minimum value: `0`

### `region-compact-tombstones-percent`

> **Warning:**
>
> Starting from v7.5.7 and v9.0.0, this configuration item is deprecated and replaced by [`gc.auto-compaction.tombstone-percent-threshold`](#tombstone-percent-threshold-new-in-v757-and-v900).

+ The proportion of tombstone required to trigger RocksDB compaction
+ Default value: `30`
+ Minimum value: `1`
+ Maximum value: `100`

### `region-compact-min-redundant-rows` <span class="version-mark">New in v7.1.0</span>

> **Warning:**
>
> Starting from v7.5.7 and v9.0.0, this configuration item is deprecated and replaced by [`gc.auto-compaction.redundant-rows-threshold`](#redundant-rows-threshold-new-in-v757-and-v900).

+ The number of redundant MVCC rows required to trigger RocksDB compaction.
+ Default value: `50000`
+ Minimum value: `0`

### `region-compact-redundant-rows-percent` <span class="version-mark">New in v7.1.0</span>

> **Warning:**
>
> Starting from v7.5.7 and v9.0.0, this configuration item is deprecated and replaced by [`gc.auto-compaction.redundant-rows-percent-threshold`](#redundant-rows-percent-threshold-new-in-v757-and-v900).

+ The percentage of redundant MVCC rows required to trigger RocksDB compaction.
+ Default value: `20`
+ Minimum value: `1`
Expand Down Expand Up @@ -2255,6 +2279,50 @@ Configuration items related to TiDB Lightning import and BR restore.
+ The number of GC threads when `enable-compaction-filter` is `false`.
+ Default value: `1`

## gc.auto-compaction

Configures the behavior of TiKV automatic compaction.

### `check-interval` <span class="version-mark">New in v7.5.7 and v9.0.0</span>

+ The interval for checking whether TiKV needs to trigger automatic compaction. Within this interval, Regions that meet the automatic compaction conditions are processed based on priority. When the interval elapses, TiKV rescans Region information and recalculates priorities.
+ Default value: `"300s"`

### `tombstone-num-threshold` <span class="version-mark">New in v7.5.7 and v9.0.0</span>

+ The number of RocksDB tombstones required to trigger TiKV automatic compaction. TiKV triggers automatic compaction when either this threshold for the number of tombstones is met or the condition for [`tombstone-percent-threshold`](#tombstone-percent-threshold-new-in-v757-and-v900) is satisfied.
+ This configuration item takes effect only when [Compaction Filter](/garbage-collection-configuration.md) is disabled.
+ Default value: `10000`
+ Minimum value: `0`

### `tombstone-percent-threshold` <span class="version-mark">New in v7.5.7 and v9.0.0</span>

+ The percentage of RocksDB tombstones required to trigger TiKV automatic compaction. TiKV triggers automatic compaction when either this threshold for the percentage of tombstones is met or the condition for [`tombstone-num-threshold`](#tombstone-num-threshold-new-in-v757-and-v900) is satisfied.
+ This configuration item takes effect only when [Compaction Filter](/garbage-collection-configuration.md) is disabled.
+ Default value: `30`
+ Minimum value: `0`
+ Maximum value: `100`

### `redundant-rows-threshold` <span class="version-mark">New in v7.5.7 and v9.0.0</span>

+ The number of redundant MVCC rows required to trigger TiKV automatic compaction, including RocksDB tombstones, TiKV stale versions, and TiKV deletion tombstones. TiKV triggers automatic compaction when either this threshold for the number of redundant rows is met or the condition for [`redundant-rows-percent-threshold`](#redundant-rows-percent-threshold-new-in-v757-and-v900) is satisfied.
+ This configuration item takes effect only when [Compaction Filter](/garbage-collection-configuration.md) is enabled.
+ Default value: `50000`
+ Minimum value: `0`

### `redundant-rows-percent-threshold` <span class="version-mark">New in v7.5.7 and v9.0.0</span>

+ The percentage of redundant MVCC rows required to trigger TiKV automatic compaction, including RocksDB tombstones, TiKV stale versions, and TiKV deletion tombstones. TiKV triggers automatic compaction when either this threshold for the percentage of redundant rows is met or the condition for [`redundant-rows-threshold`](#redundant-rows-threshold-new-in-v757-and-v900) is satisfied.
+ This configuration item takes effect only when [Compaction Filter](/garbage-collection-configuration.md) is enabled.
+ Default value: `20`
+ Minimum value: `0`
+ Maximum value: `100`

### `bottommost-level-force` <span class="version-mark">New in v7.5.7 and v9.0.0</span>

+ Controls whether to force the bottommost files in RocksDB to participate in compaction.
+ Default value: `true`

## backup

Configuration items related to BR backup.
Expand Down
Loading