-
Notifications
You must be signed in to change notification settings - Fork 6.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
379 changed files
with
3,160 additions
and
2,682 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| # Progressive TTL | ||
|
|
||
| Progressive TTL provides a capability to manage data retention with different TTL settings and Segment Creation Policies based on the time granularity of metrics. | ||
|
|
||
| BanyanDB employs a Time-To-Live (TTL) mechanism to automatically delete data older than the specified duration. When using BanyanDB as the storage backend, **the `recordDataTTL` and `metricsDataTTL` configurations are deprecated**. Instead, TTL settings should be configured directly within `storage.banyandb`. | ||
|
|
||
| For detailed information, please refer to the [Storage BanyanDB](../setup/backend/storages/banyandb.md) documentation. | ||
|
|
||
| ## Segment Interval and TTL | ||
|
|
||
| BanyanDB's data rotation mechanism manages data storage based on **Segment Interval** and **TTL** settings: | ||
|
|
||
| - **Segment Interval (`SIDays`)**: Specifies the time interval in days for creating a new data segment. Segments are time-based, facilitating efficient data retention and querying. | ||
| - **TTL (`TTLDays`)**: Defines the time-to-live for data within a group, in days. Data that exceeds the TTL will be automatically deleted. | ||
|
|
||
| ### Best Practices for Setting `SIDays` and `TTLDays` | ||
|
|
||
| - **Data Retention Requirements**: Set the TTL based on how long you need to retain your data. For instance, to retain data for 30 days, set the TTL to 30 days. | ||
| - **Segment Management**: Avoid generating too many segments, as this increases the overhead for data management and querying. | ||
| - **Query Requirements**: Align segment intervals with your query patterns. For example: | ||
| - If you frequently query data for the last 30 minutes, set `SIDays` to 1 day. | ||
| - For querying data from the last 7 days, set `SIDays` to 7 days. | ||
|
|
||
| ## Configuration Guidelines | ||
|
|
||
| ### Record Data | ||
|
|
||
| For both standard and super datasets: | ||
|
|
||
| - **Recommended `SIDays`**: `1` | ||
| - Most queries are performed within a day. | ||
| - **`TTLDays`**: Set according to your data retention needs. | ||
|
|
||
| ### Metrics Data | ||
|
|
||
| Configure `SIDays` and `TTLDays` based on data retention and query requirements. Recommended settings include: | ||
|
|
||
| | Group | `SIDays` | `TTLDays` | | ||
| |----------------|----------|-----------| | ||
| | Minute (`gmMinute`) | 1 | 7 | | ||
| | Hour (`gmHour`) | 5 | 15 | | ||
| | Day (`gmDay`) | 15 | 15 | | ||
| | Index (`gmIndex`) | 15 | 15 | | ||
|
|
||
| **Group Descriptions:** | ||
|
|
||
| - **Minute (`gmMinute`)**: Stores metrics with a 1-minute granularity. Suitable for recent data queries requiring minute-level detail. Consequently, it has shorter `SIDays` and `TTLDays` compared to other groups. | ||
| - **Hour (`gmHour`)**: Stores metrics with a 1-hour granularity. Designed for queries that need hour-level detail over a longer period than minute-level data. | ||
| - **Day (`gmDay`)**: Stores metrics with a 1-day granularity. This group handles the longest segment intervals and TTLs among all granularity groups. | ||
| - **Index (`gmIndex`)**: Stores metrics used solely for indexing without value columns. Since queries often scan all segments in the `index` group, it shares the same `SIDays` and `TTLDays` as the `day` group to optimize performance. This group's `TTL` must be set to the **max** value of all groups. | ||
|
|
||
| For more details on configuring `segmentIntervalDays` and `ttlDays`, refer to the [BanyanDB Rotation](https://skywalking.apache.org/docs/skywalking-banyandb/latest/concept/rotation/) documentation. |
Oops, something went wrong.