diff --git a/releases/release-8.5.0.md b/releases/release-8.5.0.md new file mode 100644 index 0000000000000..0ffd99400a0ef --- /dev/null +++ b/releases/release-8.5.0.md @@ -0,0 +1,389 @@ +--- +title: TiDB 8.5.0 Release Notes +summary: Learn about the new features, compatibility changes, improvements, and bug fixes in TiDB 8.5.0. +--- + +# TiDB 8.5.0 Release Notes + + + +Release date: xx xx, 2024 + +TiDB version: 8.5.0 + +Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.5/quick-start-with-tidb) | [Production deployment](https://docs.pingcap.com/tidb/v8.5/production-deployment-using-tiup) + +TiDB 8.5.0 is a Long-Term Support Release (LTS). + +Compared with the previous LTS 8.1.0, 8.5.0 includes new features, improvements, and bug fixes released in [8.2.0-DMR](/releases/release-8.2.0.md), [8.3.0-DMR](/releases/release-8.3.0.md), and [8.4.0-DMR](/releases/release-8.4.0.md). When you upgrade from 8.1.x to 8.5.0, you can download the [TiDB Release Notes PDF](https://download.pingcap.org/tidb-v8.1-to-v8.5-en-release-notes.pdf) to view all release notes between the two LTS versions. The following table lists some highlights from 8.1.0 to 8.5.0: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
分类功能描述
可扩展性与性能 多维度降低数据处理延迟 **tw@qiancai**通过不断挖掘数据处理的细节，TiDB 持续提升自身性能，力求满足金融场景对 SQL 处理时延的要求。 包括以下关键更新： +
  • 并行排序 (v8.2.0 引入)
    • +
  • 优化 KV 请求批处理策略 (v8.3.0 引入)
    • +
  • 并行获取 TSO (v8.4.0 引入)
    • +
  • 删除语句只获取必要的列 (v8.4.0 引入)
    • +
  • 优化缓存表场景性能 (v8.4.0 引入)
    • +
  • Hash Join 算法演进 (v8.4.0 引入)
    • +
    Use Active PD Followers to enhance PD's Region information query service (GA in v8.5.0) **tw@Oreoxmt 2015**TiDB v7.6.0 introduces an experimental feature "Active PD Follower", which allows PD followers to provide Region information query services. This feature improves the capability of the PD cluster to handle GetRegion and ScanRegions requests in clusters with a large number of TiDB nodes and Regions, thereby reducing the CPU pressure on PD leaders. In v8.5.0, this feature becomes generally available (GA).
    实例级执行计划缓存（实验特性）（v8.4.0 引入）实例级执行计划缓存允许同一个 TiDB 实例的所有会话共享执行计划缓存。与现有的会话级执行计划缓存相比，实例级执行计划缓存能够在内存中缓存更多执行计划，减少 SQL 编译时间，从而降低 SQL 整体运行时间，提升 OLTP 的性能和吞吐，同时更好地控制内存使用，提升数据库稳定性。
    分区表全局索引（v8.4.0 起成为正式功能）全局索引可以有效提高检索非分区列的效率，并且消除了唯一键必须包含分区键的限制。该功能扩展了 TiDB 分区表的使用场景，避免了数据迁移过程中的一些应用修改工作。
    默认允许将 Projection 算子下推到存储引擎（v8.3.0 引入） Projection 算子下推可以将负载分散到存储节点，同时减少节点间的数据传输。这有助于降低部分 SQL 的执行时间，提升数据库的整体性能。
    统计信息收集忽略不必要的列（v8.3.0 引入） 在保证优化器能够获取到必要信息的前提下，加快了统计信息收集的速度，提升统计信息的时效性，进而保证选择最优的执行计划，提升集群性能。同时也降低了系统开销，改善了资源利用率。
    稳定性与高可用提升超大规模集群的稳定性 **tw@hfxsd 1976**对于使用 TiDB 运行多租户应用或者 SaaS 应用的公司，经常需要存储大量的表，TiDB 在 v8.5.0 着力增强了大规模集群的稳定性。 Schema 缓存控制以及Stats 缓存控制已经成为正式功能，减少了内存过度消耗带来的稳定性问题。 PD 通过 Active Follower 应对大量 Region 带来的压力，并将 PD 所承担的服务逐步解耦，独立部署。通过增加并发度，以及减少收集对象的数量，统计信息收集和加载效率得到提升，保证了大集群执行计划的稳定性。
    Runaway Queries 支持更多触发条件，并能够切换资源组 （v8.4.0 引入）Runaway Queries 提供了有效的手段来降低突发的 SQL 性能问题对系统产生的影响。v8.4.0 中新增 Coprocessor 处理的 Key 的数量 (PROCESSED_KEYS) 和 Request Unit (RU) 作为识别条件，并可以将识别到的查询置入指定资源组，对 Runaway Queries 进行更精确的识别与控制。
    支持为资源管控的后台任务设置资源使用上限 (实验特性)（v8.4.0 引入）为资源管控的后台任务设置百分比上限，针对不同业务系统的需求，控制后台任务的消耗，从而将后台任务的消耗限制在一个很低的水平，保证在线业务的服务质量。
    Enhance and expand TiProxy use cases **tw@Oreoxmt**As a crucial component of the high availability of TiDB, TiProxy extends its capabilities beyond SQL traffic access and forwarding to support cluster change evaluation. Key features include: +
  • TiProxy supports traffic capture and replay (experimental, introduced in v8.4.0)
    • +
  • Built-in virtual IP management in TiProxy (introduced in v8.3.0)
    • +
  • TiProxy supports multiple load balancing policies (introduced in v8.2.0)
    • +
    并行 HashAgg 算法支持数据落盘 （v8.2.0 起成为正式功能）HashAgg 是 TiDB 中常用的聚合算子，用于快速聚合具有相同字段值的行。TiDB v8.0.0 引入并行 HashAgg 作为实验特性，以进一步提升处理速度。当内存资源不足时，并行 HashAgg 可以将临时排序数据落盘，避免因内存使用过度而导致的 OOM 风险，从而提升查询性能和节点稳定性。该功能在 v8.2.0 成为正式功能，并默认开启，用户可以通过 tidb_executor_concurrency 安全地设置并行 HashAgg 的并发度。
    SQL 外键约束成为正式功能 **tw@lilin90 1894**外键（Foreign Key）是数据库中的一种约束，用于建立表与表之间的关联关系，确保数据一致性和完整性。它可以限制子表中引用的数据必须存在于主表中，防止无效数据插入。同时，外键支持级联操作（如删除或更新时自动同步），简化了业务逻辑的实现，减少了手动维护数据关联的复杂性。
    支持向量搜索功能（实验特性）(v8.4.0 引入）向量搜索是一种基于数据语义的搜索方法，可以提供更相关的搜索结果。作为 AI 和大语言模型 (LLM) 的核心功能之一，向量搜索可用于检索增强生成 (Retrieval-Augmented Generation, RAG)、语义搜索、推荐系统等多种场景。
    数据库管理与可观测性在内存表中显示 TiKV 和 TiDB 的 CPU 时间 （v8.4.0 引入）将 CPU 时间合入系统表中展示，与会话或 SQL 的其他指标并列，方便你从多角度对高 CPU 消耗的操作进行观测，提升诊断效率。尤其适用于诊断实例 CPU 飙升或集群读写热点等场景。
    按表或数据库维度聚合 TiKV 消耗的 CPU 时间（v8.4.0 引入）当热点问题不是由个别 SQL 语句引起时，利用 Top SQL 中按表或者数据库聚合的 CPU 时间，能够协助用户快速发现造成热点的表或者应用程序，从而大大提升热点问题和 CPU 消耗问题的诊断效率。
    支持对开启了 IMDSv2 服务的 TiKV 实例做备份 （v8.4.0 引入）目前 AWS EC2 的默认元数据服务是 IMDSv2。TiDB 支持从开启了 IMDSv2 的 TiKV 实例中备份数据，协助你更好地在公有云服务中运行 TiDB 集群。
    安全日志备份数据支持客户端加密在上传日志备份到备份存储之前，你可以对日志备份数据进行加密，确保数据在存储和传输过程中的安全性。
    + +## Feature details + +### Scalability + +* The schema cache memory limit feature is now generally available (GA), reducing memory usage in scenarios with hundreds of thousands or even millions of databases or tables. [#50959](https://github.com/pingcap/tidb/issues/50959) @[tiancaiamao](https://github.com/tiancaiamao) @[wjhuang2016](https://github.com/wjhuang2016) @[gmhdbjd](https://github.com/gmhdbjd) @[tangenta](https://github.com/tangenta) tw@hfxsd + + In some SaaS scenarios, when the number of tables reaches hundreds of thousands or even millions, the schema meta can consume significant memory. Enabling this feature allows TiDB to use the LRU algorithm to cache and evict the corresponding schema meta information, effectively reducing memory usage. + + Starting from v8.4.0, this feature is enabled by default with a default value of `536870912` (that is, 512MiB). You can adjust it as needed through the variable [`tidb_schema_cache_size`](/system-variables.md#tidb_schema_cache_size-new-in-v800). + + For more information, see [Documentation](/schema-cache.md). + +* Use the Active PD Follower feature to enhance the scalability of PD's Region information query service (GA) [#7431](https://github.com/tikv/pd/issues/7431) @[okJiang](https://github.com/okJiang) tw@Oreoxmt + + In a TiDB cluster with a large number of Regions, the PD leader might experience high CPU load due to the increased overhead of handling heartbeats and scheduling tasks. If the cluster has many TiDB instances, and there is a high concurrency of requests for Region information, the CPU pressure on the PD leader increases further and might cause PD services to become unavailable. + + To ensure high availability, TiDB v7.6.0 introduces Active PD Follower as an experimental feature to enhance the scalability of PD's Region information query service. In v8.5.0, this feature becomes generally available (GA). You can enable the Active PD Follower feature by setting the system variable [`pd_enable_follower_handle_region`](/system-variables.md#pd_enable_follower_handle_region-new-in-v760) to `ON`. After this feature is enabled, TiDB evenly distributes Region information requests to all PD servers, and PD followers can also handle Region requests, thereby reducing the CPU pressure on the PD leader. + + For more information, see [documentation](/tune-region-performance.md#use-the-active-pd-follower-feature-to-enhance-the-scalability-of-pds-region-information-query-service). + +### Performance + +* Placeholder for feature summary [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) **tw@xxx** + + Provide a concise overview of what the feature is, the value it offers to users, and include a brief sentence on how to use it effectively. If there are any particularly important aspects of this feature, be sure to mention them as well. + + For more information, see [Documentation](link). + +* TiDB accelerated table creation becomes generally available (GA), significantly reducing data migration and cluster initialization time [#50052](https://github.com/pingcap/tidb/issues/50052) @[D3Hunter](https://github.com/D3Hunter) @[gmhdbjd](https://github.com/gmhdbjd) tw@Oreoxmt + + TiDB v7.6.0 introduces accelerated table creation as an experimental feature, controlled by the system variable [`tidb_ddl_version`](https://docs.pingcap.com/tidb/v7.6/system-variables#tidb_ddl_version-new-in-v760). Staring from v8.0.0, this system variable is renamed to [`tidb_enable_fast_create_table`](/system-variables.md#tidb_enable_fast_create_table-new-in-v800). + + In v8.5.0, TiDB accelerated table creation becomes generally available (GA) and is enabled by default. During data migration and cluster initialization, this feature supports rapid creation of millions of tables, significantly reducing operation time. + + For more information, see [Documentation](/accelerated-table-creation.md). + +### Reliability + +* Support configuring the rate limit and concurrency for gRPC API requests sent to PD [#5739](https://github.com/tikv/pd/issues/5739) @[rleungx](https://github.com/rleungx) **tw@qiancai** + + When a sudden influx of requests is sent to PD, it can lead to high workloads and potentially affect PD performance. Starting from v8.5.0, you can use [`pd-ctl`](/pd-control.md) to configure rate limits and concurrency for gRPC API requests sent to PD, improving its stability. + + For more information, see [Documentation](/pd-control.md). + +### Availability + +* Placeholder for feature summary [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) **tw@xxx** + + Provide a concise overview of what the feature is, the value it offers to users, and include a brief sentence on how to use it effectively. If there are any particularly important aspects of this feature, be sure to mention them as well. + + For more information, see [Documentation](link). + +### SQL + +* Placeholder for feature summary [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) **tw@xxx** + + Provide a concise overview of what the feature is, the value it offers to users, and include a brief sentence on how to use it effectively. If there are any particularly important aspects of this feature, be sure to mention them as well. + + For more information, see [Documentation](link). + +* The Foreign Key feature is now Generally Available (GA) [#36982](https://github.com/pingcap/tidb/issues/36982) @[YangKeao](https://github.com/YangKeao) @[crazycs520](https://github.com/crazycs520) tw@lilin90 + + The Foreign Key feature is now GA, enabling the use of foreign key constraints to enhance data consistency and integrity. Users can easily create foreign key constraints between tables, with support for cascading updates and deletions, making data management more convenient. This feature provides better support for applications with complex data constraints. + + For more information, see [Documentation](link). + +* Introduce the `ADMIN ALTER DDL JOBS` statement to support modifying the DDL jobs online [#57229](https://github.com/pingcap/tidb/issues/57229) @[fzzf678](https://github.com/fzzf678) @[tangenta](https://github.com/tangenta) tw@hfxsd + + Starting from v8.3.0, you can set the variables [`tidb_ddl_reorg_batch_size`](/system-variables#tidb_ddl_reorg_batch_size) and [`tidb_ddl_reorg_worker_cnt`](/system-variables#tidb_ddl_reorg_worker_cnt) at the session level. As a result, setting these two variables globally no longer affects all running DDL jobs. To modify the values of these variables, you need to cancel the DDL job first, adjust the variables, and then resubmit the job. + + TiDB v8.5.0 introduces the `ADMIN ALTER DDL JOBS` statement, allowing online adjustment of variable values for specific DDL jobs. This enables flexible balancing of resource consumption and performance, with changes limited to an individual job, making the impact more controllable. For example: + + - `ADMIN ALTER DDL JOBS job_id THREAD = 8;`: adjusts the `tidb_ddl_reorg_worker_cnt` for the specified DDL task. + - `ADMIN ALTER DDL JOBS job_id BATCH_SIZE = 256;`: adjusts the `tidb_ddl_reorg_batch_size` for the specified DDL task. + - `ADMIN ALTER DDL JOBS job_id MAX_WRITE_SPEED = '200MiB';`: adjusts the write traffic size for index data on each TiKV node. + + For more information, see [Documentation](/sql-statements/sql-statement-admin-alter-ddl.md). + +### DB operations + +* Placeholder for feature summary [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) **tw@xxx** + + Provide a concise overview of what the feature is, the value it offers to users, and include a brief sentence on how to use it effectively. If there are any particularly important aspects of this feature, be sure to mention them as well. + + For more information, see [Documentation](link). + +### Observability + +* Placeholder for feature summary [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) **tw@xxx** + + Provide a concise overview of what the feature is, the value it offers to users, and include a brief sentence on how to use it effectively. If there are any particularly important aspects of this feature, be sure to mention them as well. + + For more information, see [Documentation](link). + +### Security + +* BR supports client-side encryption of both full backup data and log backup data (GA) [#28640](https://github.com/pingcap/tidb/issues/28640) [#56433] (https://github.com/pingcap/tidb/issues/56433) @[joccau](https://github.com/joccau) @[Tristan1900](https://github.com/Tristan1900) tw@qiancai + + * Client-side encryption of full backup data (introduced as experimental in TiDB v5.3.0) enables you to encrypt backup data on the client side using a custom fixed key. + + * Client-side encryption of log backup data (introduced as experimental in TiDB v8.4.0) enables you to encrypt log backup data on the client side using one of the following methods: + + * Encrypt using a custom fixed key + * Encrypt using a master key stored on a local disk + * Encrypt using a master key managed by a Key Management Service (KMS) + + Starting from v8.5.0, both features become generally available (GA), offering enhanced client-side data security. + + For more information, see [Encrypt the backup data](/br/br-snapshot-manual.md#encrypt-the-backup-data) and [Encrypt the log backup data](/br/br-pitr-manual.md#encrypt-the-log-backup-data). + +* TiKV encryption at rest supports Google [Key Management Service (Cloud KMS)](https://cloud.google.com/docs/security/key-management-deep-dive?hl) (GA) [#8906](https://github.com/tikv/tikv/issues/8906) @[glorv](https://github.com/glorv) **tw@qiancai** + + TiKV ensures data security by using the encryption at rest technique to encrypt stored data. The core aspect of this technique is proper key management. In v8.0.0, TiKV encryption at rest experimentally supports using Google Cloud KMS for master key management. + + Starting from v8.5.0, encryption at rest using Google Cloud KMS becomes generally available (GA). To use this feature, first create a key on Google Cloud, and then configure the `[security.encryption.master-key]` section in the TiKV configuration file. + + For more information, see [documentation](/encryption-at-rest.md#tikv-encryption-at-rest). + +### Data migration + +* Placeholder for feature summary [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) **tw@xxx** + + Provide a concise overview of what the feature is, the value it offers to users, and include a brief sentence on how to use it effectively. If there are any particularly important aspects of this feature, be sure to mention them as well. + + For more information, see [Documentation](link). + +## Compatibility changes + +> **Note:** +> +> This section provides compatibility changes you need to know when you upgrade from v8.1.0 to the current version (v8.2.0). If you are upgrading from v8.0.0 or earlier versions to the current version, you might also need to check the compatibility changes introduced in intermediate versions. + +### Behavior changes + +### MySQL compatibility + +### System variables + +| Variable name | Change type | Description | +|--------|------------------------------|------| +|tidb_ddl_reorg_max_write_speed | Newly added |Used to control the speed at which TiDB writes index data to a single TiKV node. For example, setting the value to 200 MiB limits the maximum write speed to 200 MiB/s. | +| [`tidb_enable_fast_create_table`](/system-variables.md#tidb_enable_fast_create_table-new-in-v800) | Modified | Changes the default value from `OFF` to `ON` after further tests, meaning that the [accelerated table creation](/accelerated-table-creation.md) feature is enabled by default. | +| | | | + +### Configuration parameters + +| Configuration file or component | Configuration parameter | Change type | Description | +| -------- | -------- | -------- | -------- | +| | | | | +| | | | | +| | | | | + +### System tables + +### Other changes + +## Offline package changes + +## Removed features + +## Deprecated features + +The following features are planned for deprecation in future versions: + +* TiDB introduces the system variable [`tidb_enable_auto_analyze_priority_queue`](/system-variables.md#tidb_enable_auto_analyze_priority_queue-new-in-v800), which controls whether priority queues are enabled to optimize the ordering of tasks that automatically collect statistics. In future releases, the priority queue will be the only way to order tasks for automatically collecting statistics, so this system variable will be deprecated. +* TiDB introduces the system variable [`tidb_enable_async_merge_global_stats`](/system-variables.md#tidb_enable_async_merge_global_stats-new-in-v750) in v7.5.0. You can use it to set TiDB to use asynchronous merging of partition statistics to avoid OOM issues. In future releases, partition statistics will be merged asynchronously, so this system variable will be deprecated. +* It is planned to redesign [the automatic evolution of execution plan bindings](/sql-plan-management.md#baseline-evolution) in subsequent releases, and the related variables and behavior will change. +* In v8.0.0, TiDB introduces the [`tidb_enable_parallel_hashagg_spill`](/system-variables.md#tidb_enable_parallel_hashagg_spill-new-in-v800) system variable to control whether TiDB supports disk spill for the concurrent HashAgg algorithm. In future versions, the [`tidb_enable_parallel_hashagg_spill`](/system-variables.md#tidb_enable_parallel_hashagg_spill-new-in-v800) system variable will be deprecated. +* The TiDB Lightning parameter [`conflict.max-record-rows`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) is planned for deprecation in a future release and will be subsequently removed. This parameter will be replaced by [`conflict.threshold`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task), which means that the maximum number of conflicting records is consistent with the maximum number of conflicting records that can be tolerated in a single import task. +* Starting from v6.3.0, partitioned tables use [dynamic pruning mode](/partitioned-table.md#dynamic-pruning-mode) by default. Compared with static pruning mode, dynamic pruning mode supports features such as IndexJoin and plan cache with better performance. Therefore, static pruning mode will be deprecated. + +## Improvements + ++ TiDB + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + ++ TiKV + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + ++ PD + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + ++ TiFlash + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + ++ Tools + + + Backup & Restore (BR) + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + + + TiCDC + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + + + TiDB Data Migration (DM) + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + + + TiDB Lightning + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + + + TiUP + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + +## Bug fixes + ++ TiDB + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + ++ TiKV + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + ++ PD + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + ++ TiFlash + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + ++ Tools + + + Backup & Restore (BR) + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + + + TiCDC + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + + + TiDB Data Migration (DM) + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + + + TiDB Lightning + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + + + TiUP + + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + - note [#Issue-number](issue-link) @[Contributor-GitHub-ID](id-link) + +## Contributors + +We would like to thank the following contributors from the TiDB community: + +- [Contributor-GitHub-ID](id-link)