Release Note 2.1.6

[yiguolei]

Search before asking

• I had searched in the issues and found no similar issues.

Description

Behavior changes

• Removed the delete_if_exists option from create repository. [chore](backup) Remove delete_if_exists properties for creating repository #38192
• Added the enable_prepared_stmt_audit_log session variable to control whether JDBC prepared statements record audit logs, with the default being no recording. [Cherry-pick](branch-2.1) Pick "[Enhancement](audit log) Add print audit log sesssion variable #38419" #38624 [cherry-pick](branch-2.1) Pick "[Enhancement](audit log) Set print audit log sesssion variable default value to false #38865" #39009
• Implemented fd limit and memory constraints for segment cache. [improvement](segmentcache) limit segment cache by fd limit or memory… #39689
• When the FE configuration item sys_log_mode is set to BRIEF, file location information is added to the logs. ([fix](fe-log) add position info in async mode #39419 #39571)
• Changed the default value of the session variable max_allowed_packet to 16MB. ([opt](variables) enlarge the default value of max_allowed_packet #38697)
• When a single request contains multiple statements, semicolons must be used to separate them. [fix](syntax) multi statements must delim with semicolon #38670
• Added support for statements to begin with a semicolon. [opt](Nereids) support parse sql with semicolon at beginning #39399
• Aligned type formatting with MySQL in statements such as show create table. [fix](compatibility) type toSql should return lowercase string #38012
• When the new optimizer planning times out, it no longer falls back to prevent the old optimizer from using longer planning times. [opt](Nereids) do not fallback if nereids failed because timeout #39499

New features

Lakehouse

• Supported writeback for Iceberg tables: https://doris.apache.org/zh-CN/docs/2.1/lakehouse/datalake-building/iceberg-build
• SQL interception rules now support external tables: https://doris.apache.org/zh-CN/docs/2.1/admin-manual/query-admin/sql-interception
• Added the system table file_cache_statistics to view BE data cache metrics: https://doris.apache.org/docs/2.1/admin-manual/system-tables/file_cache_statistics
  Async Materialized View
• Supported transparent rewriting during inserts: [feature](mtmv) Support querying rewrite by materialized view when insert and insert overwrite dml #38115
• Supported transparent rewriting when variant types exist in queries: [feature](mtmv) Support variants rewrite by materialized view #37929
  Semi-Structured Data Management
• Supported casting ARRAY MAP to JSON type: [feature](cast) Support cast complex type to json #36548
• Supported the json_keys function: [Feature](json_functions) support json-keys  #36411
• Supported specifying the JSON path $. when importing JSON: [Feature](JsonReader) support sepecify $. as root column in json path #38213
• ARRAY MAP STRUCT types now support replace_if_not_null: [improve](nested_types)support nested type with agg replace_if_not_null #38304
• ARRAY MAP STRUCT types now support adjusting column order: [improve](nestedtype) support schema change for nested type #39210
• Added the multi_match function to match keywords across multiple fields, with support for inverted index acceleration: [feature](inverted index) multi_match function add #37722

Query Optimizer

• Filled in the original database name, table name, column name, and alias for returned columns in the MySQL protocol. [opt](compatibility) fill schema and table name in protocol column def #38126
• Supported the aggregation function group_concat with both order by and distinct simultaneously. [fix](nereids)support group_concat with distinct and order by #38080
• SQL cache now supports reusing cached results for queries with different comments. [enhancement](Nereids) support reuse sql cache between different comment #40049
• In partition pruning, supported including date_trunc and date functions in filter conditions. [Feat](nereids)Support date_trunc function in partition prune #38025 [Feat](nereids) support date function in partition prune #38743
• Allowed using the database name where the table resides as a qualifier prefix for table aliases. [chore](nereids) Added compatibility with mysql alias filter #38640
• Supported hint-style comments. [feat](Nereids) support nereids hint position detaction #39113
  Others
• Added the system table table_properties for viewing table properties: https://doris.apache.org/docs/2.1/admin-manual/system-tables/table_properties
• Introduced deadlock and slow lock detection in FE: https://doris.apache.org/zh-CN/docs/2.1/admin-manual/maint-monitor/frontend-lock-manager

Improvements

Lakehouse

• Reimplemented the external table metadata caching mechanism. For details, refer to the documentation: https://doris.apache.org/zh-CN/docs/2.1/lakehouse/metacache/
• Added the session variable keep_carriage_return with a default value of false. By default, reading Hive Text format tables treats both \r\n and \n as newline characters. ([feature](csv)Supports reading CSV data using LF and CRLF as line separators. (#37687) #38099)
• Optimized memory statistics for Parquet/ORC file read/write operations. ([Feature](multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。 #37257)
• Supported pushing down IN/NOT IN predicates for Paimon tables. ([bugfix](paimon)add support for 'in' and 'not in' #38390)
• Enhanced the optimizer to support Time Travel syntax for Hudi tables. ([bugfix](external)Modify the default value of pushdowncount for 2.1 (#37754) #38591)
• Optimized Kerberos authentication-related processes. ([fix](kerberos)fix and refactor ugi login for kerberos and simple authentication #37301)
• Enabled reading Hive tables after Rename column operations. ([feature](hive)Support reading renamed Parquet Hive and Orc Hive tables. (#38432) #38809)
• Optimized the reading performance of partition columns for external tables. ([opt](serde)Optimize the filling of fixed values ​​into block columns without repeated deserialization. (#37377) (#38245) #38810)
• Improved the data shard merging strategy during external table query planning to avoid performance degradation caused by a large number of small shards. ([opt](catalog) merge scan range to avoid too many splits (#38311) #38964)
• Added attributes such as location to SHOW CREATE DATABASE/TABLE. ([improvement](iceberg)add some description for show create for 2.1 #39179 #39644)
• Supported complex types in MaxCompute Catalog. ([fix](multi-catalog)fix max compute array and map type read offset #39822)
• Optimized the file cache loading strategy by using asynchronous loading to avoid long BE startup times. ([improve](cache) File cache async init #39036)
• Improved the file cache eviction strategy, such as evicting locks held for extended periods. ([opt](file-cache) add evict file number per round #39721)

Async Materialized View

• Supported hourly, weekly, and quarterly partition roll-up construction. [enhance](mtmv)mtmv date trunc support hour #37678
• For materialized views based on Hive external tables, the metadata cache is now updated before refresh to ensure the latest data is obtained during each refresh. [enhance](mtmv)refresh hms table before run mtmv task #38212
• Improved the performance of transparent rewrite planning in storage-compute separation mode by batch fetching metadata. [enhance](mtmv)Improve the performance of obtaining partition/table versions #39301
• Enhanced the performance of transparent rewrite planning by prohibiting duplicate enumerations. [improvement](mtmv) Only Generate rewritten plan when generate mv plan for performance #39541
• Improved the performance of transparent rewrite for refreshing materialized views based on Hive external table partitions. [enhance](mtmv)hive cache add partitionId to partitionName Map #38525

Semi-Structured Data Management

• Optimized memory allocation for TOPN queries to improve performance. [opt](inverted index) Optimization of the initialization process in topn #37429
• Enhanced the performance of string processing in inverted indexes. [opt](inverted index) opt value extraction from column to string #37395
• Optimized the performance of inverted indexes in MOW tables. [opt](inverted index) mow supports index optimization #37428
• Supported specifying the row-store page_size during table creation to control compression effectiveness. [Optimize](Row store) allow to set row_store_page_size for tables, change default value to 16KB #37145

Query Optimizer

• Adjusted the row count estimation algorithm for mark joins, resulting in more accurate cardinality estimates for mark joins. [fix](nereids) refine row count estimation for mark join #38270
• Optimized the cost estimation algorithm for semi/anti joins, enabling more accurate selection of semi/anti join orders. [opt](nereids) refine left semi/anti cost under short-cut opt #37951
• Adjusted the filter estimation algorithm for cases where some columns have no statistical information, leading to more accurate cardinality estimates. [fix](nereids)filter estimation for slot=unknown #39592
• Modified the instance calculation logic for set operation operators to prevent insufficient parallelism in extreme cases. [fix](nereids) fix union all instance number #39999
• Adjusted the usage strategy of bucket shuffle, achieving better performance when data is not sufficiently shuffled. [opt](nereids) recover adoptive bucket shuffle #36784
• Enabled early filtering of window function data, supporting multiple window functions in a single projection. [opt](nereids) support partitionTopn for multi window exprs #38393
• When a NullLiteral exists in a filter condition, it can now be folded into False, further converted to an EmptySet to reduce unnecessary data scanning and computation. [improvement](Nereids) add where Null rule to create empty relation as where false #38135
• Expanded the scope of predicate derivation, reducing data scanning in queries with specific patterns. [Feat](nereids) add pull up literal when infer predicates #37314
• Supported partial short-circuit evaluation logic in partition pruning to improve partition pruning performance, achieving over 100% improvement in specific scenarios. [enhancement](nereids) Speedup partition pruner #38191
• Enabled the computation of arbitrary scalar functions within user variables. [fix](variable) support all type functions #39144
• Maintained error messages consistent with MySQL when alias conflicts exist in queries. [chore](nereids) Added compatibility with mysql alias conflict #38104

Query Execution

• Adapted AggState for compatibility from 2.1 to 3.x and fixed coredump issues. [Compatibility](agg-state) do not throw eception when DataTypeAggState meet old be exec version #37104
• Refactored the strategy selection for local shuffle when no joins are involved. [Improvement](scan) Refine ignoring data distribution without join #37282
• Modified the scanner for internal table queries to an asynchronous approach to prevent blocking during internal table queries. [Improvement](schema scan) Use async scanner for schema scanners #38403
• Optimized the block merge process when building hash tables in Join operators. [opt](hash_join) Merging all build blocks at once will cause performance issue #37471
• Reduced the lock holding time for MultiCast operations. [opt](MultiCast) Avoid copying while holding a lock #37462
• Optimized gRPC's keepAliveTime and added a connection monitoring mechanism, reducing the probability of query failures due to RPC errors during query execution. [Improvement](rpc) set grpc channel's keepAliveTime and remove proxy on InterruptedExcep… #37304
• Cleaned up all dirty pages in jemalloc when memory limits are exceeded. [opt](memory) Purge all unused Jemalloc dirty pages for all arenas when exceed limit #37164
• Improved the performance of aes_encrypt/decrypt functions when handling constant types. [improve](function) opt aes_encrypt/decrypt function to handle const column #37194
• Optimized the performance of json_extract functions when processing constant data. [opt](fucntion) improve json_extract function handle const column #36927
• Optimized the performance of ParseUrl functions when processing constant data. [opt](function) opt ParseUrl function by process the input const column #36882

Backup Recovery / Cross-Cluster Synchronization

• Restore now supports deleting redundant tablets and partition options. [feature](restore) Support clean_tables/clean_partitions properties for restore job #39028 #39363
• Checks storage connectivity when creating a repository. [enhance](Backup) Do connectivity check when creating repository (#38350) #39538
• Enables binlog to support DROP TABLE, allowing CCR to incrementally synchronize DROP TABLE operations. [chore](fe) Returns dropped tables in GetMeta request #38541

Compaction

• Improves the issue where high-priority compaction tasks were not subject to task concurrency control limits. [enhancement](compaction) Control the parallelism for urgent compacton tasks (#37782) #38189
• Automatically reduces compaction memory consumption based on data characteristics. [enhancement](compaction) optimizing memory usage for compaction (#37099) #37486
• Fixes an issue where the sequential data optimization strategy could lead to incorrect data in aggregate tables or MOR UNIQUE tables. [fix](compaction) fix duplicate key in agg/mor table caused by ordered data compaction (#38224) #38299
• Optimizes the rowset selection strategy during compaction during replica replenishment to avoid triggering -235 errors. [fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets (#38728) #39262

MOW (Merge-On-Write)

• Optimizes slow column updates caused by concurrent column updates and compactions. [branch-2.1] Picks "[opt](merge-on-write) Skip the alignment process of some rowsets in partial update #38487" #38682
• Fixes an issue where segcompaction during bulk data imports could lead to incorrect MOW data. [fix](MoW) fix MoW & segcompaction conflict on cache of temp segment … #38992 [fix](merge-on-write) segcompaction should process delete bitmap if necessary (#38369) #39707
• Fixes data loss in column updates that may occur after BE restarts. [branch-2.1] Picks "[Fix](partial update) Persist partial_update_info in RocksDB in case of BE restart after a partial update has commited #38331" #39035

Storage Management

• Adds FE configuration to control whether queries under hot-cold tiering prefer local data replicas. Pick "[enhance](Cooldown) Use config to control whether use cooldown replica for scanning first" #38322
• Optimizes expired BE report messages to include newly created tablets. [fix](clone) fix stale tablet report miss the new cloning replica #38695 #38839 [improvement](report) report handler discard old report tasks #39469 #39605
• Optimizes replica scheduling priority strategy to prioritize replicas with missing data. [improvement](tablet scheduler) Adjust tablet sched priority to help load data succ #38528 #38884
• Prevents tablets with unfinished ALTER jobs from being balanced. [improvement](balance) don't balance tablet which has unfinish alter job #39121 #39202
• Enables modifying the number of buckets for tables with list partitioning. [fix](fe) LIST partition table support modify default bucket num #39688
• Prefers querying from online disk services. [improvement](query) prefer to chose tablet on alive disk #39467 #39654
• Improves error messages for materialized view base tables that do not support deletion during synchronization. [enhancement](delete) Add a hint msg for forbidden delete when MV or rollup exists (#39505) #39857
• Improves error messages for single columns exceeding 4GB. [improvemen](overflow) Provide the user with a suggestion to avoid th… (#39631) #39897
• Fixes an issue where aborted transactions were omitted when plan errors occurred during INSERT statements. [fix](insert) Pick Insert init plan error may forget to abort txn (#34220) #38260
• Fixes exceptions during SSL connection closure. [cherry-pick](SSL) Fix ssl connection close 2.1 (#38587) #38677
• Fixes an issue where table locks were not held when aborting transactions using labels. [fix](txn) fix abortTxn by label does not acquire table write lock (#38777) #38842
• Fixes gson pretty causing large image issues. [fix](gson) Fix GSON pretty printting setting #37211 #39135
• Fixes an issue where the new optimizer did not check for bucket values of 0 in CREATE TABLE statements. [fix](create table) fix nereids create table no check buckets=0 #38971 #38999
• Fixes errors when Chinese column names are included in DELETE condition predicates. [fix](delete) Only apply regex check on delete condition str for non-lsc tables (#39357) #39500
• Fixes frequent tablet balancing issues in partition balancing mode. [fix](partition rebalancer) fix migrate tablets between backends back and forth #39333 #39606
• Fixes an issue where partition storage policy attributes were lost. [BUG] fix partition storage policy info lost (#38700) #39677
• Fixes incorrect statistics when importing multiple tables within a transaction. [fix](statistics) correct update rows when doing multi-table load #39548
• Fixes errors when deleting random bucket tables. [cherry-pick](branch-2.1) fix delete random distributed tbl #39830
• Fixes issues where FE fails to start due to non-existent UDFs. [fix](udf) Fix exception thrown when replayDropFunction #39868
• Fixes inconsistencies in the last failed version between FE master and slave. [fix](replay) fix replay ReplicaPersistInfo no update lastFailedVersion and lastSuccVersion #39918 #39947
• Fixes an issue where related tablets may still be in schema change state when schema change jobs are canceled. [fix](schema-change) Fix potential data race when a schema change jobs is set to cancelled but the table state is still SCHEMA_CHANGE (#39164) #39327
• Fixes errors when modifying type and column order in a single statement schema change (SC). [fix](schema-change) Fix wrong intput column for cast validity check (#38894) #39107

Data Loading

• Improves error messages for -238 errors during imports. [chore](rowset writer) print rowset rows number when meet too many segments (#39091) #39182
• Allows importing to other partitions while restoring a partition. [enhancement](Load)allow load data to the other partitions when some partitions are restoring #39915
• Optimizes the strategy for FE to select BEs during group commit. [cherry-pick](branch-2.1) Pick "[Enhancement](group commit)Optimize be select for group commit #35558" #37830 [cherry-pick](branch-2.1) Pick "[Fix](group commit) Fix multiple cluster group commit BE select strategy (#38644)" #39010
• Avoids printing stack traces for some common streamload error messages. [Opt](load) don't print stack when some errors occur for stream load #38332 #38418
• Improves handling of issues where offline BEs may affect import errors. [fix](load data) decommission replica don't load data when it misses versions #38198 #38256

Permissions

• Optimizes access performance after enabling the Ranger authentication plugin. ([enhance](auth)support cache ranger datamask and row filter (#37723) #38575)
• Optimizes permission strategies for Refresh Catalog/Database/Table operations, allowing users to perform these operations with only SHOW permissions. ([enhance](auth)modify priv of refresh catalog/db/table #39008)

Bug fixes

Lakehouse

• Fixes the issue where switching catalogs may result in an error of not finding the database. ([fix](multicatalog) make lastdbofcatalog a session variable #38114)
• Addresses exceptions caused by attempting to read non-existent data on S3. ([fix](external) record not found file number #38253)
• Resolves the issue where specifying an abnormal path during export operations may lead to incorrect export locations. ([Fix](outfile) FE check the hdfs URI of outfile #38602)
• Fixes the timezone issue for time columns in Paimon tables. ([bugfix](paimon)Fixed the reading of timestamp with time zone type data #37716)
• Temporarily disables the Parquet PageIndex feature to avoid certain erroneous behaviors.
• Corrects the selection of Backend nodes in the blacklist during external table queries. ([fix](catalog) remove backend in black list from candidate backends for external table #38984)
• Resolves errors caused by missing subcolumns in Parquet Struct column types. ([bug](parquet)Fix the problem that the parquet reader reads the missing sub-columns of the struct and fails. (#38718) #39192)
• Addresses several issues with predicate pushdown in Jdbc Catalog. ([branch-2.1][fix](expr) Enhance SQL Expression Handling by Introducing printSqlInParens to CompoundPredicate #39082)
• Fixes issues where some historical Parquet formats led to incorrect query results. ([Fix](parquet-reader) Fix and optimize parquet min-max filtering.  #39375)
• Improves compatibility with ojdbc6 drivers for Oracle JDBC Catalog. ([branch-2.1][improvement](jdbc catalog) Compatible with ojdbc6 by adding version check  #39408)
• Resolves potential FE memory leaks caused by Refresh Catalog/Database/Table operations. ([improvement](external catalog)Optimize the process of refreshing catalog for 2.1 (#39205) #39186) ([fix](ctas) fix NPE when ctas with old planner and varchar issue (#39744) #39871)
• Fixes thread leaks in JDBC Catalog under certain conditions. ([branch-2.1][improvement](jdbc catalog) Force all resources to be closed in the close method #39666) ([fix](jdbc catalog) Fix Memory Leak by Enabling Weak References in HikariCP #39582)
• Addresses potential event processing failures after enabling Hive Metastore event subscription. ([fix](hive)Modify the Hive notification event processing method when using meta cache and add parameters to the Hive catalog. #39239)
• Disables reading Hive Text format tables with custom escape characters and null formats to prevent data errors. ([fix](hive) report error with escape char and null format (#39700) #39869)
• Resolves issues accessing Iceberg tables created via the Iceberg API under certain conditions. ([fix](catalog) refactor location path and support default fs #39116 #39203)
• Fixes the inability to read Paimon tables stored on HDFS clusters with high availability enabled. ([fix](paimon) fix not able to read paimon data from hdfs with HA (#39806) #39876)
• Addresses errors that may occur when reading Paimon table Deletion Vectors after enabling file caching. ([fix](paimon) fix crash when enable cache with paimon deletion vector(#39877) #39875)
• Resolves potential deadlocks when reading Parquet files under certain conditions. ([Fix](parquet-reader) Fix definition level rle decode dead loop in parquet-reader. (#39523) #39945)

Async Materialized View

• Fixes the inability to use SHOW CREATE MATERIALIZED VIEW on follower FEs. ([fix](mtmv)fix can not show create mtmv use follower fe #38794)
• Unifies the object type of asynchronous materialized views in metadata as tables to enable proper display in data tools. ([enhance](mtmv)change mysql table type of mtmv to table #38797)
• Resolves the issue where nested asynchronous materialized views always perform full refreshes. ([fix](mtmv)fix second level MTMV will refresh all partition #38698)
• Fixes the issue where canceled tasks may show as running after restarting FEs. ([fix](mtmv)Fix cancelled tasks with running status #39424)
• Addresses incorrect use of contexts, which may lead to unexpected failures of materialized view refresh tasks. ([fix](mtmv) mtmv getPlanBySql should not reuse ctx's StatementContext #39690)
• Resolves issues that may cause varchar type write failures due to unreasonable lengths when creating asynchronous materialized views based on external tables. ([fix](mtmv)transfer col in mysql varchar to text when create MTMV #37668)
• Fixes the potential invalidation of asynchronous materialized views based on external tables after FE restarts or catalog rebuilds. ([fix](mtmv)use name instead of id in meta of MTMV #39355)
• Prohibits the use of partition rollup for materialized views with list partitions to prevent the generation of incorrect data. ([enhance](mtmv)Disable mtmv list rollup #38124)
• Fixes incorrect results when literals exist in the select list during transparent rewriting for aggregation rollup. ([fix](mtmv) Fix select literal result wrongly in group by when use materialized view #38958)
• Addresses potential errors during transparent rewriting when queries contain filters like a = a. ([fix](mtmv) Fix result wrong when query rewrite by mv if query contains null_unsafe equals expression #39629)
• Fixes issues where transparent rewriting for direct external table queries fails. ([fix](mtmv) Fix rewrite fail when query direct external table without group by #39041)

Semi-Structured Data Management

• Removes support for prepared statements in the old optimizer. ([Refactor](Prepared Statement) remove implementation in legacy planner #39465)
• Fixes issues with JSON escape character handling. ([fix](serde)fix string deserialize with unescaped char #37251)
• Resolves issues with duplicate processing of JSON fields. ([fix](simdjson) fix simdjson reader will parse given root twice when data is object array  #38490)
• Fixes issues with some ARRAY and MAP functions. ([fix](arrayfuncs) fix array_min/array_max func  #39307, [fix](funcs) map struct construct funcs #39699, [fix](table-func) fix explode_map with alias  #39757)
• Resolves complex combinations of inverted index queries and LIKE queries. ([Fix](inverted index) fix wrong no need read data when same column in inverted index and like function #36687)

Query Optimizer

• Fixed the potential partition pruning error issue when 'OR' condition exists in partition filter conditions. [Fix](nereids) fix partition_prune or expression evaluate wrongly #38897
• Fixed the potential partition pruning error issue when complex expressions are involved. [Fix](nereids) fix condition function partition prune #39298
• Fixed the issue where nullable in agg_state subtypes might be planned incorrectly, leading to execution errors. [fix](agg_state) adjust nullable should apply on agg_state inner type too #37489
• Fixed the issue where nullable in set operation operators might be planned incorrectly, leading to execution errors. [fix](Nereids) set operation output nullable maybe wrong #39109
• Fixed the incorrect execution priority issue of intersect operator. [opt](set operation) INTERSECT should evaluated before others #39095
• Fixed the NPE issue that may occur when the maximum valid date literal exists in the query. [fix](nereids)prevent null pointer exception if datetime value overflows #39482
• Fixed the occasional planning error that results in an illegal slot error during execution. [fix](Nereids) should push project through limit after eliminate union node #39640
• Fixed the issue where repeatedly referencing columns in cte may lead to missing data in some columns in the result. [fix](Nereids) producer to consumer should be multimap in cte #39850
• Fixed the occasional planning error issue when 'case when' exists in the query. [fix](nereids) fix The children format needs to be [WhenClause+, DefaultValue?] #38491
• Fixed the issue where IP types cannot be implicitly converted to string types. [fix](Nereids) support implicit cast ip types to string #39318
• Fixed the potential planning error issue when using multi-dimensional aggregation and the same column and its alias exist in the select list. [Fix](nereids) fix normalize repeat alias rewrite #38166
• Fixed the issue where boolean types might be handled incorrectly when using be constant folding. [fix](Nereids) cast to boolean wrong when constant folding by be #39019
• Fixed the planning error issue caused by 'default_cluster:' as a prefix for the database name in expressions. [Fix](nereids) fix bind expression compare dbname ignore cluster #39114
• Fixed the potential deadlock issue caused by 'insert into'. [fix](Nereids) remove db readlock before get table from db #38660
• Fixed the potential planning error issue caused by not holding table locks throughout the planning process. [fix](Nereids) lock table when generate distribute plan #38950
• Fixed the issue where char(0), varchar(0) are not handled correctly when creating tables. [Fix](nereids) change char(0) to char(1), varchar(0) to varchar(65533) when create table #38427
• Fixed the issue where 'show create table' may incorrectly display hidden columns. [chore](sql) Forbid show hidden columns and create table with hidden column #38796
• Fixed the issue where columns with the same name as hidden columns are not prohibited when creating tables. [chore](sql) Forbid show hidden columns and create table with hidden column #38796
• Fixed the occasional planning error issue when executing 'insert into as select' with CTEs. [fix](nereids) fix insert stmt throw MultiCastDataSink cannot be cast to DataStreamSink #38526
• Fixed the issue where 'insert into values' cannot automatically fill null default values. [fix](Nereids) fix insert into table with null literal default value #39122
• Fixed the NPE issue caused by using cte in delete without using using. [fix](Nereids) npe when delete with cte and without using #39379
• Fixed the issue where deleting from a randomly distributed aggregation model table fails. [bug](delete) fix delete random distributed tbl #37985

Query Execution

• Fixed the issue where the pipeline execution engine gets stuck in multiple scenarios, causing queries not to end. [fix](local exchange) Fix local exchange blocked by a huge data block #38657 [fix](pipeline) Fix blocked tasks if query is canceled before opening #38206 [fix](pipeline) Fix mem control in local exchanger #38885
• Fixed the coredump issue caused by null and non-null columns in set difference calculations. [Bug](fix) fix coredump case in (not null, null) execpt (not null, t null) case #38737
• Fixed the incorrect result issue of the width_bucket function. [fix](function) the bucket number arg of width_bucket should be a positive integer value #37892
• Fixed the query error issue when a single row of data is large and the result set is also large (exceeding 2GB). [fix](sink) The issue with 2GB limit of protocol buffer #37990
• Fixed the incorrect result issue of stddev with DecimalV2 type. [fix](function) stddev with DecimalV2 type will result in an error #38731
• Fixed the coredump issue caused by the MULTI_MATCH_ANY function. [Fix](function) fix coredump for MULTI_MATCH_ANY #37959
• Fixed the issue where insert overwrite auto partition causes transaction rollback. [Fix](insert-overwrite) Fix insert overwrite auto detect transaction safe #38103
• Fixed the incorrect result issue of the convert_tz function. [fix](function) fix wrong result when convert_tz is out of bound #37358 [fix](function) Fix wrong nullable processing for convert_tz #38764
• Fixed the coredump issue when using the collect_set function with window functions. [Bug](agg) fix collect_set function core dump without arena pool #38234
• Fixed the coredump issue caused by the mod function with abnormal input. [Bug](function) fix mod function cause core dump #37999
• Fixed the issue where executing the same expression in multiple threads may lead to incorrect Java UDF results. [bug](expr)fix operator do_projections should use local_state intermediate_projections #38612
• Fixed the overflow issue caused by the incorrect return type of the conv function. [bug](function) fix conv function get wrong result as parse overflow  #38001
• Fixed the unstable result issue of the histogram function. [fix](hist) Fix unstable result of aggregrate function hist #38608

Backup & Recovery / CCR

• Fixed the issue where the data version after backup and recovery may be incorrect, leading to unreadability. [fix](restore) Reset next version for restored partitions #38343
• Fixed the issue of using restore version across versions. [fix](backup) Read BackupMeta with the specified meta version #38396
• Fixed the issue where the job is not canceled when backup fails. [bugfix](backup)(cooldown) cancel backup properly when be backup failed (#38724) #38993
• Fixed the NPE issue in ccr during the upgrade from 2.1.4 to 2.1.5, causing the FE to fail to start. [fix](binlog) Fix NPE when recover binlogs (#39909) #39910
• Fixed the issue where views and materialized views cannot be used after restore. [chore](backup) Fix the db name of the restored view #38072 [fix](restore) Reset the db name of the materialized index stmt #39710 #39848

Storage Management

• Fixed possible memory leaks in routineload when loading multiple tables from a single stream. [fix](multi table) fix single stream multi table memory leak (#38255) #38824
• Fixed the issue where delimiters and escape characters in routineload were not effective. [fix](routine load) fix enclose and escape can not set in routine load job (#38402) #38825
• Fixed incorrect show routineload results when the routineload task name contained uppercase letters. [fix](routine load) fix show routine load task result incorrect (#38523) #38826
• Fixed the issue where the offset cache was not reset when changing the routineload topic. [fix](routine load) reset Kafka progress cache when routine load job topic change #38474
• Fixed the potential exception triggered by show routineload under concurrent scenarios. [fix](routine load) add read lock to fix some concurrent bugs (#39242) #39525
• Fixed the issue where routineload might import data repeatedly. [fix](routine load) should update progress before handle transaction state transform (#39311) #39526
• Fixed the data error caused by setNull when enabling group commit via JDBC. [fix](group commit) Fix prepare stmt setNull return too many filtered rows error (#38262) #38276
• Fixed the potential NPE issue when enabling group commit insert to a non-master FE. [cherry-pick](branch-2.1) Fix some group commit forward to master problems #38345
• Fixed incorrect error handling during internal data writing in group commit. [fix](group commit) fix repaly wal check label status (#38883) #38997
• Fixed the coredump that might be triggered when the group commit execution plan failed. [fix](group commit) fix group commit core if be inject FragmentMgr.exec_plan_fragment.failed (#39339) #39396
• Fixed the issue where concurrent imports into auto partition tables might report non-existent tablets. [branch-2.1](load) fix miss writer in concurrency incremental open (#38605) #38793
• Fixed potential load stream leakage issues. [fix](move-memtable) close stream when cancel load stream stub (#38912) #39039
• Fixed the issue where transactions were opened for insert into select with no data. [fix](planner) Empty table source should not begin a transaction to avoid infinite transaction (#38991) #39108
• Ignored the single-replica import configuration when using memtable prefetching. [fix](move-memtable) ignore single replica load when move memtable (#32845) #39154
• Fixed the issue where background imports of stream load records might be abnormally aborted upon encountering db deletion. [fix](stream load) do not throw exception but skip record when can not find database (#39360) #39527
• Fixed inaccurate error messages when data errors occurred in strict mode. [Fix](load) Fix the incorrect src value printed in the error log when strict mode is true #39447 #39587
• Fixed the issue where streamload did not return an error URL upon encountering erroneous data. [fix](load) fix no error url for stream load #38325 #38417
• Fixed the issue with the combined use of insert overwrite and auto partition. [branch-2.1](insert-overwrite) Fix insert overwrite auto detect transaction safe (#38103) #38442
• Fixed parsing errors when CSV encountered data where the line delimiter was enclosed by the enclosing character. [fix](csv reader) fix csv parser incorrect if enclosing line_delimiter (#38347) #38445

Data Exporting

• Fixed the issue where enabling the delete_existing_files property during export operations might result in duplicate deletion of exported data. ([Fix](Export) Export delete multiple times when specify the delete_existing_files property () #39304)

Permissions

• Fixed the incorrect requirement of ALTER TABLE permission when creating a materialized view. [fix](auth)fix create mv not need alter priv of table #38011
• Fixed the issue where the db was explicitly displayed as empty when showing routine load. [fix](auth)fix show routine load db is null #38365
• Fixed the incorrect requirement of CREATE permission on the original table when using CREATE TABLE LIKE. [fix](auth)fix create table like need create_priv of existed table #37879
• Fixed the issue where grant operations did not check if the object existed. [enhance](privilege)Grant check name #39597

Use case

No response

Related issues

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[w41ter]

In Apache Doris 2.0.15, we introduced a new parameter: max_backup_tablets_per_job, which is used to limit the maximum number of tablets involved in a backup, to avoid OOM issues caused by excessive tablets. See doris.apache.org/docs/2.0/admin-manual/config/fe-config?_highlight=fe#max_backup_tablets_per_job.

This might prevent some existing backup tasks from running. If you have such issues, please adjust the parameter's upper limit accordingly.