Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](nereids)remove literal partition by and order by expression in window function #26899

Merged
merged 3 commits into from
Nov 23, 2023

Conversation

starocean999
Copy link
Contributor

@starocean999 starocean999 commented Nov 13, 2023

Proposed changes

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@starocean999
Copy link
Contributor Author

run buildall

1 similar comment
@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.47 seconds
stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162279374 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 11c6ba0640b55147a2c8ba23d55ee54c564f5fc5, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5251	5114	5136	5114
q2	362	149	167	149
q3	2056	1987	2019	1987
q4	1412	1357	1361	1357
q5	3959	3957	4060	3957
q6	258	130	135	130
q7	1444	885	920	885
q8	2799	2791	2782	2782
q9	9692	9459	9516	9459
q10	3463	3519	3511	3511
q11	385	263	254	254
q12	446	293	285	285
q13	4592	4188	4105	4105
q14	328	286	294	286
q15	625	566	562	562
q16	679	590	592	590
q17	1149	1088	1063	1063
q18	8055	7644	7634	7634
q19	1671	1699	1699	1699
q20	572	307	315	307
q21	4730	4388	4411	4388
q22	506	406	389	389
Total cold run time: 54434 ms
Total hot run time: 50893 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5145	5072	5030	5030
q2	336	248	253	248
q3	4128	3984	3980	3980
q4	2811	2817	2770	2770
q5	9542	9572	9558	9558
q6	250	121	123	121
q7	2693	2280	2315	2280
q8	4883	4839	4849	4839
q9	13064	12888	12905	12888
q10	4060	4125	4154	4125
q11	780	665	663	663
q12	1017	846	815	815
q13	4324	3898	3892	3892
q14	390	364	359	359
q15	660	530	555	530
q16	801	712	711	711
q17	3899	3951	3924	3924
q18	9688	9402	9651	9402
q19	1872	1787	1788	1787
q20	2409	2080	2050	2050
q21	8783	8764	8709	8709
q22	955	871	845	845
Total cold run time: 82490 ms
Total hot run time: 79526 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.98 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162191075 Bytes

@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 5bbee7545c1185d79d37ae40da928c66f64428fb, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5234	5092	5152	5092
q2	356	158	158	158
q3	2054	2049	2040	2040
q4	1405	1364	1363	1363
q5	3957	3943	4029	3943
q6	254	135	130	130
q7	1471	886	892	886
q8	2778	2764	2755	2755
q9	9765	9522	9342	9342
q10	3444	3543	3509	3509
q11	389	245	254	245
q12	430	276	288	276
q13	4550	4126	4116	4116
q14	314	286	279	279
q15	613	546	556	546
q16	688	590	600	590
q17	1125	1092	1072	1072
q18	8107	7621	7634	7621
q19	1687	1687	1676	1676
q20	534	308	326	308
q21	4700	4329	4383	4329
q22	502	406	408	406
Total cold run time: 54357 ms
Total hot run time: 50682 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5082	5010	5054	5010
q2	329	239	246	239
q3	4145	4023	4042	4023
q4	2795	2779	2776	2776
q5	10059	9955	9994	9955
q6	243	123	127	123
q7	3032	2499	2481	2481
q8	4906	4850	4840	4840
q9	13102	12967	13014	12967
q10	4093	4167	4183	4167
q11	783	662	684	662
q12	980	843	819	819
q13	4308	3838	3849	3838
q14	377	344	374	344
q15	585	558	561	558
q16	743	691	696	691
q17	3900	3908	3892	3892
q18	9592	9278	9267	9267
q19	1866	1774	1765	1765
q20	2380	2046	2056	2046
q21	8972	8765	8687	8687
q22	957	851	841	841
Total cold run time: 83229 ms
Total hot run time: 79991 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.3 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17161214053 Bytes

@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit fcbbf8cf26ce7b6dd2a02b24177a8966b776f06a, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5303	5156	5183	5156
q2	363	151	160	151
q3	2027	2023	1999	1999
q4	1402	1360	1369	1360
q5	3952	3960	3998	3960
q6	254	130	129	129
q7	1471	888	899	888
q8	2775	2798	2764	2764
q9	9760	9783	9504	9504
q10	3473	3539	3506	3506
q11	390	254	248	248
q12	436	280	281	280
q13	4612	4161	4139	4139
q14	311	280	280	280
q15	623	565	573	565
q16	693	586	580	580
q17	1133	1098	1078	1078
q18	8078	7585	7565	7565
q19	1693	1682	1671	1671
q20	523	294	294	294
q21	4692	4373	4383	4373
q22	516	419	435	419
Total cold run time: 54480 ms
Total hot run time: 50909 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5051	5219	5057	5057
q2	353	227	251	227
q3	4210	4030	4063	4030
q4	2823	2838	2803	2803
q5	9344	9298	9313	9298
q6	248	126	124	124
q7	3128	2499	2527	2499
q8	4837	4874	4844	4844
q9	13342	13264	13176	13176
q10	4059	4145	4171	4145
q11	782	657	654	654
q12	1059	818	832	818
q13	4339	3903	3850	3850
q14	371	363	358	358
q15	608	564	578	564
q16	817	707	722	707
q17	3852	3945	3859	3859
q18	9815	9532	9544	9532
q19	1921	1771	1776	1771
q20	2423	2059	2095	2059
q21	9102	8869	8733	8733
q22	940	833	871	833
Total cold run time: 83424 ms
Total hot run time: 79941 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.1 seconds
stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17094440784 Bytes

@starocean999 starocean999 changed the title [fix](nereids)window function's partition by and order by clause can't be literal [fix](nereids)window function's order by clause can't be literal Nov 16, 2023
@@ -70,6 +72,17 @@ private void checkUnexpectedExpression(Plan plan) {
if (plan.getExpressions().stream().anyMatch(e -> e.anyMatch(SubqueryExpr.class::isInstance))) {
throw new AnalysisException("Subquery is not allowed in " + plan.getType());
}
if (plan instanceof LogicalWindow) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why check here? not in ExtractAndNormalizeWindowExpression?

.stream()
.anyMatch(expression -> ((WindowExpression) expression).getOrderKeys().stream()
.map(orderExpression -> orderExpression.getOrderKey().getExpr())
.anyMatch(Literal.class::isInstance))) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not push them down lick partition by key?

@starocean999 starocean999 changed the title [fix](nereids)window function's order by clause can't be literal [fix](nereids)remove literal partition by and order by expression in window function Nov 16, 2023
@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.69 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17098354995 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 3ba24d901c5964f5e09d5d0914e917e8d5b4f885, data reload: false

run tpch-sf100 query with default conf and session variables
q1	10241	4747	4664	4664
q2	524	143	136	136
q3	3241	1870	1894	1870
q4	6373	1149	1321	1149
q5	6497	3982	3964	3964
q6	254	132	128	128
q7	1411	867	899	867
q8	8711	2838	2804	2804
q9	10273	9795	9731	9731
q10	9886	3531	3500	3500
q11	404	250	232	232
q12	453	293	297	293
q13	21859	3816	3770	3770
q14	325	283	295	283
q15	598	538	524	524
q16	680	581	582	581
q17	1176	969	937	937
q18	7836	7325	7282	7282
q19	1722	1708	1683	1683
q20	522	316	310	310
q21	4383	4004	4005	4004
q22	472	382	370	370
Total cold run time: 97841 ms
Total hot run time: 49082 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4599	4564	4579	4564
q2	350	241	238	238
q3	4001	3999	3989	3989
q4	2699	2689	2673	2673
q5	9660	9443	9436	9436
q6	245	125	120	120
q7	2582	2281	2279	2279
q8	4432	4464	4417	4417
q9	13369	13163	13249	13163
q10	4094	4159	4124	4124
q11	762	655	652	652
q12	967	816	826	816
q13	4289	3607	3603	3603
q14	382	341	341	341
q15	570	522	518	518
q16	736	678	676	676
q17	3864	3865	3799	3799
q18	9543	8961	9013	8961
q19	1850	1785	1764	1764
q20	2406	2083	2062	2062
q21	8795	8562	8646	8562
q22	893	774	846	774
Total cold run time: 81088 ms
Total hot run time: 77531 ms

jackwener
jackwener previously approved these changes Nov 17, 2023
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 17, 2023
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 17, 2023
@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.33 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 27.8 seconds inserted 10000000 Rows, about 359K ops/s
storage size: 17096852728 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit c7db235784126cfdfa8846576c2f6d2214e22346, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4951	4693	4671	4671
q2	355	160	156	156
q3	2060	1885	1888	1885
q4	1398	1252	1250	1250
q5	3998	3940	4017	3940
q6	251	132	133	132
q7	1397	869	889	869
q8	2773	2795	2799	2795
q9	10101	10072	9776	9776
q10	3452	3528	3534	3528
q11	397	260	228	228
q12	446	296	300	296
q13	4557	3806	3799	3799
q14	317	285	283	283
q15	588	536	528	528
q16	670	584	588	584
q17	1139	967	946	946
q18	7902	7523	7546	7523
q19	1680	1692	1672	1672
q20	538	320	308	308
q21	4410	4006	3963	3963
q22	468	375	375	375
Total cold run time: 53848 ms
Total hot run time: 49507 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4601	4591	4595	4591
q2	351	228	254	228
q3	4045	4017	4005	4005
q4	2707	2699	2712	2699
q5	9811	9802	9934	9802
q6	243	126	126	126
q7	2618	2272	2246	2246
q8	4487	4483	4521	4483
q9	13247	13035	13158	13035
q10	4115	4203	4182	4182
q11	730	726	669	669
q12	989	811	813	811
q13	4293	3567	3501	3501
q14	389	357	355	355
q15	570	517	517	517
q16	726	662	670	662
q17	3859	3833	3899	3833
q18	9727	9069	9122	9069
q19	1819	1787	1791	1787
q20	2407	2046	2059	2046
q21	8696	8523	8376	8376
q22	914	761	798	761
Total cold run time: 81344 ms
Total hot run time: 77784 ms

@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit cf2f63bc8114a5edfa1d3ed030df0cae5d0f0b7f, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4857	4682	4630	4630
q2	368	158	164	158
q3	2033	1913	1855	1855
q4	1382	1259	1214	1214
q5	3918	3878	3982	3878
q6	255	128	126	126
q7	1415	882	884	882
q8	2755	2783	2768	2768
q9	9510	9492	9341	9341
q10	3423	3513	3506	3506
q11	380	258	247	247
q12	434	298	297	297
q13	4507	3836	3842	3836
q14	312	287	286	286
q15	588	532	514	514
q16	663	590	586	586
q17	1144	944	910	910
q18	7902	7544	7403	7403
q19	1647	1677	1652	1652
q20	537	293	295	293
q21	4436	3959	4003	3959
q22	474	378	370	370
Total cold run time: 52940 ms
Total hot run time: 48711 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4560	4561	4557	4557
q2	335	225	263	225
q3	4023	3993	3998	3993
q4	2705	2693	2706	2693
q5	9643	9647	9612	9612
q6	248	122	123	122
q7	3051	2503	2455	2455
q8	4438	4478	4461	4461
q9	12893	12784	12812	12784
q10	4081	4162	4153	4153
q11	748	638	663	638
q12	975	809	833	809
q13	4261	3560	3594	3560
q14	390	366	346	346
q15	572	510	516	510
q16	737	677	675	675
q17	3920	3861	3823	3823
q18	9567	9220	9121	9121
q19	1784	1744	1770	1744
q20	2399	2086	2055	2055
q21	8827	8669	8763	8669
q22	882	806	773	773
Total cold run time: 81039 ms
Total hot run time: 77778 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.17 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17098293494 Bytes

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 23, 2023
@starocean999 starocean999 merged commit 1555b11 into apache:master Nov 23, 2023
26 of 28 checks passed
eldenmoon added a commit that referenced this pull request Nov 23, 2023
* [keyword](decimalv2) Add DecimalV2 keyword #26283 (#26319)

* [fix](planner) Fix sample partition table #25912 (#26399)

In the past, two conditions needed to be met when sampling a partitioned table: 1. Data is evenly distributed between partitions; 2. Data is evenly distributed between buckets. Finally, the number of sampled rows in each partition and each bucket is the same.

Now, sampling will be proportional to the number of partitioned and bucketed rows.

* [fix](spark-load)fix-Unique-key-with-MOR-by-sparkload #26383 (#26414)

* [fix](nereids)fix bug of select mv in nereids #26235 (#26415)

* [improvement](show trash) Fix be restart slow when too many trash files #26147 (#26417)

* [fix](planner)should keep at least one slot materialized in agg node #26116 (#26419)

* [fix](multi-catalog)add the FAQ for Aliyun DLF and add the fs.xx.impl check #25594 (#26422)

* [coverage](pipeline) Remove unless code and add call method for coverage #25552 (#26423)

Remove unless code and add call method for coverage

* [Fix](statistics)Fix analyze min max sql syntax error. #26240 (#26443)

backport #26240

* [fix](auditlog) fix without lock in QueryStatisticsRecvr find  (#26441)

* [fix](invert index) Fix the timing error when opening the searcher #26401 (#26472)

* [fix](nereids)only enable colocate scan for one phase global parttion topn in some condition #26473 (#26481)

* [branch-2.0](cherry-pick) Add more indexed column reader be unit test #25652 (#26430)

* [enhancement](regression) fault injection for segcompaction test (#25709) (#26305)

1. generalized debug point facilities from docker suites for
   fault-injection/stubbing cases
2. add segcompaction fault-injection cases for demonstration
3. add -238 TOO_MANY_SEGMENTS fault-injection case for good

Co-authored-by: zhengyu <freeman.zhang1992@gmail.com>

* [fix](case) rm non-visiable charactor null in out file (#26540)

* [fix](load) fix merged row number miscounting because of race condition (#26516)

row numbers miscounting because of race condition, will cause load to
fail sometimes with warning 'the rows number written doesn't match'.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

* [test](regression) Add more regression test for FE (#26539)

* [test](coverage) Improve test coverage for runtime filter (#26314) (#26547)

* [fix](Nereids) RewriteCteChildren not work with cost based rewritter (#26326) (#26530)

we use a map to record rewrite cte children result to avoid rewrite
twice in cost based rewritter. However, we record cte outer and
inner in one map, and use null as outer result's key, use cte id as
inner result's key. This is wrong, because every anchor has an outer,
and we could only record one outer. So when we use the cache in cost
based rewritter, we get wrong outer plan from the cache. Then the error
will be thrown as below:

```
Caused by: java.lang.IllegalArgumentException: Stats for CTE: CTEId#1 not found
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143) ~[guava-32.1.2-jre.jar:?]
    at org.apache.doris.nereids.stats.StatsCalculator.visitLogicalCTEConsumer(StatsCalculator.java:1049) ~[classes/:?]
    at org.apache.doris.nereids.stats.StatsCalculator.visitLogicalCTEConsumer(StatsCalculator.java:147) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.logical.LogicalCTEConsumer.accept(LogicalCTEConsumer.java:111) ~[classes/:?]
    at org.apache.doris.nereids.stats.StatsCalculator.estimate(StatsCalculator.java:222) ~[classes/:?]
    at org.apache.doris.nereids.stats.StatsCalculator.estimate(StatsCalculator.java:200) ~[classes/:?]
    at org.apache.doris.nereids.jobs.cascades.DeriveStatsJob.execute(DeriveStatsJob.java:108) ~[classes/:?]
    at org.apache.doris.nereids.jobs.scheduler.SimpleJobScheduler.executeJobPool(SimpleJobScheduler.java:39) ~[classes/:?]
    at org.apache.doris.nereids.jobs.executor.Optimizer.execute(Optimizer.java:51) ~[classes/:?]
    at org.apache.doris.nereids.jobs.rewrite.CostBasedRewriteJob.getCost(CostBasedRewriteJob.java:98) ~[classes/:?]
    at org.apache.doris.nereids.jobs.rewrite.CostBasedRewriteJob.execute(CostBasedRewriteJob.java:64) ~[classes/:?]
    at org.apache.doris.nereids.jobs.executor.AbstractBatchJobExecutor.execute(AbstractBatchJobExecutor.java:119) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visit(RewriteCteChildren.java:72) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visit(RewriteCteChildren.java:56) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.visitor.PlanVisitor.visitLogicalSink(PlanVisitor.java:118) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.visitor.SinkVisitor.visitLogicalResultSink(SinkVisitor.java:72) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.logical.LogicalResultSink.accept(LogicalResultSink.java:58) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:86) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:56) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.logical.LogicalCTEAnchor.accept(LogicalCTEAnchor.java:60) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:86) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:56) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.logical.LogicalCTEAnchor.accept(LogicalCTEAnchor.java:60) ~[classes/:?]
    at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.rewriteRoot(RewriteCteChildren.java:67) ~[classes/:?]
    at org.apache.doris.nereids.jobs.rewrite.CustomRewriteJob.execute(CustomRewriteJob.java:58) ~[classes/:?]
    at org.apache.doris.nereids.jobs.executor.AbstractBatchJobExecutor.execute(AbstractBatchJobExecutor.java:119) ~[classes/:?]
    at org.apache.doris.nereids.NereidsPlanner.rewrite(NereidsPlanner.java:275) ~[classes/:?]
    at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:218) ~[classes/:?]
    at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:118) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.commands.ExplainCommand.run(ExplainCommand.java:81) ~[classes/:?]
    at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:550) ~[classes/:?]
```

* [fix](Nereids) could not run query with repeat node in cte (#26330) (#26531)

pick from master
PR: #26330 
commit id: a89477e

ExpressionDeepCopier not process VirtualReference, so we generate inline
plan with mistake.

* [opt](Nereids) remove Nondeterministic trait from date related functions (#26444) (#26568)

* change version to 2.0.3-rc03dev

* [fix](regression-test)  Fix regiressin test syncer suit use master fe directly (#26456) (#26583)

Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>

* Revert "[improvement](scanner_schedule) reduce memory consumption of scanner #24199 (#25547)" (#26613)

This reverts commit 9a19581 to investigate ANALYZE TABLE WITH SYNC problem

* [enhancement](Nereids): add LOG info to show the phase of NereidsPlanner. (#26542)

* [opt](regression test) Add string-like column order by test #26379 (#26533)

* [Feature](auditloader) Plugin auditloader use auth token to avoid using cleartext passwords in config (#26278) (#26532)

Doris FE will check if stream load http request has auth token after checking password failed;
Plugin audit-log loader can use auth token if plugin config set use_auth_token to true

Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com>

* [branch-2.0](JdbcCatalog) fix that the predicate column name does not have back quote when querying the JDBC appearance (#26479) (#26560)

master pr: #26479

* [fix](prepare statement) Not supported such prepared statement if prepare a forward master sql (#26512) (#26638)

* [Pick-2.0](regression) add failure injection in inverted index writer #26121 (#26376)

* [fix](regression) fix regression framework bug: if real test result is negative, it will miss check test result #25734 (#25734) (#26551)

* [Branch-2.0](regression-test) Add tvf regression tests #26322 #26455 (#26566)

* [fix](BE)Branch-2.0 unknown runtime filter when get filter from _consumer_map (#26570)

* [regression-test](framework) support Non concurrent mode #26487  (#26574)

* [regression-test](fix) fix case bug #26561  (#26578)

* [fix](backup) Add repo id to local meta/info files to avoid overwriting #26536 (#26622)

The local meta/info files generated during backup are not distinguished
by repo names. If two backup jobs with the same name are submitted to
different repos at the same time, meta/info may be overwritten by another
backup job.

* [cases](regression-test) Add backup & restore test case #26490 #26491 (#26623)

* [case](regression) Adapt show create table and views to 2.0 (#26624)

* [fix](regression-test) add more check to address flaky test_partial_update_with_delete_stmt #26474 (#26628)

* [feature](Nereids): push down topN through join #24720 (#26634)

Push TopN through Join.

JoinType just can be left/right outer join or cross join, because data of their one child can't be filtered.

new TopN is (original limit + original offset, 0) as limit and offset.

(cherry picked from commit 3c9ff7a)

* [Test](statistics) Add test cases for external table statistics #26511 (#26636)

1. Test for close and open auto collection for external catalog.
2. Test for analyze table table_name (column) and whole table.

* [fix](runtime filter) append late arrival runtime filters in vfilecanner #25996 (#26640)

`VFileScanner` will try to append late arrival runtime filters in each loop of `ScannerScheduler::_scanner_scan`.  However, `VFileScanner::_get_next_reader` only generates the `_push_down_conjuncts` in the first loop, so the late arrival runtime filters are ignored.

* [fix](information_schema)fix bug that metadata_name_ids error tableid and append information_schema case #26238 (#26646)

fix bug that  #24059 .
Added some information_schema scanner tests.
files
schema_privileges
table_privileges
partitions
rowsets
statistics
table_constraints

Based on infodb_support_ext_catalog=false, it currently includes tests for all tables under the information_schema database.

* [Improve](map)Map impli cast #26126 (#26654)

* [chore](regression) Do stale resource reclaim before executing cold heat separation p2 case #26596 (#26660)

* fix shrink in topN for complext type #26609 (#26661)

* [fix](planner) Fix decimal precision and scale wrong when create table like #25802 (#26666)

Use field datatype such as decimal(10, 0) to create table like. Because the scale is 0, the precision and scale will lost when create table like done. this will fix the bug.

**Before fix, create table with following SQL**:
CREATE TABLE IF NOT EXISTS db_test.table_test
(
    `name` varchar COMMENT "1m size",
    `id` SMALLINT COMMENT "[-32768, 32767]",
    `timestamp0` decimal null comment "c0",
    `timestamp1` decimal(38, 0) null comment "c1"
)
DISTRIBUTED BY HASH(`id`) BUCKETS 1
PROPERTIES ('replication_num' = '1');

**and Then run**
CREATE TABLE db_test.table_test_like LIKE db_test.table_test
SHOW CREATE TABLE db_test.table_test_like;

the field `timestamp1` will be decimal(9, 0), it's wrong. this will fix it.

Co-authored-by: JingDas <114388747+JingDas@users.noreply.github.com>

* [fix](test) fix sql block rule test (#26671)

* [Coverage](BE) Delete vinfo_func in BE #26562 (#26674)

* [Fix](partial update) Fix core when successfully schema change and load during a partial update #26210 (#26518)

* [typo] copy branch master docs to branch-2.0 (#26703)

* [typo] update sql-functions to upper-case (#26706)

* [Bug](cherry-pick) Add status dispose in branch 2.0 beta rowset reader (#26684)

* (selectdb-cloud) Reduce FE db lock range for ShowDataStmt #26588 (#26621)

Reduce read lock critical sections and avoid execution timeouts

* [brach-2.0](pick)use 2 phase agg above union all #26245 (#26664)

* [bug](bitmap) fix bitmap value copy operator not call reset #26451 (#26681)

when a empty bitmap assign to other bitmap
the other bitmap should reset self firstly, and then set empty type.

* [fix](planner)isnull predicate can't be safely constant folded in inlineview #25377 (#26685)

* [fix](nereids)unnest in-subquery with agg node in proper condition #25800 (#26687)

* [fix](nereids)add visitMarkJoinReference method in ExpressionDeepCopier #25874 (#26688)

* [fix](nereids)don't normalize column name for base index #26476 (#26690)

* [fix](planner)cast floating point type to bigint for bit functions #26598 (#26691)

* [fix](Nereids) storage later agg rule process agg children by mistake #26101 (#26698)

pick from master
PR #26101
commit id c0ed5f7

update Project#findProject
agg function's children could be any expression rather than only slot.
we use Project#findProject to process them. But this util could only
process slot. This PR update this util to let it could process all type
expression.

* [fix](Nereids) time extract function constant folding core (#26292) (#26699)

pick from master
PR: #26292
commit id: 74fd5da

some time extract function changed return type in the previous PR #18369
but it is not change FE constant folding function signature.
This is let them have same signature to avoid BE core.

* [fix](Nereids) only search internal funcftion when dbName is empty (#26296) (#26700)

pick from master
PR: #26296
commit id: 6892fc9

if call function with database name. we should only search UDF

* [fix](Nereids) ban right outer, right anti, full outer with bucket shuffle (#26529) (#26702)

pick from master
PR: #26529
commit id: f80495d

if left bucket has no data, we do not generate left bucket instance.
These join should reserve all right side data. But because left instance
is not exists. So right data will be discard since no dest be set.

We ban these join temporarily until we could generate all instance
for left side in Coordinator.

* [test](statistics)Add hive statistics all data type p0 test (#26676) (#26715)

* [test](serialisation) Serialise some cases and enable str_to_date tests #26651 (#26716)

1 enable the cases about str_to_date, which have been muted because some parallel config influence.
2 serialise some cases which called admin set config

* Revert "[Coverage](BE) Delete vinfo_func in BE #26562 (#26674)" (#26724)

This reverts commit 22eafa4.

* [fix](regression-test) add tests for jdbc catalog (#26608) (#26719)

* [fix](nereids)SimplifyRange rule may mess up and/or predicate #26304 (#26693)

* [Fix](fs_benchmark_tools) Fix `run_fs_benchmark.sh` classpath issue. (#26183) (#26704)

Backport from #26183.

* [Fix](partial update) Fix core when doing partial update on tables with row column after schema change #26632 (#26695)

* [Opt](orc-reader) Optimize orc string dict filter in not_single_conjunct case. (#26386) (#26696)

Optimize orc/parquet string dict filter in not_single_conjunct case. We can optimize this processing to filter block firstly by dict code, then filter by not_single_conjunct. Because dict code is int, it will filter faster than string.

For example:
```
select count(l_receiptdate) from lineitem_date_as_string where l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate  and l_receiptdate >= '1994-01-01' and l_receiptdate < '1995-01-01';
```
 `l_receiptdate` and `l_shipmode` will using string dict filtering, and `l_commitdate < l_receiptdate` is the an not_single_conjunct which contains dict filter field. We can optimize this processing to filter block firstly by dict code, then filter by not_single_conjunct. Because dict code is int, it will filter faster than string.

Before:
 mysql> select count(l_receiptdate) from lineitem_date_as_string where l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate  and l_receiptdate >= '1994-01-01' and l_receiptdate < '1995-01-01';
+----------------------+
| count(l_receiptdate) |
+----------------------+
|             49314694 |
+----------------------+
1 row in set (6.87 sec)

After:
mysql> select count(l_receiptdate) from lineitem_date_as_string where l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate  and l_receiptdate >= '1994-01-01' and l_receiptdate < '1995-01-01';
+----------------------+
| count(l_receiptdate) |
+----------------------+
|             49314694 |
+----------------------+
1 row in set (4.85 sec)

* [docs](docs) Update Files of Branch-2.0 (#26737)

* [date](parser) Support DateV1 keyword (#25414) (#26746)

* [Fix](orc-reader) Fix orc complex types when late materialization was turned on by disabling late materialization in this case. (#26548) (#26743)

Fix orc complex types when late materialization was turned on in orc reader by disabling late materialization in this case.

* [fix](udf)java udf does not support overloaded evaluate method (#22681) (#26768)

Co-authored-by: HB <hubiao01@corp.netease.com>

* [fix](show_proc) fix show statistic proc dir to ensure that result only contains dbs in internal catalog (#26254) (#26763)

backport #26254
Co-authored-by: caiconghui <55968745+caiconghui@users.noreply.github.com>

* [Enhancement](sql-cache) Use update time of hive to avoid cache miss through multi fe nodes. (#26424) (#26762)

backport #26424

* [Fix](partial update) Fix partial update info loss when the delete bitmaps of the committed transactions are calculated by the compaction #26556 (#26735)

* [hotfix](editlog) Fix upsert replay on follower not contains loadedTableIndexIds (#26597) (#26756)

* [chore](regression-test) Fix error add partition operation due to duplicate partition range #26742 (#26758)

* [Bug](materialized-view) fix some bugs on create mv with percentile_approx (#26528) (#26764)

1. percentile_approx have wrong symbol
2. fnCall.getParams() get obsolete childrens

* [Bug](agg-state) fix file load insert wrong data to agg_state (#26581) (#26765)

* [Bug](decimalv2)  getCmpType return decimalv2 when lhs/rhs type both is decimalv2  (#26705) (#26767)

* [fix](Nereids) fix plan shape of query64 unstable  (#26012) (#26775)

don't remove the physical plan after optimizing the plan in dphyper.

* [FIX](complextype) fxi array nested struct literal #26270 (#26778)

* [improvement](disk balance) Prevent duplicate disk balance tasks afte… (#25990) (#26745)

* [branch-2.0](transaction) Fix publish txn wait too long when not meet quorum #26659 (#26759)

* [bugfix](clickhouse) fix datetime convert error. (#26128) (#26766)

Co-authored-by: Guangdong Liu <liugddx@gmail.com>

* [Fix](row store) cache invalidate key should not include sequence column #26771 (#26780)

* [branch-2.0](pick) support HTTP request with chunked transfer (#26520) (#26785)

* [feature](nestedType) add nested data type to create table tool (#26787)

* [fix](hudi) fix wrong schema when query hudi table on obs #26789 (#26791)

* [fix](decimal) fix undefined behaviour of divide by zero when cast string to decimal (#26792)

* [fix](refresh) fix priv issue of refresh database and table operation #26793 (#26794)

* [minor] add disable swap command tip (#26798)

* [fix](information_schema) fix test_query_sys_tables schema_privileges  regression case #26753 (#26800)

* [branch-2.0] fix test result (#26801)

fix output error from #26743
On master branch, the value in struct field is wrapped by quota,
but on branch 2.0, the value in struct field is NOT wrapped by quota

* fix: restore load job progress before retry load task (#26802)

Co-authored-by: chenboyang.922 <chenboyang.922@bytedance.com>

* [fix](thrift)limit be and fe thrift server max pkg size,avoid accepting error or too large package causing OOM #26179 (#26805)

* [fix](Planner): don't push down isNull predicate into view (#26288) (#26773)

* [opt](scanner) increase the connection num of s3 client #26795 (#26796)

* [enhancement](metrics)  enhance visibility of flush thread pool (#26544) (#26819)

* [fix](regression) move fault-injection data to the right place (#26825)

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

* [feature](binlog) Add ingest_binlog/http_get_snapshot limit download speed && Add async ingest_binlog (#26323) (#26733)

* [fix](jdbc catalog) fix mysql zero date (#26569) (#26837)

* [ci](pipeline) add tpch sff100 test on branch-2.0 (#26824)

* [pick](nerieds) make AGG_SCALAR_SUBQUERY_TO_WINDOW_FUNCTION rewrite rule #25969 (#26852)

* [enhancement](230) print max version and spec version when -230 happens (#26643) (#26854)

* [chore](fs) Don't print the stack for file system and it's derived class #26814 (#26838)

* [compile](gcc) fix gcc compile error #26863

* [test](jdbc) pick some jdbc test from branch master (#26860)

* [pipeline](exec) disable shared scan in default and disable shared scan in limit with where scan (#25952) (#26815)

* [regression](partial update) Add cases when the deleted rows have non nullable columns without default value #26776 (#26848)

* [feature](fe) Add coverage tool for FE UT (#26203) (#26857)

* [fix](map) the implementation of ColumnMap::replicate was incorrect (#26647) (#26868)

* [fix](broker load) pass loadToSingleTablet to olapTableSink (#26680) (#26869)

* [regression-test](framework) Support running tests multiple times and reporting correctly to TeamCity (#26606) (#26871)

* [refactor](stats) refactor collection logic and opt some config #26163 (#26858)

picked from #26163

* [bug](user login)fix PASSWORD_LOCK_TIME setting UNBOUNDED does not take effect #26585 (#26859)

* [Improvement](statistics)Improve stats sample strategy (#26435) (#26890)

backport #26435
Improve the accuracy of sample stats collection. For non distribution columns, use
`n*d / (n - f1 + f1*n/N)`

where `f1` is the number of distinct values that occurred exactly once in our sample of n rows (from a total of N),
and `d` is the total number of distinct values in the sample.

For distribution columns, use `ndv(n) * fraction of tablets sampled` for NDV.

For very large tablet to sample, use limit to control the total lines to scan (for non key column only, because key column is sorted and will be inaccurate using limit).

* [fix](partial update) Fix NPE when the query statement of an update statement is a point query in OriginPlanner #26881 (#26900)

* [bug](function) add signature for precentile function (#26867) (#26926)

Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com>

* enable pipeline and nereids in test-pipeline (#26918)

* [Fix](Planner) fix varchar does not show real length #25171 (#26850)

* [improvement](statistics)Multi bucket columns using DUJ1 to collect ndv #26950 (#26976)

backport #26950

* [fix](statistics)Fix external table show column stats type bug #26910  (#26921)

backport: #26910

* [minor](stats) rename stats related session variable name #26936 (#26928)

* [nereids](datetime) fix wrong result type of datetime add with interval as first arg (#26957) (#26987)

* [fix](Nereids) column pruning under union broken unexpectedly (#26884) (#26985)

* [fix](catalog) Fix ClickHouse DataTime64 precision parsing (#26980)

* [opt](MergeIO) use equivalent merge size to measure merge effectiveness (#26741) (#26923)

backport #26741

* add defensive code in runtime predicate to avoid crash due to column not in tablet schema #26990 (#26991)

* [fix](stats) fix auto collector always create sample job no matter the table size #26968 (#26972)

* [Enhance](regression)enhance docker network by add docker network subnet (#26872)

* [fix](case) regression-test/suites/show_p0/test_show_statistic_proc.groovy (#26925)

Co-authored-by: stephen <hello-stephen@qq.com>

* [fix](auth) fix overwrite logic of user with domain (#27003)

backport #27002

* [Branch-2.0](Serde) Fix content displayed by complex types in MySQL Client (#26880)

backport #25946 and #26301

* [test](tvf) append tvf read hive_text file  regression case. (#26790) (#26989)

backport #26790

* [test](information_schema)append information_schema external_table_p0 case. (#27029)

backport : #26846

* [fix](parquet) compressed_page_size has the same meaning in page v1 and v2 (#26783) (#26922)

backport #26783

* [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes (#26933) (#27018)

* [Enhance](regression) skip test_information_schema_external (#27058)

* [improvement](pipeline) task group scan entity (#19924) (#27040)

Co-authored-by: Lijia Liu <liutang123@yeah.net>

* [opt](pipeline) Return InternalError to FE instead of doing a useless DCHECK in ExecNode #27035 (#27057)

Effect: Client will see error message like below when BE meeting plan logical error.

RROR 1105 (HY000): errCode = 2, detailMessage = ([xxx]())[CANCELLED]Logical error during processing VNewOlapScanNode(dr_case_tag), output of projections 2 mismatches with exec node output 3

* [fix](nereids)Fix nereids fail to parse tablesample bug (#26982)

backport #26981

* [branch2.0](test) fix external table test case with nested type display (#27092)

* [fix](load) skip cancel already cancelled channels (#27109)

* [fix](Nereids) store user variable in connect context (#26655) (#26920)

pick from master #26655

1. user variable should be case insensitive
2. user variable should be cleared after the connection reset

* [test](parquet)append parquet reader byte_array_decimal and rle_bool case (#26751) (#27026)

backport #26751

* [fix](nereids) support uncorrelated subquery in join condition (#26893)

pick from master #26672 
commit id: 17b1108

* [Bug](pipeline) try fix the exchange sink buffer result error (#27087)

* [fix](function)return NULL rather than 'null' if path not found #25880 (#26823)

* [enhancement](nereids)make error message more readable when bind logicalRepeat node #26744 (#26895)

* [regression](delete) add delete case for every type (#26961)

* [branch-2.0](paimon)disable paimon decimal case (#26971)

* [regression](partial update) Add row store cases for all existing partial update cases #26924 (#27017)

* [fix](statistics) fix updated rows incorrect due to typo in code #26979 (#27034)

* [fix](typo) Use minutes as auto analyze schedule interval #26968 (#27041)

* [Improvement](function) opt for case when #23068 (#27054)

* [fix](planner)scan node should project all required expr from parent node #26886 (#27096)

* [fix](nereids)count in correlated subquery shoud not output null value #27064 (#27097)

* [fix](load) add lock in active_memtable_mem_consumption #25207 (#27100)

* [branch-2.0](suites) Enable test_cast_with_scale_type since Nereids is ON (#26986)

* [Fix](multi-catalog) Fix NPE when replaying hms events #26803 (#26997)

Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com>

* [Opt](scanner-scheduler) Optimize `BlockingQueue`, `BlockingPriorityQueue` and change remote scan thread pool #26784 (#27053)

- Optimize `BlockingQueue`, `BlockingPriorityQueue` by swapping `notify` and `unlock` to reduce lock competition. Ref: https://www.boost.org/doc/libs/1_54_0/boost/thread/sync_bounded_queue.hpp
- Change remote scan thread pool to `PriorityQueue`.

* [fix](errmsg) fix multiple FE processes start err msg (#27009) (#27080)

* [FIX](regresstest) fix test_load_with_map_nested_array csv for id #27105 (#27107)

* [FIX](map)fix map nested decimal with element at #27030 (#27110)

* [feature](tvf)(jni-avro)jni-avro scanner add complex data types (#26236) (#26731)

* [fix](nereids)fix bug that query infomation_schema.rowsets fe send fragment to one of muilti be. (#27025) (#27090)

Fixed the bug of incomplete query results when querying information_schema.rowsets in the case of multiple BEs.

The reason is that the schema scanner sends the scan fragment to one of multiple bes, and be queries the information of fe through rpc. Since the rowsets information requires information about all BEs, the scan fragment needs to be sent to all BEs.

* [Config](statistics)Set enable_auto_analyze default value to true. #27146

* [branch2.0](test) fix doris jdbc catalog test case (#27150)

1. Fix doris_jdbc_catalog test case out file
2. Add log to debug 2 unstable test cases: pg_jdbc_catalog and oracle_jdbc_catalog

* [bugfix](tablet)fix the tablet will be deleted when clone due to concurrency #25784 (#26777)

* [fix](sink) crash caused by wild pointer of counter in VDataStreamSender (#26947) (#27148)

If preparation fails, the counter _peak_memory_usage_counter will be a wild pointer.

*** SIGSEGV address not mapped to object (@0x454d49545f) received by PID 16992 (TID 18856 OR 0x7f4d05444700) from PID 1296651359; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:417
 1# os::Linux::chained_handler(int, siginfo*, void*) in /app/doris/Nexchip-doris-1.2.4.2-bin-x86_64/java8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /app/doris/Nexchip-doris-1.2.4.2-bin-x86_64/java8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /app/doris/Nexchip-doris-1.2.4.2-bin-x86_64/java8/jre/lib/amd64/server/libjvm.so
 4# 0x00007F55C85B9400 in /lib64/libc.so.6
 5# doris::vectorized::VDataStreamSender::close(doris::RuntimeState*, doris::Status) at /root/doris/be/src/vec/sink/vdata_stream_sender.cpp:734
 6# doris::PlanFragmentExecutor::close() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:543
 7# doris::PlanFragmentExecutor::~PlanFragmentExecutor() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:95
 8# doris::FragmentExecState::~FragmentExecState() at /root/doris/be/src/runtime/fragment_mgr.cpp:112
 9# std::_Sp_counted_ptr<doris::FragmentExecState*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at /root/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:348
10# doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&, std::function<void (doris::RuntimeState*, doris::Status*)> const&) at /root/doris/be/src/runtime/fragment_mgr.cpp:855
11# doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&) at /root/doris/be/src/runtime/fragment_mgr.cpp:592
12# doris::PInternalServiceImpl::_exec_plan_fragment_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, doris::PFragmentRequestVersion, bool) at /root/doris/be/src/service/internal_service.cpp:463
13# doris::PInternalServiceImpl::_exec_plan_fragment_in_pthread(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*) at /root/doris/be/src/service/internal_service.cpp:305
14# doris::WorkThreadPool<false>::work_thread(int) at /root/doris/be/src/util/work_thread_pool.hpp:160
15# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
16# start_thread in /lib64/libpthread.so.0
17# clone in /lib64/libc.so.6

* [improvement](log) log desensitization without displaying user info (#26912) (#27167)

* [branch-2.0](cherry-pick)  add chunked transfer json test (#26902) (#27164)

* [fix](statistics)Fix alter column stats bug (#27093) (#27189)

backport #27093

* (fix)[schema change] fix incorrect setting of schema change jobstate when replay editlog (#26992) (#27139)

* [fix](jni) avoid BE crash and NPE when close paimon reader #27129 (#27204)

bp #27129

* [enhancement](jdbc catalog) Add lowercase column name mapping to Jdbc data source & optimize database and table mapping #27124 (#27130)

* [case] Load json data with enable_simdjson_reader=false (#26601) (#27158)

Co-authored-by: HowardQin <hao.qin@esgyn.cn>

* [fix](function) fix error when use negative number in explode_numbers #27020 (#27180)

* [fix](iceberg) iceberg use customer method to encode special characters of field name (#27108) (#27205)

Fix two bugs:
1. Missing column is case sensitive, change the column name to lower case in FE for hive/iceberg/hudi
2. Iceberg use custom method to encode special characters in column name. Decode the column name to match the right column in parquet reader.

* [enhancement](binlog)  Add dbName && tableName in CreateTableRecord (#26901) (#27208)

* [Branch2.0](Export) add show export regression testes #27140 (#27160)

* [log](tablet invert)  add preconditition check failed log (#26770) (#27171)

* [branch-2.0](publish version) publish version task no need return VERSION_NOT_EXIST #27005 (#27174)

* [minor](stats) Add start/end time for analyze job, precise to seconds of TableStats update time #27123 (#27185)

* [test](regression) Add more alter stmt regression case (#26988) (#27193)

* [test](external_table_p0)append log in external_table_p0 for debug unknown table case #27212 (#27213)

* [Improve](txn) Add some fuzzy test stub in txn (#26712) (#27144)

* [branch-2.0](fe ut) fix decommission test #27082 (#27175)

* [Fix](multi-catalog) Fix complex type crash when using dict filter facility in the parquet-reader. (#27151) (#27187)

- Fix complex type crash when using the dict filter facility in the parquet-reader by turning off the dict filter facility in this case.
- Add orc complex types regression test.

* [Optimize](point query) clear names to reduce mem consumption and cpu cost related to block column name (#26931) (#27157)

* [fix](fe) Fix `enable_nereids_planner` forward not take effect (#26782) (#27159)

* The java reflection method `getFields()` only return public fields,
  but enable_nereids_planner is private

* [fix](fe ut) Fix borrow oject throw npe (#27072) (#27207)

occasional failure of fe ut, borrowObject throw npe
```
get agent task request. type: CREATE, signature: 10008, fe addr: null
java.lang.NullPointerException
	at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
	at org.apache.commons.pool2.impl.GenericKeyedObjectPool.register(GenericKeyedObjectPool.java:1079)
	at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:347)
get agent task request. type: CREATE, signature: 10012, fe addr: TNetworkAddress(hostname:127.0.0.1, port:56072)
	at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277)
	at org.apache.doris.common.GenericPool.borrowObject(GenericPool.java:99)
	at org.apache.doris.utframe.MockedBackendFactory$DefaultBeThriftServiceImpl$1.run(MockedBackendFactory.java:219)
	at java.lang.Thread.run(Thread.java:750)
```

* [regression](conf) Make checkpoint/clean thread trigger more frequent (#26883) (#27194)

* When run p0, we want some checkpoint/clean thread in FE work more
  frequently

* [hotfix](priv) Fix restore snapshot user priv with add cluster in UserIdentity (#26969) (#27210)

Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>

* [branch-2.0](fe ut) fix unstable test DecommissionBackendTest  (#27173)

* [fix](disk migrate) migrate ignore not exists tablet (#26779) (#27172)

* [fix](build)macos clang 15 version compilation error (#25457)

* [fix](tablet sched) fix sched delete stale remain replica (#27050) (#27179)

* Revert "[Branch2.0](Export) add show export regression testes #27140 (#27160)" (#27217)

This reverts commit d76581d, since it caused test_show_export testcase fail.

* Revert "[test](regression) Add more alter stmt regression case (#26988) (#27193)" (#27216)

This reverts commit 42d4806, since it caused test_alter_table_drop_column and test_alter_table_modify_column testcases fail.

* [fix](nereids)remove literal partition by and order by expression in window function #26899 (#27214)

* [fix](agg) fix coredump of multi distinct of decimal128I (#27014) (#27228)

* [fix](agg) fix coredump of multi distinct of decimal128

* fix

* Revert "[enhancement](jdbc catalog) Add lowercase column name mapping to Jdbc data source & optimize database and table mapping #27124 (#27130)" (#27230)

This reverts commit 087fccd.

* [feature](Nereids): eliminate sort under subquery (#26993) (#27218)

* [fix](ccr) Mark getBinlog,getBinlogLag,getMeta,getBackendMeta as from master (#27211) (#27227)

Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>

* [fix](Nereids): NullSafeEqual should be in HashJoinCondition #27127 (#27232)

* [fix](planner)the data type should be the same between input slot and sort slot #27137 (#27215)

* [branch2.0](nereids)Pick #26873 #25769: partition prune fix (#27222)

* [improvement](fe and broker) support specify broker to getSplits, check isSplitable, file scan for HMS Multi-catalog (#24830) (#27236)

bp #24830

* [fix](fe ut) fix unstable ut TabletRepairAndBalanceTest (#27044) (#27239)

* [minor](stats) Report error with more friendly meesage when timeout #27197 (#27240)

* [fix](build index) Fix inverted index hardlink leak and missing problem #26903 (#27244)

* [fix](multi-catalog)add the max compute fe ut and fix download expired  #27007 (#27220)

bp #27007

* [cherry-pick](regression) add hms catalog broker scan case (#25453) (#27253)

* [cherry-pick](fe) select BE local broker to scan Hive table when 'broker.name' in hms catalog is specified (#27122) (#27252)

Since #24830 introduce `broker.name` in hms catalog, data scan will run on specified brokers.
And [doris operator](https://github.com/selectdb/doris-operator) support BE and broker deployed in same pod, BE access local broker is the fastest approach to access data.
In previous logic, every inputSplit will select one BE to execute,  then randomly select one broker for actual data access, BE and related broker are always located on  separate K8S pod.
This pr optimizes the broker select strategy to prioritize BE-local broker when `broker.name` is specified in hms catalog.

* [improvement](statistics)Use count as ndv for unique/agg olap table single key column (#27186) (#27275)

Single key column of unique/agg olap table has the same value of count and ndv, for this kind of column,
don't need to calculate ndv, simply use count as ndv.
backport #27186

* [minor](stats) Fix potential npe when loading stats #27200 (#27241)

* [fix](tablesample) Fix computeSampleTabletIds NullPointerException (#27165) (#27258)

* [fix](partial update) keep case insensitivity and use the columns' origin names in partialUpdateCols in origin planner #27223 (#27255)

* [chore](fix) sync check-pr-if-need-run-build.sh with master branch (#27250)

* [fix](compile) fix BE compile failure on Mac (#27206) (#27281)

* [chore](clucene) coverage compilation option added #27162 (#27284)

* [FIX]Fix complex type meta schema in information database  #27203 (#27286)

* [feature](Nereids): Pushdown LimitDistinct Through Join (#25113) (#27288)

Push down limit-distinct through left/right outer join or cross join.
such as select t1.c1 from t1 left join t2 on t1.c1 = t2.c1 order by t1.c1 limit 1;

* [fix](inverted index) reset fs_writer to nullptr before throw exception (#27202) (#27289)

* [fix](planner)output slot should be materialized as intermediate slot in agg node #27282 (#27285)

* [FIX](complextype)Fix complex nested and add regress test #26973 (#27293)

* [fix](test) disable forbid_unknown_col_stats (#27303)

* [fix](stats) Release analyze tasks once job finished #27310 (#27309)

* [doc](fix) a new docs for k8s deploy by operator to 2.0 (#26927)

* [doc](fix) fix date trunc doc (#27320)

* [Fix](statistics)Fix analyze sql including key word bug  #27321 (#27322)

backport #27321

* [cherry-pick](function) improve compoundPred optimization work with children is nullable #26160 (#27354)

* Revert "[improvement](routine-load) add routine load rows check (#25818)" (#27336)

* [refactor](planner) filter empty partitions in a unified location (#27190) (#27256)

* [fix](hms) fix compatibility issue of hive metastore client #27327 (#27328)

* [Branch2.0](Export) add show export regression testes (#27330)

* [fix](stats) Fix thread leaks when doing checkpoint #27334 #27335

* [fix](stats) Fix creating too many tasks on new env (#27362)

* [fix](build index) fix core when build index for a new column which without data (#27276)

* change version to 2.0.3-rc04 (#27392)

* fix merge

* update clucene

---------

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com>
Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com>
Co-authored-by: deardeng <565620795@qq.com>
Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
Co-authored-by: HappenLee <happenlee@hotmail.com>
Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com>
Co-authored-by: zzzxl <33418555+zzzxl1993@users.noreply.github.com>
Co-authored-by: abmdocrt <Yukang.Lian2022@gmail.com>
Co-authored-by: HHoflittlefish777 <77738092+HHoflittlefish777@users.noreply.github.com>
Co-authored-by: zhengyu <freeman.zhang1992@gmail.com>
Co-authored-by: Dongyang Li <hello_stephen@qq.com>
Co-authored-by: walter <w41ter.l@gmail.com>
Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
Co-authored-by: Kang <kxiao.tiger@gmail.com>
Co-authored-by: Jack Drogon <jack.xsuperman@gmail.com>
Co-authored-by: jakevin <jakevingoo@gmail.com>
Co-authored-by: zhiqiang <seuhezhiqiang@163.com>
Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com>
Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
Co-authored-by: meiyi <myimeiyi@gmail.com>
Co-authored-by: airborne12 <airborne08@gmail.com>
Co-authored-by: TengJianPing <18241664+jacktengg@users.noreply.github.com>
Co-authored-by: minghong <englefly@gmail.com>
Co-authored-by: shuke <37901441+shuke987@users.noreply.github.com>
Co-authored-by: walter <patricknicholas@foxmail.com>
Co-authored-by: zhannngchen <48427519+zhannngchen@users.noreply.github.com>
Co-authored-by: Ashin Gau <AshinGau@users.noreply.github.com>
Co-authored-by: daidai <2017501503@qq.com>
Co-authored-by: amory <wangqiannan@selectdb.com>
Co-authored-by: AlexYue <yj976240184@gmail.com>
Co-authored-by: seawinde <149132972+seawinde@users.noreply.github.com>
Co-authored-by: JingDas <114388747+JingDas@users.noreply.github.com>
Co-authored-by: zclllyybb <zhaochangle@selectdb.com>
Co-authored-by: bobhan1 <bh2444151092@outlook.com>
Co-authored-by: Jeffrey <color.dove@gmail.com>
Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com>
Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
Co-authored-by: KassieZ <139741991+KassieZ@users.noreply.github.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
Co-authored-by: HB <hubiao01@corp.netease.com>
Co-authored-by: Pxl <pxl290@qq.com>
Co-authored-by: 谢健 <jianxie0@gmail.com>
Co-authored-by: yujun <yu.jun.reach@gmail.com>
Co-authored-by: Guangdong Liu <liugddx@gmail.com>
Co-authored-by: zfr95 <87513668+zfr9527@users.noreply.github.com>
Co-authored-by: chen <czjourney@163.com>
Co-authored-by: TsukiokaKogane <cby141994@gmail.com>
Co-authored-by: chenboyang.922 <chenboyang.922@bytedance.com>
Co-authored-by: ryanzryu <143597717+ryanzryu@users.noreply.github.com>
Co-authored-by: Siyang Tang <82279870+TangSiyang2001@users.noreply.github.com>
Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
Co-authored-by: Lei Zhang <27994433+SWJTU-ZhangLei@users.noreply.github.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: qiye <jianliang5669@gmail.com>
Co-authored-by: AKIRA <33112463+Kikyou1997@users.noreply.github.com>
Co-authored-by: Liqf <109049295+LemonLiTree@users.noreply.github.com>
Co-authored-by: LiBinfeng <46676950+LiBinfeng-01@users.noreply.github.com>
Co-authored-by: zhangguoqiang <18372634969@163.com>
Co-authored-by: stephen <hello-stephen@qq.com>
Co-authored-by: GoGoWen <82132356+GoGoWen@users.noreply.github.com>
Co-authored-by: wangbo <wangbo@apache.org>
Co-authored-by: Lijia Liu <liutang123@yeah.net>
Co-authored-by: Kaijie Chen <ckj@apache.org>
Co-authored-by: Yulei-Yang <yulei.yang0699@gmail.com>
Co-authored-by: lsy3993 <110876560+lsy3993@users.noreply.github.com>
Co-authored-by: zhangdong <493738387@qq.com>
Co-authored-by: Xiangyu Wang <dut.xiangyu@gmail.com>
Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com>
Co-authored-by: wudongliang <46414265+DongLiang-0@users.noreply.github.com>
Co-authored-by: Houliang Qi <neuyilan@163.com>
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: HowardQin <hao.qin@esgyn.cn>
Co-authored-by: jiafeng.zhang <zhangjf1@gmail.com>
Co-authored-by: DuRipeng <453243496@qq.com>
Co-authored-by: catpineapple <42031973+catpineapple@users.noreply.github.com>
Co-authored-by: YueW <45946325+Tanya-W@users.noreply.github.com>
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.3-merged p0_b p0_c reviewed usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants