Skip to content

Conversation

@github-actions
Copy link
Contributor

Cherry-picked from #54983

…ve auto partitions (#54983)

optimize parallel of insert command when olap table have auto partitions
@github-actions github-actions bot requested a review from dataroaring as a code owner August 20, 2025 02:21
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Aug 20, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40101 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0abbba7f49132eddaf97616114ccd0c795948b6, data reload: false

------ Round 1 ----------------------------------
q1	17583	6780	6748	6748
q2	2029	202	161	161
q3	10518	1115	1111	1111
q4	10407	754	829	754
q5	7726	2915	2803	2803
q6	212	129	136	129
q7	962	621	592	592
q8	9582	1970	1989	1970
q9	8018	6408	6446	6408
q10	7014	2266	2297	2266
q11	454	269	263	263
q12	392	208	214	208
q13	17808	2982	2986	2982
q14	231	207	213	207
q15	512	473	478	473
q16	463	377	376	376
q17	972	617	583	583
q18	7188	6700	6749	6700
q19	1386	1057	1083	1057
q20	469	189	193	189
q21	3859	3177	3173	3173
q22	1088	1005	948	948
Total cold run time: 108873 ms
Total hot run time: 40101 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6660	6628	6601	6601
q2	328	227	227	227
q3	2948	2869	2936	2869
q4	2038	1786	1774	1774
q5	5787	5743	5874	5743
q6	210	129	130	129
q7	2238	1887	1858	1858
q8	3419	3616	3575	3575
q9	8853	8920	8885	8885
q10	3536	3544	3536	3536
q11	598	500	491	491
q12	811	602	643	602
q13	6592	3127	3178	3127
q14	306	268	277	268
q15	513	465	464	464
q16	487	436	434	434
q17	1881	1685	1656	1656
q18	8222	7796	7855	7796
q19	1646	1541	1565	1541
q20	2067	1886	1870	1870
q21	5233	5004	5121	5004
q22	1112	1050	1009	1009
Total cold run time: 65485 ms
Total hot run time: 59459 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192378 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0abbba7f49132eddaf97616114ccd0c795948b6, data reload: false

query1	932	414	425	414
query2	6281	1995	1942	1942
query3	8679	198	202	198
query4	33670	23635	23481	23481
query5	3482	456	439	439
query6	283	187	181	181
query7	4205	312	310	310
query8	298	233	218	218
query9	9476	2537	2542	2537
query10	472	264	261	261
query11	18274	15134	15160	15134
query12	155	104	101	101
query13	1547	423	421	421
query14	9065	7010	6695	6695
query15	249	170	174	170
query16	7947	498	391	391
query17	1636	624	600	600
query18	2059	326	348	326
query19	258	162	155	155
query20	121	119	114	114
query21	203	107	105	105
query22	4702	4510	4329	4329
query23	34883	34559	34514	34514
query24	11669	2986	2856	2856
query25	703	408	403	403
query26	1757	176	172	172
query27	3079	339	354	339
query28	7719	2111	2157	2111
query29	1049	435	429	429
query30	264	159	162	159
query31	1029	837	818	818
query32	89	53	56	53
query33	780	300	300	300
query34	1126	524	522	522
query35	882	737	739	737
query36	1143	998	993	993
query37	194	72	68	68
query38	4110	3947	3948	3947
query39	1491	1504	1479	1479
query40	253	96	99	96
query41	49	49	51	49
query42	107	97	106	97
query43	548	529	497	497
query44	1280	817	830	817
query45	186	165	170	165
query46	1153	749	707	707
query47	2004	1909	1947	1909
query48	479	381	384	381
query49	1056	396	395	395
query50	832	416	430	416
query51	7348	7335	7260	7260
query52	104	90	90	90
query53	267	184	186	184
query54	1300	475	469	469
query55	77	78	80	78
query56	274	259	239	239
query57	1336	1217	1215	1215
query58	223	213	229	213
query59	3177	3050	3091	3050
query60	297	246	257	246
query61	115	118	118	118
query62	897	713	673	673
query63	238	192	187	187
query64	5215	681	654	654
query65	3392	3322	3312	3312
query66	1346	298	299	298
query67	16365	15515	15524	15515
query68	4725	574	570	570
query69	446	268	267	267
query70	1161	1112	1087	1087
query71	348	258	252	252
query72	6183	4087	4086	4086
query73	748	344	344	344
query74	10556	9180	9224	9180
query75	3341	2664	2629	2629
query76	2743	977	1059	977
query77	385	273	280	273
query78	10495	9576	9606	9576
query79	1703	590	605	590
query80	999	428	438	428
query81	540	222	215	215
query82	609	87	86	86
query83	251	146	141	141
query84	235	81	74	74
query85	1339	289	288	288
query86	422	306	301	301
query87	4351	4265	4237	4237
query88	3674	2589	2375	2375
query89	404	297	297	297
query90	1936	181	184	181
query91	182	146	151	146
query92	58	48	50	48
query93	1954	542	543	542
query94	830	289	294	289
query95	358	250	257	250
query96	616	281	277	277
query97	3295	3136	3188	3136
query98	210	201	197	197
query99	1485	1320	1287	1287
Total cold run time: 302461 ms
Total hot run time: 192378 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.42 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0abbba7f49132eddaf97616114ccd0c795948b6, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.10	0.10
query5	0.53	0.50	0.49
query6	1.13	0.75	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.59	0.55	0.58
query11	0.15	0.10	0.10
query12	0.12	0.10	0.10
query13	0.61	0.59	0.60
query14	0.78	0.80	0.79
query15	0.84	0.82	0.82
query16	0.40	0.38	0.39
query17	0.99	1.09	1.08
query18	0.24	0.23	0.21
query19	1.98	1.86	1.83
query20	0.02	0.01	0.01
query21	15.38	0.61	0.57
query22	2.23	1.80	1.50
query23	16.92	0.88	0.72
query24	3.77	0.59	1.55
query25	0.20	0.10	0.07
query26	0.51	0.13	0.14
query27	0.04	0.04	0.04
query28	9.96	0.53	0.46
query29	12.56	3.23	3.18
query30	0.24	0.07	0.06
query31	2.85	0.38	0.39
query32	3.22	0.47	0.46
query33	3.05	2.99	2.99
query34	16.81	4.49	4.60
query35	4.58	4.65	4.56
query36	0.67	0.47	0.50
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.43 s
Total hot run time: 29.42 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 07ef370 into branch-3.0 Sep 5, 2025
24 of 26 checks passed
@github-actions github-actions bot deleted the auto-pick-54983-branch-3.0 branch September 5, 2025 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants