Skip to content

[fix](cloud-mow) Make some timeout about mow more reasonable #46365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zhannngchen merged 1 commit into from
Jan 29, 2025
Merged

[fix](cloud-mow) Make some timeout about mow more reasonable #46365

zhannngchen merged 1 commit into from
Jan 29, 2025

Conversation

@hust-hhb
Copy link
Contributor

@hust-hhb hust-hhb commented Jan 3, 2025

Now some timeout config is not suitable for high concurrency loading on Mow table, which may lead to frequent lock timeouts.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Jan 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 3, 2025

TPC-H: Total hot run time: 33516 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b5cc27720e16b842f193e2e6a7b3a300f2e4266, data reload: false

------ Round 1 ----------------------------------
q1	17689	6295	6135	6135
q2	2058	298	163	163
q3	10512	1357	758	758
q4	10222	888	461	461
q5	7657	2277	2083	2083
q6	217	186	153	153
q7	925	788	641	641
q8	9267	1457	1305	1305
q9	5334	5041	5044	5041
q10	6756	2358	1867	1867
q11	460	286	259	259
q12	355	379	236	236
q13	17766	3775	3173	3173
q14	239	233	224	224
q15	598	525	525	525
q16	626	618	584	584
q17	592	879	343	343
q18	7576	6816	6378	6378
q19	1330	981	598	598
q20	317	339	195	195
q21	3076	2407	2069	2069
q22	371	345	325	325
Total cold run time: 103943 ms
Total hot run time: 33516 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6323	6313	6296	6296
q2	246	335	236	236
q3	2342	2736	2368	2368
q4	1484	1928	1382	1382
q5	4375	4884	5048	4884
q6	192	180	147	147
q7	2206	2044	1813	1813
q8	2698	2866	2791	2791
q9	7309	7252	7315	7252
q10	3090	3398	2852	2852
q11	615	531	499	499
q12	693	790	615	615
q13	3536	3899	3342	3342
q14	290	318	276	276
q15	599	536	520	520
q16	654	679	637	637
q17	1266	1794	1242	1242
q18	8003	7515	7261	7261
q19	858	1086	1145	1086
q20	1943	2029	1858	1858
q21	5653	5130	4953	4953
q22	606	608	589	589
Total cold run time: 54981 ms
Total hot run time: 52899 ms

@doris-robot
Copy link

doris-robot commented Jan 3, 2025

TPC-DS: Total hot run time: 189816 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b5cc27720e16b842f193e2e6a7b3a300f2e4266, data reload: false

query1	962	379	368	368
query2	6544	2376	2319	2319
query3	6705	214	208	208
query4	34066	23542	23583	23542
query5	4324	614	463	463
query6	296	201	178	178
query7	4628	507	308	308
query8	325	246	242	242
query9	9567	2787	2756	2756
query10	443	309	245	245
query11	18389	15611	15223	15223
query12	159	111	115	111
query13	1669	548	403	403
query14	10832	7435	6883	6883
query15	240	211	200	200
query16	8149	587	445	445
query17	1583	753	571	571
query18	2044	394	291	291
query19	216	177	147	147
query20	118	108	132	108
query21	206	121	101	101
query22	4289	4514	4208	4208
query23	34417	33279	33140	33140
query24	6344	2240	2269	2240
query25	490	442	391	391
query26	1209	274	152	152
query27	2098	467	328	328
query28	5456	2469	2466	2466
query29	753	539	420	420
query30	225	184	152	152
query31	992	877	814	814
query32	88	60	61	60
query33	504	335	304	304
query34	767	835	506	506
query35	798	832	734	734
query36	1023	1021	938	938
query37	123	98	74	74
query38	4361	4493	4052	4052
query39	1493	1466	1432	1432
query40	208	112	101	101
query41	48	43	44	43
query42	117	105	103	103
query43	523	539	514	514
query44	1337	829	832	829
query45	180	173	166	166
query46	856	1034	641	641
query47	1917	1906	1843	1843
query48	387	412	326	326
query49	773	481	413	413
query50	650	654	384	384
query51	7162	7122	6974	6974
query52	102	105	88	88
query53	226	254	197	197
query54	464	481	427	427
query55	81	76	78	76
query56	246	267	253	253
query57	1225	1193	1159	1159
query58	239	226	230	226
query59	3078	3009	2950	2950
query60	277	268	269	268
query61	115	112	113	112
query62	859	798	767	767
query63	227	194	191	191
query64	4575	1012	665	665
query65	3290	3188	3196	3188
query66	1059	433	322	322
query67	15938	15922	15430	15430
query68	8164	716	523	523
query69	467	291	266	266
query70	1198	1128	1101	1101
query71	457	289	263	263
query72	6179	3902	3851	3851
query73	658	746	354	354
query74	10073	9193	8844	8844
query75	3978	3168	2661	2661
query76	3702	1178	783	783
query77	769	369	273	273
query78	10093	9975	9488	9488
query79	3692	785	590	590
query80	648	531	435	435
query81	479	280	224	224
query82	602	201	121	121
query83	162	170	147	147
query84	256	92	73	73
query85	819	354	371	354
query86	404	326	300	300
query87	4419	4692	4362	4362
query88	4736	2199	2173	2173
query89	422	332	296	296
query90	1863	186	187	186
query91	135	147	108	108
query92	68	56	55	55
query93	1888	871	534	534
query94	644	393	288	288
query95	335	262	251	251
query96	491	614	286	286
query97	2905	2938	2805	2805
query98	224	205	204	204
query99	1674	1549	1438	1438
Total cold run time: 295658 ms
Total hot run time: 189816 ms

@doris-robot
Copy link

doris-robot commented Jan 3, 2025

ClickBench: Total hot run time: 32.31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3b5cc27720e16b842f193e2e6a7b3a300f2e4266, data reload: false

query1	0.04	0.04	0.05
query2	0.08	0.03	0.04
query3	0.23	0.07	0.07
query4	1.61	0.11	0.10
query5	0.43	0.43	0.41
query6	1.15	0.64	0.65
query7	0.03	0.02	0.02
query8	0.03	0.04	0.03
query9	0.59	0.49	0.49
query10	0.56	0.56	0.54
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.60	0.60	0.60
query14	2.83	2.83	2.86
query15	0.90	0.82	0.81
query16	0.39	0.39	0.38
query17	1.06	1.02	1.04
query18	0.22	0.21	0.21
query19	1.86	1.83	1.94
query20	0.02	0.02	0.02
query21	15.36	0.90	0.58
query22	0.76	0.92	0.64
query23	15.22	1.46	0.50
query24	3.36	1.99	2.12
query25	0.14	0.09	0.17
query26	0.28	0.16	0.14
query27	0.05	0.05	0.05
query28	14.99	1.44	1.04
query29	12.59	3.93	3.21
query30	0.26	0.09	0.06
query31	2.81	0.60	0.38
query32	3.23	0.54	0.46
query33	3.14	3.17	3.16
query34	16.47	5.06	4.49
query35	4.43	4.44	4.48
query36	0.63	0.50	0.48
query37	0.10	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.13 s
Total hot run time: 32.31 s

@doris-robot
Copy link

doris-robot commented Jan 3, 2025

TeamCity be ut coverage result:
Function Coverage: 38.88% (10128/26048)
Line Coverage: 29.91% (85653/286371)
Region Coverage: 29.02% (43715/150642)
Branch Coverage: 25.56% (22316/87292)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3b5cc27720e16b842f193e2e6a7b3a300f2e4266_3b5cc27720e16b842f193e2e6a7b3a300f2e4266/report/index.html

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Jan 6, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 6, 2025

TPC-H: Total hot run time: 32813 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b5cc27720e16b842f193e2e6a7b3a300f2e4266, data reload: false

------ Round 1 ----------------------------------
q1	17589	6202	6051	6051
q2	2054	308	174	174
q3	10507	1257	741	741
q4	10221	872	434	434
q5	8062	2227	1963	1963
q6	207	182	145	145
q7	922	756	592	592
q8	9241	1374	1185	1185
q9	5111	4988	4950	4950
q10	6747	2318	1867	1867
q11	488	293	262	262
q12	343	372	220	220
q13	17757	3854	3106	3106
q14	237	251	218	218
q15	571	522	485	485
q16	632	630	592	592
q17	581	855	328	328
q18	7183	6451	6483	6451
q19	2181	980	553	553
q20	303	320	186	186
q21	2774	2173	2003	2003
q22	364	328	307	307
Total cold run time: 104075 ms
Total hot run time: 32813 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6305	6360	6319	6319
q2	235	331	232	232
q3	2252	2642	2361	2361
q4	1437	1888	1353	1353
q5	4351	4778	4863	4778
q6	185	179	143	143
q7	2142	1982	1848	1848
q8	2643	2849	2737	2737
q9	7365	7307	7402	7307
q10	3069	3381	2752	2752
q11	572	505	492	492
q12	645	739	601	601
q13	3512	3863	3294	3294
q14	295	320	302	302
q15	587	536	502	502
q16	653	691	639	639
q17	1217	1751	1291	1291
q18	7736	7494	7513	7494
q19	820	1162	1084	1084
q20	2002	2042	1925	1925
q21	5788	5276	5064	5064
q22	639	600	582	582
Total cold run time: 54450 ms
Total hot run time: 53100 ms

@doris-robot
Copy link

doris-robot commented Jan 6, 2025

TPC-DS: Total hot run time: 197928 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b5cc27720e16b842f193e2e6a7b3a300f2e4266, data reload: false

query1	1306	952	903	903
query2	6123	2325	2340	2325
query3	11121	4837	4704	4704
query4	33060	23840	23465	23465
query5	3815	603	465	465
query6	277	208	194	194
query7	3980	496	311	311
query8	318	251	257	251
query9	9504	2742	2721	2721
query10	441	308	245	245
query11	18036	15339	15240	15240
query12	160	111	105	105
query13	1595	543	410	410
query14	11063	7716	8239	7716
query15	247	201	210	201
query16	7835	646	448	448
query17	1531	762	606	606
query18	2088	407	330	330
query19	197	186	157	157
query20	118	114	115	114
query21	206	130	122	122
query22	4818	4676	4411	4411
query23	36151	34966	33726	33726
query24	6329	2298	2292	2292
query25	450	437	385	385
query26	754	257	160	160
query27	2042	474	337	337
query28	5255	2535	2491	2491
query29	558	539	426	426
query30	213	188	148	148
query31	1011	951	841	841
query32	77	59	60	59
query33	491	371	311	311
query34	758	891	516	516
query35	834	852	756	756
query36	1018	1059	947	947
query37	116	90	73	73
query38	4390	4383	4253	4253
query39	1545	1503	1457	1457
query40	208	120	113	113
query41	45	44	46	44
query42	129	102	104	102
query43	528	530	486	486
query44	1363	855	854	854
query45	187	176	176	176
query46	892	1078	660	660
query47	1989	2022	1954	1954
query48	402	427	351	351
query49	708	481	399	399
query50	629	675	399	399
query51	7361	7187	7227	7187
query52	105	102	100	100
query53	232	262	187	187
query54	495	510	429	429
query55	89	77	76	76
query56	269	282	270	270
query57	1251	1223	1196	1196
query58	256	232	236	232
query59	3263	3421	3093	3093
query60	277	278	256	256
query61	129	111	109	109
query62	862	814	776	776
query63	245	191	192	191
query64	3498	1035	690	690
query65	3329	3311	3269	3269
query66	876	419	303	303
query67	16728	15862	15619	15619
query68	9201	695	535	535
query69	466	294	246	246
query70	1235	1169	1119	1119
query71	447	275	253	253
query72	6465	3835	3875	3835
query73	649	742	362	362
query74	10335	9253	9014	9014
query75	4312	3157	2659	2659
query76	3926	1267	761	761
query77	756	374	297	297
query78	10041	10162	9395	9395
query79	5255	789	597	597
query80	754	517	452	452
query81	493	270	234	234
query82	326	150	126	126
query83	187	163	148	148
query84	282	93	70	70
query85	757	366	309	309
query86	344	326	296	296
query87	4585	4420	4470	4420
query88	3652	2217	2179	2179
query89	441	332	294	294
query90	2162	187	192	187
query91	138	135	104	104
query92	65	55	51	51
query93	3191	858	527	527
query94	690	401	291	291
query95	334	267	258	258
query96	517	611	277	277
query97	3055	2951	2863	2863
query98	220	216	192	192
query99	1656	1574	1434	1434
Total cold run time: 303113 ms
Total hot run time: 197928 ms

@doris-robot
Copy link

doris-robot commented Jan 6, 2025

ClickBench: Total hot run time: 31.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3b5cc27720e16b842f193e2e6a7b3a300f2e4266, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.61	0.10	0.11
query5	0.43	0.43	0.41
query6	1.18	0.67	0.70
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.49	0.49
query10	0.54	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.61
query14	2.71	2.84	2.76
query15	0.91	0.84	0.83
query16	0.40	0.38	0.38
query17	1.06	1.07	1.00
query18	0.22	0.20	0.21
query19	1.98	1.77	2.03
query20	0.02	0.01	0.01
query21	15.35	0.95	0.59
query22	0.76	0.83	0.55
query23	15.36	1.41	0.62
query24	2.92	0.98	1.65
query25	0.26	0.13	0.07
query26	0.22	0.14	0.13
query27	0.08	0.06	0.05
query28	13.92	1.53	1.05
query29	12.59	3.96	3.32
query30	0.24	0.09	0.07
query31	2.81	0.57	0.37
query32	3.22	0.54	0.46
query33	3.08	3.14	3.08
query34	16.98	5.14	4.48
query35	4.52	4.49	4.49
query36	0.64	0.53	0.50
query37	0.09	0.07	0.06
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.36 s
Total hot run time: 31.27 s

@doris-robot
Copy link

doris-robot commented Jan 6, 2025

TeamCity be ut coverage result:
Function Coverage: 38.89% (10129/26048)
Line Coverage: 29.92% (85682/286410)
Region Coverage: 29.02% (43732/150674)
Branch Coverage: 25.56% (22315/87318)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3b5cc27720e16b842f193e2e6a7b3a300f2e4266_3b5cc27720e16b842f193e2e6a7b3a300f2e4266/report/index.html

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Jan 6, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 6, 2025

TPC-H: Total hot run time: 32461 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4abb09d723ff3b54ff2b791de66d62a7f3c1d189, data reload: false

------ Round 1 ----------------------------------
q1	17589	6154	6046	6046
q2	2054	293	172	172
q3	10486	1240	744	744
q4	10266	874	435	435
q5	8604	2180	1969	1969
q6	213	185	145	145
q7	884	761	596	596
q8	9237	1330	1109	1109
q9	5181	4923	4937	4923
q10	6752	2306	1863	1863
q11	514	274	259	259
q12	343	364	216	216
q13	17776	3678	3092	3092
q14	238	242	214	214
q15	560	512	487	487
q16	649	606	576	576
q17	563	844	319	319
q18	7039	6589	6306	6306
q19	3073	957	547	547
q20	294	306	183	183
q21	2771	2145	1956	1956
q22	365	337	304	304
Total cold run time: 105451 ms
Total hot run time: 32461 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6344	6266	6228	6228
q2	236	322	231	231
q3	2253	2646	2363	2363
q4	1439	1834	1452	1452
q5	4374	4791	4900	4791
q6	181	178	140	140
q7	2046	1989	1840	1840
q8	2617	2807	2646	2646
q9	7342	7358	7264	7264
q10	3084	3414	2774	2774
q11	566	516	490	490
q12	665	777	613	613
q13	3482	3856	3278	3278
q14	293	298	292	292
q15	565	530	489	489
q16	647	695	634	634
q17	1221	1719	1268	1268
q18	7718	7384	7384	7384
q19	841	1168	1062	1062
q20	1977	2052	1921	1921
q21	5732	5173	4882	4882
q22	686	635	603	603
Total cold run time: 54309 ms
Total hot run time: 52645 ms

@doris-robot
Copy link

doris-robot commented Jan 6, 2025

TPC-DS: Total hot run time: 197719 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4abb09d723ff3b54ff2b791de66d62a7f3c1d189, data reload: false

query1	1289	960	945	945
query2	6196	2401	2355	2355
query3	11124	4738	4828	4738
query4	32840	23706	23427	23427
query5	4227	649	450	450
query6	296	214	201	201
query7	4005	494	297	297
query8	310	258	236	236
query9	9222	2722	2707	2707
query10	472	324	250	250
query11	18078	15405	15415	15405
query12	160	108	106	106
query13	1659	530	405	405
query14	10671	7503	8057	7503
query15	233	215	195	195
query16	8126	649	477	477
query17	1573	771	646	646
query18	2139	399	308	308
query19	211	187	162	162
query20	121	119	109	109
query21	210	127	143	127
query22	4903	4912	4625	4625
query23	34501	33833	33470	33470
query24	6177	2400	2301	2301
query25	486	473	390	390
query26	713	289	157	157
query27	2036	465	344	344
query28	5509	2515	2486	2486
query29	638	545	424	424
query30	211	177	145	145
query31	964	915	822	822
query32	85	58	78	58
query33	480	352	305	305
query34	749	858	510	510
query35	836	842	737	737
query36	1044	1076	990	990
query37	120	102	79	79
query38	4213	4338	4172	4172
query39	1548	1650	1457	1457
query40	199	118	102	102
query41	45	42	44	42
query42	124	101	100	100
query43	508	526	493	493
query44	1335	830	843	830
query45	180	177	172	172
query46	873	1072	669	669
query47	2042	2015	2010	2010
query48	380	412	330	330
query49	711	472	387	387
query50	641	668	403	403
query51	7314	7151	7268	7151
query52	103	108	90	90
query53	227	258	179	179
query54	474	510	408	408
query55	77	80	76	76
query56	245	275	258	258
query57	1273	1252	1189	1189
query58	233	232	246	232
query59	3244	3284	3310	3284
query60	272	258	254	254
query61	134	111	125	111
query62	865	817	772	772
query63	228	192	204	192
query64	3224	999	655	655
query65	3324	3263	3299	3263
query66	779	410	305	305
query67	16612	15949	15536	15536
query68	9263	693	520	520
query69	476	292	237	237
query70	1240	1103	1148	1103
query71	420	296	255	255
query72	6186	3838	3897	3838
query73	644	763	360	360
query74	10555	9162	9142	9142
query75	4520	3174	2643	2643
query76	3649	1180	749	749
query77	781	364	300	300
query78	9879	10045	9375	9375
query79	3247	797	585	585
query80	754	546	422	422
query81	476	268	226	226
query82	346	149	118	118
query83	193	163	149	149
query84	277	89	75	75
query85	802	376	303	303
query86	355	310	309	309
query87	4751	4569	4347	4347
query88	3223	2219	2178	2178
query89	433	329	285	285
query90	2010	187	188	187
query91	150	127	105	105
query92	66	54	51	51
query93	2048	862	540	540
query94	665	382	282	282
query95	335	264	246	246
query96	482	607	270	270
query97	2909	2957	2875	2875
query98	222	201	188	188
query99	1713	1523	1442	1442
Total cold run time: 297111 ms
Total hot run time: 197719 ms

@doris-robot
Copy link

doris-robot commented Jan 6, 2025

ClickBench: Total hot run time: 31.28 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4abb09d723ff3b54ff2b791de66d62a7f3c1d189, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.11
query5	0.41	0.42	0.40
query6	1.16	0.66	0.67
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.51	0.51
query10	0.56	0.57	0.54
query11	0.14	0.10	0.10
query12	0.13	0.11	0.10
query13	0.60	0.61	0.59
query14	2.85	2.73	2.81
query15	0.88	0.82	0.82
query16	0.38	0.38	0.39
query17	1.04	1.00	0.98
query18	0.22	0.20	0.20
query19	1.94	1.89	1.96
query20	0.01	0.02	0.01
query21	15.36	0.91	0.59
query22	0.76	0.81	0.69
query23	15.29	1.47	0.52
query24	3.36	0.99	1.27
query25	0.20	0.19	0.09
query26	0.29	0.15	0.13
query27	0.07	0.05	0.04
query28	13.95	1.54	1.05
query29	12.60	4.02	3.27
query30	0.24	0.09	0.07
query31	2.82	0.60	0.38
query32	3.22	0.54	0.47
query33	3.09	3.02	3.18
query34	16.57	5.14	4.51
query35	4.57	4.49	4.55
query36	0.66	0.48	0.49
query37	0.09	0.06	0.05
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.47 s
Total hot run time: 31.28 s

@doris-robot
Copy link

doris-robot commented Jan 6, 2025

TeamCity be ut coverage result:
Function Coverage: 38.89% (10129/26048)
Line Coverage: 29.93% (85721/286405)
Region Coverage: 29.02% (43729/150671)
Branch Coverage: 25.56% (22320/87316)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4abb09d723ff3b54ff2b791de66d62a7f3c1d189_4abb09d723ff3b54ff2b791de66d62a7f3c1d189/report/index.html

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Jan 8, 2025

run buildall

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Jan 8, 2025

@zhannngchen take a look plz

@doris-robot
Copy link

doris-robot commented Jan 8, 2025

TPC-H: Total hot run time: 32591 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9e130bd5e98bc9ba6bdccbcb7714176609005710, data reload: false

------ Round 1 ----------------------------------
q1	17620	6198	6021	6021
q2	2044	332	179	179
q3	10533	1256	730	730
q4	10197	880	441	441
q5	7547	2188	1978	1978
q6	201	185	154	154
q7	901	755	599	599
q8	9231	1359	1183	1183
q9	5193	4929	4964	4929
q10	6765	2312	1855	1855
q11	474	289	255	255
q12	340	356	218	218
q13	17771	3682	3046	3046
q14	246	228	210	210
q15	565	508	500	500
q16	628	614	574	574
q17	579	863	340	340
q18	7193	6542	6362	6362
q19	2421	984	579	579
q20	304	313	192	192
q21	2819	2203	1935	1935
q22	356	334	311	311
Total cold run time: 103928 ms
Total hot run time: 32591 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6353	6234	6263	6234
q2	246	328	228	228
q3	2244	2663	2335	2335
q4	1439	1802	1364	1364
q5	4320	4772	4858	4772
q6	182	182	144	144
q7	2097	1980	1825	1825
q8	2673	2849	2711	2711
q9	7364	7337	7289	7289
q10	3116	3327	2895	2895
q11	580	502	500	500
q12	652	743	609	609
q13	3562	3886	3222	3222
q14	307	317	294	294
q15	579	522	507	507
q16	676	687	635	635
q17	1224	1742	1248	1248
q18	7683	7495	7354	7354
q19	847	1034	1165	1034
q20	2004	2041	1931	1931
q21	5626	5308	5095	5095
q22	627	616	550	550
Total cold run time: 54401 ms
Total hot run time: 52776 ms

@doris-robot
Copy link

doris-robot commented Jan 8, 2025

TPC-DS: Total hot run time: 195915 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9e130bd5e98bc9ba6bdccbcb7714176609005710, data reload: false

query1	1297	966	925	925
query2	6459	2344	2296	2296
query3	11166	4679	4915	4679
query4	32837	24020	23615	23615
query5	4136	588	442	442
query6	299	205	193	193
query7	3974	491	307	307
query8	315	267	262	262
query9	9533	2707	2674	2674
query10	465	306	245	245
query11	17756	15270	15409	15270
query12	169	109	109	109
query13	1579	534	424	424
query14	10507	7151	7211	7151
query15	243	209	185	185
query16	7791	626	473	473
query17	1581	786	592	592
query18	2111	436	327	327
query19	251	198	160	160
query20	119	120	110	110
query21	206	130	108	108
query22	4489	4738	4436	4436
query23	35035	33301	33248	33248
query24	6403	2339	2337	2337
query25	450	451	405	405
query26	729	235	149	149
query27	2138	450	335	335
query28	5647	2508	2467	2467
query29	571	505	419	419
query30	223	189	149	149
query31	940	881	806	806
query32	70	61	58	58
query33	493	338	294	294
query34	743	838	521	521
query35	813	830	780	780
query36	1009	1001	959	959
query37	113	92	78	78
query38	4093	4191	4336	4191
query39	1489	1486	1460	1460
query40	201	116	105	105
query41	52	41	45	41
query42	117	101	99	99
query43	522	529	493	493
query44	1328	840	840	840
query45	182	173	168	168
query46	872	1036	649	649
query47	1916	1935	1888	1888
query48	393	404	323	323
query49	711	473	402	402
query50	651	647	402	402
query51	7126	7117	6952	6952
query52	109	105	91	91
query53	230	247	187	187
query54	480	479	417	417
query55	82	82	78	78
query56	253	254	251	251
query57	1214	1213	1127	1127
query58	250	224	216	216
query59	3205	3171	2989	2989
query60	267	259	248	248
query61	109	105	109	105
query62	828	785	714	714
query63	226	196	184	184
query64	3014	1037	648	648
query65	3322	3200	3218	3200
query66	776	419	304	304
query67	16321	15869	15425	15425
query68	7841	691	517	517
query69	471	296	260	260
query70	1148	1140	1149	1140
query71	419	292	252	252
query72	6634	3906	3834	3834
query73	654	754	375	375
query74	10333	9139	8990	8990
query75	4028	3145	2696	2696
query76	3646	1290	745	745
query77	772	343	272	272
query78	10130	10005	9427	9427
query79	3491	822	583	583
query80	719	524	428	428
query81	487	260	234	234
query82	638	148	125	125
query83	238	161	146	146
query84	293	88	77	77
query85	738	362	300	300
query86	350	310	308	308
query87	4418	4462	4384	4384
query88	4538	2210	2196	2196
query89	417	328	285	285
query90	1953	187	190	187
query91	139	134	109	109
query92	71	54	52	52
query93	1860	844	531	531
query94	649	401	300	300
query95	331	269	254	254
query96	482	603	283	283
query97	2844	2869	2845	2845
query98	218	198	201	198
query99	1602	1519	1397	1397
Total cold run time: 295551 ms
Total hot run time: 195915 ms

@doris-robot
Copy link

doris-robot commented Jan 8, 2025

ClickBench: Total hot run time: 31.39 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9e130bd5e98bc9ba6bdccbcb7714176609005710, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.04	0.03
query3	0.23	0.07	0.06
query4	1.61	0.10	0.10
query5	0.41	0.42	0.43
query6	1.18	0.65	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.52	0.51
query10	0.56	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.10	0.12
query13	0.60	0.58	0.61
query14	2.73	2.75	2.74
query15	0.89	0.82	0.82
query16	0.39	0.37	0.38
query17	1.05	1.07	1.05
query18	0.23	0.21	0.21
query19	1.96	1.82	1.99
query20	0.01	0.02	0.01
query21	15.36	0.92	0.60
query22	0.75	0.85	0.67
query23	15.24	1.46	0.54
query24	3.04	1.16	0.86
query25	0.22	0.09	0.17
query26	0.27	0.15	0.13
query27	0.08	0.04	0.05
query28	13.47	1.57	1.05
query29	12.57	3.94	3.26
query30	0.25	0.08	0.06
query31	2.83	0.62	0.37
query32	3.24	0.56	0.46
query33	3.18	3.22	3.27
query34	16.80	5.13	4.52
query35	4.58	4.54	4.54
query36	0.64	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.89 s
Total hot run time: 31.39 s

@doris-robot
Copy link

doris-robot commented Jan 8, 2025

TeamCity be ut coverage result:
Function Coverage: 38.86% (10127/26061)
Line Coverage: 29.91% (85673/286475)
Region Coverage: 29.01% (43721/150704)
Branch Coverage: 25.55% (22318/87340)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9e130bd5e98bc9ba6bdccbcb7714176609005710_9e130bd5e98bc9ba6bdccbcb7714176609005710/report/index.html

zhannngchen
zhannngchen previously approved these changes Jan 13, 2025
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 13, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jan 13, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 13, 2025

PR approved by anyone and no changes requested.

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Jan 26, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 26, 2025
@doris-robot
Copy link

doris-robot commented Jan 26, 2025

TPC-H: Total hot run time: 32552 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f161a77772526876671e0e7154077e887346d7d7, data reload: false

------ Round 1 ----------------------------------
q1	17575	5509	5380	5380
q2	2044	311	166	166
q3	10459	1332	737	737
q4	10253	959	557	557
q5	8575	2435	2151	2151
q6	205	172	136	136
q7	939	771	635	635
q8	9262	1365	1179	1179
q9	5910	4972	4927	4927
q10	6846	2369	1897	1897
q11	474	278	256	256
q12	351	367	224	224
q13	17783	3716	3117	3117
q14	233	242	223	223
q15	510	480	472	472
q16	624	629	591	591
q17	574	887	339	339
q18	6995	6487	6447	6447
q19	2120	958	555	555
q20	304	322	188	188
q21	2958	2201	2063	2063
q22	367	325	312	312
Total cold run time: 105361 ms
Total hot run time: 32552 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5625	5721	5483	5483
q2	259	333	229	229
q3	2277	2634	2320	2320
q4	1437	1846	1381	1381
q5	4327	4725	4894	4725
q6	170	160	128	128
q7	2142	2011	1845	1845
q8	2658	2816	2725	2725
q9	7372	7271	7170	7170
q10	3015	3331	2771	2771
q11	581	505	491	491
q12	683	778	615	615
q13	3429	3935	3295	3295
q14	278	304	279	279
q15	526	472	459	459
q16	659	708	665	665
q17	1228	1743	1277	1277
q18	7673	7467	7328	7328
q19	807	1180	1090	1090
q20	2059	2026	1871	1871
q21	5747	5224	4861	4861
q22	612	608	552	552
Total cold run time: 53564 ms
Total hot run time: 51560 ms

@doris-robot
Copy link

doris-robot commented Jan 26, 2025

TPC-DS: Total hot run time: 185149 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f161a77772526876671e0e7154077e887346d7d7, data reload: false

query1	967	386	379	379
query2	6531	2057	2057	2057
query3	6780	218	216	216
query4	33146	23802	23225	23225
query5	4358	632	468	468
query6	292	199	188	188
query7	4600	499	311	311
query8	306	243	256	243
query9	9509	2723	2716	2716
query10	481	323	250	250
query11	17711	15212	14862	14862
query12	153	109	106	106
query13	1668	543	411	411
query14	9656	7060	6891	6891
query15	232	194	187	187
query16	8050	613	478	478
query17	1615	760	572	572
query18	2131	413	324	324
query19	208	185	165	165
query20	123	114	108	108
query21	211	120	102	102
query22	4209	4304	4278	4278
query23	33993	32967	33117	32967
query24	6562	2239	2303	2239
query25	498	448	383	383
query26	1207	268	152	152
query27	2024	467	335	335
query28	5283	2492	2462	2462
query29	750	547	415	415
query30	226	186	155	155
query31	958	861	776	776
query32	81	66	58	58
query33	533	345	300	300
query34	758	865	523	523
query35	801	814	741	741
query36	1004	1047	927	927
query37	125	95	76	76
query38	4144	4168	4079	4079
query39	1622	1395	1387	1387
query40	204	116	100	100
query41	53	60	51	51
query42	128	104	101	101
query43	512	528	495	495
query44	1292	828	828	828
query45	194	170	162	162
query46	852	1047	648	648
query47	1820	1811	1789	1789
query48	380	404	339	339
query49	776	487	410	410
query50	639	663	394	394
query51	4231	4215	4120	4120
query52	102	103	99	99
query53	235	259	190	190
query54	482	507	416	416
query55	84	82	83	82
query56	291	262	249	249
query57	1148	1171	1090	1090
query58	245	232	238	232
query59	2913	2987	2871	2871
query60	269	261	260	260
query61	124	111	112	111
query62	790	760	669	669
query63	223	193	184	184
query64	4419	981	666	666
query65	3229	3142	3170	3142
query66	1069	401	308	308
query67	15859	15409	15460	15409
query68	4556	819	538	538
query69	487	301	257	257
query70	1212	1118	1128	1118
query71	387	292	267	267
query72	5852	3782	3893	3782
query73	671	757	369	369
query74	10320	9029	8718	8718
query75	3174	3158	2634	2634
query76	3233	1173	789	789
query77	514	376	281	281
query78	10020	10039	9372	9372
query79	2478	803	617	617
query80	721	532	448	448
query81	524	273	244	244
query82	337	161	139	139
query83	180	173	165	165
query84	246	92	82	82
query85	745	341	292	292
query86	399	330	312	312
query87	4481	4477	4412	4412
query88	4307	2210	2156	2156
query89	398	332	294	294
query90	1968	194	194	194
query91	136	145	107	107
query92	69	55	55	55
query93	2804	877	532	532
query94	680	407	310	310
query95	348	260	261	260
query96	499	643	298	298
query97	2800	2808	2719	2719
query98	227	195	198	195
query99	1291	1378	1266	1266
Total cold run time: 282133 ms
Total hot run time: 185149 ms

@doris-robot
Copy link

doris-robot commented Jan 26, 2025

ClickBench: Total hot run time: 30.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f161a77772526876671e0e7154077e887346d7d7, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.02
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.41	0.41	0.42
query6	1.14	0.67	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.60	0.50	0.49
query10	0.56	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.74	2.78	2.88
query15	0.90	0.83	0.83
query16	0.40	0.38	0.37
query17	1.06	1.00	1.06
query18	0.22	0.21	0.21
query19	1.90	1.80	1.98
query20	0.01	0.01	0.01
query21	15.39	0.90	0.60
query22	0.76	0.76	0.71
query23	15.27	1.40	0.54
query24	2.85	0.77	0.72
query25	0.21	0.12	0.05
query26	0.23	0.16	0.13
query27	0.06	0.06	0.05
query28	13.58	1.03	0.45
query29	12.61	3.92	3.23
query30	0.26	0.09	0.06
query31	2.83	0.58	0.38
query32	3.22	0.55	0.46
query33	2.99	2.94	3.03
query34	16.48	5.17	4.47
query35	4.54	4.50	4.53
query36	0.64	0.48	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.25 s
Total hot run time: 30.27 s

@doris-robot
Copy link

doris-robot commented Jan 26, 2025

TeamCity be ut coverage result:
Function Coverage: 42.06% (10971/26082)
Line Coverage: 32.32% (92703/286786)
Region Coverage: 31.49% (47550/151021)
Branch Coverage: 27.52% (24077/87490)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f161a77772526876671e0e7154077e887346d7d7_f161a77772526876671e0e7154077e887346d7d7/report/index.html

dataroaring
dataroaring approved these changes Jan 27, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jan 27, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 27, 2025
@zhannngchen zhannngchen merged commit c3e8c64 into apache:master Jan 29, 2025
25 of 27 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 29, 2025
@hust-hhb
[fix](cloud-mow) Make some timeout about mow more reasonable (#46365)
e2e618a 
Now some timeout config is not suitable for high concurrency loading on
Mow table, which may lead to frequent lock timeouts.
@github-actions github-actions bot mentioned this pull request Jan 29, 2025
Merged
zhannngchen pushed a commit that referenced this pull request Jan 29, 2025
@hust-hhb
[fix](cloud-mow) Make delete bitmap cache expired time more reasonable (
29055f2 
#47295)

after pr #46365, commit mow table
may cost 1800s at most (rpc timeout*retry times), so the delete bitmap
cache expiration time should be set to 1800s.This cache will be
automatically deleted after the loading task is completed. Only when
task fails, this cache will be clean in background.
github-actions bot pushed a commit that referenced this pull request Jan 29, 2025
@hust-hhb
[fix](cloud-mow) Make delete bitmap cache expired time more reasonable (
3b246ac 
#47295)

after pr #46365, commit mow table
may cost 1800s at most (rpc timeout*retry times), so the delete bitmap
cache expiration time should be set to 1800s.This cache will be
automatically deleted after the loading task is completed. Only when
task fails, this cache will be clean in background.
dataroaring pushed a commit that referenced this pull request Jan 30, 2025
@github-actions @hust-hhb
branch-3.0: [fix](cloud-mow) Make some timeout about mow more reasona…
7950af3 
…ble #46365 (#47489)

Cherry-picked from #46365

Co-authored-by: huanghaibin <huanghaibin@selectdb.com>
@hust-hhb hust-hhb mentioned this pull request Feb 10, 2025
Merged
16 tasks
@gavinchou gavinchou mentioned this pull request Feb 18, 2025
Closed
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@hust-hhb @lzyy2024
[fix](cloud-mow) Make some timeout about mow more reasonable (apache#…
48068ab 
…46365)

Now some timeout config is not suitable for high concurrency loading on
Mow table, which may lead to frequent lock timeouts.
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@hust-hhb @lzyy2024
[fix](cloud-mow) Make delete bitmap cache expired time more reasonable (
856e6f0 
apache#47295)

after pr apache#46365, commit mow table
may cost 1800s at most (rpc timeout*retry times), so the delete bitmap
cache expiration time should be set to 1800s.This cache will be
automatically deleted after the loading task is completed. Only when
task fails, this cache will be clean in background.
dataroaring pushed a commit that referenced this pull request Feb 21, 2025
@hust-hhb
[improve](cloud-mow) Make mow stream load retry times more reasonable (…
6c1778f 
…#47666)

pr #46365 adjust timeout about mow load, thers is no need so much retry
times, can reduce to 5.
github-actions bot pushed a commit that referenced this pull request Feb 21, 2025
@hust-hhb
[improve](cloud-mow) Make mow stream load retry times more reasonable (…
cb6b7e4 
…#47666)

pr #46365 adjust timeout about mow load, thers is no need so much retry
times, can reduce to 5.
seawinde pushed a commit to seawinde/doris that referenced this pull request Feb 28, 2025
@hust-hhb @seawinde
[improve](cloud-mow) Make mow stream load retry times more reasonable (…
60f48df 
…apache#47666)

pr apache#46365 adjust timeout about mow load, thers is no need so much retry
times, can reduce to 5.
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@hust-hhb @koarz
[fix](cloud-mow) Make some timeout about mow more reasonable (apache#…
cf51688 
…46365)

Now some timeout config is not suitable for high concurrency loading on
Mow table, which may lead to frequent lock timeouts.
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@hust-hhb @koarz
[fix](cloud-mow) Make delete bitmap cache expired time more reasonable (
561e7d8 
apache#47295)

after pr apache#46365, commit mow table
may cost 1800s at most (rpc timeout*retry times), so the delete bitmap
cache expiration time should be set to 1800s.This cache will be
automatically deleted after the loading task is completed. Only when
task fails, this cache will be clean in background.
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@hust-hhb @koarz
[improve](cloud-mow) Make mow stream load retry times more reasonable (…
f05aabd 
…apache#47666)

pr apache#46365 adjust timeout about mow load, thers is no need so much retry
times, can reduce to 5.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mymeiyi mymeiyi mymeiyi left review comments

@zhannngchen zhannngchen zhannngchen approved these changes

@dataroaring dataroaring dataroaring approved these changes

@liaoxin01 liaoxin01 liaoxin01 left review comments

+1 more reviewer

@cosven cosven cosven left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.4-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@hust-hhb @Thearas @doris-robot @mymeiyi @cosven @liaoxin01 @zhannngchen @dataroaring