Skip to content

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Dec 2, 2024

Cherry-picked from #44846

… `PublishVersionTask.partitionVersionInfos` (#44846)

### What problem does this PR solve?

Problem Summary: When FE handles BEs' tablet report and finds that there
exists some expired txns on BE, it will generate publish version task.
`TPartitionVersionInfo` with same values may be added to
`transactionsToPublish` under same txn id many times when partitions
involved in this failed txn involves many tablets on this BE. Because it
uses `ArrayListMultimap`, these duplicate values may occupy a lot of
memories when the number of tablets is large.

### Release note

Fixed the issue of FE's memory occupation growing too fast in cases of
persistent load and clone failures on merge-on-write tables.

修复了在merge-on-write表上有持续的导入失败和clone失败的情况下,FE使用内存增长过快的问题。
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Dec 2, 2024
@dataroaring dataroaring reopened this Dec 2, 2024
@doris-robot
Copy link

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40455 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ba13133966c1c11639bb6f6e6ef69604cf5c797, data reload: false

------ Round 1 ----------------------------------
q1	17576	7410	7329	7329
q2	2351	183	174	174
q3	11068	1044	1159	1044
q4	10553	778	810	778
q5	7736	2775	2800	2775
q6	232	148	145	145
q7	978	625	612	612
q8	9574	1948	1968	1948
q9	6573	6396	6407	6396
q10	6972	2285	2303	2285
q11	454	263	278	263
q12	404	218	215	215
q13	17790	2960	2986	2960
q14	231	209	209	209
q15	548	496	511	496
q16	678	600	601	600
q17	968	509	576	509
q18	7216	6524	6497	6497
q19	3100	1014	1003	1003
q20	468	209	201	201
q21	3896	3185	3068	3068
q22	1101	956	948	948
Total cold run time: 110467 ms
Total hot run time: 40455 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7411	7228	7235	7228
q2	321	231	236	231
q3	2958	2881	2863	2863
q4	2077	1830	1810	1810
q5	5711	5694	5741	5694
q6	227	143	152	143
q7	2175	1769	1771	1769
q8	3306	3577	3390	3390
q9	8893	8862	8863	8862
q10	3530	3500	3513	3500
q11	609	507	494	494
q12	812	658	601	601
q13	16608	3150	3170	3150
q14	314	264	271	264
q15	572	520	537	520
q16	710	679	666	666
q17	1845	1627	1582	1582
q18	8150	7753	7350	7350
q19	3828	1660	1572	1572
q20	2093	1874	1830	1830
q21	5475	5250	5233	5233
q22	1145	1006	1021	1006
Total cold run time: 78770 ms
Total hot run time: 59758 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195474 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ba13133966c1c11639bb6f6e6ef69604cf5c797, data reload: false

query1	1256	912	919	912
query2	6226	2135	2067	2067
query3	10840	4068	4094	4068
query4	66149	29318	23392	23392
query5	5220	453	447	447
query6	461	170	171	170
query7	5672	317	329	317
query8	321	235	239	235
query9	9505	2690	2671	2671
query10	500	274	266	266
query11	17699	15127	15833	15127
query12	166	100	101	100
query13	1539	446	428	428
query14	10680	7614	7751	7614
query15	227	178	187	178
query16	7340	544	503	503
query17	1249	585	598	585
query18	1907	364	318	318
query19	224	175	164	164
query20	120	116	114	114
query21	222	104	108	104
query22	4474	4585	4350	4350
query23	35044	34130	34219	34130
query24	6268	3018	2878	2878
query25	559	460	433	433
query26	673	176	179	176
query27	1941	299	311	299
query28	4383	2571	2552	2552
query29	684	482	463	463
query30	249	173	170	170
query31	1016	824	837	824
query32	71	57	55	55
query33	472	289	283	283
query34	911	505	507	505
query35	844	747	729	729
query36	1067	932	963	932
query37	118	72	73	72
query38	4067	4089	4017	4017
query39	1604	1478	1462	1462
query40	219	103	100	100
query41	52	50	47	47
query42	108	102	108	102
query43	549	503	502	502
query44	1173	809	803	803
query45	183	170	178	170
query46	1176	735	760	735
query47	2000	1884	1870	1870
query48	479	391	380	380
query49	733	393	385	385
query50	827	428	421	421
query51	7097	7346	7140	7140
query52	104	86	83	83
query53	259	180	185	180
query54	573	448	452	448
query55	79	75	74	74
query56	253	239	228	228
query57	1190	1078	1113	1078
query58	216	224	206	206
query59	3128	2958	2930	2930
query60	278	258	247	247
query61	107	110	111	110
query62	773	670	662	662
query63	212	195	209	195
query64	1765	660	637	637
query65	3268	3205	3186	3186
query66	715	301	299	299
query67	15852	15295	15210	15210
query68	4502	554	552	552
query69	426	267	263	263
query70	1149	1135	1146	1135
query71	360	258	266	258
query72	6510	2627	3753	2627
query73	754	339	341	339
query74	10191	8950	8817	8817
query75	3327	2647	2631	2631
query76	1923	1047	1193	1047
query77	529	272	266	266
query78	10810	9782	9614	9614
query79	8343	591	586	586
query80	2190	430	418	418
query81	544	238	239	238
query82	1298	114	114	114
query83	284	145	141	141
query84	287	78	83	78
query85	1215	306	293	293
query86	464	303	288	288
query87	4543	4147	4221	4147
query88	5706	2407	2495	2407
query89	538	284	294	284
query90	2094	188	184	184
query91	178	146	144	144
query92	68	49	47	47
query93	6595	545	542	542
query94	882	286	293	286
query95	354	255	251	251
query96	619	279	281	279
query97	3366	3174	3175	3174
query98	219	202	195	195
query99	1605	1331	1291	1291
Total cold run time: 337209 ms
Total hot run time: 195474 ms

@bobhan1
Copy link
Contributor

bobhan1 commented Dec 3, 2024

run performance

@dataroaring dataroaring merged commit 66b9b8f into branch-3.0 Dec 3, 2024
12 checks passed
@github-actions github-actions bot deleted the auto-pick-44846-branch-3.0 branch December 3, 2024 01:57
@doris-robot
Copy link

TPC-H: Total hot run time: 40466 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ba13133966c1c11639bb6f6e6ef69604cf5c797, data reload: false

------ Round 1 ----------------------------------
q1	17570	7375	7335	7335
q2	2054	186	167	167
q3	10643	1053	1171	1053
q4	10558	746	813	746
q5	7751	2776	2798	2776
q6	234	146	143	143
q7	951	602	593	593
q8	9352	1879	2010	1879
q9	6472	6370	6376	6370
q10	6974	2286	2291	2286
q11	455	266	264	264
q12	404	214	200	200
q13	17768	2949	2979	2949
q14	251	211	208	208
q15	565	521	515	515
q16	666	596	610	596
q17	959	558	563	558
q18	7024	6537	6581	6537
q19	6386	1047	1073	1047
q20	480	203	204	203
q21	3861	3111	3049	3049
q22	1070	1014	992	992
Total cold run time: 112448 ms
Total hot run time: 40466 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7409	7295	7209	7209
q2	322	235	241	235
q3	3016	2907	2908	2907
q4	2060	1736	1747	1736
q5	5672	5679	5721	5679
q6	228	146	145	145
q7	2190	1755	1768	1755
q8	3347	3428	3417	3417
q9	8906	8813	8798	8798
q10	3533	3459	3494	3459
q11	586	483	497	483
q12	839	594	598	594
q13	16538	3124	3146	3124
q14	320	273	288	273
q15	560	520	521	520
q16	711	690	654	654
q17	1822	1611	1614	1611
q18	8198	7703	7633	7633
q19	1679	1552	1506	1506
q20	2031	1846	1795	1795
q21	5474	5291	5391	5291
q22	1097	1025	1017	1017
Total cold run time: 76538 ms
Total hot run time: 59841 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195948 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ba13133966c1c11639bb6f6e6ef69604cf5c797, data reload: false

query1	1219	931	903	903
query2	6256	2082	2014	2014
query3	10869	3987	4009	3987
query4	66184	29302	23330	23330
query5	5403	438	456	438
query6	440	176	202	176
query7	5654	319	322	319
query8	321	231	223	223
query9	9231	2676	2667	2667
query10	520	279	253	253
query11	17863	15225	15768	15225
query12	158	108	111	108
query13	1540	433	424	424
query14	9906	7256	7263	7256
query15	221	183	175	175
query16	7312	538	437	437
query17	1133	572	585	572
query18	1924	320	303	303
query19	209	146	146	146
query20	115	107	112	107
query21	203	104	105	104
query22	4489	4579	4400	4400
query23	34625	33928	34604	33928
query24	6045	2925	2890	2890
query25	533	401	399	399
query26	680	171	170	170
query27	1899	296	293	293
query28	4553	2548	2531	2531
query29	668	450	419	419
query30	249	166	167	166
query31	1018	807	817	807
query32	64	53	53	53
query33	420	275	286	275
query34	903	487	497	487
query35	832	747	748	747
query36	1097	925	959	925
query37	118	71	70	70
query38	4074	4009	3940	3940
query39	1505	1454	1613	1454
query40	212	102	101	101
query41	52	53	49	49
query42	114	102	100	100
query43	535	513	514	513
query44	1170	829	808	808
query45	200	168	168	168
query46	1141	730	728	728
query47	2021	1911	1895	1895
query48	459	385	385	385
query49	768	406	403	403
query50	834	416	420	416
query51	7318	7069	7067	7067
query52	94	87	86	86
query53	256	184	183	183
query54	562	461	461	461
query55	80	78	76	76
query56	263	235	249	235
query57	1168	1119	1101	1101
query58	209	209	211	209
query59	3203	3184	3068	3068
query60	293	281	265	265
query61	139	145	105	105
query62	760	650	646	646
query63	205	185	192	185
query64	1748	649	628	628
query65	3246	3157	3140	3140
query66	625	289	297	289
query67	15715	15589	15865	15589
query68	5070	555	569	555
query69	403	270	245	245
query70	1168	1123	1109	1109
query71	390	248	257	248
query72	6467	3837	4077	3837
query73	780	347	337	337
query74	10126	8992	8835	8835
query75	3358	2703	2649	2649
query76	2248	1077	1082	1077
query77	507	254	269	254
query78	10675	9682	9421	9421
query79	8351	586	587	586
query80	2090	418	424	418
query81	558	243	244	243
query82	1042	111	117	111
query83	292	145	141	141
query84	296	90	77	77
query85	1734	312	286	286
query86	487	270	301	270
query87	4526	4137	4196	4137
query88	5524	2420	2435	2420
query89	539	294	286	286
query90	2164	192	186	186
query91	178	147	168	147
query92	66	47	46	46
query93	6537	548	545	545
query94	999	290	284	284
query95	344	254	256	254
query96	635	280	278	278
query97	3501	3151	3199	3151
query98	220	200	201	200
query99	1987	1295	1312	1295
Total cold run time: 337105 ms
Total hot run time: 195948 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants