Skip to content

Conversation

@github-actions
Copy link
Contributor

Cherry-picked from #47074

![image](https://github.com/user-attachments/assets/ad83cf90-e148-414c-9278-2cad8f2cd9ed)
The Memory of fe observer node increase over time and cannot
significantly reduce after tigger full GC.


![image](https://github.com/user-attachments/assets/ac607b0c-73d1-44bc-a0ef-1a0e91c910fe)
In follower node, when bulk load job replay, A key-value(jobId,-job)
will be added to` CallBackFactory` to record the job but will not
`removeCallback`, which cause memory leak in FE observer node.
@github-actions github-actions bot requested a review from dataroaring as a code owner January 20, 2025 07:20
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jan 20, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40831 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f421b434282b54df3a2b50b9d4dba8c6ae4fecca, data reload: false

------ Round 1 ----------------------------------
q1	17601	7515	7297	7297
q2	2072	183	170	170
q3	10962	1080	1184	1080
q4	10545	746	692	692
q5	7759	2862	2860	2860
q6	235	152	151	151
q7	959	618	620	618
q8	9352	1934	2026	1934
q9	6489	6396	6363	6363
q10	7012	2283	2323	2283
q11	464	264	258	258
q12	403	219	221	219
q13	17793	2983	3016	2983
q14	241	208	216	208
q15	567	523	517	517
q16	689	629	628	628
q17	984	564	536	536
q18	7363	6673	6704	6673
q19	1402	1117	1000	1000
q20	474	207	203	203
q21	3999	3401	3169	3169
q22	1115	989	1005	989
Total cold run time: 108480 ms
Total hot run time: 40831 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7379	7188	7298	7188
q2	336	233	235	233
q3	2944	3007	2986	2986
q4	2076	1842	1814	1814
q5	5726	5763	5788	5763
q6	223	143	144	143
q7	2271	1828	1845	1828
q8	3347	3570	3567	3567
q9	8847	8912	8890	8890
q10	3622	3574	3522	3522
q11	612	517	503	503
q12	792	596	583	583
q13	9483	3169	3172	3169
q14	296	278	279	278
q15	577	534	531	531
q16	727	692	702	692
q17	1824	1642	1634	1634
q18	8329	7666	7725	7666
q19	1659	1519	1517	1517
q20	2064	1872	1901	1872
q21	5688	5413	5292	5292
q22	1173	1082	1044	1044
Total cold run time: 69995 ms
Total hot run time: 60715 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197747 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f421b434282b54df3a2b50b9d4dba8c6ae4fecca, data reload: false

query1	1269	915	928	915
query2	6237	2065	2030	2030
query3	10990	4453	4333	4333
query4	62112	29399	23504	23504
query5	5214	474	445	445
query6	414	180	203	180
query7	5472	326	310	310
query8	306	220	231	220
query9	8351	2675	2649	2649
query10	457	274	258	258
query11	17668	15140	15781	15140
query12	167	107	104	104
query13	1446	448	433	433
query14	10566	7171	7606	7171
query15	205	193	186	186
query16	7325	499	520	499
query17	1155	624	609	609
query18	2007	337	315	315
query19	214	165	168	165
query20	116	107	108	107
query21	201	104	109	104
query22	4581	4745	4526	4526
query23	34807	34154	34505	34154
query24	6153	2909	2888	2888
query25	553	449	433	433
query26	673	177	179	177
query27	1930	354	367	354
query28	4229	2527	2502	2502
query29	688	443	419	419
query30	249	165	168	165
query31	1021	803	828	803
query32	71	55	57	55
query33	432	299	294	294
query34	910	521	528	521
query35	841	739	752	739
query36	1066	981	933	933
query37	120	72	78	72
query38	4130	4005	4081	4005
query39	1512	1472	1466	1466
query40	200	102	101	101
query41	48	49	46	46
query42	118	101	101	101
query43	547	495	509	495
query44	1226	821	834	821
query45	182	166	168	166
query46	1134	733	743	733
query47	2030	1914	1949	1914
query48	469	376	392	376
query49	747	386	387	386
query50	846	431	436	431
query51	7445	7180	7342	7180
query52	109	89	94	89
query53	272	188	192	188
query54	605	466	475	466
query55	82	79	80	79
query56	267	245	248	245
query57	1272	1181	1166	1166
query58	229	215	214	214
query59	3320	3105	2879	2879
query60	274	255	257	255
query61	107	108	139	108
query62	767	671	657	657
query63	217	189	193	189
query64	1416	671	634	634
query65	3289	3222	3219	3219
query66	713	296	299	296
query67	16076	15626	15603	15603
query68	4089	584	550	550
query69	443	267	264	264
query70	1210	1126	1124	1124
query71	351	277	259	259
query72	6341	4030	4119	4030
query73	749	347	359	347
query74	10392	8942	9039	8942
query75	3361	2641	2669	2641
query76	1945	1123	1007	1007
query77	526	271	278	271
query78	10605	9563	9668	9563
query79	2235	607	614	607
query80	1374	424	430	424
query81	525	240	238	238
query82	1244	122	121	121
query83	275	150	143	143
query84	282	89	85	85
query85	1034	313	296	296
query86	385	297	307	297
query87	4582	4252	4406	4252
query88	3862	2397	2362	2362
query89	419	291	287	287
query90	2030	186	186	186
query91	185	153	150	150
query92	69	49	54	49
query93	2741	549	539	539
query94	787	296	301	296
query95	356	258	256	256
query96	625	279	276	276
query97	3364	3199	3173	3173
query98	209	215	234	215
query99	1731	1297	1277	1277
Total cold run time: 318191 ms
Total hot run time: 197747 ms

@dataroaring dataroaring merged commit 5e90377 into branch-3.0 Jan 23, 2025
21 of 22 checks passed
@github-actions github-actions bot deleted the auto-pick-47074-branch-3.0 branch January 23, 2025 05:15
@gavinchou gavinchou mentioned this pull request Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants