Skip to content

branch-3.1: [Bug](routine load) Fix routine load task failed with MEM_LIMIT_EXCEED never be scheduled again #55481 #55616

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Sep 4, 2025
Merged

branch-3.1: [Bug](routine load) Fix routine load task failed with MEM_LIMIT_EXCEED never be scheduled again #55481 #55616

morrySnow merged 1 commit into from
Sep 4, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Sep 3, 2025

Cherry-picked from #55481

@xy720
[Bug](routine load) Fix routine load task failed with MEM_LIMIT_EXCEE…
b163eb8 
…D never be scheduled again (#55481)

A routine load task which submit failed with following msg will never be
scheduled again, causing the corresponding kafka partition to no longer
be consumed.

```
2025-08-29 03:42:24:errCode = 2, detailMessage = failed to send task: errCode = 2, 
detailMessage = failed to submit task. error code: MEM_LIMIT_EXCEEDED, 
msg: (10.0.72.47)[MEM_LIMIT_EXCEEDED]
is_exceed_soft_mem_limit: 0 
current_load_mem_value: 114837740081 
_load_mem_limit: 112722727424
```
@github-actions github-actions bot requested a review from morrySnow as a code owner September 3, 2025 04:19
@Thearas
Copy link
Contributor

Thearas commented Sep 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Sep 3, 2025
@dataroaring dataroaring reopened this Sep 3, 2025
@Thearas
Copy link
Contributor

Thearas commented Sep 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 3, 2025

TPC-H: Total hot run time: 32722 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b163eb82cce87e8505743e5a8993cb5e7a42a891, data reload: false

------ Round 1 ----------------------------------
q1	17582	5475	5448	5448
q2	2018	414	292	292
q3	12666	1284	736	736
q4	10577	873	459	459
q5	9708	2407	2144	2144
q6	199	162	129	129
q7	895	764	611	611
q8	9354	1419	1169	1169
q9	5280	4990	4946	4946
q10	6773	2303	1792	1792
q11	489	284	271	271
q12	346	366	211	211
q13	17776	3567	3006	3006
q14	225	230	210	210
q15	526	458	473	458
q16	425	437	375	375
q17	621	860	359	359
q18	6866	6370	6367	6367
q19	1209	962	559	559
q20	330	342	207	207
q21	2960	2171	1968	1968
q22	1078	1050	1005	1005
Total cold run time: 107903 ms
Total hot run time: 32722 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5543	5503	5514	5503
q2	235	326	236	236
q3	2210	2660	2315	2315
q4	1424	1851	1394	1394
q5	4414	5101	5005	5005
q6	167	158	128	128
q7	2098	1962	1803	1803
q8	2669	2912	2721	2721
q9	7314	7239	7210	7210
q10	3020	3206	2796	2796
q11	577	519	489	489
q12	716	772	589	589
q13	3420	3783	3155	3155
q14	274	319	263	263
q15	519	479	467	467
q16	451	486	425	425
q17	1246	1746	1262	1262
q18	7696	7513	7339	7339
q19	840	1119	1126	1119
q20	2030	2076	1898	1898
q21	5396	4930	4525	4525
q22	1065	1087	1022	1022
Total cold run time: 53324 ms
Total hot run time: 51664 ms

@doris-robot
Copy link

doris-robot commented Sep 3, 2025

TPC-DS: Total hot run time: 192901 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b163eb82cce87e8505743e5a8993cb5e7a42a891, data reload: false

query1	938	389	387	387
query2	6222	1919	1886	1886
query3	8690	208	210	208
query4	33583	24133	23505	23505
query5	3924	603	461	461
query6	301	197	182	182
query7	4195	509	331	331
query8	335	248	229	229
query9	9441	2606	2598	2598
query10	446	312	255	255
query11	18455	15473	15303	15303
query12	165	105	103	103
query13	1540	551	402	402
query14	9007	7187	7344	7187
query15	237	189	183	183
query16	8069	704	522	522
query17	1577	775	588	588
query18	2141	425	339	339
query19	227	186	163	163
query20	130	126	122	122
query21	209	128	109	109
query22	4531	4635	4446	4446
query23	35420	34418	34057	34057
query24	7688	2734	2703	2703
query25	545	517	432	432
query26	1170	282	170	170
query27	2213	493	372	372
query28	5511	2249	2228	2228
query29	774	645	496	496
query30	241	189	166	166
query31	1010	913	855	855
query32	90	61	64	61
query33	519	381	366	366
query34	776	860	532	532
query35	776	798	756	756
query36	1048	1048	939	939
query37	101	90	67	67
query38	4053	3951	3966	3951
query39	1684	1467	1499	1467
query40	206	115	105	105
query41	50	51	54	51
query42	128	120	114	114
query43	509	511	483	483
query44	1363	830	845	830
query45	180	178	171	171
query46	883	1061	693	693
query47	1941	1986	1927	1927
query48	420	433	360	360
query49	786	508	423	423
query50	683	703	436	436
query51	7412	7372	7243	7243
query52	103	100	98	98
query53	243	267	200	200
query54	568	575	494	494
query55	80	76	79	76
query56	271	269	256	256
query57	1259	1269	1234	1234
query58	234	227	218	218
query59	3099	3161	3083	3083
query60	314	281	269	269
query61	116	118	116	116
query62	795	760	719	719
query63	237	192	201	192
query64	4497	1021	647	647
query65	3374	3336	3327	3327
query66	978	410	307	307
query67	16380	16175	15624	15624
query68	7727	829	551	551
query69	477	298	268	268
query70	1178	1167	1137	1137
query71	385	302	259	259
query72	5879	3764	3774	3764
query73	645	741	359	359
query74	10581	9276	8798	8798
query75	3279	3122	2666	2666
query76	3182	1161	783	783
query77	732	373	292	292
query78	10379	10340	9588	9588
query79	3647	873	599	599
query80	784	531	439	439
query81	482	263	219	219
query82	577	118	91	91
query83	156	161	146	146
query84	241	98	83	83
query85	788	352	323	323
query86	385	318	277	277
query87	4284	4323	4261	4261
query88	5188	2442	2423	2423
query89	402	335	303	303
query90	1799	191	188	188
query91	140	147	121	121
query92	70	56	49	49
query93	2505	926	555	555
query94	672	418	299	299
query95	342	276	275	275
query96	499	612	285	285
query97	3170	3262	3151	3151
query98	222	207	201	201
query99	1595	1415	1325	1325
Total cold run time: 297483 ms
Total hot run time: 192901 ms

@doris-robot
Copy link

doris-robot commented Sep 3, 2025

ClickBench: Total hot run time: 29.15 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b163eb82cce87e8505743e5a8993cb5e7a42a891, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.06	0.07
query4	1.62	0.11	0.10
query5	0.51	0.50	0.53
query6	1.14	0.74	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.50
query10	0.55	0.56	0.55
query11	0.14	0.10	0.10
query12	0.13	0.12	0.11
query13	0.62	0.59	0.59
query14	0.78	0.78	0.79
query15	0.86	0.83	0.83
query16	0.39	0.38	0.38
query17	1.03	1.06	0.98
query18	0.24	0.23	0.23
query19	1.96	1.86	1.85
query20	0.02	0.01	0.01
query21	15.41	0.89	0.59
query22	0.73	0.85	0.57
query23	15.14	1.43	0.53
query24	3.36	2.55	1.24
query25	0.21	0.32	0.08
query26	0.25	0.14	0.14
query27	0.04	0.03	0.06
query28	14.00	0.95	0.43
query29	12.56	3.89	3.25
query30	0.25	0.09	0.06
query31	2.83	0.60	0.39
query32	3.22	0.54	0.46
query33	2.97	3.02	3.09
query34	16.64	5.22	4.56
query35	4.55	4.61	4.65
query36	0.65	0.51	0.49
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.26 s
Total hot run time: 29.15 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 3, 2025

FE UT Coverage Report

Increment line coverage 0.00% (0/4) 🎉
Increment coverage report
Complete coverage report

@xy720
Copy link
Member

xy720 commented Sep 3, 2025

run cloud_p0

@xy720
Copy link
Member

xy720 commented Sep 3, 2025

run p0

@xy720
Copy link
Member

xy720 commented Sep 3, 2025

run Non_Concurrent

@xy720
Copy link
Member

xy720 commented Sep 3, 2025

run NonConcurrent

@morrySnow
Copy link
Contributor

morrySnow commented Sep 4, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 4, 2025

TPC-H: Total hot run time: 32839 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b163eb82cce87e8505743e5a8993cb5e7a42a891, data reload: false

------ Round 1 ----------------------------------
q1	17844	5565	5576	5565
q2	2024	389	288	288
q3	11931	1266	749	749
q4	10277	894	456	456
q5	9001	2424	2160	2160
q6	182	164	132	132
q7	921	747	618	618
q8	9332	1452	1210	1210
q9	5276	5034	4904	4904
q10	6794	2294	1828	1828
q11	469	284	275	275
q12	348	356	211	211
q13	17771	3608	3022	3022
q14	229	227	208	208
q15	535	470	469	469
q16	427	428	368	368
q17	602	894	356	356
q18	7054	6360	6350	6350
q19	1218	979	561	561
q20	330	325	197	197
q21	2863	2202	1949	1949
q22	1063	1031	963	963
Total cold run time: 106491 ms
Total hot run time: 32839 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5584	5578	5834	5578
q2	236	326	236	236
q3	2235	2636	2324	2324
q4	1325	1769	1373	1373
q5	4454	5028	4994	4994
q6	170	160	132	132
q7	2141	1949	1836	1836
q8	2640	2879	2725	2725
q9	7302	7290	7180	7180
q10	3061	3243	2700	2700
q11	591	513	496	496
q12	695	756	631	631
q13	3369	3859	3183	3183
q14	275	304	275	275
q15	536	464	473	464
q16	426	475	457	457
q17	1254	1744	1258	1258
q18	7588	7418	7372	7372
q19	806	1101	1141	1101
q20	2028	2063	1887	1887
q21	5465	4986	4536	4536
q22	1049	1067	1018	1018
Total cold run time: 53230 ms
Total hot run time: 51756 ms

@doris-robot
Copy link

doris-robot commented Sep 4, 2025

TPC-DS: Total hot run time: 192043 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b163eb82cce87e8505743e5a8993cb5e7a42a891, data reload: false

query1	954	394	391	391
query2	6212	1997	1898	1898
query3	8698	202	198	198
query4	33042	23884	23441	23441
query5	3740	611	457	457
query6	276	193	186	186
query7	4196	494	308	308
query8	291	254	240	240
query9	9539	2610	2602	2602
query10	491	328	263	263
query11	18118	15470	15216	15216
query12	159	110	110	110
query13	1548	548	438	438
query14	9396	6632	6635	6632
query15	213	205	183	183
query16	7951	665	539	539
query17	1577	761	607	607
query18	2088	430	329	329
query19	197	185	184	184
query20	127	121	118	118
query21	218	128	108	108
query22	4443	4520	4339	4339
query23	35272	33817	33879	33817
query24	7805	2730	2680	2680
query25	500	463	433	433
query26	1158	275	175	175
query27	2609	473	349	349
query28	5877	2239	2202	2202
query29	589	595	453	453
query30	246	185	165	165
query31	968	927	852	852
query32	100	62	64	62
query33	497	385	315	315
query34	762	875	546	546
query35	794	849	721	721
query36	1030	1079	959	959
query37	105	95	70	70
query38	4091	4046	4019	4019
query39	1532	1506	1476	1476
query40	213	124	110	110
query41	55	53	51	51
query42	139	112	106	106
query43	505	524	480	480
query44	1396	832	872	832
query45	207	192	178	178
query46	906	1087	685	685
query47	1970	1995	1897	1897
query48	426	446	355	355
query49	805	550	435	435
query50	700	681	446	446
query51	7305	7366	7208	7208
query52	115	116	96	96
query53	243	270	203	203
query54	577	586	496	496
query55	84	88	91	88
query56	305	290	278	278
query57	1263	1246	1221	1221
query58	260	227	241	227
query59	3123	3142	2967	2967
query60	314	310	272	272
query61	121	109	143	109
query62	797	747	705	705
query63	237	193	194	193
query64	4601	982	685	685
query65	3405	3301	3297	3297
query66	964	416	314	314
query67	16757	16066	15680	15680
query68	7937	870	562	562
query69	493	303	273	273
query70	1204	1095	1121	1095
query71	434	318	261	261
query72	5120	3786	3837	3786
query73	658	764	360	360
query74	10226	9203	9303	9203
query75	3536	3163	2714	2714
query76	3543	1224	788	788
query77	763	393	284	284
query78	10339	10523	9494	9494
query79	3393	930	603	603
query80	755	524	428	428
query81	498	263	225	225
query82	341	121	91	91
query83	167	167	147	147
query84	290	104	85	85
query85	731	360	298	298
query86	352	322	291	291
query87	4296	4344	4210	4210
query88	3390	2490	2436	2436
query89	439	341	296	296
query90	2074	195	194	194
query91	137	157	108	108
query92	72	54	53	53
query93	2126	918	548	548
query94	662	420	305	305
query95	345	283	270	270
query96	503	623	285	285
query97	3191	3293	3130	3130
query98	222	228	204	204
query99	1499	1399	1321	1321
Total cold run time: 294938 ms
Total hot run time: 192043 ms

@doris-robot
Copy link

doris-robot commented Sep 4, 2025

ClickBench: Total hot run time: 28.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b163eb82cce87e8505743e5a8993cb5e7a42a891, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.11
query5	0.54	0.52	0.53
query6	1.13	0.74	0.73
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.59	0.52	0.49
query10	0.56	0.53	0.58
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	0.78	0.80	0.81
query15	0.84	0.83	0.85
query16	0.41	0.37	0.38
query17	1.04	1.04	0.98
query18	0.23	0.21	0.23
query19	1.95	1.90	1.80
query20	0.02	0.02	0.01
query21	15.38	0.89	0.58
query22	0.74	0.78	0.61
query23	15.16	1.48	0.53
query24	3.44	1.16	0.96
query25	0.23	0.08	0.05
query26	0.25	0.15	0.13
query27	0.05	0.05	0.06
query28	13.72	1.01	0.45
query29	12.57	3.90	3.23
query30	0.26	0.09	0.06
query31	2.82	0.61	0.39
query32	3.23	0.55	0.47
query33	3.00	3.01	3.05
query34	16.49	5.20	4.58
query35	4.52	4.55	4.54
query36	0.62	0.51	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 103.97 s
Total hot run time: 28.76 s

@morrySnow morrySnow merged commit 64d71e1 into branch-3.1 Sep 4, 2025
23 checks passed
@github-actions github-actions bot deleted the auto-pick-55481-branch-3.1 branch September 4, 2025 09:19
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Thearas @doris-robot @hello-stephen @xy720 @morrySnow @dataroaring