Skip to content

branch-3.0: [Bug](routine load) Fix routine load task failed with MEM_LIMIT_EXCEED never be scheduled again #55481 #55614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Sep 5, 2025
Merged

branch-3.0: [Bug](routine load) Fix routine load task failed with MEM_LIMIT_EXCEED never be scheduled again #55481 #55614

dataroaring merged 1 commit into from
Sep 5, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Sep 3, 2025

Cherry-picked from #55481

@xy720
[Bug](routine load) Fix routine load task failed with MEM_LIMIT_EXCEE…
dade2e4 
…D never be scheduled again (#55481)

A routine load task which submit failed with following msg will never be
scheduled again, causing the corresponding kafka partition to no longer
be consumed.

```
2025-08-29 03:42:24:errCode = 2, detailMessage = failed to send task: errCode = 2, 
detailMessage = failed to submit task. error code: MEM_LIMIT_EXCEEDED, 
msg: (10.0.72.47)[MEM_LIMIT_EXCEEDED]
is_exceed_soft_mem_limit: 0 
current_load_mem_value: 114837740081 
_load_mem_limit: 112722727424
```
@dataroaring dataroaring closed this Sep 3, 2025
@dataroaring dataroaring reopened this Sep 3, 2025
@dataroaring
Copy link
Contributor

dataroaring commented Sep 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 3, 2025

TPC-H: Total hot run time: 39908 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dade2e4517613fe7af7947ab3fcc7a6a0dfce9d7, data reload: false

------ Round 1 ----------------------------------
q1	17637	7408	6688	6688
q2	2031	205	168	168
q3	10647	1150	1187	1150
q4	10479	769	721	721
q5	7700	2860	2837	2837
q6	215	132	133	132
q7	973	619	627	619
q8	9389	1970	1976	1970
q9	6652	6396	6395	6395
q10	7029	2259	2339	2259
q11	470	263	253	253
q12	391	207	213	207
q13	17782	2974	2992	2974
q14	227	206	204	204
q15	507	448	454	448
q16	475	369	378	369
q17	976	586	565	565
q18	7259	6577	6613	6577
q19	1404	1101	1044	1044
q20	481	196	202	196
q21	3907	3180	3144	3144
q22	1047	1015	988	988
Total cold run time: 107678 ms
Total hot run time: 39908 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6642	6593	6546	6546
q2	329	227	232	227
q3	2894	2935	2884	2884
q4	2047	1826	1820	1820
q5	5673	5726	5691	5691
q6	209	130	128	128
q7	2185	1746	1821	1746
q8	3400	3567	3527	3527
q9	8723	8880	8790	8790
q10	3533	3536	3487	3487
q11	602	481	492	481
q12	783	606	622	606
q13	9443	3132	3152	3132
q14	318	274	278	274
q15	509	478	484	478
q16	505	439	457	439
q17	1887	1676	1629	1629
q18	8497	8233	7686	7686
q19	1690	1559	1593	1559
q20	2112	1869	1872	1869
q21	5140	5034	5034	5034
q22	1144	1054	1037	1037
Total cold run time: 68265 ms
Total hot run time: 59070 ms

@doris-robot
Copy link

doris-robot commented Sep 3, 2025

TPC-DS: Total hot run time: 192374 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dade2e4517613fe7af7947ab3fcc7a6a0dfce9d7, data reload: false

query1	941	389	372	372
query2	6227	1904	1855	1855
query3	8687	213	204	204
query4	33676	24049	24077	24049
query5	4283	469	440	440
query6	275	189	199	189
query7	4235	316	323	316
query8	280	218	232	218
query9	9383	2588	2569	2569
query10	494	266	267	266
query11	17978	15323	15240	15240
query12	156	105	100	100
query13	1584	431	436	431
query14	8698	7429	6699	6699
query15	273	173	180	173
query16	8080	515	535	515
query17	1628	585	595	585
query18	2178	344	325	325
query19	277	163	169	163
query20	134	116	122	116
query21	206	116	109	109
query22	4698	4330	4416	4330
query23	35303	34038	34501	34038
query24	11843	2926	2870	2870
query25	703	420	439	420
query26	1719	178	182	178
query27	2892	361	368	361
query28	7702	2158	2160	2158
query29	1026	470	459	459
query30	269	167	169	167
query31	1045	817	826	817
query32	104	57	80	57
query33	744	302	302	302
query34	978	516	520	516
query35	891	735	739	735
query36	1133	944	941	941
query37	154	68	69	68
query38	4154	3952	3941	3941
query39	1514	1491	1484	1484
query40	260	118	109	109
query41	54	47	51	47
query42	119	108	108	108
query43	538	506	494	494
query44	1288	825	823	823
query45	181	165	174	165
query46	1156	722	729	722
query47	2006	1901	1916	1901
query48	488	412	390	390
query49	1139	402	396	396
query50	831	432	457	432
query51	7418	7304	7265	7265
query52	103	90	94	90
query53	264	193	192	192
query54	1346	470	484	470
query55	79	76	77	76
query56	262	253	242	242
query57	1365	1207	1192	1192
query58	230	211	216	211
query59	3223	3109	2988	2988
query60	283	268	249	249
query61	109	108	112	108
query62	852	687	706	687
query63	234	198	198	198
query64	5152	672	638	638
query65	3356	3323	3276	3276
query66	1330	330	309	309
query67	16243	15593	15827	15593
query68	4960	572	589	572
query69	441	272	265	265
query70	1171	1105	1110	1105
query71	340	260	261	260
query72	6210	4182	3738	3738
query73	789	355	353	353
query74	10365	9092	9027	9027
query75	3378	2674	2617	2617
query76	2698	955	1041	955
query77	389	299	282	282
query78	10607	9650	9597	9597
query79	2421	621	615	615
query80	993	436	440	436
query81	556	221	225	221
query82	617	88	84	84
query83	224	141	145	141
query84	240	81	81	81
query85	1914	304	298	298
query86	481	300	299	299
query87	4374	4253	4201	4201
query88	4207	2419	2393	2393
query89	421	299	299	299
query90	1951	188	187	187
query91	183	148	153	148
query92	67	51	49	49
query93	2509	561	551	551
query94	825	290	300	290
query95	366	262	260	260
query96	623	288	286	286
query97	3348	3125	3144	3125
query98	235	204	197	197
query99	1524	1295	1300	1295
Total cold run time: 305881 ms
Total hot run time: 192374 ms

@doris-robot
Copy link

doris-robot commented Sep 3, 2025

ClickBench: Total hot run time: 29.57 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dade2e4517613fe7af7947ab3fcc7a6a0dfce9d7, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.22	0.07	0.07
query4	1.63	0.10	0.10
query5	0.51	0.52	0.51
query6	1.12	0.72	0.74
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.49	0.49
query10	0.56	0.54	0.56
query11	0.14	0.10	0.10
query12	0.12	0.11	0.12
query13	0.62	0.61	0.59
query14	0.78	0.81	0.80
query15	0.84	0.83	0.83
query16	0.38	0.39	0.39
query17	0.99	1.00	1.07
query18	0.24	0.23	0.22
query19	1.98	1.80	1.87
query20	0.02	0.02	0.01
query21	15.42	0.60	0.58
query22	2.68	2.13	1.56
query23	16.85	0.86	0.82
query24	3.36	1.50	0.57
query25	0.16	0.06	0.05
query26	0.65	0.15	0.14
query27	0.04	0.04	0.03
query28	10.11	0.54	0.46
query29	12.59	3.18	3.19
query30	0.25	0.06	0.05
query31	2.86	0.39	0.39
query32	3.23	0.47	0.46
query33	3.03	3.02	3.04
query34	16.68	4.55	4.60
query35	4.59	4.55	4.59
query36	0.66	0.47	0.48
query37	0.09	0.06	0.05
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.12	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.51 s
Total hot run time: 29.57 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 3, 2025

FE UT Coverage Report

Increment line coverage 0.00% (0/4) 🎉
Increment coverage report
Complete coverage report

@xy720
Copy link
Member

xy720 commented Sep 4, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 4, 2025

TPC-H: Total hot run time: 40063 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dade2e4517613fe7af7947ab3fcc7a6a0dfce9d7, data reload: false

------ Round 1 ----------------------------------
q1	17608	7081	6755	6755
q2	2038	176	172	172
q3	10651	1156	1193	1156
q4	10466	742	779	742
q5	7759	2888	2799	2799
q6	214	131	137	131
q7	963	634	610	610
q8	9362	1963	2027	1963
q9	6606	6354	6423	6354
q10	7058	2311	2342	2311
q11	477	275	262	262
q12	393	215	217	215
q13	17778	2991	3052	2991
q14	243	214	207	207
q15	508	458	451	451
q16	483	389	379	379
q17	986	593	544	544
q18	7408	6689	6649	6649
q19	1415	1077	1059	1059
q20	474	207	202	202
q21	3861	3154	3164	3154
q22	1084	985	957	957
Total cold run time: 107835 ms
Total hot run time: 40063 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6691	6630	6627	6627
q2	333	229	239	229
q3	2944	2959	2951	2951
q4	2050	1845	1781	1781
q5	5730	5725	5694	5694
q6	216	130	132	130
q7	2231	1828	1822	1822
q8	3398	3588	3553	3553
q9	8861	8886	8797	8797
q10	3596	3553	3556	3553
q11	592	490	498	490
q12	831	597	621	597
q13	7129	3119	3180	3119
q14	297	268	283	268
q15	513	467	467	467
q16	496	449	439	439
q17	1837	1662	1623	1623
q18	8195	7734	7703	7703
q19	1659	1645	1520	1520
q20	2122	1891	1894	1891
q21	5241	4945	5026	4945
q22	1131	1066	999	999
Total cold run time: 66093 ms
Total hot run time: 59198 ms

@doris-robot
Copy link

doris-robot commented Sep 4, 2025

TPC-DS: Total hot run time: 192235 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dade2e4517613fe7af7947ab3fcc7a6a0dfce9d7, data reload: false

query1	941	398	370	370
query2	6267	1958	1951	1951
query3	8682	216	208	208
query4	33564	23633	23561	23561
query5	3613	459	442	442
query6	282	185	174	174
query7	4210	306	305	305
query8	292	228	234	228
query9	9578	2615	2607	2607
query10	471	254	260	254
query11	17895	15011	15220	15011
query12	167	102	105	102
query13	1572	436	423	423
query14	9583	6722	7114	6722
query15	257	175	185	175
query16	8035	476	487	476
query17	1611	628	590	590
query18	2150	334	330	330
query19	274	166	174	166
query20	129	126	122	122
query21	206	111	112	111
query22	4672	4480	4466	4466
query23	34623	34756	34098	34098
query24	12155	2972	2939	2939
query25	701	425	431	425
query26	1802	178	183	178
query27	2865	349	355	349
query28	7754	2196	2206	2196
query29	1062	476	456	456
query30	269	168	164	164
query31	1037	782	827	782
query32	105	59	59	59
query33	790	303	295	295
query34	1134	531	543	531
query35	918	731	739	731
query36	1074	949	974	949
query37	272	78	69	69
query38	4080	3975	3964	3964
query39	1488	1478	1484	1478
query40	254	100	101	100
query41	50	50	47	47
query42	120	106	110	106
query43	543	494	492	492
query44	1235	827	829	827
query45	203	173	172	172
query46	1177	729	739	729
query47	2029	1890	1921	1890
query48	481	386	385	385
query49	1127	419	405	405
query50	837	444	436	436
query51	7328	7293	7148	7148
query52	101	93	91	91
query53	268	193	186	186
query54	1209	470	483	470
query55	77	77	83	77
query56	265	259	264	259
query57	1333	1236	1206	1206
query58	223	209	218	209
query59	3211	3118	2975	2975
query60	294	253	251	251
query61	109	117	133	117
query62	883	680	726	680
query63	231	199	195	195
query64	5061	652	631	631
query65	3450	3302	3309	3302
query66	1451	302	298	298
query67	16193	15511	15600	15511
query68	5608	582	567	567
query69	438	269	267	267
query70	1164	1055	1089	1055
query71	363	262	268	262
query72	6192	4106	3967	3967
query73	753	357	364	357
query74	10407	9063	9014	9014
query75	3372	2682	2677	2677
query76	3347	1100	1039	1039
query77	401	272	281	272
query78	10454	9597	9631	9597
query79	2435	602	624	602
query80	1087	431	423	423
query81	537	225	223	223
query82	618	97	86	86
query83	228	147	149	147
query84	237	84	87	84
query85	1400	296	288	288
query86	452	308	293	293
query87	4400	4206	4269	4206
query88	4110	2430	2426	2426
query89	410	289	291	289
query90	2034	191	190	190
query91	180	152	152	152
query92	60	50	48	48
query93	2369	562	556	556
query94	982	307	304	304
query95	362	259	263	259
query96	639	293	291	291
query97	3339	3121	3166	3121
query98	226	206	202	202
query99	1511	1319	1322	1319
Total cold run time: 306438 ms
Total hot run time: 192235 ms

@doris-robot
Copy link

doris-robot commented Sep 4, 2025

ClickBench: Total hot run time: 29.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dade2e4517613fe7af7947ab3fcc7a6a0dfce9d7, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.63	0.11	0.11
query5	0.52	0.54	0.51
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.52
query10	0.56	0.57	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	0.76	0.83	0.79
query15	0.84	0.83	0.82
query16	0.38	0.41	0.41
query17	1.01	1.05	1.05
query18	0.24	0.22	0.22
query19	1.85	1.82	1.79
query20	0.01	0.01	0.01
query21	15.40	0.62	0.62
query22	2.32	1.85	2.37
query23	16.91	0.95	0.92
query24	3.21	0.62	0.38
query25	0.12	0.10	0.18
query26	0.37	0.15	0.13
query27	0.04	0.04	0.04
query28	11.32	0.48	0.46
query29	12.61	3.23	3.19
query30	0.25	0.06	0.06
query31	2.87	0.38	0.40
query32	3.24	0.45	0.46
query33	2.98	3.00	3.04
query34	17.05	4.52	4.57
query35	4.59	4.51	4.54
query36	0.65	0.47	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.15	0.13	0.13
query41	0.09	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.2 s
Total hot run time: 29.85 s

dataroaring
dataroaring approved these changes Sep 5, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 5d37090 into branch-3.0 Sep 5, 2025
25 of 26 checks passed
@github-actions github-actions bot deleted the auto-pick-55481-branch-3.0 branch September 5, 2025 01:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@dataroaring @doris-robot @hello-stephen @xy720