[enhancement](memoryfailed) throw exception instead of log fatal during memory allocate failed #49876

yiguolei · 2025-04-08T08:28:11Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…ng memory allocate failed

Thearas · 2025-04-08T08:28:31Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

yiguolei · 2025-04-08T08:28:56Z

run buildall

mrhhsg

lgtm

yiguolei · 2025-04-08T08:35:04Z

run buildall

doris-robot · 2025-04-08T09:00:17Z

TPC-H: Total hot run time: 34204 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7b688f71f20283e0a021003d24f43e41e0377a2b, data reload: false

------ Round 1 ----------------------------------
q1	25964	5524	5081	5081
q2	2071	276	184	184
q3	10388	1247	679	679
q4	10225	986	519	519
q5	7522	2400	2279	2279
q6	190	162	134	134
q7	924	734	613	613
q8	9323	1290	1044	1044
q9	6935	5147	5150	5147
q10	6877	2326	1876	1876
q11	492	280	264	264
q12	353	358	219	219
q13	17803	3678	3128	3128
q14	239	226	211	211
q15	526	491	488	488
q16	614	623	589	589
q17	582	862	353	353
q18	7636	7239	7212	7212
q19	1880	975	559	559
q20	334	342	223	223
q21	4003	2640	2432	2432
q22	1082	1019	970	970
Total cold run time: 115963 ms
Total hot run time: 34204 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5290	5123	5095	5095
q2	240	327	260	260
q3	2182	2604	2307	2307
q4	1389	1821	1471	1471
q5	4455	4404	4372	4372
q6	219	165	130	130
q7	1958	1914	1764	1764
q8	2587	2587	2489	2489
q9	7278	7234	7169	7169
q10	3008	3209	2732	2732
q11	585	511	479	479
q12	667	752	617	617
q13	3586	3896	3381	3381
q14	272	304	275	275
q15	559	499	485	485
q16	625	699	640	640
q17	1139	1544	1360	1360
q18	7752	7589	7510	7510
q19	839	789	910	789
q20	1939	1992	1864	1864
q21	5390	4841	4842	4841
q22	1130	1054	1026	1026
Total cold run time: 53089 ms
Total hot run time: 51056 ms

doris-robot · 2025-04-08T09:12:33Z

TPC-DS: Total hot run time: 192377 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7b688f71f20283e0a021003d24f43e41e0377a2b, data reload: false

query1	1371	1065	1045	1045
query2	6155	1920	1875	1875
query3	11178	4636	4469	4469
query4	53190	25684	23061	23061
query5	5120	503	447	447
query6	346	204	184	184
query7	4884	506	284	284
query8	322	257	240	240
query9	5638	2599	2621	2599
query10	441	319	264	264
query11	15161	15090	14825	14825
query12	165	112	110	110
query13	1049	512	406	406
query14	10192	6266	6533	6266
query15	188	192	187	187
query16	7069	652	527	527
query17	1121	777	613	613
query18	1537	430	344	344
query19	204	195	173	173
query20	129	121	127	121
query21	210	127	110	110
query22	4533	4296	4374	4296
query23	33872	33095	33035	33035
query24	6332	2427	2409	2409
query25	482	461	402	402
query26	720	269	153	153
query27	2147	499	348	348
query28	3137	2491	2458	2458
query29	626	570	438	438
query30	295	232	200	200
query31	875	860	817	817
query32	76	62	62	62
query33	455	360	311	311
query34	780	863	530	530
query35	790	832	771	771
query36	993	976	910	910
query37	125	103	75	75
query38	4246	4233	4130	4130
query39	1510	1441	1463	1441
query40	208	121	125	121
query41	53	53	53	53
query42	127	117	111	111
query43	480	522	476	476
query44	1330	821	820	820
query45	182	182	176	176
query46	853	1036	646	646
query47	1842	1888	1802	1802
query48	393	421	317	317
query49	683	506	416	416
query50	667	701	418	418
query51	4301	4308	4219	4219
query52	112	117	102	102
query53	241	271	192	192
query54	581	574	537	537
query55	84	87	81	81
query56	325	311	296	296
query57	1142	1225	1117	1117
query58	264	267	268	267
query59	2691	2881	2695	2695
query60	329	324	315	315
query61	137	128	129	128
query62	753	742	670	670
query63	225	185	185	185
query64	1786	1086	741	741
query65	4416	4330	4309	4309
query66	768	405	303	303
query67	15938	15706	15159	15159
query68	7275	824	511	511
query69	523	304	264	264
query70	1255	1051	1080	1051
query71	495	322	299	299
query72	6125	4881	5003	4881
query73	1422	732	353	353
query74	9323	9262	8833	8833
query75	3884	3218	2685	2685
query76	4246	1197	754	754
query77	636	384	277	277
query78	10096	9999	9265	9265
query79	3742	814	562	562
query80	712	522	439	439
query81	497	259	221	221
query82	491	127	92	92
query83	360	256	233	233
query84	299	107	83	83
query85	810	343	307	307
query86	383	322	279	279
query87	4437	4542	4437	4437
query88	3482	2246	2206	2206
query89	444	317	277	277
query90	1898	218	208	208
query91	140	154	114	114
query92	75	62	56	56
query93	2963	923	577	577
query94	669	419	315	315
query95	369	296	283	283
query96	488	558	277	277
query97	3307	3226	3122	3122
query98	235	201	207	201
query99	1435	1424	1307	1307
Total cold run time: 301091 ms
Total hot run time: 192377 ms

doris-robot · 2025-04-08T09:17:51Z

ClickBench: Total hot run time: 30.7 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7b688f71f20283e0a021003d24f43e41e0377a2b, data reload: false

query1	0.03	0.03	0.03
query2	0.12	0.10	0.10
query3	0.25	0.19	0.19
query4	1.59	0.19	0.19
query5	0.59	0.57	0.59
query6	1.18	0.73	0.72
query7	0.03	0.01	0.01
query8	0.04	0.04	0.03
query9	0.58	0.51	0.51
query10	0.58	0.59	0.57
query11	0.15	0.10	0.11
query12	0.15	0.11	0.11
query13	0.61	0.60	0.59
query14	2.69	2.80	2.82
query15	0.95	0.86	0.85
query16	0.40	0.39	0.40
query17	1.05	1.02	1.03
query18	0.21	0.20	0.19
query19	1.93	1.96	1.81
query20	0.02	0.01	0.02
query21	15.35	0.88	0.54
query22	0.75	1.13	0.63
query23	15.01	1.37	0.63
query24	7.46	0.99	0.36
query25	0.38	0.28	0.08
query26	0.64	0.16	0.12
query27	0.06	0.05	0.04
query28	9.32	0.94	0.45
query29	12.53	3.95	3.27
query30	0.25	0.10	0.06
query31	2.82	0.59	0.39
query32	3.22	0.55	0.47
query33	3.16	3.06	3.03
query34	15.81	5.12	4.49
query35	4.59	4.59	4.55
query36	0.66	0.49	0.50
query37	0.08	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.64 s
Total hot run time: 30.7 s

hello-stephen · 2025-04-08T10:55:38Z

BE UT Coverage Report

Increment line coverage 0.00% (0/3) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.26% (14032/26849)
Line Coverage	41.04% (121082/295018)
Region Coverage	39.79% (61632/154908)
Branch Coverage	34.46% (30852/89542)

zclllyybb · 2025-04-08T11:11:46Z

be/src/vec/aggregate_functions/aggregate_function_sort.h

+        // If memory allocate failed during deserialize, st is not ok, throw exception here to
+        // stop the query.
+        if (!st.ok()) {
+            throw doris::Exception(ErrorCode::INTERNAL_ERROR, st.to_string());


we have more proper ErrorCode MemoryLimitExceeded

throw doris::Exception(st.code(), st.to_string());

HappenLee

LGTM

github-actions · 2025-04-08T11:12:38Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-04-08T11:12:40Z

PR approved by anyone and no changes requested.

github-actions · 2025-04-08T11:19:34Z

PR approved by at least one committer and no changes requested.

HappenLee

LGTM

yiguolei · 2025-04-09T03:30:24Z

run buildall

doris-robot · 2025-04-09T03:49:27Z

TPC-H: Total hot run time: 34229 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eec0e3391cfba5c24dcceb7f3f3db55cdf897055, data reload: false

------ Round 1 ----------------------------------
q1	26096	5117	5010	5010
q2	2062	276	180	180
q3	10394	1276	685	685
q4	10229	1011	533	533
q5	7532	2770	2337	2337
q6	193	162	131	131
q7	925	741	598	598
q8	9315	1285	1125	1125
q9	6850	5178	5175	5175
q10	6811	2307	1875	1875
q11	474	283	267	267
q12	339	358	216	216
q13	17762	3687	3103	3103
q14	236	231	221	221
q15	535	495	492	492
q16	626	614	583	583
q17	609	862	389	389
q18	7560	7336	7103	7103
q19	1099	952	561	561
q20	360	345	230	230
q21	4537	3401	2491	2491
q22	1044	990	924	924
Total cold run time: 115588 ms
Total hot run time: 34229 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5169	5121	5126	5121
q2	237	327	223	223
q3	2142	2647	2310	2310
q4	1465	1813	1495	1495
q5	4521	4443	4359	4359
q6	209	165	123	123
q7	1920	1927	1741	1741
q8	2564	2646	2484	2484
q9	7225	7086	7175	7086
q10	2974	3198	2706	2706
q11	567	502	493	493
q12	713	761	609	609
q13	3498	3920	3323	3323
q14	262	295	270	270
q15	507	478	478	478
q16	647	680	672	672
q17	1222	1563	1371	1371
q18	7717	7505	7381	7381
q19	823	793	855	793
q20	2001	2044	1853	1853
q21	5122	4569	4502	4502
q22	1032	1039	963	963
Total cold run time: 52537 ms
Total hot run time: 50356 ms

doris-robot · 2025-04-09T04:01:05Z

TPC-DS: Total hot run time: 186589 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eec0e3391cfba5c24dcceb7f3f3db55cdf897055, data reload: false

query1	1008	476	468	468
query2	6528	1967	2002	1967
query3	6743	221	225	221
query4	25908	23784	22865	22865
query5	5107	642	474	474
query6	310	226	192	192
query7	4619	486	279	279
query8	293	227	232	227
query9	8641	2550	2561	2550
query10	485	327	280	280
query11	15875	15144	14971	14971
query12	175	111	107	107
query13	1657	532	405	405
query14	9634	6238	6164	6164
query15	223	203	165	165
query16	7647	587	445	445
query17	1153	741	586	586
query18	2032	403	319	319
query19	200	196	158	158
query20	122	118	119	118
query21	211	126	107	107
query22	4272	4352	4175	4175
query23	33834	33172	33065	33065
query24	8453	2392	2445	2392
query25	534	443	395	395
query26	1235	268	156	156
query27	2720	491	325	325
query28	4335	2391	2424	2391
query29	729	565	435	435
query30	289	226	191	191
query31	948	853	770	770
query32	73	64	67	64
query33	608	370	313	313
query34	820	865	511	511
query35	789	811	743	743
query36	975	1019	881	881
query37	124	103	78	78
query38	4187	4286	4084	4084
query39	1434	1376	1375	1375
query40	212	126	110	110
query41	57	54	52	52
query42	125	103	112	103
query43	480	516	478	478
query44	1310	790	786	786
query45	177	175	165	165
query46	838	1031	628	628
query47	1752	1810	1744	1744
query48	379	418	301	301
query49	802	497	406	406
query50	669	672	398	398
query51	4110	4325	4271	4271
query52	104	108	96	96
query53	229	251	184	184
query54	571	574	494	494
query55	95	79	82	79
query56	331	299	286	286
query57	1095	1133	1102	1102
query58	274	258	276	258
query59	2628	2806	2629	2629
query60	331	313	313	313
query61	129	133	131	131
query62	806	722	654	654
query63	235	190	192	190
query64	4302	1017	693	693
query65	4312	4223	4283	4223
query66	1074	417	307	307
query67	15519	15592	15437	15437
query68	8765	896	506	506
query69	477	309	263	263
query70	1155	1164	1106	1106
query71	461	329	297	297
query72	5269	4777	4569	4569
query73	695	566	337	337
query74	9119	8940	8872	8872
query75	4071	3205	2712	2712
query76	3774	1212	767	767
query77	787	395	291	291
query78	9941	9991	9250	9250
query79	2777	810	561	561
query80	666	520	447	447
query81	477	257	229	229
query82	487	126	96	96
query83	287	252	240	240
query84	292	108	85	85
query85	765	362	316	316
query86	343	333	295	295
query87	4549	4444	4413	4413
query88	3308	2216	2209	2209
query89	396	309	281	281
query90	2006	219	222	219
query91	151	145	118	118
query92	80	61	60	60
query93	1701	956	564	564
query94	672	403	319	319
query95	381	293	287	287
query96	483	560	282	282
query97	3139	3215	3174	3174
query98	249	215	201	201
query99	1458	1400	1293	1293
Total cold run time: 277057 ms
Total hot run time: 186589 ms

doris-robot · 2025-04-09T04:06:23Z

ClickBench: Total hot run time: 31.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eec0e3391cfba5c24dcceb7f3f3db55cdf897055, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.10
query3	0.25	0.20	0.19
query4	1.59	0.19	0.18
query5	0.58	0.58	0.59
query6	1.20	0.72	0.72
query7	0.02	0.01	0.02
query8	0.03	0.04	0.04
query9	0.58	0.53	0.50
query10	0.57	0.58	0.56
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.61	0.61	0.61
query14	2.80	2.66	2.72
query15	0.91	0.86	0.86
query16	0.37	0.38	0.38
query17	1.06	1.03	1.06
query18	0.21	0.19	0.20
query19	1.93	1.91	1.78
query20	0.01	0.01	0.01
query21	15.37	0.91	0.55
query22	0.78	1.23	0.96
query23	14.70	1.41	0.61
query24	6.81	2.35	0.65
query25	0.53	0.23	0.07
query26	0.62	0.16	0.14
query27	0.05	0.06	0.04
query28	9.10	0.92	0.42
query29	12.58	3.97	3.28
query30	0.26	0.09	0.07
query31	2.83	0.61	0.38
query32	3.23	0.54	0.46
query33	3.03	3.03	3.10
query34	15.76	5.10	4.49
query35	4.51	4.54	4.54
query36	0.66	0.51	0.48
query37	0.08	0.06	0.07
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.15	0.13
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.47 s
Total hot run time: 31.15 s

hello-stephen · 2025-04-09T05:58:36Z

BE UT Coverage Report

Increment line coverage 0.00% (0/4) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.26% (14032/26849)
Line Coverage	41.02% (121034/295087)
Region Coverage	39.78% (61625/154932)
Branch Coverage	34.44% (30846/89556)

…ng memory allocate failed (#49876)

…g fatal during memory allocate failed #49876 (#49897) Cherry-picked from #49876 Co-authored-by: yiguolei <guolei@selectdb.com>

…g fatal during memory allocate failed #49876 (#49896) Cherry-picked from #49876 Co-authored-by: yiguolei <guolei@selectdb.com>

…ng memory allocate failed (apache#49876)

[enhancement](memoryfailed) throw exception instead of log fatal duri…

dd1ca5a

…ng memory allocate failed

yiguolei added usercase Important user case type label p0_c dev/2.1.x dev/3.0.x labels Apr 8, 2025

mrhhsg previously approved these changes Apr 8, 2025

View reviewed changes

f

7b688f7

yiguolei dismissed mrhhsg’s stale review via 7b688f7 April 8, 2025 08:34

zclllyybb reviewed Apr 8, 2025

View reviewed changes

HappenLee previously approved these changes Apr 8, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 8, 2025

github-actions bot added the reviewed label Apr 8, 2025

f

eec0e33

yiguolei dismissed HappenLee’s stale review via eec0e33 April 8, 2025 11:16

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 8, 2025

jacktengg approved these changes Apr 8, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 8, 2025

HappenLee approved these changes Apr 8, 2025

View reviewed changes

yiguolei merged commit 10749eb into apache:master Apr 9, 2025
23 of 27 checks passed

github-actions bot pushed a commit that referenced this pull request Apr 9, 2025

[enhancement](memoryfailed) throw exception instead of log fatal duri…

7489c39

…ng memory allocate failed (#49876)

github-actions bot mentioned this pull request Apr 9, 2025

branch-3.0: [enhancement](memoryfailed) throw exception instead of log fatal during memory allocate failed #49876 #49896

Merged

github-actions bot pushed a commit that referenced this pull request Apr 9, 2025

[enhancement](memoryfailed) throw exception instead of log fatal duri…

ba75991

…ng memory allocate failed (#49876)

github-actions bot mentioned this pull request Apr 9, 2025

branch-2.1: [enhancement](memoryfailed) throw exception instead of log fatal during memory allocate failed #49876 #49897

Merged

yiguolei added a commit that referenced this pull request Apr 9, 2025

branch-2.1: [enhancement](memoryfailed) throw exception instead of lo…

18dc820

…g fatal during memory allocate failed #49876 (#49897) Cherry-picked from #49876 Co-authored-by: yiguolei <guolei@selectdb.com>

yiguolei added dev/2.1.10-merged and removed dev/2.1.x labels Apr 9, 2025

dataroaring pushed a commit that referenced this pull request Apr 22, 2025

branch-3.0: [enhancement](memoryfailed) throw exception instead of lo…

11fdbb5

…g fatal during memory allocate failed #49876 (#49896) Cherry-picked from #49876 Co-authored-by: yiguolei <guolei@selectdb.com>

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels Apr 22, 2025

BiteTheDDDDt added the dev/2.0.x label May 28, 2025

BiteTheDDDDt mentioned this pull request May 28, 2025

[enhancement](memoryfailed) throw exception if block.serialize meet error status #51335

Merged

16 tasks

BiteTheDDDDt pushed a commit to BiteTheDDDDt/incubator-doris that referenced this pull request May 29, 2025

[enhancement](memoryfailed) throw exception instead of log fatal duri…

d46621f

…ng memory allocate failed (apache#49876)

BiteTheDDDDt mentioned this pull request May 29, 2025

[enhancement](memoryfailed) pick #49876 #51335 #51366

Closed

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025

[enhancement](memoryfailed) throw exception instead of log fatal duri…

c3d9e50

…ng memory allocate failed (apache#49876)

[enhancement](memoryfailed) throw exception instead of log fatal during memory allocate failed #49876

[enhancement](memoryfailed) throw exception instead of log fatal during memory allocate failed #49876

Uh oh!

Conversation

yiguolei commented Apr 8, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 8, 2025

Uh oh!

yiguolei commented Apr 8, 2025

Uh oh!

mrhhsg left a comment

Choose a reason for hiding this comment

Uh oh!

yiguolei commented Apr 8, 2025

Uh oh!

doris-robot commented Apr 8, 2025

Uh oh!

doris-robot commented Apr 8, 2025

Uh oh!

doris-robot commented Apr 8, 2025

Uh oh!

hello-stephen commented Apr 8, 2025

BE UT Coverage Report

Uh oh!

zclllyybb Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

jacktengg Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

yiguolei commented Apr 9, 2025

Uh oh!

doris-robot commented Apr 9, 2025

Uh oh!

doris-robot commented Apr 9, 2025

Uh oh!

doris-robot commented Apr 9, 2025

Uh oh!

hello-stephen commented Apr 9, 2025

BE UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants