Skip to content

Conversation

@hubgeter
Copy link
Contributor

Related PR: #45937
branch-3.1: #55500

Problem Summary:
Fix the error case on ingestion load and the core in parquet reader.

==8898==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62f0020603fc at pc 0x55f634e64ded bp 0x7fba0d03c410 sp 0x7fba0d03bbd8 READ of size 4 at 0x62f0020603fc thread T768 (PUSH-9699)
    #0 0x55f634e64dec in __asan_memcpy (/mnt/hdd01/ci/doris-deploy-branch-3.1-local/be/lib/doris_be+0x39a24dec) (BuildId: 9b04e7f7d3075dac)
    #1 0x55f634eca93f in std::char_traits::copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/char_traits.h:409:33
    #2 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:351:4
    #3 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy_chars(char*, char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:398:9
    #4 0x55f634eca93f in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:225:6
    #5 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct_aux(char const*, char const*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11
    #6 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4
    #7 0x55f654a4f74d in std::__cxx11::basic_string, std::allocator>::basic_string(char const*, unsigned long, std::allocator const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:513:9
    #8 0x55f654a4f74d in doris::vectorized::parse_thrift_footer(std::shared_ptr, doris::vectorized::FileMetaData**, unsigned long*, doris::io::IOContext*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/parquet_thrift_util.h:55:17

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…pache#55500)

Related PR: apache#45937

Problem Summary:
Fix the error case on ingestion load and the core in parquet reader.

==8898==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62f0020603fc at pc 0x55f634e64ded bp 0x7fba0d03c410 sp 0x7fba0d03bbd8
READ of size 4 at 0x62f0020603fc thread T768 (PUSH-9699)
    #0 0x55f634e64dec in __asan_memcpy (/mnt/hdd01/ci/doris-deploy-branch-3.1-local/be/lib/doris_be+0x39a24dec) (BuildId: 9b04e7f7d3075dac)
    apache#1 0x55f634eca93f in std::char_traits::copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/char_traits.h:409:33
    apache#2 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:351:4
    apache#3 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy_chars(char*, char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:398:9
    apache#4 0x55f634eca93f in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:225:6
    apache#5 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct_aux(char const*, char const*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11
    apache#6 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4
    apache#7 0x55f654a4f74d in std::__cxx11::basic_string, std::allocator>::basic_string(char const*, unsigned long, std::allocator const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:513:9
    apache#8 0x55f654a4f74d in doris::vectorized::parse_thrift_footer(std::shared_ptr, doris::vectorized::FileMetaData**, unsigned long*, doris::io::IOContext*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/parquet_thrift_util.h:55:17
@Thearas
Copy link
Contributor

Thearas commented Sep 11, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34793 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 824bc8fb118076d238df95536a7b751fb97d65fb, data reload: false

------ Round 1 ----------------------------------
q1	17603	5212	5067	5067
q2	2018	324	206	206
q3	10260	1271	702	702
q4	10224	1013	529	529
q5	7492	2500	2321	2321
q6	190	181	143	143
q7	931	771	633	633
q8	9356	1318	1098	1098
q9	7072	5274	5264	5264
q10	6949	2404	2000	2000
q11	497	301	281	281
q12	369	360	237	237
q13	17793	3651	3001	3001
q14	231	253	222	222
q15	570	493	489	489
q16	1019	1019	948	948
q17	601	840	387	387
q18	7500	7156	7069	7069
q19	1448	955	579	579
q20	352	349	239	239
q21	3962	2660	2407	2407
q22	1084	1017	971	971
Total cold run time: 107521 ms
Total hot run time: 34793 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5152	5108	5069	5069
q2	255	337	230	230
q3	2166	2692	2310	2310
q4	1369	1740	1353	1353
q5	4231	4529	4526	4526
q6	225	179	136	136
q7	2036	2016	1879	1879
q8	2715	2636	2578	2578
q9	7507	7365	7343	7343
q10	3167	3315	2891	2891
q11	612	523	509	509
q12	661	810	650	650
q13	3612	3982	3273	3273
q14	304	317	294	294
q15	531	490	483	483
q16	1073	1089	1043	1043
q17	1179	1519	1407	1407
q18	7991	7899	7327	7327
q19	755	756	779	756
q20	1908	1952	1848	1848
q21	4996	4421	4343	4343
q22	1097	1031	998	998
Total cold run time: 53542 ms
Total hot run time: 51246 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189343 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 824bc8fb118076d238df95536a7b751fb97d65fb, data reload: false

query1	1045	428	447	428
query2	6568	1703	1686	1686
query3	6749	228	227	227
query4	26094	23624	22803	22803
query5	4449	650	531	531
query6	352	254	230	230
query7	4654	509	312	312
query8	311	262	270	262
query9	8676	2929	2943	2929
query10	504	367	307	307
query11	15900	15095	14683	14683
query12	177	126	118	118
query13	1695	570	446	446
query14	10826	9231	9217	9217
query15	210	217	176	176
query16	7731	655	512	512
query17	1251	791	651	651
query18	2055	424	338	338
query19	200	196	176	176
query20	134	128	123	123
query21	220	129	117	117
query22	4163	4356	4323	4323
query23	34009	33189	33072	33072
query24	8262	2389	2421	2389
query25	569	514	474	474
query26	1236	275	165	165
query27	2725	516	367	367
query28	4398	2277	2239	2239
query29	786	641	498	498
query30	290	243	199	199
query31	917	806	711	711
query32	90	80	82	80
query33	578	398	358	358
query34	809	863	518	518
query35	827	825	752	752
query36	975	1013	928	928
query37	128	112	96	96
query38	3590	3573	3505	3505
query39	1473	1418	1420	1418
query40	234	148	126	126
query41	67	64	61	61
query42	129	128	124	124
query43	542	517	458	458
query44	1372	870	877	870
query45	182	177	173	173
query46	847	1027	688	688
query47	1852	1844	1817	1817
query48	388	421	323	323
query49	752	517	415	415
query50	662	693	407	407
query51	4017	3919	3844	3844
query52	120	115	110	110
query53	252	283	211	211
query54	646	600	576	576
query55	100	93	90	90
query56	363	330	327	327
query57	1216	1214	1140	1140
query58	291	291	283	283
query59	2546	2596	2576	2576
query60	370	357	357	357
query61	167	157	157	157
query62	839	732	663	663
query63	241	200	196	196
query64	4414	1165	850	850
query65	4056	3989	3993	3989
query66	1136	460	362	362
query67	15558	15225	15170	15170
query68	8136	932	584	584
query69	503	342	301	301
query70	1462	1191	1264	1191
query71	571	357	321	321
query72	5878	5032	5019	5019
query73	743	625	361	361
query74	9035	9099	8964	8964
query75	3944	3325	2815	2815
query76	3632	1157	747	747
query77	812	409	341	341
query78	9631	9933	8817	8817
query79	2138	810	602	602
query80	660	661	508	508
query81	496	257	235	235
query82	457	163	138	138
query83	272	268	249	249
query84	252	112	96	96
query85	909	469	431	431
query86	348	327	322	322
query87	3760	3701	3651	3651
query88	3312	2217	2247	2217
query89	391	327	295	295
query90	1973	229	222	222
query91	173	169	133	133
query92	98	77	72	72
query93	1532	1027	641	641
query94	717	421	332	332
query95	415	344	328	328
query96	487	591	285	285
query97	2921	2975	2857	2857
query98	254	227	220	220
query99	1368	1415	1321	1321
Total cold run time: 276265 ms
Total hot run time: 189343 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.61 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 824bc8fb118076d238df95536a7b751fb97d65fb, data reload: false

query1	0.06	0.06	0.05
query2	0.09	0.05	0.05
query3	0.25	0.08	0.08
query4	1.63	0.12	0.11
query5	0.29	0.26	0.25
query6	1.16	0.64	0.64
query7	0.03	0.03	0.03
query8	0.06	0.05	0.05
query9	0.63	0.52	0.51
query10	0.58	0.57	0.57
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.67	0.63	0.63
query14	1.03	1.05	1.03
query15	0.87	0.85	0.86
query16	0.41	0.38	0.41
query17	1.06	1.05	1.03
query18	0.21	0.20	0.20
query19	1.98	1.83	1.91
query20	0.02	0.01	0.01
query21	15.40	0.96	0.58
query22	0.79	1.17	0.69
query23	14.88	1.40	0.66
query24	6.61	2.10	0.44
query25	0.53	0.16	0.19
query26	0.61	0.18	0.13
query27	0.07	0.05	0.05
query28	10.25	0.95	0.45
query29	12.67	3.93	3.24
query30	0.28	0.14	0.13
query31	2.83	0.60	0.39
query32	3.27	0.56	0.47
query33	3.12	3.09	3.07
query34	16.06	5.49	4.86
query35	4.91	4.89	4.91
query36	0.69	0.52	0.50
query37	0.11	0.08	0.07
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.09	0.04	0.03
query42	0.04	0.04	0.03
query43	0.05	0.03	0.03
Total cold run time: 104.89 s
Total hot run time: 29.61 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 40.00% (2/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 51.98% (17301/33284)
Line Coverage 37.33% (157773/422606)
Region Coverage 31.98% (120482/376716)
Branch Coverage 33.35% (52869/158525)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 40.00% (2/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.75% (23121/32682)
Line Coverage 57.15% (241358/422351)
Region Coverage 52.58% (200921/382144)
Branch Coverage 54.15% (86324/159428)

@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34663 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0899a493fd3b0819344abc830d2d756d892e44e5, data reload: false

------ Round 1 ----------------------------------
q1	17631	5290	5108	5108
q2	1988	323	206	206
q3	10289	1278	716	716
q4	10226	1026	526	526
q5	7513	2400	2315	2315
q6	184	165	141	141
q7	932	765	662	662
q8	9362	1344	1129	1129
q9	6845	5117	5108	5108
q10	6970	2368	1987	1987
q11	486	304	279	279
q12	360	356	236	236
q13	17797	3704	3011	3011
q14	246	255	218	218
q15	554	500	482	482
q16	999	1000	959	959
q17	607	870	375	375
q18	7320	7181	7092	7092
q19	1380	950	560	560
q20	344	342	226	226
q21	3668	2531	2341	2341
q22	1073	1033	986	986
Total cold run time: 106774 ms
Total hot run time: 34663 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5278	5190	5023	5023
q2	251	334	227	227
q3	2208	2670	2301	2301
q4	1367	1774	1351	1351
q5	4253	4472	4527	4472
q6	223	171	129	129
q7	2052	1995	1893	1893
q8	2675	2686	2939	2686
q9	7376	7222	7328	7222
q10	3103	3276	2861	2861
q11	606	567	517	517
q12	699	859	726	726
q13	3410	3853	3320	3320
q14	304	299	283	283
q15	529	494	485	485
q16	1309	1124	1054	1054
q17	1189	1641	1307	1307
q18	8007	7756	7563	7563
q19	807	803	931	803
q20	2013	2065	1880	1880
q21	4796	4344	4270	4270
q22	1074	1051	989	989
Total cold run time: 53529 ms
Total hot run time: 51362 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189681 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0899a493fd3b0819344abc830d2d756d892e44e5, data reload: false

query1	1044	458	439	439
query2	6578	1687	1659	1659
query3	6750	239	228	228
query4	26583	23508	22999	22999
query5	4393	662	502	502
query6	344	264	215	215
query7	4648	526	302	302
query8	305	252	251	251
query9	8677	2932	2955	2932
query10	495	382	306	306
query11	16231	15298	15000	15000
query12	186	125	118	118
query13	1676	561	446	446
query14	10874	9450	9325	9325
query15	212	196	183	183
query16	7391	689	512	512
query17	1281	784	697	697
query18	2032	429	339	339
query19	205	202	186	186
query20	141	134	128	128
query21	216	145	121	121
query22	4027	4186	4038	4038
query23	33815	32919	32864	32864
query24	8219	2415	2464	2415
query25	599	542	450	450
query26	1238	277	176	176
query27	2731	521	357	357
query28	4400	2253	2228	2228
query29	852	635	506	506
query30	304	238	200	200
query31	888	803	723	723
query32	98	80	80	80
query33	593	424	362	362
query34	836	874	533	533
query35	871	849	778	778
query36	981	1042	918	918
query37	138	121	104	104
query38	3529	3636	3520	3520
query39	1680	1417	1469	1417
query40	255	144	136	136
query41	64	62	63	62
query42	137	118	128	118
query43	553	496	476	476
query44	1406	880	879	879
query45	186	180	175	175
query46	931	1037	669	669
query47	1759	1805	1728	1728
query48	415	425	322	322
query49	752	510	437	437
query50	694	705	409	409
query51	3975	3965	3902	3902
query52	126	117	111	111
query53	251	279	210	210
query54	619	635	581	581
query55	102	91	98	91
query56	382	336	336	336
query57	1181	1182	1114	1114
query58	303	289	288	288
query59	2567	2649	2598	2598
query60	366	358	359	358
query61	166	186	160	160
query62	824	747	655	655
query63	245	202	205	202
query64	4483	1197	820	820
query65	4037	3965	3989	3965
query66	1157	448	359	359
query67	15460	15188	15253	15188
query68	7910	957	601	601
query69	514	353	342	342
query70	1357	1335	1235	1235
query71	618	359	328	328
query72	6029	5083	5428	5083
query73	715	714	367	367
query74	8923	9097	8661	8661
query75	3531	3213	2817	2817
query76	3345	1166	759	759
query77	684	425	340	340
query78	9636	9732	8905	8905
query79	2144	844	603	603
query80	713	611	575	575
query81	500	271	238	238
query82	233	178	141	141
query83	281	289	261	261
query84	264	130	104	104
query85	1019	471	426	426
query86	384	319	323	319
query87	3659	3737	3693	3693
query88	2956	2224	2225	2224
query89	394	331	300	300
query90	1998	238	231	231
query91	168	174	135	135
query92	99	77	75	75
query93	2026	988	656	656
query94	699	408	313	313
query95	414	350	329	329
query96	490	591	284	284
query97	2925	2969	2899	2899
query98	270	227	219	219
query99	1340	1436	1312	1312
Total cold run time: 275695 ms
Total hot run time: 189681 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.88 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0899a493fd3b0819344abc830d2d756d892e44e5, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.06	0.06
query3	0.25	0.08	0.08
query4	1.60	0.12	0.12
query5	0.26	0.26	0.25
query6	1.18	0.66	0.65
query7	0.04	0.03	0.03
query8	0.06	0.05	0.05
query9	0.63	0.52	0.53
query10	0.56	0.58	0.57
query11	0.17	0.11	0.11
query12	0.16	0.13	0.12
query13	0.66	0.63	0.63
query14	1.02	1.06	1.04
query15	0.89	0.87	0.86
query16	0.41	0.40	0.39
query17	1.07	1.05	1.06
query18	0.22	0.21	0.20
query19	1.95	1.88	1.88
query20	0.02	0.01	0.01
query21	15.40	0.96	0.58
query22	0.77	1.27	0.73
query23	14.85	1.39	0.64
query24	6.39	1.97	0.52
query25	0.44	0.39	0.08
query26	0.46	0.15	0.14
query27	0.06	0.06	0.06
query28	10.14	0.95	0.43
query29	12.65	3.98	3.30
query30	0.30	0.15	0.14
query31	2.84	0.60	0.39
query32	3.25	0.56	0.47
query33	3.14	3.04	3.09
query34	15.82	5.50	4.91
query35	4.94	4.90	4.91
query36	0.70	0.52	0.51
query37	0.11	0.07	0.07
query38	0.06	0.05	0.05
query39	0.03	0.03	0.03
query40	0.18	0.14	0.14
query41	0.09	0.03	0.02
query42	0.03	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 104 s
Total hot run time: 29.88 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 40.00% (2/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.21% (17421/33369)
Line Coverage 37.43% (158328/423034)
Region Coverage 32.01% (120744/377257)
Branch Coverage 33.37% (52981/158747)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 40.00% (2/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.81% (23103/32628)
Line Coverage 57.17% (241273/422020)
Region Coverage 52.48% (200589/382248)
Branch Coverage 54.13% (86311/159448)

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 12, 2025
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 40.00% (2/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.81% (23103/32628)
Line Coverage 57.17% (241273/422020)
Region Coverage 52.48% (200589/382248)
Branch Coverage 54.13% (86311/159448)

@morningman morningman merged commit bf74dbb into apache:master Sep 12, 2025
27 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.1-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants