Skip to content

Conversation

@swjtu-zhanglei
Copy link
Contributor

@swjtu-zhanglei swjtu-zhanglei commented Jun 24, 2025

=================================================================
==3271527==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 216 byte(s) in 1 object(s) allocated from:
    #0 0x555f9ecf55ee in malloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x370815ee) (BuildId: 22b637395dd039b0)
    #1 0x555fd8e19bd7 in CRYPTO_zalloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x711a5bd7) (BuildId: 22b637395dd039b0)
    #2 0x555fda7c9d14 in aws_mqtt_library_init (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72b55d14) (BuildId: 22b637395dd039b0)
    #3 0x555fda74ed1f in Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72adad1f) (BuildId: 22b637395dd039b0)
    #4 0x555fa365f0c9 in doris::S3ClientFactory::instance() /root/doris/be/src/util/s3_util.cpp:188:28
    #5 0x555f9f13b87b in doris::io::ObjClientHolder::init() /root/doris/be/src/io/fs/s3_file_system.cpp:78:15
    #6 0x555f9f140eee in doris::io::S3FileSystem::init() /root/doris/be/src/io/fs/s3_file_system.cpp:182:21
    ......

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

* aws/aws-sdk-cpp#2314
* In the aws-sdk-cpp code, there is a suggestion:
  "Please call ShutdownAPI from the same thread from which InitAPI() has been called"
  otherwill it will cause memory leak.

```
=================================================================
==3271527==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 216 byte(s) in 1 object(s) allocated from:
    #0 0x555f9ecf55ee in malloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x370815ee) (BuildId: 22b637395dd039b0)
    apache#1 0x555fd8e19bd7 in CRYPTO_zalloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x711a5bd7) (BuildId: 22b637395dd039b0)
    apache#2 0x555fda7c9d14 in aws_mqtt_library_init (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72b55d14) (BuildId: 22b637395dd039b0)
    apache#3 0x555fda74ed1f in Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72adad1f) (BuildId: 22b637395dd039b0)
    apache#4 0x555fa365f0c9 in doris::S3ClientFactory::instance() /root/doris/be/src/util/s3_util.cpp:188:28
    apache#5 0x555f9f13b87b in doris::io::ObjClientHolder::init() /root/doris/be/src/io/fs/s3_file_system.cpp:78:15
    apache#6 0x555f9f140eee in doris::io::S3FileSystem::init() /root/doris/be/src/io/fs/s3_file_system.cpp:182:21
    ......
```
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@swjtu-zhanglei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34760 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3943b43a22cba5c8fee433ed565cd02d9a95e4a6, data reload: false

------ Round 1 ----------------------------------
q1	17626	5155	5049	5049
q2	1941	295	209	209
q3	10258	1349	734	734
q4	10218	1011	548	548
q5	7542	2350	2385	2350
q6	185	167	134	134
q7	893	746	598	598
q8	9309	1321	1064	1064
q9	6756	5053	5131	5053
q10	6899	2397	1970	1970
q11	492	295	287	287
q12	356	348	222	222
q13	17787	3662	3133	3133
q14	225	220	225	220
q15	543	489	482	482
q16	432	432	372	372
q17	614	874	398	398
q18	7620	7241	7169	7169
q19	1220	940	565	565
q20	355	349	231	231
q21	3952	3143	2997	2997
q22	1055	1041	975	975
Total cold run time: 106278 ms
Total hot run time: 34760 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5132	5109	5115	5109
q2	256	326	224	224
q3	2167	2640	2287	2287
q4	1341	1795	1336	1336
q5	4232	4107	4423	4107
q6	216	171	132	132
q7	2159	1940	1735	1735
q8	2635	2658	2557	2557
q9	7189	7138	7209	7138
q10	3084	3226	2903	2903
q11	583	515	498	498
q12	661	770	585	585
q13	3627	3871	3255	3255
q14	283	306	268	268
q15	523	475	478	475
q16	432	485	450	450
q17	1192	1565	1391	1391
q18	7797	7550	7366	7366
q19	798	821	847	821
q20	1988	2104	1899	1899
q21	4943	4528	4293	4293
q22	1062	1027	1021	1021
Total cold run time: 52300 ms
Total hot run time: 49850 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188213 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3943b43a22cba5c8fee433ed565cd02d9a95e4a6, data reload: false

query1	996	412	406	406
query2	6535	1823	1835	1823
query3	6746	221	221	221
query4	26673	23724	23274	23274
query5	4376	657	502	502
query6	316	222	211	211
query7	4626	527	297	297
query8	275	231	216	216
query9	8632	2654	2653	2653
query10	501	346	269	269
query11	15703	15655	14937	14937
query12	176	121	114	114
query13	1677	572	418	418
query14	9767	6706	6666	6666
query15	244	204	197	197
query16	7264	648	489	489
query17	1211	780	620	620
query18	2004	432	314	314
query19	206	209	172	172
query20	132	127	120	120
query21	219	144	118	118
query22	4085	4155	4107	4107
query23	34453	33198	33332	33198
query24	8613	2429	2496	2429
query25	562	479	411	411
query26	1251	286	157	157
query27	2719	538	354	354
query28	4318	2153	2135	2135
query29	791	615	449	449
query30	291	238	207	207
query31	965	871	781	781
query32	77	64	64	64
query33	582	399	333	333
query34	812	923	555	555
query35	832	876	754	754
query36	1012	1027	917	917
query37	117	105	81	81
query38	4171	4334	4068	4068
query39	1529	1489	1544	1489
query40	223	125	115	115
query41	64	63	64	63
query42	133	122	112	112
query43	501	522	477	477
query44	1375	846	850	846
query45	187	180	188	180
query46	912	1107	633	633
query47	1776	1809	1744	1744
query48	385	462	312	312
query49	764	504	408	408
query50	677	735	406	406
query51	4431	4351	4245	4245
query52	121	114	104	104
query53	233	282	186	186
query54	586	581	519	519
query55	90	90	88	88
query56	316	322	297	297
query57	1233	1228	1129	1129
query58	294	275	269	269
query59	2577	2638	2536	2536
query60	335	336	323	323
query61	130	126	124	124
query62	861	789	693	693
query63	239	196	193	193
query64	4345	1002	678	678
query65	4381	4274	4249	4249
query66	1164	463	346	346
query67	16037	15532	15246	15246
query68	7988	924	539	539
query69	474	317	288	288
query70	1223	1201	1091	1091
query71	477	348	314	314
query72	5753	4744	4732	4732
query73	686	589	362	362
query74	9048	9142	8961	8961
query75	3931	3411	2833	2833
query76	3675	1338	814	814
query77	778	394	289	289
query78	10268	10232	9484	9484
query79	4338	840	605	605
query80	668	542	465	465
query81	486	281	238	238
query82	574	133	103	103
query83	294	261	241	241
query84	299	109	103	103
query85	789	359	321	321
query86	335	324	296	296
query87	4462	4525	4313	4313
query88	3468	2314	2298	2298
query89	460	351	307	307
query90	1966	222	231	222
query91	145	147	118	118
query92	73	64	63	63
query93	2869	954	594	594
query94	681	422	316	316
query95	391	302	288	288
query96	500	640	287	287
query97	2792	2856	2712	2712
query98	246	214	206	206
query99	1554	1495	1363	1363
Total cold run time: 281520 ms
Total hot run time: 188213 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.34 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3943b43a22cba5c8fee433ed565cd02d9a95e4a6, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.60	0.10	0.10
query5	0.43	0.43	0.42
query6	1.16	0.67	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.53	0.50
query10	0.57	0.59	0.57
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.63	0.61	0.60
query14	0.80	0.80	0.81
query15	0.91	0.86	0.87
query16	0.39	0.39	0.40
query17	1.09	1.05	1.09
query18	0.22	0.21	0.21
query19	1.93	1.83	1.82
query20	0.01	0.02	0.01
query21	15.40	0.91	0.54
query22	0.74	1.14	0.62
query23	15.08	1.39	0.65
query24	7.16	1.70	0.57
query25	0.51	0.15	0.21
query26	0.64	0.16	0.14
query27	0.05	0.05	0.05
query28	9.44	0.84	0.43
query29	12.61	4.11	3.35
query30	0.26	0.09	0.06
query31	2.83	0.59	0.40
query32	3.25	0.57	0.49
query33	3.14	3.09	3.09
query34	16.06	5.38	4.80
query35	4.85	4.83	4.79
query36	0.69	0.51	0.48
query37	0.09	0.07	0.07
query38	0.05	0.05	0.04
query39	0.03	0.02	0.03
query40	0.17	0.15	0.14
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.21 s
Total hot run time: 29.34 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 24, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.64% (15131/26715)
Line Coverage 45.45% (135704/298581)
Region Coverage 44.66% (68354/153065)
Branch Coverage 39.24% (35141/89546)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 62.18% (16344/26283)
Line Coverage 51.74% (154386/298399)
Region Coverage 49.23% (88639/180043)
Branch Coverage 42.70% (43688/102320)

@hello-stephen hello-stephen merged commit 4f955c6 into apache:master Jun 24, 2025
32 of 34 checks passed
github-actions bot pushed a commit that referenced this pull request Jun 24, 2025
* aws/aws-sdk-cpp#2314
* In the aws-sdk-cpp code, there is a suggestion: "Please call
ShutdownAPI from the same thread from which InitAPI() has been called"
otherwise it will cause memory leak.

```
=================================================================
==3271527==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 216 byte(s) in 1 object(s) allocated from:
    #0 0x555f9ecf55ee in malloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x370815ee) (BuildId: 22b637395dd039b0)
    #1 0x555fd8e19bd7 in CRYPTO_zalloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x711a5bd7) (BuildId: 22b637395dd039b0)
    #2 0x555fda7c9d14 in aws_mqtt_library_init (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72b55d14) (BuildId: 22b637395dd039b0)
    #3 0x555fda74ed1f in Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72adad1f) (BuildId: 22b637395dd039b0)
    #4 0x555fa365f0c9 in doris::S3ClientFactory::instance() /root/doris/be/src/util/s3_util.cpp:188:28
    #5 0x555f9f13b87b in doris::io::ObjClientHolder::init() /root/doris/be/src/io/fs/s3_file_system.cpp:78:15
    #6 0x555f9f140eee in doris::io::S3FileSystem::init() /root/doris/be/src/io/fs/s3_file_system.cpp:182:21
    ......
```
github-actions bot pushed a commit that referenced this pull request Jun 24, 2025
* aws/aws-sdk-cpp#2314
* In the aws-sdk-cpp code, there is a suggestion: "Please call
ShutdownAPI from the same thread from which InitAPI() has been called"
otherwise it will cause memory leak.

```
=================================================================
==3271527==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 216 byte(s) in 1 object(s) allocated from:
    #0 0x555f9ecf55ee in malloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x370815ee) (BuildId: 22b637395dd039b0)
    #1 0x555fd8e19bd7 in CRYPTO_zalloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x711a5bd7) (BuildId: 22b637395dd039b0)
    #2 0x555fda7c9d14 in aws_mqtt_library_init (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72b55d14) (BuildId: 22b637395dd039b0)
    #3 0x555fda74ed1f in Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72adad1f) (BuildId: 22b637395dd039b0)
    #4 0x555fa365f0c9 in doris::S3ClientFactory::instance() /root/doris/be/src/util/s3_util.cpp:188:28
    #5 0x555f9f13b87b in doris::io::ObjClientHolder::init() /root/doris/be/src/io/fs/s3_file_system.cpp:78:15
    #6 0x555f9f140eee in doris::io::S3FileSystem::init() /root/doris/be/src/io/fs/s3_file_system.cpp:182:21
    ......
```
dataroaring pushed a commit that referenced this pull request Jun 24, 2025
Cherry-picked from #52185

Co-authored-by: Lei Zhang <zhanglei@selectdb.com>
morrySnow pushed a commit that referenced this pull request Jun 24, 2025
Cherry-picked from #52185

Co-authored-by: Lei Zhang <zhanglei@selectdb.com>
github-actions bot pushed a commit that referenced this pull request Jun 25, 2025
* aws/aws-sdk-cpp#2314
* In the aws-sdk-cpp code, there is a suggestion: "Please call
ShutdownAPI from the same thread from which InitAPI() has been called"
otherwise it will cause memory leak.

```
=================================================================
==3271527==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 216 byte(s) in 1 object(s) allocated from:
    #0 0x555f9ecf55ee in malloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x370815ee) (BuildId: 22b637395dd039b0)
    #1 0x555fd8e19bd7 in CRYPTO_zalloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x711a5bd7) (BuildId: 22b637395dd039b0)
    #2 0x555fda7c9d14 in aws_mqtt_library_init (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72b55d14) (BuildId: 22b637395dd039b0)
    #3 0x555fda74ed1f in Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72adad1f) (BuildId: 22b637395dd039b0)
    #4 0x555fa365f0c9 in doris::S3ClientFactory::instance() /root/doris/be/src/util/s3_util.cpp:188:28
    #5 0x555f9f13b87b in doris::io::ObjClientHolder::init() /root/doris/be/src/io/fs/s3_file_system.cpp:78:15
    #6 0x555f9f140eee in doris::io::S3FileSystem::init() /root/doris/be/src/io/fs/s3_file_system.cpp:182:21
    ......
```
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Jun 25, 2025
* aws/aws-sdk-cpp#2314
* In the aws-sdk-cpp code, there is a suggestion: "Please call
ShutdownAPI from the same thread from which InitAPI() has been called"
otherwise it will cause memory leak.

```
=================================================================
==3271527==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 216 byte(s) in 1 object(s) allocated from:
    #0 0x555f9ecf55ee in malloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x370815ee) (BuildId: 22b637395dd039b0)
    apache#1 0x555fd8e19bd7 in CRYPTO_zalloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x711a5bd7) (BuildId: 22b637395dd039b0)
    apache#2 0x555fda7c9d14 in aws_mqtt_library_init (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72b55d14) (BuildId: 22b637395dd039b0)
    apache#3 0x555fda74ed1f in Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72adad1f) (BuildId: 22b637395dd039b0)
    apache#4 0x555fa365f0c9 in doris::S3ClientFactory::instance() /root/doris/be/src/util/s3_util.cpp:188:28
    apache#5 0x555f9f13b87b in doris::io::ObjClientHolder::init() /root/doris/be/src/io/fs/s3_file_system.cpp:78:15
    apache#6 0x555f9f140eee in doris::io::S3FileSystem::init() /root/doris/be/src/io/fs/s3_file_system.cpp:182:21
    ......
```
yiguolei pushed a commit that referenced this pull request Jun 28, 2025
* aws/aws-sdk-cpp#2314
* In the aws-sdk-cpp code, there is a suggestion: "Please call
ShutdownAPI from the same thread from which InitAPI() has been called"
otherwise it will cause memory leak.

```
=================================================================
==3271527==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 216 byte(s) in 1 object(s) allocated from:
    #0 0x555f9ecf55ee in malloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x370815ee) (BuildId: 22b637395dd039b0)
    #1 0x555fd8e19bd7 in CRYPTO_zalloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x711a5bd7) (BuildId: 22b637395dd039b0)
    #2 0x555fda7c9d14 in aws_mqtt_library_init (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72b55d14) (BuildId: 22b637395dd039b0)
    #3 0x555fda74ed1f in Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72adad1f) (BuildId: 22b637395dd039b0)
    #4 0x555fa365f0c9 in doris::S3ClientFactory::instance() /root/doris/be/src/util/s3_util.cpp:188:28
    #5 0x555f9f13b87b in doris::io::ObjClientHolder::init() /root/doris/be/src/io/fs/s3_file_system.cpp:78:15
    #6 0x555f9f140eee in doris::io::S3FileSystem::init() /root/doris/be/src/io/fs/s3_file_system.cpp:182:21
    ......
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.7-merged dev/3.1.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants