Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement](memory) Allocator support address sanitizers #33396

Merged
merged 9 commits into from
Apr 18, 2024

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Apr 8, 2024

Proposed changes

  1. If DEBUG build type, record size of each memory alloc and free, print no free size or no alloc size when query MemTracker is destructed, if necessary, record stack trace.

  2. add global PointQueryExecutor memory tracker in ExecEnv, because memory may be shared between PointQueryExecutors of different pointer queries, but memory will not be shared between PointQueryExecutor and Fragment of the same pointer query.

  3. If DEBUG build type, if query memory tracker not equal to 0 when query ends, BE will crash.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Apr 8, 2024

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/runtime/memory/mem_tracker_limiter.cpp Outdated Show resolved Hide resolved
be/src/runtime/memory/mem_tracker_limiter.cpp Outdated Show resolved Hide resolved
@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Apr 9, 2024

run buildall

2 similar comments
@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Apr 9, 2024

run buildall

@xinyiZzz
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@xinyiZzz
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@xinyiZzz
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@xinyiZzz
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@xinyiZzz
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@xinyiZzz
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@xinyiZzz
Copy link
Contributor Author

run p0

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@xinyiZzz
Copy link
Contributor Author

run p0

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@xinyiZzz
Copy link
Contributor Author

run p0

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@xinyiZzz
Copy link
Contributor Author

run buildall

@xinyiZzz
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/common/pod_array.h Outdated Show resolved Hide resolved
be/src/vec/common/pod_array.h Outdated Show resolved Hide resolved
@xinyiZzz
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

ClickBench: Total hot run time: 29.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f991e14ff43dff01992c239365e11a2b0babf50b, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.06	0.07
query5	0.50	0.49	0.51
query6	1.48	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.46
query10	0.52	0.53	0.54
query11	0.16	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.59	0.58
query14	0.77	0.77	0.77
query15	0.82	0.80	0.80
query16	0.37	0.34	0.37
query17	0.93	0.94	0.95
query18	0.21	0.26	0.25
query19	1.80	1.74	1.78
query20	0.01	0.02	0.02
query21	15.43	0.65	0.64
query22	4.47	7.99	1.61
query23	18.29	1.36	1.32
query24	1.81	0.27	0.21
query25	0.16	0.08	0.07
query26	0.25	0.16	0.16
query27	0.08	0.08	0.08
query28	13.28	0.99	0.97
query29	12.63	3.21	3.23
query30	0.26	0.07	0.06
query31	2.85	0.37	0.38
query32	3.28	0.46	0.47
query33	2.81	2.83	2.85
query34	17.10	4.46	4.39
query35	4.49	4.44	4.48
query36	0.66	0.46	0.47
query37	0.19	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.18	0.13	0.13
query41	0.09	0.04	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.54 s
Total hot run time: 29.94 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f991e14ff43dff01992c239365e11a2b0babf50b with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

@xinyiZzz
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38752 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4f25badbbbad57a4f0330291beaff7d61f2c72d, data reload: false

------ Round 1 ----------------------------------
q1	17607	4480	4422	4422
q2	2003	181	178	178
q3	10475	1111	1167	1111
q4	10193	771	831	771
q5	7535	2669	2647	2647
q6	214	133	134	133
q7	1054	635	588	588
q8	9243	2073	2053	2053
q9	7348	6606	6543	6543
q10	8562	3538	3515	3515
q11	474	223	235	223
q12	398	221	212	212
q13	17758	2922	2952	2922
q14	276	237	220	220
q15	503	478	471	471
q16	532	396	386	386
q17	962	756	668	668
q18	7563	6970	6828	6828
q19	7367	1512	1463	1463
q20	665	301	305	301
q21	3463	2823	2791	2791
q22	358	318	306	306
Total cold run time: 114553 ms
Total hot run time: 38752 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4391	4238	4207	4207
q2	371	275	264	264
q3	3003	2774	2749	2749
q4	1901	1626	1561	1561
q5	5403	5390	5308	5308
q6	209	123	124	123
q7	2201	1831	1877	1831
q8	3248	3388	3367	3367
q9	8592	8600	8577	8577
q10	3883	3688	3692	3688
q11	594	479	490	479
q12	758	586	582	582
q13	16870	2967	2926	2926
q14	289	283	287	283
q15	512	470	476	470
q16	477	419	433	419
q17	1793	1483	1463	1463
q18	8144	7953	7911	7911
q19	1684	1621	1674	1621
q20	2069	1869	1828	1828
q21	10526	4943	4989	4943
q22	529	470	458	458
Total cold run time: 77447 ms
Total hot run time: 55058 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.45% (8909/25133)
Line Coverage: 27.14% (73180/269658)
Region Coverage: 26.28% (37828/143964)
Branch Coverage: 23.07% (19270/83540)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c4f25badbbbad57a4f0330291beaff7d61f2c72d_c4f25badbbbad57a4f0330291beaff7d61f2c72d/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38486 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4f25badbbbad57a4f0330291beaff7d61f2c72d, data reload: false

------ Round 1 ----------------------------------
q1	17608	4403	4243	4243
q2	2015	189	179	179
q3	10472	1145	1157	1145
q4	10193	794	837	794
q5	7551	2676	2606	2606
q6	213	129	130	129
q7	982	613	593	593
q8	9226	2063	2051	2051
q9	7272	6546	6555	6546
q10	8647	3523	3536	3523
q11	448	228	225	225
q12	407	215	219	215
q13	17773	2930	2954	2930
q14	273	232	237	232
q15	528	481	473	473
q16	517	380	369	369
q17	962	723	661	661
q18	7309	6793	6775	6775
q19	6731	1492	1494	1492
q20	635	309	294	294
q21	3422	2708	2795	2708
q22	375	303	303	303
Total cold run time: 113559 ms
Total hot run time: 38486 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4382	4254	4233	4233
q2	359	273	272	272
q3	2958	2740	2800	2740
q4	1866	1559	1548	1548
q5	5373	5388	5324	5324
q6	209	123	125	123
q7	2246	1874	1879	1874
q8	3214	3330	3346	3330
q9	8593	8624	8607	8607
q10	4072	3925	3971	3925
q11	594	490	502	490
q12	810	633	621	621
q13	17133	3224	3135	3135
q14	329	296	313	296
q15	533	491	488	488
q16	486	448	436	436
q17	1826	1529	1494	1494
q18	8269	7982	7944	7944
q19	1675	1541	1539	1539
q20	2083	1831	1847	1831
q21	8983	4916	4914	4914
q22	556	487	484	484
Total cold run time: 76549 ms
Total hot run time: 55648 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185549 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c4f25badbbbad57a4f0330291beaff7d61f2c72d, data reload: false

query1	884	382	363	363
query2	6198	2592	2434	2434
query3	6660	209	207	207
query4	22778	21239	21226	21226
query5	4171	403	408	403
query6	262	177	173	173
query7	4581	301	293	293
query8	229	173	172	172
query9	8545	2320	2301	2301
query10	417	250	270	250
query11	14769	14131	14259	14131
query12	132	86	82	82
query13	1624	375	368	368
query14	8641	8254	7406	7406
query15	260	185	185	185
query16	8186	256	257	256
query17	1946	566	539	539
query18	2104	277	263	263
query19	328	150	152	150
query20	89	81	86	81
query21	197	137	122	122
query22	4985	4799	4854	4799
query23	33866	33049	33164	33049
query24	10824	3115	3038	3038
query25	654	417	413	413
query26	1175	164	160	160
query27	2381	376	403	376
query28	6975	2075	2068	2068
query29	908	678	641	641
query30	285	183	180	180
query31	1003	776	778	776
query32	97	54	60	54
query33	794	265	257	257
query34	1332	492	504	492
query35	847	722	727	722
query36	1076	913	929	913
query37	127	71	70	70
query38	3516	3411	3339	3339
query39	1646	1601	1581	1581
query40	178	133	131	131
query41	47	49	44	44
query42	108	100	101	100
query43	594	537	542	537
query44	1222	760	758	758
query45	287	288	279	279
query46	1109	749	778	749
query47	2039	1947	1978	1947
query48	389	302	310	302
query49	870	406	388	388
query50	798	408	406	406
query51	6794	6717	6804	6717
query52	109	89	95	89
query53	357	282	281	281
query54	308	240	240	240
query55	76	73	73	73
query56	246	229	232	229
query57	1220	1162	1106	1106
query58	226	205	214	205
query59	3423	3167	3079	3079
query60	258	262	235	235
query61	127	104	103	103
query62	606	453	445	445
query63	315	285	279	279
query64	5167	3928	3947	3928
query65	3060	3002	3040	3002
query66	752	325	319	319
query67	15498	15041	15155	15041
query68	9479	565	571	565
query69	594	301	316	301
query70	1444	1140	1115	1115
query71	1489	1267	1271	1267
query72	6983	2598	2434	2434
query73	1642	328	328	328
query74	6788	6465	6403	6403
query75	4486	2662	2622	2622
query76	5921	1064	999	999
query77	647	270	274	270
query78	11106	10137	10213	10137
query79	12362	527	527	527
query80	2834	434	442	434
query81	538	240	241	240
query82	282	103	95	95
query83	213	164	165	164
query84	263	83	82	82
query85	1059	256	256	256
query86	343	317	303	303
query87	3543	3301	3283	3283
query88	5343	2430	2490	2430
query89	506	384	375	375
query90	2380	178	179	178
query91	121	93	95	93
query92	60	45	47	45
query93	8116	509	504	504
query94	1525	173	174	173
query95	379	284	286	284
query96	602	270	268	268
query97	3140	2961	2973	2961
query98	245	215	216	215
query99	1217	865	853	853
Total cold run time: 307325 ms
Total hot run time: 185549 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c4f25badbbbad57a4f0330291beaff7d61f2c72d, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.49	0.47
query6	1.47	0.74	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.54	0.51	0.49
query10	0.56	0.55	0.54
query11	0.16	0.12	0.12
query12	0.14	0.12	0.13
query13	0.61	0.58	0.59
query14	0.75	0.77	0.78
query15	0.82	0.81	0.81
query16	0.38	0.37	0.36
query17	0.95	0.93	0.97
query18	0.19	0.29	0.24
query19	1.85	1.71	1.66
query20	0.02	0.01	0.01
query21	15.40	0.65	0.64
query22	4.56	6.71	2.20
query23	18.33	1.35	1.20
query24	1.81	0.22	0.22
query25	0.14	0.08	0.08
query26	0.25	0.16	0.16
query27	0.09	0.07	0.07
query28	13.37	1.00	0.97
query29	12.62	3.24	3.22
query30	0.26	0.06	0.06
query31	2.89	0.38	0.37
query32	3.30	0.45	0.46
query33	2.83	2.77	2.83
query34	17.06	4.42	4.45
query35	4.45	4.46	4.48
query36	0.64	0.45	0.46
query37	0.18	0.15	0.15
query38	0.15	0.13	0.14
query39	0.04	0.03	0.04
query40	0.18	0.14	0.15
query41	0.10	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.03	0.03
Total cold run time: 109.79 s
Total hot run time: 30.4 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c4f25badbbbad57a4f0330291beaff7d61f2c72d with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       14.5 seconds inserted 10000000 Rows, about 689K ops/s

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 18, 2024
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit 34c57fa into apache:master Apr 18, 2024
25 of 29 checks passed
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 9, 2024
)

If DEBUG build type, record size of each memory alloc and free, print no free size or no alloc size when query MemTracker is destructed, if necessary, record stack trace.

add global PointQueryExecutor memory tracker in ExecEnv, because memory may be shared between PointQueryExecutors of different pointer queries, but memory will not be shared between PointQueryExecutor and Fragment of the same pointer query.

If DEBUG build type, if query memory tracker not equal to 0 when query ends, BE will crash.
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 9, 2024
)

If DEBUG build type, record size of each memory alloc and free, print no free size or no alloc size when query MemTracker is destructed, if necessary, record stack trace.

add global PointQueryExecutor memory tracker in ExecEnv, because memory may be shared between PointQueryExecutors of different pointer queries, but memory will not be shared between PointQueryExecutor and Fragment of the same pointer query.

If DEBUG build type, if query memory tracker not equal to 0 when query ends, BE will crash.
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 10, 2024
)

If DEBUG build type, record size of each memory alloc and free, print no free size or no alloc size when query MemTracker is destructed, if necessary, record stack trace.

add global PointQueryExecutor memory tracker in ExecEnv, because memory may be shared between PointQueryExecutors of different pointer queries, but memory will not be shared between PointQueryExecutor and Fragment of the same pointer query.

If DEBUG build type, if query memory tracker not equal to 0 when query ends, BE will crash.
yiguolei pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 10, 2024
)

If DEBUG build type, record size of each memory alloc and free, print no free size or no alloc size when query MemTracker is destructed, if necessary, record stack trace.

add global PointQueryExecutor memory tracker in ExecEnv, because memory may be shared between PointQueryExecutors of different pointer queries, but memory will not be shared between PointQueryExecutor and Fragment of the same pointer query.

If DEBUG build type, if query memory tracker not equal to 0 when query ends, BE will crash.
yiguolei pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 11, 2024
)

If DEBUG build type, record size of each memory alloc and free, print no free size or no alloc size when query MemTracker is destructed, if necessary, record stack trace.

add global PointQueryExecutor memory tracker in ExecEnv, because memory may be shared between PointQueryExecutors of different pointer queries, but memory will not be shared between PointQueryExecutor and Fragment of the same pointer query.

If DEBUG build type, if query memory tracker not equal to 0 when query ends, BE will crash.
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 13, 2024
)

If DEBUG build type, record size of each memory alloc and free, print no free size or no alloc size when query MemTracker is destructed, if necessary, record stack trace.

add global PointQueryExecutor memory tracker in ExecEnv, because memory may be shared between PointQueryExecutors of different pointer queries, but memory will not be shared between PointQueryExecutor and Fragment of the same pointer query.

If DEBUG build type, if query memory tracker not equal to 0 when query ends, BE will crash.
yiguolei added a commit that referenced this pull request Sep 14, 2024
pick
#33396
#33862
#33853
#33732
#33841
#33933
#34901
#35014

---------

Co-authored-by: yiguolei <676222867@qq.com>
Co-authored-by: yiguolei <yiguolei@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants