Skip to content

branch-3.0: [fix](memory) Fix ThreadMemTrackerMgr::limiter_mem_tracker() performance #50525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
May 21, 2025
Merged

branch-3.0: [fix](memory) Fix ThreadMemTrackerMgr::limiter_mem_tracker() performance #50525

dataroaring merged 1 commit into from
May 21, 2025

Conversation

@xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Apr 28, 2025 •
edited
Loading

pick #50462

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@xinyiZzz xinyiZzz requested a review from dataroaring as a code owner April 28, 2025 15:26
@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Apr 28, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 28, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xinyiZzz
[fix](memory) Fix ThreadMemTrackerMgr::limiter_mem_tracker() perfor…
ca21f28 
…mance (apache#50462)

Prefer use `limiter_mem_tracker`, which is faster than
`limiter_mem_tracker_sptr`.
when multiple threads hold the same `std::shared_ptr` at the same time,
modifying the `std::shared_ptr` reference count will be expensive when
there is high concurrency.
@xinyiZzz xinyiZzz force-pushed the branch-3.0_20250428 branch from 70cb99a to ca21f28 Compare April 28, 2025 15:39
@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Apr 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 28, 2025

TPC-H: Total hot run time: 40232 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca21f2842b041b83b64853d263435cf5359d2f4d, data reload: false

------ Round 1 ----------------------------------
q1	17600	6797	6623	6623
q2	2077	204	176	176
q3	10530	1078	1173	1078
q4	10533	781	867	781
q5	7747	2859	2888	2859
q6	218	139	137	137
q7	974	619	615	615
q8	9358	1987	2034	1987
q9	6749	6466	6469	6466
q10	7046	2297	2242	2242
q11	457	263	264	263
q12	396	209	208	208
q13	17804	2980	2997	2980
q14	252	207	238	207
q15	510	466	463	463
q16	497	411	383	383
q17	979	534	548	534
q18	7428	6789	6742	6742
q19	1395	1077	1052	1052
q20	470	205	208	205
q21	4021	3238	3234	3234
q22	1122	997	1010	997
Total cold run time: 108163 ms
Total hot run time: 40232 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6605	7065	6582	6582
q2	334	224	228	224
q3	2934	2791	2968	2791
q4	2049	1876	1815	1815
q5	5754	5719	5770	5719
q6	201	129	130	129
q7	2228	1810	1794	1794
q8	3419	3583	3541	3541
q9	8927	8953	8848	8848
q10	3576	3544	3545	3544
q11	593	501	485	485
q12	805	597	638	597
q13	9153	3155	3197	3155
q14	296	267	268	267
q15	519	472	470	470
q16	494	452	438	438
q17	1815	1638	1624	1624
q18	8340	7920	7697	7697
q19	1679	1619	1590	1590
q20	2108	1815	1820	1815
q21	5558	5348	5361	5348
q22	1135	1070	1024	1024
Total cold run time: 68522 ms
Total hot run time: 59497 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 28, 2025

BE UT Coverage Report

Increment line coverage 80.00% (16/20) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 40.51% (10638/26259)
Line Coverage 31.25% (90140/288471)
Region Coverage 30.38% (46539/153181)
Branch Coverage 26.89% (23812/88568)

@doris-robot
Copy link

doris-robot commented Apr 28, 2025

TPC-DS: Total hot run time: 197342 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca21f2842b041b83b64853d263435cf5359d2f4d, data reload: false

query1	1290	902	918	902
query2	6274	1891	1923	1891
query3	10912	4536	4369	4369
query4	61845	29612	23437	23437
query5	5233	472	464	464
query6	401	181	203	181
query7	5493	308	318	308
query8	311	226	227	226
query9	8641	2626	2608	2608
query10	461	277	261	261
query11	17456	15405	15820	15405
query12	162	110	102	102
query13	1413	417	420	417
query14	10467	7316	6750	6750
query15	205	179	174	174
query16	7157	511	486	486
query17	1119	600	604	600
query18	1839	328	343	328
query19	213	164	159	159
query20	118	108	110	108
query21	203	103	104	103
query22	4698	4454	4655	4454
query23	34979	34404	34105	34105
query24	6110	2871	2934	2871
query25	549	442	427	427
query26	664	172	175	172
query27	2067	365	350	350
query28	4070	2217	2165	2165
query29	724	478	457	457
query30	245	162	169	162
query31	979	814	825	814
query32	73	59	61	59
query33	420	303	303	303
query34	927	499	512	499
query35	842	731	752	731
query36	1082	978	967	967
query37	109	63	67	63
query38	4119	4014	3974	3974
query39	1527	1509	1444	1444
query40	204	103	107	103
query41	49	49	51	49
query42	115	108	104	104
query43	520	487	479	479
query44	1168	820	829	820
query45	185	168	168	168
query46	1140	736	738	736
query47	1970	1941	1943	1941
query48	481	389	385	385
query49	751	419	427	419
query50	871	443	420	420
query51	7294	7316	7305	7305
query52	108	91	94	91
query53	263	192	191	191
query54	573	499	481	481
query55	86	82	76	76
query56	273	244	254	244
query57	1291	1194	1128	1128
query58	219	239	227	227
query59	3223	2974	3003	2974
query60	276	253	256	253
query61	117	116	130	116
query62	764	670	679	670
query63	222	190	185	185
query64	1396	658	635	635
query65	3246	3249	3215	3215
query66	722	293	317	293
query67	15848	15669	15639	15639
query68	4146	599	575	575
query69	431	259	264	259
query70	1115	1072	1129	1072
query71	347	287	243	243
query72	6351	4134	4201	4134
query73	753	356	364	356
query74	10149	9363	9109	9109
query75	3397	2586	2682	2586
query76	2069	1036	1106	1036
query77	506	278	272	272
query78	10610	9670	9572	9572
query79	1989	607	608	607
query80	1392	434	432	432
query81	514	222	224	222
query82	1247	89	87	87
query83	278	147	147	147
query84	281	82	86	82
query85	1046	315	292	292
query86	385	295	310	295
query87	4447	4278	4277	4277
query88	4070	2394	2356	2356
query89	413	293	289	289
query90	2019	187	187	187
query91	187	150	154	150
query92	67	53	49	49
query93	2925	565	561	561
query94	814	300	295	295
query95	358	261	266	261
query96	619	285	288	285
query97	3314	3127	3165	3127
query98	216	210	198	198
query99	1604	1286	1306	1286
Total cold run time: 316831 ms
Total hot run time: 197342 ms

@doris-robot
Copy link

doris-robot commented Apr 28, 2025

ClickBench: Total hot run time: 31.04 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca21f2842b041b83b64853d263435cf5359d2f4d, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.03	0.03
query3	0.22	0.06	0.07
query4	1.63	0.11	0.10
query5	0.53	0.50	0.52
query6	1.14	0.72	0.73
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.57	0.48	0.51
query10	0.55	0.54	0.55
query11	0.14	0.10	0.10
query12	0.14	0.12	0.12
query13	0.60	0.61	0.59
query14	0.78	0.79	0.79
query15	0.83	0.83	0.84
query16	0.38	0.41	0.38
query17	1.02	1.00	1.06
query18	0.24	0.22	0.22
query19	1.97	1.79	1.84
query20	0.01	0.02	0.01
query21	15.41	0.60	0.57
query22	2.45	2.29	2.84
query23	16.98	1.05	1.01
query24	3.13	1.10	1.31
query25	0.21	0.17	0.20
query26	0.30	0.13	0.13
query27	0.04	0.04	0.05
query28	10.15	0.49	0.51
query29	12.58	3.20	3.19
query30	0.24	0.06	0.06
query31	2.88	0.39	0.39
query32	3.23	0.47	0.45
query33	2.97	2.98	3.06
query34	17.07	4.50	4.54
query35	4.59	4.60	4.55
query36	0.67	0.48	0.50
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.27 s
Total hot run time: 31.04 s

dataroaring
dataroaring approved these changes May 21, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 42a3466 into apache:branch-3.0 May 21, 2025
21 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@xinyiZzz @hello-stephen @doris-robot @dataroaring