Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](load) reduce memory reserved in memtable limiter #37511

Merged
merged 2 commits into from
Jul 10, 2024
Merged

[improve](load) reduce memory reserved in memtable limiter #37511

merged 2 commits into from
Jul 10, 2024

Conversation

kaijchen
Copy link
Contributor

@kaijchen kaijchen commented Jul 9, 2024 •
edited
Loading

Proposed changes

#37174 increased max sys_mem_available_low_water_mark from 1.6G to 6.4G, causing memtable memory limiter reserving too much memory.

This PR adds a BE config memtable_limiter_reserved_memory_bytes, defaults to 800MB, which defines bytes reserved in memtable memory limiter.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@kaijchen
Copy link
Contributor Author

kaijchen commented Jul 9, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40139 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c7eaf95d3c6a82479c8622e7060f03ae824adb1, data reload: false

------ Round 1 ----------------------------------
q1	17606	4430	4281	4281
q2	2007	197	200	197
q3	10441	1162	1097	1097
q4	10205	824	737	737
q5	7489	2715	2615	2615
q6	221	137	139	137
q7	949	601	617	601
q8	9222	2082	2114	2082
q9	9017	6525	6541	6525
q10	8905	3751	3742	3742
q11	463	235	239	235
q12	472	235	236	235
q13	17773	2986	2984	2984
q14	268	234	244	234
q15	522	480	488	480
q16	498	403	391	391
q17	1002	643	703	643
q18	8013	7622	7577	7577
q19	5329	1532	1535	1532
q20	657	329	327	327
q21	4957	3154	3847	3154
q22	384	338	333	333
Total cold run time: 116400 ms
Total hot run time: 40139 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4359	4254	4234	4234
q2	358	268	258	258
q3	2975	2892	2901	2892
q4	1996	1658	1684	1658
q5	5703	5564	5485	5485
q6	228	139	135	135
q7	2244	1902	1878	1878
q8	3267	3420	3459	3420
q9	8780	8788	8869	8788
q10	4109	3865	3801	3801
q11	627	499	490	490
q12	839	622	638	622
q13	16802	3153	3201	3153
q14	300	277	283	277
q15	534	474	503	474
q16	487	443	431	431
q17	1861	1548	1486	1486
q18	8188	7930	7908	7908
q19	1806	1703	1549	1549
q20	2134	1841	1847	1841
q21	5073	5029	4860	4860
q22	639	560	567	560
Total cold run time: 73309 ms
Total hot run time: 56200 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174882 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c7eaf95d3c6a82479c8622e7060f03ae824adb1, data reload: false

query1	912	383	371	371
query2	6437	2473	2393	2393
query3	6628	208	218	208
query4	27384	17717	17310	17310
query5	3660	511	497	497
query6	251	167	171	167
query7	4575	289	287	287
query8	310	283	284	283
query9	8504	2416	2419	2416
query10	568	309	278	278
query11	11957	10033	10102	10033
query12	130	88	79	79
query13	1642	367	374	367
query14	10155	7703	7669	7669
query15	227	184	186	184
query16	7713	304	295	295
query17	1843	555	518	518
query18	1956	271	274	271
query19	189	151	151	151
query20	87	81	82	81
query21	209	138	129	129
query22	4322	4077	4165	4077
query23	33924	33826	33667	33667
query24	10655	2876	2939	2876
query25	570	381	401	381
query26	703	148	147	147
query27	2230	279	281	279
query28	6098	2195	2183	2183
query29	890	636	634	634
query30	252	156	154	154
query31	986	773	791	773
query32	99	54	58	54
query33	671	303	305	303
query34	892	483	501	483
query35	687	575	588	575
query36	1151	1008	955	955
query37	147	84	90	84
query38	3022	2840	2789	2789
query39	896	852	825	825
query40	200	122	119	119
query41	55	51	51	51
query42	115	98	102	98
query43	567	572	542	542
query44	1069	730	728	728
query45	193	163	161	161
query46	1087	714	765	714
query47	1858	1779	1761	1761
query48	383	296	302	296
query49	886	436	443	436
query50	795	395	403	395
query51	6805	6889	6796	6796
query52	109	97	92	92
query53	363	289	334	289
query54	851	456	448	448
query55	74	72	74	72
query56	295	270	267	267
query57	1142	1059	1070	1059
query58	256	259	247	247
query59	3479	3429	3327	3327
query60	304	276	283	276
query61	96	92	93	92
query62	817	636	634	634
query63	326	299	297	297
query64	9115	2193	1622	1622
query65	3153	3142	3107	3107
query66	741	330	329	329
query67	15375	15053	14856	14856
query68	4532	556	546	546
query69	650	474	362	362
query70	1177	1133	1139	1133
query71	398	284	287	284
query72	7327	5718	5508	5508
query73	762	324	325	324
query74	5861	5467	5489	5467
query75	3421	2663	2664	2663
query76	2676	937	995	937
query77	632	312	311	311
query78	9549	8899	8847	8847
query79	3270	523	514	514
query80	2140	496	481	481
query81	598	226	223	223
query82	1012	144	138	138
query83	324	179	169	169
query84	274	90	87	87
query85	1370	315	300	300
query86	466	306	323	306
query87	3276	3094	3092	3092
query88	4278	2356	2362	2356
query89	487	387	391	387
query90	1729	190	191	190
query91	130	101	104	101
query92	57	49	49	49
query93	4987	524	531	524
query94	1086	206	206	206
query95	406	373	312	312
query96	612	267	270	267
query97	3166	3038	3012	3012
query98	218	200	207	200
query99	1647	1285	1249	1249
Total cold run time: 282559 ms
Total hot run time: 174882 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c7eaf95d3c6a82479c8622e7060f03ae824adb1, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.50	0.49
query6	1.14	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.50
query10	0.55	0.54	0.55
query11	0.14	0.11	0.11
query12	0.15	0.12	0.13
query13	0.59	0.59	0.58
query14	0.76	0.78	0.78
query15	0.85	0.83	0.82
query16	0.37	0.37	0.36
query17	1.04	1.00	0.96
query18	0.23	0.22	0.22
query19	1.92	1.75	1.80
query20	0.01	0.00	0.01
query21	15.38	0.76	0.66
query22	4.74	6.65	2.12
query23	18.28	1.35	1.23
query24	2.08	0.24	0.22
query25	0.16	0.09	0.08
query26	0.29	0.21	0.21
query27	0.45	0.23	0.23
query28	13.25	1.01	1.02
query29	12.62	3.28	3.33
query30	0.26	0.06	0.05
query31	2.90	0.40	0.40
query32	3.27	0.48	0.46
query33	2.86	2.85	2.91
query34	17.20	4.36	4.42
query35	4.46	4.38	4.43
query36	0.64	0.48	0.50
query37	0.19	0.15	0.14
query38	0.15	0.14	0.15
query39	0.05	0.03	0.04
query40	0.15	0.12	0.13
query41	0.09	0.05	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.47 s
Total hot run time: 30.76 s

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40171 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f2ec6a41feeea4613407ee660e8753b85ab6315, data reload: false

------ Round 1 ----------------------------------
q1	17861	4619	4408	4408
q2	2653	193	200	193
q3	11697	1191	1060	1060
q4	10649	828	860	828
q5	8256	2815	2714	2714
q6	227	140	139	139
q7	969	605	607	605
q8	9225	2074	2134	2074
q9	9018	6528	6501	6501
q10	8927	3751	3719	3719
q11	496	240	238	238
q12	408	237	228	228
q13	17757	2994	3006	2994
q14	271	231	226	226
q15	530	478	488	478
q16	502	383	370	370
q17	978	710	686	686
q18	8181	7539	7394	7394
q19	7979	1467	1438	1438
q20	690	331	332	331
q21	4953	3308	3215	3215
q22	393	332	332	332
Total cold run time: 122620 ms
Total hot run time: 40171 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4438	4246	4285	4246
q2	392	264	261	261
q3	2995	2782	2726	2726
q4	1906	1618	1611	1611
q5	5303	5340	5346	5340
q6	230	134	133	133
q7	2166	1719	1746	1719
q8	3228	3403	3354	3354
q9	8434	8330	8407	8330
q10	3871	3676	3685	3676
q11	561	490	475	475
q12	800	641	596	596
q13	17690	2987	2979	2979
q14	302	268	250	250
q15	508	487	477	477
q16	477	400	432	400
q17	1804	1497	1454	1454
q18	7606	7526	7540	7526
q19	1730	1624	1625	1624
q20	2005	1781	1754	1754
q21	4984	4825	4676	4676
q22	646	500	523	500
Total cold run time: 72076 ms
Total hot run time: 54107 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172771 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f2ec6a41feeea4613407ee660e8753b85ab6315, data reload: false

query1	906	371	371	371
query2	6455	2346	2290	2290
query3	6649	203	216	203
query4	28502	17618	17227	17227
query5	4202	497	498	497
query6	294	180	152	152
query7	4585	294	293	293
query8	298	301	285	285
query9	8612	2412	2390	2390
query10	429	290	275	275
query11	11314	10136	10089	10089
query12	137	85	88	85
query13	1643	374	376	374
query14	9685	7472	7451	7451
query15	239	185	180	180
query16	7543	310	305	305
query17	1389	540	525	525
query18	1901	267	266	266
query19	215	144	148	144
query20	87	87	79	79
query21	209	130	130	130
query22	4238	4017	4089	4017
query23	33886	33189	34708	33189
query24	12355	2813	2833	2813
query25	535	355	363	355
query26	1109	149	150	149
query27	2854	275	269	269
query28	7184	2121	2114	2114
query29	851	630	612	612
query30	290	146	148	146
query31	999	745	737	737
query32	94	50	51	50
query33	767	296	300	296
query34	965	486	482	482
query35	705	571	550	550
query36	1108	911	943	911
query37	218	78	75	75
query38	2872	2714	2661	2661
query39	867	790	808	790
query40	277	116	120	116
query41	53	51	55	51
query42	112	98	103	98
query43	611	535	526	526
query44	1265	725	721	721
query45	197	157	163	157
query46	1101	754	718	718
query47	1839	1772	1759	1759
query48	392	298	294	294
query49	1170	427	407	407
query50	794	398	385	385
query51	6888	6804	6760	6760
query52	105	92	103	92
query53	361	298	291	291
query54	880	476	447	447
query55	75	74	72	72
query56	302	282	270	270
query57	1162	1049	1094	1049
query58	250	252	245	245
query59	3601	3081	3144	3081
query60	313	275	276	275
query61	98	91	96	91
query62	849	670	657	657
query63	328	292	289	289
query64	10498	2159	1618	1618
query65	3238	3152	3105	3105
query66	1234	332	334	332
query67	15304	14911	14985	14911
query68	4558	553	552	552
query69	466	326	317	317
query70	1139	1133	1160	1133
query71	407	279	272	272
query72	6596	5121	5225	5121
query73	774	334	323	323
query74	6039	5459	5498	5459
query75	3433	2646	2636	2636
query76	2825	952	957	952
query77	487	293	304	293
query78	9372	8928	9490	8928
query79	3043	513	520	513
query80	2175	489	483	483
query81	578	219	217	217
query82	1420	140	136	136
query83	324	182	173	173
query84	281	93	90	90
query85	1596	315	297	297
query86	491	296	328	296
query87	3247	3117	3108	3108
query88	3830	2363	2399	2363
query89	487	404	382	382
query90	1859	267	182	182
query91	130	102	101	101
query92	66	48	51	48
query93	3836	529	519	519
query94	1260	211	212	211
query95	411	311	309	309
query96	600	265	263	263
query97	3195	3033	3007	3007
query98	221	206	197	197
query99	1784	1246	1298	1246
Total cold run time: 287418 ms
Total hot run time: 172771 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f2ec6a41feeea4613407ee660e8753b85ab6315, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.09	0.09
query5	0.51	0.48	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.55	0.49	0.49
query10	0.56	0.55	0.54
query11	0.15	0.11	0.12
query12	0.14	0.13	0.13
query13	0.60	0.58	0.58
query14	0.75	0.80	0.78
query15	0.87	0.80	0.82
query16	0.37	0.34	0.37
query17	1.00	1.04	1.03
query18	0.22	0.21	0.21
query19	1.80	1.78	1.71
query20	0.01	0.01	0.01
query21	15.39	0.76	0.64
query22	4.54	7.58	1.77
query23	18.28	1.34	1.29
query24	2.09	0.24	0.22
query25	0.17	0.09	0.08
query26	0.37	0.22	0.21
query27	0.45	0.23	0.24
query28	13.23	1.00	1.00
query29	12.57	3.39	3.30
query30	0.26	0.06	0.05
query31	2.86	0.38	0.38
query32	3.29	0.47	0.46
query33	2.91	2.87	2.97
query34	17.23	4.38	4.36
query35	4.47	4.38	4.46
query36	0.65	0.46	0.47
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.12	0.12
query41	0.08	0.04	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 110.23 s
Total hot run time: 30.48 s

liaoxin01
liaoxin01 approved these changes Jul 10, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 10, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jul 10, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 3197f9b into apache:master Jul 10, 2024
25 of 30 checks passed
kaijchen added a commit to kaijchen/doris that referenced this pull request Jul 12, 2024
@kaijchen
[improve](load) reduce memory reserved in memtable limiter (apache#37511
fef9384 
)

causing memtable memory limiter reserving too much memory.

This PR adds a BE config `memtable_limiter_reserved_memory_bytes`,
defaults to 800MB, which defines bytes reserved in memtable memory
limiter.
@kaijchen kaijchen mentioned this pull request Jul 12, 2024
Merged
liaoxin01 pushed a commit that referenced this pull request Jul 15, 2024
@kaijchen
[improve](load) reduce memory reserved in memtable limiter (#37511) (#…
232202b 
…37699)

cherry-pick #37511
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@kaijchen @seawinde
[improve](load) reduce memory reserved in memtable limiter (apache#37511
bf02a47 
)

## Proposed changes

apache#37174 increased max sys_mem_available_low_water_mark from 1.6G to 6.4G,
causing memtable memory limiter reserving too much memory.

This PR adds a BE config `memtable_limiter_reserved_memory_bytes`,
defaults to 800MB, which defines bytes reserved in memtable memory
limiter.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@kaijchen @dataroaring
[improve](load) reduce memory reserved in memtable limiter (#37511)
4dcde30 
## Proposed changes

#37174 increased max sys_mem_available_low_water_mark from 1.6G to 6.4G,
causing memtable memory limiter reserving too much memory.

This PR adds a BE config `memtable_limiter_reserved_memory_bytes`,
defaults to 800MB, which defines bytes reserved in memtable memory
limiter.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kaijchen @doris-robot @liaoxin01 @dataroaring