Skip to content

[Fix](inverted index) fix memory leak when inverted index writer not close #33724

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 1 commit into from
Apr 17, 2024
Merged

[Fix](inverted index) fix memory leak when inverted index writer not close #33724

yiguolei merged 1 commit into from
Apr 17, 2024

Conversation

@airborne12
Copy link
Member

@airborne12 airborne12 commented Apr 16, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Apr 16, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@airborne12
Copy link
Member Author

airborne12 commented Apr 16, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Apr 16, 2024

clang-tidy review says "All clean, LGTM! 👍"

dataroaring
dataroaring approved these changes Apr 16, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Apr 16, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 16, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Apr 16, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Apr 16, 2024

TeamCity be ut coverage result:
Function Coverage: 35.50% (8908/25092)
Line Coverage: 27.22% (73154/268731)
Region Coverage: 26.35% (37821/143550)
Branch Coverage: 23.13% (19272/83334)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6f89a12933b6506b6a9b1c83584ea056728547b1_6f89a12933b6506b6a9b1c83584ea056728547b1/report/index.html

@doris-robot
Copy link

doris-robot commented Apr 16, 2024

TPC-H: Total hot run time: 38684 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f89a12933b6506b6a9b1c83584ea056728547b1, data reload: false

------ Round 1 ----------------------------------
q1	17612	4383	4216	4216
q2	2005	197	189	189
q3	10632	1208	1231	1208
q4	10201	727	801	727
q5	7512	2663	2637	2637
q6	217	130	130	130
q7	1007	594	577	577
q8	9224	2042	2034	2034
q9	7556	6674	6593	6593
q10	8582	3584	3653	3584
q11	429	234	240	234
q12	469	220	222	220
q13	17965	3000	3034	3000
q14	281	233	237	233
q15	525	480	488	480
q16	531	378	391	378
q17	943	698	687	687
q18	7373	6766	6626	6626
q19	8787	1514	1586	1514
q20	694	329	317	317
q21	3614	2783	2882	2783
q22	361	317	322	317
Total cold run time: 116520 ms
Total hot run time: 38684 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4481	4410	4348	4348
q2	377	276	263	263
q3	3146	2866	2865	2865
q4	1983	1672	1702	1672
q5	5471	5335	5335	5335
q6	205	123	122	122
q7	2265	1834	1919	1834
q8	3215	3359	3342	3342
q9	8587	8533	8792	8533
q10	4109	4005	3932	3932
q11	591	506	501	501
q12	783	621	684	621
q13	17460	3219	3205	3205
q14	323	274	288	274
q15	530	485	492	485
q16	509	424	445	424
q17	1816	1500	1493	1493
q18	8361	7966	7652	7652
q19	1686	1540	1537	1537
q20	2090	1842	1864	1842
q21	10324	4969	4901	4901
q22	562	476	469	469
Total cold run time: 78874 ms
Total hot run time: 55650 ms

@doris-robot
Copy link

doris-robot commented Apr 16, 2024

TPC-DS: Total hot run time: 184877 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f89a12933b6506b6a9b1c83584ea056728547b1, data reload: false

query1	895	372	389	372
query2	6181	2856	2345	2345
query3	6665	206	204	204
query4	23617	21288	21326	21288
query5	4124	402	405	402
query6	274	197	179	179
query7	4581	284	292	284
query8	235	174	184	174
query9	8421	2378	2341	2341
query10	417	255	279	255
query11	14702	14226	14188	14188
query12	137	89	83	83
query13	1645	354	351	351
query14	9313	7335	7746	7335
query15	255	178	181	178
query16	7872	256	267	256
query17	1918	561	543	543
query18	1403	275	264	264
query19	197	156	145	145
query20	90	85	83	83
query21	200	122	122	122
query22	5026	4900	4858	4858
query23	33751	33013	33141	33013
query24	11130	2996	3112	2996
query25	581	389	388	388
query26	703	157	162	157
query27	2481	356	372	356
query28	5779	2116	2106	2106
query29	864	618	612	612
query30	310	185	180	180
query31	982	752	744	744
query32	97	53	53	53
query33	662	247	269	247
query34	883	494	506	494
query35	852	704	702	702
query36	1104	958	917	917
query37	112	73	70	70
query38	3553	3345	3338	3338
query39	1612	1592	1572	1572
query40	173	129	126	126
query41	46	42	41	41
query42	107	95	100	95
query43	606	558	531	531
query44	1102	745	738	738
query45	278	299	280	280
query46	1078	742	768	742
query47	2028	1949	1936	1936
query48	378	303	313	303
query49	837	374	398	374
query50	810	407	394	394
query51	6931	6819	6646	6646
query52	94	93	87	87
query53	341	273	278	273
query54	306	236	231	231
query55	77	69	73	69
query56	250	234	230	230
query57	1227	1107	1131	1107
query58	228	200	203	200
query59	3589	3305	3148	3148
query60	268	249	239	239
query61	109	133	106	106
query62	603	451	453	451
query63	305	274	277	274
query64	5009	4103	3906	3906
query65	3046	3020	3028	3020
query66	744	326	333	326
query67	15407	15030	14947	14947
query68	6858	532	545	532
query69	569	302	296	296
query70	1284	1150	1177	1150
query71	1448	1258	1265	1258
query72	6450	2669	2421	2421
query73	724	321	316	316
query74	6846	6446	6349	6349
query75	3734	2614	2571	2571
query76	4263	950	963	950
query77	623	266	264	264
query78	10825	10147	10501	10147
query79	7692	516	512	512
query80	1402	438	418	418
query81	512	242	239	239
query82	1605	96	98	96
query83	204	164	165	164
query84	268	87	81	81
query85	1293	269	320	269
query86	448	318	298	298
query87	3492	3247	3311	3247
query88	4961	2403	2399	2399
query89	484	375	358	358
query90	1899	178	179	178
query91	118	95	98	95
query92	55	51	46	46
query93	6324	490	492	490
query94	999	177	173	173
query95	388	293	299	293
query96	595	269	273	269
query97	3125	2929	2933	2929
query98	235	222	213	213
query99	1185	841	889	841
Total cold run time: 290509 ms
Total hot run time: 184877 ms

@doris-robot
Copy link

doris-robot commented Apr 16, 2024

ClickBench: Total hot run time: 30.57 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f89a12933b6506b6a9b1c83584ea056728547b1, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.08	0.08
query5	0.50	0.48	0.50
query6	1.45	0.73	0.72
query7	0.02	0.01	0.02
query8	0.06	0.05	0.04
query9	0.54	0.49	0.50
query10	0.55	0.55	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.60
query14	0.76	0.77	0.77
query15	0.83	0.82	0.80
query16	0.36	0.37	0.37
query17	0.98	1.00	0.96
query18	0.20	0.27	0.23
query19	1.76	1.69	1.72
query20	0.01	0.02	0.01
query21	15.40	0.65	0.65
query22	3.56	7.34	2.24
query23	18.32	1.43	1.27
query24	1.65	0.29	0.21
query25	0.15	0.08	0.08
query26	0.25	0.17	0.16
query27	0.08	0.08	0.08
query28	13.39	1.01	0.97
query29	12.71	3.20	3.21
query30	0.27	0.08	0.06
query31	2.83	0.39	0.38
query32	3.27	0.46	0.47
query33	2.83	2.85	2.86
query34	17.16	4.37	4.37
query35	4.46	4.42	4.51
query36	0.65	0.45	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.18	0.15	0.13
query41	0.10	0.05	0.05
query42	0.07	0.05	0.05
query43	0.04	0.03	0.03
Total cold run time: 108.68 s
Total hot run time: 30.57 s

@doris-robot
Copy link

doris-robot commented Apr 16, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6f89a12933b6506b6a9b1c83584ea056728547b1 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      34 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.3 seconds inserted 10000000 Rows, about 751K ops/s

@yiguolei yiguolei merged commit 22bfe7e into apache:master Apr 17, 2024
dataroaring pushed a commit that referenced this pull request Apr 17, 2024
@airborne12 @dataroaring
[Fix](inverted index) fix memory leak when inverted index writer not …
1dceb89 
…close (#33724)
yiguolei pushed a commit that referenced this pull request Apr 17, 2024
@airborne12 @Doris-Extras
[Fix](inverted index) fix memory leak when inverted index writer not …
e6725e9 
…close (#33724)
yiguolei pushed a commit that referenced this pull request Apr 17, 2024
@airborne12 @Doris-Extras
[Fix](inverted index) fix memory leak when inverted index writer not …
cb255f6 
…close (#33724)
@xiaokang xiaokang removed the p0_b label Apr 18, 2024
xiaokang pushed a commit to xiaokang/doris that referenced this pull request May 1, 2024
@airborne12 @xiaokang
[Fix](inverted index) fix memory leak when inverted index writer not …
67ffe40 
…close (apache#33724)
@xiaokang xiaokang mentioned this pull request May 1, 2024
Merged
@xiaokang xiaokang added the p0_b label May 1, 2024
xiaokang added a commit that referenced this pull request May 1, 2024
@xiaokang @airborne12
[Fix](inverted index) fix memory leak when inverted index writer not …
d0e89c0 
…close (#33724) (#34374)

Co-authored-by: airborne12 <airborne08@gmail.com>
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@xiaokang @airborne12
[Fix](inverted index) fix memory leak when inverted index writer not …
199595c 
…close (apache#33724) (apache#34374)

Co-authored-by: airborne12 <airborne08@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.10-merged dev/3.0.0-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@airborne12 @doris-robot @yiguolei @dataroaring @xiaokang