Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](group commit) fix group commit can not get block queue and may stuck #37260

Merged
merged 1 commit into from
Jul 6, 2024

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Jul 4, 2024

Proposed changes

  1. fix can not get block queue in low frequency
  2. fix the get_block may stuck

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mymeiyi mymeiyi changed the title fix group commit [fix](group commit) fix group commit can not get block queue and may stuck Jul 4, 2024
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Jul 4, 2024

run buildall

Copy link
Contributor

github-actions bot commented Jul 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Jul 4, 2024

run buildall

Copy link
Contributor

github-actions bot commented Jul 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39740 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2765b4737153e8a5a91a60ab9cc717f7d45ec813, data reload: false

------ Round 1 ----------------------------------
q1	17601	4320	4327	4320
q2	2016	193	187	187
q3	10446	1206	1036	1036
q4	10191	774	792	774
q5	7479	2587	2653	2587
q6	219	137	135	135
q7	948	586	608	586
q8	9226	2053	2077	2053
q9	8755	6505	6467	6467
q10	8987	3736	3729	3729
q11	461	237	235	235
q12	463	237	226	226
q13	17798	2968	2984	2968
q14	259	221	214	214
q15	524	502	476	476
q16	510	379	387	379
q17	949	675	659	659
q18	7969	7487	7427	7427
q19	4212	1402	1401	1401
q20	660	313	332	313
q21	4953	3757	3224	3224
q22	404	348	344	344
Total cold run time: 115030 ms
Total hot run time: 39740 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4311	4221	4266	4221
q2	368	267	266	266
q3	2983	2759	2918	2759
q4	2050	1688	1754	1688
q5	5583	5521	5469	5469
q6	230	130	129	129
q7	2217	1793	1847	1793
q8	3247	3408	3418	3408
q9	8697	8700	8788	8700
q10	4113	3877	3766	3766
q11	594	505	491	491
q12	790	635	654	635
q13	15941	3231	3163	3163
q14	306	305	282	282
q15	532	508	497	497
q16	499	439	442	439
q17	1783	1527	1500	1500
q18	8200	7895	7991	7895
q19	2331	1670	1601	1601
q20	2067	1924	1846	1846
q21	5063	4914	4962	4914
q22	598	588	589	588
Total cold run time: 72503 ms
Total hot run time: 56050 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172391 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2765b4737153e8a5a91a60ab9cc717f7d45ec813, data reload: false

query1	926	383	374	374
query2	6455	2493	2415	2415
query3	6639	202	210	202
query4	19282	17541	17304	17304
query5	3630	478	480	478
query6	265	177	160	160
query7	4599	312	291	291
query8	332	312	319	312
query9	8635	2402	2395	2395
query10	576	300	290	290
query11	10362	10110	9916	9916
query12	118	90	84	84
query13	1653	378	368	368
query14	10199	7586	6381	6381
query15	227	194	186	186
query16	7721	329	313	313
query17	1808	551	560	551
query18	1951	284	283	283
query19	194	156	159	156
query20	92	82	92	82
query21	213	137	128	128
query22	4342	4001	4033	4001
query23	33831	33751	33639	33639
query24	10574	2942	2888	2888
query25	632	401	402	401
query26	714	162	158	158
query27	2196	326	336	326
query28	6159	2172	2152	2152
query29	909	669	666	666
query30	257	154	161	154
query31	990	772	754	754
query32	100	58	56	56
query33	688	315	297	297
query34	877	475	500	475
query35	736	645	659	645
query36	1154	984	971	971
query37	143	83	83	83
query38	2948	2897	2857	2857
query39	886	832	830	830
query40	207	133	128	128
query41	58	55	54	54
query42	118	107	99	99
query43	605	535	527	527
query44	1080	741	726	726
query45	198	170	167	167
query46	1071	708	713	708
query47	1868	1760	1746	1746
query48	380	308	313	308
query49	947	407	413	407
query50	758	393	382	382
query51	6826	6788	6718	6718
query52	111	94	96	94
query53	378	293	286	286
query54	852	450	455	450
query55	78	74	72	72
query56	276	257	273	257
query57	1126	1012	1050	1012
query58	268	255	242	242
query59	3440	3196	3033	3033
query60	306	293	282	282
query61	97	95	96	95
query62	621	434	444	434
query63	321	297	300	297
query64	8515	2239	1809	1809
query65	3146	3098	3108	3098
query66	747	326	331	326
query67	15798	15147	14949	14949
query68	8214	541	549	541
query69	761	501	337	337
query70	1228	1078	1116	1078
query71	514	282	274	274
query72	9131	5336	5565	5336
query73	872	325	321	321
query74	5845	5497	5445	5445
query75	5093	2677	2656	2656
query76	4711	1019	953	953
query77	781	303	296	296
query78	9671	8803	8821	8803
query79	10280	521	515	515
query80	1266	470	530	470
query81	549	219	216	216
query82	769	104	103	103
query83	329	167	166	166
query84	273	83	86	83
query85	1316	292	289	289
query86	416	328	320	320
query87	3420	3047	3052	3047
query88	4796	2375	2382	2375
query89	534	382	377	377
query90	2057	186	186	186
query91	130	98	99	98
query92	59	87	47	47
query93	7172	506	492	492
query94	1271	214	204	204
query95	411	315	311	311
query96	598	270	264	264
query97	3178	3009	3009	3009
query98	224	198	200	198
query99	1153	858	853	853
Total cold run time: 291231 ms
Total hot run time: 172391 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2765b4737153e8a5a91a60ab9cc717f7d45ec813, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.07	0.07
query5	0.49	0.48	0.48
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.49	0.50
query10	0.53	0.55	0.54
query11	0.15	0.12	0.11
query12	0.16	0.13	0.12
query13	0.59	0.58	0.58
query14	0.76	0.79	0.78
query15	0.86	0.81	0.82
query16	0.36	0.35	0.37
query17	0.97	1.03	0.99
query18	0.21	0.26	0.26
query19	1.92	1.79	1.71
query20	0.02	0.01	0.01
query21	15.41	0.74	0.66
query22	4.45	6.89	1.92
query23	18.31	1.50	1.23
query24	2.07	0.25	0.21
query25	0.15	0.09	0.09
query26	0.29	0.21	0.21
query27	0.46	0.23	0.24
query28	13.31	1.02	1.00
query29	12.66	3.28	3.27
query30	0.26	0.07	0.06
query31	2.86	0.38	0.38
query32	3.28	0.47	0.48
query33	2.90	2.89	2.91
query34	17.00	4.39	4.37
query35	4.40	4.42	4.45
query36	0.65	0.48	0.49
query37	0.19	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.15	0.13	0.11
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.99 s
Total hot run time: 30.64 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 5, 2024
Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit c037302 into apache:master Jul 6, 2024
25 of 29 checks passed
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
…stuck (#37260)

## Proposed changes

1. fix `can not get block queue` in low frequency 
2. fix the get_block may stuck
dataroaring pushed a commit that referenced this pull request Jul 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants