Skip to content

branch-3.0: [fix](group commit) reduce cpu cost for group_commit get_block #49822 #51079

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
May 21, 2025
Merged

branch-3.0: [fix](group commit) reduce cpu cost for group_commit get_block #49822 #51079

dataroaring merged 2 commits into from
May 21, 2025

Conversation

@mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented May 20, 2025

pick #49822

@mymeiyi mymeiyi requested a review from dataroaring as a code owner May 20, 2025 09:29
@hello-stephen
Copy link
Contributor

hello-stephen commented May 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 20, 2025

TPC-H: Total hot run time: 39951 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c6a7a2b5d500b0a53984f29e347be7e02f5d69f5, data reload: false

------ Round 1 ----------------------------------
q1	17579	6734	6581	6581
q2	2069	173	170	170
q3	10618	1096	1170	1096
q4	10554	797	900	797
q5	7735	2863	2778	2778
q6	224	135	134	134
q7	967	615	601	601
q8	9362	1911	2061	1911
q9	6642	6442	6397	6397
q10	7006	2305	2295	2295
q11	466	268	261	261
q12	399	220	210	210
q13	17766	3000	2995	2995
q14	238	212	203	203
q15	511	454	460	454
q16	694	582	597	582
q17	978	617	590	590
q18	7315	6577	6620	6577
q19	1392	1065	1033	1033
q20	475	203	204	203
q21	3952	3209	3113	3113
q22	1093	1005	970	970
Total cold run time: 108035 ms
Total hot run time: 39951 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6509	6528	6536	6528
q2	325	229	231	229
q3	2905	2775	2873	2775
q4	2041	1865	1779	1779
q5	5789	5711	5764	5711
q6	209	131	131	131
q7	2196	1818	1835	1818
q8	3390	3605	3541	3541
q9	8980	8826	8853	8826
q10	3540	3531	3509	3509
q11	593	498	506	498
q12	793	618	615	615
q13	9025	3140	3181	3140
q14	302	266	269	266
q15	508	472	469	469
q16	704	649	673	649
q17	1846	1613	1631	1613
q18	8214	7777	7828	7777
q19	1730	1503	1451	1451
q20	2033	1828	1820	1820
q21	5531	5409	5300	5300
q22	1124	1064	1015	1015
Total cold run time: 68287 ms
Total hot run time: 59460 ms

@doris-robot
Copy link

doris-robot commented May 20, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/22) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.11% (10859/26413)
Line Coverage 31.90% (92668/290505)
Region Coverage 30.98% (47818/154362)
Branch Coverage 27.44% (24483/89210)

@doris-robot
Copy link

doris-robot commented May 20, 2025

TPC-DS: Total hot run time: 199345 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c6a7a2b5d500b0a53984f29e347be7e02f5d69f5, data reload: false

query1	1294	894	903	894
query2	6313	2015	1998	1998
query3	10832	4359	4491	4359
query4	61685	35487	24161	24161
query5	5161	478	466	466
query6	306	183	178	178
query7	4899	312	308	308
query8	286	239	229	229
query9	6827	2628	2626	2626
query10	436	272	263	263
query11	15894	15348	15824	15348
query12	169	105	111	105
query13	1047	450	431	431
query14	10930	7386	7419	7386
query15	202	186	181	181
query16	7173	444	479	444
query17	1141	587	587	587
query18	1884	341	337	337
query19	221	168	177	168
query20	116	107	109	107
query21	207	104	107	104
query22	4613	4689	4472	4472
query23	34702	34154	34757	34154
query24	6230	2969	2921	2921
query25	523	419	411	411
query26	649	161	165	161
query27	2072	356	348	348
query28	3681	2483	2452	2452
query29	691	452	437	437
query30	245	170	175	170
query31	991	812	811	811
query32	67	61	55	55
query33	488	284	274	274
query34	923	511	515	511
query35	871	719	738	719
query36	1092	965	965	965
query37	118	67	68	67
query38	4063	3968	3942	3942
query39	1521	1483	1467	1467
query40	196	96	101	96
query41	50	51	49	49
query42	116	104	106	104
query43	544	493	509	493
query44	1190	830	823	823
query45	189	175	169	169
query46	1158	745	730	730
query47	2042	1925	1914	1914
query48	471	384	386	384
query49	722	391	383	383
query50	842	431	453	431
query51	7317	7187	7344	7187
query52	106	102	99	99
query53	258	188	185	185
query54	569	471	500	471
query55	83	77	81	77
query56	269	252	253	252
query57	1259	1162	1137	1137
query58	226	207	224	207
query59	3275	3040	3086	3040
query60	291	270	257	257
query61	119	136	113	113
query62	799	681	684	681
query63	217	191	187	187
query64	1418	669	656	656
query65	3244	3227	3218	3218
query66	717	292	299	292
query67	15937	15722	15879	15722
query68	3499	605	580	580
query69	421	259	267	259
query70	1154	1132	1096	1096
query71	342	268	254	254
query72	6349	4126	3991	3991
query73	751	357	354	354
query74	10374	9266	9286	9266
query75	3371	2656	2677	2656
query76	1714	1059	1001	1001
query77	477	276	286	276
query78	10892	9598	9603	9598
query79	1115	603	606	603
query80	815	432	445	432
query81	509	245	242	242
query82	1310	90	91	90
query83	245	153	154	153
query84	290	86	85	85
query85	909	370	363	363
query86	329	309	312	309
query87	4453	4391	4248	4248
query88	3378	2432	2403	2403
query89	409	298	293	293
query90	2028	188	185	185
query91	202	149	151	149
query92	58	49	51	49
query93	1057	557	560	557
query94	753	297	300	297
query95	349	264	257	257
query96	609	283	282	282
query97	3285	3150	3150	3150
query98	216	207	198	198
query99	1553	1295	1316	1295
Total cold run time: 307403 ms
Total hot run time: 199345 ms

@doris-robot
Copy link

doris-robot commented May 20, 2025

ClickBench: Total hot run time: 32.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c6a7a2b5d500b0a53984f29e347be7e02f5d69f5, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.10
query5	0.53	0.51	0.51
query6	1.14	0.73	0.74
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.51	0.50
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.74	2.85	2.76
query15	0.89	0.82	0.83
query16	0.38	0.37	0.38
query17	1.08	1.00	1.06
query18	0.24	0.22	0.22
query19	2.00	1.81	2.01
query20	0.01	0.01	0.02
query21	15.36	0.59	0.59
query22	2.96	2.80	1.73
query23	16.92	1.04	0.98
query24	2.89	1.26	1.69
query25	0.27	0.17	0.08
query26	0.49	0.14	0.14
query27	0.04	0.04	0.04
query28	9.76	0.55	0.47
query29	12.58	3.32	3.26
query30	0.24	0.06	0.06
query31	2.86	0.39	0.38
query32	3.24	0.46	0.46
query33	2.98	3.05	3.04
query34	16.97	4.45	4.51
query35	4.48	4.54	4.55
query36	0.69	0.48	0.48
query37	0.09	0.06	0.06
query38	0.06	0.03	0.03
query39	0.03	0.03	0.02
query40	0.17	0.12	0.13
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.31 s
Total hot run time: 32.63 s

dataroaring
dataroaring approved these changes May 21, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 212cd8f into apache:branch-3.0 May 21, 2025
22 of 24 checks passed
mymeiyi added a commit to mymeiyi/doris that referenced this pull request May 28, 2025
@mymeiyi
Revert "branch-3.0: [fix](group commit) reduce cpu cost for group_com…
e79a574 
…mit get_block apache#49822 (apache#51079)"

This reverts commit 212cd8f.
mymeiyi added a commit to mymeiyi/doris that referenced this pull request May 28, 2025
@mymeiyi
Revert "Revert "branch-3.0: [fix](group commit) reduce cpu cost for g…
dc340a6 
…roup_commit get_block apache#49822 (apache#51079)""

This reverts commit e79a574.
Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Jul 21, 2025
@mymeiyi
Select db core 4.0.5 group commit (apache#4057)
c88f80e 
pick apache#51079 and
apache#51375

---------

Co-authored-by: zhannngchen <zhangchen@selectdb.com>
Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Jul 30, 2025
@mymeiyi
Select db core 4.0.5 group commit (apache#4057)
dceb1ee 
pick apache#51079 and
apache#51375

---------

Co-authored-by: zhannngchen <zhangchen@selectdb.com>
@mymeiyi mymeiyi deleted the pick-49822 branch September 10, 2025 06:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mymeiyi @hello-stephen @doris-robot @dataroaring