Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](merge-on-write) Reduce the version not continuous logs for merge-on-write table #40946

Merged
merged 3 commits into from
Sep 19, 2024

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Sep 18, 2024

branch-2.1-pick: #40996
branch-2.0-pick: #40997

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 18, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41412 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a4d2f6ae35a44d594e6fe2dfea935313313ca30a, data reload: false

------ Round 1 ----------------------------------
q1	17908	7406	7307	7307
q2	2065	162	148	148
q3	10719	1081	1152	1081
q4	10371	739	720	720
q5	7777	3103	3042	3042
q6	239	149	147	147
q7	1059	626	593	593
q8	9427	2057	2080	2057
q9	6937	6413	6468	6413
q10	7026	2281	2289	2281
q11	437	247	249	247
q12	419	215	217	215
q13	17773	2967	2993	2967
q14	261	214	204	204
q15	570	533	520	520
q16	668	627	614	614
q17	998	846	879	846
q18	7500	6838	6638	6638
q19	1408	1040	1026	1026
q20	609	283	280	280
q21	4009	3203	3081	3081
q22	1136	1012	985	985
Total cold run time: 109316 ms
Total hot run time: 41412 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7342	7200	7196	7196
q2	328	232	226	226
q3	3093	2916	2973	2916
q4	2048	1893	1800	1800
q5	5618	5561	5682	5561
q6	235	142	142	142
q7	2224	1792	1766	1766
q8	3297	3418	3434	3418
q9	8866	8897	8783	8783
q10	3504	3456	3502	3456
q11	576	482	471	471
q12	818	640	624	624
q13	11812	3140	3143	3140
q14	310	276	277	276
q15	578	539	521	521
q16	697	657	686	657
q17	1800	1639	1611	1611
q18	8211	7913	7690	7690
q19	1755	1690	1627	1627
q20	2150	1912	1897	1897
q21	5499	5333	5343	5333
q22	1158	1064	996	996
Total cold run time: 71919 ms
Total hot run time: 60107 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198808 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a4d2f6ae35a44d594e6fe2dfea935313313ca30a, data reload: false

query1	1296	867	926	867
query2	6379	2083	2060	2060
query3	10770	3825	3872	3825
query4	64533	28832	23346	23346
query5	5026	475	475	475
query6	409	167	163	163
query7	5517	302	302	302
query8	322	220	221	220
query9	8420	2684	2679	2679
query10	453	293	270	270
query11	17061	15143	15749	15143
query12	158	105	108	105
query13	1460	431	404	404
query14	10874	7312	7656	7312
query15	206	177	170	170
query16	6709	520	522	520
query17	1194	589	585	585
query18	1480	318	310	310
query19	207	153	158	153
query20	135	111	111	111
query21	226	105	108	105
query22	4729	4498	4869	4498
query23	34717	33996	33959	33959
query24	5993	2947	2883	2883
query25	503	409	409	409
query26	649	158	165	158
query27	1660	280	285	280
query28	4188	2480	2478	2478
query29	677	432	434	432
query30	230	156	157	156
query31	977	769	808	769
query32	73	55	54	54
query33	436	297	300	297
query34	916	488	487	487
query35	842	762	746	746
query36	1058	959	933	933
query37	156	90	82	82
query38	3977	3993	4030	3993
query39	1560	1458	1395	1395
query40	214	97	95	95
query41	49	47	48	47
query42	124	93	94	93
query43	528	483	469	469
query44	1137	818	806	806
query45	196	164	161	161
query46	1140	751	749	749
query47	1901	1828	1834	1828
query48	498	347	362	347
query49	681	396	409	396
query50	829	411	401	401
query51	7010	7078	6794	6794
query52	97	85	80	80
query53	252	175	179	175
query54	561	456	473	456
query55	77	75	76	75
query56	283	260	251	251
query57	1208	1094	1110	1094
query58	214	230	258	230
query59	3092	2990	3171	2990
query60	306	273	263	263
query61	105	104	100	100
query62	750	664	671	664
query63	214	184	183	183
query64	1362	662	628	628
query65	3272	3160	3182	3160
query66	680	306	303	303
query67	15954	15668	15380	15380
query68	1281	555	539	539
query69	457	281	292	281
query70	1154	1134	1134	1134
query71	328	275	265	265
query72	6166	4014	3977	3977
query73	756	330	322	322
query74	9208	8985	8882	8882
query75	3318	2683	2664	2664
query76	1377	877	875	875
query77	427	289	290	289
query78	10240	11476	9340	9340
query79	1227	870	869	869
query80	808	568	593	568
query81	493	253	249	249
query82	1292	230	228	228
query83	223	155	157	155
query84	284	94	103	94
query85	740	370	363	363
query86	344	328	320	320
query87	4435	4433	4352	4352
query88	4407	4042	4047	4042
query89	382	356	360	356
query90	1687	309	304	304
query91	159	162	168	162
query92	75	73	71	71
query93	952	908	897	897
query94	686	394	338	338
query95	446	409	410	409
query96	481	485	483	483
query97	3140	3123	3134	3123
query98	230	229	219	219
query99	1454	1286	1295	1286
Total cold run time: 307567 ms
Total hot run time: 198808 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.16 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a4d2f6ae35a44d594e6fe2dfea935313313ca30a, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.02
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.50	0.53	0.52
query6	1.13	0.71	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.47
query10	0.56	0.56	0.54
query11	0.14	0.11	0.11
query12	0.15	0.11	0.11
query13	0.60	0.60	0.58
query14	2.96	2.95	2.93
query15	0.88	0.81	0.83
query16	0.37	0.39	0.39
query17	1.06	1.06	1.04
query18	0.20	0.19	0.19
query19	1.87	1.86	2.06
query20	0.01	0.00	0.00
query21	15.37	0.55	0.56
query22	2.61	2.34	2.07
query23	17.54	0.91	0.86
query24	2.84	0.27	1.49
query25	0.12	0.22	0.04
query26	0.44	0.14	0.13
query27	0.04	0.03	0.05
query28	11.09	1.10	1.06
query29	12.59	3.19	3.18
query30	0.24	0.06	0.06
query31	2.86	0.38	0.37
query32	3.29	0.47	0.47
query33	2.97	2.98	3.03
query34	17.19	4.36	4.39
query35	4.43	4.45	4.47
query36	0.67	0.48	0.49
query37	0.08	0.05	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.15	0.12	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.8 s
Total hot run time: 32.16 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.33% (9583/25672)
Line Coverage: 28.72% (79237/275883)
Region Coverage: 28.18% (41013/145517)
Branch Coverage: 24.80% (20898/84258)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a4d2f6ae35a44d594e6fe2dfea935313313ca30a_a4d2f6ae35a44d594e6fe2dfea935313313ca30a/report/index.html

@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 19, 2024

run buildall

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 41799 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4e981488348c2c78d1d6bddf383175d0d28a6bfb, data reload: false

------ Round 1 ----------------------------------
q1	17570	7398	7298	7298
q2	2067	169	157	157
q3	10758	1129	1197	1129
q4	10472	781	714	714
q5	7763	3114	3122	3114
q6	234	148	145	145
q7	1018	621	586	586
q8	9417	2093	2080	2080
q9	6886	6450	6461	6450
q10	7019	2276	2308	2276
q11	439	250	244	244
q12	414	209	209	209
q13	17798	2971	2990	2971
q14	232	217	228	217
q15	577	521	514	514
q16	675	633	616	616
q17	1013	802	808	802
q18	7346	6719	6827	6719
q19	1414	991	1021	991
q20	618	300	291	291
q21	4023	3340	3277	3277
q22	1127	1034	999	999
Total cold run time: 108880 ms
Total hot run time: 41799 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7287	7189	7213	7189
q2	332	232	226	226
q3	3093	2960	2951	2951
q4	2066	1879	1802	1802
q5	5575	5614	5653	5614
q6	234	146	144	144
q7	2189	1800	1772	1772
q8	3405	3436	3433	3433
q9	8842	9046	8837	8837
q10	3445	3465	3480	3465
q11	600	483	513	483
q12	793	609	602	602
q13	9883	3174	3150	3150
q14	323	276	281	276
q15	574	522	530	522
q16	725	676	670	670
q17	1816	1623	1575	1575
q18	8318	7855	7861	7855
q19	1736	1684	1636	1636
q20	2141	1923	1877	1877
q21	5582	5282	5410	5282
q22	1144	1050	1041	1041
Total cold run time: 70103 ms
Total hot run time: 60402 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198955 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4e981488348c2c78d1d6bddf383175d0d28a6bfb, data reload: false

query1	1300	898	882	882
query2	6407	2060	2085	2060
query3	10800	3913	3926	3913
query4	63219	28645	23576	23576
query5	5015	467	457	457
query6	402	166	163	163
query7	5452	308	298	298
query8	315	224	218	218
query9	8361	2655	2658	2655
query10	449	282	275	275
query11	17008	15086	15872	15086
query12	161	101	109	101
query13	1504	436	429	429
query14	10859	7398	7701	7398
query15	202	179	176	176
query16	6693	486	414	414
query17	1192	597	605	597
query18	1476	312	322	312
query19	213	163	149	149
query20	125	111	114	111
query21	220	101	105	101
query22	4779	4723	4890	4723
query23	34741	33848	33829	33829
query24	5965	2905	2869	2869
query25	503	394	413	394
query26	656	155	157	155
query27	1666	287	287	287
query28	4199	2466	2453	2453
query29	679	440	439	439
query30	228	155	162	155
query31	993	758	807	758
query32	70	53	54	53
query33	437	294	309	294
query34	917	490	497	490
query35	863	733	729	729
query36	1044	907	969	907
query37	149	91	86	86
query38	3984	3889	3928	3889
query39	1455	1476	1413	1413
query40	208	94	94	94
query41	48	46	47	46
query42	110	95	97	95
query43	540	496	478	478
query44	1151	827	827	827
query45	196	163	169	163
query46	1131	774	756	756
query47	1935	1865	1823	1823
query48	479	367	366	366
query49	702	404	404	404
query50	828	419	420	419
query51	7010	6963	6919	6919
query52	97	85	83	83
query53	242	174	175	174
query54	554	439	447	439
query55	75	74	76	74
query56	271	253	246	246
query57	1219	1114	1077	1077
query58	223	223	229	223
query59	3125	2965	2935	2935
query60	282	261	272	261
query61	114	104	107	104
query62	751	665	673	665
query63	217	174	183	174
query64	1383	643	650	643
query65	3212	3148	3193	3148
query66	683	285	302	285
query67	16149	15670	15503	15503
query68	1283	567	551	551
query69	456	280	280	280
query70	1220	1023	1076	1023
query71	329	258	262	258
query72	6215	3934	3890	3890
query73	766	326	327	326
query74	9231	8884	8991	8884
query75	3331	2682	2607	2607
query76	1478	839	934	839
query77	462	290	291	290
query78	10215	9544	9353	9353
query79	1057	882	872	872
query80	805	563	566	563
query81	503	253	258	253
query82	687	234	232	232
query83	158	168	156	156
query84	285	103	100	100
query85	744	371	369	369
query86	319	311	318	311
query87	4446	4509	4393	4393
query88	4515	4034	4039	4034
query89	384	356	360	356
query90	1767	308	301	301
query91	163	165	157	157
query92	79	69	72	69
query93	950	895	900	895
query94	704	383	360	360
query95	432	404	403	403
query96	484	482	486	482
query97	3166	3118	3116	3116
query98	224	229	222	222
query99	1465	1293	1292	1292
Total cold run time: 305959 ms
Total hot run time: 198955 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4e981488348c2c78d1d6bddf383175d0d28a6bfb, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.02
query3	0.22	0.06	0.06
query4	1.65	0.10	0.10
query5	0.53	0.49	0.54
query6	1.13	0.74	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.58	0.52	0.49
query10	0.54	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	2.98	3.09	2.98
query15	0.88	0.83	0.81
query16	0.39	0.40	0.37
query17	0.98	1.05	0.94
query18	0.19	0.20	0.19
query19	1.95	1.77	2.02
query20	0.01	0.01	0.01
query21	15.37	0.61	0.58
query22	2.71	2.37	1.78
query23	17.31	0.89	0.80
query24	2.49	0.89	0.86
query25	0.17	0.11	0.10
query26	0.45	0.14	0.12
query27	0.04	0.04	0.03
query28	11.40	1.08	1.07
query29	12.56	3.22	3.20
query30	0.24	0.05	0.05
query31	2.88	0.37	0.38
query32	3.28	0.45	0.45
query33	3.00	2.95	3.08
query34	17.04	4.38	4.43
query35	4.46	4.38	4.46
query36	0.67	0.48	0.48
query37	0.08	0.07	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 107.6 s
Total hot run time: 32.27 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.32% (9582/25675)
Line Coverage: 28.71% (79219/275974)
Region Coverage: 28.17% (41001/145541)
Branch Coverage: 24.79% (20894/84274)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4e981488348c2c78d1d6bddf383175d0d28a6bfb_4e981488348c2c78d1d6bddf383175d0d28a6bfb/report/index.html

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dataroaring pushed a commit that referenced this pull request Sep 20, 2024
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.16-merged dev/2.1.7-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants