Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](mow) opt mow lookup with sequence column #38287

Merged
merged 2 commits into from
Jul 26, 2024

Conversation

cambyzju
Copy link
Contributor

Compare primary key without sequence column is expensive,
if full key is exact_match, the primary key without sequence column should also the same,
do not need to compare again.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@cambyzju
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40013 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 960440581ba8ca152d26171063c85f278202d347, data reload: false

------ Round 1 ----------------------------------
q1	17673	4601	4266	4266
q2	2025	194	189	189
q3	10470	1220	1142	1142
q4	10191	819	774	774
q5	7661	2730	2598	2598
q6	218	137	140	137
q7	955	612	601	601
q8	9235	2085	2087	2085
q9	8763	6545	6565	6545
q10	8588	3780	3757	3757
q11	459	247	251	247
q12	403	228	229	228
q13	18676	2999	3008	2999
q14	274	242	239	239
q15	534	482	493	482
q16	489	393	378	378
q17	972	686	710	686
q18	8049	7538	7378	7378
q19	8574	1472	1496	1472
q20	706	334	338	334
q21	4934	3180	3247	3180
q22	352	296	296	296
Total cold run time: 120201 ms
Total hot run time: 40013 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4506	4287	4248	4248
q2	379	267	268	267
q3	3013	2701	2717	2701
q4	1883	1570	1586	1570
q5	5309	5297	5321	5297
q6	225	130	132	130
q7	2172	1738	1705	1705
q8	3211	3363	3306	3306
q9	8378	8402	8334	8334
q10	3891	3728	3727	3727
q11	582	502	492	492
q12	767	570	619	570
q13	16667	2972	3012	2972
q14	300	279	275	275
q15	527	478	469	469
q16	473	421	406	406
q17	1787	1482	1486	1482
q18	7706	7419	7381	7381
q19	2073	1537	1502	1502
q20	2003	1754	1780	1754
q21	4884	4779	4760	4760
q22	606	505	499	499
Total cold run time: 71342 ms
Total hot run time: 53847 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173771 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 960440581ba8ca152d26171063c85f278202d347, data reload: false

query1	918	366	364	364
query2	6458	1904	1827	1827
query3	6651	206	219	206
query4	26087	17326	17502	17326
query5	4200	498	499	498
query6	268	173	168	168
query7	4634	295	294	294
query8	252	203	196	196
query9	8567	2446	2409	2409
query10	431	310	282	282
query11	12630	10074	10097	10074
query12	142	89	83	83
query13	1637	377	372	372
query14	10216	7961	7781	7781
query15	239	166	168	166
query16	7924	486	505	486
query17	1591	568	560	560
query18	2002	297	281	281
query19	209	157	164	157
query20	94	86	84	84
query21	208	135	130	130
query22	4329	4062	3941	3941
query23	33811	33151	33281	33151
query24	11188	2945	2846	2846
query25	648	406	398	398
query26	1538	155	153	153
query27	2901	278	282	278
query28	7672	2021	2015	2015
query29	1008	670	634	634
query30	291	153	150	150
query31	984	752	761	752
query32	104	54	59	54
query33	788	336	354	336
query34	911	495	486	486
query35	882	740	779	740
query36	1138	926	947	926
query37	164	83	83	83
query38	2846	2802	2742	2742
query39	907	820	846	820
query40	284	130	127	127
query41	49	46	51	46
query42	129	112	100	100
query43	507	474	476	474
query44	1254	742	740	740
query45	194	165	165	165
query46	1099	753	753	753
query47	1846	1773	1744	1744
query48	377	301	302	301
query49	1200	439	439	439
query50	813	407	413	407
query51	6724	6761	6710	6710
query52	105	96	98	96
query53	364	297	296	296
query54	1037	566	445	445
query55	76	76	73	73
query56	309	268	266	266
query57	1150	1024	1063	1024
query58	257	267	257	257
query59	2751	2473	2605	2473
query60	317	277	281	277
query61	95	93	98	93
query62	858	656	657	656
query63	330	293	289	289
query64	10532	2245	4143	2245
query65	3177	3111	3132	3111
query66	1408	328	330	328
query67	15543	14962	14944	14944
query68	8518	573	569	569
query69	722	441	381	381
query70	1245	1160	1150	1150
query71	533	278	271	271
query72	8777	5156	5276	5156
query73	1538	343	328	328
query74	6088	5632	5572	5572
query75	4921	2671	2749	2671
query76	5036	952	957	952
query77	750	308	315	308
query78	9639	9226	8925	8925
query79	8063	543	544	543
query80	1028	498	502	498
query81	583	228	220	220
query82	775	138	140	138
query83	306	170	165	165
query84	278	89	87	87
query85	1371	327	357	327
query86	404	317	281	281
query87	3332	3123	3085	3085
query88	4681	2370	2372	2370
query89	539	391	389	389
query90	1992	201	199	199
query91	131	110	102	102
query92	61	53	56	53
query93	6291	538	528	528
query94	1230	296	299	296
query95	412	329	333	329
query96	655	270	271	270
query97	3187	3010	3047	3010
query98	216	203	201	201
query99	1601	1274	1269	1269
Total cold run time: 305156 ms
Total hot run time: 173771 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.74 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 960440581ba8ca152d26171063c85f278202d347, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.69	0.08	0.08
query5	0.49	0.49	0.48
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.56	0.49	0.50
query10	0.55	0.53	0.54
query11	0.15	0.11	0.11
query12	0.16	0.12	0.12
query13	0.60	0.60	0.59
query14	0.76	0.77	0.78
query15	0.85	0.81	0.81
query16	0.36	0.36	0.35
query17	0.98	1.00	1.06
query18	0.23	0.23	0.22
query19	1.82	1.70	1.70
query20	0.02	0.01	0.01
query21	15.43	0.77	0.64
query22	4.30	7.08	2.01
query23	18.29	1.33	1.31
query24	2.11	0.24	0.21
query25	0.14	0.10	0.09
query26	0.31	0.22	0.21
query27	0.46	0.23	0.23
query28	13.30	1.02	1.00
query29	12.66	3.32	3.36
query30	0.25	0.07	0.05
query31	2.87	0.40	0.40
query32	3.26	0.48	0.47
query33	2.94	2.97	2.89
query34	17.21	4.29	4.34
query35	4.42	4.52	4.39
query36	0.65	0.49	0.48
query37	0.20	0.15	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.05
Total cold run time: 110.14 s
Total hot run time: 30.74 s

Copy link
Contributor

PR approved by anyone and no changes requested.

@cambyzju
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39581 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dd2181179c11b02c94dcd9ca6bc8a444b970a90b, data reload: false

------ Round 1 ----------------------------------
q1	17633	4374	4231	4231
q2	2001	187	183	183
q3	10450	1233	1079	1079
q4	10183	775	770	770
q5	7565	2695	2690	2690
q6	220	137	135	135
q7	959	588	589	588
q8	9224	2072	2068	2068
q9	8604	6537	6534	6534
q10	8874	3744	3816	3744
q11	461	233	235	233
q12	417	219	218	218
q13	19025	2970	2959	2959
q14	282	239	240	239
q15	548	483	480	480
q16	502	378	380	378
q17	955	668	723	668
q18	7943	7390	7271	7271
q19	5597	1437	1384	1384
q20	695	320	324	320
q21	4924	3203	3122	3122
q22	348	288	287	287
Total cold run time: 117410 ms
Total hot run time: 39581 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4357	4263	4224	4224
q2	378	280	279	279
q3	3015	2855	2915	2855
q4	2023	1707	1701	1701
q5	5610	5526	5528	5526
q6	230	132	127	127
q7	2202	1884	1849	1849
q8	3297	3442	3423	3423
q9	8786	8724	8900	8724
q10	4137	3808	3804	3804
q11	610	490	524	490
q12	816	641	634	634
q13	17064	3144	3246	3144
q14	307	275	279	275
q15	536	490	485	485
q16	496	434	434	434
q17	1858	1544	1507	1507
q18	7997	7814	7916	7814
q19	1767	1645	1423	1423
q20	2267	1897	1876	1876
q21	5180	4903	4761	4761
q22	600	488	489	488
Total cold run time: 73533 ms
Total hot run time: 55843 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173565 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd2181179c11b02c94dcd9ca6bc8a444b970a90b, data reload: false

query1	906	373	377	373
query2	6438	1896	1978	1896
query3	6655	203	213	203
query4	28360	17275	17343	17275
query5	3586	476	471	471
query6	255	169	160	160
query7	4580	287	277	277
query8	241	184	183	183
query9	8435	2406	2376	2376
query10	440	278	284	278
query11	10578	10051	10069	10051
query12	119	87	90	87
query13	1640	374	363	363
query14	10416	7812	7578	7578
query15	227	167	172	167
query16	7641	459	464	459
query17	1581	567	547	547
query18	1647	288	294	288
query19	211	153	157	153
query20	89	85	84	84
query21	212	132	136	132
query22	4375	4088	3975	3975
query23	34190	33748	33735	33735
query24	11116	2938	2883	2883
query25	611	391	396	391
query26	850	154	154	154
query27	2282	270	280	270
query28	7040	2076	2068	2068
query29	905	671	618	618
query30	254	153	158	153
query31	992	762	756	756
query32	93	55	55	55
query33	749	345	321	321
query34	895	494	520	494
query35	889	807	747	747
query36	1164	998	967	967
query37	143	81	84	81
query38	2912	2919	2805	2805
query39	922	867	821	821
query40	205	117	117	117
query41	47	43	43	43
query42	117	94	94	94
query43	521	473	466	466
query44	1188	719	720	719
query45	207	175	178	175
query46	1087	712	752	712
query47	1855	1800	1778	1778
query48	368	289	297	289
query49	830	410	422	410
query50	783	389	392	389
query51	6760	6567	6653	6567
query52	108	91	93	91
query53	355	289	279	279
query54	890	439	438	438
query55	72	75	74	74
query56	290	273	266	266
query57	1105	1048	1042	1042
query58	248	242	270	242
query59	2911	2774	2680	2680
query60	305	280	272	272
query61	97	93	94	93
query62	798	640	644	640
query63	321	286	291	286
query64	9418	2201	1661	1661
query65	3210	3116	3094	3094
query66	748	333	336	333
query67	15506	15143	14954	14954
query68	5649	549	559	549
query69	716	454	359	359
query70	1219	1122	1168	1122
query71	488	274	324	274
query72	8307	5467	5326	5326
query73	777	325	323	323
query74	6320	5640	5683	5640
query75	3939	2681	2702	2681
query76	3714	953	996	953
query77	618	314	305	305
query78	11097	8981	9698	8981
query79	7702	518	520	518
query80	1011	482	476	476
query81	590	219	216	216
query82	771	145	132	132
query83	276	160	163	160
query84	278	89	86	86
query85	1240	316	305	305
query86	314	278	316	278
query87	3318	3125	3084	3084
query88	4099	2371	2388	2371
query89	500	395	382	382
query90	1558	195	191	191
query91	134	100	99	99
query92	55	49	52	49
query93	5307	515	522	515
query94	615	290	369	290
query95	405	313	314	313
query96	611	276	272	272
query97	3195	3020	2989	2989
query98	228	192	194	192
query99	1509	1282	1235	1235
Total cold run time: 291549 ms
Total hot run time: 173565 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.46 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dd2181179c11b02c94dcd9ca6bc8a444b970a90b, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.23	0.04	0.05
query4	1.68	0.07	0.07
query5	0.49	0.48	0.47
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.49	0.48
query10	0.55	0.53	0.55
query11	0.14	0.12	0.11
query12	0.15	0.13	0.12
query13	0.60	0.59	0.58
query14	0.76	0.79	0.79
query15	0.86	0.82	0.82
query16	0.37	0.35	0.37
query17	0.99	1.04	0.97
query18	0.24	0.22	0.24
query19	1.76	1.70	1.70
query20	0.01	0.01	0.00
query21	15.39	0.78	0.67
query22	4.54	7.63	1.73
query23	18.30	1.40	1.34
query24	2.07	0.25	0.23
query25	0.17	0.09	0.08
query26	0.31	0.22	0.21
query27	0.46	0.24	0.24
query28	13.25	1.02	0.99
query29	12.61	3.29	3.27
query30	0.25	0.05	0.06
query31	2.87	0.39	0.38
query32	3.30	0.48	0.47
query33	2.93	2.92	2.92
query34	17.12	4.33	4.40
query35	4.41	4.48	4.45
query36	0.65	0.45	0.47
query37	0.20	0.15	0.14
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.04	0.04
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.07 s
Total hot run time: 30.46 s

@bobhan1
Copy link
Contributor

bobhan1 commented Jul 25, 2024

run external

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhannngchen zhannngchen merged commit 9a13422 into apache:master Jul 26, 2024
26 of 28 checks passed
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 26, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

cambyzju added a commit to cambyzju/incubator-doris that referenced this pull request Jul 26, 2024
Compare primary key without sequence column is expensive,
if full key is exact_match, the primary key without sequence column
should also the same,
do not need to compare again.
cambyzju added a commit to cambyzju/incubator-doris that referenced this pull request Jul 26, 2024
Compare primary key without sequence column is expensive,
if full key is exact_match, the primary key without sequence column
should also the same,
do not need to compare again.
@xiaokang xiaokang mentioned this pull request Aug 3, 2024
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
Compare primary key without sequence column is expensive,
if full key is exact_match, the primary key without sequence column
should also the same,
do not need to compare again.
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
Compare primary key without sequence column is expensive,
if full key is exact_match, the primary key without sequence column
should also the same,
do not need to compare again.
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants