Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bugfix](iceberg)Convert the datetime type in the predicate according to the target column #32923

Merged
merged 4 commits into from
Mar 30, 2024
Merged

[bugfix](iceberg)Convert the datetime type in the predicate according to the target column #32923

merged 4 commits into from
Mar 30, 2024

Conversation

wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Mar 27, 2024 •
edited
Loading

Proposed changes

Convert the datetime type in the predicate according to the target column.
And add a testcase for #32194
related #30478 #30162

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37713 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 56857ffd8e8c5b714713c48f18a220a2b86ef594, data reload: false

------ Round 1 ----------------------------------
q1	17651	4115	4106	4106
q2	2104	150	149	149
q3	10601	1106	1177	1106
q4	10223	717	738	717
q5	7451	3004	2989	2989
q6	203	126	128	126
q7	1021	593	577	577
q8	9344	2004	1979	1979
q9	7251	6598	6592	6592
q10	8416	3416	3534	3416
q11	432	234	218	218
q12	422	194	193	193
q13	17788	2826	2838	2826
q14	239	215	204	204
q15	512	465	462	462
q16	497	381	376	376
q17	944	518	625	518
q18	7103	6503	6356	6356
q19	4009	1482	1372	1372
q20	528	268	255	255
q21	3553	2956	2869	2869
q22	357	309	307	307
Total cold run time: 110649 ms
Total hot run time: 37713 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4139	4069	4038	4038
q2	325	228	224	224
q3	2939	2811	2859	2811
q4	1790	1563	1526	1526
q5	5284	5308	5319	5308
q6	201	117	119	117
q7	2233	1832	1827	1827
q8	3156	3352	3274	3274
q9	8749	8707	8682	8682
q10	3793	3757	3738	3738
q11	547	431	439	431
q12	703	582	557	557
q13	16935	2844	2841	2841
q14	291	253	250	250
q15	502	473	474	473
q16	479	418	423	418
q17	1719	1509	1477	1477
q18	7409	7043	6946	6946
q19	1603	1527	1583	1527
q20	1912	1716	1685	1685
q21	4717	4746	4665	4665
q22	548	470	460	460
Total cold run time: 69974 ms
Total hot run time: 53275 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181539 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 56857ffd8e8c5b714713c48f18a220a2b86ef594, data reload: false

query1	964	370	354	354
query2	6550	1941	1863	1863
query3	6707	212	215	212
query4	31844	21273	21356	21273
query5	4279	398	395	395
query6	266	185	180	180
query7	4626	295	284	284
query8	239	170	172	170
query9	9424	2340	2306	2306
query10	565	250	279	250
query11	15283	14255	14242	14242
query12	135	94	87	87
query13	1626	421	416	416
query14	10661	8065	7446	7446
query15	259	209	200	200
query16	8034	255	251	251
query17	2120	559	545	545
query18	1389	301	274	274
query19	348	151	154	151
query20	94	82	94	82
query21	204	126	126	126
query22	4941	4848	4769	4769
query23	33414	32842	32715	32715
query24	10973	2918	2839	2839
query25	638	389	394	389
query26	1228	159	161	159
query27	2849	355	359	355
query28	7434	1887	1868	1868
query29	906	660	627	627
query30	304	149	151	149
query31	996	728	756	728
query32	104	56	56	56
query33	771	252	252	252
query34	1030	491	498	491
query35	845	645	618	618
query36	1055	921	906	906
query37	138	66	69	66
query38	3560	3478	3437	3437
query39	1472	1463	1432	1432
query40	215	119	114	114
query41	51	48	47	47
query42	100	99	97	97
query43	497	451	450	450
query44	1156	730	748	730
query45	277	257	272	257
query46	1126	698	716	698
query47	1931	1852	1851	1851
query48	455	368	356	356
query49	1115	343	362	343
query50	765	378	388	378
query51	6699	6623	6758	6623
query52	111	87	94	87
query53	348	276	279	276
query54	297	252	243	243
query55	89	79	82	79
query56	247	234	228	228
query57	1206	1152	1158	1152
query58	233	207	217	207
query59	2820	2558	2637	2558
query60	277	241	255	241
query61	115	115	128	115
query62	643	477	448	448
query63	302	274	273	273
query64	5757	4113	4012	4012
query65	3221	3048	3028	3028
query66	971	383	381	381
query67	15374	14796	15175	14796
query68	8994	518	530	518
query69	630	388	392	388
query70	1324	1144	1196	1144
query71	529	262	270	262
query72	6424	2731	2529	2529
query73	1604	318	311	311
query74	7971	6484	6424	6424
query75	3728	2222	2203	2203
query76	5042	937	945	937
query77	652	262	255	255
query78	11057	10229	10157	10157
query79	8779	537	528	528
query80	1796	372	366	366
query81	536	213	221	213
query82	492	86	89	86
query83	212	149	149	149
query84	291	83	79	79
query85	1241	319	311	311
query86	397	293	272	272
query87	3803	3595	3571	3571
query88	4692	2291	2278	2278
query89	482	360	362	360
query90	1945	175	177	175
query91	194	137	136	136
query92	63	49	48	48
query93	6761	496	478	478
query94	1166	174	173	173
query95	444	345	327	327
query96	618	262	266	262
query97	2613	2500	2465	2465
query98	223	222	206	206
query99	1167	901	897	897
Total cold run time: 310543 ms
Total hot run time: 181539 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 56857ffd8e8c5b714713c48f18a220a2b86ef594 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s

@wuwenchi wuwenchi marked this pull request as draft March 27, 2024 13:53
@wuwenchi wuwenchi marked this pull request as ready for review March 28, 2024 13:24
@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37546 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dac72a8256c7740ed4cf790d9eaa0d7370645403, data reload: false

------ Round 1 ----------------------------------
q1	17613	4273	4152	4152
q2	2099	153	151	151
q3	10597	1164	1171	1164
q4	10235	799	759	759
q5	7443	2904	2926	2904
q6	203	125	124	124
q7	1033	578	570	570
q8	9352	1984	1974	1974
q9	6932	6346	6324	6324
q10	8454	3440	3563	3440
q11	436	225	222	222
q12	433	206	198	198
q13	17784	2866	2855	2855
q14	255	205	208	205
q15	508	458	456	456
q16	484	371	370	370
q17	941	601	520	520
q18	7112	6504	6351	6351
q19	2641	1423	1426	1423
q20	543	251	245	245
q21	3570	2860	2882	2860
q22	335	279	287	279
Total cold run time: 109003 ms
Total hot run time: 37546 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4156	4090	4093	4090
q2	326	235	229	229
q3	2945	2832	2766	2766
q4	1830	1556	1555	1555
q5	5162	5225	5196	5196
q6	195	114	120	114
q7	2182	1820	1832	1820
q8	3156	3276	3291	3276
q9	8428	8397	8420	8397
q10	3823	3748	3966	3748
q11	561	465	454	454
q12	754	583	567	567
q13	17046	3058	3024	3024
q14	288	285	266	266
q15	541	480	477	477
q16	497	444	474	444
q17	1784	1537	1515	1515
q18	8011	7552	7453	7453
q19	1944	1587	1566	1566
q20	1996	1900	1830	1830
q21	4843	4847	4896	4847
q22	544	465	461	461
Total cold run time: 71012 ms
Total hot run time: 54095 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182590 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dac72a8256c7740ed4cf790d9eaa0d7370645403, data reload: false

query1	931	348	348	348
query2	6354	2096	2075	2075
query3	6712	203	203	203
query4	31732	21273	21436	21273
query5	4326	412	400	400
query6	281	178	174	174
query7	4618	300	301	300
query8	223	167	167	167
query9	9541	2296	2281	2281
query10	416	253	288	253
query11	15303	14254	14183	14183
query12	128	91	87	87
query13	1620	423	439	423
query14	9990	8046	7918	7918
query15	256	187	184	184
query16	8264	261	261	261
query17	1966	598	554	554
query18	2098	286	292	286
query19	350	165	158	158
query20	93	89	87	87
query21	205	139	135	135
query22	5028	4848	4872	4848
query23	33443	32767	33350	32767
query24	10938	2995	2987	2987
query25	616	424	406	406
query26	1089	163	167	163
query27	2532	364	377	364
query28	7280	2020	2012	2012
query29	901	680	650	650
query30	325	156	153	153
query31	1009	790	748	748
query32	100	58	57	57
query33	803	275	260	260
query34	1195	513	515	513
query35	923	728	724	724
query36	1058	905	918	905
query37	138	70	69	69
query38	3698	3674	3640	3640
query39	1087	1070	1073	1070
query40	176	108	113	108
query41	49	47	47	47
query42	113	100	100	100
query43	483	472	473	472
query44	1151	762	773	762
query45	281	270	248	248
query46	1124	729	711	711
query47	2023	1943	1973	1943
query48	463	368	366	366
query49	958	345	342	342
query50	817	402	399	399
query51	6878	6887	6685	6685
query52	107	93	93	93
query53	354	278	291	278
query54	308	244	243	243
query55	86	85	79	79
query56	263	234	231	231
query57	1256	1187	1223	1187
query58	233	234	219	219
query59	2778	2585	2690	2585
query60	253	234	234	234
query61	96	93	92	92
query62	612	432	422	422
query63	306	277	284	277
query64	5316	4050	4055	4050
query65	3087	3026	3043	3026
query66	787	352	350	350
query67	15750	14835	14854	14835
query68	10098	521	522	521
query69	694	389	391	389
query70	1301	1120	1146	1120
query71	542	269	263	263
query72	7545	2521	2370	2370
query73	1701	313	318	313
query74	8064	6477	6401	6401
query75	5120	2229	2237	2229
query76	6913	961	854	854
query77	619	252	260	252
query78	11005	10256	10021	10021
query79	10932	536	524	524
query80	1797	403	403	403
query81	513	214	215	214
query82	168	90	83	83
query83	227	147	148	147
query84	286	76	72	72
query85	1135	311	302	302
query86	355	291	313	291
query87	3793	3575	3536	3536
query88	4596	2330	2309	2309
query89	500	363	366	363
query90	2041	180	177	177
query91	172	134	136	134
query92	55	45	46	45
query93	5001	498	496	496
query94	1285	175	178	175
query95	418	310	313	310
query96	614	274	266	266
query97	2632	2499	2520	2499
query98	228	216	224	216
query99	1060	844	825	825
Total cold run time: 315088 ms
Total hot run time: 182590 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit dac72a8256c7740ed4cf790d9eaa0d7370645403 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

morningman
morningman approved these changes Mar 28, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Yulei-Yang
Yulei-Yang approved these changes Mar 29, 2024
View reviewed changes
Copy link
Contributor

@Yulei-Yang Yulei-Yang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit c19c246 into apache:master Mar 30, 2024
28 of 31 checks passed
@wuwenchi wuwenchi mentioned this pull request Apr 1, 2024
Merged
yiguolei pushed a commit that referenced this pull request Apr 1, 2024
@wuwenchi @Doris-Extras
[bugfix](iceberg)Convert the datetime type in the predicate according…
32283c1 
… to the target column (#32923)

Convert the datetime type in the predicate according to the target column.
And add a testcase for #32194
related #30478 #30162
xiaokang pushed a commit that referenced this pull request Apr 2, 2024
@wuwenchi
[bugfix](iceberg)Convert the datetime type in the predicate according…
de7cabb 
… to the target column #32923 (#33092)
@morningman morningman mentioned this pull request Apr 6, 2024
Merged
morningman added a commit that referenced this pull request Apr 7, 2024
@morningman
[fix](iceberg) fix iceberg predicate conversion bug (#33283)
79a7718 
Followup #32923

Some cases are not covered in #32923
morningman pushed a commit to morningman/doris that referenced this pull request Apr 7, 2024
@wuwenchi @morningman
[bugfix](iceberg)Convert the datetime type in the predicate according…
190763e 
… to the target column (apache#32923)

Convert the datetime type in the predicate according to the target column.
And add a testcase for apache#32194
related apache#30478 apache#30162
morningman added a commit to morningman/doris that referenced this pull request Apr 7, 2024
@morningman
[fix](iceberg) fix iceberg predicate conversion bug (apache#33283)
d9d950d 
Followup apache#32923

Some cases are not covered in apache#32923
seawinde pushed a commit to seawinde/doris that referenced this pull request Apr 10, 2024
@morningman @seawinde
[fix](iceberg) fix iceberg predicate conversion bug (apache#33283)
d133408 
Followup apache#32923

Some cases are not covered in apache#32923
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@wuwenchi
[bugfix](iceberg)Convert the datetime type in the predicate according…
79532af 
… to the target column apache#32923 (apache#33092)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@Yulei-Yang Yulei-Yang Yulei-Yang approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.8-merged dev/2.1.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@wuwenchi @doris-robot @morningman @Yulei-Yang @xiaokang