[fix](iceberg) fix error when query iceberg table using date cast predicate #32194

Yulei-Yang · 2024-03-13T11:29:26Z

Proposed changes

Issue Number: close #xxx

when runing tpch query Q3 Q5 Q10 Q11 Q18 Q19, an error occurs. Error msg: ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: Invalid value for conversion to type string: xxxx (java.lang.Long)
these query has same type predicate in where clause like this: o_orderdate >= date '1994-01-01'

error log:

java.lang.RuntimeException: Invalid value for conversion to type string: 809996400000000 (java.lang.Long)
        at org.apache.doris.common.security.authentication.HadoopUGI.ugiDoAs(HadoopUGI.java:114) ~[fe-common-1.2-SNAPSHOT.j
ar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.hive.HiveMetaStoreClientHelper.ugiDoAs(HiveMetaStoreClientHelper.java:829) ~[doris-f
e.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.iceberg.source.IcebergScanNode.getSplits(IcebergScanNode.java:179) ~[doris-fe.jar:1.
2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.createScanRangeLocations(FileQueryScanNode.java:271) ~[doris-fe.ja
r:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.doFinalize(FileQueryScanNode.java:226) ~[doris-fe.jar:1.2-SNAPSHOT
]
        at org.apache.doris.datasource.FileQueryScanNode.finalize(FileQueryScanNode.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.PlanNode.finalize(PlanNode.java:687) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.AggregationNode.finalize(AggregationNode.java:381) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.OriginalPlanner.createPlanFragments(OriginalPlanner.java:206) ~[doris-fe.jar:1.2-SNAPSH
OT]
        at org.apache.doris.planner.OriginalPlanner.plan(OriginalPlanner.java:100) ~[doris-fe.jar:1.2-SNAPSHOT]

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-03-13T11:29:30Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Yulei-Yang · 2024-03-13T11:31:58Z

run buildall

doris-robot · 2024-03-13T12:06:44Z

TPC-H: Total hot run time: 37007 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3a8e09f383400ab5d97e43761470a5467bf62a2b, data reload: false

------ Round 1 ----------------------------------
q1	17892	4246	4134	4134
q2	2016	149	143	143
q3	10579	908	907	907
q4	4648	708	715	708
q5	7726	2831	2799	2799
q6	194	127	131	127
q7	1188	837	820	820
q8	9486	2063	2041	2041
q9	7935	6533	6611	6533
q10	8456	2709	2614	2614
q11	433	226	232	226
q12	805	325	298	298
q13	19813	3050	3036	3036
q14	283	249	251	249
q15	501	475	489	475
q16	2003	406	434	406
q17	1346	593	558	558
q18	6651	5836	5726	5726
q19	1567	1427	1445	1427
q20	548	283	282	282
q21	7400	3193	3256	3193
q22	806	308	305	305
Total cold run time: 112276 ms
Total hot run time: 37007 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4094	4108	4087	4087
q2	319	219	224	219
q3	2952	2893	2795	2795
q4	1815	1489	1553	1489
q5	5193	5238	5251	5238
q6	203	116	122	116
q7	2229	1850	1838	1838
q8	3141	3271	3269	3269
q9	8482	8525	8518	8518
q10	6117	3651	3655	3651
q11	548	435	438	435
q12	709	547	551	547
q13	13778	2866	2858	2858
q14	275	232	260	232
q15	480	451	447	447
q16	469	414	408	408
q17	1704	1458	1433	1433
q18	7527	7272	7018	7018
q19	5954	1573	1524	1524
q20	1944	1708	1703	1703
q21	4809	4602	4688	4602
q22	540	494	462	462
Total cold run time: 73282 ms
Total hot run time: 52889 ms

doris-robot · 2024-03-13T12:18:56Z

TPC-DS: Total hot run time: 186306 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3a8e09f383400ab5d97e43761470a5467bf62a2b, data reload: false

query1	907	372	347	347
query2	7359	2054	2005	2005
query3	6697	215	206	206
query4	31162	20610	20619	20610
query5	4451	401	418	401
query6	264	207	170	170
query7	4618	298	289	289
query8	228	178	160	160
query9	9955	2312	2264	2264
query10	458	242	232	232
query11	14607	14236	14173	14173
query12	134	83	88	83
query13	1630	417	447	417
query14	11961	10466	10442	10442
query15	255	204	190	190
query16	8079	261	257	257
query17	1946	529	528	528
query18	2091	296	263	263
query19	251	151	153	151
query20	90	82	84	82
query21	191	127	124	124
query22	4590	4402	4429	4402
query23	31982	30754	31066	30754
query24	10642	2833	2851	2833
query25	621	368	381	368
query26	1226	155	156	155
query27	2727	364	355	355
query28	7776	1901	1847	1847
query29	856	641	618	618
query30	304	149	153	149
query31	973	720	717	717
query32	98	58	55	55
query33	776	355	259	259
query34	978	458	462	458
query35	834	629	597	597
query36	1006	865	873	865
query37	128	77	76	76
query38	3345	3154	3100	3100
query39	1409	1361	1376	1361
query40	212	113	117	113
query41	50	48	47	47
query42	107	93	95	93
query43	478	442	465	442
query44	1341	734	710	710
query45	268	253	258	253
query46	1061	693	683	683
query47	1658	1586	1599	1586
query48	450	361	345	345
query49	1128	335	335	335
query50	753	379	367	367
query51	6679	6584	6504	6504
query52	97	89	87	87
query53	340	278	278	278
query54	314	232	237	232
query55	79	74	83	74
query56	236	246	234	234
query57	1094	1002	1003	1002
query58	243	202	201	201
query59	2785	2630	2584	2584
query60	264	265	273	265
query61	114	107	109	107
query62	587	403	388	388
query63	293	271	282	271
query64	5708	3880	4015	3880
query65	3038	2982	2986	2982
query66	844	340	331	331
query67	15095	14328	14598	14328
query68	9237	526	542	526
query69	634	390	385	385
query70	1211	1155	1166	1155
query71	583	279	274	274
query72	6785	2636	2442	2442
query73	1302	317	313	313
query74	8522	6594	6742	6594
query75	9995	7961	8046	7961
query76	5347	920	941	920
query77	764	260	250	250
query78	10364	9597	9596	9596
query79	12614	545	531	531
query80	1570	387	368	368
query81	508	207	207	207
query82	606	193	195	193
query83	200	143	137	137
query84	286	75	77	75
query85	1333	319	305	305
query86	392	299	311	299
query87	3434	3250	3224	3224
query88	5246	2270	2309	2270
query89	511	378	366	366
query90	2020	173	178	173
query91	188	144	129	129
query92	56	46	47	46
query93	7144	511	497	497
query94	1290	174	170	170
query95	427	336	334	334
query96	582	275	268	268
query97	4391	4290	4258	4258
query98	236	212	203	203
query99	1031	783	752	752
Total cold run time: 321730 ms
Total hot run time: 186306 ms

doris-robot · 2024-03-13T12:24:14Z

ClickBench: Total hot run time: 29.59 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3a8e09f383400ab5d97e43761470a5467bf62a2b, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.02
query3	0.23	0.06	0.06
query4	1.68	0.09	0.10
query5	0.50	0.53	0.48
query6	1.24	0.63	0.63
query7	0.03	0.02	0.01
query8	0.04	0.02	0.02
query9	0.56	0.49	0.49
query10	0.54	0.52	0.53
query11	0.12	0.08	0.09
query12	0.11	0.10	0.09
query13	0.56	0.57	0.56
query14	0.76	0.73	0.74
query15	0.80	0.78	0.77
query16	0.39	0.40	0.37
query17	1.07	1.05	0.93
query18	0.17	0.30	0.20
query19	1.73	1.68	1.73
query20	0.01	0.00	0.00
query21	15.41	0.57	0.58
query22	2.50	3.21	1.53
query23	17.22	0.72	0.90
query24	3.47	1.39	0.73
query25	0.27	0.06	0.07
query26	0.62	0.12	0.11
query27	0.04	0.03	0.03
query28	10.74	0.85	0.85
query29	12.61	3.27	3.20
query30	0.68	0.56	0.59
query31	2.83	0.34	0.34
query32	3.41	0.45	0.43
query33	2.78	2.80	2.84
query34	16.13	4.31	4.36
query35	4.30	4.32	4.31
query36	1.07	1.02	1.01
query37	0.07	0.05	0.05
query38	0.04	0.02	0.03
query39	0.02	0.01	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.02	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 105.14 s
Total hot run time: 29.59 s

doris-robot · 2024-03-13T12:27:17Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3a8e09f383400ab5d97e43761470a5467bf62a2b with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s

morningman

LGTM
I will add a test for this case later

github-actions · 2024-03-13T13:36:46Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-13T13:36:48Z

PR approved by anyone and no changes requested.

#32194)

… to the target column (#32923) Convert the datetime type in the predicate according to the target column. And add a testcase for #32194 related #30478 #30162

… to the target column (apache#32923) Convert the datetime type in the predicate according to the target column. And add a testcase for apache#32194 related apache#30478 apache#30162

[fix](iceberg) fix error when query iceberg table using date predicate

3a8e09f

Yulei-Yang changed the title ~~[fix](iceberg) fix error when query iceberg table using date predicate~~ [fix](iceberg) fix error when query iceberg table using date cast predicate Mar 13, 2024

morningman approved these changes Mar 13, 2024

View reviewed changes

morningman added the dev/2.0.x label Mar 13, 2024

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2024

github-actions bot added the reviewed label Mar 13, 2024

yiguolei approved these changes Mar 14, 2024

View reviewed changes

Yulei-Yang mentioned this pull request Mar 14, 2024

[fix](iceberg) fix error when query iceberg table using date cast predicate #32211

Merged

yiguolei merged commit 72a133c into apache:master Mar 14, 2024
27 of 31 checks passed

wm1581066 added dev/2.0.7-merged and removed dev/2.0.x labels Mar 14, 2024

yiguolei pushed a commit that referenced this pull request Mar 15, 2024

[fix](iceberg) fix error when query iceberg table using date predicate (

c85af1b

#32194)

yiguolei mentioned this pull request Mar 24, 2024

Release notes 2.1.1 #32736

Open

wuwenchi mentioned this pull request Mar 27, 2024

[bugfix](iceberg)Convert the datetime type in the predicate according to the target column #32923

Merged

Yulei-Yang deleted the fix_iceberg_date_predicate branch May 20, 2024 09:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](iceberg) fix error when query iceberg table using date cast predicate #32194

[fix](iceberg) fix error when query iceberg table using date cast predicate #32194

Yulei-Yang commented Mar 13, 2024

doris-robot commented Mar 13, 2024

Yulei-Yang commented Mar 13, 2024

doris-robot commented Mar 13, 2024

doris-robot commented Mar 13, 2024

doris-robot commented Mar 13, 2024

doris-robot commented Mar 13, 2024

morningman left a comment

github-actions bot commented Mar 13, 2024

github-actions bot commented Mar 13, 2024

[fix](iceberg) fix error when query iceberg table using date cast predicate #32194

[fix](iceberg) fix error when query iceberg table using date cast predicate #32194

Conversation

Yulei-Yang commented Mar 13, 2024

Proposed changes

Further comments

doris-robot commented Mar 13, 2024

Yulei-Yang commented Mar 13, 2024

doris-robot commented Mar 13, 2024

doris-robot commented Mar 13, 2024

doris-robot commented Mar 13, 2024

doris-robot commented Mar 13, 2024

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 13, 2024

github-actions bot commented Mar 13, 2024