[branch-2.0](datetime) refactor and support more format for str_to_date #30719

zclllyybb · 2024-02-01T12:29:57Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-02-01T12:30:01Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

zclllyybb · 2024-02-01T12:30:03Z

run buildall

github-actions · 2024-02-01T12:36:45Z

clang-tidy review says "All clean, LGTM! 👍"

zclllyybb · 2024-02-04T09:58:08Z

run buildall

github-actions · 2024-02-04T10:04:15Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-02-04T10:54:52Z

TPC-H: Total hot run time: 50377 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bcaaec9c675c886a8acb30450b018c8fc2b0d7e3, data reload: false

------ Round 1 ----------------------------------
q1	17588	5134	4343	4343
q2	2043	157	154	154
q3	10718	1942	1903	1903
q4	10235	1261	1322	1261
q5	8583	4060	3971	3971
q6	224	123	120	120
q7	2055	1604	1576	1576
q8	9329	2724	2723	2723
q9	11216	10727	10765	10727
q10	8646	3542	3537	3537
q11	429	246	243	243
q12	468	290	304	290
q13	18342	3929	4024	3929
q14	352	324	331	324
q15	516	454	459	454
q16	691	593	594	593
q17	1125	963	961	961
q18	7344	6943	6931	6931
q19	1648	1562	1515	1515
q20	532	322	309	309
q21	4460	4120	4156	4120
q22	499	411	393	393
Total cold run time: 117043 ms
Total hot run time: 50377 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4315	4311	4300	4300
q2	319	226	218	218
q3	4181	4166	4137	4137
q4	2771	2759	2767	2759
q5	7392	7270	7162	7162
q6	238	116	118	116
q7	3200	2853	2870	2853
q8	4376	4459	4494	4459
q9	17605	17399	17100	17100
q10	4252	4240	4275	4240
q11	783	678	684	678
q12	1016	860	850	850
q13	7331	3752	3767	3752
q14	443	438	418	418
q15	504	459	461	459
q16	754	727	698	698
q17	3839	3839	3898	3839
q18	8809	8737	8855	8737
q19	1704	1691	1622	1622
q20	2402	2119	2139	2119
q21	8538	8486	8509	8486
q22	1067	915	922	915
Total cold run time: 85839 ms
Total hot run time: 79917 ms

doris-robot · 2024-02-04T11:01:08Z

TeamCity be ut coverage result:
Function Coverage: 37.84% (8032/21224)
Line Coverage: 29.53% (65481/221721)
Region Coverage: 28.96% (33631/116120)
Branch Coverage: 24.82% (17260/69542)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bcaaec9c675c886a8acb30450b018c8fc2b0d7e3_bcaaec9c675c886a8acb30450b018c8fc2b0d7e3/report/index.html

doris-robot · 2024-02-04T11:09:24Z

TPC-DS: Total hot run time: 241354 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bcaaec9c675c886a8acb30450b018c8fc2b0d7e3, data reload: false

query1	910	384	384	384
query2	6527	2197	2085	2085
query3	6915	206	208	206
query4	19772	17966	17949	17949
query5	19714	6306	6261	6261
query6	291	205	224	205
query7	4144	290	299	290
query8	251	239	247	239
query9	3108	2721	2635	2635
query10	419	286	306	286
query11	11338	10750	10746	10746
query12	122	75	71	71
query13	5650	633	627	627
query14	17696	13457	13380	13380
query15	371	232	241	232
query16	6457	259	273	259
query17	3225	870	833	833
query18	2254	416	394	394
query19	200	144	142	142
query20	71	73	78	73
query21	181	106	88	88
query22	5460	5204	5254	5204
query23	32557	31874	31959	31874
query24	6990	6478	6469	6469
query25	498	432	442	432
query26	513	168	162	162
query27	1788	291	296	291
query28	6094	2252	2216	2216
query29	2959	2867	2707	2707
query30	243	163	163	163
query31	906	720	735	720
query32	67	57	55	55
query33	395	251	258	251
query34	832	475	471	471
query35	1093	962	911	911
query36	1507	1480	1473	1473
query37	88	60	59	59
query38	3140	2932	2922	2922
query39	1366	1311	1319	1311
query40	205	97	93	93
query41	35	36	31	31
query42	80	93	86	86
query43	645	543	536	536
query44	1101	712	731	712
query45	245	229	226	226
query46	1225	977	952	952
query47	1816	1705	1795	1705
query48	973	664	667	664
query49	611	366	360	360
query50	852	581	582	581
query51	5534	5510	5456	5456
query52	89	77	79	77
query53	449	322	308	308
query54	2422	2258	2250	2250
query55	78	79	76	76
query56	227	198	190	190
query57	1178	1131	1076	1076
query58	202	189	205	189
query59	3507	3147	3425	3147
query60	200	192	197	192
query61	85	81	89	81
query62	802	463	482	463
query63	463	334	330	330
query64	1957	1324	1276	1276
query65	3703	3605	3631	3605
query66	821	369	350	350
query67	16132	17137	16422	16422
query68	8284	631	686	631
query69	552	330	343	330
query70	1903	1677	1862	1677
query71	386	317	312	312
query72	4470	3341	3361	3341
query73	727	315	319	315
query74	6329	5968	5848	5848
query75	4307	3598	3567	3567
query76	4696	1182	1227	1182
query77	600	263	253	253
query78	33388	49619	50836	49619
query79	16097	628	614	614
query80	5026	373	378	373
query81	554	227	231	227
query82	1021	96	96	96
query83	418	139	136	136
query84	253	74	68	68
query85	2607	278	281	278
query86	479	384	380	380
query87	3285	3028	3014	3014
query88	7136	2360	2336	2336
query89	463	274	276	274
query90	2492	194	207	194
query91	152	121	116	116
query92	59	50	52	50
query93	6414	575	571	571
query94	1857	207	209	207
query95	1095	1062	1050	1050
query96	649	337	334	334
query97	6669	6384	6435	6384
query98	193	173	179	173
query99	3926	918	894	894
Total cold run time: 348236 ms
Total hot run time: 241354 ms

doris-robot · 2024-02-04T11:14:48Z

ClickBench: Total hot run time: 30.91 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bcaaec9c675c886a8acb30450b018c8fc2b0d7e3, data reload: false

query1	0.03	0.02	0.01
query2	0.06	0.03	0.02
query3	0.25	0.05	0.04
query4	1.84	0.07	0.06
query5	0.53	0.52	0.52
query6	1.27	0.62	0.60
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.53	0.49	0.50
query10	0.56	0.54	0.55
query11	0.12	0.09	0.09
query12	0.12	0.10	0.10
query13	0.62	0.61	0.62
query14	0.78	0.80	0.79
query15	0.78	0.76	0.75
query16	0.38	0.37	0.36
query17	1.02	1.02	1.03
query18	0.22	0.27	0.24
query19	1.95	1.77	1.78
query20	0.01	0.01	0.00
query21	15.48	0.56	0.55
query22	1.84	2.01	1.51
query23	17.25	0.95	0.98
query24	4.67	1.22	1.33
query25	1.57	0.12	0.11
query26	0.25	0.14	0.15
query27	0.10	0.11	0.11
query28	7.79	0.74	0.72
query29	12.78	2.22	2.23
query30	0.57	0.59	0.52
query31	2.81	0.37	0.40
query32	3.41	0.50	0.50
query33	3.09	3.03	3.10
query34	15.24	4.83	4.82
query35	4.89	4.85	4.84
query36	1.09	1.02	1.01
query37	0.05	0.05	0.04
query38	0.04	0.02	0.02
query39	0.01	0.01	0.01
query40	0.16	0.14	0.14
query41	0.06	0.02	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 104.32 s
Total hot run time: 30.91 s

doris-robot · 2024-02-04T11:18:00Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit bcaaec9c675c886a8acb30450b018c8fc2b0d7e3 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

zclllyybb · 2024-02-04T16:10:42Z

run buildall

github-actions · 2024-02-04T16:17:04Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-02-04T16:35:48Z

TPC-H: Total hot run time: 50183 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e7a9bd476654b49a7e125cf21e40ad9556885eb0, data reload: false

------ Round 1 ----------------------------------
q1	18141	4489	4347	4347
q2	2031	156	139	139
q3	10274	1925	1955	1925
q4	10101	1270	1338	1270
q5	8376	4020	3988	3988
q6	236	126	125	125
q7	2064	1604	1594	1594
q8	9299	2746	2733	2733
q9	10895	10480	10533	10480
q10	8637	3534	3547	3534
q11	426	252	234	234
q12	464	304	304	304
q13	18360	3994	4058	3994
q14	354	322	323	322
q15	512	465	449	449
q16	686	602	596	596
q17	1126	958	956	956
q18	7414	6821	6865	6821
q19	1693	1572	1552	1552
q20	524	316	298	298
q21	4509	4140	4128	4128
q22	506	404	394	394
Total cold run time: 116628 ms
Total hot run time: 50183 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4331	4313	4336	4313
q2	322	222	218	218
q3	4195	4165	4159	4159
q4	2770	2762	2775	2762
q5	7238	7169	7156	7156
q6	234	125	116	116
q7	3229	2853	2898	2853
q8	4361	4476	4461	4461
q9	17124	17096	16953	16953
q10	4266	4269	4301	4269
q11	760	684	682	682
q12	1046	849	853	849
q13	6571	3780	3722	3722
q14	446	420	419	419
q15	504	456	458	456
q16	752	716	707	707
q17	3944	3815	3842	3815
q18	8843	8907	8807	8807
q19	1710	1725	1668	1668
q20	2391	2128	2150	2128
q21	8517	8517	8442	8442
q22	1019	947	940	940
Total cold run time: 84573 ms
Total hot run time: 79895 ms

doris-robot · 2024-02-04T16:46:26Z

TeamCity be ut coverage result:
Function Coverage: 37.85% (8034/21225)
Line Coverage: 29.54% (65508/221763)
Region Coverage: 28.97% (33656/116161)
Branch Coverage: 24.83% (17274/69564)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e7a9bd476654b49a7e125cf21e40ad9556885eb0_e7a9bd476654b49a7e125cf21e40ad9556885eb0/report/index.html

doris-robot · 2024-02-04T16:50:15Z

TPC-DS: Total hot run time: 237177 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e7a9bd476654b49a7e125cf21e40ad9556885eb0, data reload: false

query1	931	391	380	380
query2	6511	2169	2181	2169
query3	6916	207	204	204
query4	20246	18068	18164	18068
query5	19727	6305	6223	6223
query6	271	209	229	209
query7	4147	298	296	296
query8	239	232	222	222
query9	3071	2681	2623	2623
query10	420	284	299	284
query11	11367	10720	10582	10582
query12	122	76	69	69
query13	5635	640	617	617
query14	17275	13246	13193	13193
query15	355	231	242	231
query16	6454	254	275	254
query17	3206	879	865	865
query18	2258	422	400	400
query19	203	146	141	141
query20	75	81	73	73
query21	181	97	89	89
query22	5267	5067	5057	5057
query23	32489	31969	31778	31778
query24	7000	6512	6458	6458
query25	552	422	416	416
query26	525	167	153	153
query27	1907	285	285	285
query28	6018	2212	2173	2173
query29	2981	2872	2822	2822
query30	234	159	162	159
query31	895	722	783	722
query32	61	53	60	53
query33	391	252	263	252
query34	853	450	473	450
query35	1113	920	882	882
query36	1408	1664	1924	1664
query37	90	59	63	59
query38	3064	2916	2907	2907
query39	1380	1333	1318	1318
query40	205	94	94	94
query41	35	35	32	32
query42	91	76	81	76
query43	631	641	626	626
query44	1122	705	722	705
query45	237	227	223	223
query46	1231	966	956	956
query47	2116	1704	1683	1683
query48	978	673	664	664
query49	618	376	364	364
query50	863	609	608	608
query51	5501	5463	5390	5390
query52	81	84	82	82
query53	447	323	320	320
query54	2444	2276	2271	2271
query55	79	75	85	75
query56	188	195	197	195
query57	1108	1105	1313	1105
query58	213	195	194	194
query59	3523	3214	3153	3153
query60	207	184	207	184
query61	82	89	80	80
query62	824	507	541	507
query63	475	332	340	332
query64	2024	1283	1270	1270
query65	3667	3652	3630	3630
query66	825	373	357	357
query67	15753	15288	17687	15288
query68	8575	624	677	624
query69	560	328	319	319
query70	2208	1726	1870	1726
query71	379	303	313	303
query72	4369	3262	3274	3262
query73	728	318	327	318
query74	6368	5861	5729	5729
query75	4663	3541	3531	3531
query76	5110	1182	1210	1182
query77	865	251	245	245
query78	32329	52506	46953	46953
query79	15265	678	633	633
query80	4862	368	379	368
query81	568	230	227	227
query82	1012	100	91	91
query83	430	132	133	132
query84	259	68	66	66
query85	2376	281	270	270
query86	446	411	433	411
query87	3248	2946	3012	2946
query88	6690	2363	2332	2332
query89	423	280	282	280
query90	2520	204	205	204
query91	163	133	119	119
query92	65	53	53	53
query93	5689	594	603	594
query94	1702	210	204	204
query95	1108	1062	1064	1062
query96	649	338	332	332
query97	6416	6312	6401	6312
query98	200	180	170	170
query99	3832	915	947	915
Total cold run time: 345483 ms
Total hot run time: 237177 ms

doris-robot · 2024-02-04T16:55:46Z

ClickBench: Total hot run time: 30.82 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e7a9bd476654b49a7e125cf21e40ad9556885eb0, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.05	0.05
query4	1.80	0.08	0.06
query5	0.54	0.52	0.53
query6	1.25	0.65	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.52	0.48	0.46
query10	0.53	0.54	0.55
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.61	0.61	0.62
query14	0.79	0.80	0.78
query15	0.77	0.76	0.76
query16	0.36	0.36	0.37
query17	1.03	1.01	1.00
query18	0.22	0.27	0.22
query19	1.90	1.87	1.85
query20	0.01	0.01	0.00
query21	15.49	0.57	0.57
query22	2.06	2.11	1.35
query23	17.13	1.00	1.04
query24	7.48	1.05	1.21
query25	1.46	0.10	0.11
query26	0.32	0.15	0.12
query27	0.12	0.11	0.11
query28	6.43	0.75	0.70
query29	12.58	2.43	2.41
query30	0.56	0.53	0.52
query31	2.82	0.38	0.39
query32	3.39	0.50	0.49
query33	3.03	3.07	3.08
query34	15.26	4.78	4.78
query35	4.84	4.86	4.83
query36	1.06	1.03	1.01
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.13
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.35 s
Total hot run time: 30.82 s

doris-robot · 2024-02-04T16:59:01Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e7a9bd476654b49a7e125cf21e40ad9556885eb0 with default session variables
Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

…te (apache#30719)

github-actions bot added area/planner Issues or PRs related to the query planner kind/docs Categorizes issue or PR as related to documentation. kind/test labels Feb 1, 2024

two pr

bcaaec9

zclllyybb force-pushed the pick_date_str branch from fe6f9a2 to bcaaec9 Compare February 4, 2024 09:57

fix case

e7a9bd4

xiaokang marked this pull request as draft February 6, 2024 09:36

zclllyybb marked this pull request as ready for review February 7, 2024 13:23

yiguolei merged commit 79b693c into apache:branch-2.0 Feb 7, 2024

zclllyybb deleted the pick_date_str branch February 13, 2024 14:29

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[branch-2.0](datetime) refactor and support more format for str_to_da…

3eaa849

…te (apache#30719)

[branch-2.0](datetime) refactor and support more format for str_to_date #30719

[branch-2.0](datetime) refactor and support more format for str_to_date #30719

Uh oh!

Conversation

zclllyybb commented Feb 1, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Feb 1, 2024

Uh oh!

zclllyybb commented Feb 1, 2024

Uh oh!

github-actions bot commented Feb 1, 2024

Uh oh!

zclllyybb commented Feb 4, 2024

Uh oh!

github-actions bot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

zclllyybb commented Feb 4, 2024

Uh oh!

github-actions bot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants