Skip to content

branch-3.0: [enhancement](maxcompute)support maxcompute timestamp column type. #48768 #48841

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Mar 19, 2025
Merged

branch-3.0: [enhancement](maxcompute)support maxcompute timestamp column type. #48768 #48841

dataroaring merged 1 commit into from
Mar 19, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Mar 9, 2025

Cherry-picked from #48768

@hubgeter
[enhancement](maxcompute)support maxcompute timestamp column type. (#…
31d7f8a 
…48768)

### What problem does this PR solve?

Problem Summary:
1. support maxcompute timestamp column type.
2. Add the parameter `mc.datetime_predicate_push_down` to disable
predicate pushdown for odps catalog datetime type, because the timestamp
precision of odps is 9, while the mapping precision of Doris is 6. If we
insert `2023-02-02 00:00:00.123456789` into odps, doris will read it as
`2023-02-02 00:00:00.123456`. Due to the lack of "789", we cannot push
it down correctly. If you don't need such a high precision (greater than
6) on odps, it will not affect your normal use.
@github-actions github-actions bot requested a review from dataroaring as a code owner March 9, 2025 13:55
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Mar 9, 2025
@dataroaring dataroaring reopened this Mar 9, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 9, 2025

TPC-H: Total hot run time: 40768 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 31d7f8a02ce29bfbef8d06fc73006354c7f1ca1c, data reload: false

------ Round 1 ----------------------------------
q1	17642	7276	6677	6677
q2	2070	176	162	162
q3	10581	1098	1203	1098
q4	10730	754	807	754
q5	8821	2866	2831	2831
q6	223	138	135	135
q7	1017	656	622	622
q8	9343	1983	2041	1983
q9	6691	6443	6472	6443
q10	7048	2278	2311	2278
q11	474	267	268	267
q12	401	212	209	209
q13	19587	3172	3159	3159
q14	257	232	228	228
q15	525	466	473	466
q16	693	596	607	596
q17	1006	538	584	538
q18	9055	6973	6725	6725
q19	1437	1029	1053	1029
q20	472	221	210	210
q21	4253	3388	3327	3327
q22	1164	1056	1031	1031
Total cold run time: 113490 ms
Total hot run time: 40768 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6695	6620	6657	6620
q2	361	238	247	238
q3	3141	2982	3043	2982
q4	2171	1939	1853	1853
q5	5825	5860	5864	5860
q6	207	131	129	129
q7	2224	1802	1833	1802
q8	3382	3437	3520	3437
q9	8875	8880	8806	8806
q10	3580	3584	3537	3537
q11	590	492	486	486
q12	820	613	588	588
q13	13908	3162	3169	3162
q14	300	288	286	286
q15	527	472	469	469
q16	671	660	651	651
q17	1837	1606	1590	1590
q18	8274	7614	7461	7461
q19	1648	1643	1594	1594
q20	2104	1903	1898	1898
q21	5591	5313	5363	5313
q22	1134	1036	1033	1033
Total cold run time: 73865 ms
Total hot run time: 59795 ms

@doris-robot
Copy link

doris-robot commented Mar 9, 2025

TPC-DS: Total hot run time: 197488 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 31d7f8a02ce29bfbef8d06fc73006354c7f1ca1c, data reload: false

query1	1308	937	908	908
query2	6212	2119	2044	2044
query3	10951	4313	4342	4313
query4	65510	28814	23779	23779
query5	5059	448	444	444
query6	407	175	169	169
query7	5644	317	303	303
query8	289	214	211	211
query9	9315	2592	2587	2587
query10	466	275	275	275
query11	17813	15053	15628	15053
query12	156	103	109	103
query13	1520	441	457	441
query14	9782	7800	7680	7680
query15	207	189	189	189
query16	7226	519	485	485
query17	1097	595	578	578
query18	1852	311	318	311
query19	216	167	166	166
query20	115	111	111	111
query21	209	99	104	99
query22	4713	4450	4443	4443
query23	34219	33832	33935	33832
query24	6124	2905	2877	2877
query25	520	406	399	399
query26	656	166	168	166
query27	1773	356	360	356
query28	3788	2445	2416	2416
query29	732	477	462	462
query30	259	180	171	171
query31	985	834	841	834
query32	69	57	58	57
query33	449	293	303	293
query34	915	503	516	503
query35	878	738	727	727
query36	1101	943	963	943
query37	124	74	66	66
query38	4118	4156	3950	3950
query39	1544	1475	1501	1475
query40	210	102	107	102
query41	51	54	50	50
query42	123	101	98	98
query43	539	499	512	499
query44	1195	824	850	824
query45	194	170	170	170
query46	1126	724	727	724
query47	2044	1932	1956	1932
query48	477	383	404	383
query49	765	411	427	411
query50	848	438	426	426
query51	7330	7161	7119	7119
query52	100	89	91	89
query53	257	182	183	182
query54	571	471	460	460
query55	82	77	79	77
query56	271	250	266	250
query57	1258	1105	1148	1105
query58	225	217	229	217
query59	3191	2977	2795	2795
query60	271	247	257	247
query61	124	144	104	104
query62	763	677	685	677
query63	215	184	197	184
query64	1419	680	633	633
query65	3218	3161	3226	3161
query66	716	313	300	300
query67	15727	15497	15504	15497
query68	4324	555	569	555
query69	425	272	266	266
query70	1111	1126	1135	1126
query71	363	268	250	250
query72	6381	4017	4043	4017
query73	752	351	346	346
query74	10214	8906	9027	8906
query75	3359	2682	2667	2667
query76	1896	1078	1101	1078
query77	499	273	271	271
query78	10455	9686	9628	9628
query79	1232	593	597	593
query80	836	450	430	430
query81	512	242	235	235
query82	1317	88	86	86
query83	241	151	142	142
query84	289	89	77	77
query85	872	309	296	296
query86	339	300	293	293
query87	4431	4349	4257	4257
query88	3572	2393	2341	2341
query89	412	292	292	292
query90	2002	184	183	183
query91	192	168	150	150
query92	69	48	53	48
query93	1306	544	539	539
query94	778	302	271	271
query95	362	262	263	262
query96	603	279	283	279
query97	3285	3192	3194	3192
query98	217	205	194	194
query99	1793	1297	1283	1283
Total cold run time: 316070 ms
Total hot run time: 197488 ms

@doris-robot
Copy link

doris-robot commented Mar 9, 2025

ClickBench: Total hot run time: 32.47 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 31d7f8a02ce29bfbef8d06fc73006354c7f1ca1c, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.04
query3	0.24	0.06	0.07
query4	1.64	0.10	0.10
query5	0.51	0.52	0.51
query6	1.14	0.73	0.71
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.56	0.50	0.50
query10	0.55	0.55	0.55
query11	0.14	0.11	0.10
query12	0.15	0.12	0.12
query13	0.62	0.60	0.59
query14	2.90	2.76	2.82
query15	0.90	0.82	0.82
query16	0.38	0.38	0.37
query17	1.06	1.07	1.05
query18	0.23	0.22	0.22
query19	1.99	1.84	2.02
query20	0.01	0.01	0.00
query21	15.38	0.57	0.58
query22	2.89	1.66	2.38
query23	16.92	0.84	0.82
query24	2.85	1.12	1.75
query25	0.12	0.22	0.19
query26	0.51	0.14	0.13
query27	0.04	0.05	0.05
query28	9.89	0.54	0.45
query29	12.56	3.31	3.27
query30	0.25	0.06	0.07
query31	2.87	0.39	0.38
query32	3.23	0.46	0.46
query33	2.95	3.01	3.02
query34	17.15	4.54	4.53
query35	4.53	4.60	4.58
query36	0.68	0.50	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.5 s
Total hot run time: 32.47 s

dataroaring
dataroaring approved these changes Mar 19, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit da284e0 into branch-3.0 Mar 19, 2025
23 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-48768-branch-3.0 branch March 19, 2025 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hello-stephen @doris-robot @dataroaring @hubgeter