Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bugfix](paimon)Fixed the reading of timestamp with time zone type data #37716

Merged
merged 6 commits into from
Jul 15, 2024
Merged

[bugfix](paimon)Fixed the reading of timestamp with time zone type data #37716

merged 6 commits into from
Jul 15, 2024

Conversation

wuwenchi
Copy link
Contributor

Proposed changes

  1. When using jni to read timestamps with time zones, the time needs to be converted to local time
  2. In version 0.8 of paimon, the time zone (isAdjustToUTC) information of parquet files is added, and doris can parse data directly according to the time zone information

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi
Copy link
Contributor Author

run buildall

@wuwenchi
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39665 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ed9eefd57f4506786d406b6228fc313e126e7e9, data reload: false

------ Round 1 ----------------------------------
q1	17629	4392	4263	4263
q2	2011	185	187	185
q3	10454	1148	1016	1016
q4	10194	832	787	787
q5	7569	2694	2857	2694
q6	218	135	140	135
q7	979	599	618	599
q8	9203	2073	2066	2066
q9	8730	6541	6534	6534
q10	8864	3799	3738	3738
q11	452	237	239	237
q12	466	225	227	225
q13	18945	2979	2945	2945
q14	277	236	237	236
q15	529	481	500	481
q16	501	390	376	376
q17	965	630	737	630
q18	8115	7541	7339	7339
q19	6255	1428	1475	1428
q20	682	328	331	328
q21	4867	3085	3231	3085
q22	401	338	338	338
Total cold run time: 118306 ms
Total hot run time: 39665 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4360	4259	4232	4232
q2	366	292	281	281
q3	3005	2933	2901	2901
q4	2043	1686	1680	1680
q5	5571	5580	5521	5521
q6	231	135	132	132
q7	2211	1913	1874	1874
q8	3263	3457	3398	3398
q9	8729	8746	8791	8746
q10	4109	3811	3753	3753
q11	591	514	491	491
q12	802	632	632	632
q13	16576	3150	3229	3150
q14	309	298	285	285
q15	533	496	511	496
q16	497	443	425	425
q17	1817	1523	1482	1482
q18	8046	7876	7901	7876
q19	1848	1654	1579	1579
q20	2983	1871	1884	1871
q21	9313	4927	4570	4570
q22	638	562	560	560
Total cold run time: 77841 ms
Total hot run time: 55935 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174195 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ed9eefd57f4506786d406b6228fc313e126e7e9, data reload: false

query1	917	374	362	362
query2	6438	2414	2320	2320
query3	6633	208	218	208
query4	28300	17658	17329	17329
query5	3780	494	482	482
query6	267	181	155	155
query7	4576	297	296	296
query8	305	284	287	284
query9	8446	2453	2436	2436
query10	451	308	274	274
query11	11433	9890	10205	9890
query12	116	86	84	84
query13	1635	380	380	380
query14	10151	7253	7609	7253
query15	242	188	196	188
query16	7734	335	333	333
query17	1412	566	550	550
query18	1973	283	285	283
query19	199	158	158	158
query20	92	80	86	80
query21	213	129	126	126
query22	4348	4173	3974	3974
query23	33949	33648	34081	33648
query24	10803	3029	3014	3014
query25	594	388	382	382
query26	708	155	148	148
query27	2210	283	293	283
query28	5716	2178	2174	2174
query29	875	643	614	614
query30	260	150	154	150
query31	980	793	758	758
query32	101	58	58	58
query33	660	288	295	288
query34	889	498	498	498
query35	701	586	592	586
query36	1132	998	964	964
query37	159	90	86	86
query38	2936	2814	2842	2814
query39	898	815	819	815
query40	197	119	117	117
query41	55	55	55	55
query42	113	98	94	94
query43	568	561	564	561
query44	1089	714	712	712
query45	198	164	168	164
query46	1076	730	728	728
query47	1874	1772	1810	1772
query48	363	298	303	298
query49	859	417	412	412
query50	777	401	389	389
query51	7027	6750	6678	6678
query52	112	91	92	91
query53	346	293	293	293
query54	891	448	473	448
query55	81	78	76	76
query56	296	274	269	269
query57	1115	1082	1063	1063
query58	240	234	257	234
query59	3233	3019	3146	3019
query60	305	269	278	269
query61	119	94	123	94
query62	753	649	636	636
query63	310	282	279	279
query64	9127	2206	1643	1643
query65	3113	3109	3084	3084
query66	698	328	328	328
query67	15372	14824	14904	14824
query68	4607	545	552	545
query69	683	497	361	361
query70	1162	1128	1154	1128
query71	404	285	279	279
query72	9205	5466	6032	5466
query73	754	338	330	330
query74	5917	5550	5501	5501
query75	4139	2682	2710	2682
query76	2930	943	960	943
query77	651	318	327	318
query78	10454	9038	9621	9038
query79	3497	518	518	518
query80	1668	470	469	469
query81	590	220	220	220
query82	768	141	139	139
query83	355	168	171	168
query84	270	91	87	87
query85	1520	381	309	309
query86	458	303	308	303
query87	3304	3166	3137	3137
query88	4593	2439	2410	2410
query89	488	383	379	379
query90	2014	189	189	189
query91	132	102	103	102
query92	57	48	50	48
query93	4979	519	511	511
query94	1456	217	225	217
query95	409	321	318	318
query96	603	269	275	269
query97	3228	3061	3041	3041
query98	223	198	201	198
query99	1553	1255	1217	1217
Total cold run time: 286610 ms
Total hot run time: 174195 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3ed9eefd57f4506786d406b6228fc313e126e7e9, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.22	0.04	0.05
query4	1.68	0.06	0.07
query5	0.50	0.48	0.49
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.56	0.49	0.49
query10	0.55	0.55	0.53
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.58	0.58	0.58
query14	0.77	0.78	0.79
query15	0.86	0.81	0.81
query16	0.37	0.36	0.36
query17	0.99	0.99	1.04
query18	0.24	0.22	0.22
query19	1.80	1.73	1.79
query20	0.01	0.01	0.01
query21	15.43	0.74	0.64
query22	3.91	7.69	2.18
query23	18.30	1.37	1.24
query24	2.14	0.24	0.22
query25	0.15	0.08	0.09
query26	0.30	0.22	0.21
query27	0.45	0.22	0.22
query28	13.15	1.02	0.99
query29	12.61	3.33	3.29
query30	0.26	0.06	0.06
query31	2.86	0.38	0.40
query32	3.26	0.48	0.48
query33	2.84	2.96	2.93
query34	16.95	4.31	4.38
query35	4.36	4.48	4.48
query36	0.65	0.47	0.48
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.04
query42	0.06	0.04	0.04
query43	0.04	0.05	0.04
Total cold run time: 109.11 s
Total hot run time: 30.91 s

@wuwenchi
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39807 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f2ebe95138931a628cad5cf0d20ebb5754bf2cf, data reload: false

------ Round 1 ----------------------------------
q1	17608	4457	4330	4330
q2	2031	189	188	188
q3	10452	1176	1005	1005
q4	10187	747	795	747
q5	7542	2681	2679	2679
q6	232	140	140	140
q7	967	599	595	595
q8	9218	2071	2073	2071
q9	6885	6566	6539	6539
q10	8805	3722	3755	3722
q11	453	240	240	240
q12	496	221	216	216
q13	18752	2978	2989	2978
q14	275	239	246	239
q15	555	495	486	486
q16	515	384	387	384
q17	967	615	706	615
q18	8253	7405	7498	7405
q19	1567	1421	1473	1421
q20	672	318	326	318
q21	5022	3210	3257	3210
q22	349	289	279	279
Total cold run time: 111803 ms
Total hot run time: 39807 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4303	4250	4210	4210
q2	384	274	264	264
q3	3039	2815	2737	2737
q4	1872	1561	1626	1561
q5	5333	5328	5309	5309
q6	216	131	128	128
q7	2139	1720	1743	1720
q8	3191	3332	3338	3332
q9	8445	8380	8381	8380
q10	3908	3739	3688	3688
q11	576	483	485	483
q12	782	603	601	601
q13	17704	2974	2995	2974
q14	305	272	281	272
q15	524	479	483	479
q16	484	424	429	424
q17	1774	1466	1470	1466
q18	7559	7515	7356	7356
q19	1651	1579	1633	1579
q20	1997	1813	1787	1787
q21	4922	4765	4661	4661
q22	606	499	539	499
Total cold run time: 71714 ms
Total hot run time: 53910 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173381 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f2ebe95138931a628cad5cf0d20ebb5754bf2cf, data reload: false

query1	922	379	366	366
query2	6453	1825	1774	1774
query3	6670	208	221	208
query4	28280	17712	17455	17455
query5	4159	480	479	479
query6	279	167	160	160
query7	4585	288	293	288
query8	236	198	202	198
query9	8532	2465	2445	2445
query10	458	290	282	282
query11	10501	9969	10091	9969
query12	141	83	82	82
query13	1641	385	368	368
query14	10182	7502	7714	7502
query15	225	167	170	167
query16	7830	325	318	318
query17	1799	575	571	571
query18	1842	285	285	285
query19	193	163	160	160
query20	93	83	83	83
query21	220	132	130	130
query22	4274	4160	4029	4029
query23	33703	33333	33250	33250
query24	12056	2889	2825	2825
query25	677	384	386	384
query26	1813	151	150	150
query27	2880	277	277	277
query28	7642	2028	2026	2026
query29	1128	647	623	623
query30	268	147	147	147
query31	948	748	764	748
query32	99	59	57	57
query33	786	317	312	312
query34	914	493	481	481
query35	689	601	579	579
query36	1099	904	943	904
query37	233	82	83	82
query38	2855	2746	2727	2727
query39	840	809	802	802
query40	284	122	121	121
query41	50	47	46	46
query42	119	99	102	99
query43	510	482	474	474
query44	1197	745	734	734
query45	193	161	165	161
query46	1083	729	718	718
query47	1883	1881	1768	1768
query48	362	295	294	294
query49	1198	409	421	409
query50	767	407	398	398
query51	6906	6779	6770	6770
query52	107	90	91	90
query53	361	292	288	288
query54	1036	449	452	449
query55	76	75	73	73
query56	289	275	292	275
query57	1152	1061	1026	1026
query58	265	236	248	236
query59	2878	2656	2511	2511
query60	309	271	286	271
query61	100	99	96	96
query62	817	655	631	631
query63	323	308	277	277
query64	10478	2228	2378	2228
query65	3194	3107	3093	3093
query66	1372	341	342	341
query67	15585	15033	14964	14964
query68	6360	555	561	555
query69	702	443	357	357
query70	1186	1067	1091	1067
query71	499	271	275	271
query72	8065	5680	5743	5680
query73	803	328	326	326
query74	5919	5470	5537	5470
query75	4325	2674	2719	2674
query76	4661	968	932	932
query77	722	326	317	317
query78	9706	10064	8922	8922
query79	7991	526	544	526
query80	1230	486	484	484
query81	588	217	224	217
query82	804	144	133	133
query83	226	168	167	167
query84	278	85	86	85
query85	1443	396	298	298
query86	457	311	326	311
query87	3257	3085	3117	3085
query88	5275	2448	2462	2448
query89	522	399	375	375
query90	2019	193	194	193
query91	132	120	101	101
query92	61	51	52	51
query93	6066	519	516	516
query94	1308	216	215	215
query95	409	313	322	313
query96	617	276	270	270
query97	3185	3037	3067	3037
query98	218	197	194	194
query99	1551	1314	1257	1257
Total cold run time: 301624 ms
Total hot run time: 173381 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f2ebe95138931a628cad5cf0d20ebb5754bf2cf, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.71	0.08	0.08
query5	0.49	0.49	0.49
query6	1.14	0.72	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.47
query10	0.55	0.54	0.55
query11	0.15	0.12	0.11
query12	0.14	0.13	0.12
query13	0.60	0.60	0.59
query14	0.78	0.79	0.78
query15	0.84	0.81	0.82
query16	0.37	0.35	0.37
query17	0.99	1.00	1.04
query18	0.22	0.21	0.21
query19	1.83	1.68	1.72
query20	0.02	0.01	0.01
query21	15.43	0.77	0.65
query22	4.39	6.99	2.08
query23	18.26	1.30	1.27
query24	2.09	0.24	0.23
query25	0.15	0.09	0.09
query26	0.29	0.21	0.21
query27	0.45	0.24	0.23
query28	13.25	1.02	1.01
query29	12.64	3.30	3.27
query30	0.26	0.06	0.05
query31	2.88	0.39	0.39
query32	3.24	0.47	0.47
query33	2.89	2.96	2.90
query34	17.13	4.35	4.34
query35	4.46	4.41	4.37
query36	0.65	0.50	0.48
query37	0.18	0.16	0.15
query38	0.16	0.15	0.16
query39	0.04	0.03	0.03
query40	0.15	0.12	0.12
query41	0.10	0.05	0.06
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.98 s
Total hot run time: 30.75 s

morningman
morningman approved these changes Jul 12, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 12, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 87962aa into apache:master Jul 15, 2024
25 of 29 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@wuwenchi @seawinde
[bugfix](paimon)Fixed the reading of timestamp with time zone type da…
dde0f7a 
…ta (apache#37716)

## Proposed changes

1. When using jni to read timestamps with time zones, the time needs to
be converted to local time
2. In version 0.8 of paimon, the time zone (isAdjustToUTC) information
of parquet files is added, and doris can parse data directly according
to the time zone information
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@wuwenchi @dataroaring
[bugfix](paimon)Fixed the reading of timestamp with time zone type da…
fd5f0d4 
…ta (#37716)

## Proposed changes

1. When using jni to read timestamps with time zones, the time needs to
be converted to local time
2. In version 0.8 of paimon, the time zone (isAdjustToUTC) information
of parquet files is added, and doris can parse data directly according
to the time zone information
wuwenchi added a commit to wuwenchi/doris_new that referenced this pull request Jul 31, 2024
@wuwenchi
[bugfix](paimon)Fixed the reading of timestamp with time zone type da…
dae011d 
…ta (apache#37716)

## Proposed changes

1. When using jni to read timestamps with time zones, the time needs to
be converted to local time
2. In version 0.8 of paimon, the time zone (isAdjustToUTC) information
of parquet files is added, and doris can parse data directly according
to the time zone information

(cherry picked from commit 87962aa)
@wuwenchi wuwenchi mentioned this pull request Jul 31, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 1, 2024
@wuwenchi
[bugfix](paimon)Fixed the reading of timestamp with time zone type da…
41fa7bc 
…ta for 2.1 (#37716) (#38592)

bp: #37716
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@wuwenchi @doris-robot @morningman @yiguolei @dataroaring