Skip to content

branch-3.1: [fix][paimon] Fix Paimon date partition conversion failure #54043 #54250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Aug 4, 2025
Merged

branch-3.1: [fix][paimon] Fix Paimon date partition conversion failure #54043 #54250

morrySnow merged 1 commit into from
Aug 4, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 3, 2025

Cherry-picked from #54043

[fix][paimon] Fix Paimon date partition conversion failure (#54043)
d328065 
### What problem does this PR solve?

**Background**: PR [#46641](#46641)
worked around Paimon date partition failures by treating date partitions
as unpartitioned tables.

**Root Cause**: Paimon stores date partition values as integers (e.g.,
`20231225` for `2023-12-25`), but the previous implementation failed to
convert these to Doris-compatible date formats, resulting in:
```
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = date literal [19737] is invalid: null
```
@github-actions github-actions bot requested a review from morrySnow as a code owner August 3, 2025 22:06
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Aug 3, 2025
@dataroaring dataroaring reopened this Aug 3, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 3, 2025

TPC-H: Total hot run time: 32729 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d328065bd6565df4997dadca18689888400d29dc, data reload: false

------ Round 1 ----------------------------------
q1	17583	5454	5482	5454
q2	2064	272	164	164
q3	10547	1247	752	752
q4	10265	854	450	450
q5	8671	2361	2098	2098
q6	184	166	130	130
q7	896	746	610	610
q8	9325	1439	1161	1161
q9	5175	4871	4885	4871
q10	6796	2253	1846	1846
q11	476	277	253	253
q12	330	338	209	209
q13	17993	3609	3014	3014
q14	244	246	221	221
q15	544	474	478	474
q16	418	430	382	382
q17	609	873	376	376
q18	7028	6566	6546	6546
q19	1326	957	565	565
q20	317	347	198	198
q21	2881	2092	1949	1949
q22	1038	1006	1007	1006
Total cold run time: 104710 ms
Total hot run time: 32729 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5576	5520	5506	5506
q2	245	325	228	228
q3	2259	2630	2281	2281
q4	1319	1788	1340	1340
q5	4434	4849	4961	4849
q6	169	157	129	129
q7	2040	1981	1839	1839
q8	2607	2790	2663	2663
q9	7268	7236	7192	7192
q10	3002	3335	2811	2811
q11	570	499	485	485
q12	617	709	582	582
q13	3375	3846	3213	3213
q14	278	292	273	273
q15	522	469	463	463
q16	460	498	444	444
q17	1210	1708	1252	1252
q18	7570	7399	7457	7399
q19	824	1104	1078	1078
q20	2002	2045	1924	1924
q21	5368	4948	4820	4820
q22	1094	1047	1014	1014
Total cold run time: 52809 ms
Total hot run time: 51785 ms

@doris-robot
Copy link

doris-robot commented Aug 3, 2025

TPC-DS: Total hot run time: 198139 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d328065bd6565df4997dadca18689888400d29dc, data reload: false

query1	1304	946	914	914
query2	6230	2043	1959	1959
query3	10887	4438	4508	4438
query4	32766	24192	23957	23957
query5	4901	609	450	450
query6	315	191	197	191
query7	4027	502	322	322
query8	302	254	237	237
query9	9582	2590	2570	2570
query10	496	326	253	253
query11	18122	15377	15150	15150
query12	156	122	104	104
query13	1557	531	417	417
query14	9544	7363	6849	6849
query15	224	221	183	183
query16	8040	652	505	505
query17	1525	779	601	601
query18	2110	430	327	327
query19	215	194	174	174
query20	128	120	115	115
query21	212	124	112	112
query22	4682	4769	4589	4589
query23	35076	34310	34269	34269
query24	7205	2683	2706	2683
query25	507	466	402	402
query26	1173	280	179	179
query27	2414	456	351	351
query28	5635	2221	2182	2182
query29	727	588	458	458
query30	246	196	172	172
query31	983	921	852	852
query32	83	61	56	56
query33	493	365	317	317
query34	749	863	539	539
query35	774	835	709	709
query36	988	1064	946	946
query37	103	94	69	69
query38	3987	4007	4000	4000
query39	1539	1480	1465	1465
query40	205	119	114	114
query41	50	48	52	48
query42	119	105	107	105
query43	505	518	497	497
query44	1353	845	832	832
query45	198	176	175	175
query46	881	1062	684	684
query47	1968	2005	1927	1927
query48	424	441	353	353
query49	791	497	428	428
query50	689	702	457	457
query51	7424	7413	7312	7312
query52	102	101	91	91
query53	239	272	197	197
query54	540	540	476	476
query55	81	81	77	77
query56	271	280	258	258
query57	1292	1246	1179	1179
query58	250	240	232	232
query59	3177	3235	3142	3142
query60	299	319	280	280
query61	115	112	155	112
query62	786	769	699	699
query63	240	210	205	205
query64	4369	1002	648	648
query65	3338	3276	3324	3276
query66	993	436	330	330
query67	16222	15877	15633	15633
query68	2659	867	588	588
query69	467	297	278	278
query70	1235	1085	1070	1070
query71	370	297	263	263
query72	5745	3699	3876	3699
query73	643	744	350	350
query74	9952	9076	8908	8908
query75	3214	3160	2715	2715
query76	2972	1180	772	772
query77	485	378	297	297
query78	10450	10352	9695	9695
query79	2599	891	611	611
query80	1707	555	460	460
query81	548	264	223	223
query82	348	118	92	92
query83	251	168	160	160
query84	244	111	82	82
query85	860	460	375	375
query86	469	324	304	304
query87	4323	4365	4226	4226
query88	3869	2433	2377	2377
query89	402	341	298	298
query90	1655	195	196	195
query91	154	150	120	120
query92	62	58	86	58
query93	1439	888	547	547
query94	660	399	300	300
query95	345	283	270	270
query96	484	620	294	294
query97	3204	3319	3159	3159
query98	235	221	206	206
query99	1344	1438	1314	1314
Total cold run time: 290322 ms
Total hot run time: 198139 ms

@doris-robot
Copy link

doris-robot commented Aug 3, 2025

ClickBench: Total hot run time: 28.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d328065bd6565df4997dadca18689888400d29dc, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.02
query3	0.24	0.06	0.06
query4	1.62	0.11	0.10
query5	0.52	0.52	0.53
query6	1.12	0.73	0.73
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.50	0.51
query10	0.56	0.56	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	0.76	0.82	0.79
query15	0.85	0.83	0.81
query16	0.38	0.37	0.39
query17	1.01	1.08	1.00
query18	0.23	0.22	0.21
query19	1.86	1.78	1.81
query20	0.01	0.02	0.00
query21	15.39	0.92	0.57
query22	0.74	0.82	0.72
query23	15.00	1.44	0.53
query24	2.62	1.70	0.73
query25	0.25	0.09	0.06
query26	0.18	0.14	0.13
query27	0.06	0.07	0.05
query28	13.86	1.02	0.43
query29	12.59	3.96	3.32
query30	0.25	0.09	0.06
query31	2.84	0.63	0.38
query32	3.23	0.54	0.45
query33	2.99	3.03	3.03
query34	16.68	5.17	4.54
query35	4.56	4.55	4.51
query36	0.65	0.49	0.50
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 103.16 s
Total hot run time: 28.61 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 3, 2025

FE UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉
Increment coverage report
Complete coverage report

@vinlee19
Copy link
Contributor

vinlee19 commented Aug 4, 2025

run p0

@morrySnow morrySnow merged commit f14b453 into branch-3.1 Aug 4, 2025
23 checks passed
@github-actions github-actions bot deleted the auto-pick-54043-branch-3.1 branch August 4, 2025 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @vinlee19 @morrySnow @dataroaring