Skip to content

[enhance](mtmv)When obtaining the partition list fails, treat the paimon table as an unpartitioned table #46641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 5 commits into from
Jan 9, 2025
Merged

[enhance](mtmv)When obtaining the partition list fails, treat the paimon table as an unpartitioned table #46641

morrySnow merged 5 commits into from
Jan 9, 2025

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented Jan 8, 2025 •
edited
Loading

What problem does this PR solve?

When retrieving data of type Paimon Date in version 0.9 from the system table, the value is an integer and cannot be converted to type Date.

This issue has been fixed in Paimon's latest code.

This PR downgrades this situation without affecting user data queries

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
When obtaining the partition list fails, treat the paimon table as an unpartitioned table

Release note

When obtaining the partition list fails, treat the paimon table as an unpartitioned table

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 8, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Jan 8, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 8, 2025

TPC-H: Total hot run time: 32683 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed7d320c0624c9ae04bb81d6fd6ccf9d78fb2271, data reload: false

------ Round 1 ----------------------------------
q1	17634	6180	6059	6059
q2	2044	309	166	166
q3	10409	1241	715	715
q4	10235	847	432	432
q5	7546	2182	1916	1916
q6	206	190	154	154
q7	895	744	610	610
q8	9237	1351	1144	1144
q9	5150	4912	4933	4912
q10	6761	2283	1838	1838
q11	463	282	257	257
q12	341	359	224	224
q13	17786	3738	3110	3110
q14	247	234	206	206
q15	568	522	499	499
q16	629	635	590	590
q17	551	844	332	332
q18	6989	6524	6543	6524
q19	1234	958	561	561
q20	309	313	200	200
q21	2796	2129	1926	1926
q22	365	326	308	308
Total cold run time: 102395 ms
Total hot run time: 32683 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6252	6259	6261	6259
q2	249	328	234	234
q3	2253	2687	2340	2340
q4	1406	1809	1409	1409
q5	4299	4703	4707	4703
q6	190	183	146	146
q7	2113	1987	1823	1823
q8	2539	2767	2699	2699
q9	7346	7196	7189	7189
q10	3048	3280	2843	2843
q11	612	518	496	496
q12	663	757	595	595
q13	3460	3783	3307	3307
q14	278	315	305	305
q15	566	509	510	509
q16	662	679	641	641
q17	1207	1716	1268	1268
q18	7882	7422	7354	7354
q19	788	838	1105	838
q20	1901	1970	1828	1828
q21	5583	5053	4941	4941
q22	606	600	540	540
Total cold run time: 53903 ms
Total hot run time: 52267 ms

@doris-robot
Copy link

doris-robot commented Jan 8, 2025

TPC-DS: Total hot run time: 188251 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed7d320c0624c9ae04bb81d6fd6ccf9d78fb2271, data reload: false

query1	965	378	388	378
query2	6511	2375	2319	2319
query3	6709	218	225	218
query4	33729	23290	23212	23212
query5	4307	588	439	439
query6	272	206	183	183
query7	4637	490	311	311
query8	297	273	232	232
query9	9659	2737	2717	2717
query10	474	315	259	259
query11	17916	15235	14939	14939
query12	159	109	111	109
query13	1673	545	399	399
query14	10022	6450	6617	6450
query15	241	199	192	192
query16	8150	565	484	484
query17	1582	742	582	582
query18	2124	432	289	289
query19	205	176	162	162
query20	128	110	112	110
query21	205	121	124	121
query22	4119	4333	4058	4058
query23	34081	33111	33224	33111
query24	7225	2248	2287	2248
query25	485	461	377	377
query26	1201	270	156	156
query27	2659	460	338	338
query28	5830	2465	2445	2445
query29	741	545	409	409
query30	231	179	149	149
query31	954	867	804	804
query32	105	61	60	60
query33	526	344	288	288
query34	724	833	498	498
query35	774	801	721	721
query36	1071	1031	959	959
query37	122	104	75	75
query38	3980	4094	3931	3931
query39	1493	1437	1433	1433
query40	203	117	101	101
query41	46	47	45	45
query42	120	103	103	103
query43	511	531	486	486
query44	1264	810	817	810
query45	180	172	159	159
query46	868	1036	651	651
query47	1809	1825	1739	1739
query48	385	425	344	344
query49	792	473	412	412
query50	610	650	396	396
query51	6916	6870	6933	6870
query52	99	104	92	92
query53	218	254	181	181
query54	471	492	409	409
query55	81	81	85	81
query56	253	286	250	250
query57	1138	1155	1101	1101
query58	241	235	226	226
query59	3058	3146	2991	2991
query60	268	267	259	259
query61	106	107	107	107
query62	860	767	703	703
query63	228	204	189	189
query64	4536	996	655	655
query65	3187	3150	3183	3150
query66	1086	428	297	297
query67	15802	15809	15319	15319
query68	8702	756	506	506
query69	459	290	266	266
query70	1232	1136	1103	1103
query71	450	280	264	264
query72	5819	3894	3876	3876
query73	664	763	360	360
query74	9548	9111	8911	8911
query75	4205	3126	2688	2688
query76	3976	1178	760	760
query77	776	368	276	276
query78	10082	10044	9888	9888
query79	2876	786	589	589
query80	698	501	435	435
query81	473	282	229	229
query82	398	152	123	123
query83	198	164	140	140
query84	287	89	66	66
query85	794	347	338	338
query86	349	311	298	298
query87	4523	4501	4270	4270
query88	4210	2196	2175	2175
query89	399	328	298	298
query90	1900	186	186	186
query91	136	137	105	105
query92	62	60	53	53
query93	944	847	513	513
query94	663	390	291	291
query95	331	262	252	252
query96	491	603	284	284
query97	2898	2936	2836	2836
query98	237	203	198	198
query99	1614	1509	1383	1383
Total cold run time: 292016 ms
Total hot run time: 188251 ms

@doris-robot
Copy link

doris-robot commented Jan 8, 2025

ClickBench: Total hot run time: 31.77 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed7d320c0624c9ae04bb81d6fd6ccf9d78fb2271, data reload: false

query1	0.03	0.03	0.02
query2	0.08	0.03	0.04
query3	0.25	0.07	0.07
query4	1.61	0.11	0.10
query5	0.43	0.42	0.41
query6	1.17	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.50	0.49
query10	0.55	0.56	0.54
query11	0.15	0.10	0.10
query12	0.14	0.10	0.10
query13	0.61	0.62	0.59
query14	2.83	2.83	2.80
query15	0.89	0.82	0.83
query16	0.38	0.39	0.39
query17	1.08	1.05	1.07
query18	0.23	0.21	0.20
query19	1.98	1.79	2.01
query20	0.02	0.01	0.01
query21	15.36	0.93	0.58
query22	0.75	0.71	0.64
query23	15.36	1.40	0.52
query24	3.02	1.79	1.38
query25	0.27	0.18	0.13
query26	0.19	0.14	0.13
query27	0.06	0.04	0.06
query28	14.18	1.51	1.05
query29	12.60	3.94	3.26
query30	0.25	0.08	0.06
query31	2.82	0.60	0.38
query32	3.22	0.54	0.46
query33	3.21	3.10	3.17
query34	16.62	5.10	4.54
query35	4.52	4.50	4.50
query36	0.65	0.49	0.48
query37	0.10	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.04
Total cold run time: 106.65 s
Total hot run time: 31.77 s

@zddr
Copy link
Contributor Author

zddr commented Jan 9, 2025

run p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 9, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jan 9, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 9, 2025

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit c206a6b into apache:master Jan 9, 2025
31 of 33 checks passed
zddr added a commit to zddr/incubator-doris that referenced this pull request Jan 9, 2025
@zddr
[enhance](mtmv)When obtaining the partition list fails, treat the pai…
06bd408 
…mon table as an unpartitioned table (apache#46641)

When retrieving data of type Paimon Date in version 0.9 from the system
table, the value is an integer and cannot be converted to type Date.

This issue has been fixed in Paimon's latest code.

This PR downgrades this situation without affecting user data queries
@zddr zddr mentioned this pull request Jan 9, 2025
Merged
zddr added a commit to zddr/incubator-doris that referenced this pull request Jan 9, 2025
@zddr
[enhance](mtmv)When obtaining the partition list fails, treat the pai…
97ccad2 
…mon table as an unpartitioned table (apache#46641)

When retrieving data of type Paimon Date in version 0.9 from the system
table, the value is an integer and cannot be converted to type Date.

This issue has been fixed in Paimon's latest code.

This PR downgrades this situation without affecting user data queries
@zddr zddr mentioned this pull request Jan 9, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jan 9, 2025
@zddr
branch-3.0: [enhance](mtmv)When obtaining the partition list fails, t…
172b859 
…reat the paimon table as an unpartitioned table #46641 (#46709)

pick from master #46641
yiguolei pushed a commit that referenced this pull request Jan 10, 2025
@zddr
[enhance](mtmv)When obtaining the partition list fails, treat the pai… (
c016eb4 
#46708)

…mon table as an unpartitioned table  (#46641)

pick: #46641
zddr added a commit to zddr/incubator-doris that referenced this pull request Jun 20, 2025
@zddr
[enhance](mtmv)When obtaining the partition list fails, treat the pai…
039436e 
…mon table as an unpartitioned table (apache#46641)

When retrieving data of type Paimon Date in version 0.9 from the system
table, the value is an integer and cannot be converted to type Date.

This issue has been fixed in Paimon's latest code.

This PR downgrades this situation without affecting user data queries
@zddr zddr mentioned this pull request Jun 20, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 23, 2025
@zddr @Jibing-Li @BePPPower
branch-3.1: [enhance](mtmv)MTMV support iceberg/hudi/paimon (#52040)
4234a27 
Cherry-pick from
#43959
#44419
#44415
#44567
#44673
#44998
#45273
#44911
#44726
#45652
#45659
#46257
#46641
#47026
#47166
#48172
#49956
#50979

---------

Co-authored-by: James <lijibing@selectdb.com>
Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>
This was referenced Jul 29, 2025
Merged
Merged
morningman pushed a commit that referenced this pull request Aug 3, 2025
[fix][paimon] Fix Paimon date partition conversion failure (#54043)
1b7f43d 
### What problem does this PR solve?

**Background**: PR [#46641](#46641)
worked around Paimon date partition failures by treating date partitions
as unpartitioned tables.

**Root Cause**: Paimon stores date partition values as integers (e.g.,
`20231225` for `2023-12-25`), but the previous implementation failed to
convert these to Doris-compatible date formats, resulting in:
```
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = date literal [19737] is invalid: null
```
github-actions bot pushed a commit that referenced this pull request Aug 3, 2025
[fix][paimon] Fix Paimon date partition conversion failure (#54043)
d328065 
### What problem does this PR solve?

**Background**: PR [#46641](#46641)
worked around Paimon date partition failures by treating date partitions
as unpartitioned tables.

**Root Cause**: Paimon stores date partition values as integers (e.g.,
`20231225` for `2023-12-25`), but the previous implementation failed to
convert these to Doris-compatible date formats, resulting in:
```
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = date literal [19737] is invalid: null
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Jibing-Li Jibing-Li Jibing-Li approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zddr @Thearas @doris-robot @Jibing-Li @morrySnow @yiguolei