Skip to content

[enhance](mtmv)MTMV allow paimon table has multi partition keys #45652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 23 commits into from
Dec 23, 2024
Merged

[enhance](mtmv)MTMV allow paimon table has multi partition keys #45652

morrySnow merged 23 commits into from
Dec 23, 2024

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented Dec 19, 2024 •
edited
Loading

What problem does this PR solve?

  • MTMV allow paimon table has multi partition keys
  • add case

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
MTMV allow paimon table has multi partition keys

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 19, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Dec 19, 2024

run buildall

morrySnow
morrySnow previously approved these changes Dec 19, 2024
View reviewed changes
@zddr
Copy link
Contributor Author

zddr commented Dec 19, 2024

run buildall

1 similar comment
@zddr
Copy link
Contributor Author

zddr commented Dec 19, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Dec 19, 2024

PR approved by anyone and no changes requested.

@zddr
Copy link
Contributor Author

zddr commented Dec 23, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Dec 23, 2024

TPC-H: Total hot run time: 40428 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2064acd8b1586b0c714e1e1c0529b50b0b2685ee, data reload: false

------ Round 1 ----------------------------------
q1	17637	7479	7425	7425
q2	2045	183	178	178
q3	10681	1236	1163	1163
q4	10918	698	737	698
q5	7607	2709	2723	2709
q6	240	151	151	151
q7	985	641	630	630
q8	9491	1886	1927	1886
q9	6775	6468	6563	6468
q10	7174	2302	2349	2302
q11	499	259	274	259
q12	465	225	226	225
q13	18807	3002	3017	3002
q14	253	218	213	213
q15	585	518	494	494
q16	1940	587	596	587
q17	1178	531	524	524
q18	7263	6713	6740	6713
q19	1346	966	1039	966
q20	454	191	181	181
q21	4097	3417	3331	3331
q22	379	328	323	323
Total cold run time: 110819 ms
Total hot run time: 40428 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7272	7169	7190	7169
q2	338	231	234	231
q3	3051	2962	2980	2962
q4	2166	1875	1818	1818
q5	5675	5685	5765	5685
q6	237	153	152	152
q7	2265	1895	1826	1826
q8	3365	3542	3470	3470
q9	8835	8898	8985	8898
q10	3592	3588	3553	3553
q11	606	513	496	496
q12	814	610	627	610
q13	13420	3155	3141	3141
q14	305	289	271	271
q15	575	524	517	517
q16	693	679	648	648
q17	1877	1603	1613	1603
q18	8448	7840	7642	7642
q19	1739	1552	1536	1536
q20	2123	1884	1901	1884
q21	5687	5524	5537	5524
q22	666	627	634	627
Total cold run time: 73749 ms
Total hot run time: 60263 ms

@doris-robot
Copy link

doris-robot commented Dec 23, 2024

TPC-DS: Total hot run time: 197074 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2064acd8b1586b0c714e1e1c0529b50b0b2685ee, data reload: false

query1	1310	1015	923	923
query2	6241	2456	2449	2449
query3	11005	4665	4698	4665
query4	33266	23485	23607	23485
query5	4205	486	449	449
query6	316	206	190	190
query7	4015	314	308	308
query8	310	237	243	237
query9	9320	2706	2708	2706
query10	465	260	252	252
query11	18157	15041	15172	15041
query12	154	115	102	102
query13	1580	428	413	413
query14	9853	7106	7347	7106
query15	262	187	214	187
query16	8186	446	453	446
query17	1554	583	624	583
query18	2159	337	311	311
query19	362	160	155	155
query20	123	115	114	114
query21	206	109	108	108
query22	4885	4702	4435	4435
query23	35187	33678	33618	33618
query24	10552	2464	2521	2464
query25	623	428	448	428
query26	1260	166	170	166
query27	2350	353	353	353
query28	7461	2472	2493	2472
query29	831	426	432	426
query30	226	153	149	149
query31	1064	870	850	850
query32	99	55	55	55
query33	766	307	301	301
query34	1215	532	511	511
query35	920	766	799	766
query36	1155	962	1002	962
query37	139	83	79	79
query38	4548	4056	4309	4056
query39	1489	1484	1467	1467
query40	223	113	104	104
query41	44	45	41	41
query42	119	109	108	108
query43	552	513	498	498
query44	1255	874	853	853
query45	197	165	172	165
query46	1215	753	757	753
query47	2082	1934	1897	1897
query48	433	338	325	325
query49	925	407	400	400
query50	848	398	408	398
query51	7313	7227	7174	7174
query52	108	95	95	95
query53	258	196	189	189
query54	1273	436	446	436
query55	83	79	83	79
query56	273	251	236	236
query57	1325	1174	1158	1158
query58	245	235	219	219
query59	3383	3258	3126	3126
query60	279	245	245	245
query61	103	105	110	105
query62	876	674	683	674
query63	220	201	212	201
query64	4139	794	787	787
query65	3282	3235	3227	3227
query66	825	384	311	311
query67	16149	15715	15485	15485
query68	5489	566	561	561
query69	517	262	252	252
query70	1201	1222	1158	1158
query71	480	252	250	250
query72	7119	4178	4329	4178
query73	805	361	375	361
query74	10216	8903	8855	8855
query75	3571	2699	2678	2678
query76	3929	1097	1108	1097
query77	568	269	281	269
query78	10320	9492	9343	9343
query79	2830	631	681	631
query80	1407	448	434	434
query81	518	234	224	224
query82	679	123	124	123
query83	207	143	143	143
query84	295	76	73	73
query85	1360	314	329	314
query86	408	305	296	296
query87	4607	4449	4395	4395
query88	4460	2193	2193	2193
query89	447	296	297	296
query90	2072	196	192	192
query91	146	108	104	104
query92	71	53	52	52
query93	3807	557	545	545
query94	859	293	279	279
query95	353	254	256	254
query96	644	291	280	280
query97	2876	2684	2666	2666
query98	228	195	209	195
query99	1672	1306	1319	1306
Total cold run time: 310007 ms
Total hot run time: 197074 ms

@doris-robot
Copy link

doris-robot commented Dec 23, 2024

ClickBench: Total hot run time: 33.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2064acd8b1586b0c714e1e1c0529b50b0b2685ee, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.63	0.11	0.10
query5	0.43	0.41	0.42
query6	1.15	0.65	0.68
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.52	0.52
query10	0.54	0.57	0.56
query11	0.14	0.11	0.11
query12	0.15	0.12	0.10
query13	0.60	0.61	0.58
query14	2.80	2.75	2.86
query15	0.91	0.82	0.83
query16	0.40	0.37	0.37
query17	1.04	0.99	1.05
query18	0.23	0.21	0.21
query19	1.93	1.89	2.01
query20	0.01	0.01	0.01
query21	15.37	0.57	0.60
query22	2.55	2.26	2.07
query23	16.94	1.06	0.69
query24	3.09	1.79	1.38
query25	0.23	0.27	0.08
query26	0.43	0.14	0.14
query27	0.04	0.05	0.05
query28	9.57	1.14	1.08
query29	12.63	3.23	3.25
query30	0.26	0.07	0.06
query31	2.86	0.40	0.38
query32	3.27	0.47	0.46
query33	3.12	3.13	3.13
query34	17.01	4.44	4.48
query35	4.53	4.44	4.49
query36	0.66	0.48	0.51
query37	0.10	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.18	0.14	0.15
query41	0.09	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.02 s
Total hot run time: 33.25 s

@zddr
Copy link
Contributor Author

zddr commented Dec 23, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Dec 23, 2024

TPC-H: Total hot run time: 39820 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ace187dbb1df4af156d552f5092c847b1d3b0ff1, data reload: false

------ Round 1 ----------------------------------
q1	17584	8125	7325	7325
q2	2049	180	184	180
q3	10607	1094	1186	1094
q4	10570	794	794	794
q5	7625	2729	2707	2707
q6	243	148	151	148
q7	991	638	607	607
q8	9266	1822	1980	1822
q9	6588	6399	6413	6399
q10	7023	2311	2308	2308
q11	471	268	255	255
q12	436	227	218	218
q13	17776	2910	2965	2910
q14	251	211	211	211
q15	563	506	497	497
q16	663	602	588	588
q17	996	604	530	530
q18	7287	6601	6712	6601
q19	1365	1079	1004	1004
q20	481	186	184	184
q21	4005	3233	3128	3128
q22	369	314	310	310
Total cold run time: 107209 ms
Total hot run time: 39820 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7199	7217	7217	7217
q2	334	231	239	231
q3	2920	2996	2936	2936
q4	2104	1864	1933	1864
q5	5657	5699	5643	5643
q6	234	150	153	150
q7	2226	1803	1861	1803
q8	3397	3587	3546	3546
q9	8831	9079	8979	8979
q10	3598	3539	3558	3539
q11	609	525	513	513
q12	812	606	615	606
q13	13384	3190	3128	3128
q14	308	272	282	272
q15	555	505	520	505
q16	719	661	654	654
q17	1866	1628	1644	1628
q18	8351	7833	7882	7833
q19	1734	1579	1518	1518
q20	2107	1862	1874	1862
q21	5553	5439	5480	5439
q22	633	585	575	575
Total cold run time: 73131 ms
Total hot run time: 60441 ms

@doris-robot
Copy link

doris-robot commented Dec 23, 2024

TPC-DS: Total hot run time: 197147 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ace187dbb1df4af156d552f5092c847b1d3b0ff1, data reload: false

query1	1302	933	962	933
query2	6248	2324	2415	2324
query3	11111	4733	4915	4733
query4	32952	23501	23465	23465
query5	3603	459	437	437
query6	288	197	195	195
query7	3982	301	306	301
query8	306	246	232	232
query9	9431	2774	2765	2765
query10	468	261	253	253
query11	17802	15337	15101	15101
query12	163	105	106	105
query13	1597	453	419	419
query14	9387	6762	8355	6762
query15	447	206	219	206
query16	8783	462	492	462
query17	1871	574	591	574
query18	2155	311	347	311
query19	373	154	175	154
query20	120	121	111	111
query21	216	103	106	103
query22	4911	4747	4652	4652
query23	34781	33799	33791	33791
query24	10523	2562	2552	2552
query25	649	416	401	401
query26	1282	152	158	152
query27	2633	338	333	333
query28	6908	2494	2477	2477
query29	878	424	419	419
query30	237	148	145	145
query31	1040	847	803	803
query32	92	56	60	56
query33	774	308	307	307
query34	982	544	522	522
query35	940	771	774	771
query36	1109	956	982	956
query37	152	76	79	76
query38	4322	4338	4284	4284
query39	1485	1458	1471	1458
query40	209	98	98	98
query41	47	42	44	42
query42	110	97	96	96
query43	562	513	507	507
query44	1254	863	837	837
query45	204	174	168	168
query46	1230	719	730	719
query47	2062	1940	1915	1915
query48	420	324	330	324
query49	1049	428	391	391
query50	819	422	398	398
query51	7419	7220	7243	7220
query52	103	95	93	93
query53	259	183	189	183
query54	1262	420	423	420
query55	96	79	121	79
query56	266	256	233	233
query57	1300	1212	1167	1167
query58	232	237	225	225
query59	3481	3245	3332	3245
query60	300	249	257	249
query61	122	104	110	104
query62	908	706	701	701
query63	225	197	197	197
query64	3832	690	714	690
query65	3278	3149	3214	3149
query66	848	303	302	302
query67	15950	15732	15462	15462
query68	4783	568	576	568
query69	431	250	254	250
query70	1205	1152	1140	1140
query71	409	257	272	257
query72	6960	4034	4117	4034
query73	780	366	362	362
query74	10463	8904	9095	8904
query75	3442	2632	2657	2632
query76	2742	1103	1059	1059
query77	384	269	279	269
query78	10435	9445	9677	9445
query79	1844	614	604	604
query80	867	432	448	432
query81	557	240	232	232
query82	677	119	125	119
query83	270	148	142	142
query84	228	72	74	72
query85	1301	311	302	302
query86	462	279	296	279
query87	4742	4344	4395	4344
query88	3765	2255	2223	2223
query89	435	302	296	296
query90	2011	190	189	189
query91	145	107	102	102
query92	67	49	51	49
query93	2143	553	554	553
query94	920	289	280	280
query95	356	256	258	256
query96	610	276	284	276
query97	2863	2658	2689	2658
query98	219	197	210	197
query99	1563	1307	1317	1307
Total cold run time: 302229 ms
Total hot run time: 197147 ms

@doris-robot
Copy link

doris-robot commented Dec 23, 2024

ClickBench: Total hot run time: 32.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ace187dbb1df4af156d552f5092c847b1d3b0ff1, data reload: false

query1	0.03	0.04	0.03
query2	0.10	0.03	0.03
query3	0.24	0.07	0.07
query4	1.60	0.11	0.11
query5	0.43	0.41	0.43
query6	1.17	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.57	0.58	0.54
query11	0.14	0.10	0.10
query12	0.14	0.10	0.11
query13	0.60	0.60	0.60
query14	2.77	2.72	2.84
query15	0.89	0.82	0.83
query16	0.38	0.37	0.39
query17	1.07	1.04	0.98
query18	0.23	0.21	0.21
query19	1.98	1.80	1.99
query20	0.02	0.01	0.01
query21	15.36	0.57	0.58
query22	2.63	1.95	2.54
query23	16.91	0.90	0.91
query24	3.27	0.88	1.69
query25	0.42	0.26	0.05
query26	0.31	0.14	0.15
query27	0.04	0.03	0.05
query28	10.02	1.12	1.07
query29	12.58	3.25	3.24
query30	0.25	0.06	0.06
query31	2.86	0.40	0.38
query32	3.23	0.46	0.47
query33	3.09	3.16	3.04
query34	17.20	4.53	4.55
query35	4.52	4.48	4.56
query36	0.67	0.48	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.17	0.12	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.87 s
Total hot run time: 32.67 s

@zddr
Copy link
Contributor Author

zddr commented Dec 23, 2024

run p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 23, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Dec 23, 2024

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit fa97350 into apache:master Dec 23, 2024
22 of 23 checks passed
zddr added a commit to zddr/incubator-doris that referenced this pull request Jun 19, 2025
@zddr
[enhance](mtmv)MTMV allow paimon table has multi partition keys (apac…
1c87279 
…he#45652)

- MTMV allow paimon table has multi partition keys
- add case
zddr added a commit to zddr/incubator-doris that referenced this pull request Jun 19, 2025
@zddr
[enhance](mtmv)MTMV allow paimon table has multi partition keys (apac…
9c27a90 
…he#45652)

- MTMV allow paimon table has multi partition keys
- add case
@zddr zddr mentioned this pull request Jun 20, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 23, 2025
@zddr @Jibing-Li @BePPPower
branch-3.1: [enhance](mtmv)MTMV support iceberg/hudi/paimon (#52040)
4234a27 
Cherry-pick from
#43959
#44419
#44415
#44567
#44673
#44998
#45273
#44911
#44726
#45652
#45659
#46257
#46641
#47026
#47166
#48172
#49956
#50979

---------

Co-authored-by: James <lijibing@selectdb.com>
Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

+1 more reviewer

@seawinde seawinde seawinde approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zddr @Thearas @doris-robot @morrySnow @seawinde