[regression](mtmv)Support show iceberg external table partition. Add more test case for iceberg mtmv. #46257

Jibing-Li · 2025-01-02T03:29:07Z

What problem does this PR solve?

Support show iceberg external table partition.
We convert iceberg partition to doris range partition in IcebergExternalTable. This PR add show partition function for IcebergExternalTable, this make it possible to add regression test.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-01-02T03:29:11Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Jibing-Li · 2025-01-02T08:00:33Z

run buildall

doris-robot · 2025-01-02T09:13:06Z

TPC-H: Total hot run time: 33079 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f43bd9e97158c549e91aaaff59a90f3adc0b416, data reload: false

------ Round 1 ----------------------------------
q1	17604	6255	6079	6079
q2	2063	313	173	173
q3	10421	1247	781	781
q4	10289	866	443	443
q5	8947	2227	2029	2029
q6	216	183	151	151
q7	899	760	607	607
q8	9238	1428	1228	1228
q9	5417	5055	4910	4910
q10	6794	2331	1860	1860
q11	474	288	264	264
q12	367	384	231	231
q13	17790	3592	2990	2990
q14	233	233	242	233
q15	560	509	501	501
q16	651	629	593	593
q17	590	878	350	350
q18	7052	6606	6507	6507
q19	1890	966	566	566
q20	311	327	190	190
q21	3196	2416	2081	2081
q22	364	350	312	312
Total cold run time: 105366 ms
Total hot run time: 33079 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6257	6250	6238	6238
q2	241	329	233	233
q3	2286	2702	2293	2293
q4	1401	1859	1368	1368
q5	4355	4831	5013	4831
q6	182	180	142	142
q7	2086	1958	1891	1891
q8	2686	2863	2761	2761
q9	7323	7316	7345	7316
q10	3140	3297	2825	2825
q11	576	521	520	520
q12	683	769	592	592
q13	3418	3759	3184	3184
q14	310	332	282	282
q15	571	526	496	496
q16	648	733	653	653
q17	1277	1759	1257	1257
q18	7564	7639	7525	7525
q19	876	1241	1138	1138
q20	2006	2031	1913	1913
q21	5828	5481	5111	5111
q22	604	635	585	585
Total cold run time: 54318 ms
Total hot run time: 53154 ms

doris-robot · 2025-01-02T09:25:26Z

TPC-DS: Total hot run time: 196829 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f43bd9e97158c549e91aaaff59a90f3adc0b416, data reload: false

query1	1300	961	907	907
query2	6402	2480	2349	2349
query3	10995	4968	4692	4692
query4	33162	23805	23339	23339
query5	4088	617	466	466
query6	282	204	194	194
query7	3997	495	306	306
query8	307	257	236	236
query9	9370	2643	2640	2640
query10	475	316	247	247
query11	17915	15475	15035	15035
query12	166	112	107	107
query13	1582	560	408	408
query14	10855	7004	7870	7004
query15	263	211	197	197
query16	8192	630	501	501
query17	1565	788	564	564
query18	2109	422	335	335
query19	232	181	154	154
query20	122	114	120	114
query21	206	126	107	107
query22	4646	4804	4649	4649
query23	34504	33788	33396	33396
query24	6393	2331	2317	2317
query25	500	474	392	392
query26	775	275	151	151
query27	2258	470	373	373
query28	5385	2456	2456	2456
query29	629	565	421	421
query30	222	184	151	151
query31	1009	937	899	899
query32	75	58	58	58
query33	464	373	292	292
query34	777	845	522	522
query35	831	845	764	764
query36	1050	1061	991	991
query37	123	106	71	71
query38	4281	4338	4275	4275
query39	1508	1471	1473	1471
query40	212	120	106	106
query41	49	46	43	43
query42	117	99	103	99
query43	552	548	526	526
query44	1412	818	814	814
query45	183	182	171	171
query46	942	1093	663	663
query47	2005	2025	1940	1940
query48	400	412	325	325
query49	727	480	394	394
query50	663	703	400	400
query51	7307	7301	7289	7289
query52	109	102	96	96
query53	228	260	188	188
query54	491	493	436	436
query55	85	82	81	81
query56	269	262	271	262
query57	1284	1238	1167	1167
query58	240	257	239	239
query59	3384	3639	3325	3325
query60	271	263	254	254
query61	112	112	113	112
query62	860	824	758	758
query63	220	196	191	191
query64	3015	1069	676	676
query65	3299	3265	3286	3265
query66	784	423	329	329
query67	16448	15835	15473	15473
query68	8020	707	592	592
query69	479	312	256	256
query70	1209	1183	1124	1124
query71	392	314	257	257
query72	6526	3882	3835	3835
query73	636	744	359	359
query74	10470	9032	8991	8991
query75	3422	3155	2633	2633
query76	3351	1342	786	786
query77	827	353	272	272
query78	10113	10111	9476	9476
query79	3624	813	583	583
query80	660	512	423	423
query81	496	266	229	229
query82	623	145	119	119
query83	157	160	148	148
query84	248	88	72	72
query85	784	360	359	359
query86	406	324	287	287
query87	4577	4525	4429	4429
query88	4774	2156	2149	2149
query89	427	337	296	296
query90	1828	198	187	187
query91	148	133	103	103
query92	68	54	50	50
query93	2235	873	535	535
query94	687	391	294	294
query95	336	269	261	261
query96	479	610	278	278
query97	2726	2830	2639	2639
query98	229	204	194	194
query99	1804	1586	1440	1440
Total cold run time: 297444 ms
Total hot run time: 196829 ms

doris-robot · 2025-01-02T09:30:48Z

ClickBench: Total hot run time: 30.79 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9f43bd9e97158c549e91aaaff59a90f3adc0b416, data reload: false

query1	0.06	0.04	0.04
query2	0.07	0.03	0.04
query3	0.23	0.07	0.06
query4	1.61	0.10	0.11
query5	0.41	0.43	0.41
query6	1.13	0.65	0.65
query7	0.03	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.53
query10	0.56	0.56	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.72	2.87	2.71
query15	0.90	0.81	0.82
query16	0.39	0.37	0.37
query17	1.04	1.04	1.02
query18	0.23	0.21	0.21
query19	1.83	1.98	1.79
query20	0.01	0.01	0.01
query21	15.37	1.00	0.60
query22	0.76	0.79	0.73
query23	15.17	1.48	0.58
query24	2.96	1.70	0.64
query25	0.19	0.10	0.07
query26	0.23	0.15	0.14
query27	0.05	0.05	0.05
query28	13.69	1.51	1.04
query29	12.57	3.84	3.22
query30	0.25	0.09	0.06
query31	2.82	0.62	0.37
query32	3.22	0.54	0.45
query33	3.12	3.20	3.03
query34	16.64	5.07	4.44
query35	4.51	4.45	4.46
query36	0.67	0.51	0.48
query37	0.10	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.43 s
Total hot run time: 30.79 s

zddr · 2025-01-06T02:35:04Z

fe/fe-core/src/main/java/org/apache/doris/analysis/ShowPartitionsStmt.java

            for (String col : result.getColumnNames()) {
                builder.addColumn(new Column(col, ScalarType.createVarchar(30)));
            }
+        } else if (catalog instanceof IcebergExternalCatalog) {


need also support in PartitionsTableValuedFunction?

How about support TVF in a separate PR?

github-actions · 2025-01-06T02:42:43Z

PR approved by anyone and no changes requested.

github-actions · 2025-01-06T07:54:07Z

PR approved by at least one committer and no changes requested.

…more test case for iceberg mtmv. (apache#46257) ### What problem does this PR solve? Support show iceberg external table partition. We convert iceberg partition to doris range partition in IcebergExternalTable. This PR add show partition function for IcebergExternalTable, this make it possible to add regression test. Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None

Cherry-pick from #43959 #44419 #44415 #44567 #44673 #44998 #45273 #44911 #44726 #45652 #45659 #46257 #46641 #47026 #47166 #48172 #49956 #50979 --------- Co-authored-by: James <lijibing@selectdb.com> Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>

Jibing-Li force-pushed the show_iceberg_partition branch 2 times, most recently from 94e41de to 09d877f Compare January 2, 2025 07:54

Support show iceberg external table partition.

9f43bd9

Jibing-Li force-pushed the show_iceberg_partition branch from 09d877f to 9f43bd9 Compare January 2, 2025 07:57

Jibing-Li changed the title ~~[improvement](mtmv)Support show iceberg external table partition.~~ [regression](mtmv)Support show iceberg external table partition. Add more test case for iceberg mtmv. Jan 2, 2025

Jibing-Li marked this pull request as ready for review January 2, 2025 08:00

zddr reviewed Jan 6, 2025

View reviewed changes

zddr approved these changes Jan 6, 2025

View reviewed changes

github-actions bot added the reviewed label Jan 6, 2025

morrySnow approved these changes Jan 6, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 6, 2025

Jibing-Li merged commit 624bbff into apache:master Jan 6, 2025
33 of 34 checks passed

Jibing-Li deleted the show_iceberg_partition branch January 6, 2025 07:55

zddr mentioned this pull request Jun 20, 2025

branch-3.1:[enhance](mtmv)MTMV support iceberg/hudi/paimon #52040

Merged

morrySnow added the dev/3.1.0-merged label Jun 23, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[regression](mtmv)Support show iceberg external table partition. Add more test case for iceberg mtmv. #46257

[regression](mtmv)Support show iceberg external table partition. Add more test case for iceberg mtmv. #46257

Uh oh!

Jibing-Li commented Jan 2, 2025 •

edited

Loading

Uh oh!

Thearas commented Jan 2, 2025

Uh oh!

Jibing-Li commented Jan 2, 2025

Uh oh!

doris-robot commented Jan 2, 2025

Uh oh!

doris-robot commented Jan 2, 2025

Uh oh!

doris-robot commented Jan 2, 2025

Uh oh!

zddr Jan 6, 2025

Uh oh!

Jibing-Li Jan 6, 2025

Uh oh!

github-actions bot commented Jan 6, 2025

Uh oh!

github-actions bot commented Jan 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[regression](mtmv)Support show iceberg external table partition. Add more test case for iceberg mtmv. #46257

[regression](mtmv)Support show iceberg external table partition. Add more test case for iceberg mtmv. #46257

Uh oh!

Conversation

Jibing-Li commented Jan 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jan 2, 2025

Uh oh!

Jibing-Li commented Jan 2, 2025

Uh oh!

doris-robot commented Jan 2, 2025

Uh oh!

doris-robot commented Jan 2, 2025

Uh oh!

doris-robot commented Jan 2, 2025

Uh oh!

zddr Jan 6, 2025

Choose a reason for hiding this comment

Uh oh!

Jibing-Li Jan 6, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 6, 2025

Uh oh!

github-actions bot commented Jan 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Jibing-Li commented Jan 2, 2025 •

edited

Loading