Skip to content

branch-3.1: [opt](iceberg) support create branch/tag for iceberg #51727 #52852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 5 commits into from
Jul 8, 2025
Merged

branch-3.1: [opt](iceberg) support create branch/tag for iceberg #51727 #52852

morrySnow merged 5 commits into from
Jul 8, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Jul 7, 2025

bp #51727

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Jul 7, 2025

run buildall

morningman and others added 5 commits July 7, 2025 15:15
@morningman
add old parser
60e9159
@wuwenchi @morningman
[opt](iceberg) support create branch/tag for iceberg (apache#51727)
93dc0e4 
Related Issue: apache#48285

Problem Summary:

You can now create Iceberg branches and tags through Doris.

```sql

-- branch
alter table tb1 create branch b1;
alter table tb1 create branch if not exists b1;
alter table tb1 create or replace branch b1;
alter table tb1 create or replace branch b1 AS OF VERSION <version>;
alter table tb1 create or replace branch b1 AS OF VERSION <version> RETAIN 1 HOURS;
-- CREATE b1 at snapshot 1234, retain b1 for 30 days, and retain the latest 30 days. The latest 3 snapshot snapshots, and 2 days worth of snapshots.
alter table tb1 CREATE BRANCH b1 AS OF VERSION 1234 RETAIN 30 DAYS WITH SNAPSHOT RETENTION 3 SNAPSHOTS 2 DAYS;

-- tag
alter table tb1 create tag t1;
alter table tb1 create tag if not eists t1;
alter table tb1 create or replace tag t1;
alter table tb1 create or replace tag t1 AS OF VERSION <version>;
-- CREATE t1 at snapshot 1234 and retain it for 1 year.
alter table tb1 create tag t1 AS OF VERSION 1234 RETAIN 365 DAYS
```

The supported time units include: DAYS, HOURS, MINUTES
@morningman
merge
5cb8f39
@morningman
fix keywords
3a06a4c
@morningman
fix
6219bb8
@morningman
Copy link
Contributor Author

morningman commented Jul 7, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-H: Total hot run time: 40001 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6219bb804e12d30dfdddc64449a0cb14940cdc5c, data reload: false

------ Round 1 ----------------------------------
q1	17913	7648	6650	6650
q2	2074	162	172	162
q3	10636	1132	1192	1132
q4	10275	787	826	787
q5	7766	2822	2833	2822
q6	219	134	138	134
q7	991	628	625	625
q8	9371	1952	2014	1952
q9	6580	6360	6371	6360
q10	6965	2274	2297	2274
q11	453	268	261	261
q12	391	216	216	216
q13	17778	2986	2974	2974
q14	242	204	209	204
q15	502	468	468	468
q16	442	382	374	374
q17	960	569	553	553
q18	7407	6708	6731	6708
q19	1366	1015	1007	1007
q20	488	210	196	196
q21	3901	3169	3196	3169
q22	1062	986	973	973
Total cold run time: 107782 ms
Total hot run time: 40001 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6633	6558	6570	6558
q2	319	227	229	227
q3	2938	2916	2932	2916
q4	2083	1874	1849	1849
q5	5721	5750	5697	5697
q6	208	125	131	125
q7	2239	1784	1849	1784
q8	3350	3506	3526	3506
q9	8778	8829	8832	8829
q10	3536	3518	3567	3518
q11	574	499	501	499
q12	839	616	601	601
q13	8827	3243	3175	3175
q14	303	258	273	258
q15	504	458	455	455
q16	470	424	455	424
q17	1856	1652	1621	1621
q18	8202	7768	7708	7708
q19	1686	1624	1587	1587
q20	2085	1818	1859	1818
q21	5232	5015	5162	5015
q22	1138	1047	1042	1042
Total cold run time: 67521 ms
Total hot run time: 59212 ms

@morningman morningman marked this pull request as ready for review July 8, 2025 04:55
@morningman morningman requested a review from morrySnow as a code owner July 8, 2025 04:55
@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-H: Total hot run time: 39823 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6219bb804e12d30dfdddc64449a0cb14940cdc5c, data reload: false

------ Round 1 ----------------------------------
q1	17848	6734	6584	6584
q2	2101	187	172	172
q3	10585	1157	1186	1157
q4	10229	738	745	738
q5	7705	3102	2846	2846
q6	215	129	134	129
q7	1005	623	617	617
q8	9378	2009	2040	2009
q9	6650	6368	6408	6368
q10	6953	2277	2317	2277
q11	467	264	282	264
q12	416	212	212	212
q13	17823	2951	2966	2951
q14	242	209	209	209
q15	499	478	468	468
q16	476	387	374	374
q17	998	599	542	542
q18	7405	6764	6681	6681
q19	1330	1059	977	977
q20	478	208	207	207
q21	3977	3062	3306	3062
q22	1082	1001	979	979
Total cold run time: 107862 ms
Total hot run time: 39823 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6554	6547	6578	6547
q2	323	238	234	234
q3	2931	2950	2976	2950
q4	2006	1851	1827	1827
q5	5687	5724	5729	5724
q6	212	128	130	128
q7	2202	1791	1839	1791
q8	3389	3524	3577	3524
q9	8816	8958	8933	8933
q10	3559	3530	3537	3530
q11	598	499	490	490
q12	812	655	591	591
q13	9284	3183	3221	3183
q14	308	270	277	270
q15	506	482	462	462
q16	489	432	431	431
q17	1867	1628	1605	1605
q18	8224	7931	7937	7931
q19	1643	1521	1494	1494
q20	2047	1844	1860	1844
q21	5229	5037	5043	5037
q22	1135	1009	1030	1009
Total cold run time: 67821 ms
Total hot run time: 59535 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-DS: Total hot run time: 197641 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6219bb804e12d30dfdddc64449a0cb14940cdc5c, data reload: false

query1	1288	945	917	917
query2	6344	1941	1885	1885
query3	10829	4402	4161	4161
query4	33141	23540	23964	23540
query5	4179	489	462	462
query6	297	184	188	184
query7	4011	325	333	325
query8	295	239	231	231
query9	9613	2626	2618	2618
query10	486	270	259	259
query11	18104	15396	15242	15242
query12	162	101	105	101
query13	1553	430	420	420
query14	9460	7082	7238	7082
query15	255	194	189	189
query16	8014	490	486	486
query17	1582	614	634	614
query18	2193	334	314	314
query19	240	168	174	168
query20	123	119	120	119
query21	205	108	110	108
query22	4878	4561	4408	4408
query23	35165	34593	34201	34201
query24	12483	3032	3120	3032
query25	702	428	436	428
query26	1741	190	173	173
query27	2694	355	358	355
query28	7612	2249	2233	2233
query29	1103	442	432	432
query30	252	162	160	160
query31	1096	804	864	804
query32	95	57	55	55
query33	786	305	326	305
query34	1148	508	526	508
query35	890	733	738	733
query36	1129	968	950	950
query37	276	67	67	67
query38	4097	4041	4056	4041
query39	1551	1475	1476	1475
query40	254	105	105	105
query41	47	53	48	48
query42	114	107	101	101
query43	528	483	477	477
query44	1319	819	814	814
query45	184	173	175	173
query46	1190	736	766	736
query47	2038	1910	1930	1910
query48	435	367	370	367
query49	1013	412	414	412
query50	847	435	467	435
query51	7547	7287	7292	7287
query52	101	92	90	90
query53	260	187	186	186
query54	1288	470	477	470
query55	81	89	78	78
query56	266	253	239	239
query57	1388	1201	1223	1201
query58	223	223	214	214
query59	3224	3003	2984	2984
query60	284	262	269	262
query61	114	136	154	136
query62	882	708	696	696
query63	215	203	189	189
query64	4972	684	636	636
query65	3393	3327	3331	3327
query66	1302	302	298	298
query67	16191	15642	15725	15642
query68	4817	601	590	590
query69	424	263	264	263
query70	1169	1113	1101	1101
query71	328	279	255	255
query72	6234	4065	4003	4003
query73	752	352	364	352
query74	10676	8795	8995	8795
query75	3389	2654	2735	2654
query76	2556	1158	1036	1036
query77	391	273	279	273
query78	10576	9764	9607	9607
query79	1685	617	613	613
query80	1107	425	436	425
query81	549	220	228	220
query82	919	88	90	88
query83	234	145	148	145
query84	239	85	78	78
query85	1385	305	304	304
query86	445	286	263	263
query87	4374	4235	4327	4235
query88	3719	2397	2388	2388
query89	423	297	298	297
query90	2051	187	187	187
query91	143	108	104	104
query92	70	48	53	48
query93	2438	568	562	562
query94	975	285	304	285
query95	373	268	267	267
query96	623	286	278	278
query97	3286	3123	3248	3123
query98	221	216	201	201
query99	1506	1303	1307	1303
Total cold run time: 307849 ms
Total hot run time: 197641 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

ClickBench: Total hot run time: 31.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6219bb804e12d30dfdddc64449a0cb14940cdc5c, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.10	0.10
query5	0.51	0.51	0.52
query6	1.13	0.77	0.74
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.50	0.50
query10	0.57	0.56	0.57
query11	0.16	0.12	0.12
query12	0.15	0.13	0.13
query13	0.62	0.60	0.60
query14	0.78	0.81	0.82
query15	0.84	0.85	0.84
query16	0.38	0.37	0.37
query17	0.99	1.06	1.00
query18	0.18	0.18	0.18
query19	1.86	1.82	1.76
query20	0.02	0.01	0.01
query21	15.40	0.66	0.65
query22	3.72	7.39	2.31
query23	18.31	1.48	1.31
query24	2.20	0.23	0.21
query25	0.15	0.09	0.09
query26	0.28	0.19	0.18
query27	0.08	0.08	0.09
query28	13.27	0.62	0.58
query29	12.68	3.34	3.36
query30	0.25	0.05	0.05
query31	2.85	0.41	0.41
query32	3.23	0.49	0.49
query33	2.93	3.07	3.09
query34	17.01	4.53	4.57
query35	4.56	4.54	4.59
query36	0.67	0.48	0.47
query37	0.20	0.16	0.17
query38	0.16	0.15	0.16
query39	0.05	0.04	0.04
query40	0.17	0.12	0.14
query41	0.09	0.05	0.06
query42	0.07	0.05	0.05
query43	0.05	0.05	0.05
Total cold run time: 109.24 s
Total hot run time: 31.34 s

@morrySnow morrySnow changed the title [opt](iceberg) support create branch/tag for iceberg (#51727) branch-3.1: [opt](iceberg) support create branch/tag for iceberg #51727 Jul 8, 2025
@morrySnow morrySnow merged commit 0b19a54 into apache:branch-3.1 Jul 8, 2025
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morningman @hello-stephen @doris-robot @morrySnow @wuwenchi