Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](binlog) Fix add partition record sql #35461

Merged
merged 1 commit into from
May 28, 2024
Merged

[fix](binlog) Fix add partition record sql #35461

merged 1 commit into from
May 28, 2024

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented May 27, 2024

Proposed changes

Issue Number: close #xxx

  1. support adding a temporary partition
  2. remove extra parentheses in the list partition value set
  3. support unpartitioned partition item

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@w41ter
[fix](binlog) Fix add partition record sql
8edf104 
1. support adding temporary partition
2. remove extra parentheses in list partition value set
3. support unpartitioned partition item
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41679 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8edf10444e228aadb82182a76f4ea7d188b20438, data reload: false

------ Round 1 ----------------------------------
q1	17605	4285	4228	4228
q2	2017	185	187	185
q3	10505	1238	1180	1180
q4	10196	945	800	800
q5	7476	2691	2653	2653
q6	223	130	135	130
q7	966	605	611	605
q8	9224	2122	2125	2122
q9	9292	6669	6670	6669
q10	9038	3897	3835	3835
q11	471	239	246	239
q12	405	217	225	217
q13	17199	3287	3230	3230
q14	255	219	246	219
q15	530	493	490	490
q16	515	409	401	401
q17	999	645	645	645
q18	8187	7787	7764	7764
q19	7463	1549	1513	1513
q20	652	319	315	315
q21	5107	3993	3970	3970
q22	361	269	281	269
Total cold run time: 118686 ms
Total hot run time: 41679 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4503	4366	4411	4366
q2	397	276	266	266
q3	3079	2914	2898	2898
q4	1961	1759	1625	1625
q5	5297	5493	5497	5493
q6	209	122	121	121
q7	2228	1852	1801	1801
q8	3231	3379	3405	3379
q9	8599	8641	8690	8641
q10	4082	3836	3728	3728
q11	580	509	488	488
q12	817	629	658	629
q13	16652	3141	3163	3141
q14	304	271	293	271
q15	529	478	479	478
q16	513	435	447	435
q17	1839	1547	1506	1506
q18	7694	7562	7395	7395
q19	1668	1565	1551	1551
q20	2004	1812	1819	1812
q21	10567	4677	4733	4677
q22	566	483	478	478
Total cold run time: 77319 ms
Total hot run time: 55179 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168662 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8edf10444e228aadb82182a76f4ea7d188b20438, data reload: false

query1	913	377	373	373
query2	6438	2406	2409	2406
query3	6649	201	203	201
query4	21660	17359	17324	17324
query5	4131	414	413	413
query6	244	167	153	153
query7	4584	295	287	287
query8	234	183	179	179
query9	8455	2349	2324	2324
query10	459	280	274	274
query11	10503	10132	10123	10123
query12	127	88	85	85
query13	1628	352	350	350
query14	10096	6057	7469	6057
query15	205	166	162	162
query16	7734	264	261	261
query17	1490	532	508	508
query18	1962	273	269	269
query19	200	149	154	149
query20	97	84	91	84
query21	220	127	130	127
query22	4270	3911	4039	3911
query23	33718	33367	33133	33133
query24	11061	2829	2807	2807
query25	559	362	353	353
query26	705	154	152	152
query27	2257	314	318	314
query28	5846	2016	2012	2012
query29	850	613	589	589
query30	217	149	150	149
query31	961	759	750	750
query32	93	52	54	52
query33	738	277	278	277
query34	915	482	489	482
query35	733	618	632	618
query36	1089	897	893	893
query37	107	66	69	66
query38	2893	2817	2746	2746
query39	856	795	812	795
query40	199	130	128	128
query41	77	51	49	49
query42	108	99	92	92
query43	578	565	542	542
query44	1176	730	744	730
query45	175	158	165	158
query46	1079	731	704	704
query47	1861	1792	1802	1792
query48	378	301	289	289
query49	837	375	378	375
query50	762	403	404	403
query51	6945	6731	6822	6731
query52	99	93	84	84
query53	350	294	285	285
query54	891	426	423	423
query55	76	72	72	72
query56	265	244	241	241
query57	1108	1083	1022	1022
query58	256	214	208	208
query59	3361	3381	3254	3254
query60	271	261	258	258
query61	84	85	88	85
query62	630	446	473	446
query63	311	279	287	279
query64	8451	2248	1671	1671
query65	3173	3106	3130	3106
query66	775	328	326	326
query67	15151	14731	14679	14679
query68	4522	520	526	520
query69	473	260	262	260
query70	1167	1067	1142	1067
query71	433	267	270	267
query72	7126	5676	2713	2713
query73	716	319	321	319
query74	5996	5636	5640	5636
query75	3442	2656	2625	2625
query76	3014	993	919	919
query77	600	315	270	270
query78	10235	9838	9855	9838
query79	2288	509	504	504
query80	1284	437	415	415
query81	485	220	225	220
query82	668	95	90	90
query83	201	167	165	165
query84	271	87	87	87
query85	1206	265	256	256
query86	396	303	302	302
query87	3310	3208	3113	3113
query88	3648	2410	2424	2410
query89	478	397	381	381
query90	2072	185	184	184
query91	119	96	99	96
query92	58	52	49	49
query93	1906	502	486	486
query94	1149	192	192	192
query95	415	311	320	311
query96	581	273	270	270
query97	3300	2994	3040	2994
query98	246	218	219	218
query99	1188	859	845	845
Total cold run time: 268317 ms
Total hot run time: 168662 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.28 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8edf10444e228aadb82182a76f4ea7d188b20438, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.24	0.07	0.06
query4	1.64	0.06	0.07
query5	0.50	0.50	0.51
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.05
query9	0.53	0.47	0.49
query10	0.54	0.54	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.60
query14	0.77	0.77	0.77
query15	0.83	0.81	0.80
query16	0.37	0.35	0.34
query17	1.00	0.94	0.94
query18	0.22	0.23	0.26
query19	1.75	1.69	1.73
query20	0.01	0.01	0.02
query21	15.57	0.67	0.63
query22	4.10	7.64	1.69
query23	18.35	1.37	1.28
query24	2.04	0.22	0.21
query25	0.14	0.09	0.07
query26	0.26	0.17	0.18
query27	0.08	0.07	0.07
query28	13.32	1.01	0.99
query29	13.80	3.29	3.29
query30	0.24	0.06	0.06
query31	2.91	0.38	0.38
query32	3.24	0.47	0.47
query33	2.90	3.01	2.91
query34	17.10	4.44	4.46
query35	4.49	4.48	4.66
query36	0.64	0.46	0.47
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.03
query40	0.16	0.16	0.15
query41	0.09	0.04	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.52 s
Total hot run time: 30.28 s

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes May 28, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

dataroaring
dataroaring approved these changes May 28, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit f39bd11 into apache:master May 28, 2024
28 of 31 checks passed
@w41ter w41ter deleted the fix_add_partition_record branch May 28, 2024 07:58
yiguolei pushed a commit that referenced this pull request May 28, 2024
@w41ter @Doris-Extras
[fix](binlog) Fix add partition record sql (#35461)
efdce7e 
1. support adding a temporary partition
2. remove extra parentheses in the list partition value set
3. support unpartitioned partition item
w41ter added a commit to w41ter/incubator-doris that referenced this pull request May 29, 2024
@w41ter
[fix](binlog) Fix add partition record sql
6994ce0 
1. support adding a temporary partition
2. remove extra parentheses in the list partition value set
3. support unpartitioned partition item

Cherry-pick apache#35461
@w41ter w41ter mentioned this pull request May 29, 2024
Merged
dataroaring pushed a commit that referenced this pull request May 29, 2024
@w41ter
[fix](binlog) Fix add partition record sql (#35564)
3f3d834 
1. support adding a temporary partition
2. remove extra parentheses in the list partition value set
3. support unpartitioned partition item

Cherry-pick #35461
dataroaring pushed a commit that referenced this pull request May 31, 2024
@w41ter @dataroaring
[fix](binlog) Fix add partition record sql (#35461)
95e87c4 
1. support adding a temporary partition
2. remove extra parentheses in the list partition value set
3. support unpartitioned partition item
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 4, 2024
@w41ter
[fix](fe) Fix the sql of AddPartitionRecord
5c00132 
The range field is accidentally compared to DUMMY_ITEM.

It was introduced by apache#35461.
@w41ter w41ter mentioned this pull request Jul 4, 2024
Merged
w41ter added a commit that referenced this pull request Jul 5, 2024
@w41ter
[fix](fe) Fix the sql of AddPartitionRecord (#37295)
41501d9 
The range field is accidentally compared to DUMMY_ITEM.

It was introduced by #35461.
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 5, 2024
@w41ter
[fix](fe) Fix the sql of AddPartitionRecord
35d54fe 
Cherry-pick apache#37295.

The range field is accidentally compared to DUMMY_ITEM.

It was introduced by apache#35461.
This was referenced Jul 5, 2024
Closed
Merged
dataroaring pushed a commit that referenced this pull request Jul 5, 2024
@w41ter
[fix](fe) Fix the sql of AddPartitionRecord (#37341)
7ce4a42 
Cherry-pick #37295

The range field is accidentally compared to DUMMY_ITEM.

It was introduced by #35461.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@w41ter @dataroaring
[fix](fe) Fix the sql of AddPartitionRecord (#37295)
06110cc 
The range field is accidentally compared to DUMMY_ITEM.

It was introduced by #35461.
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@w41ter
[fix](binlog) Fix add partition record sql (apache#35564)
3229f7c 
1. support adding a temporary partition
2. remove extra parentheses in the list partition value set
3. support unpartitioned partition item

Cherry-pick apache#35461
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@w41ter @doris-robot @dataroaring @gavinchou