[Feature](partition) Support insert overwrite auto detect partition #32156

zclllyybb · 2024-03-13T02:02:08Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-03-13T02:02:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

be/src/exec/tablet_info.cpp

be/src/vec/sink/vrow_distribution.cpp

zclllyybb · 2024-03-17T21:42:22Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/vec/sink/vrow_distribution.h

doris-robot · 2024-03-17T22:49:32Z

TPC-H: Total hot run time: 38162 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b45cc829b0b40aabf10ba42ada426d631babb9eb, data reload: false

------ Round 1 ----------------------------------
q1	17669	4205	4131	4131
q2	2015	152	140	140
q3	10599	1098	894	894
q4	7775	747	724	724
q5	7481	2607	2678	2607
q6	182	125	124	124
q7	1204	802	799	799
q8	9369	2030	2012	2012
q9	7193	6429	6430	6429
q10	8511	3492	3606	3492
q11	443	234	226	226
q12	630	288	307	288
q13	17782	2893	2814	2814
q14	271	246	246	246
q15	500	457	454	454
q16	496	390	388	388
q17	964	615	521	521
q18	7253	6462	6423	6423
q19	4530	1407	1457	1407
q20	552	287	277	277
q21	6272	3472	3491	3472
q22	351	294	347	294
Total cold run time: 112042 ms
Total hot run time: 38162 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4294	4064	4048	4048
q2	323	229	217	217
q3	2982	2831	2811	2811
q4	1860	1554	1542	1542
q5	5280	5220	5234	5220
q6	193	116	116	116
q7	2228	1859	1907	1859
q8	3167	3318	3280	3280
q9	8584	8546	8582	8546
q10	3714	3672	3649	3649
q11	538	440	439	439
q12	739	550	553	550
q13	16889	2844	2863	2844
q14	276	251	256	251
q15	485	446	449	446
q16	459	388	408	388
q17	1740	1459	1465	1459
q18	7518	7292	7112	7112
q19	1634	1511	1518	1511
q20	1891	1718	1711	1711
q21	4857	4766	4657	4657
q22	556	454	461	454
Total cold run time: 70207 ms
Total hot run time: 53110 ms

doris-robot · 2024-03-17T23:13:42Z

TeamCity be ut coverage result:
Function Coverage: 34.98% (8614/24629)
Line Coverage: 26.65% (69847/262089)
Region Coverage: 25.94% (36266/139811)
Branch Coverage: 22.87% (18519/80970)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b45cc829b0b40aabf10ba42ada426d631babb9eb_b45cc829b0b40aabf10ba42ada426d631babb9eb/report/index.html

be/src/exec/tablet_info.cpp

zclllyybb · 2024-03-18T05:02:02Z

run buildall

be/src/exec/tablet_info.h

doris-robot · 2024-03-18T06:32:56Z

TeamCity be ut coverage result:
Function Coverage: 35.24% (8702/24693)
Line Coverage: 27.06% (71177/263050)
Region Coverage: 26.31% (36926/140352)
Branch Coverage: 23.22% (18882/81304)
Coverage Report: http://coverage.selectdb-in.cc/coverage/43091c79ca66644b24a4fc329713ab70ceea20ec_43091c79ca66644b24a4fc329713ab70ceea20ec/report/index.html

zclllyybb · 2024-03-18T09:10:13Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/analysis/PartitionNames.java

doris-robot · 2024-03-18T09:52:47Z

TeamCity be ut coverage result:
Function Coverage: 35.24% (8703/24693)
Line Coverage: 27.06% (71189/263061)
Region Coverage: 26.31% (36932/140367)
Branch Coverage: 23.23% (18889/81306)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3ba93a1d1a4837258a7b4959c5d0472a42f7a1a8_3ba93a1d1a4837258a7b4959c5d0472a42f7a1a8/report/index.html

.../docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT-OVERWRITE.md

fe/fe-core/src/main/java/org/apache/doris/analysis/PartitionNames.java

fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java

fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundTableSink.java

fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundTableSinkCreator.java

fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java

zclllyybb · 2024-03-18T12:09:25Z

run buildall

doris-robot · 2024-03-18T12:53:08Z

TeamCity be ut coverage result:
Function Coverage: 35.24% (8702/24697)
Line Coverage: 27.06% (71195/263082)
Region Coverage: 26.30% (36925/140382)
Branch Coverage: 23.24% (18894/81314)
Coverage Report: http://coverage.selectdb-in.cc/coverage/443c9f24ce4cedc8f8b12b79004f64939af74254_443c9f24ce4cedc8f8b12b79004f64939af74254/report/index.html

fe/fe-core/src/main/java/org/apache/doris/insertoverwrite/InsertOverwriteManager.java

github-actions · 2024-03-20T03:35:41Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-20T03:35:43Z

PR approved by anyone and no changes requested.

HappenLee

LGTM

…pache#32156)

github-actions bot added the meta-change label Mar 13, 2024

HappenLee reviewed Mar 13, 2024

View reviewed changes

be/src/exec/tablet_info.cpp Outdated Show resolved Hide resolved

HappenLee reviewed Mar 13, 2024

View reviewed changes

be/src/vec/sink/vrow_distribution.cpp Show resolved Hide resolved

zclllyybb force-pushed the overwrite branch 2 times, most recently from 5e44236 to a45ce4a Compare March 13, 2024 21:38

zclllyybb changed the title ~~[Feature-WIP](partition) Support insert overwrite auto detect partition~~ [Feature](partition) Support insert overwrite auto detect partition Mar 13, 2024

zclllyybb force-pushed the overwrite branch 3 times, most recently from d6c1492 to b45cc82 Compare March 17, 2024 21:42

github-actions bot reviewed Mar 17, 2024

View reviewed changes

be/src/vec/sink/vrow_distribution.h Show resolved Hide resolved

HappenLee reviewed Mar 18, 2024

View reviewed changes

be/src/exec/tablet_info.cpp Show resolved Hide resolved

zclllyybb force-pushed the overwrite branch from b45cc82 to 43091c7 Compare March 18, 2024 05:00

HappenLee reviewed Mar 18, 2024

View reviewed changes

be/src/exec/tablet_info.h Outdated Show resolved Hide resolved

zclllyybb force-pushed the overwrite branch from 43091c7 to a063018 Compare March 18, 2024 08:15

zclllyybb added 5 commits March 18, 2024 16:35

1

b12d13f

done

40fa862

bugfix

5f86b31

done

926a625

test

3ba93a1

zclllyybb force-pushed the overwrite branch from a063018 to 3ba93a1 Compare March 18, 2024 09:09

Jibing-Li reviewed Mar 18, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/analysis/PartitionNames.java Outdated Show resolved Hide resolved

morrySnow reviewed Mar 18, 2024

View reviewed changes

fix

443c9f2

zclllyybb requested review from HappenLee, Jibing-Li and morrySnow March 18, 2024 12:13

zddr reviewed Mar 19, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/insertoverwrite/InsertOverwriteManager.java Show resolved Hide resolved

morrySnow approved these changes Mar 20, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 20, 2024

github-actions bot added the reviewed label Mar 20, 2024

HappenLee approved these changes Mar 20, 2024

View reviewed changes

HappenLee merged commit 340b41c into apache:master Mar 20, 2024

zclllyybb deleted the overwrite branch March 20, 2024 09:21

yiguolei added the 2.1.0-conflict label Mar 21, 2024

zclllyybb added a commit to zclllyybb/doris that referenced this pull request Apr 10, 2024

[Feature](partition) Support insert overwrite auto detect partition (a…

9eb6394

…pache#32156)

zclllyybb added a commit to zclllyybb/doris that referenced this pull request Apr 11, 2024

[Feature](partition) Support insert overwrite auto detect partition (a…

2810a17

…pache#32156)

zclllyybb mentioned this pull request Apr 11, 2024

[branch-2.1](auto-partition) pick auto partition and some more prs #33523

Merged

zclllyybb added a commit to zclllyybb/doris that referenced this pull request Apr 11, 2024

[Feature](partition) Support insert overwrite auto detect partition (a…

029b0be

…pache#32156)

yiguolei added dev/2.1.3-merged and removed 2.1.0-conflict labels Apr 11, 2024

[Feature](partition) Support insert overwrite auto detect partition #32156

[Feature](partition) Support insert overwrite auto detect partition #32156

Uh oh!

Conversation

zclllyybb commented Mar 13, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Mar 13, 2024

Uh oh!

Uh oh!

Uh oh!

zclllyybb commented Mar 17, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

doris-robot commented Mar 17, 2024

Uh oh!

doris-robot commented Mar 17, 2024

Uh oh!

Uh oh!

zclllyybb commented Mar 18, 2024

Uh oh!

Uh oh!

doris-robot commented Mar 18, 2024

Uh oh!

zclllyybb commented Mar 18, 2024

Uh oh!

Uh oh!

doris-robot commented Mar 18, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zclllyybb commented Mar 18, 2024

Uh oh!

doris-robot commented Mar 18, 2024

Uh oh!

Uh oh!

github-actions bot commented Mar 20, 2024

Uh oh!

github-actions bot commented Mar 20, 2024

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants