Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bugfix](hive/iceberg)align with Hive insert overwrite table functionality #39840

Merged
merged 6 commits into from
Sep 5, 2024
Merged

[bugfix](hive/iceberg)align with Hive insert overwrite table functionality #39840

merged 6 commits into from
Sep 5, 2024

Conversation

wuwenchi
Copy link
Contributor

Proposed changes

When execute: insert overwrite table tb1 select * from tb2 and tb2 is an empty table:

  1. if tb1 is a partition table, there will be nothing to do.
  2. if tb1 is a unpartition table, the data in the table will be cleared.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37728 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ccc2e95ddd1233dfff143c3dfb2da95bea89aa9, data reload: false

------ Round 1 ----------------------------------
q1	17656	4321	4359	4321
q2	2024	186	171	171
q3	11708	941	1018	941
q4	10509	792	815	792
q5	7755	2845	2826	2826
q6	223	140	134	134
q7	948	615	599	599
q8	9334	2039	2107	2039
q9	7167	6548	6511	6511
q10	6999	2273	2200	2200
q11	466	245	246	245
q12	390	218	217	217
q13	17756	3034	3057	3034
q14	269	242	239	239
q15	510	480	486	480
q16	490	386	393	386
q17	969	679	635	635
q18	7214	6881	6734	6734
q19	1396	1074	993	993
q20	700	324	336	324
q21	3826	3086	2906	2906
q22	1103	1039	1001	1001
Total cold run time: 109412 ms
Total hot run time: 37728 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4339	4296	4255	4255
q2	381	268	278	268
q3	2877	2660	2645	2645
q4	1949	1668	1619	1619
q5	5502	5652	5666	5652
q6	224	142	137	137
q7	2282	1798	1856	1798
q8	3296	3413	3402	3402
q9	8833	8897	8864	8864
q10	3586	3416	3380	3380
q11	596	528	531	528
q12	835	658	682	658
q13	17021	3130	3232	3130
q14	310	294	301	294
q15	548	514	498	498
q16	484	452	438	438
q17	1847	1543	1523	1523
q18	8005	7841	7825	7825
q19	1730	1600	1626	1600
q20	2136	1904	1906	1904
q21	5680	5531	5482	5482
q22	1156	1042	1039	1039
Total cold run time: 73617 ms
Total hot run time: 56939 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192413 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6ccc2e95ddd1233dfff143c3dfb2da95bea89aa9, data reload: false

query1	1240	909	868	868
query2	6301	1927	1836	1836
query3	10620	4023	3978	3978
query4	59730	26181	23261	23261
query5	5437	500	505	500
query6	405	176	156	156
query7	5757	299	292	292
query8	276	212	214	212
query9	8980	2508	2483	2483
query10	507	287	269	269
query11	18196	15127	15376	15127
query12	149	107	99	99
query13	1542	403	385	385
query14	10567	7310	7608	7310
query15	225	182	169	169
query16	7512	464	469	464
query17	1108	551	569	551
query18	2002	290	291	290
query19	272	142	156	142
query20	120	115	114	114
query21	201	111	104	104
query22	4570	4378	4461	4378
query23	34615	33608	33351	33351
query24	6184	2843	2864	2843
query25	522	411	395	395
query26	670	159	160	159
query27	1740	292	292	292
query28	3775	2062	2044	2044
query29	704	424	422	422
query30	236	155	145	145
query31	919	753	779	753
query32	81	59	56	56
query33	449	296	296	296
query34	875	481	482	481
query35	813	707	722	707
query36	1050	937	960	937
query37	144	88	88	88
query38	3949	3819	3927	3819
query39	1453	1378	1392	1378
query40	214	121	121	121
query41	48	48	45	45
query42	116	99	99	99
query43	520	482	476	476
query44	1093	755	760	755
query45	198	164	165	164
query46	1088	759	736	736
query47	1906	1787	1810	1787
query48	397	301	288	288
query49	782	452	443	443
query50	845	406	409	406
query51	7211	7031	7007	7007
query52	99	98	90	90
query53	257	185	182	182
query54	591	487	466	466
query55	82	84	79	79
query56	296	273	262	262
query57	1186	1047	1066	1047
query58	230	226	233	226
query59	2940	2821	2716	2716
query60	302	281	288	281
query61	125	120	123	120
query62	776	639	628	628
query63	218	188	183	183
query64	2872	1820	1811	1811
query65	3248	3170	3131	3131
query66	615	344	338	338
query67	15349	15468	15248	15248
query68	4443	567	568	567
query69	405	396	277	277
query70	1204	1085	1028	1028
query71	344	268	271	268
query72	6414	2346	1965	1965
query73	755	318	327	318
query74	9183	8684	8798	8684
query75	3382	2692	2706	2692
query76	1720	1005	1003	1003
query77	582	324	326	324
query78	9775	10323	9811	9811
query79	1776	548	538	538
query80	869	512	502	502
query81	565	228	222	222
query82	319	141	135	135
query83	181	149	148	148
query84	271	78	78	78
query85	718	289	318	289
query86	396	297	260	260
query87	4356	4234	4366	4234
query88	3244	2296	2279	2279
query89	395	288	282	282
query90	1847	190	186	186
query91	122	98	98	98
query92	69	50	53	50
query93	1770	556	555	555
query94	798	296	284	284
query95	353	254	253	253
query96	598	274	264	264
query97	3189	3059	3075	3059
query98	223	203	198	198
query99	1545	1273	1293	1273
Total cold run time: 309311 ms
Total hot run time: 192413 ms

@wuwenchi wuwenchi marked this pull request as ready for review August 26, 2024 01:59
@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38371 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 010e293835b4b099ac51c1eba51b3984ac2d444b, data reload: false

------ Round 1 ----------------------------------
q1	18277	4800	4387	4387
q2	2633	184	175	175
q3	11077	1166	1191	1166
q4	10274	765	656	656
q5	7830	2854	2906	2854
q6	232	139	142	139
q7	975	634	613	613
q8	9712	2090	2036	2036
q9	6982	6499	6526	6499
q10	7003	2289	2223	2223
q11	443	255	256	255
q12	399	236	230	230
q13	17789	3038	3093	3038
q14	291	235	235	235
q15	533	490	493	490
q16	497	394	404	394
q17	968	697	739	697
q18	7476	6858	6888	6858
q19	1389	1071	966	966
q20	681	335	322	322
q21	4312	3223	3136	3136
q22	1139	1002	1023	1002
Total cold run time: 110912 ms
Total hot run time: 38371 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4377	4290	4223	4223
q2	382	274	282	274
q3	2898	2696	2675	2675
q4	1891	1675	1647	1647
q5	5387	5436	5402	5402
q6	214	131	134	131
q7	2110	1748	1717	1717
q8	3201	3365	3364	3364
q9	8396	8430	8342	8342
q10	3460	3205	3190	3190
q11	599	499	481	481
q12	823	632	611	611
q13	8498	3055	3054	3054
q14	314	278	272	272
q15	531	486	484	484
q16	467	435	419	419
q17	1781	1509	1473	1473
q18	7764	7370	7467	7370
q19	1649	1538	1504	1504
q20	2072	1843	1842	1842
q21	5537	5095	5279	5095
q22	1151	1050	1010	1010
Total cold run time: 63502 ms
Total hot run time: 54580 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187124 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 010e293835b4b099ac51c1eba51b3984ac2d444b, data reload: false

query1	911	375	356	356
query2	6478	1969	1931	1931
query3	6642	218	226	218
query4	33717	23137	23004	23004
query5	4150	485	487	485
query6	265	167	160	160
query7	4585	312	306	306
query8	255	210	212	210
query9	8508	2491	2471	2471
query10	434	268	281	268
query11	17799	15016	15065	15016
query12	146	105	99	99
query13	1644	392	404	392
query14	9343	7137	6561	6561
query15	285	169	168	168
query16	8089	478	456	456
query17	1572	552	529	529
query18	2118	278	289	278
query19	266	142	146	142
query20	116	110	114	110
query21	205	105	101	101
query22	4443	4190	4086	4086
query23	34093	33931	33268	33268
query24	11207	2860	2889	2860
query25	634	380	402	380
query26	1179	157	156	156
query27	2458	283	278	278
query28	7324	2034	2033	2033
query29	817	417	406	406
query30	289	162	151	151
query31	976	752	788	752
query32	99	54	59	54
query33	751	290	279	279
query34	975	496	502	496
query35	840	735	719	719
query36	1099	931	952	931
query37	158	91	78	78
query38	3968	3843	3880	3843
query39	1478	1399	1394	1394
query40	193	117	114	114
query41	47	47	44	44
query42	118	102	100	100
query43	504	470	467	467
query44	1228	749	750	749
query45	196	167	166	166
query46	1100	740	731	731
query47	1840	1784	1781	1781
query48	379	308	301	301
query49	1072	443	443	443
query50	820	426	416	416
query51	7270	7030	7084	7030
query52	100	86	88	86
query53	257	189	187	187
query54	903	492	486	486
query55	80	80	79	79
query56	285	280	266	266
query57	1172	1108	1077	1077
query58	237	229	220	220
query59	3050	2969	2743	2743
query60	297	272	268	268
query61	102	100	103	100
query62	792	653	647	647
query63	217	189	185	185
query64	4104	718	655	655
query65	3227	3167	3155	3155
query66	724	337	349	337
query67	15622	15165	15405	15165
query68	3558	587	582	582
query69	399	286	281	281
query70	1170	1134	1079	1079
query71	332	273	276	273
query72	6394	3964	4051	3964
query73	750	337	345	337
query74	9252	8913	8855	8855
query75	3374	2687	2739	2687
query76	2020	1010	989	989
query77	550	317	322	317
query78	10199	9115	8979	8979
query79	1016	538	538	538
query80	673	507	507	507
query81	502	234	229	229
query82	236	142	136	136
query83	170	157	155	155
query84	226	85	80	80
query85	857	296	279	279
query86	314	314	299	299
query87	4472	4246	4247	4246
query88	3223	2371	2366	2366
query89	388	299	290	290
query90	1876	199	195	195
query91	124	105	99	99
query92	61	52	53	52
query93	1034	540	544	540
query94	695	353	299	299
query95	360	259	261	259
query96	593	276	268	268
query97	3177	3033	3125	3033
query98	213	211	201	201
query99	1516	1287	1333	1287
Total cold run time: 285976 ms
Total hot run time: 187124 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 010e293835b4b099ac51c1eba51b3984ac2d444b, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.05
query3	0.23	0.06	0.06
query4	1.67	0.09	0.09
query5	0.50	0.48	0.50
query6	1.14	0.74	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.54	0.50	0.49
query10	0.53	0.53	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.76	0.77	0.80
query15	0.91	0.83	0.82
query16	0.37	0.36	0.36
query17	0.97	0.98	1.05
query18	0.21	0.20	0.20
query19	1.90	1.71	1.73
query20	0.00	0.01	0.02
query21	15.39	0.65	0.65
query22	4.25	6.50	2.46
query23	18.34	1.54	1.33
query24	2.30	0.23	0.22
query25	0.16	0.08	0.09
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.23	1.02	1.00
query29	12.61	3.34	3.28
query30	0.24	0.05	0.06
query31	2.88	0.39	0.39
query32	3.27	0.47	0.46
query33	2.97	3.04	3.00
query34	16.96	4.34	4.41
query35	4.44	4.40	4.44
query36	0.66	0.50	0.49
query37	0.20	0.16	0.16
query38	0.17	0.15	0.15
query39	0.05	0.03	0.03
query40	0.16	0.13	0.14
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 109.68 s
Total hot run time: 31.2 s

@wuwenchi
Copy link
Contributor Author

run feut

morningman
morningman approved these changes Sep 4, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 4, 2024

PR approved by anyone and no changes requested.

@morningman morningman merged commit 67531f6 into apache:master Sep 5, 2024
28 of 30 checks passed
wuwenchi added a commit to wuwenchi/doris_new that referenced this pull request Sep 12, 2024
@wuwenchi
[bugfix](hive/iceberg)align with Hive insert overwrite table function…
80021dc 
…ality (apache#39840)

## Proposed changes

When execute: `insert overwrite table tb1 select * from tb2` and `tb2`
is an empty table:
1. if `tb1` is a partition table, there will be nothing to do.
2. if `tb1` is a unpartition table, the data in the table will be
cleared.
@wuwenchi wuwenchi mentioned this pull request Sep 12, 2024
Merged
yiguolei pushed a commit that referenced this pull request Sep 12, 2024
@wuwenchi
[bugfix](hive/iceberg)align with Hive insert overwrite table function…
4b7b43b 
…ality (#39840) (#40724)

bp #39840
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@wuwenchi @dataroaring
[bugfix](hive/iceberg)align with Hive insert overwrite table function…
23805df 
…ality (#39840)

## Proposed changes

When execute: `insert overwrite table tb1 select * from tb2` and `tb2`
is an empty table:
1. if `tb1` is a partition table, there will be nothing to do.
2. if `tb1` is a unpartition table, the data in the table will be
cleared.
@yiguolei yiguolei mentioned this pull request Nov 6, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@wuwenchi @doris-robot @morningman @CalvinKirs @yiguolei