[Feature](partition) Support OLAP table null partition #31827

zclllyybb · 2024-03-05T18:27:23Z

Proposed changes

Issue Number: close #xxx

for auto partition, support nullable partition column.
for auto list partition, support create real null partition for null values.
for auto range partition, null value will raise a error now. but maybe we can improve this in the future

doc pr: apache/doris-website#452

Further comments

zclllyybb · 2024-03-05T19:08:56Z

run buildall

doris-robot · 2024-03-05T20:51:44Z

TeamCity be ut coverage result:
Function Coverage: 35.42% (8556/24153)
Line Coverage: 27.17% (69440/255558)
Region Coverage: 26.39% (36025/136513)
Branch Coverage: 23.22% (18407/79262)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a48395ff99525094360aa2ce691ea99253c80199_a48395ff99525094360aa2ce691ea99253c80199/report/index.html

zclllyybb · 2024-03-06T08:06:35Z

run buildall

be/src/vec/sink/vrow_distribution.cpp

doris-robot · 2024-03-06T09:43:32Z

TeamCity be ut coverage result:
Function Coverage: 35.41% (8559/24174)
Line Coverage: 27.16% (69486/255819)
Region Coverage: 26.37% (36037/136639)
Branch Coverage: 23.21% (18415/79342)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cfaf4663eea12e8bc9efe11149e685f91da78912_cfaf4663eea12e8bc9efe11149e685f91da78912/report/index.html

zclllyybb · 2024-03-06T14:42:05Z

run buildall

doris-robot · 2024-03-06T16:09:46Z

TeamCity be ut coverage result:
Function Coverage: 35.41% (8559/24174)
Line Coverage: 27.16% (69485/255824)
Region Coverage: 26.37% (36034/136643)
Branch Coverage: 23.21% (18419/79344)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d426c543e1be2c8ba3a8a5d4b01b350638ea36b7_d426c543e1be2c8ba3a8a5d4b01b350638ea36b7/report/index.html

zclllyybb · 2024-03-07T19:27:38Z

run buildall

zclllyybb · 2024-03-07T19:59:19Z

run buildall

doris-robot · 2024-03-07T21:03:48Z

TeamCity be ut coverage result:
Function Coverage: 35.37% (8555/24186)
Line Coverage: 27.15% (69518/256052)
Region Coverage: 26.37% (36079/136843)
Branch Coverage: 23.21% (18442/79446)
Coverage Report: http://coverage.selectdb-in.cc/coverage/125e526d5bf857abc3558ad6d8f8c37fbc6dbbcd_125e526d5bf857abc3558ad6d8f8c37fbc6dbbcd/report/index.html

zclllyybb · 2024-03-08T02:34:14Z

run buildall

doris-robot · 2024-03-08T04:15:09Z

TeamCity be ut coverage result:
Function Coverage: 35.37% (8554/24186)
Line Coverage: 27.15% (69514/256054)
Region Coverage: 26.36% (36077/136847)
Branch Coverage: 23.21% (18438/79446)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0810c578b64f9151d25903ee31270e0e00ccf2cf_0810c578b64f9151d25903ee31270e0e00ccf2cf/report/index.html

zclllyybb · 2024-03-11T18:07:41Z

run buildall

doris-robot · 2024-03-11T19:32:25Z

TPC-H: Total hot run time: 38162 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39eae89849e935e37c255c5415dfcd9077136d9d, data reload: false

------ Round 1 ----------------------------------
q1	17639	4107	4049	4049
q2	2026	150	145	145
q3	10554	943	929	929
q4	4667	937	948	937
q5	7577	2908	3023	2908
q6	181	126	126	126
q7	1281	826	823	823
q8	9492	2025	2080	2025
q9	7310	6531	6513	6513
q10	8221	2557	2549	2549
q11	420	218	213	213
q12	769	320	321	320
q13	17960	3003	2980	2980
q14	271	254	250	250
q15	504	466	446	446
q16	488	387	390	387
q17	965	904	880	880
q18	6710	6112	5885	5885
q19	1583	1532	1521	1521
q20	547	284	290	284
q21	7508	3691	3696	3691
q22	793	314	301	301
Total cold run time: 107466 ms
Total hot run time: 38162 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4007	4004	4009	4004
q2	319	221	233	221
q3	2958	2989	2926	2926
q4	1867	1837	1798	1798
q5	5242	5323	5239	5239
q6	202	117	124	117
q7	2266	1822	1814	1814
q8	3241	3300	3288	3288
q9	8523	8505	8496	8496
q10	6170	3715	3738	3715
q11	532	434	438	434
q12	693	516	556	516
q13	11831	2858	2816	2816
q14	275	251	252	251
q15	495	461	454	454
q16	433	403	395	395
q17	1705	1671	1676	1671
q18	7832	7641	7381	7381
q19	3140	1563	1613	1563
q20	1899	1680	1675	1675
q21	4936	4701	4824	4701
q22	501	446	463	446
Total cold run time: 69067 ms
Total hot run time: 53921 ms

doris-robot · 2024-03-11T19:42:40Z

TeamCity be ut coverage result:
Function Coverage: 34.98% (8556/24459)
Line Coverage: 26.82% (69537/259275)
Region Coverage: 26.06% (36077/138438)
Branch Coverage: 23.03% (18442/80076)
Coverage Report: http://coverage.selectdb-in.cc/coverage/39eae89849e935e37c255c5415dfcd9077136d9d_39eae89849e935e37c255c5415dfcd9077136d9d/report/index.html

doris-robot · 2024-03-11T19:44:16Z

TPC-DS: Total hot run time: 185981 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39eae89849e935e37c255c5415dfcd9077136d9d, data reload: false

query1	915	346	349	346
query2	7377	1907	1850	1850
query3	6703	211	208	208
query4	24825	20883	20766	20766
query5	4311	471	533	471
query6	265	172	172	172
query7	4621	304	306	304
query8	233	167	169	167
query9	8409	2257	2279	2257
query10	464	237	259	237
query11	15070	14632	14287	14287
query12	130	94	87	87
query13	1652	432	456	432
query14	11091	10021	10420	10021
query15	260	208	194	194
query16	7712	278	268	268
query17	1875	568	553	553
query18	1959	271	272	271
query19	190	161	155	155
query20	93	89	83	83
query21	199	132	121	121
query22	4703	4386	4411	4386
query23	31372	30683	30702	30683
query24	10446	3093	3055	3055
query25	647	405	405	405
query26	1184	166	177	166
query27	2424	364	368	364
query28	6411	1846	1857	1846
query29	1017	607	613	607
query30	311	151	152	151
query31	933	744	720	720
query32	103	64	64	64
query33	767	264	256	256
query34	1052	488	499	488
query35	850	633	637	633
query36	960	831	818	818
query37	134	78	75	75
query38	3257	3171	3164	3164
query39	1529	1379	1379	1379
query40	214	119	115	115
query41	66	65	62	62
query42	112	107	101	101
query43	436	403	385	385
query44	1079	717	710	710
query45	275	257	257	257
query46	1047	817	771	771
query47	1606	1545	1557	1545
query48	446	358	365	358
query49	1129	347	334	334
query50	789	386	386	386
query51	6635	6582	6587	6582
query52	106	92	98	92
query53	357	285	283	283
query54	379	278	249	249
query55	94	85	85	85
query56	245	221	226	221
query57	1077	980	1008	980
query58	244	218	225	218
query59	2432	2281	2400	2281
query60	272	252	259	252
query61	113	111	113	111
query62	653	440	404	404
query63	307	282	322	282
query64	5225	3444	3254	3254
query65	3064	3000	2996	2996
query66	845	340	323	323
query67	14882	14323	14597	14323
query68	7000	572	575	572
query69	638	391	393	391
query70	1240	1106	1119	1106
query71	517	284	287	284
query72	6548	2665	2506	2506
query73	792	327	331	327
query74	6898	6534	6475	6475
query75	9648	8065	8226	8065
query76	4894	1102	1163	1102
query77	661	256	263	256
query78	10210	9529	9605	9529
query79	10193	524	515	515
query80	1783	381	371	371
query81	544	206	208	206
query82	796	207	197	197
query83	207	145	138	138
query84	279	77	79	77
query85	1506	336	317	317
query86	470	308	256	256
query87	3497	3238	3215	3215
query88	4385	2359	2351	2351
query89	557	361	362	361
query90	1944	179	175	175
query91	154	124	128	124
query92	57	49	49	49
query93	7214	540	526	526
query94	1227	188	186	186
query95	457	358	351	351
query96	618	263	262	262
query97	4426	4271	4265	4265
query98	229	205	202	202
query99	1156	757	754	754
Total cold run time: 301328 ms
Total hot run time: 185981 ms

doris-robot · 2024-03-11T19:49:36Z

ClickBench: Total hot run time: 30.99 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39eae89849e935e37c255c5415dfcd9077136d9d, data reload: false

query1	0.04	0.03	0.03
query2	0.05	0.02	0.02
query3	0.23	0.06	0.06
query4	1.67	0.10	0.10
query5	0.52	0.49	0.51
query6	1.38	0.63	0.67
query7	0.01	0.02	0.02
query8	0.03	0.03	0.03
query9	0.57	0.53	0.51
query10	0.57	0.55	0.58
query11	0.13	0.10	0.10
query12	0.12	0.10	0.10
query13	0.58	0.59	0.57
query14	0.74	0.75	0.75
query15	0.82	0.81	0.80
query16	0.38	0.37	0.38
query17	0.97	0.97	0.99
query18	0.26	0.25	0.25
query19	1.84	1.68	1.69
query20	0.02	0.01	0.01
query21	15.42	0.61	0.59
query22	3.24	3.80	2.56
query23	17.44	1.06	0.98
query24	2.22	0.56	0.52
query25	0.23	0.05	0.06
query26	0.19	0.15	0.14
query27	0.03	0.03	0.03
query28	11.95	0.85	0.84
query29	12.69	3.27	3.44
query30	0.63	0.58	0.53
query31	2.79	0.34	0.34
query32	3.39	0.44	0.43
query33	2.77	2.84	2.82
query34	15.48	4.30	4.31
query35	4.31	4.32	4.28
query36	1.09	1.02	1.01
query37	0.07	0.05	0.06
query38	0.04	0.02	0.03
query39	0.02	0.02	0.01
query40	0.19	0.16	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.27 s
Total hot run time: 30.99 s

doris-robot · 2024-03-11T19:52:43Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 39eae89849e935e37c255c5415dfcd9077136d9d with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       16.2 seconds inserted 10000000 Rows, about 617K ops/s

zclllyybb · 2024-03-18T14:41:53Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/vec/columns/column_const.h

doris-robot · 2024-03-22T01:11:54Z

TeamCity be ut coverage result:
Function Coverage: 35.28% (8733/24753)
Line Coverage: 27.09% (71519/264037)
Region Coverage: 26.32% (37099/140932)
Branch Coverage: 23.22% (18973/81694)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8f4a4904f509eb83193efb1bf29d516f9062ed9a_8f4a4904f509eb83193efb1bf29d516f9062ed9a/report/index.html

be/src/exec/tablet_info.cpp

HappenLee

Add doc change

zclllyybb · 2024-03-22T03:42:57Z

run buildall

doris-robot · 2024-03-22T04:38:04Z

TPC-H: Total hot run time: 38044 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8b90b2d9f94602c21555f976383e2633d778c56, data reload: false

------ Round 1 ----------------------------------
q1	17619	4361	4139	4139
q2	2031	158	147	147
q3	10667	1150	1212	1150
q4	10642	766	780	766
q5	7466	3032	2956	2956
q6	201	123	121	121
q7	1059	596	577	577
q8	9521	2054	2018	2018
q9	7254	6602	6489	6489
q10	8431	3437	3545	3437
q11	431	222	218	218
q12	370	193	198	193
q13	17817	2848	2879	2848
q14	228	202	206	202
q15	507	470	458	458
q16	494	365	363	363
q17	985	645	536	536
q18	7278	6520	6606	6520
q19	2387	1444	1453	1444
q20	538	257	256	256
q21	3568	2904	3059	2904
q22	353	302	307	302
Total cold run time: 109847 ms
Total hot run time: 38044 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4215	4098	4094	4094
q2	330	231	230	230
q3	2963	2874	2846	2846
q4	1878	1563	1541	1541
q5	5286	5293	5240	5240
q6	196	116	117	116
q7	2259	1851	1834	1834
q8	3188	3334	3320	3320
q9	8618	8603	8609	8603
q10	3688	3726	3701	3701
q11	566	444	450	444
q12	727	543	565	543
q13	16897	2878	2826	2826
q14	287	266	249	249
q15	494	446	458	446
q16	484	418	403	403
q17	1758	1507	1456	1456
q18	7645	7234	7185	7185
q19	1657	1566	1542	1542
q20	1925	1734	1707	1707
q21	4825	4646	4639	4639
q22	545	456	441	441
Total cold run time: 70431 ms
Total hot run time: 53406 ms

doris-robot · 2024-03-22T04:49:54Z

TPC-DS: Total hot run time: 181189 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8b90b2d9f94602c21555f976383e2633d778c56, data reload: false

query1	945	369	350	350
query2	7459	2082	1925	1925
query3	6709	209	210	209
query4	31189	20756	20606	20606
query5	4333	405	395	395
query6	258	177	170	170
query7	4621	297	285	285
query8	235	163	165	163
query9	9392	2305	2299	2299
query10	466	243	271	243
query11	15349	14091	14526	14091
query12	142	84	89	84
query13	1622	412	410	410
query14	11917	10919	10488	10488
query15	262	193	187	187
query16	8157	264	257	257
query17	1974	577	549	549
query18	1889	294	287	287
query19	342	154	162	154
query20	92	89	91	89
query21	197	128	130	128
query22	4856	4594	4580	4580
query23	31847	31087	31342	31087
query24	10675	2752	2829	2752
query25	610	384	393	384
query26	1128	157	159	157
query27	2960	357	359	357
query28	7554	1915	1902	1902
query29	883	650	631	631
query30	302	151	151	151
query31	967	731	733	731
query32	93	58	57	57
query33	762	246	334	246
query34	972	487	478	478
query35	831	611	638	611
query36	990	857	858	857
query37	128	78	80	78
query38	3558	3428	3444	3428
query39	1408	1384	1383	1383
query40	207	120	117	117
query41	51	48	47	47
query42	105	98	96	96
query43	473	441	451	441
query44	1165	724	711	711
query45	273	263	269	263
query46	1098	681	673	673
query47	1678	1604	1621	1604
query48	454	385	357	357
query49	1102	336	351	336
query50	776	371	365	365
query51	6606	6569	6650	6569
query52	102	94	90	90
query53	347	278	273	273
query54	314	237	266	237
query55	85	83	83	83
query56	254	231	230	230
query57	1103	1016	1022	1016
query58	241	211	212	211
query59	2668	2561	2539	2539
query60	286	254	263	254
query61	115	133	137	133
query62	602	396	419	396
query63	301	275	282	275
query64	5828	4064	3861	3861
query65	3068	2995	2982	2982
query66	1381	350	337	337
query67	15165	14520	14372	14372
query68	7825	525	516	516
query69	625	379	385	379
query70	1239	1144	1120	1120
query71	519	280	275	275
query72	6480	2679	2513	2513
query73	739	317	327	317
query74	8254	6572	6600	6572
query75	4111	2803	2770	2770
query76	5052	931	945	931
query77	650	247	256	247
query78	10428	9920	9735	9735
query79	9861	512	520	512
query80	1511	385	372	372
query81	517	214	209	209
query82	696	199	194	194
query83	220	138	142	138
query84	289	80	78	78
query85	1296	323	315	315
query86	413	287	303	287
query87	3674	3543	3586	3543
query88	4442	2296	2297	2296
query89	493	363	361	361
query90	2055	172	175	172
query91	179	132	135	132
query92	60	49	48	48
query93	6855	500	488	488
query94	1184	177	172	172
query95	427	323	327	323
query96	611	272	264	264
query97	3067	2865	2876	2865
query98	226	209	203	203
query99	1121	752	735	735
Total cold run time: 309041 ms
Total hot run time: 181189 ms

doris-robot · 2024-03-22T04:53:06Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a8b90b2d9f94602c21555f976383e2633d778c56 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

doris-robot · 2024-03-22T05:16:13Z

TeamCity be ut coverage result:
Function Coverage: 35.26% (8730/24756)
Line Coverage: 27.07% (71499/264151)
Region Coverage: 26.30% (37088/141003)
Branch Coverage: 23.21% (18970/81746)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a8b90b2d9f94602c21555f976383e2633d778c56_a8b90b2d9f94602c21555f976383e2633d778c56/report/index.html

HappenLee

LGTM

github-actions · 2024-03-22T16:37:00Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-22T16:37:03Z

PR approved by anyone and no changes requested.

for auto partition, support nullable partition column. for auto list partition, support create real null partition for null values. for auto range partition, null value will raise a error now. but maybe we can improve this in the future

be/src/vec/common/assert_cast.h

github-actions bot added the meta-change label Mar 5, 2024

zclllyybb force-pushed the ap_restrict branch from 8fe3f77 to a48395f Compare March 5, 2024 18:36

zclllyybb force-pushed the ap_restrict branch from a48395f to cfaf466 Compare March 6, 2024 08:03

HappenLee reviewed Mar 6, 2024

View reviewed changes

be/src/vec/sink/vrow_distribution.cpp Show resolved Hide resolved

zclllyybb force-pushed the ap_restrict branch from d426c54 to 73c7956 Compare March 7, 2024 19:26

zclllyybb changed the title ~~[Feature-WIP](partition) Suppor null range partition~~ [Feature](partition) Support OLAP table null partition Mar 7, 2024

zclllyybb force-pushed the ap_restrict branch from 73c7956 to 125e526 Compare March 7, 2024 19:58

zclllyybb force-pushed the ap_restrict branch from 125e526 to 0810c57 Compare March 8, 2024 02:33

zclllyybb force-pushed the ap_restrict branch 2 times, most recently from 8d28e76 to 39eae89 Compare March 11, 2024 18:07

zclllyybb force-pushed the ap_restrict branch from 39eae89 to 24cd86b Compare March 18, 2024 14:41

github-actions bot reviewed Mar 18, 2024

View reviewed changes

be/src/vec/columns/column_const.h Show resolved Hide resolved

be/src/vec/columns/column_const.h Outdated Show resolved Hide resolved

zclllyybb force-pushed the ap_restrict branch 2 times, most recently from af21993 to 6c9edf8 Compare March 22, 2024 03:30

HappenLee reviewed Mar 22, 2024

View reviewed changes

be/src/exec/tablet_info.cpp Outdated Show resolved Hide resolved

HappenLee reviewed Mar 22, 2024

View reviewed changes

all

a8b90b2

zclllyybb force-pushed the ap_restrict branch from 6c9edf8 to a8b90b2 Compare March 22, 2024 03:42

HappenLee approved these changes Mar 22, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 22, 2024

BiteTheDDDDt approved these changes Mar 24, 2024

View reviewed changes

BiteTheDDDDt merged commit a9563c6 into apache:master Mar 24, 2024

zclllyybb deleted the ap_restrict branch March 24, 2024 12:13

yiguolei added the 2.1.0-conflict label Mar 25, 2024

zclllyybb mentioned this pull request Mar 28, 2024

[branch-2.1.2](partition) Support OLAP table null partition (#31827) #32936

Closed

zclllyybb mentioned this pull request Apr 11, 2024

[branch-2.1](auto-partition) pick auto partition and some more prs #33523

Merged

yiguolei added dev/2.1.3-merged and removed 2.1.0-conflict labels Apr 11, 2024

zhiqiang-hhhh reviewed Aug 9, 2024

View reviewed changes

be/src/vec/common/assert_cast.h Show resolved Hide resolved

[Feature](partition) Support OLAP table null partition #31827

[Feature](partition) Support OLAP table null partition #31827

Uh oh!

Conversation

zclllyybb commented Mar 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

zclllyybb commented Mar 5, 2024

Uh oh!

doris-robot commented Mar 5, 2024

Uh oh!

zclllyybb commented Mar 6, 2024

Uh oh!

Uh oh!

doris-robot commented Mar 6, 2024

Uh oh!

zclllyybb commented Mar 6, 2024

Uh oh!

doris-robot commented Mar 6, 2024

Uh oh!

zclllyybb commented Mar 7, 2024

Uh oh!

zclllyybb commented Mar 7, 2024

Uh oh!

doris-robot commented Mar 7, 2024

Uh oh!

zclllyybb commented Mar 8, 2024

Uh oh!

doris-robot commented Mar 8, 2024

Uh oh!

zclllyybb commented Mar 11, 2024

Uh oh!

doris-robot commented Mar 11, 2024

Uh oh!

doris-robot commented Mar 11, 2024

Uh oh!

doris-robot commented Mar 11, 2024

Uh oh!

doris-robot commented Mar 11, 2024

Uh oh!

doris-robot commented Mar 11, 2024

Uh oh!

zclllyybb commented Mar 18, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

zclllyybb commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 22, 2024

Uh oh!

github-actions bot commented Mar 22, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

zclllyybb commented Mar 5, 2024 •

edited

Loading