Skip to content

[opt](Pick)clear min/max column stats if table is partially analyzed (#33685) branch-2.0 #35533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
May 29, 2024
Merged

[opt](Pick)clear min/max column stats if table is partially analyzed (#33685) branch-2.0 #35533

morrySnow merged 1 commit into from
May 29, 2024

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented May 28, 2024

clear min/max column stats if table is partially analyzed (#33685) ed (#33685)

(cherry picked from commit 3d14f66)

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented May 28, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xiaokang
Copy link
Contributor

xiaokang commented May 28, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 28, 2024

TPC-H: Total hot run time: 49635 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e3a1a8acc21572b95a7a16f01acc8fd8dfde5fd5, data reload: false

------ Round 1 ----------------------------------
q1	18094	4405	4341	4341
q2	2069	155	144	144
q3	10440	1854	1907	1854
q4	10323	1239	1325	1239
q5	8466	3854	3937	3854
q6	232	124	148	124
q7	2066	1592	1605	1592
q8	9490	2731	2693	2693
q9	11277	10301	10241	10241
q10	8651	3512	3519	3512
q11	425	246	247	246
q12	463	308	304	304
q13	18366	3965	4008	3965
q14	355	328	328	328
q15	517	454	466	454
q16	669	567	576	567
q17	1115	953	956	953
q18	7203	6929	6992	6929
q19	1734	1609	1530	1530
q20	533	313	309	309
q21	4443	4096	4065	4065
q22	478	406	391	391
Total cold run time: 117409 ms
Total hot run time: 49635 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4291	4344	4290	4290
q2	323	227	226	226
q3	4179	4127	4157	4127
q4	2756	2733	2731	2731
q5	7133	7061	7037	7037
q6	238	122	118	118
q7	3222	2817	2885	2817
q8	4364	4403	4429	4403
q9	16758	16659	16722	16659
q10	4204	4262	4247	4247
q11	774	678	653	653
q12	1020	833	855	833
q13	7409	3776	3735	3735
q14	464	425	420	420
q15	513	463	470	463
q16	740	684	675	675
q17	3757	3826	3849	3826
q18	8813	8744	8870	8744
q19	1743	1713	1672	1672
q20	2461	2149	2119	2119
q21	8531	8481	8459	8459
q22	1034	926	913	913
Total cold run time: 84727 ms
Total hot run time: 79167 ms

@doris-robot
Copy link

doris-robot commented May 28, 2024

TPC-DS: Total hot run time: 203073 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e3a1a8acc21572b95a7a16f01acc8fd8dfde5fd5, data reload: false

query1	925	389	380	380
query2	6583	2831	2553	2553
query3	6917	207	207	207
query4	20816	17993	17991	17991
query5	19740	6516	6549	6516
query6	283	220	231	220
query7	4152	301	304	301
query8	263	255	267	255
query9	3168	2730	2637	2637
query10	427	286	310	286
query11	11400	10763	10760	10760
query12	133	74	73	73
query13	5591	694	699	694
query14	18274	13259	12960	12960
query15	356	220	238	220
query16	6439	284	263	263
query17	1711	1445	893	893
query18	2303	422	425	422
query19	202	160	155	155
query20	81	81	80	80
query21	194	101	97	97
query22	5321	5161	5153	5153
query23	32632	31817	32060	31817
query24	6893	6546	6541	6541
query25	522	443	426	426
query26	526	172	163	163
query27	1841	300	302	300
query28	6157	2375	2330	2330
query29	2905	2792	2599	2599
query30	238	165	172	165
query31	928	774	723	723
query32	67	65	61	61
query33	403	261	261	261
query34	862	472	486	472
query35	1140	894	960	894
query36	1265	1320	1091	1091
query37	93	60	63	60
query38	3076	2903	2928	2903
query39	1379	1323	1328	1323
query40	214	89	96	89
query41	40	38	37	37
query42	84	85	77	77
query43	798	714	670	670
query44	1166	722	713	713
query45	250	230	234	230
query46	1236	960	982	960
query47	1914	1709	1677	1677
query48	1033	734	717	717
query49	645	384	384	384
query50	867	644	630	630
query51	4851	4719	4599	4599
query52	98	90	86	86
query53	451	320	316	316
query54	2681	2518	2452	2452
query55	98	86	87	86
query56	247	221	217	217
query57	1158	1079	1101	1079
query58	230	195	203	195
query59	4106	4406	4046	4046
query60	222	214	211	211
query61	99	100	97	97
query62	853	458	493	458
query63	483	341	345	341
query64	2540	1549	1512	1512
query65	3632	3558	3510	3510
query66	775	394	393	393
query67	16399	17085	15164	15164
query68	8550	647	653	647
query69	583	348	333	333
query70	1510	1597	1590	1590
query71	414	324	310	310
query72	6516	3517	3545	3517
query73	751	326	320	320
query74	6311	5866	5879	5866
query75	5199	3740	3603	3603
query76	5194	1150	1205	1150
query77	884	266	259	259
query78	12523	11803	11548	11548
query79	7992	648	651	648
query80	1091	410	406	406
query81	495	242	237	237
query82	1715	101	99	99
query83	165	131	140	131
query84	258	71	71	71
query85	878	329	321	321
query86	332	323	285	285
query87	3193	3062	3057	3057
query88	4955	2356	2341	2341
query89	405	286	318	286
query90	1964	203	216	203
query91	178	141	142	141
query92	57	54	55	54
query93	5538	593	569	569
query94	702	213	224	213
query95	1087	1041	1044	1041
query96	667	340	330	330
query97	6567	6250	6380	6250
query98	200	186	177	177
query99	2915	944	879	879
Total cold run time: 313165 ms
Total hot run time: 203073 ms

@doris-robot
Copy link

doris-robot commented May 28, 2024

ClickBench: Total hot run time: 30.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3a1a8acc21572b95a7a16f01acc8fd8dfde5fd5, data reload: false

query1	0.03	0.02	0.03
query2	0.07	0.03	0.03
query3	0.24	0.05	0.05
query4	1.80	0.07	0.06
query5	0.54	0.52	0.51
query6	1.23	0.61	0.61
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.53	0.49	0.49
query10	0.54	0.53	0.54
query11	0.12	0.09	0.08
query12	0.12	0.10	0.09
query13	0.62	0.62	0.60
query14	0.78	0.79	0.80
query15	0.78	0.76	0.75
query16	0.38	0.39	0.39
query17	1.02	0.98	1.01
query18	0.22	0.26	0.24
query19	1.96	1.86	1.88
query20	0.02	0.01	0.01
query21	15.46	0.55	0.56
query22	2.10	2.00	1.92
query23	17.06	0.95	0.96
query24	5.53	1.47	0.49
query25	0.35	0.08	0.06
query26	0.59	0.15	0.15
query27	0.04	0.04	0.04
query28	7.97	0.70	0.71
query29	12.60	2.30	2.15
query30	0.57	0.55	0.53
query31	2.80	0.38	0.38
query32	3.38	0.52	0.50
query33	3.08	3.07	3.06
query34	15.24	4.81	4.79
query35	4.82	4.86	4.82
query36	1.08	0.99	1.02
query37	0.05	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.01	0.02
Total cold run time: 104.1 s
Total hot run time: 30.44 s

@doris-robot
Copy link

doris-robot commented May 28, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e3a1a8acc21572b95a7a16f01acc8fd8dfde5fd5 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.1 seconds inserted 10000000 Rows, about 452K ops/s

@morrySnow morrySnow merged commit 7cc003e into apache:branch-2.0 May 29, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@englefly
[opt](nereids) clear min/max column stats if table is partially analy…
79d6b60 
…zed (apache#35533)

cherry picked from master PR apache#33685
commit 3d14f66

if user queries newly loaded data (the new data are not analyzed), optimizer may generate inefficient plan because the newly loaded data is out of column stats min-max range.

In this pr, we will ignore min-max if there are newly loaded data.
@englefly englefly deleted the pick33685-br2.0 branch November 30, 2024 03:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/nereids

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@englefly @doris-robot @xiaokang @morrySnow