Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](auto bucket) Fix hit not support alter estimate_partition_size #33670

Merged
merged 1 commit into from
Apr 15, 2024
Merged

[fix](auto bucket) Fix hit not support alter estimate_partition_size #33670

merged 1 commit into from
Apr 15, 2024

Conversation

deardeng
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring
Copy link
Contributor

run buildall

dataroaring
dataroaring approved these changes Apr 15, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 15, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@xiaokang
Copy link
Contributor

@deardeng I think it should be allowed to modified.

@doris-robot
Copy link

TPC-H: Total hot run time: 38546 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8b1a7af9783c0927f69a7973075c0d7f45ab12d6, data reload: false

------ Round 1 ----------------------------------
q1	17599	4349	4338	4338
q2	2013	190	182	182
q3	10454	1161	1186	1161
q4	10188	734	720	720
q5	7539	2684	2681	2681
q6	217	131	130	130
q7	1006	612	589	589
q8	9228	2038	2021	2021
q9	7927	6627	6565	6565
q10	8571	3544	3522	3522
q11	459	236	232	232
q12	485	224	212	212
q13	18877	2886	2991	2886
q14	287	231	237	231
q15	517	488	466	466
q16	506	394	378	378
q17	959	622	686	622
q18	7303	6820	6666	6666
q19	6312	1558	1513	1513
q20	683	329	311	311
q21	3427	2817	2916	2817
q22	358	303	307	303
Total cold run time: 114915 ms
Total hot run time: 38546 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4388	4245	4286	4245
q2	361	256	265	256
q3	2948	2790	2786	2786
q4	1872	1591	1577	1577
q5	5293	5302	5332	5302
q6	207	121	122	121
q7	2228	1812	1868	1812
q8	3194	3302	3319	3302
q9	8569	8602	8763	8602
q10	4049	3938	4004	3938
q11	618	491	484	484
q12	799	637	612	612
q13	16313	3225	3115	3115
q14	331	290	267	267
q15	502	462	497	462
q16	513	436	441	436
q17	1830	1548	1516	1516
q18	8121	7854	7793	7793
q19	1697	1616	1561	1561
q20	2034	1838	1870	1838
q21	8937	4921	4893	4893
q22	542	509	441	441
Total cold run time: 75346 ms
Total hot run time: 55359 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184158 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8b1a7af9783c0927f69a7973075c0d7f45ab12d6, data reload: false

query1	1223	1125	1122	1122
query2	6223	2560	2317	2317
query3	6639	201	201	201
query4	37242	21705	21670	21670
query5	4145	390	390	390
query6	244	186	185	185
query7	4063	287	287	287
query8	222	177	177	177
query9	5811	2273	2260	2260
query10	363	243	240	240
query11	14785	14391	14217	14217
query12	140	88	91	88
query13	978	369	351	351
query14	9474	6912	6885	6885
query15	211	179	171	171
query16	6882	264	250	250
query17	1593	570	548	548
query18	1398	290	275	275
query19	200	154	165	154
query20	92	87	88	87
query21	197	129	128	128
query22	5082	4841	4885	4841
query23	33619	33176	33538	33176
query24	8865	3015	3002	3002
query25	535	404	411	404
query26	823	159	157	157
query27	3100	373	362	362
query28	6789	2085	2108	2085
query29	880	627	634	627
query30	275	180	171	171
query31	940	774	754	754
query32	61	53	52	52
query33	517	267	245	245
query34	898	492	488	488
query35	796	716	704	704
query36	1109	952	940	940
query37	105	70	67	67
query38	3544	3399	3317	3317
query39	1088	1077	1047	1047
query40	174	132	126	126
query41	51	42	45	42
query42	103	98	94	94
query43	559	534	529	529
query44	1156	736	746	736
query45	286	267	259	259
query46	1099	743	739	739
query47	2041	2046	1981	1981
query48	373	297	297	297
query49	857	370	377	370
query50	802	403	403	403
query51	6870	6786	6628	6628
query52	111	86	90	86
query53	345	276	270	270
query54	265	231	215	215
query55	74	70	71	70
query56	236	218	233	218
query57	1215	1160	1148	1148
query58	216	209	203	203
query59	3260	3284	3069	3069
query60	253	231	233	231
query61	90	88	93	88
query62	585	453	424	424
query63	313	271	282	271
query64	4192	3866	4109	3866
query65	3141	3099	3081	3081
query66	737	319	316	316
query67	16400	15224	14855	14855
query68	7973	546	550	546
query69	606	311	315	311
query70	1232	1158	1098	1098
query71	459	270	277	270
query72	6521	2664	2425	2425
query73	821	321	317	317
query74	7292	6356	6425	6356
query75	3622	2672	2696	2672
query76	4201	1081	1141	1081
query77	664	246	242	242
query78	10875	10352	10290	10290
query79	8251	521	521	521
query80	1628	412	426	412
query81	525	236	233	233
query82	824	96	90	90
query83	211	167	177	167
query84	264	83	81	81
query85	931	271	291	271
query86	394	297	313	297
query87	3479	3270	3269	3269
query88	5775	2240	2271	2240
query89	526	371	387	371
query90	1899	174	178	174
query91	119	97	96	96
query92	60	49	47	47
query93	6198	498	497	497
query94	1029	183	224	183
query95	381	279	288	279
query96	593	266	263	263
query97	3122	2933	2967	2933
query98	233	228	222	222
query99	1179	847	866	847
Total cold run time: 298842 ms
Total hot run time: 184158 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.28 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8b1a7af9783c0927f69a7973075c0d7f45ab12d6, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.49	0.47
query6	1.46	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.55	0.55	0.56
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.59	0.59
query14	0.76	0.78	0.76
query15	0.83	0.81	0.81
query16	0.36	0.38	0.38
query17	0.94	0.98	0.95
query18	0.19	0.27	0.21
query19	1.77	1.74	1.75
query20	0.02	0.01	0.01
query21	15.39	0.65	0.65
query22	4.08	7.01	2.67
query23	18.29	1.36	1.27
query24	2.10	0.20	0.20
query25	0.14	0.08	0.09
query26	0.27	0.17	0.16
query27	0.08	0.07	0.07
query28	13.34	1.00	0.99
query29	12.56	3.30	3.26
query30	0.25	0.07	0.06
query31	2.88	0.39	0.38
query32	3.26	0.46	0.46
query33	2.84	2.88	2.92
query34	17.01	4.50	4.43
query35	4.57	4.51	4.58
query36	0.65	0.47	0.46
query37	0.18	0.15	0.15
query38	0.16	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.15	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.4 s
Total hot run time: 31.28 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 8b1a7af9783c0927f69a7973075c0d7f45ab12d6 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.2 seconds inserted 10000000 Rows, about 757K ops/s

@dataroaring dataroaring merged commit 93879db into apache:master Apr 15, 2024
27 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Apr 24, 2024
@deardeng @dataroaring
[fix](auto bucket) Fix hit not support alter estimate_partition_size (#…
db4f978 
…33670)
@deardeng deardeng deleted the fix-hint branch April 30, 2024 10:54
deardeng added a commit to deardeng/incubator-doris that referenced this pull request Jul 10, 2024
@deardeng
[fix](auto bucket) Fix hit not support alter estimate_partition_size (a…
d76628d 
…pache#33670)
@deardeng deardeng mentioned this pull request Jul 10, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 10, 2024
@deardeng
[fix](auto bucket) Fix hit not support alter estimate_partition_size #…
68c4ce1 
…33670 (#37587)

cherry pick from #33670
deardeng added a commit to deardeng/incubator-doris that referenced this pull request Jul 11, 2024
@deardeng
[fix](auto bucket) Fix hit not support alter estimate_partition_size (a…
bec67f4 
…pache#33670)
@deardeng deardeng mentioned this pull request Jul 11, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 13, 2024
@deardeng
[fix](auto bucket) Fix hit not support alter estimate_partition_size #…
ec8467f 
…33670 (#37633)

cherry pick from #33670
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@deardeng
[fix](auto bucket) Fix hit not support alter estimate_partition_size a…
8d4a7b4 
…pache#33670 (apache#37587)

cherry pick from apache#33670
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@deardeng @doris-robot @dataroaring @xiaokang @gavinchou @yiguolei