Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](nereids) fix union all instance number #39999

Merged
merged 2 commits into from
Aug 29, 2024
Merged

[fix](nereids) fix union all instance number #39999

merged 2 commits into from
Aug 29, 2024

Conversation

xzj7019
Copy link
Contributor

@xzj7019 xzj7019 commented Aug 27, 2024 •
edited
Loading

In following plan pattern, fragment 11 has 3 child fragments aligning with 3 exchange nodes and the union plan node only have 2 plan children, i.e, the aggr node and the exchange node(4). In current union node's parallelism computing, it used the plan children's size to do iteration to find the max parallel but ignore the last fragment's. Actually, it should use fragment's children(exclude unpartitioned ones) instead of plan node's children to do the iteration.

11:VUNION(703)
|
|----10:VAGGREGATE (update finalize)(697)
| | group by: brand_name[#23], skc[#24]
| | sortByGroupKey:false
| | cardinality=1
| | final projections: '1', brand_name[#25], skc[#26]
| | final project output tuple id: 11
| | distribute expr lists: skc[#24]
| |
| 9:VHASH JOIN(691)
| | join op: INNER JOIN(PARTITIONED)[]
| | equal join conjunct: (skc[#19] = skc_code[#12])
| | cardinality=1
| | vec output tuple id: 9
| | output tuple id: 9
| | vIntermediate tuple ids: 8
| | hash output slot ids: 18 19
| | final projections: brand_name[#20], skc[#21]
| | final project output tuple id: 9
| | distribute expr lists: skc[#19]
| | distribute expr lists: skc_code[#12]
| |
| |----6:VEXCHANGE
| | offset: 0
| | distribute expr lists: skc_code[#12]
| |
| 8:VEXCHANGE
| offset: 0
| distribute expr lists:
|
4:VEXCHANGE
offset: 0
distribute expr lists:

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xzj7019
Copy link
Contributor Author

xzj7019 commented Aug 27, 2024

run buildall

@wm1581066 wm1581066 added the usercase Important user case type label label Aug 27, 2024
@xzj7019
Copy link
Contributor Author

xzj7019 commented Aug 28, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38505 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3892263c1788ff011b7a6947300b5e8cac459060, data reload: false

------ Round 1 ----------------------------------
q1	18028	4639	4404	4404
q2	2605	182	181	181
q3	11509	1180	1224	1180
q4	10307	691	751	691
q5	7976	2980	2899	2899
q6	234	140	139	139
q7	997	622	608	608
q8	9405	2097	2058	2058
q9	7334	6626	6587	6587
q10	7012	2209	2239	2209
q11	456	243	253	243
q12	408	226	221	221
q13	17829	3039	2989	2989
q14	273	235	237	235
q15	530	494	487	487
q16	586	496	493	493
q17	990	689	676	676
q18	7398	6866	6795	6795
q19	1400	1120	1060	1060
q20	699	334	331	331
q21	4154	3107	3004	3004
q22	1095	1015	1030	1015
Total cold run time: 111225 ms
Total hot run time: 38505 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4394	4323	4280	4280
q2	378	284	270	270
q3	2902	2672	2701	2672
q4	1940	1669	1642	1642
q5	5393	5403	5450	5403
q6	226	131	131	131
q7	2124	1738	1721	1721
q8	3221	3369	3387	3369
q9	8443	8489	8464	8464
q10	3447	3188	3140	3140
q11	598	494	501	494
q12	784	617	619	617
q13	10005	3063	3033	3033
q14	301	270	268	268
q15	518	470	485	470
q16	599	553	556	553
q17	1804	1513	1490	1490
q18	7764	7452	7532	7452
q19	1699	1631	1462	1462
q20	2044	1808	1855	1808
q21	5491	5254	5281	5254
q22	1109	1032	1036	1032
Total cold run time: 65184 ms
Total hot run time: 55025 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188263 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3892263c1788ff011b7a6947300b5e8cac459060, data reload: false

query1	907	372	367	367
query2	6470	2022	1946	1946
query3	6649	216	221	216
query4	31877	23144	23185	23144
query5	4160	504	492	492
query6	254	168	182	168
query7	4590	309	296	296
query8	261	223	219	219
query9	8368	2524	2490	2490
query10	428	286	278	278
query11	17814	15092	15180	15092
query12	152	100	106	100
query13	1652	400	390	390
query14	9998	7381	7213	7213
query15	267	170	176	170
query16	8021	469	434	434
query17	1622	547	562	547
query18	1896	286	281	281
query19	339	141	142	141
query20	117	109	111	109
query21	211	104	99	99
query22	4433	4193	4203	4193
query23	33949	33448	33406	33406
query24	11135	2979	2909	2909
query25	618	381	396	381
query26	1169	159	156	156
query27	2843	294	279	279
query28	7816	2143	2122	2122
query29	806	414	397	397
query30	304	165	153	153
query31	973	778	759	759
query32	102	56	62	56
query33	766	279	285	279
query34	981	482	491	482
query35	846	744	721	721
query36	1079	942	943	942
query37	162	92	93	92
query38	4113	3873	3823	3823
query39	1434	1403	1368	1368
query40	207	118	118	118
query41	48	47	47	47
query42	116	97	102	97
query43	518	490	484	484
query44	1291	758	759	758
query45	190	167	164	164
query46	1097	754	724	724
query47	1891	1766	1818	1766
query48	387	295	299	295
query49	1026	420	412	412
query50	827	412	416	412
query51	7226	6994	7075	6994
query52	98	90	92	90
query53	285	185	190	185
query54	919	472	469	469
query55	82	75	77	75
query56	288	267	256	256
query57	1141	1068	1087	1068
query58	247	223	229	223
query59	3163	2890	2806	2806
query60	294	265	281	265
query61	100	97	96	96
query62	830	660	667	660
query63	217	190	180	180
query64	4155	716	650	650
query65	3211	3117	3150	3117
query66	1358	328	342	328
query67	15454	15127	15175	15127
query68	3112	580	568	568
query69	403	284	308	284
query70	1217	1158	1065	1065
query71	332	285	275	275
query72	6439	3955	3950	3950
query73	745	339	335	335
query74	9035	8799	8830	8799
query75	3384	2712	2715	2712
query76	1865	1071	959	959
query77	473	324	319	319
query78	9549	9112	9056	9056
query79	1027	540	541	540
query80	764	525	529	525
query81	464	303	234	234
query82	250	145	147	145
query83	176	155	156	155
query84	228	77	74	74
query85	682	283	282	282
query86	316	275	299	275
query87	4368	4303	4281	4281
query88	3146	2348	2347	2347
query89	383	288	286	286
query90	1853	191	192	191
query91	126	111	101	101
query92	58	53	53	53
query93	1045	547	540	540
query94	668	308	298	298
query95	326	262	324	262
query96	588	271	273	271
query97	3178	3111	3046	3046
query98	220	202	196	196
query99	1479	1269	1263	1263
Total cold run time: 284147 ms
Total hot run time: 188263 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3892263c1788ff011b7a6947300b5e8cac459060, data reload: false

query1	0.05	0.05	0.04
query2	0.09	0.04	0.03
query3	0.22	0.06	0.05
query4	1.67	0.10	0.10
query5	0.51	0.48	0.50
query6	1.13	0.74	0.74
query7	0.01	0.01	0.02
query8	0.04	0.04	0.05
query9	0.54	0.50	0.49
query10	0.55	0.55	0.56
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.62	0.59	0.58
query14	2.03	2.05	2.13
query15	0.86	0.82	0.83
query16	0.37	0.38	0.37
query17	1.01	1.03	0.98
query18	0.21	0.20	0.21
query19	1.83	1.76	1.72
query20	0.01	0.00	0.01
query21	15.42	0.67	0.66
query22	4.27	7.87	1.55
query23	18.27	1.36	1.25
query24	2.13	0.25	0.21
query25	0.15	0.09	0.08
query26	0.26	0.18	0.18
query27	0.07	0.08	0.07
query28	13.21	1.01	1.01
query29	12.57	3.31	3.31
query30	0.24	0.05	0.05
query31	2.88	0.40	0.39
query32	3.26	0.48	0.47
query33	2.98	3.01	3.05
query34	17.00	4.41	4.35
query35	4.40	4.46	4.45
query36	0.65	0.50	0.47
query37	0.19	0.17	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.13
query41	0.10	0.05	0.05
query42	0.06	0.06	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.57 s
Total hot run time: 31.59 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 9dbd4d9 into apache:master Aug 29, 2024
31 of 35 checks passed
This was referenced Aug 29, 2024
Merged
Merged
yiguolei pushed a commit that referenced this pull request Aug 29, 2024
@xzj7019
[fix](nereids) fix union all instance number (#40099)
2054cc9 
pick from #39999

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
morrySnow pushed a commit that referenced this pull request Sep 2, 2024
@xzj7019
[fix](nereids) fix union all instance number (#39999) (#40100)
a4d124e 
pick from master #39999
dataroaring pushed a commit that referenced this pull request Sep 3, 2024
@xzj7019 @dataroaring
[fix](coordinator) fix union all instance number (#39999)
d1cf33f 
In following plan pattern, fragment 11 has 3 child fragments aligning
with 3 exchange nodes and the union plan node only have 2 plan children,
i.e, the aggr node and the exchange node(4). In current union node's
parallelism computing, it used the plan children's size to do iteration
to find the max parallel but ignore the last fragment's. Actually, it
should use fragment's children(exclude unpartitioned ones) instead of
plan node's children to do the iteration.

11:VUNION(703)
  |
  |----10:VAGGREGATE (update finalize)(697)
  |    |  group by: brand_name[#23], skc[#24]
  |    |  sortByGroupKey:false
  |    |  cardinality=1
  |    |  final projections: '1', brand_name[#25], skc[#26]
  |    |  final project output tuple id: 11
  |    |  distribute expr lists: skc[#24]
  |    |
  |    9:VHASH JOIN(691)
  |    |  join op: INNER JOIN(PARTITIONED)[]
  |    |  equal join conjunct: (skc[#19] = skc_code[#12])
  |    |  cardinality=1
  |    |  vec output tuple id: 9
  |    |  output tuple id: 9
  |    |  vIntermediate tuple ids: 8
  |    |  hash output slot ids: 18 19
  |    |  final projections: brand_name[#20], skc[#21]
  |    |  final project output tuple id: 9
  |    |  distribute expr lists: skc[#19]
  |    |  distribute expr lists: skc_code[#12]
  |    |
  |    |----6:VEXCHANGE
  |    |       offset: 0
  |    |       distribute expr lists: skc_code[#12]
  |    |
  |    8:VEXCHANGE
  |       offset: 0
  |       distribute expr lists:
  |
  4:VEXCHANGE
     offset: 0
     distribute expr lists:
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@924060929 924060929 924060929 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.2-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@xzj7019 @doris-robot @924060929 @morrySnow @yiguolei @wm1581066 @dataroaring