Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nerieds) column prune should retain at least one column for union all #41613

Merged
merged 1 commit into from
Oct 11, 2024
Merged

[fix](Nerieds) column prune should retain at least one column for union all #41613

merged 1 commit into from
Oct 11, 2024

Conversation

feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented Oct 9, 2024 •
edited by morrySnow
Loading

introduce by #31811 and #39450

select count(1) from(select 3, 6 union all select 1, 3) t

wrong LogicalUnion plan:

LogicalUnion( qualifier=ALL, outputs=[3#6], regularChildrenOutputs=[], constantExprsList=[[], []], hasPushedFilter=false

this sql will report error in explain, because the logical union outputs has a slot, but the logical union has no child and has a empty constantExprList, which is wrong set in column prune.
this pr fixes it by consider when require columns is empty and keep the min slot and min slot corresponding const expressions.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@feiniaofeiafei feiniaofeiafei force-pushed the fix_column_prune_union_const_expr branch from 76b3b57 to edc3a97 Compare October 9, 2024 09:50
@feiniaofeiafei
Copy link
Contributor Author

run buildall

@morrySnow morrySnow changed the title fix bug in column prune with union const expr [fix](Nerieds) column prune should retain at least one column for union all Oct 9, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 40833 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit edc3a9744d63de962c660984572e481eae6112fa, data reload: false

------ Round 1 ----------------------------------
q1	17575	7379	7237	7237
q2	2018	285	278	278
q3	12050	1094	1165	1094
q4	10568	726	693	693
q5	7740	2856	2871	2856
q6	238	154	152	152
q7	968	632	604	604
q8	9348	1901	1952	1901
q9	6532	6427	6400	6400
q10	6948	2263	2319	2263
q11	442	253	247	247
q12	409	224	222	222
q13	17799	3004	3002	3002
q14	232	217	214	214
q15	586	525	512	512
q16	643	583	590	583
q17	977	552	522	522
q18	7198	6666	6799	6666
q19	1350	957	986	957
q20	472	199	196	196
q21	4225	3276	3258	3258
q22	1068	976	1003	976
Total cold run time: 109386 ms
Total hot run time: 40833 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7252	7244	7202	7202
q2	332	230	223	223
q3	2987	2992	2983	2983
q4	2139	1816	1844	1816
q5	5789	5724	5729	5724
q6	238	144	150	144
q7	2270	1860	1832	1832
q8	3370	3545	3423	3423
q9	8923	8956	8773	8773
q10	3597	3580	3513	3513
q11	587	524	507	507
q12	874	680	625	625
q13	10545	3231	3164	3164
q14	317	281	277	277
q15	582	539	529	529
q16	714	631	623	623
q17	1857	1633	1612	1612
q18	8285	7648	7715	7648
q19	1723	1532	1427	1427
q20	2097	1887	1904	1887
q21	5586	5382	5361	5361
q22	1108	1021	1036	1021
Total cold run time: 71172 ms
Total hot run time: 60314 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192716 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit edc3a9744d63de962c660984572e481eae6112fa, data reload: false

query1	917	409	413	409
query2	6256	2176	2036	2036
query3	8686	196	203	196
query4	34161	23532	23715	23532
query5	3443	473	469	469
query6	272	177	175	175
query7	4188	313	322	313
query8	291	228	230	228
query9	9358	2664	2667	2664
query10	440	284	287	284
query11	17636	15341	15242	15242
query12	160	104	101	101
query13	1584	481	451	451
query14	8969	7663	7602	7602
query15	258	174	174	174
query16	7630	476	493	476
query17	1687	637	603	603
query18	2053	315	320	315
query19	360	160	157	157
query20	123	115	114	114
query21	219	107	106	106
query22	4809	4680	4609	4609
query23	35411	34190	33897	33897
query24	10980	2940	2851	2851
query25	633	403	415	403
query26	1164	162	159	159
query27	2232	306	296	296
query28	7207	2415	2424	2415
query29	813	439	422	422
query30	266	149	153	149
query31	1014	808	793	793
query32	94	54	56	54
query33	767	307	296	296
query34	907	499	517	499
query35	844	742	736	736
query36	1091	922	948	922
query37	153	83	90	83
query38	3941	3932	3868	3868
query39	1508	1427	1436	1427
query40	200	97	98	97
query41	47	44	47	44
query42	117	96	95	95
query43	530	503	485	485
query44	1238	814	810	810
query45	197	166	167	166
query46	1147	715	716	715
query47	1916	1840	1844	1840
query48	452	371	355	355
query49	876	416	409	409
query50	815	417	433	417
query51	7110	6898	6905	6898
query52	96	83	85	83
query53	259	178	181	178
query54	1005	459	478	459
query55	80	74	77	74
query56	268	264	252	252
query57	1261	1164	1185	1164
query58	229	235	233	233
query59	3241	3057	2974	2974
query60	297	273	273	273
query61	103	101	97	97
query62	822	675	646	646
query63	222	191	190	190
query64	4094	631	634	631
query65	3267	3187	3185	3185
query66	724	326	314	314
query67	15759	16141	15736	15736
query68	4575	597	564	564
query69	479	294	295	294
query70	1159	1142	1044	1044
query71	346	278	280	278
query72	7161	3938	3950	3938
query73	773	350	348	348
query74	10053	8999	9138	8999
query75	3376	2719	2674	2674
query76	2726	967	962	962
query77	607	290	296	290
query78	10568	9675	9539	9539
query79	2072	595	609	595
query80	876	459	443	443
query81	575	239	244	239
query82	318	137	144	137
query83	291	157	138	138
query84	274	78	75	75
query85	990	302	295	295
query86	329	305	265	265
query87	4421	4272	4424	4272
query88	3867	2437	2395	2395
query89	390	287	276	276
query90	1947	187	185	185
query91	139	108	107	107
query92	65	50	50	50
query93	1152	545	545	545
query94	691	293	298	293
query95	354	256	258	256
query96	608	279	282	279
query97	3246	3114	3104	3104
query98	216	202	208	202
query99	1567	1289	1329	1289
Total cold run time: 295108 ms
Total hot run time: 192716 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit edc3a9744d63de962c660984572e481eae6112fa, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.05
query4	1.65	0.10	0.10
query5	0.51	0.52	0.51
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.56	0.53	0.50
query10	0.55	0.54	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.60
query14	2.86	2.73	2.85
query15	0.90	0.84	0.85
query16	0.38	0.37	0.38
query17	1.03	1.07	1.06
query18	0.20	0.20	0.20
query19	1.95	1.79	1.97
query20	0.02	0.01	0.01
query21	15.35	0.60	0.61
query22	2.35	1.78	1.38
query23	17.11	0.91	0.82
query24	2.94	0.39	0.87
query25	0.25	0.17	0.05
query26	0.31	0.13	0.14
query27	0.04	0.04	0.04
query28	11.39	1.09	1.07
query29	12.55	3.21	3.21
query30	0.24	0.06	0.06
query31	2.88	0.38	0.38
query32	3.29	0.46	0.45
query33	3.02	3.00	3.06
query34	16.97	4.42	4.46
query35	4.51	4.52	4.54
query36	0.65	0.52	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.11	0.12
query41	0.07	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 107.3 s
Total hot run time: 31.61 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 10, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 473007d into apache:master Oct 11, 2024
27 of 31 checks passed
feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Oct 12, 2024
@feiniaofeiafei
[fix](Nerieds) column prune should retain at least one column for uni…
23f7f00 
…on all (apache#41613)


select count(1) from(select 3, 6 union all select 1, 3) t

wrong LogicalUnion plan:

LogicalUnion( qualifier=ALL, outputs=[3#6], regularChildrenOutputs=[], constantExprsList=[[], []], hasPushedFilter=false

this sql will report error in explain, because the logical union outputs
has a slot, but the logical union has no child and has a empty
constantExprList, which is wrong set in column prune.
this pr fixes it by consider when require columns is empty and keep the
min slot and min slot corresponding const expressions.
@feiniaofeiafei feiniaofeiafei mentioned this pull request Oct 12, 2024
Merged
yiguolei pushed a commit that referenced this pull request Oct 14, 2024
@feiniaofeiafei
[fix](Nerieds) column prune should retain at least one column for uni…
afdc680 
…on all (#41613) (#41771)

cherry-pick #41613 to branch-2.1
feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Oct 15, 2024
@feiniaofeiafei
[fix](Nerieds) column prune should retain at least one column for uni…
3ce6d47 
…on all (apache#41613)


select count(1) from(select 3, 6 union all select 1, 3) t

wrong LogicalUnion plan:

LogicalUnion( qualifier=ALL, outputs=[3#6], regularChildrenOutputs=[], constantExprsList=[[], []], hasPushedFilter=false

this sql will report error in explain, because the logical union outputs
has a slot, but the logical union has no child and has a empty
constantExprList, which is wrong set in column prune.
this pr fixes it by consider when require columns is empty and keep the
min slot and min slot corresponding const expressions.
@feiniaofeiafei feiniaofeiafei mentioned this pull request Oct 15, 2024
Merged
924060929 pushed a commit that referenced this pull request Oct 24, 2024
@feiniaofeiafei
[fix](Nerieds) column prune should retain at least one column for uni…
12eb9c3 
…on all (#41613) (#41909)

introduce by #31811 and #39450
```sql
select count(1) from(select 3, 6 union all select 1, 3) t
```
wrong LogicalUnion plan:
```sql
LogicalUnion( qualifier=ALL, outputs=[3#6], regularChildrenOutputs=[], constantExprsList=[[], []], hasPushedFilter=false
```
this sql will report error in explain, because the logical union outputs has a slot, but the logical union has no child and has a empty constantExprList, which is wrong set in column prune.
this pr fixes it by consider when require columns is empty and keep the min slot and min slot corresponding const expressions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@924060929 924060929 924060929 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@feiniaofeiafei @doris-robot @924060929 @morrySnow @yiguolei