Skip to content

[fix](nereids) fix can not prune dereference expression #58657

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
924060929 merged 5 commits into from
Dec 3, 2025
Merged

[fix](nereids) fix can not prune dereference expression #58657

924060929 merged 5 commits into from
Dec 3, 2025

Conversation

@924060929
Copy link
Contributor

@924060929 924060929 commented Dec 3, 2025

What problem does this PR solve?

fix can not prune dereference expression, introduced by #57532

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929
Copy link
Contributor Author

924060929 commented Dec 3, 2025

run buildall

morrySnow
morrySnow previously approved these changes Dec 3, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 3, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 3, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 3, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Dec 3, 2025

TPC-H: Total hot run time: 34099 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 986f8cf76bda729871a12d51c3b95fdcaf8c2d0c, data reload: false

------ Round 1 ----------------------------------
q1	17626	4984	4889	4889
q2	2084	321	204	204
q3	10195	1311	741	741
q4	10242	871	312	312
q5	7514	2407	2116	2116
q6	183	168	136	136
q7	958	785	625	625
q8	9358	1441	1085	1085
q9	6951	5345	5324	5324
q10	6780	2179	1783	1783
q11	528	312	284	284
q12	341	369	218	218
q13	17756	3659	2998	2998
q14	243	236	208	208
q15	574	515	510	510
q16	887	865	809	809
q17	666	791	545	545
q18	7425	7087	7027	7027
q19	1091	965	602	602
q20	366	364	226	226
q21	3932	3382	2504	2504
q22	1057	1014	953	953
Total cold run time: 106757 ms
Total hot run time: 34099 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4962	4918	4903	4903
q2	338	400	328	328
q3	2142	2638	2276	2276
q4	1300	1751	1277	1277
q5	4178	4366	4648	4366
q6	219	184	129	129
q7	2043	1979	1863	1863
q8	2665	2580	2541	2541
q9	7452	7511	7463	7463
q10	3044	3276	2811	2811
q11	595	520	505	505
q12	706	773	633	633
q13	3447	3882	3349	3349
q14	310	322	302	302
q15	555	491	535	491
q16	925	910	869	869
q17	1207	1472	1434	1434
q18	8160	7708	7494	7494
q19	882	885	903	885
q20	2041	2086	1919	1919
q21	4891	4356	4125	4125
q22	1095	1076	975	975
Total cold run time: 53157 ms
Total hot run time: 50938 ms

@doris-robot
Copy link

doris-robot commented Dec 3, 2025

TPC-DS: Total hot run time: 181286 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 986f8cf76bda729871a12d51c3b95fdcaf8c2d0c, data reload: false

query1	1100	410	384	384
query2	6581	1185	1166	1166
query3	6743	228	229	228
query4	25234	23274	22480	22480
query5	5131	642	504	504
query6	344	235	221	221
query7	4653	512	310	310
query8	312	264	238	238
query9	8725	2615	2606	2606
query10	554	393	325	325
query11	15733	14990	15106	14990
query12	184	116	110	110
query13	1673	555	424	424
query14	9843	5865	5972	5865
query15	215	198	181	181
query16	7663	687	514	514
query17	1187	752	614	614
query18	2080	427	326	326
query19	216	198	168	168
query20	126	123	122	122
query21	216	136	117	117
query22	3895	4044	3911	3911
query23	32975	31809	32078	31809
query24	8391	2362	2401	2362
query25	607	519	469	469
query26	1246	279	168	168
query27	2734	488	335	335
query28	4372	2131	2172	2131
query29	810	618	475	475
query30	320	248	212	212
query31	828	700	618	618
query32	83	70	69	69
query33	603	366	330	330
query34	831	866	530	530
query35	789	847	752	752
query36	916	936	866	866
query37	130	117	92	92
query38	3829	3891	3811	3811
query39	1486	1409	1393	1393
query40	229	132	118	118
query41	65	61	62	61
query42	129	113	112	112
query43	447	444	427	427
query44	1319	758	749	749
query45	201	195	186	186
query46	908	1013	649	649
query47	1650	1730	1646	1646
query48	399	423	334	334
query49	788	487	402	402
query50	680	697	415	415
query51	3967	3991	3868	3868
query52	121	116	111	111
query53	250	262	197	197
query54	330	314	285	285
query55	97	103	94	94
query56	353	362	353	353
query57	1135	1168	1097	1097
query58	298	287	288	287
query59	2340	2622	2296	2296
query60	383	361	361	361
query61	214	184	191	184
query62	776	714	653	653
query63	234	199	194	194
query64	4670	1307	1015	1015
query65	4040	4001	3978	3978
query66	1118	462	361	361
query67	15320	14871	14938	14871
query68	8300	976	629	629
query69	537	356	331	331
query70	1085	1032	1002	1002
query71	458	362	318	318
query72	6052	4829	4853	4829
query73	682	576	343	343
query74	8833	8724	8657	8657
query75	3028	3032	2561	2561
query76	3327	1162	734	734
query77	525	403	315	315
query78	9458	9698	8795	8795
query79	2269	851	596	596
query80	644	573	485	485
query81	512	271	240	240
query82	493	141	115	115
query83	270	262	240	240
query84	266	121	107	107
query85	914	487	456	456
query86	387	311	279	279
query87	4014	4061	3974	3974
query88	4184	2257	2263	2257
query89	395	329	289	289
query90	2005	220	216	216
query91	171	171	147	147
query92	82	71	63	63
query93	1913	1017	655	655
query94	755	447	342	342
query95	505	405	412	405
query96	543	558	286	286
query97	2612	2678	2612	2612
query98	238	221	212	212
query99	1326	1397	1266	1266
Total cold run time: 272602 ms
Total hot run time: 181286 ms

@doris-robot
Copy link

doris-robot commented Dec 3, 2025

ClickBench: Total hot run time: 27.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 986f8cf76bda729871a12d51c3b95fdcaf8c2d0c, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.06	0.05
query3	0.26	0.09	0.09
query4	1.62	0.11	0.11
query5	0.26	0.24	0.25
query6	1.17	0.65	0.63
query7	0.03	0.03	0.02
query8	0.05	0.04	0.04
query9	0.59	0.50	0.50
query10	0.56	0.56	0.55
query11	0.17	0.11	0.11
query12	0.15	0.11	0.13
query13	0.62	0.59	0.60
query14	0.99	0.98	0.99
query15	0.83	0.79	0.79
query16	0.40	0.41	0.39
query17	1.04	1.04	1.08
query18	0.23	0.21	0.21
query19	1.91	1.84	1.78
query20	0.01	0.02	0.01
query21	15.45	0.28	0.14
query22	4.98	0.05	0.05
query23	16.08	0.27	0.10
query24	1.27	0.66	0.71
query25	0.09	0.10	0.04
query26	0.14	0.15	0.12
query27	0.06	0.06	0.05
query28	5.06	1.19	1.01
query29	12.56	4.02	3.20
query30	0.29	0.14	0.12
query31	2.81	0.62	0.38
query32	3.24	0.55	0.46
query33	3.04	3.04	3.01
query34	16.90	5.22	4.56
query35	4.59	4.50	4.52
query36	0.67	0.50	0.48
query37	0.11	0.07	0.07
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.15
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 98.84 s
Total hot run time: 27.54 s

@924060929
Copy link
Contributor Author

924060929 commented Dec 3, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 3, 2025
@924060929
Copy link
Contributor Author

924060929 commented Dec 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 3, 2025

TPC-H: Total hot run time: 34462 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 31421a8699316a7afde5865295e15c6f9fc17c34, data reload: false

------ Round 1 ----------------------------------
q1	17647	5143	4860	4860
q2	2065	313	212	212
q3	10249	1346	754	754
q4	10224	870	319	319
q5	7545	2522	2200	2200
q6	191	177	139	139
q7	972	805	645	645
q8	9361	1454	1102	1102
q9	7196	5370	5330	5330
q10	6859	2242	1817	1817
q11	535	309	300	300
q12	355	382	242	242
q13	17769	3705	3059	3059
q14	230	236	218	218
q15	586	519	511	511
q16	902	858	822	822
q17	675	764	543	543
q18	8136	7125	7005	7005
q19	1098	956	608	608
q20	363	357	237	237
q21	3990	3502	2584	2584
q22	1048	1033	955	955
Total cold run time: 107996 ms
Total hot run time: 34462 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5028	4994	4971	4971
q2	338	401	320	320
q3	2184	2674	2283	2283
q4	1362	1805	1342	1342
q5	4269	4721	4596	4596
q6	217	180	130	130
q7	2100	2029	1893	1893
q8	2713	2603	2557	2557
q9	7591	7492	7496	7492
q10	3097	3272	2899	2899
q11	590	536	506	506
q12	853	743	614	614
q13	3541	3927	3357	3357
q14	300	323	279	279
q15	563	514	502	502
q16	934	947	905	905
q17	1204	1384	1457	1384
q18	7928	7697	7581	7581
q19	908	858	896	858
q20	2079	2071	1957	1957
q21	5000	4401	4238	4238
q22	1090	1028	978	978
Total cold run time: 53889 ms
Total hot run time: 51642 ms

@doris-robot
Copy link

doris-robot commented Dec 3, 2025

TPC-DS: Total hot run time: 182710 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 31421a8699316a7afde5865295e15c6f9fc17c34, data reload: false

query1	1050	411	402	402
query2	6611	1186	1191	1186
query3	6741	237	226	226
query4	25575	23401	22773	22773
query5	4463	678	522	522
query6	346	255	237	237
query7	4647	546	320	320
query8	323	253	237	237
query9	8695	2674	2670	2670
query10	557	353	313	313
query11	15143	15109	14963	14963
query12	188	126	119	119
query13	1711	586	443	443
query14	9775	6100	6067	6067
query15	210	198	191	191
query16	7691	710	547	547
query17	1257	791	660	660
query18	2048	442	354	354
query19	224	218	193	193
query20	128	126	125	125
query21	222	140	112	112
query22	3982	3917	3766	3766
query23	33161	32180	32138	32138
query24	8498	2486	2503	2486
query25	648	562	505	505
query26	1251	289	179	179
query27	2702	526	353	353
query28	4355	2223	2199	2199
query29	835	661	552	552
query30	326	246	217	217
query31	874	723	632	632
query32	94	78	77	77
query33	626	407	394	394
query34	832	897	567	567
query35	808	836	739	739
query36	886	941	838	838
query37	126	114	91	91
query38	3890	3927	3776	3776
query39	1464	1422	1405	1405
query40	238	140	121	121
query41	76	63	62	62
query42	126	120	121	120
query43	447	461	422	422
query44	1352	780	771	771
query45	202	200	183	183
query46	906	1019	656	656
query47	1667	1715	1618	1618
query48	412	437	337	337
query49	782	495	422	422
query50	687	695	427	427
query51	3878	3948	3928	3928
query52	114	119	107	107
query53	264	271	199	199
query54	320	305	302	302
query55	98	95	95	95
query56	333	318	340	318
query57	1128	1150	1096	1096
query58	292	281	267	267
query59	2292	2422	2282	2282
query60	351	388	329	329
query61	160	154	159	154
query62	778	731	667	667
query63	240	199	204	199
query64	4460	1281	937	937
query65	4086	3947	3955	3947
query66	1107	456	349	349
query67	15126	15154	14797	14797
query68	8589	995	648	648
query69	524	371	314	314
query70	1109	976	981	976
query71	505	349	322	322
query72	5813	5091	4873	4873
query73	737	584	349	349
query74	8967	8890	8654	8654
query75	3666	3073	2548	2548
query76	3783	1148	767	767
query77	818	415	318	318
query78	9581	9609	8948	8948
query79	2224	836	588	588
query80	668	621	488	488
query81	513	273	245	245
query82	467	148	113	113
query83	274	264	267	264
query84	258	116	96	96
query85	953	497	445	445
query86	401	331	277	277
query87	4102	4156	3971	3971
query88	4049	2269	2253	2253
query89	390	338	295	295
query90	1952	220	215	215
query91	175	171	141	141
query92	85	70	68	68
query93	1810	1040	677	677
query94	730	459	354	354
query95	506	406	413	406
query96	546	564	285	285
query97	2638	2684	2627	2627
query98	249	214	220	214
query99	1432	1415	1285	1285
Total cold run time: 273399 ms
Total hot run time: 182710 ms

@doris-robot
Copy link

doris-robot commented Dec 3, 2025

ClickBench: Total hot run time: 27.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 31421a8699316a7afde5865295e15c6f9fc17c34, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.05	0.05
query3	0.25	0.09	0.08
query4	1.61	0.11	0.11
query5	0.28	0.26	0.25
query6	1.18	0.65	0.65
query7	0.04	0.03	0.02
query8	0.06	0.04	0.05
query9	0.57	0.52	0.50
query10	0.56	0.55	0.56
query11	0.16	0.10	0.11
query12	0.15	0.11	0.12
query13	0.62	0.61	0.61
query14	0.99	1.00	0.99
query15	0.81	0.80	0.80
query16	0.42	0.42	0.40
query17	1.00	1.03	1.08
query18	0.23	0.22	0.21
query19	1.87	1.94	1.78
query20	0.01	0.01	0.01
query21	15.44	0.30	0.14
query22	4.85	0.05	0.05
query23	16.02	0.28	0.10
query24	0.95	0.46	0.40
query25	0.09	0.08	0.04
query26	0.14	0.13	0.13
query27	0.06	0.04	0.06
query28	4.09	1.23	1.04
query29	12.66	4.09	3.25
query30	0.28	0.14	0.12
query31	2.82	0.62	0.41
query32	3.23	0.58	0.46
query33	3.10	3.05	3.05
query34	16.85	5.23	4.57
query35	4.53	4.55	4.56
query36	0.68	0.50	0.49
query37	0.10	0.07	0.07
query38	0.08	0.04	0.04
query39	0.05	0.03	0.04
query40	0.19	0.16	0.14
query41	0.08	0.03	0.04
query42	0.05	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 97.35 s
Total hot run time: 27.55 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 3, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 3, 2025

PR approved by at least one committer and no changes requested.

@924060929 924060929 merged commit 2b23693 into apache:master Dec 3, 2025
26 of 28 checks passed
@924060929 924060929 deleted the fix-prune-dereference branch December 3, 2025 10:04
@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 3, 2025

FE Regression Coverage Report

Increment line coverage 100.00% (3/3) 🎉
Increment coverage report
Complete coverage report

nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@924060929 @nagisa-kunhah
[fix](nereids) fix can not prune dereference expression (apache#58657)
99307b1 
fix can not prune dereference expression, introduced by apache#57532
924060929 added a commit that referenced this pull request Dec 23, 2025
@924060929
[fix](nereids) fix can not prune dereference expression (#58657)
645394e 
fix can not prune dereference expression, introduced by #57532

(cherry picked from commit 2b23693)
924060929 added a commit that referenced this pull request Dec 23, 2025
@924060929
[fix](nereids) fix can not prune dereference expression (#58657)
eac26e9 
fix can not prune dereference expression, introduced by #57532

(cherry picked from commit 2b23693)
@kaka11chen kaka11chen mentioned this pull request Dec 23, 2025
Merged
16 tasks
yiguolei pushed a commit that referenced this pull request Dec 24, 2025
@kaka11chen @924060929
@mrhhsg @eldenmoon
[feature](reader) Optimize Complex Type Column Reading with Column Pu…
eea2586 
…rning (#59286)

### What problem does this PR solve?

Problem Summary:

### Release note

Cherry-pick #58370 #58354 #59043 #58851 #58485 #58682 #58614 #58373
#57204 #58719 #58471 #58573 #58657

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---------

Co-authored-by: 924060929 <lanhuajian@selectdb.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: Jerry Hu <hushenggang@selectdb.com>
Co-authored-by: lihangyu <lihangyu@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@924060929 @Thearas @doris-robot @hello-stephen @starocean999 @morrySnow @yiguolei