Skip to content

[fix](nereids) fix prune map type cause backend core #58573

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
924060929 merged 3 commits into from
Dec 2, 2025
Merged

[fix](nereids) fix prune map type cause backend core #58573

924060929 merged 3 commits into from
Dec 2, 2025

Conversation

@924060929
Copy link
Contributor

@924060929 924060929 commented Dec 1, 2025 •
edited
Loading

What problem does this PR solve?

fix prune map type cause backend core, when the map type is changed, we should not prune the nested column type, introduced by #57204

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 1, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929
Copy link
Contributor Author

924060929 commented Dec 1, 2025

run buildall

@924060929
Copy link
Contributor Author

924060929 commented Dec 1, 2025

run buildall

@924060929
Copy link
Contributor Author

924060929 commented Dec 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 1, 2025

TPC-H: Total hot run time: 35768 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ced77873b3049be43563e0d15acf96276a4b5f77, data reload: false

------ Round 1 ----------------------------------
q1	17602	5008	4975	4975
q2	2040	346	205	205
q3	10222	1340	732	732
q4	10235	925	334	334
q5	7525	2501	2153	2153
q6	187	169	136	136
q7	983	820	645	645
q8	9345	1418	1078	1078
q9	7122	5342	5341	5341
q10	6835	2185	1774	1774
q11	539	308	297	297
q12	371	373	233	233
q13	17783	3723	3080	3080
q14	247	233	213	213
q15	601	530	514	514
q16	905	860	806	806
q17	605	743	574	574
q18	7514	7524	7994	7524
q19	1221	1004	593	593
q20	380	362	244	244
q21	4265	3853	3282	3282
q22	1108	1190	1035	1035
Total cold run time: 107635 ms
Total hot run time: 35768 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5250	5001	5165	5001
q2	328	429	342	342
q3	2386	2933	2556	2556
q4	1489	1876	1430	1430
q5	4570	4688	4357	4357
q6	209	166	126	126
q7	2103	1985	1809	1809
q8	2629	2538	2487	2487
q9	7594	7441	7641	7441
q10	3103	3223	2834	2834
q11	591	510	510	510
q12	701	786	619	619
q13	3521	3783	3070	3070
q14	264	274	254	254
q15	539	501	492	492
q16	867	891	854	854
q17	1114	1363	1388	1363
q18	7436	7138	7167	7138
q19	779	749	740	740
q20	1960	1974	1790	1790
q21	4617	4303	4189	4189
q22	1099	1000	974	974
Total cold run time: 53149 ms
Total hot run time: 50376 ms

@doris-robot
Copy link

doris-robot commented Dec 1, 2025

TPC-DS: Total hot run time: 181994 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ced77873b3049be43563e0d15acf96276a4b5f77, data reload: false

query1	1079	408	396	396
query2	6592	1190	1196	1190
query3	6747	232	226	226
query4	25480	23465	22932	22932
query5	4675	643	529	529
query6	337	240	231	231
query7	4664	539	306	306
query8	315	249	231	231
query9	8741	2656	2635	2635
query10	513	352	304	304
query11	15683	15270	14898	14898
query12	176	116	111	111
query13	1678	547	436	436
query14	9594	5941	5999	5941
query15	205	198	183	183
query16	7479	726	540	540
query17	1207	765	646	646
query18	2013	421	324	324
query19	206	200	174	174
query20	135	129	127	127
query21	220	139	114	114
query22	3860	3930	3854	3854
query23	33172	31995	31818	31818
query24	8518	2381	2367	2367
query25	595	505	455	455
query26	1249	278	167	167
query27	2745	527	342	342
query28	4320	2160	2140	2140
query29	811	635	501	501
query30	301	244	218	218
query31	825	705	623	623
query32	89	76	81	76
query33	602	403	335	335
query34	821	899	537	537
query35	784	842	738	738
query36	913	921	821	821
query37	120	113	88	88
query38	3868	3874	3844	3844
query39	1498	1411	1414	1411
query40	223	131	124	124
query41	68	63	64	63
query42	127	118	113	113
query43	454	470	403	403
query44	1265	758	750	750
query45	204	193	189	189
query46	864	993	633	633
query47	1683	1737	1652	1652
query48	397	415	329	329
query49	777	499	406	406
query50	664	689	418	418
query51	3863	3943	3866	3866
query52	112	115	111	111
query53	245	259	196	196
query54	339	310	297	297
query55	99	97	96	96
query56	341	356	348	348
query57	1138	1183	1105	1105
query58	312	287	282	282
query59	2345	2420	2287	2287
query60	389	357	357	357
query61	198	193	198	193
query62	794	737	664	664
query63	231	201	202	201
query64	4699	1314	986	986
query65	4086	3986	3959	3959
query66	1167	478	369	369
query67	15267	15080	14884	14884
query68	8372	968	634	634
query69	530	364	329	329
query70	1070	1080	994	994
query71	451	339	333	333
query72	6062	4961	4942	4942
query73	678	572	343	343
query74	8780	8954	8564	8564
query75	3238	3030	2536	2536
query76	3425	1133	732	732
query77	584	394	307	307
query78	9443	9734	8898	8898
query79	2399	821	582	582
query80	645	576	517	517
query81	525	265	241	241
query82	462	138	118	118
query83	284	263	245	245
query84	255	120	105	105
query85	918	492	445	445
query86	390	304	278	278
query87	4091	4043	3968	3968
query88	4121	2290	2287	2287
query89	386	336	299	299
query90	1946	230	222	222
query91	166	167	142	142
query92	85	72	65	65
query93	1927	989	657	657
query94	762	452	362	362
query95	511	435	412	412
query96	521	550	286	286
query97	2632	2734	2564	2564
query98	244	220	217	217
query99	1398	1428	1280	1280
Total cold run time: 272482 ms
Total hot run time: 181994 ms

@doris-robot
Copy link

doris-robot commented Dec 1, 2025

ClickBench: Total hot run time: 27.49 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ced77873b3049be43563e0d15acf96276a4b5f77, data reload: false

query1	0.06	0.04	0.04
query2	0.10	0.05	0.06
query3	0.25	0.09	0.08
query4	1.61	0.11	0.12
query5	0.28	0.27	0.26
query6	1.17	0.64	0.66
query7	0.04	0.03	0.03
query8	0.05	0.04	0.04
query9	0.57	0.52	0.49
query10	0.56	0.56	0.56
query11	0.16	0.10	0.11
query12	0.15	0.11	0.11
query13	0.63	0.62	0.61
query14	1.00	0.97	0.99
query15	0.82	0.80	0.82
query16	0.40	0.40	0.39
query17	1.04	1.00	1.06
query18	0.24	0.25	0.21
query19	1.95	1.88	1.85
query20	0.01	0.01	0.02
query21	15.44	0.28	0.15
query22	4.62	0.05	0.05
query23	15.99	0.27	0.11
query24	1.65	1.30	0.27
query25	0.09	0.07	0.05
query26	0.14	0.13	0.15
query27	0.08	0.04	0.06
query28	4.97	1.19	1.04
query29	12.59	3.95	3.27
query30	0.28	0.14	0.13
query31	2.82	0.60	0.40
query32	3.24	0.55	0.47
query33	3.03	3.06	3.13
query34	16.93	5.16	4.54
query35	4.55	4.59	4.58
query36	0.66	0.50	0.49
query37	0.10	0.06	0.06
query38	0.06	0.04	0.04
query39	0.05	0.03	0.04
query40	0.17	0.14	0.13
query41	0.08	0.03	0.04
query42	0.04	0.04	0.03
query43	0.04	0.04	0.04
Total cold run time: 98.71 s
Total hot run time: 27.49 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 1, 2025

FE Regression Coverage Report

Increment line coverage 0.00% (0/29) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 2, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2025

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 2, 2025

FE Regression Coverage Report

Increment line coverage 0.00% (0/29) 🎉
Increment coverage report
Complete coverage report

@924060929 924060929 merged commit 1d7f6c4 into apache:master Dec 2, 2025
27 of 28 checks passed
@924060929 924060929 deleted the fix-prune-map branch December 2, 2025 06:31
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@924060929 @nagisa-kunhah
[fix](nereids) fix prune map type cause backend core (apache#58573)
f2c4858 
fix prune map type cause backend core, when the map type is changed, we
should not prune the nested column type, introduced by apache#57204
924060929 added a commit that referenced this pull request Dec 23, 2025
@924060929
[fix](nereids) fix prune map type cause backend core (#58573)
f61d01c 
fix prune map type cause backend core, when the map type is changed, we
should not prune the nested column type, introduced by #57204

(cherry picked from commit 1d7f6c4)
924060929 added a commit that referenced this pull request Dec 23, 2025
@924060929
[fix](nereids) fix prune map type cause backend core (#58573)
4a29af8 
fix prune map type cause backend core, when the map type is changed, we
should not prune the nested column type, introduced by #57204

(cherry picked from commit 1d7f6c4)
@kaka11chen kaka11chen mentioned this pull request Dec 23, 2025
Merged
16 tasks
yiguolei pushed a commit that referenced this pull request Dec 24, 2025
@kaka11chen @924060929
@mrhhsg @eldenmoon
[feature](reader) Optimize Complex Type Column Reading with Column Pu…
eea2586 
…rning (#59286)

### What problem does this PR solve?

Problem Summary:

### Release note

Cherry-pick #58370 #58354 #59043 #58851 #58485 #58682 #58614 #58373
#57204 #58719 #58471 #58573 #58657

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---------

Co-authored-by: 924060929 <lanhuajian@selectdb.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: Jerry Hu <hushenggang@selectdb.com>
Co-authored-by: lihangyu <lihangyu@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@924060929 @hello-stephen @doris-robot @starocean999 @morrySnow @yiguolei