[fix](maxcompute)Fix the NereidsException caused by non-existent table columns when pushing down mc predicates. #55635

hubgeter · 2025-09-03T11:15:42Z

What problem does this PR solve?

Problem Summary:
fix: maxcompute catalog push down predicates when column not exist in table.

Caused by: java.lang.NullPointerException: Cannot invoke "com.aliyun.odps.Column.getType()" because the return value of "java.util.Map.get(Object)" is null
        at org.apache.doris.datasource.maxcompute.source.MaxComputeScanNode.convertExprToOdpsPredicate(MaxComputeScanNode.java:422) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.maxcompute.source.MaxComputeScanNode.convertPredicate(MaxComputeScanNode.java:303) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.doFinalize(FileQueryScanNode.java:221) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.finalizeForNereids(FileQueryScanNode.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:285) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.splitFragments(NereidsPlanner.java:441) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.distribute(NereidsPlanner.java:527) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.lambda$plan$0(NereidsPlanner.java:149) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.planWithLock(NereidsPlanner.java:225) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:145) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.commands.insert.InsertOverwriteTableCommand.run(InsertOverwriteTableCommand.java:133) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:767) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 12 more

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…e columns when pushing down mc predicates.

Thearas · 2025-09-03T11:15:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

hubgeter · 2025-09-03T11:15:53Z

run buildall

doris-robot · 2025-09-03T11:46:51Z

TPC-H: Total hot run time: 34108 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cb3cf9b10b36be3e0fadaff6873dd8b8ff69fa0e, data reload: false

------ Round 1 ----------------------------------
q1	17642	5250	5121	5121
q2	2014	320	209	209
q3	10269	1260	722	722
q4	10229	1021	549	549
q5	7556	2487	2334	2334
q6	190	166	136	136
q7	921	758	646	646
q8	9377	1335	1124	1124
q9	6923	5081	5091	5081
q10	6935	2380	1989	1989
q11	484	302	284	284
q12	358	369	227	227
q13	17810	3692	3037	3037
q14	233	256	239	239
q15	580	506	488	488
q16	435	426	386	386
q17	597	864	363	363
q18	7415	7199	7035	7035
q19	1350	965	572	572
q20	351	337	243	243
q21	3832	3191	2326	2326
q22	1072	1040	997	997
Total cold run time: 106573 ms
Total hot run time: 34108 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5262	5169	5183	5169
q2	248	343	231	231
q3	2156	2662	2327	2327
q4	1331	1772	1320	1320
q5	4229	4340	4584	4340
q6	229	180	133	133
q7	2121	1893	1899	1893
q8	2711	2711	2611	2611
q9	7331	7250	7404	7250
q10	3129	3262	3016	3016
q11	620	545	502	502
q12	711	815	654	654
q13	3610	3984	3277	3277
q14	287	359	281	281
q15	533	481	478	478
q16	449	503	669	503
q17	1193	1573	1416	1416
q18	8007	7479	7742	7479
q19	859	808	882	808
q20	1916	2094	2027	2027
q21	5173	4471	4468	4468
q22	1076	1058	1007	1007
Total cold run time: 53181 ms
Total hot run time: 51190 ms

doris-robot · 2025-09-03T11:58:31Z

TPC-DS: Total hot run time: 186742 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cb3cf9b10b36be3e0fadaff6873dd8b8ff69fa0e, data reload: false

query1	1038	488	394	394
query2	6597	1721	1712	1712
query3	6747	232	225	225
query4	26271	23573	23237	23237
query5	4390	663	527	527
query6	348	248	244	244
query7	4652	519	318	318
query8	307	262	253	253
query9	8657	2930	2955	2930
query10	468	335	289	289
query11	15963	15275	15049	15049
query12	179	119	118	118
query13	1665	581	437	437
query14	9553	5775	5768	5768
query15	217	189	175	175
query16	7494	670	482	482
query17	1244	784	650	650
query18	2060	450	354	354
query19	220	208	182	182
query20	136	128	126	126
query21	217	137	121	121
query22	4056	4199	4026	4026
query23	34037	33024	32770	32770
query24	8156	2389	2425	2389
query25	615	552	486	486
query26	1250	355	164	164
query27	2640	518	351	351
query28	4347	2331	2235	2235
query29	779	611	488	488
query30	294	219	195	195
query31	914	792	731	731
query32	91	80	83	80
query33	574	399	352	352
query34	852	874	527	527
query35	835	818	781	781
query36	992	1037	924	924
query37	121	114	97	97
query38	4073	4018	4052	4018
query39	1521	1419	1437	1419
query40	226	137	130	130
query41	67	62	62	62
query42	131	118	115	115
query43	530	523	460	460
query44	1361	874	868	868
query45	188	178	175	175
query46	862	1016	651	651
query47	1765	1825	1737	1737
query48	397	424	339	339
query49	764	521	422	422
query50	657	696	403	403
query51	4129	4222	4136	4136
query52	118	119	102	102
query53	243	266	202	202
query54	621	599	547	547
query55	95	86	92	86
query56	361	332	332	332
query57	1245	1212	1151	1151
query58	297	284	291	284
query59	2647	2685	2596	2596
query60	391	355	343	343
query61	171	164	169	164
query62	800	732	669	669
query63	232	200	205	200
query64	4534	1162	824	824
query65	4367	4208	4151	4151
query66	1145	449	358	358
query67	15522	15242	15002	15002
query68	8789	949	593	593
query69	494	337	299	299
query70	1261	1177	1142	1142
query71	577	348	338	338
query72	5865	4906	4960	4906
query73	728	604	364	364
query74	9236	9138	8965	8965
query75	4273	3071	2617	2617
query76	3703	1221	771	771
query77	813	428	350	350
query78	9591	9785	8882	8882
query79	2418	904	602	602
query80	674	584	517	517
query81	477	275	223	223
query82	417	139	117	117
query83	295	272	244	244
query84	363	123	98	98
query85	881	463	426	426
query86	348	323	314	314
query87	4353	4317	4178	4178
query88	3103	2232	2247	2232
query89	424	342	292	292
query90	1955	231	236	231
query91	171	169	137	137
query92	91	76	71	71
query93	1217	1009	658	658
query94	701	416	327	327
query95	421	342	339	339
query96	499	590	286	286
query97	2701	2715	2578	2578
query98	252	224	225	224
query99	1483	1431	1288	1288
Total cold run time: 277010 ms
Total hot run time: 186742 ms

doris-robot · 2025-09-03T12:04:11Z

ClickBench: Total hot run time: 32.77 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cb3cf9b10b36be3e0fadaff6873dd8b8ff69fa0e, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.06
query3	0.25	0.08	0.08
query4	1.60	0.11	0.11
query5	0.47	0.43	0.41
query6	1.18	0.65	0.64
query7	0.03	0.03	0.03
query8	0.06	0.04	0.04
query9	0.60	0.54	0.53
query10	0.59	0.58	0.58
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.64	0.61
query14	0.80	0.84	0.86
query15	0.87	0.87	0.87
query16	0.39	0.43	0.39
query17	1.08	1.03	1.06
query18	0.23	0.20	0.20
query19	1.97	1.83	1.82
query20	0.02	0.02	0.01
query21	15.40	0.96	0.57
query22	0.81	1.08	0.87
query23	14.81	1.37	0.62
query24	6.63	1.59	0.60
query25	0.47	0.17	0.19
query26	0.67	0.16	0.14
query27	0.06	0.06	0.05
query28	10.18	0.98	0.43
query29	12.56	3.94	3.25
query30	3.10	3.08	3.00
query31	2.82	0.58	0.40
query32	3.24	0.56	0.46
query33	3.07	3.20	3.09
query34	16.10	5.46	4.83
query35	4.88	4.90	4.94
query36	0.73	0.51	0.50
query37	0.11	0.08	0.07
query38	0.06	0.05	0.05
query39	0.05	0.03	0.02
query40	0.18	0.16	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 107.32 s
Total hot run time: 32.77 s

hello-stephen · 2025-09-03T12:20:29Z

FE UT Coverage Report

Increment line coverage 0.00% (0/7) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-09-03T14:13:03Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/7) 🎉
Increment coverage report
Complete coverage report

github-actions · 2025-09-06T06:07:21Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-09-06T06:07:23Z

PR approved by anyone and no changes requested.

…e columns when pushing down mc predicates. (#55635) ### What problem does this PR solve? Problem Summary: fix: maxcompute catalog push down predicates when column not exist in table. ``` Caused by: java.lang.NullPointerException: Cannot invoke "com.aliyun.odps.Column.getType()" because the return value of "java.util.Map.get(Object)" is null at org.apache.doris.datasource.maxcompute.source.MaxComputeScanNode.convertExprToOdpsPredicate(MaxComputeScanNode.java:422) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.maxcompute.source.MaxComputeScanNode.convertPredicate(MaxComputeScanNode.java:303) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.FileQueryScanNode.doFinalize(FileQueryScanNode.java:221) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.FileQueryScanNode.finalizeForNereids(FileQueryScanNode.java:213) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:285) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.NereidsPlanner.splitFragments(NereidsPlanner.java:441) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.NereidsPlanner.distribute(NereidsPlanner.java:527) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.NereidsPlanner.lambda$plan$0(NereidsPlanner.java:149) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.NereidsPlanner.planWithLock(NereidsPlanner.java:225) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:145) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.trees.plans.commands.insert.InsertOverwriteTableCommand.run(InsertOverwriteTableCommand.java:133) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:767) ~[doris-fe.jar:1.2-SNAPSHOT] ... 12 more ```

…e columns when pushing down mc predicates. (apache#55635) ### What problem does this PR solve? Problem Summary: fix: maxcompute catalog push down predicates when column not exist in table. ``` Caused by: java.lang.NullPointerException: Cannot invoke "com.aliyun.odps.Column.getType()" because the return value of "java.util.Map.get(Object)" is null at org.apache.doris.datasource.maxcompute.source.MaxComputeScanNode.convertExprToOdpsPredicate(MaxComputeScanNode.java:422) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.maxcompute.source.MaxComputeScanNode.convertPredicate(MaxComputeScanNode.java:303) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.FileQueryScanNode.doFinalize(FileQueryScanNode.java:221) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.FileQueryScanNode.finalizeForNereids(FileQueryScanNode.java:213) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:285) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.NereidsPlanner.splitFragments(NereidsPlanner.java:441) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.NereidsPlanner.distribute(NereidsPlanner.java:527) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.NereidsPlanner.lambda$plan$0(NereidsPlanner.java:149) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.NereidsPlanner.planWithLock(NereidsPlanner.java:225) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:145) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.trees.plans.commands.insert.InsertOverwriteTableCommand.run(InsertOverwriteTableCommand.java:133) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:767) ~[doris-fe.jar:1.2-SNAPSHOT] ... 12 more ```

…xistent table columns when pushing down mc predicates. #55635 (#55746) Cherry-picked from #55635 Co-authored-by: daidai <changyuwei@selectdb.com>

…xistent table columns when pushing down mc predicates. #55635 (#55744) Cherry-picked from #55635 Co-authored-by: daidai <changyuwei@selectdb.com>

…57474) ### What problem does this PR solve? Problem Summary: This PR is a continuation of PR #55635. Based on the current investigation, the conclusion in PR #55635 appears to be incorrect — the reason why the predicate column cannot be found in the columnMap remains unclear. To help with further debugging, this PR adds additional log information. In addition, during the `refresh table` process, the MaxComputeExternalTable object remains unchanged, so the objects it holds also stay the same. When initializing the table schema, the map maintained by this object may have repeated `put` operations. So, move this map to `MaxComputeSchemaCacheValue`, which better matches its lifecycle and avoids repeated updates during table refresh.

…pache#57474) ### What problem does this PR solve? Problem Summary: This PR is a continuation of PR apache#55635. Based on the current investigation, the conclusion in PR apache#55635 appears to be incorrect — the reason why the predicate column cannot be found in the columnMap remains unclear. To help with further debugging, this PR adds additional log information. In addition, during the `refresh table` process, the MaxComputeExternalTable object remains unchanged, so the objects it holds also stay the same. When initializing the table schema, the map maintained by this object may have repeated `put` operations. So, move this map to `MaxComputeSchemaCacheValue`, which better matches its lifecycle and avoids repeated updates during table refresh.

[fix](maxcompute)Fix the NereidsException caused by non-existent tabl…

cb3cf9b

…e columns when pushing down mc predicates.

morningman added dev/2.1.x dev/3.0.x dev/3.1.x labels Sep 3, 2025

morningman approved these changes Sep 6, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 6, 2025

github-actions bot added the reviewed label Sep 6, 2025

CalvinKirs approved these changes Sep 6, 2025

View reviewed changes

morningman merged commit b68bb92 into apache:master Sep 7, 2025
30 of 32 checks passed

github-actions bot mentioned this pull request Sep 7, 2025

branch-3.0: [fix](maxcompute)Fix the NereidsException caused by non-existent table columns when pushing down mc predicates. #55635 #55744

Merged

github-actions bot mentioned this pull request Sep 7, 2025

branch-2.1: [fix](maxcompute)Fix the NereidsException caused by non-existent table columns when pushing down mc predicates. #55635 #55745

Open

github-actions bot mentioned this pull request Sep 7, 2025

branch-3.1: [fix](maxcompute)Fix the NereidsException caused by non-existent table columns when pushing down mc predicates. #55635 #55746

Merged

morrySnow added dev/3.1.1-merged and removed dev/3.1.x labels Sep 8, 2025

dataroaring added the dev/3.0.9-merged label Sep 9, 2025

dataroaring removed the dev/3.0.x label Sep 9, 2025

morrySnow mentioned this pull request Sep 22, 2025

Release Note 3.1.1 #56289

Closed

hubgeter mentioned this pull request Oct 30, 2025

[fix](maxcompute)fix maxcompute push down predicates null pointer. #57474

Merged

16 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[fix](maxcompute)Fix the NereidsException caused by non-existent table columns when pushing down mc predicates. #55635

[fix](maxcompute)Fix the NereidsException caused by non-existent table columns when pushing down mc predicates. #55635

Uh oh!

hubgeter commented Sep 3, 2025

Uh oh!

Thearas commented Sep 3, 2025

Uh oh!

hubgeter commented Sep 3, 2025

Uh oh!

doris-robot commented Sep 3, 2025

Uh oh!

doris-robot commented Sep 3, 2025

Uh oh!

doris-robot commented Sep 3, 2025

Uh oh!

hello-stephen commented Sep 3, 2025

Uh oh!

hello-stephen commented Sep 3, 2025

Uh oh!

github-actions bot commented Sep 6, 2025

Uh oh!

github-actions bot commented Sep 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

[fix](maxcompute)Fix the NereidsException caused by non-existent table columns when pushing down mc predicates. #55635

[fix](maxcompute)Fix the NereidsException caused by non-existent table columns when pushing down mc predicates. #55635

Uh oh!

Conversation

hubgeter commented Sep 3, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Sep 3, 2025

Uh oh!

hubgeter commented Sep 3, 2025

Uh oh!

doris-robot commented Sep 3, 2025

Uh oh!

doris-robot commented Sep 3, 2025

Uh oh!

doris-robot commented Sep 3, 2025

Uh oh!

hello-stephen commented Sep 3, 2025

FE UT Coverage Report

Uh oh!

hello-stephen commented Sep 3, 2025

FE Regression Coverage Report

Uh oh!

github-actions bot commented Sep 6, 2025

Uh oh!

github-actions bot commented Sep 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants