Skip to content

branch-3.0: [fix](maxcompute)Fix the NereidsException caused by non-existent table columns when pushing down mc predicates. #55635 #55744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Sep 9, 2025
Merged

branch-3.0: [fix](maxcompute)Fix the NereidsException caused by non-existent table columns when pushing down mc predicates. #55635 #55744

dataroaring merged 1 commit into from
Sep 9, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Sep 7, 2025

Cherry-picked from #55635

@hubgeter
[fix](maxcompute)Fix the NereidsException caused by non-existent tabl…
6c5f6f4 
…e columns when pushing down mc predicates. (#55635)

### What problem does this PR solve?

Problem Summary:
fix: maxcompute catalog push down predicates when column not exist in
table.
```
Caused by: java.lang.NullPointerException: Cannot invoke "com.aliyun.odps.Column.getType()" because the return value of "java.util.Map.get(Object)" is null
        at org.apache.doris.datasource.maxcompute.source.MaxComputeScanNode.convertExprToOdpsPredicate(MaxComputeScanNode.java:422) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.maxcompute.source.MaxComputeScanNode.convertPredicate(MaxComputeScanNode.java:303) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.doFinalize(FileQueryScanNode.java:221) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.finalizeForNereids(FileQueryScanNode.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:285) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.splitFragments(NereidsPlanner.java:441) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.distribute(NereidsPlanner.java:527) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.lambda$plan$0(NereidsPlanner.java:149) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.planWithLock(NereidsPlanner.java:225) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:145) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.commands.insert.InsertOverwriteTableCommand.run(InsertOverwriteTableCommand.java:133) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:767) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 12 more 
```
@Thearas
Copy link
Contributor

Thearas commented Sep 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Sep 7, 2025
@dataroaring dataroaring reopened this Sep 7, 2025
@Thearas
Copy link
Contributor

Thearas commented Sep 7, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 7, 2025

TPC-H: Total hot run time: 39935 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c5f6f456904d9349d54e4d996b824893bb06efd, data reload: false

------ Round 1 ----------------------------------
q1	17579	6821	6616	6616
q2	2046	203	164	164
q3	10496	1134	1140	1134
q4	10228	755	722	722
q5	7724	2959	2859	2859
q6	210	131	134	131
q7	969	623	615	615
q8	9379	1924	2063	1924
q9	6753	6430	6437	6430
q10	7035	2306	2260	2260
q11	474	262	262	262
q12	405	222	219	219
q13	17799	2971	2972	2971
q14	232	209	210	209
q15	505	461	461	461
q16	500	376	382	376
q17	987	584	562	562
q18	7390	6754	6627	6627
q19	1392	1091	1055	1055
q20	479	198	194	194
q21	3879	3163	3314	3163
q22	1106	981	998	981
Total cold run time: 107567 ms
Total hot run time: 39935 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6613	6568	6562	6562
q2	333	230	236	230
q3	2884	2906	2929	2906
q4	2014	1835	1810	1810
q5	5746	5717	5767	5717
q6	214	131	129	129
q7	2265	1784	1820	1784
q8	3395	3570	3565	3565
q9	8878	9001	8883	8883
q10	3571	3551	3535	3535
q11	599	503	507	503
q12	810	602	578	578
q13	8356	3162	3157	3157
q14	315	265	275	265
q15	521	467	464	464
q16	488	439	426	426
q17	1846	1622	1612	1612
q18	8347	7794	7749	7749
q19	1679	1560	1479	1479
q20	2068	1904	1887	1887
q21	5310	5028	5049	5028
q22	1207	1086	1055	1055
Total cold run time: 67459 ms
Total hot run time: 59324 ms

@doris-robot
Copy link

doris-robot commented Sep 7, 2025

TPC-DS: Total hot run time: 193324 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c5f6f456904d9349d54e4d996b824893bb06efd, data reload: false

query1	942	411	383	383
query2	7026	1967	1912	1912
query3	8682	199	188	188
query4	33916	23900	23751	23751
query5	3456	454	433	433
query6	273	177	168	168
query7	4202	313	339	313
query8	301	223	223	223
query9	9387	2579	2581	2579
query10	484	285	261	261
query11	17837	15563	15454	15454
query12	153	103	100	100
query13	1550	409	407	407
query14	8564	7377	7444	7377
query15	254	170	175	170
query16	8074	495	509	495
query17	1595	617	591	591
query18	2246	334	322	322
query19	286	181	168	168
query20	124	123	117	117
query21	210	111	116	111
query22	4910	4671	4747	4671
query23	34879	34148	34500	34148
query24	11911	2873	2934	2873
query25	728	438	442	438
query26	1817	177	168	168
query27	2873	355	371	355
query28	7623	2156	2176	2156
query29	1087	462	447	447
query30	271	159	162	159
query31	1066	854	845	845
query32	114	84	60	60
query33	793	300	312	300
query34	1038	509	525	509
query35	866	741	725	725
query36	1140	954	980	954
query37	263	69	65	65
query38	4126	3952	3963	3952
query39	1534	1674	1502	1502
query40	261	103	101	101
query41	55	48	49	48
query42	111	100	100	100
query43	524	466	490	466
query44	1268	818	823	818
query45	191	185	182	182
query46	1160	724	749	724
query47	2025	1955	1943	1943
query48	473	379	379	379
query49	1131	385	402	385
query50	833	430	446	430
query51	7398	7186	7325	7186
query52	107	87	88	87
query53	260	188	181	181
query54	1365	466	473	466
query55	82	76	83	76
query56	271	250	249	249
query57	1327	1212	1198	1198
query58	229	209	205	205
query59	3272	3064	3039	3039
query60	281	261	254	254
query61	110	109	137	109
query62	900	685	697	685
query63	221	193	190	190
query64	4943	646	642	642
query65	3393	3293	3371	3293
query66	1529	298	332	298
query67	16460	15543	15582	15543
query68	5731	565	553	553
query69	436	264	268	264
query70	1167	1081	1062	1062
query71	349	252	270	252
query72	6194	4060	3864	3864
query73	740	347	353	347
query74	10466	8957	8957	8957
query75	3390	2622	2634	2622
query76	3271	1055	1040	1040
query77	391	274	269	269
query78	10602	9620	9588	9588
query79	1699	585	612	585
query80	1067	421	417	417
query81	533	214	227	214
query82	628	86	84	84
query83	227	141	140	140
query84	236	77	78	77
query85	1326	309	291	291
query86	400	296	306	296
query87	4428	4300	4253	4253
query88	3573	2414	2352	2352
query89	407	291	291	291
query90	1977	185	183	183
query91	182	150	150	150
query92	63	50	52	50
query93	1761	551	542	542
query94	904	292	287	287
query95	361	256	263	256
query96	614	281	277	277
query97	3325	3154	3155	3154
query98	210	198	199	198
query99	1508	1303	1282	1282
Total cold run time: 304947 ms
Total hot run time: 193324 ms

@doris-robot
Copy link

doris-robot commented Sep 7, 2025

ClickBench: Total hot run time: 28.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c5f6f456904d9349d54e4d996b824893bb06efd, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.06	0.06
query4	1.63	0.10	0.11
query5	0.53	0.50	0.54
query6	1.12	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.55	0.52	0.49
query10	0.56	0.56	0.58
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.59	0.60
query14	0.78	0.80	0.82
query15	0.84	0.83	0.82
query16	0.38	0.38	0.40
query17	1.03	1.00	1.04
query18	0.23	0.23	0.22
query19	1.89	1.84	1.83
query20	0.01	0.02	0.02
query21	15.38	0.60	0.59
query22	2.32	2.29	0.82
query23	17.03	0.88	0.75
query24	3.61	1.49	0.47
query25	0.24	0.18	0.04
query26	0.42	0.13	0.13
query27	0.04	0.05	0.04
query28	10.03	0.53	0.50
query29	12.62	3.17	3.18
query30	0.25	0.08	0.06
query31	2.86	0.39	0.38
query32	3.27	0.46	0.45
query33	3.00	3.00	3.02
query34	17.10	4.53	4.52
query35	4.59	4.61	4.58
query36	0.68	0.48	0.47
query37	0.08	0.07	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.76 s
Total hot run time: 28.67 s

dataroaring
dataroaring approved these changes Sep 9, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 090091a into branch-3.0 Sep 9, 2025
25 of 26 checks passed
@github-actions github-actions bot deleted the auto-pick-55635-branch-3.0 branch September 9, 2025 02:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @dataroaring @hubgeter