Skip to content

branch-3.1: [fix](maxcompute)Fix the NereidsException caused by non-existent table columns when pushing down mc predicates. #55635 #55746

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Sep 8, 2025
Merged

branch-3.1: [fix](maxcompute)Fix the NereidsException caused by non-existent table columns when pushing down mc predicates. #55635 #55746

morrySnow merged 1 commit into from
Sep 8, 2025
+36 −8

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Sep 7, 2025

Cherry-picked from #55635

@hubgeter
[fix](maxcompute)Fix the NereidsException caused by non-existent tabl…
4196fec 
…e columns when pushing down mc predicates. (#55635)

### What problem does this PR solve?

Problem Summary:
fix: maxcompute catalog push down predicates when column not exist in
table.
```
Caused by: java.lang.NullPointerException: Cannot invoke "com.aliyun.odps.Column.getType()" because the return value of "java.util.Map.get(Object)" is null
        at org.apache.doris.datasource.maxcompute.source.MaxComputeScanNode.convertExprToOdpsPredicate(MaxComputeScanNode.java:422) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.maxcompute.source.MaxComputeScanNode.convertPredicate(MaxComputeScanNode.java:303) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.doFinalize(FileQueryScanNode.java:221) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.finalizeForNereids(FileQueryScanNode.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:285) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.splitFragments(NereidsPlanner.java:441) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.distribute(NereidsPlanner.java:527) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.lambda$plan$0(NereidsPlanner.java:149) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.planWithLock(NereidsPlanner.java:225) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:145) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.commands.insert.InsertOverwriteTableCommand.run(InsertOverwriteTableCommand.java:133) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:767) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 12 more 
```
@github-actions github-actions bot requested a review from morrySnow as a code owner September 7, 2025 04:25
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Sep 7, 2025
@dataroaring dataroaring reopened this Sep 7, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 7, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 7, 2025

TPC-H: Total hot run time: 32386 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4196fec707af87a482378bcc71b0894d7e9422fb, data reload: false

------ Round 1 ----------------------------------
q1	17585	5411	5374	5374
q2	2016	385	284	284
q3	12102	1248	724	724
q4	10241	854	453	453
q5	8475	2364	2094	2094
q6	180	162	131	131
q7	887	755	606	606
q8	9321	1425	1144	1144
q9	5233	4973	4937	4937
q10	6783	2269	1828	1828
q11	479	281	262	262
q12	335	347	208	208
q13	17774	3598	2980	2980
q14	230	226	215	215
q15	545	459	457	457
q16	405	428	366	366
q17	604	849	379	379
q18	6913	6366	6288	6288
q19	1217	973	534	534
q20	318	339	205	205
q21	2775	2163	1961	1961
q22	1056	1025	956	956
Total cold run time: 105474 ms
Total hot run time: 32386 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5448	5436	5471	5436
q2	234	328	237	237
q3	2240	2634	2308	2308
q4	1341	1773	1334	1334
q5	4361	4864	4977	4864
q6	174	163	132	132
q7	2075	2016	1870	1870
q8	2670	2883	2720	2720
q9	7480	7467	7360	7360
q10	3075	3375	2753	2753
q11	572	513	515	513
q12	679	795	664	664
q13	3414	3792	3213	3213
q14	280	295	274	274
q15	514	465	478	465
q16	430	505	456	456
q17	1231	1732	1240	1240
q18	7526	7400	7391	7391
q19	792	1010	1125	1010
q20	2013	2046	1865	1865
q21	5251	4978	4564	4564
q22	1063	1036	978	978
Total cold run time: 52863 ms
Total hot run time: 51647 ms

@doris-robot
Copy link

doris-robot commented Sep 7, 2025

TPC-DS: Total hot run time: 191564 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4196fec707af87a482378bcc71b0894d7e9422fb, data reload: false

query1	928	396	410	396
query2	6210	2000	1924	1924
query3	8686	197	192	192
query4	33872	23647	23466	23466
query5	3661	603	440	440
query6	291	189	174	174
query7	4202	504	311	311
query8	310	245	236	236
query9	9577	2625	2613	2613
query10	468	323	262	262
query11	18016	15480	15206	15206
query12	153	111	104	104
query13	1555	535	412	412
query14	8797	7751	7023	7023
query15	229	189	179	179
query16	8155	674	512	512
query17	1610	780	637	637
query18	2208	429	342	342
query19	236	207	176	176
query20	129	125	120	120
query21	209	132	107	107
query22	4546	4629	4578	4578
query23	34785	34498	33713	33713
query24	7235	2648	2666	2648
query25	561	503	446	446
query26	830	298	172	172
query27	1928	484	355	355
query28	5199	2225	2187	2187
query29	722	613	460	460
query30	242	194	170	170
query31	1012	926	819	819
query32	95	61	58	58
query33	494	382	340	340
query34	737	870	520	520
query35	784	806	717	717
query36	1016	1054	969	969
query37	108	99	67	67
query38	3955	3978	3963	3963
query39	1514	1457	1456	1456
query40	211	121	104	104
query41	46	46	51	46
query42	126	104	103	103
query43	521	499	488	488
query44	1300	822	827	822
query45	191	182	171	171
query46	900	1086	674	674
query47	1962	2053	1920	1920
query48	420	425	346	346
query49	730	493	407	407
query50	681	728	422	422
query51	7336	7460	7234	7234
query52	107	105	93	93
query53	234	260	192	192
query54	558	553	473	473
query55	87	82	75	75
query56	280	262	242	242
query57	1293	1253	1205	1205
query58	234	216	224	216
query59	3077	3188	3121	3121
query60	300	273	267	267
query61	108	107	117	107
query62	823	760	685	685
query63	234	190	197	190
query64	4141	1040	647	647
query65	3352	3307	3257	3257
query66	779	420	302	302
query67	16566	15951	15621	15621
query68	8025	821	533	533
query69	487	302	263	263
query70	1224	1148	1117	1117
query71	428	301	264	264
query72	5185	3724	3877	3724
query73	656	744	350	350
query74	10492	9100	8825	8825
query75	3930	3144	2700	2700
query76	3566	1175	771	771
query77	770	365	268	268
query78	10255	10441	9454	9454
query79	5185	865	592	592
query80	631	524	425	425
query81	469	267	218	218
query82	234	118	86	86
query83	164	155	143	143
query84	284	95	77	77
query85	743	357	290	290
query86	347	319	304	304
query87	4324	4271	4164	4164
query88	4350	2384	2371	2371
query89	428	339	293	293
query90	2072	186	188	186
query91	134	147	106	106
query92	65	59	53	53
query93	3625	911	548	548
query94	661	408	293	293
query95	346	277	266	266
query96	491	625	284	284
query97	3176	3280	3150	3150
query98	210	213	203	203
query99	1464	1421	1343	1343
Total cold run time: 296283 ms
Total hot run time: 191564 ms

@doris-robot
Copy link

doris-robot commented Sep 7, 2025

ClickBench: Total hot run time: 29.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4196fec707af87a482378bcc71b0894d7e9422fb, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.11
query5	0.51	0.50	0.50
query6	1.20	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.03	0.03
query9	0.57	0.51	0.49
query10	0.55	0.55	0.55
query11	0.15	0.11	0.09
query12	0.14	0.11	0.11
query13	0.62	0.61	0.60
query14	0.77	0.79	0.79
query15	0.85	0.83	0.81
query16	0.37	0.38	0.40
query17	1.06	1.01	0.98
query18	0.24	0.23	0.23
query19	1.97	1.83	1.86
query20	0.01	0.01	0.02
query21	15.39	0.94	0.57
query22	0.73	0.80	0.57
query23	15.23	1.45	0.53
query24	2.70	1.76	1.79
query25	0.15	0.20	0.05
query26	0.20	0.14	0.13
query27	0.04	0.05	0.04
query28	14.47	0.95	0.42
query29	12.61	3.93	3.29
query30	0.25	0.09	0.06
query31	2.81	0.57	0.39
query32	3.24	0.54	0.46
query33	3.00	3.06	3.05
query34	16.72	5.21	4.55
query35	4.49	4.63	4.51
query36	0.63	0.50	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.03	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.16 s
Total hot run time: 29.53 s

@morrySnow morrySnow merged commit 7ad22cd into branch-3.1 Sep 8, 2025
23 checks passed
@github-actions github-actions bot deleted the auto-pick-55635-branch-3.1 branch September 8, 2025 02:56
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @morrySnow @dataroaring @hubgeter