Skip to content

[fix](paimon) Fix paimon native reader don't use late materialization. #55690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Sep 8, 2025
Merged

[fix](paimon) Fix paimon native reader don't use late materialization. #55690

morningman merged 1 commit into from
Sep 8, 2025

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Sep 4, 2025 •
edited by morningman
Loading

bp #55894

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 4, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen
Copy link
Contributor Author

kaka11chen commented Sep 4, 2025

run buildall

@kaka11chen kaka11chen changed the title [fix](paimon) Fix paimon native table don't use late materialization. [fix](paimon) Fix paimon native reader don't use late materialization. Sep 4, 2025
@doris-robot
Copy link

doris-robot commented Sep 4, 2025

TPC-H: Total hot run time: 39670 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8e0afd275d364b368da98bdc5cbe2f71628c3f9b, data reload: false

------ Round 1 ----------------------------------
q1	17895	6904	6698	6698
q2	2037	187	162	162
q3	10663	1108	1148	1108
q4	10516	692	762	692
q5	7770	2814	2835	2814
q6	207	130	131	130
q7	952	630	599	599
q8	9351	1931	2039	1931
q9	6627	6367	6370	6367
q10	7030	2207	2309	2207
q11	476	267	268	267
q12	400	215	220	215
q13	17781	2996	2990	2990
q14	249	206	214	206
q15	518	464	459	459
q16	477	375	374	374
q17	969	576	608	576
q18	7254	6601	6522	6522
q19	1403	1132	1005	1005
q20	488	202	205	202
q21	3863	3226	3178	3178
q22	1145	970	968	968
Total cold run time: 108071 ms
Total hot run time: 39670 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6625	6625	6529	6529
q2	324	231	236	231
q3	2855	2894	2875	2875
q4	2053	1883	1809	1809
q5	5708	5767	5727	5727
q6	211	127	133	127
q7	2299	1818	1829	1818
q8	3392	3483	3543	3483
q9	8760	8867	8878	8867
q10	3557	3545	3545	3545
q11	589	512	498	498
q12	808	580	569	569
q13	9072	3225	3181	3181
q14	290	270	288	270
q15	526	464	479	464
q16	507	459	443	443
q17	1838	1652	1638	1638
q18	8527	7713	7861	7713
q19	1680	1554	1546	1546
q20	2080	1887	1874	1874
q21	5216	5000	5044	5000
q22	1138	1067	1024	1024
Total cold run time: 68055 ms
Total hot run time: 59231 ms

@doris-robot
Copy link

doris-robot commented Sep 4, 2025

BE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 42.08% (11242/26713)
Line Coverage 32.59% (96209/295168)
Region Coverage 30.54% (55253/180938)
Branch Coverage 26.86% (27346/101820)

@doris-robot
Copy link

doris-robot commented Sep 4, 2025

TPC-DS: Total hot run time: 191721 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8e0afd275d364b368da98bdc5cbe2f71628c3f9b, data reload: false

query1	924	401	395	395
query2	6254	1882	1968	1882
query3	8686	206	211	206
query4	33768	23893	23368	23368
query5	3644	470	445	445
query6	290	171	176	171
query7	4230	308	308	308
query8	279	219	232	219
query9	9351	2596	2575	2575
query10	474	285	262	262
query11	17709	15064	15102	15064
query12	162	102	104	102
query13	1585	436	433	433
query14	9042	7016	6583	6583
query15	264	169	183	169
query16	8032	491	502	491
query17	1634	643	591	591
query18	2185	335	346	335
query19	308	160	192	160
query20	125	119	115	115
query21	215	105	105	105
query22	4678	4427	4338	4338
query23	35004	34261	34355	34261
query24	12274	2943	2923	2923
query25	716	420	436	420
query26	1812	179	182	179
query27	3256	355	376	355
query28	7504	2125	2172	2125
query29	1099	464	462	462
query30	272	165	165	165
query31	1076	810	850	810
query32	105	60	58	58
query33	775	313	330	313
query34	1021	541	522	522
query35	870	718	746	718
query36	1124	922	958	922
query37	262	66	69	66
query38	4080	3974	4025	3974
query39	1518	1513	1473	1473
query40	280	100	99	99
query41	51	51	49	49
query42	118	109	111	109
query43	513	485	502	485
query44	1301	810	811	810
query45	202	187	178	178
query46	1159	740	749	740
query47	2017	1917	1906	1906
query48	516	379	388	379
query49	1089	401	397	397
query50	849	447	442	442
query51	7466	7308	7217	7217
query52	101	93	89	89
query53	259	184	188	184
query54	1397	477	482	477
query55	86	76	82	76
query56	279	247	256	247
query57	1324	1208	1207	1207
query58	225	213	218	213
query59	3250	3103	3049	3049
query60	307	271	282	271
query61	111	113	106	106
query62	868	718	703	703
query63	217	191	192	191
query64	4933	668	623	623
query65	3407	3283	3339	3283
query66	1349	300	307	300
query67	15930	15472	15722	15472
query68	4931	582	598	582
query69	435	263	272	263
query70	1168	1094	1047	1047
query71	349	259	254	254
query72	6162	3954	4071	3954
query73	743	343	362	343
query74	9541	8951	9173	8951
query75	3358	2689	2651	2651
query76	2786	1041	1161	1041
query77	369	282	274	274
query78	10419	9570	9520	9520
query79	2121	614	602	602
query80	1098	432	443	432
query81	532	220	221	220
query82	614	89	89	89
query83	229	144	144	144
query84	241	77	78	77
query85	1385	294	305	294
query86	461	303	288	288
query87	4486	4269	4226	4226
query88	3729	2440	2410	2410
query89	418	293	282	282
query90	2011	188	187	187
query91	182	167	145	145
query92	65	47	48	47
query93	2468	562	579	562
query94	972	299	304	299
query95	363	261	261	261
query96	612	290	280	280
query97	3331	3125	3127	3125
query98	212	198	198	198
query99	1524	1308	1361	1308
Total cold run time: 303526 ms
Total hot run time: 191721 ms

@doris-robot
Copy link

doris-robot commented Sep 4, 2025

ClickBench: Total hot run time: 30.31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8e0afd275d364b368da98bdc5cbe2f71628c3f9b, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.64	0.10	0.10
query5	0.50	0.53	0.52
query6	1.14	0.74	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.56	0.55	0.55
query11	0.14	0.11	0.10
query12	0.14	0.10	0.11
query13	0.62	0.59	0.59
query14	0.78	0.81	0.78
query15	0.84	0.81	0.81
query16	0.37	0.40	0.41
query17	1.01	1.06	1.00
query18	0.24	0.24	0.22
query19	1.93	1.87	1.85
query20	0.01	0.01	0.01
query21	15.38	0.60	0.58
query22	2.47	2.26	1.88
query23	17.04	0.95	0.84
query24	3.60	1.03	0.99
query25	0.26	0.05	0.06
query26	0.48	0.14	0.15
query27	0.06	0.05	0.05
query28	10.18	0.50	0.50
query29	12.58	3.23	3.22
query30	0.25	0.06	0.06
query31	2.86	0.38	0.39
query32	3.24	0.46	0.46
query33	3.00	2.99	3.07
query34	17.26	4.49	4.52
query35	4.58	4.51	4.58
query36	0.65	0.47	0.47
query37	0.08	0.07	0.06
query38	0.04	0.03	0.04
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.24 s
Total hot run time: 30.31 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 4, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.23% (19716/26206)
Line Coverage 68.43% (201079/293853)
Region Coverage 66.45% (120298/181042)
Branch Coverage 59.70% (60983/102144)

@liutang123
Copy link
Contributor

liutang123 commented Sep 5, 2025

We test this PR in our cluster, the query time has been reduced from 1 minute 20 seconds to 20 seconds.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 6, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 6, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 6, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Sep 7, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit f36c233 into apache:branch-3.0 Sep 8, 2025
32 of 34 checks passed
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Sep 11, 2025
@kaka11chen
[fix](paimon) Fix paimon native reader don't use late materialization. (
7baf2de 
apache#55690)
This was referenced Sep 11, 2025
Merged
Merged
morningman pushed a commit that referenced this pull request Sep 11, 2025
@kaka11chen
[fix](paimon) Fix paimon native reader don't use late materialization. (
61eed5b 
#55894)

Cherry-pick #55690
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@kaka11chen @hello-stephen @doris-robot @liutang123 @morningman @dataroaring