Skip to content

[opt](orc-reader)Turn on late materialization of orc complex types. #52079

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jun 26, 2025
Merged

[opt](orc-reader)Turn on late materialization of orc complex types. #52079

morrySnow merged 1 commit into from
Jun 26, 2025

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Jun 20, 2025

What problem does this PR solve?

Problem Summary:

Release note

Cherry-pick #49718

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@kaka11chen kaka11chen requested a review from morrySnow as a code owner June 20, 2025 11:45
@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen kaka11chen marked this pull request as draft June 20, 2025 11:46
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Jun 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 20, 2025

TPC-H: Total hot run time: 39916 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7bff73ba73cfb5126bddc9193841bd964a328ad6, data reload: false

------ Round 1 ----------------------------------
q1	17582	7030	6661	6661
q2	2187	171	182	171
q3	11855	1165	1221	1165
q4	11405	747	753	747
q5	7836	2918	2932	2918
q6	223	139	134	134
q7	978	627	618	618
q8	9691	1956	2055	1956
q9	6627	6400	6432	6400
q10	6962	2262	2295	2262
q11	464	258	255	255
q12	393	211	208	208
q13	17781	2981	2974	2974
q14	229	209	217	209
q15	506	474	456	456
q16	507	379	378	378
q17	983	555	584	555
q18	7286	6677	6565	6565
q19	1387	1090	1049	1049
q20	488	197	197	197
q21	3898	3070	3182	3070
q22	1091	968	999	968
Total cold run time: 110359 ms
Total hot run time: 39916 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6671	6553	6608	6553
q2	326	232	233	232
q3	2905	2866	2996	2866
q4	2102	1826	1782	1782
q5	5652	5635	5724	5635
q6	200	121	120	120
q7	2174	1786	1833	1786
q8	3315	3466	3502	3466
q9	8843	8899	8841	8841
q10	3575	3503	3503	3503
q11	597	485	505	485
q12	832	618	628	618
q13	7290	3125	3150	3125
q14	301	261	284	261
q15	504	457	470	457
q16	494	431	418	418
q17	1815	1613	1605	1605
q18	8277	7611	7694	7611
q19	1717	1605	1566	1566
q20	2089	1841	1859	1841
q21	5314	5027	4814	4814
q22	1095	985	955	955
Total cold run time: 66088 ms
Total hot run time: 58540 ms

@doris-robot
Copy link

doris-robot commented Jun 20, 2025

TPC-DS: Total hot run time: 190359 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7bff73ba73cfb5126bddc9193841bd964a328ad6, data reload: false

query1	978	370	354	354
query2	6506	1924	1886	1886
query3	6704	218	222	218
query4	33645	23793	23833	23793
query5	4347	444	430	430
query6	258	181	180	180
query7	4626	309	316	309
query8	283	224	222	222
query9	9599	2557	2550	2550
query10	472	269	274	269
query11	18463	15138	15163	15138
query12	155	102	100	100
query13	1630	433	406	406
query14	9557	7513	7563	7513
query15	253	168	184	168
query16	8196	483	469	469
query17	1673	590	562	562
query18	2165	313	319	313
query19	380	156	159	156
query20	119	108	108	108
query21	201	104	104	104
query22	4315	4419	4087	4087
query23	34236	33177	33500	33177
query24	11681	2837	2811	2811
query25	683	423	417	417
query26	1624	168	173	168
query27	2854	348	343	343
query28	7714	2084	2087	2084
query29	1006	463	446	446
query30	318	157	164	157
query31	1026	777	781	777
query32	100	60	60	60
query33	806	308	309	308
query34	911	504	539	504
query35	855	714	715	714
query36	1061	898	939	898
query37	140	72	67	67
query38	3997	3783	3785	3783
query39	1642	1408	1442	1408
query40	285	100	103	100
query41	55	51	51	51
query42	114	103	103	103
query43	504	476	488	476
query44	1268	794	786	786
query45	189	173	172	172
query46	1168	719	689	689
query47	1950	1822	1854	1822
query48	475	387	379	379
query49	1250	401	399	399
query50	806	419	419	419
query51	7253	7120	7095	7095
query52	99	93	90	90
query53	257	185	191	185
query54	1320	487	476	476
query55	79	80	80	80
query56	291	266	252	252
query57	1278	1161	1150	1150
query58	239	220	222	220
query59	3110	2974	2959	2959
query60	277	262	260	260
query61	113	110	109	109
query62	861	654	678	654
query63	224	189	197	189
query64	5218	644	651	644
query65	3297	3190	3181	3181
query66	1439	314	302	302
query67	15975	15440	15641	15440
query68	4820	570	570	570
query69	443	277	264	264
query70	1160	1064	1101	1064
query71	331	256	273	256
query72	6444	4046	4026	4026
query73	773	348	346	346
query74	10574	8963	9067	8963
query75	3358	2676	2665	2665
query76	2926	934	1197	934
query77	419	275	268	268
query78	10593	9685	9509	9509
query79	1696	591	597	591
query80	1191	433	422	422
query81	522	226	220	220
query82	943	87	91	87
query83	219	144	164	144
query84	241	76	79	76
query85	1322	305	283	283
query86	400	300	293	293
query87	4344	4234	4213	4213
query88	3644	2400	2352	2352
query89	409	299	287	287
query90	2028	184	185	184
query91	187	148	147	147
query92	62	48	50	48
query93	1439	555	547	547
query94	915	279	288	279
query95	363	260	257	257
query96	599	301	297	297
query97	3369	3153	3134	3134
query98	212	199	198	198
query99	1559	1372	1296	1296
Total cold run time: 302078 ms
Total hot run time: 190359 ms

@doris-robot
Copy link

doris-robot commented Jun 20, 2025

ClickBench: Total hot run time: 29.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7bff73ba73cfb5126bddc9193841bd964a328ad6, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.11	0.11
query5	0.53	0.49	0.52
query6	1.12	0.72	0.73
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.56	0.51	0.49
query10	0.55	0.54	0.56
query11	0.14	0.10	0.13
query12	0.13	0.11	0.10
query13	0.60	0.60	0.60
query14	0.76	0.79	0.80
query15	0.83	0.82	0.83
query16	0.37	0.38	0.40
query17	1.04	1.00	1.01
query18	0.23	0.22	0.21
query19	1.89	1.83	1.87
query20	0.01	0.01	0.01
query21	15.42	0.59	0.56
query22	2.29	2.05	1.56
query23	17.06	0.93	0.92
query24	2.94	0.69	2.13
query25	0.26	0.14	0.07
query26	0.50	0.13	0.14
query27	0.06	0.04	0.05
query28	9.88	0.53	0.50
query29	12.61	3.26	3.22
query30	0.24	0.06	0.06
query31	2.85	0.38	0.39
query32	3.27	0.46	0.45
query33	2.97	2.97	3.05
query34	17.05	4.48	4.46
query35	4.52	4.52	4.54
query36	0.65	0.47	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.83 s
Total hot run time: 29.66 s

@doris-robot
Copy link

doris-robot commented Jun 20, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.14% (10901/26496)
Line Coverage 31.95% (93266/291956)
Region Coverage 31.01% (48089/155053)
Branch Coverage 27.49% (24637/89628)

@kaka11chen
[opt](orc-reader)Turn on late materialization of orc complex types. (a…
ba98e06 
…pache#49718)

Related PR: apache#45966

[opt] (orc-reader) Turn on late materialization of orc complex types.

After implementing the new merge io function in apache#45966 to adapt the
complex type delayed materialization and the need to backtrack to solve
the reading characteristics, turn on the late materialization of orc
complex types in orc reader.
@kaka11chen kaka11chen force-pushed the cherry-pick-49718_3.1 branch from 7bff73b to ba98e06 Compare June 25, 2025 07:13
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Jun 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-H: Total hot run time: 39810 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba98e06877a8ec1e4b2bbbbd65cf0a33b625008b, data reload: false

------ Round 1 ----------------------------------
q1	17611	6818	6656	6656
q2	2421	197	197	197
q3	11819	1167	1239	1167
q4	11249	751	685	685
q5	7720	2907	2861	2861
q6	219	131	133	131
q7	984	622	610	610
q8	9336	1944	2030	1944
q9	6654	6383	6419	6383
q10	7011	2245	2361	2245
q11	468	245	257	245
q12	406	216	207	207
q13	17774	2939	2953	2939
q14	229	211	217	211
q15	506	461	465	461
q16	453	389	385	385
q17	972	555	580	555
q18	7376	6775	6703	6703
q19	1349	988	985	985
q20	478	196	201	196
q21	4084	3039	3196	3039
q22	1094	1009	1005	1005
Total cold run time: 110213 ms
Total hot run time: 39810 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6623	6533	6507	6507
q2	336	224	231	224
q3	2918	2774	2927	2774
q4	2033	1862	1774	1774
q5	5722	5709	5704	5704
q6	204	124	123	123
q7	2174	1805	1769	1769
q8	3326	3484	3479	3479
q9	8862	8825	8871	8825
q10	3602	3579	3531	3531
q11	594	489	497	489
q12	831	603	612	603
q13	8862	3139	3154	3139
q14	304	277	283	277
q15	492	467	466	466
q16	494	423	426	423
q17	1838	1591	1586	1586
q18	8251	7778	7678	7678
q19	1709	1609	1578	1578
q20	2151	1861	1889	1861
q21	5167	4947	4957	4947
q22	1092	1015	1022	1015
Total cold run time: 67585 ms
Total hot run time: 58772 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 26, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 42.38% (11535/27215)
Line Coverage 33.37% (100219/300362)
Region Coverage 32.66% (52194/159811)
Branch Coverage 29.72% (28031/94304)

@kaka11chen kaka11chen marked this pull request as ready for review June 26, 2025 06:27
@morrySnow morrySnow merged commit ea3cb31 into apache:branch-3.1 Jun 26, 2025
18 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kaka11chen @hello-stephen @doris-robot @morrySnow