Skip to content

[fix](parquet)fix parquet topn lazy mat complex data error result #58785

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 3 commits into from
Dec 16, 2025
Merged

[fix](parquet)fix parquet topn lazy mat complex data error result #58785

morningman merged 3 commits into from
Dec 16, 2025

Conversation

@hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Dec 7, 2025 •
edited
Loading

What problem does this PR solve?

Related PR: #51329
Problem Summary:
This PR primarily enables the Parquet reader to use page indexes when reading complex columns, and also fixes a data reading error in PR #51329 when materializing data based on row number in the second stage of topn.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter hubgeter closed this Dec 8, 2025
@hubgeter hubgeter reopened this Dec 8, 2025
@hubgeter
Copy link
Contributor Author

hubgeter commented Dec 8, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 8, 2025

TPC-H: Total hot run time: 34889 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d701319b71c7519418e4805880fa50ea44176a8f, data reload: false

------ Round 1 ----------------------------------
q1	17631	5285	4978	4978
q2	2056	322	200	200
q3	10241	1297	757	757
q4	10220	859	321	321
q5	7517	2223	2269	2223
q6	206	179	140	140
q7	943	840	639	639
q8	9375	1618	1291	1291
q9	7198	5445	5414	5414
q10	6861	2217	1814	1814
q11	549	325	295	295
q12	369	379	238	238
q13	17802	3734	3025	3025
q14	241	233	216	216
q15	597	524	517	517
q16	899	892	808	808
q17	672	849	499	499
q18	7455	7160	7052	7052
q19	1100	962	641	641
q20	394	361	224	224
q21	4058	3486	2608	2608
q22	1045	1018	989	989
Total cold run time: 107429 ms
Total hot run time: 34889 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5053	5024	5155	5024
q2	322	398	313	313
q3	2197	2743	2281	2281
q4	1347	1815	1330	1330
q5	4660	4550	4748	4550
q6	231	186	142	142
q7	2072	1990	1778	1778
q8	2676	2570	2582	2570
q9	7558	7510	7570	7510
q10	3078	3260	2849	2849
q11	644	564	513	513
q12	783	885	605	605
q13	3555	3840	3560	3560
q14	282	310	288	288
q15	567	514	507	507
q16	897	958	893	893
q17	1313	1547	1471	1471
q18	8092	7601	7445	7445
q19	951	846	907	846
q20	2010	2110	1930	1930
q21	4919	4230	4126	4126
q22	1118	1052	996	996
Total cold run time: 54325 ms
Total hot run time: 51527 ms

@doris-robot
Copy link

doris-robot commented Dec 8, 2025

TPC-DS: Total hot run time: 179870 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d701319b71c7519418e4805880fa50ea44176a8f, data reload: false

query5	4367	607	497	497
query6	349	228	213	213
query7	4666	482	293	293
query8	336	253	247	247
query9	8741	2663	2669	2663
query10	526	321	278	278
query11	15378	14816	14630	14630
query12	187	119	122	119
query13	1692	488	387	387
query14	5903	3371	3118	3118
query14_1	2950	3036	2941	2941
query15	216	199	183	183
query16	7494	484	454	454
query17	1230	706	608	608
query18	2029	434	331	331
query19	213	194	166	166
query20	134	119	118	118
query21	209	138	117	117
query22	3851	3930	3873	3873
query23	16582	16113	15865	15865
query23_1	16032	16011	15967	15967
query24	7187	1646	1244	1244
query24_1	1225	1225	1254	1225
query25	662	522	458	458
query26	1258	294	171	171
query27	2899	470	314	314
query28	4418	2212	2193	2193
query29	863	597	476	476
query30	320	239	220	220
query31	834	705	605	605
query32	89	74	75	74
query33	696	358	312	312
query34	869	914	529	529
query35	815	847	735	735
query36	901	911	840	840
query37	133	97	81	81
query38	3797	3856	3780	3780
query39	767	753	715	715
query39_1	707	711	710	710
query40	234	135	123	123
query41	74	68	68	68
query42	126	102	101	101
query43	449	439	401	401
query44	1352	795	784	784
query45	202	200	184	184
query46	911	988	624	624
query47	1700	1731	1584	1584
query48	429	324	240	240
query49	810	452	370	370
query50	688	313	232	232
query51	3849	3931	3849	3849
query52	116	99	89	89
query53	236	237	185	185
query54	340	286	270	270
query55	97	81	79	79
query56	371	317	318	317
query57	1163	1150	1117	1117
query58	307	266	281	266
query59	2379	2436	2421	2421
query60	388	330	323	323
query61	216	211	149	149
query62	798	685	636	636
query63	232	180	185	180
query64	4623	1195	902	902
query65	4032	3965	3961	3961
query66	1232	439	342	342
query67	15179	14978	14830	14830
query68	8227	952	666	666
query69	508	297	271	271
query70	1073	994	965	965
query71	465	302	272	272
query72	6010	4966	4919	4919
query73	665	541	299	299
query74	8671	8781	8522	8522
query75	3435	3044	2548	2548
query76	3522	1150	756	756
query77	788	390	296	296
query78	9406	9504	8828	8828
query79	1643	845	576	576
query80	648	559	464	464
query81	487	269	236	236
query82	442	126	108	108
query83	280	276	264	264
query84	254	122	98	98
query85	942	542	462	462
query86	348	310	299	299
query87	4006	4039	3984	3984
query88	3920	2159	2151	2151
query89	406	331	298	298
query90	2062	160	160	160
query91	174	167	145	145
query92	78	67	65	65
query93	1336	1056	694	694
query94	769	318	293	293
query95	573	381	333	333
query96	530	499	215	215
query97	2636	2657	2602	2602
query98	259	208	199	199
query99	1378	1313	1242	1242
Total cold run time: 270179 ms
Total hot run time: 179870 ms

@doris-robot
Copy link

doris-robot commented Dec 8, 2025

ClickBench: Total hot run time: 27.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d701319b71c7519418e4805880fa50ea44176a8f, data reload: false

query1	0.05	0.05	0.04
query2	0.11	0.05	0.04
query3	0.26	0.09	0.09
query4	1.62	0.10	0.11
query5	0.28	0.24	0.26
query6	1.16	0.63	0.62
query7	0.04	0.03	0.02
query8	0.06	0.04	0.04
query9	0.56	0.52	0.51
query10	0.55	0.56	0.56
query11	0.15	0.10	0.11
query12	0.15	0.12	0.11
query13	0.61	0.61	0.60
query14	0.99	0.98	0.99
query15	0.82	0.79	0.78
query16	0.42	0.38	0.42
query17	0.96	1.00	1.02
query18	0.24	0.21	0.21
query19	1.93	1.88	1.90
query20	0.02	0.01	0.01
query21	15.43	0.32	0.14
query22	4.64	0.05	0.05
query23	16.12	0.28	0.10
query24	1.53	0.29	0.47
query25	0.08	0.08	0.09
query26	0.15	0.13	0.12
query27	0.06	0.04	0.06
query28	4.39	1.24	1.03
query29	12.62	3.97	3.21
query30	0.29	0.14	0.11
query31	2.84	0.63	0.38
query32	3.23	0.56	0.46
query33	3.00	3.04	3.01
query34	17.05	5.24	4.51
query35	4.61	4.61	4.57
query36	0.67	0.51	0.49
query37	0.12	0.07	0.07
query38	0.07	0.05	0.04
query39	0.05	0.03	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.04
query42	0.04	0.04	0.03
query43	0.04	0.04	0.03
Total cold run time: 98.26 s
Total hot run time: 27.26 s

@hubgeter
Copy link
Contributor Author

hubgeter commented Dec 8, 2025

run buildall

@hubgeter
Copy link
Contributor Author

hubgeter commented Dec 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 9, 2025

TPC-DS: Total hot run time: 182047 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb1123a0b27c387511194df4d111e57549ebea02, data reload: false

query5	5769	621	484	484
query6	320	236	208	208
query7	4220	453	269	269
query8	313	241	229	229
query9	8765	2630	2633	2630
query10	587	366	315	315
query11	15620	15054	14515	14515
query12	187	120	112	112
query13	1260	502	391	391
query14	6932	3240	2946	2946
query14_1	2888	2896	2991	2896
query15	211	202	183	183
query16	910	474	475	474
query17	1194	675	558	558
query18	2697	431	342	342
query19	224	219	202	202
query20	125	112	114	112
query21	223	130	116	116
query22	4004	3932	3826	3826
query23	17349	16812	16402	16402
query23_1	16544	16417	16323	16323
query24	7553	1675	1377	1377
query24_1	1274	1312	1284	1284
query25	589	505	410	410
query26	1248	277	172	172
query27	3368	481	324	324
query28	4741	2261	2235	2235
query29	864	609	481	481
query30	352	261	213	213
query31	844	762	657	657
query32	118	78	69	69
query33	548	353	310	310
query34	951	937	579	579
query35	820	852	760	760
query36	936	925	863	863
query37	160	93	81	81
query38	4182	4187	4226	4187
query39	753	739	720	720
query39_1	701	706	692	692
query40	228	135	114	114
query41	62	59	57	57
query42	101	99	101	99
query43	424	442	408	408
query44	1319	759	770	759
query45	198	186	182	182
query46	910	968	608	608
query47	1648	1704	1610	1610
query48	323	341	248	248
query49	621	427	334	334
query50	672	283	215	215
query51	3827	3868	3823	3823
query52	106	110	92	92
query53	309	351	294	294
query54	285	264	248	248
query55	71	73	72	72
query56	290	292	319	292
query57	1185	1150	1066	1066
query58	276	256	250	250
query59	2328	2353	2271	2271
query60	313	314	289	289
query61	159	155	149	149
query62	691	667	627	627
query63	327	295	295	295
query64	4943	1294	1010	1010
query65	4014	3958	3927	3927
query66	1359	427	324	324
query67	15556	14939	15093	14939
query68	4819	1067	767	767
query69	504	349	308	308
query70	1105	1041	924	924
query71	371	323	289	289
query72	5691	5077	5015	5015
query73	666	581	304	304
query74	8879	8796	8658	8658
query75	3552	3491	3136	3136
query76	3795	1128	737	737
query77	516	381	292	292
query78	9483	9737	8742	8742
query79	1097	870	615	615
query80	730	672	563	563
query81	505	270	233	233
query82	416	130	103	103
query83	260	261	230	230
query84	260	120	95	95
query85	887	497	473	473
query86	335	295	273	273
query87	3965	4013	3924	3924
query88	3187	2269	2298	2269
query89	467	414	384	384
query90	2005	155	147	147
query91	175	170	142	142
query92	68	60	61	60
query93	1090	913	575	575
query94	525	317	276	276
query95	562	330	362	330
query96	587	467	208	208
query97	2599	2660	2577	2577
query98	208	193	190	190
query99	1282	1323	1182	1182
Total cold run time: 263063 ms
Total hot run time: 182047 ms

@doris-robot
Copy link

doris-robot commented Dec 9, 2025

ClickBench: Total hot run time: 27.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fb1123a0b27c387511194df4d111e57549ebea02, data reload: false

query1	0.06	0.04	0.04
query2	0.10	0.05	0.05
query3	0.26	0.09	0.08
query4	1.62	0.11	0.11
query5	0.27	0.24	0.27
query6	1.17	0.64	0.62
query7	0.03	0.02	0.03
query8	0.06	0.04	0.04
query9	0.57	0.51	0.51
query10	0.56	0.54	0.55
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.62	0.60
query14	0.99	0.98	0.98
query15	0.81	0.81	0.81
query16	0.39	0.41	0.41
query17	1.01	0.98	1.06
query18	0.23	0.21	0.22
query19	1.94	1.88	1.90
query20	0.02	0.01	0.02
query21	15.43	0.27	0.15
query22	4.95	0.05	0.05
query23	16.10	0.28	0.10
query24	1.31	0.35	0.35
query25	0.10	0.07	0.06
query26	0.16	0.14	0.13
query27	0.04	0.06	0.07
query28	3.48	1.22	1.02
query29	12.60	3.99	3.24
query30	0.28	0.14	0.12
query31	2.82	0.63	0.39
query32	3.23	0.54	0.47
query33	3.05	3.04	3.09
query34	16.85	5.20	4.57
query35	4.56	4.58	4.59
query36	0.68	0.50	0.49
query37	0.12	0.07	0.06
query38	0.07	0.04	0.04
query39	0.04	0.02	0.03
query40	0.17	0.14	0.14
query41	0.09	0.04	0.03
query42	0.04	0.04	0.04
query43	0.04	0.04	0.03
Total cold run time: 97.23 s
Total hot run time: 27.53 s

@hubgeter
Copy link
Contributor Author

hubgeter commented Dec 9, 2025

run buildall

@hubgeter
Copy link
Contributor Author

hubgeter commented Dec 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

TPC-H: Total hot run time: 35339 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 938350d1e209eb06be005978fb6b0bffd6a765cb, data reload: false

------ Round 1 ----------------------------------
q1	17631	4168	4069	4069
q2	2061	351	234	234
q3	10207	1317	742	742
q4	10214	861	320	320
q5	7514	2171	1951	1951
q6	187	173	140	140
q7	1000	876	709	709
q8	9353	1511	1210	1210
q9	7291	5332	5340	5332
q10	6860	2406	1973	1973
q11	509	334	295	295
q12	700	717	550	550
q13	17785	3682	3013	3013
q14	283	301	270	270
q15	593	506	505	505
q16	945	907	873	873
q17	715	866	525	525
q18	7436	7044	7044	7044
q19	1098	965	646	646
q20	395	362	248	248
q21	4323	3972	3724	3724
q22	1037	969	966	966
Total cold run time: 108137 ms
Total hot run time: 35339 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4079	4221	4056	4056
q2	339	409	314	314
q3	2149	2724	2328	2328
q4	1357	1787	1302	1302
q5	4283	4708	4522	4522
q6	256	187	144	144
q7	2145	1986	1845	1845
q8	2716	2507	2529	2507
q9	7654	7573	7434	7434
q10	3049	3247	2896	2896
q11	582	531	505	505
q12	845	782	585	585
q13	3506	4032	3465	3465
q14	382	289	269	269
q15	550	528	512	512
q16	884	911	885	885
q17	1160	1453	1394	1394
q18	7922	7690	7719	7690
q19	876	825	875	825
q20	1892	1977	1795	1795
q21	4695	4365	4245	4245
q22	1085	1009	974	974
Total cold run time: 52406 ms
Total hot run time: 50492 ms

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

TPC-DS: Total hot run time: 181983 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 938350d1e209eb06be005978fb6b0bffd6a765cb, data reload: false

query5	4586	619	460	460
query6	329	246	229	229
query7	4226	472	284	284
query8	320	253	254	253
query9	8787	2650	2628	2628
query10	551	359	337	337
query11	15248	14980	15072	14980
query12	183	119	112	112
query13	1275	497	406	406
query14	6253	3279	3038	3038
query14_1	2931	2923	2947	2923
query15	216	198	189	189
query16	947	482	473	473
query17	1156	743	613	613
query18	2505	452	366	366
query19	235	236	216	216
query20	124	118	120	118
query21	219	140	119	119
query22	3951	4063	3927	3927
query23	16534	16219	16156	16156
query23_1	16074	16046	16139	16046
query24	7348	1677	1287	1287
query24_1	1269	1262	1303	1262
query25	608	463	418	418
query26	1244	266	162	162
query27	2744	474	306	306
query28	4475	2185	2176	2176
query29	777	562	444	444
query30	315	245	212	212
query31	839	695	626	626
query32	78	69	68	68
query33	539	326	300	300
query34	893	925	539	539
query35	783	833	723	723
query36	845	904	822	822
query37	128	94	81	81
query38	3861	3822	3772	3772
query39	754	732	766	732
query39_1	704	699	706	699
query40	224	138	120	120
query41	66	63	61	61
query42	109	109	109	109
query43	441	443	428	428
query44	1350	802	779	779
query45	195	188	186	186
query46	890	974	627	627
query47	1676	1664	1604	1604
query48	324	351	257	257
query49	633	443	351	351
query50	667	303	227	227
query51	3835	3801	3814	3801
query52	105	109	102	102
query53	326	355	301	301
query54	294	263	277	263
query55	85	83	75	75
query56	311	308	302	302
query57	1148	1125	1072	1072
query58	269	259	260	259
query59	2309	2431	2259	2259
query60	323	323	296	296
query61	158	159	163	159
query62	696	660	652	652
query63	337	303	316	303
query64	4918	1325	1025	1025
query65	4062	3953	3952	3952
query66	1448	449	333	333
query67	15272	14971	14693	14693
query68	4705	1038	750	750
query69	506	353	309	309
query70	1072	993	996	993
query71	374	334	296	296
query72	6056	4997	4917	4917
query73	653	569	314	314
query74	8780	8813	8674	8674
query75	3559	3528	3171	3171
query76	3943	1144	744	744
query77	527	398	297	297
query78	9508	9738	8895	8895
query79	1076	888	632	632
query80	728	680	562	562
query81	507	270	232	232
query82	407	138	108	108
query83	264	261	246	246
query84	259	126	97	97
query85	910	522	475	475
query86	327	307	280	280
query87	4020	4086	3953	3953
query88	3283	2312	2323	2312
query89	468	438	392	392
query90	1981	168	161	161
query91	178	166	140	140
query92	78	68	68	68
query93	1096	904	572	572
query94	511	318	293	293
query95	584	395	313	313
query96	590	463	210	210
query97	2628	2673	2563	2563
query98	205	201	193	193
query99	1275	1330	1217	1217
Total cold run time: 258172 ms
Total hot run time: 181983 ms

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

ClickBench: Total hot run time: 27.83 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 938350d1e209eb06be005978fb6b0bffd6a765cb, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.05	0.04
query3	0.25	0.09	0.08
query4	1.60	0.12	0.11
query5	0.27	0.25	0.26
query6	1.17	0.73	0.69
query7	0.04	0.03	0.02
query8	0.06	0.05	0.04
query9	0.57	0.52	0.51
query10	0.55	0.55	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	1.00	0.98	0.98
query15	0.82	0.80	0.81
query16	0.42	0.40	0.42
query17	1.06	0.99	0.99
query18	0.24	0.22	0.22
query19	1.96	1.88	1.86
query20	0.02	0.01	0.01
query21	15.43	0.28	0.14
query22	4.86	0.05	0.04
query23	16.12	0.28	0.10
query24	1.66	0.63	0.95
query25	0.10	0.15	0.06
query26	0.14	0.13	0.13
query27	0.06	0.04	0.05
query28	4.58	1.23	1.03
query29	12.58	4.00	3.31
query30	0.27	0.14	0.13
query31	2.83	0.62	0.40
query32	3.23	0.55	0.45
query33	2.95	2.96	3.07
query34	17.02	5.24	4.62
query35	4.57	4.61	4.52
query36	0.68	0.50	0.49
query37	0.11	0.06	0.06
query38	0.08	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.13
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 98.74 s
Total hot run time: 27.83 s

@hubgeter
Copy link
Contributor Author

hubgeter commented Dec 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

TPC-H: Total hot run time: 35515 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3a79755720f00e2dae4e4caac968b015292ff530, data reload: false

------ Round 1 ----------------------------------
q1	17676	4235	4093	4093
q2	2063	357	249	249
q3	10140	1348	785	785
q4	10227	928	327	327
q5	7582	2164	1957	1957
q6	190	167	137	137
q7	1011	864	714	714
q8	9351	1462	1168	1168
q9	7090	5391	5331	5331
q10	6795	2392	1963	1963
q11	513	321	298	298
q12	640	722	575	575
q13	17763	3648	3082	3082
q14	313	292	269	269
q15	583	524	508	508
q16	917	919	863	863
q17	707	840	502	502
q18	7789	7067	7073	7067
q19	1104	959	632	632
q20	402	361	254	254
q21	4255	3889	3768	3768
q22	1069	995	973	973
Total cold run time: 108180 ms
Total hot run time: 35515 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4091	4080	4053	4053
q2	325	399	317	317
q3	2126	2690	2332	2332
q4	1323	1748	1302	1302
q5	4228	4815	4757	4757
q6	235	174	130	130
q7	2051	1986	1868	1868
q8	2678	2567	2577	2567
q9	7689	7576	7671	7576
q10	3176	3257	2791	2791
q11	604	521	487	487
q12	664	750	592	592
q13	3635	3943	3230	3230
q14	286	300	300	300
q15	545	520	508	508
q16	885	916	899	899
q17	1340	1451	1516	1451
q18	7928	7708	7651	7651
q19	963	906	950	906
q20	1994	2077	1955	1955
q21	4842	4260	4245	4245
q22	1087	1017	963	963
Total cold run time: 52695 ms
Total hot run time: 50880 ms

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

TPC-DS: Total hot run time: 181931 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3a79755720f00e2dae4e4caac968b015292ff530, data reload: false

query5	5016	648	505	505
query6	357	252	228	228
query7	4223	476	296	296
query8	326	268	273	268
query9	8791	2631	2585	2585
query10	557	375	325	325
query11	15292	14875	14697	14697
query12	180	120	116	116
query13	1282	512	398	398
query14	6435	3325	3010	3010
query14_1	2931	2960	2994	2960
query15	213	199	193	193
query16	812	500	495	495
query17	1129	729	610	610
query18	2715	470	361	361
query19	247	235	211	211
query20	128	118	114	114
query21	222	139	119	119
query22	3887	3920	3819	3819
query23	16699	16312	16021	16021
query23_1	16038	16239	16040	16040
query24	7383	1674	1244	1244
query24_1	1289	1275	1268	1268
query25	600	496	454	454
query26	1251	270	168	168
query27	2745	476	362	362
query28	4427	2192	2173	2173
query29	813	564	436	436
query30	319	247	218	218
query31	831	687	631	631
query32	84	66	74	66
query33	523	351	300	300
query34	910	906	555	555
query35	815	811	719	719
query36	864	918	819	819
query37	132	98	81	81
query38	3886	3835	3767	3767
query39	794	733	722	722
query39_1	702	728	707	707
query40	227	144	127	127
query41	70	65	64	64
query42	113	109	108	108
query43	436	435	391	391
query44	1368	767	760	760
query45	191	191	182	182
query46	894	984	628	628
query47	1635	1701	1609	1609
query48	316	332	247	247
query49	644	431	370	370
query50	675	299	218	218
query51	3836	3850	3831	3831
query52	113	117	100	100
query53	338	353	295	295
query54	304	262	289	262
query55	80	78	72	72
query56	293	312	310	310
query57	1149	1141	1075	1075
query58	271	277	268	268
query59	2274	2425	2328	2328
query60	350	335	307	307
query61	174	164	158	158
query62	707	694	620	620
query63	334	301	307	301
query64	4992	1344	1026	1026
query65	4007	3943	3975	3943
query66	1423	456	356	356
query67	15328	14900	14867	14867
query68	8580	1024	745	745
query69	498	354	317	317
query70	1097	1033	1019	1019
query71	385	316	291	291
query72	6185	5031	5083	5031
query73	689	613	314	314
query74	8871	8812	8580	8580
query75	3597	3533	3142	3142
query76	4060	1156	789	789
query77	628	409	303	303
query78	9377	9658	8826	8826
query79	1736	883	644	644
query80	734	668	555	555
query81	540	280	232	232
query82	415	140	109	109
query83	277	258	241	241
query84	264	121	104	104
query85	934	522	479	479
query86	397	292	288	288
query87	4083	4170	3909	3909
query88	3719	2330	2297	2297
query89	469	421	392	392
query90	2187	163	148	148
query91	186	177	152	152
query92	85	69	65	65
query93	2398	934	563	563
query94	486	307	281	281
query95	583	394	317	317
query96	590	491	212	212
query97	2610	2656	2548	2548
query98	224	203	190	190
query99	1304	1328	1246	1246
Total cold run time: 266428 ms
Total hot run time: 181931 ms

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

ClickBench: Total hot run time: 27.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3a79755720f00e2dae4e4caac968b015292ff530, data reload: false

query1	0.05	0.04	0.04
query2	0.10	0.04	0.04
query3	0.25	0.09	0.08
query4	1.61	0.11	0.11
query5	0.27	0.25	0.25
query6	1.17	0.65	0.62
query7	0.03	0.03	0.03
query8	0.06	0.04	0.05
query9	0.59	0.52	0.51
query10	0.55	0.56	0.55
query11	0.15	0.11	0.12
query12	0.14	0.12	0.11
query13	0.61	0.60	0.62
query14	1.00	0.97	0.97
query15	0.82	0.79	0.80
query16	0.39	0.38	0.41
query17	1.07	1.06	1.04
query18	0.23	0.22	0.21
query19	1.93	1.80	1.78
query20	0.02	0.01	0.01
query21	15.44	0.30	0.14
query22	4.97	0.05	0.05
query23	15.98	0.29	0.10
query24	1.83	0.58	0.17
query25	0.08	0.05	0.05
query26	0.15	0.14	0.13
query27	0.05	0.06	0.04
query28	4.42	1.25	1.03
query29	12.65	4.15	3.26
query30	0.28	0.14	0.12
query31	2.82	0.66	0.41
query32	3.23	0.58	0.47
query33	3.06	3.00	2.99
query34	17.04	5.34	4.48
query35	4.58	4.55	4.54
query36	0.69	0.50	0.49
query37	0.11	0.07	0.06
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.04	0.04
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 98.88 s
Total hot run time: 27.11 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 10, 2025

BE UT Coverage Report

Increment line coverage 61.54% (400/650) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.37% (18721/35081)
Line Coverage 39.07% (173158/443236)
Region Coverage 33.75% (134335/398018)
Branch Coverage 34.65% (57708/166563)

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 10, 2025

BE Regression && UT Coverage Report

Increment line coverage 78.43% (509/649) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 72.20% (24834/34397)
Line Coverage 58.90% (260834/442858)
Region Coverage 53.71% (216351/402787)
Branch Coverage 55.25% (92520/167444)

@hubgeter hubgeter marked this pull request as ready for review December 11, 2025 03:06
morningman
morningman approved these changes Dec 15, 2025
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 15, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 15, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 15, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 0031179 into apache:master Dec 16, 2025
30 of 32 checks passed
hubgeter added a commit to hubgeter/doris that referenced this pull request Dec 17, 2025
@hubgeter
[fix](parquet)fix parquet topn lazy mat complex data error result (ap…
a6e1ff9 
…ache#58785)

Related PR: apache#51329
Problem Summary:
This PR primarily enables the Parquet reader to use page indexes when
reading complex columns, and also fixes a data reading error in PR
of topn.
@hubgeter hubgeter mentioned this pull request Dec 17, 2025
Merged
16 tasks
yiguolei pushed a commit that referenced this pull request Dec 22, 2025
@hubgeter
branch-4.0:[fix](parquet)fix parquet topn lazy mat complex data error…
38e10a4 
… result (#58785) (#59129)

bp #58785
Related PR: #51329
Problem Summary:
This PR primarily enables the Parquet reader to use page indexes when
reading complex columns, and also fixes a data reading error in PR of
topn.

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@morningman morningman mentioned this pull request Dec 22, 2025
Merged
16 tasks
kaka11chen pushed a commit to kaka11chen/doris that referenced this pull request Jan 7, 2026
@hubgeter @kaka11chen
[fix](parquet)fix parquet topn lazy mat complex data error result (ap…
57ecc97 
…ache#58785)

Related PR: apache#51329
Problem Summary:
This PR primarily enables the Parquet reader to use page indexes when
reading complex columns, and also fixes a data reading error in PR
of topn.
Copilot AI mentioned this pull request Jan 7, 2026
Closed
16 tasks
kaka11chen pushed a commit that referenced this pull request Jan 8, 2026
@hubgeter @kaka11chen
[fix](parquet)fix parquet topn lazy mat complex data error result (#5…
0375981 
…8785)

Related PR: #51329
Problem Summary:
This PR primarily enables the Parquet reader to use page indexes when
reading complex columns, and also fixes a data reading error in PR
of topn.
morningman pushed a commit that referenced this pull request Jan 8, 2026
@hubgeter @morningman
[fix](parquet)fix parquet topn lazy mat complex data error result (#5…
a2380c1 
…8785)

Related PR: #51329
Problem Summary:
This PR primarily enables the Parquet reader to use page indexes when
reading complex columns, and also fixes a data reading error in PR
of topn.
morningman pushed a commit that referenced this pull request Jan 10, 2026
@hubgeter @morningman
[fix](parquet)fix parquet topn lazy mat complex data error result (#5…
40ec6e4 
…8785)

Related PR: #51329
Problem Summary:
This PR primarily enables the Parquet reader to use page indexes when
reading complex columns, and also fixes a data reading error in PR
of topn.
This was referenced Jan 13, 2026
Closed
Merged
morningman pushed a commit that referenced this pull request Jan 14, 2026
@hubgeter @morningman
[fix](parquet)fix parquet topn lazy mat complex data error result (#5…
3d82a95 
…8785)

Related PR: #51329
Problem Summary:
This PR primarily enables the Parquet reader to use page indexes when
reading complex columns, and also fixes a data reading error in PR
of topn.
morningman pushed a commit that referenced this pull request Jan 16, 2026
@hubgeter
[fix](parqeut)Fixed a performance fallback caused by filling RL DL wh…
52c75e7 
…en reading Parquet scalar columns. (#59833)

### What problem does this PR solve?
Related PR: #58785

Problem Summary:
The performance rollback was introduced in #58785. The reason is that,
prior to #58785, reading ordinary columns did not require filling and
saving the RL DL. #58785 combined the logic for reading ordinary columns
from a struct with the logic for reading ordinary columns, and filled
and saved the RL DL to populate the null map information of the struct.
This PR re-separates the reading logic.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@hubgeter @Thearas @doris-robot @hello-stephen @morningman @CalvinKirs @yiguolei