Skip to content

branch-3.1: [fix](iceberg)Use the correct schema for query (#50376) #52219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Jul 1, 2025
Merged

branch-3.1: [fix](iceberg)Use the correct schema for query (#50376) #52219

morningman merged 1 commit into from
Jul 1, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Jun 24, 2025 •
edited
Loading

bp #50376

@morningman morningman requested a review from morrySnow as a code owner June 24, 2025 09:13
@morrySnow
Copy link
Contributor

morrySnow commented Jun 24, 2025

run buildall

@morningman morningman marked this pull request as draft June 24, 2025 10:09
@morningman
Copy link
Contributor Author

morningman commented Jun 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-H: Total hot run time: 39284 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0633d2bcc28045632b4d7fac119b52be0c306f81, data reload: false

------ Round 1 ----------------------------------
q1	17613	6687	6563	6563
q2	2080	169	167	167
q3	10610	1071	1196	1071
q4	10561	745	684	684
q5	7737	2866	2813	2813
q6	215	133	131	131
q7	1000	621	604	604
q8	9343	1933	1981	1933
q9	6595	6388	6395	6388
q10	6991	2259	2226	2226
q11	462	260	266	260
q12	398	211	206	206
q13	17780	2955	2945	2945
q14	236	209	203	203
q15	498	452	463	452
q16	467	383	366	366
q17	968	594	533	533
q18	7246	6587	6643	6587
q19	1322	1038	941	941
q20	474	200	194	194
q21	3913	3055	3154	3055
q22	1086	985	962	962
Total cold run time: 107595 ms
Total hot run time: 39284 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6569	6552	6559	6552
q2	318	234	231	231
q3	2902	2752	2819	2752
q4	2042	1795	1771	1771
q5	5688	5762	5694	5694
q6	211	127	126	126
q7	2166	1751	1769	1751
q8	3374	3534	3556	3534
q9	8868	8803	8862	8803
q10	3562	3521	3490	3490
q11	575	492	510	492
q12	793	587	600	587
q13	8961	3180	3114	3114
q14	294	280	288	280
q15	535	453	457	453
q16	475	426	432	426
q17	1829	1587	1588	1587
q18	8222	7741	7662	7662
q19	1682	1551	1523	1523
q20	2114	1865	1804	1804
q21	4947	5040	4981	4981
q22	1136	1042	1016	1016
Total cold run time: 67263 ms
Total hot run time: 58629 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-DS: Total hot run time: 196086 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0633d2bcc28045632b4d7fac119b52be0c306f81, data reload: false

query1	1277	896	894	894
query2	6293	1959	1874	1874
query3	10784	4188	4202	4188
query4	61437	28847	23867	23867
query5	5185	462	450	450
query6	415	190	181	181
query7	5508	317	307	307
query8	317	229	229	229
query9	8530	2580	2595	2580
query10	470	268	252	252
query11	17479	15273	15829	15273
query12	166	107	107	107
query13	1471	444	426	426
query14	9619	6500	6586	6500
query15	211	178	173	173
query16	7129	503	479	479
query17	1183	563	590	563
query18	1852	327	312	312
query19	219	162	179	162
query20	119	111	110	110
query21	210	106	107	106
query22	4883	4494	4749	4494
query23	34462	33978	34142	33978
query24	6803	2953	2876	2876
query25	566	436	426	426
query26	674	177	169	169
query27	2300	360	351	351
query28	4412	2158	2161	2158
query29	719	447	456	447
query30	246	166	159	159
query31	1016	808	865	808
query32	71	58	59	58
query33	454	315	329	315
query34	915	505	525	505
query35	846	747	735	735
query36	1085	961	935	935
query37	110	65	70	65
query38	4046	3896	3959	3896
query39	1524	1476	1471	1471
query40	206	98	98	98
query41	49	45	46	45
query42	109	106	102	102
query43	505	497	498	497
query44	1171	818	809	809
query45	180	167	169	167
query46	1153	716	731	716
query47	2064	1950	1971	1950
query48	448	343	336	336
query49	722	385	383	383
query50	867	420	440	420
query51	7310	7166	7216	7166
query52	99	91	89	89
query53	251	181	180	180
query54	561	460	462	460
query55	77	78	78	78
query56	274	243	254	243
query57	1359	1216	1214	1214
query58	224	216	212	212
query59	3182	3080	3146	3080
query60	282	257	274	257
query61	112	171	130	130
query62	769	712	684	684
query63	229	191	196	191
query64	1347	628	621	621
query65	3220	3171	3138	3138
query66	700	293	293	293
query67	15937	15556	15390	15390
query68	3725	611	571	571
query69	427	263	273	263
query70	1135	1123	1064	1064
query71	345	259	254	254
query72	6418	4094	4077	4077
query73	750	351	364	351
query74	10189	9187	8922	8922
query75	3408	2633	2659	2633
query76	2022	1094	1012	1012
query77	504	256	269	256
query78	10633	9587	9585	9585
query79	1471	587	588	587
query80	875	469	432	432
query81	496	216	213	213
query82	1263	88	86	86
query83	243	142	146	142
query84	280	80	79	79
query85	908	303	291	291
query86	335	292	302	292
query87	4444	4196	4229	4196
query88	3808	2369	2335	2335
query89	415	296	298	296
query90	1994	183	180	180
query91	141	106	106	106
query92	62	48	54	48
query93	1955	544	543	543
query94	735	285	277	277
query95	346	254	254	254
query96	606	272	278	272
query97	3316	3175	3141	3141
query98	213	200	199	199
query99	1559	1301	1313	1301
Total cold run time: 313434 ms
Total hot run time: 196086 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

ClickBench: Total hot run time: 29.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0633d2bcc28045632b4d7fac119b52be0c306f81, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.52	0.51	0.52
query6	1.14	0.74	0.72
query7	0.02	0.03	0.02
query8	0.04	0.03	0.04
query9	0.56	0.52	0.50
query10	0.56	0.56	0.54
query11	0.14	0.10	0.10
query12	0.15	0.11	0.11
query13	0.61	0.60	0.59
query14	0.78	0.80	0.80
query15	0.84	0.84	0.82
query16	0.39	0.38	0.38
query17	1.06	1.04	1.06
query18	0.24	0.23	0.21
query19	1.96	1.83	1.88
query20	0.02	0.01	0.01
query21	15.39	0.58	0.55
query22	2.22	1.60	1.68
query23	17.02	0.88	0.81
query24	3.33	1.08	0.66
query25	0.30	0.24	0.16
query26	0.27	0.13	0.14
query27	0.05	0.05	0.04
query28	10.45	0.49	0.47
query29	12.59	3.23	3.19
query30	0.24	0.06	0.06
query31	2.87	0.40	0.40
query32	3.23	0.46	0.45
query33	2.96	2.95	3.04
query34	17.06	4.49	4.42
query35	4.50	4.53	4.51
query36	0.70	0.47	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.18	0.13	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.65 s
Total hot run time: 29.59 s

@morningman morningman marked this pull request as ready for review June 30, 2025 03:20
@morningman
Copy link
Contributor Author

morningman commented Jun 30, 2025

run buildall

@wuwenchi @morningman
[fix](iceberg)Use the correct schema for query (apache#50376)
5394f23 
Followup apache#49956

Problem Summary:

When a snapshot is specified in the query, the corresponding schema
should be used for parsing, otherwise the latest snapshot should be used
for parsing.

1. When using the HMS type, you also need to initialize the executor
pool.
2. Set the size of the thread pool to be equal to the number of cores of
the current machine.
3. When no snapshot is specified, the latest schema is used.
4. When specifying a snapshot, you need to use the schema corresponding
to the snapshot.
5. When generating a scannode, save the schema information and no longer
obtain it from the cache to prevent the cache from being refreshed.
6. When refreshing the schema, you need to refresh all schemas of
related tables.
@morningman morningman changed the title [fix](iceberg)Use the correct schema for query (#50376) branch-3.1: [fix](iceberg)Use the correct schema for query (#50376) Jun 30, 2025
@morningman
Copy link
Contributor Author

morningman commented Jun 30, 2025

run buildall

@morningman morningman merged commit faff952 into apache:branch-3.1 Jul 1, 2025
20 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@morningman @morrySnow @doris-robot @wuwenchi