Skip to content

branch-3.0: [opt](hive) add option to get schema from table object #50038 #50268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Apr 23, 2025
Merged

branch-3.0: [opt](hive) add option to get schema from table object #50038 #50268

dataroaring merged 1 commit into from
Apr 23, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Apr 22, 2025

Cherry-picked from #50038

@morningman
[opt](hive) add option to get schema from table object (#50038)
9e81cc7 
### What problem does this PR solve?

Sometimes user may meet error:
```
Storage schema reading not supported
```
when using hive catalog to query table.

Because there are some compatibility issue in hive metastore, see
trinodb/trino#2678.
So here we provide a catalog property:

`get_schema_from_table`.

Default is false, which will still get schema from hive metastore like
before.
If set to true, the schema will be got from `table` object directly, to
avoid above error.
But notice that if set to true, the default value of column will be
ignored because the `table` object
does not store this information.
@github-actions github-actions bot requested a review from dataroaring as a code owner April 22, 2025 02:30
@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 22, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Apr 22, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 22, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 22, 2025

TPC-H: Total hot run time: 40088 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9e81cc7d947c9967cd90e734d3186d12e617bbc9, data reload: false

------ Round 1 ----------------------------------
q1	17573	6705	6588	6588
q2	2052	180	179	179
q3	10620	1064	1198	1064
q4	10561	744	683	683
q5	7755	2854	2846	2846
q6	222	133	130	130
q7	976	611	614	611
q8	9393	1939	2016	1939
q9	6509	6410	6433	6410
q10	6961	2250	2337	2250
q11	468	262	253	253
q12	390	207	208	207
q13	17806	2946	2989	2946
q14	243	205	209	205
q15	498	470	451	451
q16	650	576	568	568
q17	955	597	628	597
q18	7167	6689	6744	6689
q19	1500	1065	1065	1065
q20	469	197	199	197
q21	3943	3221	3404	3221
q22	1077	992	989	989
Total cold run time: 107788 ms
Total hot run time: 40088 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6622	6583	6501	6501
q2	332	234	224	224
q3	2895	2767	2851	2767
q4	2064	1814	1750	1750
q5	5807	5694	5739	5694
q6	217	133	132	132
q7	2219	1807	1824	1807
q8	3333	3560	3531	3531
q9	8998	8768	8943	8768
q10	3540	3544	3488	3488
q11	584	486	491	486
q12	807	580	600	580
q13	10220	3143	3201	3143
q14	296	269	260	260
q15	521	475	464	464
q16	694	639	629	629
q17	1835	1662	1607	1607
q18	8216	7863	7743	7743
q19	1675	1446	1560	1446
q20	2047	1796	1865	1796
q21	5422	5447	5337	5337
q22	1125	1034	1021	1021
Total cold run time: 69469 ms
Total hot run time: 59174 ms

@doris-robot
Copy link

doris-robot commented Apr 22, 2025

TPC-DS: Total hot run time: 197524 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9e81cc7d947c9967cd90e734d3186d12e617bbc9, data reload: false

query1	1286	919	886	886
query2	6218	1980	1975	1975
query3	10818	4403	4365	4365
query4	60794	29073	23887	23887
query5	5226	463	443	443
query6	408	177	180	177
query7	5436	328	307	307
query8	301	223	229	223
query9	8575	2622	2572	2572
query10	453	275	252	252
query11	17308	15310	15670	15310
query12	160	102	107	102
query13	1460	450	442	442
query14	9941	7339	6731	6731
query15	215	191	178	178
query16	7134	513	476	476
query17	1142	581	572	572
query18	1854	316	316	316
query19	210	157	157	157
query20	118	116	114	114
query21	210	104	104	104
query22	4792	4628	4770	4628
query23	34914	33944	34256	33944
query24	6203	2886	2955	2886
query25	545	412	429	412
query26	655	175	168	168
query27	1824	360	363	360
query28	4872	2463	2469	2463
query29	702	511	427	427
query30	258	162	162	162
query31	1005	821	848	821
query32	64	56	58	56
query33	493	299	293	293
query34	911	502	513	502
query35	858	735	745	735
query36	1093	943	946	943
query37	112	63	70	63
query38	4049	4047	3951	3951
query39	1529	1472	1471	1471
query40	205	97	95	95
query41	50	48	48	48
query42	115	100	106	100
query43	525	475	478	475
query44	1172	816	804	804
query45	182	164	173	164
query46	1151	707	731	707
query47	2011	1917	1909	1909
query48	478	392	406	392
query49	709	396	400	396
query50	859	449	415	415
query51	7409	7318	7338	7318
query52	112	87	98	87
query53	258	192	185	185
query54	582	479	481	479
query55	85	83	79	79
query56	260	252	258	252
query57	1260	1199	1159	1159
query58	220	209	244	209
query59	3198	3005	2844	2844
query60	262	257	242	242
query61	111	117	105	105
query62	786	662	690	662
query63	211	194	181	181
query64	1388	669	670	669
query65	3258	3175	3216	3175
query66	702	297	298	297
query67	16203	15851	15724	15724
query68	4246	581	592	581
query69	430	266	262	262
query70	1185	1045	1106	1045
query71	354	284	242	242
query72	6398	4129	3938	3938
query73	757	352	362	352
query74	10381	9007	8969	8969
query75	3364	2655	2648	2648
query76	2037	1093	1053	1053
query77	510	280	269	269
query78	10670	9671	9655	9655
query79	1221	594	599	594
query80	855	423	415	415
query81	516	245	236	236
query82	1314	90	88	88
query83	248	146	144	144
query84	291	80	81	80
query85	863	295	297	295
query86	318	301	306	301
query87	4391	4208	4245	4208
query88	3448	2416	2393	2393
query89	415	288	286	286
query90	2026	185	179	179
query91	184	170	152	152
query92	69	51	47	47
query93	1378	550	551	550
query94	750	302	303	302
query95	349	262	248	248
query96	609	280	276	276
query97	3308	3167	3179	3167
query98	215	209	208	208
query99	1790	1305	1295	1295
Total cold run time: 312798 ms
Total hot run time: 197524 ms

@doris-robot
Copy link

doris-robot commented Apr 22, 2025

ClickBench: Total hot run time: 32.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9e81cc7d947c9967cd90e734d3186d12e617bbc9, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.06
query4	1.61	0.11	0.10
query5	0.53	0.52	0.51
query6	1.12	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.51
query10	0.54	0.54	0.56
query11	0.14	0.11	0.10
query12	0.14	0.12	0.11
query13	0.60	0.59	0.59
query14	2.71	2.74	2.74
query15	0.89	0.82	0.82
query16	0.37	0.38	0.40
query17	1.06	1.06	1.01
query18	0.24	0.22	0.21
query19	1.95	1.91	2.03
query20	0.01	0.01	0.01
query21	15.36	0.59	0.58
query22	2.93	2.40	2.27
query23	16.93	0.93	0.76
query24	3.09	1.38	1.00
query25	0.27	0.24	0.05
query26	0.54	0.14	0.14
query27	0.03	0.04	0.05
query28	10.16	0.52	0.45
query29	12.57	3.26	3.26
query30	0.25	0.05	0.06
query31	2.86	0.39	0.37
query32	3.24	0.46	0.46
query33	2.95	3.05	3.02
query34	16.76	4.49	4.44
query35	4.52	4.56	4.51
query36	0.71	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.03	0.03
query40	0.16	0.12	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.53 s
Total hot run time: 32.61 s

dataroaring
dataroaring approved these changes Apr 23, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 8969a6b into branch-3.0 Apr 23, 2025
23 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-50038-branch-3.0 branch April 23, 2025 03:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hello-stephen @doris-robot @dataroaring @morningman