Skip to content

[Fix](catalog)Upgrade hive-catalog-shade to 2.0.2 to solve the problem that DLF cannot be used normally #34749

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
CalvinKirs merged 1 commit into from
May 13, 2024
Merged

[Fix](catalog)Upgrade hive-catalog-shade to 2.0.2 to solve the problem that DLF cannot be used normally #34749

CalvinKirs merged 1 commit into from
May 13, 2024

Conversation

@CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented May 13, 2024

Proposed changes

The exception stack encountered when using DLF is as follows:

This is because hive-catalog-sahde incorrectly renamed dlf-related packages, causing it to fail to run normally.

 java.lang.ClassCastException: class org.apache.hadoop.hive.metastore.api.NoSuchObjectException cannot be cast to class org.apache.thrift.TException

apache/doris-shade#44

@doris-robot
Copy link

doris-robot commented May 13, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@CalvinKirs
Copy link
Member Author

CalvinKirs commented May 13, 2024

run buildall

@morningman
Copy link
Contributor

morningman commented May 13, 2024

run buildall

morningman
morningman approved these changes May 13, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented May 13, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 13, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 13, 2024

PR approved by anyone and no changes requested.

@CalvinKirs
Copy link
Member Author

CalvinKirs commented May 13, 2024

Oops...let's run the test later. It seems that the jar package has not completely completed the release process.

@CalvinKirs
Copy link
Member Author

CalvinKirs commented May 13, 2024

run buildall

JNSimba
JNSimba approved these changes May 13, 2024
View reviewed changes
Copy link
Member

@JNSimba JNSimba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

doris-robot commented May 13, 2024

TPC-H: Total hot run time: 39758 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 26aa76f910246b538de4a3a065b0eeed6fc7e61a, data reload: false

------ Round 1 ----------------------------------
q1	17995	4498	4282	4282
q2	2018	190	187	187
q3	10497	1190	1161	1161
q4	10149	874	723	723
q5	7463	2694	2644	2644
q6	221	136	133	133
q7	971	612	586	586
q8	9242	2072	2045	2045
q9	8803	6528	6492	6492
q10	8958	3732	3721	3721
q11	444	244	255	244
q12	411	221	220	220
q13	18581	2926	2949	2926
q14	259	219	225	219
q15	510	479	488	479
q16	522	389	402	389
q17	950	657	638	638
q18	8089	7574	7325	7325
q19	6178	1554	1518	1518
q20	652	323	316	316
q21	4939	3228	4047	3228
q22	362	282	282	282
Total cold run time: 118214 ms
Total hot run time: 39758 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4418	4223	4219	4219
q2	382	274	264	264
q3	2972	2830	2689	2689
q4	1889	1539	1574	1539
q5	5305	5260	5348	5260
q6	213	126	125	125
q7	2261	1860	1888	1860
q8	3206	3318	3285	3285
q9	8343	8336	8366	8336
q10	3864	3664	3719	3664
q11	585	499	483	483
q12	764	563	621	563
q13	17429	2950	2957	2950
q14	298	266	274	266
q15	518	470	480	470
q16	471	433	415	415
q17	1754	1490	1462	1462
q18	7614	7410	7471	7410
q19	1660	1606	1574	1574
q20	1991	1762	1746	1746
q21	8009	4831	4896	4831
q22	569	494	517	494
Total cold run time: 74515 ms
Total hot run time: 53905 ms

@doris-robot
Copy link

doris-robot commented May 13, 2024

TPC-DS: Total hot run time: 187413 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 26aa76f910246b538de4a3a065b0eeed6fc7e61a, data reload: false

query1	922	366	347	347
query2	6442	2468	2288	2288
query3	6652	204	204	204
query4	26221	21215	21219	21215
query5	4162	412	416	412
query6	270	176	200	176
query7	4586	290	291	290
query8	247	194	198	194
query9	8727	2452	2419	2419
query10	457	248	253	248
query11	14732	14440	14192	14192
query12	143	91	90	90
query13	1676	379	382	379
query14	10092	8364	8005	8005
query15	252	177	180	177
query16	8080	262	263	262
query17	1850	563	535	535
query18	2041	280	264	264
query19	212	148	148	148
query20	90	87	86	86
query21	197	127	153	127
query22	5057	4865	4858	4858
query23	34507	33540	33580	33540
query24	12049	2934	2917	2917
query25	650	381	367	367
query26	1722	164	150	150
query27	3011	327	314	314
query28	7166	2083	2070	2070
query29	1012	613	591	591
query30	283	150	153	150
query31	981	769	735	735
query32	90	52	55	52
query33	741	251	244	244
query34	1060	483	476	476
query35	814	687	684	684
query36	1040	950	931	931
query37	279	67	68	67
query38	2901	2748	2792	2748
query39	1616	1549	1579	1549
query40	273	122	128	122
query41	42	62	42	42
query42	102	96	95	95
query43	571	532	555	532
query44	1192	733	748	733
query45	267	258	252	252
query46	1078	706	721	706
query47	1966	1910	1890	1890
query48	382	300	295	295
query49	1186	390	396	390
query50	757	378	393	378
query51	6761	6663	6633	6633
query52	97	93	86	86
query53	350	278	280	278
query54	977	428	425	425
query55	71	72	72	72
query56	243	219	221	219
query57	1242	1176	1122	1122
query58	226	203	196	196
query59	3288	3086	3232	3086
query60	249	237	257	237
query61	89	86	84	84
query62	670	474	485	474
query63	306	279	287	279
query64	9733	7406	7385	7385
query65	3200	3095	3080	3080
query66	1350	356	341	341
query67	15470	14889	15130	14889
query68	4583	529	526	526
query69	488	306	309	306
query70	1164	1145	1064	1064
query71	420	275	267	267
query72	7299	2535	2365	2365
query73	709	325	320	320
query74	6544	6138	6232	6138
query75	3388	2622	2631	2622
query76	2824	1020	973	973
query77	376	261	256	256
query78	10636	10093	10135	10093
query79	2057	507	524	507
query80	776	424	417	417
query81	512	217	224	217
query82	653	92	92	92
query83	270	159	165	159
query84	236	87	94	87
query85	1248	280	322	280
query86	447	319	302	302
query87	3285	3167	3104	3104
query88	4136	2464	2456	2456
query89	465	378	385	378
query90	1989	187	190	187
query91	140	106	109	106
query92	64	51	49	49
query93	1789	504	492	492
query94	1193	183	189	183
query95	397	306	314	306
query96	589	278	273	273
query97	3127	3023	2981	2981
query98	237	222	216	216
query99	1080	869	895	869
Total cold run time: 288289 ms
Total hot run time: 187413 ms

@CalvinKirs CalvinKirs merged commit 7fc75e3 into apache:master May 13, 2024
@CalvinKirs CalvinKirs deleted the cvec_v_ec_v branch May 13, 2024 06:52
nextdreamblue pushed a commit to nextdreamblue/doris that referenced this pull request May 13, 2024
@CalvinKirs @nextdreamblue
[Fix](catalog)Upgrade hive-catalog-shade to 2.0.2 to solve the proble…
fd3781a 
…m that DLF cannot be used normally (apache#34749)
ByteYue pushed a commit to ByteYue/doris that referenced this pull request May 15, 2024
@CalvinKirs @ByteYue
[Fix](catalog)Upgrade hive-catalog-shade to 2.0.2 to solve the proble…
c6ed978 
…m that DLF cannot be used normally (apache#34749)
@morningman morningman mentioned this pull request May 23, 2024
Merged
morningman added a commit that referenced this pull request May 23, 2024
@morningman
[branch-2.1] remove dlf dependencies (#35292)
ed464ac 
followup #35241
In #35241, we update the doris-shade version to 2.1.0, which already contains dlf dependencies.

pick part of #34749, to remove dlf dependencies in fe/pom.xml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@JNSimba JNSimba JNSimba approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@CalvinKirs @doris-robot @morningman @JNSimba @yiguolei @dataroaring