HIVE-25317: Relocate dependencies in shaded hive-exec module #2459

viirya · 2021-07-08T19:31:33Z

What changes were proposed in this pull request?

Trying to relocate dependencies which could conflict Spark.

Why are the changes needed?

When we want to use shaded version of hive-exec (i.e., w/o classifier), more dependencies conflict with Spark. We need to relocate these dependencies too.

Does this PR introduce any user-facing change?

If previously downstream projects rely on included dependencies in shaded release, they might need to explicitly include these dependencies after the relocation here.

How was this patch tested?

CI

viirya · 2021-07-13T20:19:43Z

Thanks @sunchao. Seems caused by relocating avro, protobuf. Reverted the two relocations and see what's going on.

sunchao · 2021-07-13T21:33:18Z

@viirya there are still more errors:

java.lang.NoClassDefFoundError: org/apache/hive/org/apache/commons/logging/LogFactory
	at org.apache.hadoop.hive.conf.valcoersion.JavaIOTmpdirVariableCoercion.<clinit>(JavaIOTmpdirVariableCoercion.java:35)
	at org.apache.hadoop.hive.conf.SystemVariables.<clinit>(SystemVariables.java:37)
	at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<init>(HiveConf.java:3445)
	at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<init>(HiveConf.java:3409)
	at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:436)
	at org.apache.hadoop.hive.druid.serde.DruidSerDe.initialize(DruidSerDe.java:113)
	at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
	at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
	at org.apache.hadoop.hive.druid.TestDruidSerDe.testDruidDeserializer(TestDruidSerDe.java:510)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)

Caused by: java.lang.NoClassDefFoundError: org/apache/hive/org/apache/commons/codec/language/Soundex
	at org.apache.hadoop.hive.ql.udf.generic.GenericUDFSoundex.<init>(GenericUDFSoundex.java:49)
	... 29 more

Caused by: java.lang.NoClassDefFoundError: org/apache/hive/org/apache/commons/codec/binary/Base64
	at org.apache.hadoop.hive.serde2.lazy.LazyBinary.decodeIfNeeded(LazyBinary.java:58)
	at org.apache.hadoop.hive.serde2.lazy.LazyBinary.init(LazyBinary.java:50)
	at org.apache.hadoop.hive.serde2.lazy.LazyStruct.uncheckedGetField(LazyStruct.java:226)
	at org.apache.hadoop.hive.serde2.lazy.LazyStruct.getField(LazyStruct.java:202)
	at org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldData(LazySimpleStructObjectInspector.java:128)
	at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldData(UnionStructObjectInspector.java:157)
	at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:95)
	at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
	at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
	at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
	at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:438)
	at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:430)

This reverts commit 4644b3b.

viirya · 2021-07-16T20:54:22Z

I'm not sure why this happens, or if it is caused by this change. HiveConf is not from hive-exec (ql), and this doesn't relocate any Hive internal class (we shoudn't do this too). cc @sunchao

java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.conf.HiveConf$ConfVars
	at org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitCreateTablePlusCommitDropTableWithoutPurge(TestDruidStorageHandler.java:135)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

sunchao · 2021-07-16T21:02:16Z

I'll take a look @viirya - where did you find this error? do you have a link?

I'm seeing these errors in the last run:

Caused by: java.lang.ClassNotFoundException: org.apache.hive.org.apache.commons.codec.language.Soundex
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	... 32 more

Caused by: java.lang.ClassNotFoundException: org.apache.hive.org.apache.commons.logging.LogFactory
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	... 35 more

viirya · 2021-07-16T21:11:19Z

The quoted test failure from "Testing / split-20 / Archive / testCommitCreateTablePlusCommitDropTableWithoutPurge – org.apache.hadoop.hive.druid.TestDruidStorageHandler" in http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-2459/11/tests.

viirya · 2021-07-16T21:11:45Z

Yea, I'm trying to deal with org.apache.hive.org.apache.commons.logging.LogFactory.

sunchao · 2021-07-16T21:28:19Z

I see. Yea this is very weird. I suspect there are something else that caused it to fail, for instance, when initializing some of the enum items.

kgyrtkirk · 2021-07-28T13:32:48Z

llap-server/pom.xml

      <groupId>org.apache.hive</groupId>
      <artifactId>hive-exec</artifactId>
      <version>${project.version}</version>
+      <classifier>core</classifier>


don't use the core artifact - that's just bad!

what are you trying to achieve here with this?

branch-2.3 ?
please note that changes should land on master first

As more dependencies are relocated here, some modules if they depends on non-core artifact, will cause class not found error...

The motivation is because we want to use shaded version of hive-exec (i.e., w/o classifier) in Spark to make sure it doesn't conflict guava version there. But there are more dependencies conflict with Spark. We need to relocate these dependencies too..

@sunchao do we need to have similar change on master first?

Yea I think so since this PR is trying to shade things from the hive-exec-core I believe?

we should fix the issues with using the normal hive-exec artifact if there is any - loading the core jar could cause troubles...

note: on branch2 guava is most likely not properly shaded away HIVE-22126

@kgyrtkirk Guava is shaded in branch-2.3 via https://issues.apache.org/jira/browse/HIVE-23980. The issue is, in order for Spark to use shaded hive-exec, Hive will need to relocate more classes and at the same time making sure it won't break other modules (for instance, if the shaded class appears in certain API and another module imported the unshaded version of the class by itself).

Currently we've abandoned this approach and decided to shade the hive-exec-core within Spark itself, following similar approach in Trino (see https://github.com/trinodb/trino-hive-apache).

we could relocate/shade away those deps to make it possible for other projects to use the normal artifact - seems like there is a very good list in the trino project.

viirya · 2021-09-15T07:31:26Z

Close this as we are taking a different direction to shade dependencies of Hive at Spark.

Relocate dependencies in shaded release.

4a05f32

kgyrtkirk added tests pending tests failed and removed tests pending labels Jul 8, 2021

hbase-handler uses hive-exec with core classifier.

8914bce

kgyrtkirk added tests pending and removed tests failed labels Jul 9, 2021

llap-server uses hive-exec with core classifier.

8dd4d12

kgyrtkirk added tests failed tests pending and removed tests pending tests failed labels Jul 9, 2021

service uses hive-exec with core classifier.

d0270a6

kgyrtkirk added tests pending tests failed and removed tests failed tests pending labels Jul 9, 2021

Use hive-exec with core classifier.

0020d0d

kgyrtkirk added tests pending tests failed and removed tests failed tests pending labels Jul 10, 2021

fix.

e269b76

kgyrtkirk added tests pending and removed tests failed labels Jul 10, 2021

viirya added 3 commits July 10, 2021 15:20

Don't relocate thrift.

a9a4951

fix

a8769e1

fix

7dd6427

kgyrtkirk added tests pending and removed tests failed labels Jul 13, 2021

kgyrtkirk added tests failed and removed tests pending labels Jul 14, 2021

fix.

4644b3b

kgyrtkirk added tests pending tests failed and removed tests failed tests pending labels Jul 14, 2021

Revert "fix."

3d50761

This reverts commit 4644b3b.

kgyrtkirk added tests pending tests failed and removed tests failed tests pending labels Jul 15, 2021

Remove org.apache.common.* from druid-handler's shaded package.

d49f560

kgyrtkirk added tests pending and removed tests failed labels Jul 16, 2021

kgyrtkirk added tests failed and removed tests pending labels Jul 17, 2021

kgyrtkirk reviewed Jul 28, 2021

View reviewed changes

viirya closed this Sep 15, 2021

sunchao mentioned this pull request Oct 14, 2021

[SPARK-36676][SQL][BUILD] Create shaded Hive module and upgrade Guava version to 30.1.1-jre apache/spark#33989

Closed

HIVE-25317: Relocate dependencies in shaded hive-exec module #2459

HIVE-25317: Relocate dependencies in shaded hive-exec module #2459

Uh oh!

Conversation

viirya commented Jul 8, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

viirya commented Jul 13, 2021

Uh oh!

sunchao commented Jul 13, 2021

Uh oh!

viirya commented Jul 16, 2021

Uh oh!

sunchao commented Jul 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viirya commented Jul 16, 2021

Uh oh!

viirya commented Jul 16, 2021

Uh oh!

sunchao commented Jul 16, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya commented Sep 15, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sunchao commented Jul 16, 2021 •

edited

Loading