Skip to content

HIVE-25317: Relocate dependencies in shaded hive-exec module #2459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 14 commits into from
Closed

HIVE-25317: Relocate dependencies in shaded hive-exec module #2459

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
4a05f32
Relocate dependencies in shaded release.
viirya Jul 8, 2021
8914bce
hbase-handler uses hive-exec with core classifier.
viirya Jul 9, 2021
8dd4d12
llap-server uses hive-exec with core classifier.
viirya Jul 9, 2021
d0270a6
service uses hive-exec with core classifier.
viirya Jul 9, 2021
0020d0d
Use hive-exec with core classifier.
viirya Jul 10, 2021
e269b76
fix.
viirya Jul 10, 2021
a9a4951
Don't relocate thrift.
viirya Jul 10, 2021
a8769e1
fix
viirya Jul 10, 2021
7dd6427
fix
viirya Jul 10, 2021
d573028
relocate calcite.
viirya Jul 12, 2021
e03eb77
Don't relocate avro and protobuf.
viirya Jul 13, 2021
4644b3b
fix.
viirya Jul 14, 2021
3d50761
Revert "fix."
viirya Jul 15, 2021
d49f560
Remove org.apache.common.* from druid-handler's shaded package.
viirya Jul 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion druid-handler/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -328,7 +328,6 @@
<include>it.unimi.dsi:*</include>
<include>org.jdbi:*</include>
<include>net.jpountz.lz4:*</include>
<include>org.apache.commons:*</include>
</includes>
</artifactSet>
<filters>
Expand Down
1 change: 1 addition & 0 deletions hbase-handler/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
<classifier>core</classifier>
</dependency>
<!-- inter-project -->
<dependency>
Expand Down
1 change: 1 addition & 0 deletions itests/hive-blobstore/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
<classifier>core</classifier>
<scope>test</scope>
</dependency>
<dependency>
Expand Down
1 change: 1 addition & 0 deletions itests/hive-jmh/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
<classifier>core</classifier>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
Expand Down
1 change: 1 addition & 0 deletions itests/hive-minikdc/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
<classifier>core</classifier>
<scope>test</scope>
</dependency>
<dependency>
Expand Down
1 change: 1 addition & 0 deletions itests/hive-unit-hadoop2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
<classifier>core</classifier>
</dependency>

<!-- dependencies are always listed in sorted order by groupId, artifectId -->
Expand Down
1 change: 1 addition & 0 deletions itests/hive-unit/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
<classifier>core</classifier>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
Expand Down
2 changes: 1 addition & 1 deletion itests/qtest-accumulo/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
<scope>test</scope>
<classifier>core</classifier>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.apache.calcite</groupId>
Expand Down
1 change: 1 addition & 0 deletions llap-server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
<classifier>core</classifier>
Copy link
Member

@kgyrtkirk kgyrtkirk Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use the core artifact - that's just bad!

what are you trying to achieve here with this?

Copy link
Member

@kgyrtkirk kgyrtkirk Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

branch-2.3 ?
please note that changes should land on master first

Copy link
Member Author

@viirya viirya Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As more dependencies are relocated here, some modules if they depends on non-core artifact, will cause class not found error...

The motivation is because we want to use shaded version of hive-exec (i.e., w/o classifier) in Spark to make sure it doesn't conflict guava version there. But there are more dependencies conflict with Spark. We need to relocate these dependencies too..

Copy link
Member Author

@viirya viirya Aug 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sunchao do we need to have similar change on master first?

Copy link
Member

@sunchao sunchao Aug 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea I think so since this PR is trying to shade things from the hive-exec-core I believe?

Copy link
Member

@kgyrtkirk kgyrtkirk Sep 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should fix the issues with using the normal hive-exec artifact if there is any - loading the core jar could cause troubles...

Copy link
Member

@kgyrtkirk kgyrtkirk Sep 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: on branch2 guava is most likely not properly shaded away HIVE-22126

Copy link
Member

@sunchao sunchao Sep 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kgyrtkirk Guava is shaded in branch-2.3 via https://issues.apache.org/jira/browse/HIVE-23980. The issue is, in order for Spark to use shaded hive-exec, Hive will need to relocate more classes and at the same time making sure it won't break other modules (for instance, if the shaded class appears in certain API and another module imported the unshaded version of the class by itself).

Currently we've abandoned this approach and decided to shade the hive-exec-core within Spark itself, following similar approach in Trino (see https://github.com/trinodb/trino-hive-apache).

Copy link
Member

@kgyrtkirk kgyrtkirk Sep 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could relocate/shade away those deps to make it possible for other projects to use the normal artifact - seems like there is a very good list in the trino project.

</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
Expand Down
32 changes: 32 additions & 0 deletions ql/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -884,6 +884,38 @@
</includes>
</artifactSet>
<relocations>
<relocation>
<pattern>org.apache.calcite</pattern>
<shadedPattern>org.apache.hive.org.apache.calcite</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.orc</pattern>
<shadedPattern>org.apache.hive.org.apache.orc</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons</pattern>
<shadedPattern>org.apache.hive.org.apache.commons</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.parquet</pattern>
<shadedPattern>org.apache.hive.org.apache.parquet</shadedPattern>
</relocation>
<relocation>
<pattern>joda-time</pattern>
<shadedPattern>org.apache.hive.joda-time</shadedPattern>
</relocation>
<relocation>
<pattern>commons-lang</pattern>
<shadedPattern>org.apache.hive.commons-lang</shadedPattern>
</relocation>
<relocation>
<pattern>org.jodd</pattern>
<shadedPattern>org.apache.hive.org.jodd</shadedPattern>
</relocation>
<relocation>
<pattern>org.codehaus.jackson</pattern>
<shadedPattern>org.apache.hive.org.codehaus.jackson</shadedPattern>
</relocation>
<relocation>
<pattern>com.esotericsoftware</pattern>
<shadedPattern>org.apache.hive.com.esotericsoftware</shadedPattern>
Expand Down
1 change: 1 addition & 0 deletions service/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${project.version}</version>
<classifier>core</classifier>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
Expand Down