Skip to content

[SPARK-42228][BUILD][CONNECT] Add shade and relocation rule of grpc to connect-client-jvm module #39789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LuciferYang wants to merge 8 commits into from
Closed

[SPARK-42228][BUILD][CONNECT] Add shade and relocation rule of grpc to connect-client-jvm module #39789

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e33c29a
fix
LuciferYang Jan 29, 2023
dbbf2ba
address comment
LuciferYang Feb 1, 2023
fc3165d
Revert "address comment"
LuciferYang Feb 1, 2023
92fcb0c
fix relocation of grpc
LuciferYang Feb 1, 2023
a5d0e46
add ServicesResourceTransformer relocation class names in META-INF/se…
LuciferYang Feb 1, 2023
52a2865
add comments
LuciferYang Feb 1, 2023
562a71e
to connect.client
LuciferYang Feb 1, 2023
d3ad7b8
for sbt
LuciferYang Feb 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions connector/connect/client/jvm/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,11 +105,19 @@
<artifactSet>
<includes>
<include>com.google.guava:*</include>
<include>io.grpc:*</include>
<include>com.google.protobuf:*</include>
<include>org.apache.spark:spark-connect-common_${scala.binary.version}</include>
</includes>
</artifactSet>
<relocations>
<relocation>
Copy link
Contributor Author

@LuciferYang LuciferYang Feb 1, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After

ls org/sparkproject/connect/client
grpc	guava

ls org/sparkproject/connect/client/grpc
Attributes$1.class							LoadBalancer$Helper.class
Attributes$Builder.class						LoadBalancer$PickResult.class
Attributes$Key.class							LoadBalancer$PickSubchannelArgs.class
Attributes.class							LoadBalancer$ResolvedAddresses$Builder.class
BinaryLog.class								LoadBalancer$ResolvedAddresses.class
BindableService.class							LoadBalancer$Subchannel.class
CallCredentials$MetadataApplier.class					LoadBalancer$SubchannelPicker.class
CallCredentials$RequestInfo.class					LoadBalancer$SubchannelStateListener.class
CallCredentials.class							LoadBalancer.class
CallOptions$Key.class							LoadBalancerProvider$UnknownConfig.class
....

all grpc java class relocation to org/sparkproject/connect/client/grpc

<pattern>io.grpc</pattern>
<shadedPattern>${spark.shade.packageName}.connect.client.grpc</shadedPattern>
<includes>
<include>io.grpc.**</include>
</includes>
</relocation>
<relocation>
<pattern>com.google.protobuf</pattern>
<shadedPattern>${spark.shade.packageName}.connect.protobuf</shadedPattern>
Expand All @@ -132,6 +140,10 @@
</includes>
</relocation>
</relocations>
<!--SPARK-42228: Add `ServicesResourceTransformer` to relocation class names in META-INF/services for grpc-->
Copy link
Contributor Author

@LuciferYang LuciferYang Feb 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For fix:

org.apache.spark.sql.ClientE2ETestSuite *** ABORTED ***
  org.sparkproject.connect.grpc.ManagedChannelProvider$ProviderNotFoundException: No functional channel service provider found. Try adding a dependency on the grpc-okhttp, grpc-netty, or grpc-netty-shaded artifact
  at org.sparkproject.connect.grpc.ManagedChannelProvider.provider(ManagedChannelProvider.java:45)
  at org.sparkproject.connect.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39)
  at org.apache.spark.sql.connect.client.SparkConnectClient$Builder.build(SparkConnectClient.scala:191)
  at org.apache.spark.sql.SparkSession$Builder.<init>(SparkSession.scala:143)
  at org.apache.spark.sql.SparkSession$.builder(SparkSession.scala:134)
  at org.apache.spark.sql.connect.client.util.RemoteSparkSession.beforeAll(RemoteSparkSession.scala:163)
  at org.apache.spark.sql.connect.client.util.RemoteSparkSession.beforeAll$(RemoteSparkSession.scala:160)
  at org.apache.spark.sql.ClientE2ETestSuite.beforeAll(ClientE2ETestSuite.scala:22)
  at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
  at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
  ...

Copy link
Contributor Author

@LuciferYang LuciferYang Feb 1, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before

ls META-INF/services 
io.grpc.LoadBalancerProvider	io.grpc.ManagedChannelProvider	io.grpc.NameResolverProvider	io.grpc.ServerProvider

cat META-INF/services/*
io.grpc.internal.PickFirstLoadBalancerProvider
io.grpc.util.SecretRoundRobinLoadBalancerProvider$Provider
io.grpc.netty.NettyChannelProvider
io.grpc.netty.UdsNettyChannelProvider
io.grpc.netty.UdsNameResolverProvider
io.grpc.netty.NettyServerProvider

After

ls META-INF/services/ 
org.sparkproject.connect.client.grpc.LoadBalancerProvider	org.sparkproject.connect.client.grpc.NameResolverProvider
org.sparkproject.connect.client.grpc.ManagedChannelProvider	org.sparkproject.connect.client.grpc.ServerProvider

cat META-INF/services/*
org.sparkproject.connect.client.grpc.internal.PickFirstLoadBalancerProvider
org.sparkproject.connect.client.grpc.util.SecretRoundRobinLoadBalancerProvider$Provider
org.sparkproject.connect.client.grpc.protobuf.services.internal.HealthCheckingRoundRobinLoadBalancerProvider
org.sparkproject.connect.client.grpc.netty.NettyChannelProvider
org.sparkproject.connect.client.grpc.netty.UdsNettyChannelProvider
org.sparkproject.connect.client.grpc.netty.UdsNameResolverProvider
org.sparkproject.connect.client.grpc.internal.DnsNameResolverProvider
org.sparkproject.connect.client.grpc.netty.NettyServerProvider

<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
</configuration>
</plugin>
</plugins>
Expand Down
1 change: 1 addition & 0 deletions project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -870,6 +870,7 @@ object SparkConnectClient {
},

(assembly / assemblyShadeRules) := Seq(
ShadeRule.rename("io.grpc.**" -> "org.sparkproject.connect.client.grpc.@0").inAll,
ShadeRule.rename("com.google.protobuf.**" -> "org.sparkproject.connect.protobuf.@1").inAll,
ShadeRule.rename("com.google.common.**" -> "org.sparkproject.connect.client.guava.@1").inAll,
ShadeRule.rename("com.google.thirdparty.**" -> "org.sparkproject.connect.client.guava.@1").inAll,
Expand Down