Skip to content

[DON'T MERGE][BUILD] Switch default protobuf-java version to 3.x #39255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LuciferYang wants to merge 5 commits into from
Closed

[DON'T MERGE][BUILD] Switch default protobuf-java version to 3.x #39255

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f3b817d
test use new pb
LuciferYang Dec 22, 2022
b696af9
Merge branch 'apache:master' into remove-old-protobuf
LuciferYang Dec 28, 2022
403a7aa
remove redundant declare
LuciferYang Dec 28, 2022
76b7036
upgrade deps
LuciferYang Jan 1, 2023
6c3976f
Merge branch 'upmaster' into remove-old-protobuf
LuciferYang Jan 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions connector/connect/client/jvm/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,6 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${protobuf.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.scalacheck</groupId>
Expand Down
2 changes: 0 additions & 2 deletions connector/connect/common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,6 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${protobuf.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
Expand Down
2 changes: 0 additions & 2 deletions connector/connect/server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,8 +139,6 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${protobuf.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
Expand Down
1 change: 1 addition & 0 deletions connector/kafka-0-10-assembly/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.5.0</version>
<scope>provided</scope>
</dependency>
<dependency>
Expand Down
2 changes: 0 additions & 2 deletions connector/protobuf/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,6 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${protobuf.version}</version>
<scope>compile</scope>
</dependency>
</dependencies>
<build>
Expand Down
2 changes: 0 additions & 2 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -535,8 +535,6 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${protobuf.version}</version>
<scope>compile</scope>
</dependency>
</dependencies>
<build>
Expand Down
2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-2-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ parquet-format-structures/1.12.3//parquet-format-structures-1.12.3.jar
parquet-hadoop/1.12.3//parquet-hadoop-1.12.3.jar
parquet-jackson/1.12.3//parquet-jackson-1.12.3.jar
pickle/1.3//pickle-1.3.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
protobuf-java/3.21.12//protobuf-java-3.21.12.jar
py4j/0.10.9.7//py4j-0.10.9.7.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
rocksdbjni/7.8.3//rocksdbjni-7.8.3.jar
Expand Down
2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ parquet-format-structures/1.12.3//parquet-format-structures-1.12.3.jar
parquet-hadoop/1.12.3//parquet-hadoop-1.12.3.jar
parquet-jackson/1.12.3//parquet-jackson-1.12.3.jar
pickle/1.3//pickle-1.3.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
protobuf-java/3.21.12//protobuf-java-3.21.12.jar
py4j/0.10.9.7//py4j-0.10.9.7.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
rocksdbjni/7.8.3//rocksdbjni-7.8.3.jar
Expand Down
10 changes: 1 addition & 9 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,6 @@
<log4j.version>2.19.0</log4j.version>
<!-- make sure to update IsolatedClientLoader whenever this version is changed -->
<hadoop.version>3.3.4</hadoop.version>
<!-- Protobuf version for building with Hadoop/Yarn dependencies -->
<protobuf.hadoopDependency.version>2.5.0</protobuf.hadoopDependency.version>
<!-- Actual Protobuf version in Spark modules like Spark Connect, protobuf connector, etc. -->
<!-- SPARK-41247: When updating `protobuf.version`, also need to update `protoVersion` in `SparkBuild.scala` -->
<protobuf.version>3.21.12</protobuf.version>
Expand Down Expand Up @@ -819,16 +817,10 @@
</exclusion>
</exclusions>
</dependency>
<!-- In theory we need not directly depend on protobuf since Spark does not directly
use it. However, when building with Hadoop/YARN 2.2 Maven doesn't correctly bump
the protobuf version up from the one Mesos gives. For now we include this variable
to explicitly bump the version when building with YARN. It would be nice to figure
out why Maven can't resolve this correctly (like SBT does). -->
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${protobuf.hadoopDependency.version}</version>
<scope>${hadoop.deps.scope}</scope>
<version>${protobuf.version}</version>
Copy link
Contributor Author

@LuciferYang LuciferYang Dec 28, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tested all maven test with -Phadoop-2 and -Phadoop-3, and all tests passed.

If feasible, we don't need to explicitly declare protobuf.version in each module that requires protobuf-java 3.x(such as core, sql, connect and protobuf), but this also changes protobuf-java version in dev/deps/spark-deps-hadoop-x-hive-2.3 from 2.5.0 to 3.x

@gengliangwang

Copy link
Contributor Author

@LuciferYang LuciferYang Dec 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course, it may be safer to remove -Phadoop-2 completely before doing this work

</dependency>
<dependency>
<groupId>org.roaringbitmap</groupId>
Expand Down
5 changes: 0 additions & 5 deletions sql/core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,11 +147,6 @@
<groupId>org.apache.xbean</groupId>
<artifactId>xbean-asm9-shaded</artifactId>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${protobuf.version}</version>
</dependency>
<dependency>
<groupId>org.scalacheck</groupId>
<artifactId>scalacheck_${scala.binary.version}</artifactId>
Expand Down