Skip to content

[WIP] Move ClientE2ETestSuite into a separate module #39788

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LuciferYang wants to merge 3 commits into from
Closed

[WIP] Move ClientE2ETestSuite into a separate module #39788

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d1712fc
first commit
LuciferYang Jan 29, 2023
22d6c58
add connect-client-integration-tests to modules.py
LuciferYang Jan 29, 2023
309347b
remove assembly
LuciferYang Jan 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 107 additions & 0 deletions connector/connect/client/integration-tests/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.5.0-SNAPSHOT</version>
<relativePath>../../../../pom.xml</relativePath>
</parent>

<artifactId>spark-connect-client-integration-tests_2.12</artifactId>
<packaging>jar</packaging>
<name>Spark Project Connect Client Integration Tests</name>
<url>https://spark.apache.org/</url>
<properties>
<sbt.project.name>connect-client-integration-tests</sbt.project.name>
<guava.version>31.0.1-jre</guava.version>
<protobuf.version>3.21.12</protobuf.version>
</properties>

<dependencies>
<!--
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${protobuf.version}</version>
<scope>test</scope>
</dependency>
-->
<!--
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
<scope>test</scope>
</dependency>
-->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-connect-client-jvm_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-connect-client-jvm_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
<scope>test</scope>
<version>${project.version}</version>
</dependency>
<dependency>
Copy link
Contributor Author

@LuciferYang LuciferYang Jan 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be <scope>test</scope>

<groupId>org.apache.spark</groupId>
<artifactId>spark-repl_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-connect-common_${scala.binary.version}</artifactId>
<scope>test</scope>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-connect_${scala.binary.version}</artifactId>
Copy link
Contributor Author

@LuciferYang LuciferYang Jan 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be <scope>test</scope>

<version>${project.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<build>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
</project>
39 changes: 39 additions & 0 deletions connector/connect/client/integration-tests/src/test/resources/log4j2.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Set everything to be logged to the file target/unit-tests.log
rootLogger.level = info
rootLogger.appenderRef.file.ref = ${sys:test.appender:-File}

appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Tests that launch java subprocesses can set the "test.appender" system property to
# "console" to avoid having the child process's logs overwrite the unit test's
# log file.
appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %t: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
logger.jetty.level = warn
0 ...apache/spark/sql/ClientE2ETestSuite.scala → ...apache/spark/sql/ClientE2ETestSuite.scala
View file
Open in desktop
File renamed without changes.
4 changes: 2 additions & 2 deletions ...nect/client/util/RemoteSparkSession.scala → ...nect/client/util/RemoteSparkSession.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ object SparkConnectServerUtils {
}
sys.props.getOrElse("spark.test.home", sys.env("SPARK_HOME"))
}
private val isDebug = System.getProperty(DEBUG_SC_JVM_CLIENT, "false").toBoolean
private val isDebug = System.getProperty(DEBUG_SC_JVM_CLIENT, "true").toBoolean

// Log server start stop debug info into console
// scalastyle:off println
Expand Down Expand Up @@ -121,7 +121,7 @@ object SparkConnectServerUtils {
val jars = recursiveListFiles(parentDir).filter { f =>
// SBT jar
(f.getParentFile.getName.startsWith("scala-") &&
f.getName.startsWith("spark-connect-assembly") && f.getName.endsWith(".jar")) ||
f.getName.startsWith("spark-connect") && f.getName.endsWith(".jar")) ||
// Maven Jar
(f.getParent.endsWith("target") &&
f.getName.startsWith("spark-connect") && f.getName.endsWith(".jar"))
Expand Down
27 changes: 5 additions & 22 deletions connector/connect/client/jvm/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,28 +65,6 @@
<version>${guava.version}</version>
<scope>compile</scope>
</dependency>
<!--
SPARK-42213: Add `repl` as test dependency to solve the issue that
`ClientE2ETestSuite` cannot find `repl.Main` when maven test.
-->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-repl_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
<!-- Exclude sql module to resolve compilation conflicts -->
<exclusions>
<exclusion>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.scalacheck</groupId>
<artifactId>scalacheck_${scala.binary.version}</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
Expand All @@ -105,11 +83,16 @@
<artifactSet>
<includes>
<include>com.google.guava:*</include>
<include>io.grpc:*:</include>
<include>com.google.protobuf:*</include>
<include>org.apache.spark:spark-connect-common_${scala.binary.version}</include>
</includes>
</artifactSet>
<relocations>
<relocation>
<pattern>io.grpc</pattern>
<shadedPattern>${spark.shade.packageName}.connect.grpc</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.protobuf</pattern>
<shadedPattern>${spark.shade.packageName}.connect.protobuf</shadedPattern>
Expand Down
1 change: 1 addition & 0 deletions dev/sparktestsupport/modules.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,7 @@ def __hash__(self):
sbt_test_goals=[
"connect/test",
"connect-client-jvm/test",
"connect-client-integration-tests/test",
],
)

Expand Down
1 change: 1 addition & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@
<module>connector/connect/server</module>
<module>connector/connect/common</module>
<module>connector/connect/client/jvm</module>
<module>connector/connect/client/integration-tests</module>
<module>connector/protobuf</module>
<!-- See additional modules enabled by profiles below -->
</modules>
Expand Down
32 changes: 30 additions & 2 deletions project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,13 +55,14 @@ object BuildCommons {
val connectCommon = ProjectRef(buildLocation, "connect-common")
val connect = ProjectRef(buildLocation, "connect")
val connectClient = ProjectRef(buildLocation, "connect-client-jvm")
val connectClientTests = ProjectRef(buildLocation, "connect-client-integration-tests")

val allProjects@Seq(
core, graphx, mllib, mllibLocal, repl, networkCommon, networkShuffle, launcher, unsafe, tags, sketch, kvstore, _*
) = Seq(
"core", "graphx", "mllib", "mllib-local", "repl", "network-common", "network-shuffle", "launcher", "unsafe",
"tags", "sketch", "kvstore"
).map(ProjectRef(buildLocation, _)) ++ sqlProjects ++ streamingProjects ++ Seq(connectCommon, connect, connectClient)
).map(ProjectRef(buildLocation, _)) ++ sqlProjects ++ streamingProjects ++ Seq(connectCommon, connect, connectClient, connectClientTests)

val optionallyEnabledProjects@Seq(kubernetes, mesos, yarn,
sparkGangliaLgpl, streamingKinesisAsl,
Expand Down Expand Up @@ -403,7 +404,8 @@ object SparkBuild extends PomBuild {
val mimaProjects = allProjects.filterNot { x =>
Seq(
spark, hive, hiveThriftServer, repl, networkCommon, networkShuffle, networkYarn,
unsafe, tags, tokenProviderKafka010, sqlKafka010, connectCommon, connect, connectClient, protobuf
unsafe, tags, tokenProviderKafka010, sqlKafka010, connectCommon, connect, connectClient,
connectClientTests, protobuf
).contains(x)
}

Expand Down Expand Up @@ -447,6 +449,7 @@ object SparkBuild extends PomBuild {
enable(SparkConnectCommon.settings)(connectCommon)
enable(SparkConnect.settings)(connect)
enable(SparkConnectClient.settings)(connectClient)
enable(SparkConnectClientTests.settings)(connectClientTests)

/* Protobuf settings */
enable(SparkProtobuf.settings)(protobuf)
Expand Down Expand Up @@ -870,6 +873,7 @@ object SparkConnectClient {
},

(assembly / assemblyShadeRules) := Seq(
ShadeRule.rename("io.grpc.**" -> "org.sparkproject.connect.grpc.@0").inAll,
ShadeRule.rename("com.google.protobuf.**" -> "org.sparkproject.connect.protobuf.@1").inAll,
ShadeRule.rename("com.google.common.**" -> "org.sparkproject.connect.client.guava.@1").inAll,
ShadeRule.rename("com.google.thirdparty.**" -> "org.sparkproject.connect.client.guava.@1").inAll,
Expand All @@ -891,6 +895,30 @@ object SparkConnectClient {
}
}

object SparkConnectClientTests {
lazy val settings = Seq(
excludeDependencies ++= {
Copy link
Contributor Author

@LuciferYang LuciferYang Jan 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

excludeDependencies does not work. There is a probability that sbt will fail to compile because spark-sql is in the classpath, which will cause package conflicts as follows:

[error] /${basedir}/spark-mine/connector/connect/client/integration-tests/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:27:21: value collectResult is not a member of org.apache.spark.sql.DataFrame
[error]     val schema = df.collectResult().schema
[error]                     ^
[error] /${basedir}/spark-mine/connector/connect/client/integration-tests/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:33:21: value collectResult is not a member of org.apache.spark.sql.DataFrame
[error]     val result = df.collectResult()
[error]                     ^
[error] /${basedir}/spark-mine/connector/connect/client/integration-tests/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:43:21: value collectResult is not a member of org.apache.spark.sql.Dataset[Long]
[error]     val result = df.collectResult()
[error]                     ^
[error] /${basedir}/spark-mine/connector/connect/client/integration-tests/src/test/scala/org/apache/spark/sql/connect/client/util/RemoteSparkSession.scala:152:36: value client is not a member of org.apache.spark.sql.SparkSession.Builder
[error]     spark = SparkSession.builder().client(SparkConnectClient.builder().port(port).build()).build()
[error]                                    ^
[error] /${basedir}/spark-mine/connector/connect/client/integration-tests/src/test/scala/org/apache/spark/sql/connect/client/util/RemoteSparkSession.scala:165:12: value collectResult is not a member of org.apache.spark.sql.DataFrame
[error] possible cause: maybe a semicolon is missing before `value collectResult'?
[error]           .collectResult()
[error]            ^
[error] 5 errors found
[error] (connect-client-integration-tests / Test / compileIncremental) Compilation failed
[error] Total time: 47 s, completed 2023-1-29 12:41:23

I can't solve it temporarily

val scalaBinaryVersion =
SbtPomKeys.effectivePom.value.getProperties.get("scala.binary.version").asInstanceOf[String]
Seq(
ExclusionRule("org.apache.spark", s"spark-sql_${scalaBinaryVersion}")
)
},
libraryDependencies ++= {
val guavaVersion =
SbtPomKeys.effectivePom.value.getProperties.get("guava.version").asInstanceOf[String]
Seq("com.google.guava" % "guava" % guavaVersion)
},
dependencyOverrides ++= {
val guavaVersion =
SbtPomKeys.effectivePom.value.getProperties.get("guava.version").asInstanceOf[String]
Seq(
"com.google.guava" % "guava" % guavaVersion
)
}
)
}

object SparkProtobuf {
import BuildCommons.protoVersion

Expand Down