Skip to content

[SPARK-42587][CONNECT][TESTS] Use wrapper versions for SBT and Maven in connect module tests #40180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 2 commits into from
Closed

[SPARK-42587][CONNECT][TESTS] Use wrapper versions for SBT and Maven in connect module tests #40180

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
af2bdee
[SPARK-42587][CONNECT][TESTS] Use wrapper versions for SBT and Maven …
dongjoon-hyun Feb 27, 2023
958cd47
Fix scalastyle
dongjoon-hyun Feb 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions ...ct/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/CompatibilitySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,14 +39,15 @@ import org.apache.spark.sql.connect.client.util.IntegrationTestUtils._
* spark-sql
* spark-connect-client-jvm
* }}}
* To build the above artifact, use e.g. `sbt package` or `mvn clean install -DskipTests`.
* To build the above artifact, use e.g. `build/sbt package` or
* `build/mvn clean install -DskipTests`.
*
* When debugging this test, if any changes to the client API, the client jar need to be built
* before running the test. An example workflow with SBT for this test:
* 1. Compatibility test has reported an unexpected client API change.
* 1. Fix the wrong client API.
* 1. Build the client jar: `sbt package`
* 1. Run the test again: `sbt "testOnly
* 1. Build the client jar: `build/sbt package`
* 1. Run the test again: `build/sbt "testOnly
* org.apache.spark.sql.connect.client.CompatibilitySuite"`
*/
class CompatibilitySuite extends ConnectFunSuite {
Expand Down
7 changes: 4 additions & 3 deletions ...nt/jvm/src/test/scala/org/apache/spark/sql/connect/client/util/IntegrationTestUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,9 @@ object IntegrationTestUtils {
private[connect] def debug(error: Throwable): Unit = if (isDebug) error.printStackTrace()

/**
* Find a jar in the Spark project artifacts. It requires a build first (e.g. sbt package, mvn
* clean install -DskipTests) so that this method can find the jar in the target folders.
* Find a jar in the Spark project artifacts. It requires a build first (e.g. build/sbt package,
* build/mvn clean install -DskipTests) so that this method can find the jar in the target
* folders.
*
* @return
* the jar
Expand All @@ -52,7 +53,7 @@ object IntegrationTestUtils {
targetDir.exists(),
s"Fail to locate the target folder: '${targetDir.getCanonicalPath}'. " +
s"SPARK_HOME='${new File(sparkHome).getCanonicalPath}'. " +
"Make sure the spark project jars has been built (e.g. using sbt package)" +
"Make sure the spark project jars has been built (e.g. using build/sbt package)" +
"and the env variable `SPARK_HOME` is set correctly.")
val jars = recursiveListFiles(targetDir).filter { f =>
// SBT jar
Expand Down
3 changes: 2 additions & 1 deletion ...ient/jvm/src/test/scala/org/apache/spark/sql/connect/client/util/RemoteSparkSession.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ import org.apache.spark.util.Utils

/**
* An util class to start a local spark connect server in a different process for local E2E tests.
* Pre-running the tests, the spark connect artifact needs to be built using e.g. `sbt package`.
* Pre-running the tests, the spark connect artifact needs to be built using e.g.
* `build/sbt package`.
* It is designed to start the server once but shared by all tests. It is equivalent to use the
* following command to start the connect server via command line:
*
Expand Down