From 9aec3e387a26ca8b8d33817f1f7978b9ebbb68a1 Mon Sep 17 00:00:00 2001 From: Kay Ousterhout Date: Sat, 11 Feb 2017 18:38:46 -0800 Subject: [PATCH 1/3] Add instructions for running individual tests. This is useful and I often forget how to do it. I learned some new tricks when @squito gave @jinxing64 some tips on how to do this, so I thought it was worth adding this to the website. --- developer-tools.md | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/developer-tools.md b/developer-tools.md index 77d225f4dfd..3afde9c61e0 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -32,6 +32,49 @@ $ ./bin/spark-shell $ build/sbt ~compile ``` +

Running Individual Tests

+ +When developing locally, it's often convenient to run a single test or a few tests, rather than running the entire test suite. + +

Testing with SBT

+ +The fastest way to run individual tests is to use the `sbt` console. It's fastest to keep a `sbt` console open, and use it to re-run tests as necessary. For example, to run the DAGSchedulerSuite, which is in the `core` project: + +``` +$ build/sbt +> project core +> testOnly *DAGSchedulerSuite +``` + +If you'd like to run just a single test in the DAGSchedulerSuite, e.g., a test that includes "SPARK-12345" in the name, you run the following command in the sbt console: + +``` +> testOnly *DAGSchedulerSuite -- -z "SPARK-12345" +``` + +Alternately, in the sbt console, you can run all tests in the scheduler package: + +``` +> testOnly org.apache.spark.scheduler.* +``` + +If you'd prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open cosole). To do this, you need to surround `testOnly` and the following arguments in quotes: + +``` +$ build/sbt "core/testOnly *DAGSchedulerSuite -- -z SPARK-12345" +``` + +For more about how to run individual tests with sbt, see the [sbt documentation](http://www.scala-sbt.org/0.13/docs/Testing.html). + + +

Testing with Maven

+ +You can use wildcards to run individual tests with Maven as follows: + +``` +build/mvn test -DwildcardSuites=org.apache.spark.scheduler.**Suite +``` +

Checking Out Pull Requests

Git provides a mechanism for fetching remote pull requests into your own local repository. From 608233ad7c991833fafa89e2ffb21a09fa12d451 Mon Sep 17 00:00:00 2001 From: Kay Ousterhout Date: Sat, 11 Feb 2017 18:44:37 -0800 Subject: [PATCH 2/3] Added html changes --- site/developer-tools.html | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/site/developer-tools.html b/site/developer-tools.html index 24b1ce19146..5b8523a390a 100644 --- a/site/developer-tools.html +++ b/site/developer-tools.html @@ -216,6 +216,43 @@

Reducing Build Times

$ build/sbt ~compile +

Running Individual Tests

+ +

When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.

+ +

Testing with SBT

+ +

The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run the DAGSchedulerSuite, which is in the core project:

+ +
$ build/sbt
+> project core
+> testOnly *DAGSchedulerSuite
+
+ +

If you’d like to run just a single test in the DAGSchedulerSuite, e.g., a test that includes “SPARK-12345” in the name, you run the following command in the sbt console:

+ +
> testOnly *DAGSchedulerSuite -- -z "SPARK-12345"
+
+ +

Alternately, in the sbt console, you can run all tests in the scheduler package:

+ +
> testOnly org.apache.spark.scheduler.*
+
+ +

If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open cosole). To do this, you need to surround testOnly and the following arguments in quotes:

+ +
$ build/sbt "core/testOnly *DAGSchedulerSuite -- -z SPARK-12345"
+
+ +

For more about how to run individual tests with sbt, see the sbt documentation.

+ +

Testing with Maven

+ +

You can use wildcards to run individual tests with Maven as follows:

+ +
build/mvn test -DwildcardSuites=org.apache.spark.scheduler.**Suite
+
+

Checking Out Pull Requests

Git provides a mechanism for fetching remote pull requests into your own local repository. From 41459f2c3c79b12f65f636cee0c4da5616daa981 Mon Sep 17 00:00:00 2001 From: Kay Ousterhout Date: Tue, 21 Feb 2017 14:08:58 -0800 Subject: [PATCH 3/3] Fixed mvn test desscription and added Zinc info. --- developer-tools.md | 54 ++++++++++++++++++++++++++++++++------- site/developer-tools.html | 51 +++++++++++++++++++++++++++++------- 2 files changed, 87 insertions(+), 18 deletions(-) diff --git a/developer-tools.md b/developer-tools.md index 3afde9c61e0..c8fd4373c3c 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -9,7 +9,9 @@ navigation:

Useful Developer Tools

-

Reducing Build Times

+

Reducing Build Times

+ +

SBT: Avoiding Re-Creating the Assembly JAR

Spark's default build strategy is to assemble a jar including all of its dependencies. This can be cumbersome when doing iterative development. When developing locally, it is possible to create @@ -32,32 +34,58 @@ $ ./bin/spark-shell $ build/sbt ~compile ``` -

Running Individual Tests

+

Maven: Speeding up Compilation with Zinc

+ +[Zinc](https://github.com/typesafehub/zinc) is a long-running server version of SBT's incremental +compiler. When run locally as a background process, it speeds up builds of Scala-based projects +like Spark. Developers who regularly recompile Spark with Maven will be the most interested in +Zinc. The project site gives instructions for building and running `zinc`; OS X users can +install it using `brew install zinc`. + +If using the `build/mvn` package `zinc` will automatically be downloaded and leveraged for all +builds. This process will auto-start after the first time `build/mvn` is called and bind to port +3030 unless the `ZINC_PORT` environment variable is set. The `zinc` process can subsequently be +shut down at any time by running `build/zinc-/bin/zinc -shutdown` and will automatically +restart whenever `build/mvn` is called. + +

Running Individual Tests

When developing locally, it's often convenient to run a single test or a few tests, rather than running the entire test suite.

Testing with SBT

-The fastest way to run individual tests is to use the `sbt` console. It's fastest to keep a `sbt` console open, and use it to re-run tests as necessary. For example, to run the DAGSchedulerSuite, which is in the `core` project: +The fastest way to run individual tests is to use the `sbt` console. It's fastest to keep a `sbt` console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., `core`: ``` $ build/sbt > project core -> testOnly *DAGSchedulerSuite +> test ``` -If you'd like to run just a single test in the DAGSchedulerSuite, e.g., a test that includes "SPARK-12345" in the name, you run the following command in the sbt console: +You can run a single test suite using the `testOnly` command. For example, to run the DAGSchedulerSuite: ``` -> testOnly *DAGSchedulerSuite -- -z "SPARK-12345" +> testOnly org.apache.spark.scheduler.DAGSchedulerSuite +``` + +The `testOnly` command accepts wildcards; e.g., you can also run the `DAGSchedulerSuite` with: + +``` +> testOnly *DAGSchedulerSuite ``` -Alternately, in the sbt console, you can run all tests in the scheduler package: +Or you could run all of the tests in the scheduler package: ``` > testOnly org.apache.spark.scheduler.* ``` +If you'd like to run just a single test in the `DAGSchedulerSuite`, e.g., a test that includes "SPARK-12345" in the name, you run the following command in the sbt console: + +``` +> testOnly *DAGSchedulerSuite -- -z "SPARK-12345" +``` + If you'd prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open cosole). To do this, you need to surround `testOnly` and the following arguments in quotes: ``` @@ -69,10 +97,18 @@ For more about how to run individual tests with sbt, see the [sbt documentation]

Testing with Maven

-You can use wildcards to run individual tests with Maven as follows: +With Maven, you can use the `-DwildcardSuites` flag to run individual Scala tests: + +``` +build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.scheduler.DAGSchedulerSuite test +``` + +You need `-Dtest=none` to avoid running the Java tests. For more information about the ScalaTest Maven Plugin, refer to the [ScalaTest documentation](http://www.scalatest.org/user_guide/using_the_scalatest_maven_plugin). + +To run individual Java tests, you can use the `-Dtest` flag: ``` -build/mvn test -DwildcardSuites=org.apache.spark.scheduler.**Suite +build/mvn test -DwildcardSuites=none -Dtest=org.apache.spark.streaming.JavaAPISuite test ```

Checking Out Pull Requests

diff --git a/site/developer-tools.html b/site/developer-tools.html index 5b8523a390a..095e09f28ef 100644 --- a/site/developer-tools.html +++ b/site/developer-tools.html @@ -194,7 +194,9 @@
Latest News

Useful Developer Tools

-

Reducing Build Times

+

Reducing Build Times

+ +

SBT: Avoiding Re-Creating the Assembly JAR

Spark’s default build strategy is to assemble a jar including all of its dependencies. This can be cumbersome when doing iterative development. When developing locally, it is possible to create @@ -216,29 +218,53 @@

Reducing Build Times

$ build/sbt ~compile -

Running Individual Tests

+

Maven: Speeding up Compilation with Zinc

+ +

Zinc is a long-running server version of SBT’s incremental +compiler. When run locally as a background process, it speeds up builds of Scala-based projects +like Spark. Developers who regularly recompile Spark with Maven will be the most interested in +Zinc. The project site gives instructions for building and running zinc; OS X users can +install it using brew install zinc.

+ +

If using the build/mvn package zinc will automatically be downloaded and leveraged for all +builds. This process will auto-start after the first time build/mvn is called and bind to port +3030 unless the ZINC_PORT environment variable is set. The zinc process can subsequently be +shut down at any time by running build/zinc-<version>/bin/zinc -shutdown and will automatically +restart whenever build/mvn is called.

+ +

Running Individual Tests

When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.

Testing with SBT

-

The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run the DAGSchedulerSuite, which is in the core project:

+

The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., core:

$ build/sbt
 > project core
-> testOnly *DAGSchedulerSuite
+> test
-

If you’d like to run just a single test in the DAGSchedulerSuite, e.g., a test that includes “SPARK-12345” in the name, you run the following command in the sbt console:

+

You can run a single test suite using the testOnly command. For example, to run the DAGSchedulerSuite:

-
> testOnly *DAGSchedulerSuite -- -z "SPARK-12345"
+
> testOnly org.apache.spark.scheduler.DAGSchedulerSuite
+

+
+
The testOnly command accepts wildcards; e.g., you can also run the DAGSchedulerSuite with:

+
+
> testOnly *DAGSchedulerSuite
 

 
-
Alternately, in the sbt console, you can run all tests in the scheduler package:

+
Or you could run all of the tests in the scheduler package:

 
 
> testOnly org.apache.spark.scheduler.*
 

 
+
If you’d like to run just a single test in the DAGSchedulerSuite, e.g., a test that includes “SPARK-12345” in the name, you run the following command in the sbt console:

+
+
> testOnly *DAGSchedulerSuite -- -z "SPARK-12345"
+

+
 
If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open cosole).  To do this, you need to surround testOnly and the following arguments in quotes:

 
 
$ build/sbt "core/testOnly *DAGSchedulerSuite -- -z SPARK-12345"
@@ -248,9 +274,16 @@ 
Testing with SBT

 
 
Testing with Maven

 
-
You can use wildcards to run individual tests with Maven as follows:

+
With Maven, you can use the -DwildcardSuites flag to run individual Scala tests:

+
+
build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.scheduler.DAGSchedulerSuite test
+

+
+
You need -Dtest=none to avoid running the Java tests.  For more information about the ScalaTest Maven Plugin, refer to the ScalaTest documentation.

+
+
To run individual Java tests, you can use the -Dtest flag:

 
-
build/mvn test -DwildcardSuites=org.apache.spark.scheduler.**Suite
+
build/mvn test -DwildcardSuites=none -Dtest=org.apache.spark.streaming.JavaAPISuite test
 

 
 
Checking Out Pull Requests