From 70e752f3ea5f0ebfc5ff7927a1184bc49973c852 Mon Sep 17 00:00:00 2001 From: HyukjinKwon Date: Thu, 3 Sep 2020 17:05:51 +0900 Subject: [PATCH 1/5] Development - Testing PySpark --- python/docs/source/development/index.rst | 2 +- python/docs/source/development/testing.rst | 61 ++++++++++++++++++++++ 2 files changed, 62 insertions(+), 1 deletion(-) create mode 100644 python/docs/source/development/testing.rst diff --git a/python/docs/source/development/index.rst b/python/docs/source/development/index.rst index 246822613800..ca628f73d21f 100644 --- a/python/docs/source/development/index.rst +++ b/python/docs/source/development/index.rst @@ -23,4 +23,4 @@ Development :maxdepth: 2 contributing - + testing diff --git a/python/docs/source/development/testing.rst b/python/docs/source/development/testing.rst new file mode 100644 index 000000000000..182041c18529 --- /dev/null +++ b/python/docs/source/development/testing.rst @@ -0,0 +1,61 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +=============== +Testing PySpark +=============== + +In order to run PySpark tests, you should build Spark itself first via Maven +or SBT. For example, + +.. code-block:: bash + + ./build/mvn -DskipTests clean package + +After that, the PySpark test cases can be run via using ``python/run-tests``. For example, + +.. code-block:: bash + + $ python/run-tests --python-executable=python3 + +Note that: + +* If you are running tests on Mac OS, you may set ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment variable to ``YES``. +* If you're using JDK 11, you should set ``-Dio.netty.tryReflectionSetAccessible=true`` for Arrow related features. See also `Downloading `_. + +Please see the guidance on how to `build Spark `_, +`run tests for a module, or individual tests `_. + + +Running Individual PySpark Tests +-------------------------------- + +You can run a specific test via using ``python/run-tests``, for example, as below: + +.. code-block:: bash + + $ python/run-tests --testnames pyspark.sql.tests.test_arrow + +Please refer `Testing PySpark `_ for more details. + + +Running tests using GitHub Actions +---------------------------------- + +You can run the full PySpark tests by using GitHub Actions in your own forked GitHub +repositry by few clicks. Please refer +`Running tests in your forked repository using GitHub Actions `_ for more details. From abe8a8e3391ada399e1f7b31767b50bb1b4c7bee Mon Sep 17 00:00:00 2001 From: HyukjinKwon Date: Thu, 3 Sep 2020 17:18:21 +0900 Subject: [PATCH 2/5] nits --- python/docs/source/development/testing.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/python/docs/source/development/testing.rst b/python/docs/source/development/testing.rst index 182041c18529..b1cd7bba9dbf 100644 --- a/python/docs/source/development/testing.rst +++ b/python/docs/source/development/testing.rst @@ -24,13 +24,13 @@ or SBT. For example, .. code-block:: bash - ./build/mvn -DskipTests clean package + build/mvn -DskipTests clean package After that, the PySpark test cases can be run via using ``python/run-tests``. For example, .. code-block:: bash - $ python/run-tests --python-executable=python3 + python/run-tests --python-executable=python3 Note that: @@ -48,7 +48,7 @@ You can run a specific test via using ``python/run-tests``, for example, as belo .. code-block:: bash - $ python/run-tests --testnames pyspark.sql.tests.test_arrow + python/run-tests --testnames pyspark.sql.tests.test_arrow Please refer `Testing PySpark `_ for more details. From ee723db379bf571a2f51939df60bf3958517aba3 Mon Sep 17 00:00:00 2001 From: HyukjinKwon Date: Thu, 3 Sep 2020 17:19:12 +0900 Subject: [PATCH 3/5] typo --- python/docs/source/development/testing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/docs/source/development/testing.rst b/python/docs/source/development/testing.rst index b1cd7bba9dbf..faf854665ae3 100644 --- a/python/docs/source/development/testing.rst +++ b/python/docs/source/development/testing.rst @@ -35,7 +35,7 @@ After that, the PySpark test cases can be run via using ``python/run-tests``. Fo Note that: * If you are running tests on Mac OS, you may set ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment variable to ``YES``. -* If you're using JDK 11, you should set ``-Dio.netty.tryReflectionSetAccessible=true`` for Arrow related features. See also `Downloading `_. +* If you are using JDK 11, you should set ``-Dio.netty.tryReflectionSetAccessible=true`` for Arrow related features. See also `Downloading `_. Please see the guidance on how to `build Spark `_, `run tests for a module, or individual tests `_. From 46934b3470480d7c1cda711289546ae9ba419a6b Mon Sep 17 00:00:00 2001 From: HyukjinKwon Date: Fri, 4 Sep 2020 12:53:35 +0900 Subject: [PATCH 4/5] Address comments --- python/docs/source/development/testing.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/python/docs/source/development/testing.rst b/python/docs/source/development/testing.rst index faf854665ae3..223970be4329 100644 --- a/python/docs/source/development/testing.rst +++ b/python/docs/source/development/testing.rst @@ -19,8 +19,7 @@ Testing PySpark =============== -In order to run PySpark tests, you should build Spark itself first via Maven -or SBT. For example, +In order to run PySpark tests, you should build Spark itself first via Maven or SBT. For example, .. code-block:: bash @@ -50,12 +49,12 @@ You can run a specific test via using ``python/run-tests``, for example, as belo python/run-tests --testnames pyspark.sql.tests.test_arrow -Please refer `Testing PySpark `_ for more details. +Please refer to `Testing PySpark `_ for more details. Running tests using GitHub Actions ---------------------------------- You can run the full PySpark tests by using GitHub Actions in your own forked GitHub -repositry by few clicks. Please refer +repositry with a few clicks. Please refer to `Running tests in your forked repository using GitHub Actions `_ for more details. From 9dac672d0d0545fd7060e7ee9f14ac50428bbbce Mon Sep 17 00:00:00 2001 From: HyukjinKwon Date: Fri, 4 Sep 2020 14:45:06 +0900 Subject: [PATCH 5/5] Address comments --- python/docs/source/development/testing.rst | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/python/docs/source/development/testing.rst b/python/docs/source/development/testing.rst index 223970be4329..08fd730a19f4 100644 --- a/python/docs/source/development/testing.rst +++ b/python/docs/source/development/testing.rst @@ -31,10 +31,7 @@ After that, the PySpark test cases can be run via using ``python/run-tests``. Fo python/run-tests --python-executable=python3 -Note that: - -* If you are running tests on Mac OS, you may set ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment variable to ``YES``. -* If you are using JDK 11, you should set ``-Dio.netty.tryReflectionSetAccessible=true`` for Arrow related features. See also `Downloading `_. +Note that you may set ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment variable to ``YES`` if you are running tests on Mac OS. Please see the guidance on how to `build Spark `_, `run tests for a module, or individual tests `_.