misc(Python): Replace python 2 with python 3

Python 2 has reached EOL last year and should not be used anymore. This commit replaces all references to the "python" binary with the more explicit "python3" binary. If desired, the build can still be performed for Python 2 by settings the "PYTHON_EXECUTABLE" environment variable to an appropriate version. Additionally, python wheels are the preferred way to distribute python code (see https://packaging.python.org/discussions/wheel-vs-egg/). This commit additionally builds the job-server-python wheel.

Spark-2.4 does not support python >= 3.8 (see apache/spark#26194) leading to failed test cases (TypeError: an integer is required (got type bytes)). If you encounter these issues try to state a python executable < 3.8 explicitly.

build.sbt

@@ -115,8 +115,8 @@ lazy val jobServerExtrasSettings = revolverSettings ++ Assembly.settings ++ publ
 lazy val jobServerApiSettings = Seq(libraryDependencies ++= sparkDeps ++ sparkExtraDeps)
 
 lazy val testPython = taskKey[Unit]("Launch a sub process to run the Python tests")
-lazy val buildPython = taskKey[Unit]("Build the python side of python support into a wheel")
-lazy val buildPyExamples = taskKey[Unit]("Build the examples of python jobs into a wheel")
+lazy val buildPython = taskKey[Unit]("Build the python side of python support into a wheel and egg")
+lazy val buildPyExamples = taskKey[Unit]("Build the examples of python jobs into a wheel and egg")
 
 lazy val jobServerPythonSettings = revolverSettings ++ Assembly.settings ++ publishSettings ++ Seq(
   libraryDependencies ++= sparkPythonDeps,

doc/python.md

@@ -10,6 +10,7 @@
   - [Running a job](#running-a-job)
   - [PythonSessionContext](#pythonsessioncontext)
   - [CustomContexts](#customcontexts)
+  - [Troubleshooting](#troubleshooting)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
@@ -266,3 +267,12 @@ The Python support can support arbitrary context types as long as they are based
 contexts of your custom type, your Python jobs which use this context must implement an additional method,
 `build_context(self, gateway, jvmContext, sparkConf)`, which returns the Python equivalent of the JVM Context object.
 For a simple example, see `CustomContextJob` in the `job-server-python` sub-module.
+
+
+## Troubleshooting
+
+### TypeError: an integer is required (got type bytes)
+
+Spark-2.4 does not support python >= 3.8 (see [here](https://github.com/apache/spark/pull/26194) for more information).
+If you encounter this issue, please verify that you provide a python executable < 3.8 in your
+[configuration file](../job-server/src/main/resources/application.conf#L210).

job-server/src/main/resources/application.conf

@@ -207,6 +207,7 @@ spark {
       #Any locations of python libraries to be included on the PYTHONPATH when running Python jobs
       paths = []
       #The shell command to run when launching the subprocess for python jobs
+      #Please note: Spark2 currently only supports Python < 3.8.
       executable = "python3"
     }