Commits on May 24, 2020

1. [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2 …

   This PR aims to upgrade `kubernetes-client` library to bring the JDK8 related fixes. Please note that JDK11 works fine without any problem.
   - https://github.com/fabric8io/kubernetes-client/releases/tag/v4.9.2
     - JDK8 always uses http/1.1 protocol (Prevent OkHttp from wrongly enabling http/2)
   
   OkHttp "wrongly" detects the Platform as Jdk9Platform on JDK 8u251.
   - fabric8io/kubernetes-client#2212
   - https://stackoverflow.com/questions/61565751/why-am-i-not-able-to-run-sparkpi-example-on-a-kubernetes-k8s-cluster
   
   Although there is a workaround `export HTTP2_DISABLE=true` and `Downgrade JDK or K8s`, we had better avoid this problematic situation.
   
   No. This will recover the failures on JDK 8u252.
   
   - [x] Pass the Jenkins UT (#28601 (comment))
   - [x] Pass the Jenkins K8S IT with the K8s 1.13 (#28601 (comment))
   - [x] Manual testing with K8s 1.17.3. (Below)
   
   **v1.17.6 result (on Minikube)**
   ```
   KubernetesSuite:
   - Run SparkPi with no resources
   - Run SparkPi with a very long application name.
   - Use SparkLauncher.NO_RESOURCE
   - Run SparkPi with a master URL without a scheme.
   - Run SparkPi with an argument.
   - Run SparkPi with custom labels, annotations, and environment variables.
   - All pods have the same service account by default
   - Run extraJVMOptions check on driver
   - Run SparkRemoteFileTest using a remote data file
   - Run SparkPi with env and mount secrets.
   - Run PySpark on simple pi.py example
   - Run PySpark with Python2 to test a pyfiles example
   - Run PySpark with Python3 to test a pyfiles example
   - Run PySpark with memory customization
   - Run in client mode.
   - Start pod creation from template
   - PVs with local storage
   - Launcher client dependencies
   - Test basic decommissioning
   Run completed in 8 minutes, 27 seconds.
   Total number of tests run: 19
   Suites: completed 2, aborted 0
   Tests: succeeded 19, failed 0, canceled 0, ignored 0, pending 0
   All tests passed.
   ```
   
   Closes #28601 from dongjoon-hyun/SPARK-K8S-CLIENT.
   
   Authored-by: Dongjoon Hyun <dongjoon@apache.org>
   Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
   (cherry picked from commit 64ffc66)
   Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
   (cherry picked from commit f05a26a)
   Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

   

   dongjoon-hyun committed May 24, 2020
   Configuration menu

   • View commit details
   • Copy full SHA for 8bbc46a
   • Browse repository at this point

   Copy the full SHA

   8bbc46a View commit details

   Browse the repository at this point in the history