[SPARK-31819][K8S][DOCS][TESTS][2.4] Add a workaround for Java 8u251+/K8s 1.17 and update integration test cases #28638
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…update integration test cases
|
Test build #123107 has finished for PR 28638 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
dongjoon-hyun
changed the title
dongjoon-hyun
changed the title
|
LGTM. Thanks. |
|
Thank you. Merged to branch-2.4 |
dongjoon-hyun
added a commit
that referenced
this pull request
May 26, 2020
…/K8s 1.17 and update integration test cases ### What changes were proposed in this pull request? This PR aims to add a workaround `HTTP2_DISABLE=true` to the document and to update the K8s integration test. ### Why are the changes needed? SPARK-31786 reported fabric8 kubernetes-client library fails to talk K8s 1.17.x client on Java 8u251+ environment. It's fixed at Apache Spark 3.0.0 by upgrading the library, but it turns out that we can not use the same way in `branch-2.4` (#28625) ### Does this PR introduce _any_ user-facing change? Yes. This will provide a workaround at the document and testing environment. ### How was this patch tested? This PR is irrelevant to Jenkins UT because it's only updating docs and integration tests. We need to the followings. - [x] Pass the Jenkins K8s IT with old JDK8 and K8s versions (#28638 (comment)) - [x] Manually run K8s IT on K8s 1.17/Java 8u251+ with `export HTTP2_DISABLE=true`. **K8s v1.17.6 / JDK 1.8.0_252** ``` KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. Run completed in 5 minutes, 7 seconds. Total number of tests run: 14 Suites: completed 2, aborted 0 Tests: succeeded 14, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes #28638 from dongjoon-hyun/SPARK-31819. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
lresende
pushed a commit
to jupyter-server/enterprise_gateway
that referenced
this pull request
Jul 16, 2020
A bug in the k8s client relative to Java 8 version used by Spark 2.4.5 crept in since the last test pass. Unfortunately, this wasn't caught until EG 2.2.0rc1 had been produced. These changes workaround the issue that is apparently fixed in Spark 3.0. cf: apache/spark#28638
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR aims to add a workaround
HTTP2_DISABLE=trueto the document and to update the K8s integration test.Why are the changes needed?
SPARK-31786 reported fabric8 kubernetes-client library fails to talk K8s 1.17.x client on Java 8u251+ environment. It's fixed at Apache Spark 3.0.0 by upgrading the library, but it turns out that we can not use the same way in
branch-2.4(#28625)Does this PR introduce any user-facing change?
Yes. This will provide a workaround at the document and testing environment.
How was this patch tested?
This PR is irrelevant to Jenkins UT because it's only updating docs and integration tests. We need to the followings.
export HTTP2_DISABLE=true.K8s v1.17.6 / JDK 1.8.0_252