[SPARK-30715][K8S] Bump fabric8 to 4.7.1

[onursatici]

What changes were proposed in this pull request?

Bump fabric8 kubernetes-client to 4.7.1

Why are the changes needed?

New fabric8 version brings support for Kubernetes 1.17 clusters.
Full release notes:

• https://github.com/fabric8io/kubernetes-client/releases/tag/v4.7.0
• https://github.com/fabric8io/kubernetes-client/releases/tag/v4.7.1

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing unit and integration tests cover creation of K8S objects. Adjusted them to work with the new fabric8 version

[dongjoon-hyun]

ok to test

[dongjoon-hyun]

Hi, @onursatici . Is this for improvement or for a bug fix?

[SparkQA]

Test build #117779 has finished for PR 27443 at commit 9b305b7.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22541/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22541/

[dongjoon-hyun]

@onursatici . Ur, could you run K8s integration test on this PR locally first?

[INFO] Compiling 19 Scala sources to /home/jenkins/workspace/SparkPullRequestBuilder-K8s/resource-managers/kubernetes/integration-tests/target/test-classes ...
[ERROR] [Error] /home/jenkins/workspace/SparkPullRequestBuilder-K8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:58: ')' expected but string literal found.
[ERROR] [Error] /home/jenkins/workspace/SparkPullRequestBuilder-K8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:59: ';' expected but ')' found.

[SparkQA]

Test build #117780 has finished for PR 27443 at commit 9f6fc54.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

@onursatici . Since you are active contributor in K8s area, I want to recommend to keep all sections of the PR template. Especially, ### Why are the changes needed? section. You can add the following release note at that section simply.

• https://github.com/fabric8io/kubernetes-client/releases/tag/v4.7.0
• https://github.com/fabric8io/kubernetes-client/releases/tag/v4.7.1

In short, please recover the deleted two sections.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22542/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22542/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22543/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22543/

[dongjoon-hyun]

It seems that there exist 3 failures.

KubernetesSuite:
- Run SparkPi with no resources *** FAILED ***
  The code passed to eventually never returned normally. Attempted 70 times over 2.0014648788833336 minutes. Last failure message: false was not true. (KubernetesSuite.scala:315)
- Run SparkPi with a very long application name. *** FAILED ***
  The code passed to eventually never returned normally. Attempted 70 times over 2.0008292973333335 minutes. Last failure message: false was not true. (KubernetesSuite.scala:315)
...
- Launcher client dependencies *** FAILED ***
  The code passed to eventually never returned normally. Attempted 130 times over 4.00124932035 minutes. Last failure message: false was not true. (KubernetesSuite.scala:315)

[SparkQA]

Test build #117781 has finished for PR 27443 at commit 2a3a946.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[onursatici]

Thanks for having a look @dongjoon-hyun, will fix the integrations tests and will let you know

[onursatici]

retest this please

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22612/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22612/

[dongjoon-hyun]

Oh, it seems that two failures are gone. The remaining one is known to be a flaky test.

- Launcher client dependencies *** FAILED ***
  The code passed to eventually never returned normally. Attempted 130 times over 4.0007343233 minutes. Last failure message: false was not true. (KubernetesSuite.scala:315)

[dongjoon-hyun]

+1, LGTM. Merged to master.

I verified this locally. The flaky failure is irrelevant to this PR.

KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark with Python2 to test a pyfiles example
- Run PySpark with Python3 to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- Run SparkR on simple dataframe.R example
Run completed in 13 minutes, 30 seconds.
Total number of tests run: 19
Suites: completed 2, aborted 0
Tests: succeeded 19, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

[dongjoon-hyun]

Thank you, @onursatici .

[dongjoon-hyun]

I'll backport this to branch-3.0 to address the recent JDK8 issue on K8s 1.17.x.