[SPARK-28938][K8S] Move to supported OpenJDK docker image for Kubernetes #26037
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment has been minimized.
This comment has been minimized.
dongjoon-hyun
changed the title
|
Thank you for making a PR, @viirya . |
|
Retest this please. |
|
Hmm, I was thinking to run the integration test on Jenkins. At least I remember I saw it in previous PRs. |
|
Yes. It's weird to me, too. |
|
I checked the Jenkins. And, found that it's running here. |
This comment has been minimized.
This comment has been minimized.
|
oh, thanks. @dongjoon-hyun. Seems Jenkins also posted the comment of Kubernetes integration test too. |
|
Let's see the result~ 😄 |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Could you check the K8s IT failure? |
dongjoon-hyun
requested changes
Oct 6, 2019
|
@dongjoon-hyun Thanks. Updated Python and R Docker files. Let's see K8s IT result with latest change. If it has failure, I will change to a local environment which can run K8s IT. (current local env cannot) |
This comment has been minimized.
This comment has been minimized.
dongjoon-hyun
requested changes
Oct 6, 2019
There was a problem hiding this comment.
It seems that there are two failures.
- R (Please see
pandocerror in Jenkins log. It might be irrelevant to this one.) - Python (I hit this locally.)
ensurepip is disabled in Debian/Ubuntu for the system python.
Python modules For the system python are usually handled by dpkg and apt-get.
apt-get install python-<module name>
Install the python-pip package to use pip itself. Using pip together
with the system python might have unexpected results for any system installed
module, so use it on your own risk, or make sure to only use it in virtual
environments.
The command '/bin/sh -c apt install -y python && apt install -y python3 && python -m ensurepip && python3 -m ensurepip && rm -r /usr/lib/python*/ensurepip && pip install --upgrade pip setuptools && rm -r /root/.cache && rm -rf /var/cache/apt/*' returned a non-zero code: 1
Failed to build PySpark Docker image, please refer to Docker build output for details.
|
About R (Please see pandoc error in Jenkins log. It might be irrelevant to this one.), I think it is happened before K8s IT stuff, according to the log. There is another ongoing K8s IT build. Let's see the latest result and do necessary change if it needs. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
And yes, ensurepip is a problem. Let me update and test locally. |
|
Test build #111833 has finished for PR 26037 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
dongjoon-hyun
approved these changes
Oct 7, 2019
|
Could you make a backport PR to |
|
ok.
…On Mon, Oct 7, 2019, 08:56 Dongjoon Hyun ***@***.***> wrote:
Could you make a backport to branch-2.4, @viirya
<https://github.com/viirya> ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#26037?email_source=notifications&email_token=AAAQZ52XPIWPJ6TPYVM764LQNNL2HA5CNFSM4I55M4MKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAQ3UEA#issuecomment-539081232>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAQZ573BCRVUH3E4J3R2HTQNNL2HANCNFSM4I55M4MA>
.
|
raiju
pushed a commit
to raiju/spark
that referenced
this pull request
Oct 31, 2019
The current docker image used by Kubernetes is `openjdk:8-alpine`. It was not supported and was removed with the commit docker-library/openjdk@3eb0351#diff-f95ffa3d1377774732c33f7b8368e099. This PR proposes to move to a supported docker image. I think there are at least two reasons: 1. According to the commit, Alpine/musl is not officially supported by the OpenJDK project. 2. As no more OpenJDK 8 Alpine images, new JDK updates including security fixes , are not applied to it. See below: ``` docker run -it --rm openjdk:8-alpine java -version openjdk version "1.8.0_212" OpenJDK Runtime Environment (IcedTea 3.12.0) (Alpine 8.212.04-r0) OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode) ``` ``` docker run -it --rm openjdk:8-jdk-slim java -version openjdk version "1.8.0_222" OpenJDK Runtime Environment (build 1.8.0_222-b10) OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode) ``` Yes. This changes the base docker image of Spark. Existing tests. Closes apache#26037 from viirya/SPARK-28938. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
robert3005
pushed a commit
to palantir/spark
that referenced
this pull request
Nov 7, 2019
…tes (#616) [SPARK-28938][K8S] Move to supported OpenJDK docker image for Kubernetes The current docker image used by Kubernetes is `openjdk:8-alpine`. It was not supported and was removed with the commit docker-library/openjdk@3eb0351#diff-f95ffa3d1377774732c33f7b8368e099. This PR proposes to move to a supported docker image. I think there are at least two reasons: 1. According to the commit, Alpine/musl is not officially supported by the OpenJDK project. 2. As no more OpenJDK 8 Alpine images, new JDK updates including security fixes , are not applied to it. See below: ``` docker run -it --rm openjdk:8-alpine java -version openjdk version "1.8.0_212" OpenJDK Runtime Environment (IcedTea 3.12.0) (Alpine 8.212.04-r0) OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode) ``` ``` docker run -it --rm openjdk:8-jdk-slim java -version openjdk version "1.8.0_222" OpenJDK Runtime Environment (build 1.8.0_222-b10) OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode) ``` Yes. This changes the base docker image of Spark. Existing tests. Closes apache#26037 from viirya/SPARK-28938. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The current docker image used by Kubernetes is
openjdk:8-alpine. It was not supported and was removed with the commit docker-library/openjdk@3eb0351#diff-f95ffa3d1377774732c33f7b8368e099.This PR proposes to move to a supported docker image.
Why are the changes needed?
I think there are at least two reasons:
, are not applied to it. See below:
Does this PR introduce any user-facing change?
Yes. This changes the base docker image of Spark.
How was this patch tested?
Existing tests.