[SPARK-30216][INFRA] Use python3 in Docker release image #26848

wangyum · 2019-12-11T08:53:24Z

What changes were proposed in this pull request?

Reverts commit 1f94bf4 and d6be46e
Switches python to python3 in Docker release image.

Why are the changes needed?

dev/make-distribution.sh and python/setup.py are use python3.
https://github.com/apache/spark/pull/26844/files#diff-ba2c046d92a1d2b5b417788bfb5cb5f8L236
https://github.com/apache/spark/pull/26330/files#diff-8cf6167d58ce775a08acafcfe6f40966

Does this PR introduce any user-facing change?

No.

How was this patch tested?

manual test:

yumwang@ubuntu-3513086:~/spark$ dev/create-release/do-release-docker.sh -n -d /home/yumwang/spark-release
Output directory already exists. Overwrite and continue? [y/n] y
Branch [branch-2.4]: master
Current branch version is 3.0.0-SNAPSHOT.
Release [3.0.0]: 3.0.0-preview2
RC # [1]:
This is a dry run. Please confirm the ref that will be built for testing.
Ref [master]:
ASF user [yumwang]:
Full name [Yuming Wang]:
GPG key [yumwang@apache.org]: DBD447010C1B4F7DAD3F7DFD6E1B4122F6A3A338
================
Release details:
BRANCH:     master
VERSION:    3.0.0-preview2
TAG:        v3.0.0-preview2-rc1
NEXT:       3.0.1-SNAPSHOT

ASF USER:   yumwang
GPG KEY:    DBD447010C1B4F7DAD3F7DFD6E1B4122F6A3A338
FULL NAME:  Yuming Wang
E-MAIL:     yumwang@apache.org
================
Is this info correct [y/n]? y
GPG passphrase:

========================
= Building spark-rm image with tag latest...
Command: docker build -t spark-rm:latest --build-arg UID=110302528 /home/yumwang/spark/dev/create-release/spark-rm
Log file: docker-build.log
Building v3.0.0-preview2-rc1; output will be at /home/yumwang/spark-release/output

gpg: directory '/home/spark-rm/.gnupg' created
gpg: keybox '/home/spark-rm/.gnupg/pubring.kbx' created
gpg: /home/spark-rm/.gnupg/trustdb.gpg: trustdb created
gpg: key 6E1B4122F6A3A338: public key "Yuming Wang <yumwang@apache.org>" imported
gpg: key 6E1B4122F6A3A338: secret key imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1
========================
= Creating release tag v3.0.0-preview2-rc1...
Command: /opt/spark-rm/release-tag.sh
Log file: tag.log
It may take some time for the tag to be synchronized to github.
Press enter when you've verified that the new tag (v3.0.0-preview2-rc1) is available.
========================
= Building Spark...
Command: /opt/spark-rm/release-build.sh package
Log file: build.log
========================
= Building documentation...
Command: /opt/spark-rm/release-build.sh docs
Log file: docs.log
========================
= Publishing release
Command: /opt/spark-rm/release-build.sh publish-release
Log file: publish.log

Generated doc:

SparkQA · 2019-12-11T11:43:52Z

Test build #115167 has finished for PR 26848 at commit 3039465.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dev/create-release/spark-rm/Dockerfile

SparkQA · 2019-12-11T15:51:02Z

Test build #115188 has finished for PR 26848 at commit 9832f05.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dev/create-release/spark-rm/Dockerfile

dongjoon-hyun · 2019-12-11T16:20:20Z

dev/create-release/spark-rm/Dockerfile

+  pip3 install $BASE_PIP_PKGS && \
+  pip3 install $PIP_PKGS && \
  cd && \
  virtualenv -p python3 /opt/p35 && \


Do we need to use virtualenv when we have a system python3?

I will verify it later.

srowen

LGTM if it works!

python/setup.py

HyukjinKwon · 2019-12-12T01:54:34Z

dev/make-distribution.sh

  # Delete the egg info file if it exists, this can cache older setup files.
  rm -rf pyspark.egg-info || echo "No existing egg info file, skipping deletion"
-  python3 setup.py sdist
+  python setup.py sdist


Can we keep python3 by default? I think the compatibility change was intentionally removed as of 04e99c1#diff-e77989ed21758e78331b20e477fc5582 to prepare the complete removal of Python 2 support.

If so, we also do not need to add from __future__ import print_function in python/setup.py because it is always called by python3?

Yeah, I think so.

+1 for @HyukjinKwon 's advice.

update-alternatives: error: alternative path /usr/bin/python2.7 doesn't exist

HyukjinKwon

Seems fine

wangyum · 2019-12-12T02:26:14Z

dev/create-release/spark-rm/Dockerfile

+  pip3 install $PIP_PKGS && \
  cd && \
-  virtualenv -p python3 /opt/p35 && \
+  python /opt/p35 && \


No such file or directory when building docker image:

Successfully built pypandoc pycparser Installing collected packages: six, pycparser, cffi, cryptography, pyopenssl, pypandoc, numpy, Pygments, sphinxcontrib-applehelp, pytz, babel, snowballstemmer, imagesize, urllib3, certifi, chardet, idna, requests, sphinxcontrib-serializinghtml, sphinxcontrib-devhelp, sphinxcontrib-htmlhelp, alabaster, sphinxcontrib-jsmath, sphinxcontrib-qthelp, pyparsing, packaging, MarkupSafe, Jinja2, docutils, sphinx Successfully installed Jinja2-2.10.3 MarkupSafe-1.1.1 Pygments-2.5.2 alabaster-0.7.12 babel-2.7.0 certifi-2019.11.28 cffi-1.13.2 chardet-3.0.4 cryptography-2.8 docutils-0.15.2 idna-2.8 imagesize-1.1.0 numpy-1.17.4 packaging-19.2 pycparser-2.19 pyopenssl-19.1.0 pypandoc-1.4 pyparsing-2.4.5 pytz-2019.3 requests-2.22.0 six-1.13.0 snowballstemmer-2.0.0 sphinx-2.2.2 sphinxcontrib-applehelp-1.0.1 sphinxcontrib-devhelp-1.0.1 sphinxcontrib-htmlhelp-1.0.2 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.2 sphinxcontrib-serializinghtml-1.1.3 urllib3-1.25.7 python: can't open file '/opt/p35': [Errno 2] No such file or directory

@cloud-fan Can we remove virtualenv?

I think you can remove python /opt/p35 && /opt/p35/bin/activate.

That seems in order to activate virtualenv.

Can we remove this?

spark/dev/create-release/do-release-docker.sh

Lines 139 to 140 in 8d1b5ba

# SPARK-24530: Sphinx must work with python 3 to generate doc correctly.

echo "SPHINXPYTHON=/opt/p35/bin/python" >> $ENVFILE

Thank you @HyukjinKwon . We need to do it:

= Building documentation... Command: /opt/spark-rm/release-build.sh docs Log file: docs.log Command FAILED. Check full logs for details. Copying jquery.min.js from Scala API to Java API for page post-processing of badges Copying api_javadocs.js to Java API for page post-processing of badges Appending content of api-javadocs.css to JavaDoc stylesheet.css for badge styles Moving to python/docs directory and building sphinx. make: /opt/p35/bin/python: Command not found /opt/p35/bin/python -msphinx -b html -d _build/doctrees . _build/html make: /opt/p35/bin/python: Command not found Makefile:80: recipe for target 'html' failed make: *** [html] Error 127 jekyll 3.8.6 | Error: Python doc generation failed

make: /opt/p35/bin/python: Command not found

Can we try to set SPHINXPYTHON=python and see if it works? Seems it's just because python was not found.

I reverted commit 1f94bf4 and d6be46e, I'd like to confirm is it correct?

Yeah. Seems right to revert two commits (and push both reverts into this PR).

cc @cloud-fan.

SparkQA · 2019-12-12T05:34:25Z

Test build #115206 has finished for PR 26848 at commit 02896e3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-12T05:40:12Z

Test build #115204 has finished for PR 26848 at commit 98cd80a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-12T05:52:47Z

Test build #115209 has finished for PR 26848 at commit 1f2ede8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-12-12T09:45:49Z

retest this please

SparkQA · 2019-12-12T12:08:23Z

Test build #115228 has finished for PR 26848 at commit 71b642d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-12-12T12:11:34Z

retest this please

SparkQA · 2019-12-12T14:28:49Z

Test build #115237 has finished for PR 26848 at commit 71b642d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-12-12T14:31:55Z

retest this please

SparkQA · 2019-12-12T16:55:05Z

Test build #115242 has finished for PR 26848 at commit 71b642d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon

Looks fine if it worked - I haven't tested it by myself.

wangyum · 2019-12-13T06:12:47Z

I tested it on the master branch. Do we need to test on branch-2.4?

dongjoon-hyun · 2019-12-13T19:28:04Z

No~ This is for 3.0. branch-2.4 release script is for 2.4.

I tested it on the master branch. Do we need to test on branch-2.4?

dongjoon-hyun

+1, LGTM. Merged to master.
Thank you so much, @wangyum , @srowen and @HyukjinKwon .
cc @gatorsmile and @cloud-fan

wangyum added 2 commits December 11, 2019 16:46

Use python3 in Docker release image

3039465

Change default python version to python3.

9832f05

wangyum commented Dec 11, 2019

View reviewed changes

dev/create-release/spark-rm/Dockerfile Show resolved Hide resolved

wangyum changed the title ~~[WIP][SPARK-30216][INFRA] Use python3 in Docker release image~~ [SPARK-30216][INFRA] Use python3 in Docker release image Dec 11, 2019

dongjoon-hyun reviewed Dec 11, 2019

View reviewed changes

dev/create-release/spark-rm/Dockerfile Show resolved Hide resolved

dongjoon-hyun reviewed Dec 11, 2019

View reviewed changes

dongjoon-hyun added the PROJECT INFRA label Dec 11, 2019

srowen reviewed Dec 11, 2019

View reviewed changes

Verify remove virtualenv

98cd80a

HyukjinKwon reviewed Dec 12, 2019

View reviewed changes

python/setup.py Outdated Show resolved Hide resolved

HyukjinKwon reviewed Dec 12, 2019

View reviewed changes

Fix:

02896e3

update-alternatives: error: alternative path /usr/bin/python2.7 doesn't exist

HyukjinKwon approved these changes Dec 12, 2019

View reviewed changes

wangyum commented Dec 12, 2019

View reviewed changes

Address comment

1f2ede8

This reverts commit 1f94bf4 and d6be46e

71b642d

HyukjinKwon approved these changes Dec 13, 2019

View reviewed changes

dongjoon-hyun approved these changes Dec 13, 2019

View reviewed changes

dongjoon-hyun closed this in e1ee3fb Dec 13, 2019

wangyum deleted the SPARK-30216 branch December 18, 2019 07:32

	# SPARK-24530: Sphinx must work with python 3 to generate doc correctly.
	echo "SPHINXPYTHON=/opt/p35/bin/python" >> $ENVFILE

[SPARK-30216][INFRA] Use python3 in Docker release image #26848

[SPARK-30216][INFRA] Use python3 in Docker release image #26848

Uh oh!

Conversation

wangyum commented Dec 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Dec 11, 2019

Uh oh!

Uh oh!

SparkQA commented Dec 11, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Dec 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangyum Dec 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 12, 2019

Uh oh!

SparkQA commented Dec 12, 2019

Uh oh!

SparkQA commented Dec 12, 2019

Uh oh!

wangyum commented Dec 12, 2019

Uh oh!

SparkQA commented Dec 12, 2019

Uh oh!

HyukjinKwon commented Dec 12, 2019

Uh oh!

SparkQA commented Dec 12, 2019

Uh oh!

wangyum commented Dec 12, 2019

Uh oh!

SparkQA commented Dec 12, 2019

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

wangyum commented Dec 11, 2019 •

edited

Loading

HyukjinKwon Dec 12, 2019 •

edited

Loading

wangyum Dec 12, 2019 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading