[SPARK-25079][PYTHON] preparing for python 3.5 bump #22061
Conversation
| @@ -162,7 +162,7 @@ def run_individual_python_test(target_dir, test_name, pyspark_python): | |||
|
|
|||
|
|
|||
| def get_default_python_executables(): | |||
| python_execs = [x for x in ["python2.7", "python3.4", "pypy"] if which(x)] | |||
| python_execs = [x for x in ["python2.7", "python3.4", "python3.5", "pypy"] if which(x)] | |||
There was a problem hiding this comment.
i will push another commit at some point in the future to remove the reference to python3.4.
There was a problem hiding this comment.
Just wondering why not replace python3.4 now? From the steps in jira, there will be a symlink for 3.4 -> 3.5 so it would be running the same tests twice right?
|
Test build #94515 has finished for PR 22061 at commit
|
There was a problem hiding this comment.
Change itself LGTM. Will push this in when it's ready and if my hand is needed to get this in. Please let me know later @shaneknapp.
|
this probably won't be able to get merged for a while.
…On Thu, Aug 9, 2018 at 7:01 PM, Hyukjin Kwon ***@***.***> wrote:
***@***.**** approved this pull request.
Change itself LGTM. Will push this in when it's ready and if my hand is
needed to get this in. Please let me know later @shaneknapp
<https://github.com/shaneknapp>.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#22061 (review)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABiDrITC9weSTWgqbdTiAdwBlee0L5hoks5uPOlvgaJpZM4V2HgI>
.
|
| @@ -162,7 +162,7 @@ def run_individual_python_test(target_dir, test_name, pyspark_python): | |||
|
|
|||
|
|
|||
| def get_default_python_executables(): | |||
| python_execs = [x for x in ["python2.7", "python3.4", "pypy"] if which(x)] | |||
| python_execs = [x for x in ["python2.7", "python3.4", "python3.5", "pypy"] if which(x)] | |||
There was a problem hiding this comment.
Just wondering why not replace python3.4 now? From the steps in jira, there will be a symlink for 3.4 -> 3.5 so it would be running the same tests twice right?
|
@BryanCutler the reason i'm not doing the upgrade is that the path for the current py3.4 anaconda env is hard-coded in to the python test running code across all branches. this means if i do the upgrade to 3.5 on the workers and create the symlink, all branches will be running 3.5 w/only a small amount of testing. also, i'll need to ensure that there are no package/dependency issues w/previously installed libs. if i create a new anaconda 3.5 env, then i will need to immediately change the path in dev/run-tests.py for whatever branch(es) that we want 3.5... as well as hack on the conda package definitions so that i can ansibileize things. it's a bit more work that i think you might expect... i'll be working on getting this sorted this week. |
|
also, i'll be taking the ubuntu-testing worker offline today to wipe and do a fresh install of conda w/python3.5. |
|
Ok, make sense. Thanks @shaneknapp ! |
|
i launched a PRB build against a freshly-installed python 3.5 environment: i'm sure we'll have some things to fix, etc when it finishes. |
|
another test triggered. forgot to actually test against 3.5... :( |
|
@shaneknapp should this go in now? replacing 3.4? |
|
no, not yet. i'll give a bigger update later today on the status of this,
but the TL;DR is that it's a bigger project than i expected. :\
…On Mon, Sep 24, 2018 at 9:18 AM, Sean Owen ***@***.***> wrote:
@shaneknapp <https://github.com/shaneknapp> should this go in now?
replacing 3.4?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#22061 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ABiDrI0nMUTlFHKimrukHjoO-NM5kiAEks5ueQXXgaJpZM4V2HgI>
.
|
|
closing due to #24266 |
What changes were proposed in this pull request?
preparing to bump python 3.4 to 3.5
How was this patch tested?
tested against a python3.5 installation on my ubuntu testing node, and everything passes:
https://issues.apache.org/jira/browse/SPARK-25079
this will need to be orchestrated carefully... details of steps needed to be taken are in the jira issue.
DO NOT MERGE