Skip to content

[SPARK-6216] [PySpark] check python version of worker with driver #6203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
davies wants to merge 3 commits into from
Closed

[SPARK-6216] [PySpark] check python version of worker with driver #6203

davies wants to merge 3 commits into from

Conversation

@davies
Copy link
Contributor

@davies davies commented May 15, 2015

This PR revert #5404, change to pass the version of python in driver into JVM, check it in worker before deserializing closure, then it can works with different major version of Python.

@davies
Copy link
Contributor Author

davies commented May 15, 2015

cc @JoshRosen

@AmplabJenkins
Copy link

AmplabJenkins commented May 15, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 15, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 15, 2015

Test build #32859 has started for PR 6203 at commit 47c6278.

vanzin
vanzin reviewed May 15, 2015
View reviewed changes
python/pyspark/context.py Outdated
Copy link
Contributor

@vanzin vanzin May 15, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not a huge deal, but it might be better to pass major and minor as separate parameters. You know, in case python 2.10 ever comes out. :-)

Or, at least, use a larger multiplier.

@vanzin
Copy link
Contributor

vanzin commented May 15, 2015

LGTM aside from minor suggestion.

@SparkQA
Copy link

SparkQA commented May 16, 2015

Test build #32859 has finished for PR 6203 at commit 47c6278.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32859/
Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 16, 2015

Test build #32870 has started for PR 6203 at commit 6ce5096.

@SparkQA
Copy link

SparkQA commented May 16, 2015

Test build #32870 has finished for PR 6203 at commit 6ce5096.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32870/
Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 16, 2015

Test build #32883 has started for PR 6203 at commit b8fb76e.

@SparkQA
Copy link

SparkQA commented May 16, 2015

Test build #32883 has finished for PR 6203 at commit b8fb76e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32883/
Test PASSed.

@JoshRosen
Copy link
Contributor

JoshRosen commented May 17, 2015

LGTM as well; this seems like a straightforward fix and will be useful once more people start playing around with Python 3.

@JoshRosen
Copy link
Contributor

JoshRosen commented May 18, 2015

I'm going to merge this into master and branch-1.4 (1.4.0). Thanks!

asfgit pushed a commit that referenced this pull request May 18, 2015
@JoshRosen
[SPARK-6216] [PYSPARK] check python version of worker with driver
a833209 
This PR revert #5404, change to pass the version of python in driver into JVM, check it in worker before deserializing closure, then it can works with different major version of Python.

Author: Davies Liu <davies@databricks.com>

Closes #6203 from davies/py_version and squashes the following commits:

b8fb76e [Davies Liu] fix test
6ce5096 [Davies Liu] use string for version
47c6278 [Davies Liu] check python version of worker with driver

(cherry picked from commit 32fbd29)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
@asfgit asfgit closed this in 32fbd29 May 18, 2015
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@jeanlyn
[SPARK-6216] [PYSPARK] check python version of worker with driver
6ee9228 
This PR revert apache#5404, change to pass the version of python in driver into JVM, check it in worker before deserializing closure, then it can works with different major version of Python.

Author: Davies Liu <davies@databricks.com>

Closes apache#6203 from davies/py_version and squashes the following commits:

b8fb76e [Davies Liu] fix test
6ce5096 [Davies Liu] use string for version
47c6278 [Davies Liu] check python version of worker with driver
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@jeanlyn
[SPARK-6216] [PYSPARK] check python version of worker with driver
f58e5b5 
This PR revert apache#5404, change to pass the version of python in driver into JVM, check it in worker before deserializing closure, then it can works with different major version of Python.

Author: Davies Liu <davies@databricks.com>

Closes apache#6203 from davies/py_version and squashes the following commits:

b8fb76e [Davies Liu] fix test
6ce5096 [Davies Liu] use string for version
47c6278 [Davies Liu] check python version of worker with driver
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@nemccarthy
[SPARK-6216] [PYSPARK] check python version of worker with driver
df617b9 
This PR revert apache#5404, change to pass the version of python in driver into JVM, check it in worker before deserializing closure, then it can works with different major version of Python.

Author: Davies Liu <davies@databricks.com>

Closes apache#6203 from davies/py_version and squashes the following commits:

b8fb76e [Davies Liu] fix test
6ce5096 [Davies Liu] use string for version
47c6278 [Davies Liu] check python version of worker with driver
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@davies @AmplabJenkins @SparkQA @vanzin @JoshRosen