[SPARK-12652][PySpark]Upgrade Py4J to 0.9.1 #10692

zsxwing · 2016-01-11T06:08:30Z

Upgrade Py4J to 0.9.1
SPARK-12657: Revert SPARK-12617
SPARK-12658: Revert SPARK-12511
- Still keep the change that only reading checkpoint once. This is a manual change and worth to take a look carefully. zsxwing@bfd4b5c
Verify no leak any more after reverting our workarounds

JoshRosen · 2016-01-11T08:50:27Z

Jenkins, retest this please.

SparkQA · 2016-01-11T10:33:06Z

Test build #49095 has finished for PR 10692 at commit a3e3e17.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-11T13:16:05Z

Test build #49116 has finished for PR 10692 at commit a3e3e17.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-01-11T17:44:14Z

retest this please

SparkQA · 2016-01-11T19:45:29Z

Test build #49157 has finished for PR 10692 at commit a3e3e17.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-01-11T19:49:52Z

Traceback (most recent call last):
  File "/usr/lib64/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/streaming/tests.py", line 1638, in <module>
    kafka_assembly_jar = search_kafka_assembly_jar()
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/streaming/tests.py", line 1564, in search_kafka_assembly_jar
    "remove all but one") % (", ".join(jars)))
Exception: Found multiple Spark Streaming Kafka assembly JARs: /home/jenkins/workspace/SparkPullRequestBuilder/external/kafka-assembly/target/scala-2.10/spark-streaming-kafka-assembly-2.0.0-SNAPSHOT.jar, /home/jenkins/workspace/SparkPullRequestBuilder/external/kafka-assembly/target/spark-streaming-kafka-assembly_2.10-spark-581766.jar; please remove all but one

This is pretty weird. Where did /home/jenkins/workspace/SparkPullRequestBuilder/external/kafka-assembly/target/spark-streaming-kafka-assembly_2.10-spark-581766.jar come from? @JoshRosen did you see any similar issue?

zsxwing · 2016-01-11T19:49:59Z

retest this please

JoshRosen · 2016-01-11T20:06:21Z

@zsxwing, that file is generated as part of the dummy publishing step which is used to produce POMs for use by the dev/test-dependencies checks. Ping me if it happens again and I'll look at the build logs to see if I can figure out what those aren't being cleaned up. We might have to add an additional clean step somewhere.

JoshRosen · 2016-01-11T20:07:39Z

Actually, I think I've spotted the problem. Working on a small fix now.

JoshRosen · 2016-01-11T20:21:47Z

I've opened #10704 to fix this

zsxwing · 2016-01-11T20:50:19Z

By the way, I packed py4j-0.9.1-src.zip by myself since I could not find it anywhere.

JoshRosen · 2016-01-11T21:14:36Z

Jenkins, retest this please.

JoshRosen · 2016-01-11T21:24:13Z

PY4J zip packaging and source diffs look good to me, so I think this is good-to-go pending tests.

zsxwing · 2016-01-11T21:32:03Z

PY4J zip packaging and source diffs look good to me, so I think this is good-to-go pending tests.

This is not ready for merging. I plan to revert our workarounds for Py4J issues in this PR.

…Streaming" This reverts commit 1e6648d.

This reverts commit 047a31b.

SparkQA · 2016-01-11T21:45:44Z

Test build #49164 has finished for PR 10692 at commit a3e3e17.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…nDStream.registerSerializer is called only once" This partially reverts commit 6cfe341 because the fix to make sure PySpark reads the checkpoint only once is still worth to keep

SparkQA · 2016-01-11T22:56:15Z

Test build #49170 has finished for PR 10692 at commit a3e3e17.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-12T01:11:57Z

Test build #49188 has finished for PR 10692 at commit bfd4b5c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-01-12T01:12:39Z

@JoshRosen @davies could you take a look?

davies · 2016-01-12T01:41:00Z

@zsxwing Could you also update the patch for callback server to use new API? see py4j/py4j#147

zsxwing · 2016-01-12T05:41:49Z

@zsxwing Could you also update the patch for callback server to use new API? see py4j/py4j#147

Updated

SparkQA · 2016-01-12T07:45:40Z

Test build #49215 has finished for PR 10692 at commit 3db993c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-01-12T22:14:32Z

@davies could you take a look at my latest commit?

davies · 2016-01-12T22:20:23Z

LGTM

zsxwing · 2016-01-12T22:26:46Z

Thanks, merging to master

sarathjiguru · 2016-04-26T11:04:08Z

Hello @zsxwing ,

This PR is not merged with branch-1.6. Is there a reason for it?
I am hoping latest version of py4j is needed for production ready spark streaming jobs. Please let me know if it is not the case.

Commits on Jan 13:
Master: https://github.com/apache/spark/commits/master?page=38
v1.6.1: https://github.com/apache/spark/commits/v1.6.1?page=3
branch-1.6: https://github.com/apache/spark/commits/branch-1.6?page=5

This particular commit is not merged with branch-1.6.

zsxwing · 2016-05-02T20:23:19Z

This PR is not merged with branch-1.6. Is there a reason for it?

@sarathjiguru as we have already merged workarounds to branch-1.6, it's not necessary to upgrade Py4j for 1.6.

Upgrade Py4J to 0.9.1

a3e3e17

zsxwing added 2 commits January 11, 2016 13:43

Revert "[SPARK-12617][PYSPARK] Move Py4jCallbackConnectionCleaner to …

8c968e0

…Streaming" This reverts commit 1e6648d.

Revert "[SPARK-12617] [PYSPARK] Clean up the leak sockets of Py4J"

7a64ee1

This reverts commit 047a31b.

Partially revert "[SPARK-12511] [PYSPARK] [STREAMING] Make sure Pytho…

bfd4b5c

…nDStream.registerSerializer is called only once" This partially reverts commit 6cfe341 because the fix to make sure PySpark reads the checkpoint only once is still worth to keep

zsxwing changed the title ~~[SPARK-12652][PySpark][WIP]Upgrade Py4J to 0.9.1~~ [SPARK-12652][PySpark]Upgrade Py4J to 0.9.1 Jan 11, 2016

Use resetCallbackClient to update the server port

3db993c

asfgit closed this in 4f60651 Jan 12, 2016

zsxwing deleted the py4j-0.9.1 branch January 12, 2016 22:29

[SPARK-12652][PySpark]Upgrade Py4J to 0.9.1 #10692

[SPARK-12652][PySpark]Upgrade Py4J to 0.9.1 #10692

Uh oh!

Conversation

zsxwing commented Jan 11, 2016

Uh oh!

JoshRosen commented Jan 11, 2016

Uh oh!

SparkQA commented Jan 11, 2016

Uh oh!

SparkQA commented Jan 11, 2016

Uh oh!

zsxwing commented Jan 11, 2016

Uh oh!

SparkQA commented Jan 11, 2016

Uh oh!

zsxwing commented Jan 11, 2016

Uh oh!

zsxwing commented Jan 11, 2016

Uh oh!

JoshRosen commented Jan 11, 2016

Uh oh!

JoshRosen commented Jan 11, 2016

Uh oh!

JoshRosen commented Jan 11, 2016

Uh oh!

zsxwing commented Jan 11, 2016

Uh oh!

JoshRosen commented Jan 11, 2016

Uh oh!

JoshRosen commented Jan 11, 2016

Uh oh!

zsxwing commented Jan 11, 2016

Uh oh!

SparkQA commented Jan 11, 2016

Uh oh!

SparkQA commented Jan 11, 2016

Uh oh!

SparkQA commented Jan 12, 2016

Uh oh!

zsxwing commented Jan 12, 2016

Uh oh!

davies commented Jan 12, 2016

Uh oh!

zsxwing commented Jan 12, 2016

Uh oh!

SparkQA commented Jan 12, 2016

Uh oh!

zsxwing commented Jan 12, 2016

Uh oh!

davies commented Jan 12, 2016

Uh oh!

zsxwing commented Jan 12, 2016

Uh oh!

sarathjiguru commented Apr 26, 2016

Uh oh!

zsxwing commented May 2, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants