[SPARK-25891][PYTHON] Upgrade to Py4J 0.10.8.1

[dongjoon-hyun]

What changes were proposed in this pull request?

This is a backport of #22901.

Py4J 0.10.8.1 is released on October 21st and is the first release of Py4J to support Python 3.7 officially. We had better have this to get the official support. Also, there are some patches related to garbage collections.

https://www.py4j.org/changelog.html#py4j-0-10-8-and-py4j-0-10-8-1

How was this patch tested?

Pass the Jenkins.

[dongjoon-hyun]

Since 2.4.0 passed the vote, I think we can start to discuss this. This is for Spark 2.4.1 to support Python 3.7 more officially.

@HyukjinKwon , @BryanCutler , @srowen , @cloud-fan , @gatorsmile . Could you guys give me some advice on this? And, do I need to change the JIRA issue types to a BUG since we don't allow a backport improvement?

[SparkQA]

Test build #98361 has finished for PR 22924 at commit 9ce825e.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I think it's OK to backport at 3.7 support is actually fairly important, and this also fixes bugs: https://www.py4j.org/changelog.html

It drops 2.6 support but we did so as well a long time ago in 2.2.0.

[gatorsmile]

I am against this change in 2.4 branch unless the bugs are critical for our end users.

[srowen]

Yeah I get that. I haven't heard the bugs that are fixed here impact users, but a few sound kind of bad. Maybe this should depend more on: how much do people need Python 3.7 in 2.4.x? that seems like a valid basis to back-port, if it were in demand. A 'bug fix' of a different sort, fixing incompatibilities with 3.7

[gatorsmile]

I would be more conservative in these cases. The new release of Py4J might introduce new regressions. Not all the changes are clearly documented in their release notes. It is kind of aggressive to change the dependence in our maintenance releases.

[dongjoon-hyun]

Thank you for your interests and discussion, @srowen and @gatorsmile .
Let's close for now since we have a negative opinion.

Thank you again.

[SparkQA]

Test build #98362 has finished for PR 22924 at commit 78127fd.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[fabiencelier]

Hi,
We are using python 3.7 and need py4j 0.10.8.1 in our library (0.10.7 does not work for us).
This is preventing us from using pyspark in our projects.
Are you planning to upgrade the py4j version in the 2.4 branch ?

[HyukjinKwon]

I think it's fine to backport.

@fabiencelier, as a workaround for now, you can install py4j 0.10.8.1 via pip and remove python/lib/py4j-0.10.7-src.zip. I think that will works because Python 3.7 is a Python side issue and as far as I know Java side keeps compatibility.

[dongjoon-hyun]

Hi, @fabiencelier and @HyukjinKwon .

As the original author of this PR, I'm +1 for the backporting idea.

However, as you see in the previous comments, this backporting was already considered and officially rejected by the community. (cc @gatorsmile )

For now, @HyukjinKwon 's workaround or Spark 3.0.0 will be the viable solutions.