[SPARK-16399] [PYSPARK] Force PYSPARK_PYTHON to python

[MechCoder]

What changes were proposed in this pull request?

I would like to change

if hash python2.7 2>/dev/null; then
  # Attempt to use Python 2.7, if installed:
  DEFAULT_PYTHON="python2.7"
else
  DEFAULT_PYTHON="python"
fi

to just DEFAULT_PYTHON="python"

I'm not sure if it is a great assumption that python2.7 is used by default, when python points to something else.

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

[MechCoder]

ping @srowen @JoshRosen

[JoshRosen]

If I recall, the original motivation for preferring the python2.7 executable to plain python was that python might be Python 2.6 on older CentOS / RHEL systems and certain parts of PySpark performance are adversely impacted by Python 2.6. Given that we're planning to drop Python 2.6 support, this may no longer be necessary.

[MechCoder]

Thanks for clarifying! It might be a good time to get rid of it..

[srowen]

Doesn't even need to be a variable anymore right?

[MechCoder]

@srowen fixed.

[SparkQA]

Test build #61623 has finished for PR 14016 at commit 4661493.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #61632 has finished for PR 14016 at commit 01c405b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #61633 has finished for PR 14016 at commit 871281d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Is this safe to merge for 2.0.0? I don't think Python 2.6 support is officially gone yet. It seems OK for master at least, and you can of course force a version of Python if needed.

If it's only for master we should not attach this to SPARK-15761

[MechCoder]

I agree with you, I created a new JIRA and renamed the title.

[srowen]

Merged to master