[SPARK-17336][PYSPARK] Fix appending multiple times to PYTHONPATH from spark-config.sh

[BryanCutler]

What changes were proposed in this pull request?

During startup of Spark standalone, the script file spark-config.sh appends to the PYTHONPATH and can be sourced many times, causing duplicates in the path. This change adds a env flag that is set when the PYTHONPATH is appended so it will happen only one time.

How was this patch tested?

Manually started standalone master/worker and verified PYTHONPATH has no duplicate entries.

[BryanCutler]

@srowen mind taking a look? This seems to do the trick for me. Thanks!

[srowen]

Seems reasonable, LGTM

[SparkQA]

Test build #65158 has finished for PR 15028 at commit 180a17f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

My only concern is if someone has two different versions of Spark is this seems like it might result in weird behaviour if they want to launch jobs against seperate versions in the same shell (which might not seem likely - but think about doing something like performance testing between Spark versions).

[srowen]

I think the current behavior might be worse on that dimension ... you might get several different versions of things at once on the classpath, not just redundant copies.

[srowen]

I'll go ahead with this because I don't think it makes anything any worse.

[srowen]

Merged to master/2.0

[holdenk]

Since the search order is defined the old behavior probably worked cross versions (albeit in an ugly fashion) - I'll follow up with some checks for spark-perf and fix there if necessary since I think that's really the main place which might have been dependent on this behavior.