[SPARK-3243] Don't use stale spark-driver.* system properties

[andrewor14]

If we set both spark.driver.extraClassPath and --driver-class-path, then the latter correctly overrides the former. However, the value of the system property spark.driver.extraClassPath still uses the former, which is actually not added to the class path. This may cause some confusion...

Of course, this also affects other options (i.e. java options, library path, memory...).

[SparkQA]

QA tests have started for PR 2154 at commit 0140836.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2154 at commit 0140836.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • "$FWDIR"/bin/spark-submit --class $CLASS "$
  • class ExternalSorter(object):
  • "$FWDIR"/bin/spark-submit --class $CLASS "$
  • protected class AttributeEquals(val a: Attribute)

[andrewor14]

test this please

[SparkQA]

QA tests have started for PR 2154 at commit 17ec6fc.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2154 at commit 17ec6fc.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

LGTM. Was there any side-effect to this, since it only affects client mode? (I'm guessing maybe reporting in the UI, but that's about it?)

[andrewor14]

Reporting in the UI, and if the user looks up sys.props("spark.driver.classpath") to check her class path she will get a value that is not actually used.

[pwendell]

Thanks - pulling this in.