fix 'can not find /tmp/xxx/yyy.tar.gz' error when use spark cluster mode

[Chopinxb]

when spark use cluster deploy-mode, the run_path will be created on the submitting host instead of the host where the driver is located. This will casuse error below:
Traceback (most recent call last): File "pyspark_runner.py", line 143, in <module> _get_runner_class()(*sys.argv[1:]).run() File "pyspark_runner.py", line 119, in run self.job.setup_remote(sc) File "/opt/tiger/ss_lib/python_package/lib/python2.7/site-packages/luigi/contrib/spark.py", line 307, in setup_remote self._setup_packages(sc) File "/opt/tiger/ss_lib/python_package/lib/python2.7/site-packages/luigi/contrib/spark.py", line 364, in _setup_packages tar = tarfile.open(tar_path, "w:gz") File "/usr/lib/python2.7/tarfile.py", line 1693, in open return func(name, filemode, fileobj, **kwargs) File "/usr/lib/python2.7/tarfile.py", line 1740, in gzopen fileobj = gzip.GzipFile(name, mode, compresslevel, fileobj) File "/usr/lib/python2.7/gzip.py", line 94, in __init__ fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb') IOError: [Errno 2] No such file or directory: '/tmp/xxxYcUXC/yyy.tar.gz'

Description

In this PR, we will create the parent directory before compresse and upload packages in the host where driver is located.
The driver is the role who will run _setup_packages func but not run func in class SparkSubmitTask