Skip to content

Commit

Permalink
fix can not find /tmp/xxx/yyy.tar.gz (#3111)
Browse files Browse the repository at this point in the history
when spark use cluster deploy-mode,  the run_path will be created on the submitting host instead of the host where the driver is located
  • Loading branch information
Chopinxb authored Oct 1, 2021
1 parent ef9dd77 commit 0c5c53b
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions luigi/contrib/spark.py
Original file line number Diff line number Diff line change
Expand Up @@ -356,6 +356,7 @@ def _setup_packages(self, sc):
mod_path = mod.__path__[0]
except AttributeError:
mod_path = mod.__file__
os.makedirs(self.run_path, exist_ok=True)
tar_path = os.path.join(self.run_path, package + '.tar.gz')
tar = tarfile.open(tar_path, "w:gz")
tar.add(mod_path, os.path.basename(mod_path))
Expand Down

0 comments on commit 0c5c53b

Please sign in to comment.