Skip to content

Commit

Permalink
[SPARK-16077] [PYSPARK] catch the exception from pickle.whichmodule()
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?

In the case that we don't know which module a object came from, will call pickle.whichmodule() to go throught all the loaded modules to find the object, which could fail because some modules, for example, six, see https://bitbucket.org/gutworth/six/issues/63/importing-six-breaks-pickling

We should ignore the exception here, use `__main__` as the module name (it means we can't find the module).

## How was this patch tested?

Manual tested. Can't have a unit test for this.

Author: Davies Liu <davies@databricks.com>

Closes #13788 from davies/whichmodule.

(cherry picked from commit d489354)
Signed-off-by: Davies Liu <davies.liu@gmail.com>
  • Loading branch information
Davies Liu authored and davies committed Jun 24, 2016
1 parent 4fdac3c commit d7223bb
Showing 1 changed file with 12 additions and 2 deletions.
14 changes: 12 additions & 2 deletions python/pyspark/cloudpickle.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,12 @@ def save_function(self, obj, name=None):

if name is None:
name = obj.__name__
modname = pickle.whichmodule(obj, name)
try:
# whichmodule() could fail, see
# https://bitbucket.org/gutworth/six/issues/63/importing-six-breaks-pickling
modname = pickle.whichmodule(obj, name)
except Exception:
modname = None
# print('which gives %s %s %s' % (modname, obj, name))
try:
themodule = sys.modules[modname]
Expand Down Expand Up @@ -326,7 +331,12 @@ def save_global(self, obj, name=None, pack=struct.pack):

modname = getattr(obj, "__module__", None)
if modname is None:
modname = pickle.whichmodule(obj, name)
try:
# whichmodule() could fail, see
# https://bitbucket.org/gutworth/six/issues/63/importing-six-breaks-pickling
modname = pickle.whichmodule(obj, name)
except Exception:
modname = '__main__'

if modname == '__main__':
themodule = None
Expand Down

0 comments on commit d7223bb

Please sign in to comment.