Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: remove switching directory to load results #3047

Closed
wants to merge 1 commit into from
Closed

fix: remove switching directory to load results #3047

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 23 additions & 24 deletions nipype/utils/filemanip.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -707,34 +707,33 @@ def loadpkl(infile):
pkl_metadata = None

unpkl = None
with indirectory(infile.parent):
with SoftFileLock('%s.lock' % infile.name):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to continue changing directories while correctly locking, what if you just switched (and locked the absolute path):

-    with indirectory(infile.parent):
-        with SoftFileLock('%s.lock' % infile.name):
+    with SoftFileLock('%s.lock' % infile):
+        with indirectory(infile.parent):

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still has potential race conditions when it comes to loading the actual results file.

with pklopen(infile.name, 'rb') as pkl_file:
try: # Look if pkl file contains version file
pkl_metadata_line = pkl_file.readline()
pkl_metadata = json.loads(pkl_metadata_line)
except (UnicodeDecodeError, json.JSONDecodeError):
# Could not get version info
pkl_file.seek(0)
try:
unpkl = pickle.load(pkl_file)
except UnicodeDecodeError:
# Was this pickle created with Python 2.x?
unpkl = pickle.load(pkl_file, fix_imports=True, encoding='utf-8')
fmlogger.info('Successfully loaded pkl in compatibility mode.')
# Unpickling problems
except Exception as e:
if pkl_metadata and 'version' in pkl_metadata:
from nipype import __version__ as version
if pkl_metadata['version'] != version:
fmlogger.error("""\
with SoftFileLock('%s.lock' % infile):
with pklopen(str(infile), 'rb') as pkl_file:
try: # Look if pkl file contains version file
pkl_metadata_line = pkl_file.readline()
pkl_metadata = json.loads(pkl_metadata_line)
except (UnicodeDecodeError, json.JSONDecodeError):
# Could not get version info
pkl_file.seek(0)
try:
unpkl = pickle.load(pkl_file)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this still needs to be done in the folder where outputs required to exist are.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about the other chdir?

except UnicodeDecodeError:
# Was this pickle created with Python 2.x?
unpkl = pickle.load(pkl_file, fix_imports=True, encoding='utf-8')
fmlogger.info('Successfully loaded pkl in compatibility mode.')
# Unpickling problems
except Exception as e:
if pkl_metadata and 'version' in pkl_metadata:
from nipype import __version__ as version
if pkl_metadata['version'] != version:
fmlogger.error("""\
Attempted to open a results file generated by Nipype version %s, \
with an incompatible Nipype version (%s)""", pkl_metadata['version'], version)
raise e
fmlogger.warning("""\
raise e
fmlogger.warning("""\
No metadata was found in the pkl file. Make sure you are currently using \
the same Nipype version from the generated pkl.""")
raise e
raise e

if unpkl is None:
raise ValueError('Loading %s resulted in None.' % infile)
Expand Down