Skip to content

File(exists=True) in output specs can cause failures after unused outputs are cleaned up #3493

Open
@effigies

Description

@effigies

Summary

If an output_spec contains a trait File(exists=True), and that output field is not passed as an input to another node, then the file will get cleaned up by remove_unnecessary_outputs but still be set in the traits. When re-loading a node working directory, this then causes a trait failure:

         ***********************************                                                                                                 
220718-18:01:54,394 nipype.workflow ERROR:                                                                                                                                                                                                                                                
         could not run node: nibabies_21_1_wf.single_subject_470437_1mo_wf.func_preproc_ses_1mo_task_rest_acq_AP_run_02_wf.subcortical_mni_alignment_wf.applyxfm_roi                                                  
220718-18:01:54,394 nipype.workflow INFO:                                                                                                    
         crashfile: /out/sub-470437/ses-1mo/log/20220718-172738_8d289d13-1f67-4ead-93a7-d590eb97a25f/crash-20220718-175440-root-_applyxfm_roi18-9b41d34f-48ea-4453-b226-60f0625e806e.txt
220718-18:01:54,394 nipype.workflow ERROR:                                                                                                   
         could not run node: nibabies_21_1_wf.single_subject_470437_1mo_wf.func_preproc_ses_1mo_task_rest_acq_PA_run_04_wf.subcortical_mni_alignment_wf.applyxfm_roi
220718-18:01:54,394 nipype.workflow INFO:                                                                                                    
         crashfile: /out/sub-470437/ses-1mo/log/20220718-172738_8d289d13-1f67-4ead-93a7-d590eb97a25f/crash-20220718-175440-root-_applyxfm_roi
18-5ef037cc-028f-4499-bfc1-7850b5900e63.txt                                                                                                                                                                                                                                               
220718-18:01:54,394 nipype.workflow ERROR:                                                                                                                                                                                                                                                
         could not run node: nibabies_21_1_wf.single_subject_470437_1mo_wf.func_preproc_ses_1mo_task_rest_acq_PA_run_05_wf.subcortical_mni_al                                                                                                                                             
ignment_wf.applyxfm_roi                                                                                                                                                                                                                                                                   
220718-18:01:54,394 nipype.workflow INFO:                                                                                                    
         crashfile: /out/sub-470437/ses-1mo/log/20220718-172738_8d289d13-1f67-4ead-93a7-d590eb97a25f/crash-20220718-175440-root-_applyxfm_roi0-5ec94b31-2de2-400e-872c-dbefe686a5a3.txt
220718-18:01:54,394 nipype.workflow ERROR:                                                                                                   
         could not run node: nibabies_21_1_wf.single_subject_470437_1mo_wf.func_preproc_ses_1mo_task_rest_acq_PA_run_04_wf.subcortical_mni_alignment_wf.applyxfm_roi
220718-18:01:54,394 nipype.workflow INFO:                                                                                                    
         crashfile: /out/sub-470437/ses-1mo/log/20220718-172738_8d289d13-1f67-4ead-93a7-d590eb97a25f/crash-20220718-175452-root-_applyxfm_roi17-4ba136ca-5ae1-4f59-8a8c-490f6a9b6344.txt
220718-18:01:54,394 nipype.workflow INFO:                                                                                                    
         ***********************************                                                                                                                                                                                                                                              
220718-18:01:54,472 nipype.workflow CRITICAL:                                                                                                
         nibabies failed: 4 raised. Re-raising first.                                                                                        
RuntimeError: Traceback (most recent call last):                                                                                             
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node                                   
    result["result"] = node.run(updatehash=updatehash)                                                                                       
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run                                            
    result = self._run_interface(execute=True)                                                                                               
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface                                 
    return self._run_command(execute)                                                                                                        
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command                                   
    raise NodeExecutionError(                                                                                                                
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _applyxfm_roi18.                                      
                                                                                                                                             
Traceback (most recent call last):                                                                                                           
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs                               
    setattr(outputs, key, val)                                                                                                               
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate                            
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)                                                            
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate                            
    self.error(objekt, name, str(value))                                                                                                     
  File "/opt/conda/lib/python3.9/site-packages/traits/base_trait_handler.py", line 74, in error                                              
    raise TraitError(                                                                                                                        
traits.trait_errors.TraitError: The 'out_matrix_file' trait of a FLIRTOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/scratch/nibabies_21_1_wf/single_subject_470437_1mo_wf/func_preproc_ses_1mo_task_rest_acq_AP_run_02_wf/subcor
tical_mni_alignment_wf/applyxfm_roi/mapflow/_applyxfm_roi18/vol0000_xform-00000_merged_flirt.mat' <class 'str'> was specified.                                                                                                                                                            
                                                                                                                                             
During handling of the above exception, another exception occurred:                                                                          
                                                                                                                                             
Traceback (most recent call last):                                                                                                           
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 401, in run                                                                                                                                                                                          
    outputs = self.aggregate_outputs(runtime)                                                                                                
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/fsl/preprocess.py", line 742, in aggregate_outputs                          
    outputs = super(FLIRT, self).aggregate_outputs(                                                                                          
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs                               
    raise FileNotFoundError(msg)                                                                                                             
FileNotFoundError: No such file or directory '/scratch/nibabies_21_1_wf/single_subject_470437_1mo_wf/func_preproc_ses_1mo_task_rest_acq_AP_run_02_wf/subcortical_mni_alignment_wf/applyxfm_roi/mapflow/_applyxfm_roi18/vol0000_xform-00000_merged_flirt.mat' for output 'out_matrix_file' 
of a ApplyXFM interface

I can see a few solutions:

  1. Purge exists=True from output_specs.
  2. Monkey-patch File(exists=True) in output specs to be simply File()
  3. Catch the error and replace the str value with Undefined
  4. If a file is going to be removed by remove_unnecessary_outputs, then set its value to Undefined before saving it.

Platform details:

{'commit_hash': '%h',
 'commit_source': 'archive substitution',
 'networkx_version': '2.8.2',
 'nibabel_version': '3.2.2',
 'nipype_version': '1.8.1',
 'numpy_version': '1.22.4',
 'pkg_path': '/opt/conda/lib/python3.9/site-packages/nipype',
 'scipy_version': '1.8.1',
 'sys_executable': '/opt/conda/bin/python',
 'sys_platform': 'linux',
 'sys_version': '3.9.12 | packaged by conda-forge | (main, Mar 24 2022, '
                '23:25:59) \n'
                '[GCC 10.3.0]',
 'traits_version': '6.3.2'}

Execution environment

Choose one

  • Container: nipreps/nibabies:22.1.0rc1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions