Error during brain masking

[walkerped]

I have been running fmriprep in two datasets. It was working fine, until early last week, when all subjects began failing. I have even tried rerunning participants whose data has previously been run successfully, and they have also failed. The error I am getting is:

Node Name: fmriprep_wf.single_subject_MOATMS04_wf.anat_preproc_wf.brain_extraction_wf.atropos_wf.dil_brainmask

File: /autofs/cluster/neuromod/NIFTI/MOATMS/derivatives/fmriprep/sub-MOATMS04/log/20210622-065448_73dba296-1cb8-4d50-8cd6-0addd5f5efec/crash-20210622-104208-wsp6-dil_brainmask-12ccbe41-4b0c-4b79-8c64-b555015c3a2c.txt
Working Directory: /scratch/wsp/fmriprep_wf/single_subject_MOATMS04_wf/anat_preproc_wf/brain_extraction_wf/atropos_wf/dil_brainmask
Inputs:

    args:
    copy_header: True
    dimension: 3
    environ: {'NSLOTS': '1'}
    num_threads: 1
    op1:
    op2: 2
    operation: MD
    output_image:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 814, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 745, in raise_exception
    ).format(**runtime.dictcopy())
RuntimeError: Command:
ImageMath 3 tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled_maths.nii.gz MD /scratch/wsp/fmriprep_wf/single_subject_MOATMS04_wf/anat_preproc_wf/brain_extraction_wf/thr_brainmask/tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled.nii.gz 2
Standard output:

Standard error:
Illegal instruction (core dumped)
Return code: 132

When I run the ImageMath command by hand, I get the same error. However, when I run it with "GD" in the place of "MD", it completes successfully, with an appropriately dilated tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled_maths.nii.gz file as output. I can open OASIS30ANTs_res-01_label-brain_probseg_trans_resampled.nii.gz, and it does appear to be a binary mask.

I have also checked that the working directory has permissions and space, as well as my cache.

I'm running fmriprep-20.2.1 using singularity on a computing cluster running CentOS Linux 7 (Core). The data set has some BIDS warnings, because of some extra files in the folders that past generations of researchers put there, but otherwise are BIDS valid.

The command I am using to run it is:

    singularity run --cleanenv \
        -B ${BIDS_DIR}:${BIDS_DIR},${derivatives_DIR}:${derivatives_DIR},/scratch/wsp:/scratch/wsp,${fsLicense}:${fsLicense},${filterFile}:${filterFile},${cacheNew}:${cacheNew},${cacheSym
}:${cacheSym},${cache}:${cache} \
        $fmriprep \
        $BIDS_DIR $derivatives_DIR participant \
        --participant-label sub-${subject} \
        --output-spaces fsnative fsaverage T1w MNI152NLin2009cAsym \
        --longitudinal \
        --work-dir /scratch/wsp \
        --cifti-output \
        --bids-filter-file ${filterFile} \
        --dummy-scans 3 \
        --fs-license-file ${fsLicense} \
        --mem_mb 16000 \
        --skip_bids_validation

I found that someone else reported a similar issue at https://neurostars.org/t/fmriprep-20-2-0-error-in-anat-preproc-wf-brain-extraction-wf-atropos-wf-dil-brainmask/19453/16 .

[walkerped]

I've found that I can successfully run subjects directly on the linux workstations in the lab, instead of the cluster, with this same command.

[effigies]

Sorry, think I started responding yesterday. I suspect that somehow you have an ANTs that is being called outside the container. Can you singularity shell in and check the location and version of ImageMath?

[walkerped]

The location of ImageMath is:
/usr/lib/ants/ImageMath

I'm not sure how to check the version of ImageMath specifically. If I do "antsRegistration --version" I get:
ANTs Version: 2.3.3.dev168-g29bdf
Compiled: Jun 9 2020 03:44:55

I'm getting the same output for this whether on our cluster or on our linux workstation. But is there a command to get the version of ImageMath?

[effigies]

Hmm. That should be right. If you fetch the input files and just run ImageMath with them, can you reproduce the issue (inside the singularity environment or outside)?

[walkerped]

Both in and out of the singularity shell, I can replicate the error with the command in the fmriprep output:

ImageMath 3 tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled_maths.nii.gz MD /scratch/wsp/fmriprep_wf/single_subject_MOATMS36_wf/anat_preproc_wf/brain_extraction_wf/thr_brainmask/tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled.nii.gz 2

Illegal instruction (core dumped)

But in both cases, the command completes successfully if I change the MD option to GD:

ImageMath 3 tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled_maths.nii.gz GD /scratch/wsp/fmriprep_wf/single_subject_MOATMS36_wf/anat_preproc_wf/brain_extraction_wf/thr_brainmask/tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled.nii.gz 2

[effigies]

The problem is that we're following https://github.com/ANTsX/ANTs/blob/master/Scripts/antsBrainExtraction.sh for this, and they use MD. I don't fully understand the distinction between these modes, so I'd be hesitant to switch based on "it doesn't crash". Given that you have a reproduction, is there any chance you could take this to the ANTs GitHub or mailing list and see what they can figure out? They'll be in a much better position to debug.

[walkerped]

To wrap this up, the ANTs developers (see link above) determined that the processors I was running fmriprep on were too old to be compatible with the particular version/compilation of ANTs. I have moved over to a newer cluster, and it seems to be working.

Thanks!

[effigies]

Sounds good. Conversation moved to #2437.