antsBrainExtraction.sh failing on ImageMath

[walkerped]

Describe the problem

I am running fmriprep, which uses antsBrainExtraction.sh for brain extraction. It is crashing on this step, specifically on the following command:

ImageMath 3 tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled_maths.nii.gz MD tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled.nii.gz 2

I get the following error:

Illegal instruction (core dumped)

I noticed while troubleshooting, if I run GD instead, it finished successfully, ie:

ImageMath 3 tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled_maths.nii.gz GD tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled.nii.gz 2

see nipreps/fmriprep#2433 for more context.

To Reproduce
Run

ImageMath 3 tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled_maths.nii.gz MD tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled.nii.gz 2

on:

tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled.nii.gz

System information (please complete the following information)
Many issues are specific to a particular system. Please include all information
about your computing environment.

• OS: CentOS Linux 7 (Core)
• OS version: 7
• Type of system: HPC cluster

Running via fmriprep in a singularity container, with the OS of: Ubuntu 16.04.6 LTS

ANTs version information

• ANTs code version: 2.3.3.dev168-g29bdf
• ANTs installation type: Within the fmriprep singularity container

[cookpa]

I'm thinking you have either another version of ANTs, or a shared library, getting mixed up with the container versions.

To check there's not another ANTs about, you can try

which -a ImageMath

To look at the libraries, I'd check $LD_LIBRARY_PATH and also

  ldd `which ImageMath`

and ensure the container is not mounting a conflicting version.

I've used ImageMath's MD function successfully on your image on three systems (CentOS, Ubuntu, Mac OS). I haven't tested your exact version of ANTs but will try that next. I suspect though it's something to do with the container

[walkerped]

Here is what I am getting for those, from one of the cluster's compute nodes:

singularity shell -B /scratch/wsp:/scratch/wsp fmriprep-20.2.1.simg
which -a ImageMath

    /usr/lib/ants/ImageMath

ldd which ImageMath

    linux-vdso.so.1 =>  (0x00007ffc571e1000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd0fd8f4000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd0fd6f0000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd0fd4d3000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd0fd1ca000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd0fce48000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd0fcc32000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd0fc868000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fd0fdafc000)

echo $LD_LIBRARY_PATH

/usr/lib/fsl/5.0::/.singularity.d/libs

[cookpa]

I just tried it in fmriprep, on CentOS 7:

$ singularity inspect fmriprep-20.2.1.sif
WARNING: No SIF metadata partition, searching in container...
org.label-schema.build-date: Thursday_11_February_2021_18:58:18_UTC
org.label-schema.schema-version: 1.0
org.label-schema.usage.singularity.deffile.bootstrap: docker
org.label-schema.usage.singularity.deffile.from: nipreps/fmriprep:20.2.1
org.label-schema.usage.singularity.version: 3.6.4`

$ singularity shell fmriprep-20.2.1.sif
Singularity> antsRegistration --version
ANTs Version: 2.3.3.dev168-g29bdf
Compiled: Jun  9 2020 03:44:55

Singularity> ImageMath 3 dilated.nii.gz MD tpl-OASIS30ANTs_res-01_label-brain_probseg_trans_resampled.nii.gz 2

Since it looks like we're running the same code, my next guess is that the ANTs was compiled using compiler options that are not supported on your system.

[gdevenyi]

@walkerped 'cat /proc/cpuinfo'

[walkerped]

cat /proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5472  @ 3.00GHz
stepping        : 6
microcode       : 0x60f
cpu MHz         : 3000.052
cache size      : 6144 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf eagerfpu pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority dtherm
bogomips        : 6000.10
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5472  @ 3.00GHz
stepping        : 6
microcode       : 0x60f
cpu MHz         : 3000.052
cache size      : 6144 KB
physical id     : 1
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 4
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf eagerfpu pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority dtherm
bogomips        : 6000.11
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5472  @ 3.00GHz
stepping        : 6
microcode       : 0x60f
cpu MHz         : 3000.052
cache size      : 6144 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 4
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf eagerfpu pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority dtherm
bogomips        : 6000.10
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5472  @ 3.00GHz
stepping        : 6
microcode       : 0x60f
cpu MHz         : 3000.052
cache size      : 6144 KB
physical id     : 1
siblings        : 4
core id         : 1
cpu cores       : 4
apicid          : 5
initial apicid  : 5
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf eagerfpu pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority dtherm
bogomips        : 6000.11
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 4
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5472  @ 3.00GHz
stepping        : 6
microcode       : 0x60f
cpu MHz         : 3000.052
cache size      : 6144 KB
physical id     : 0
siblings        : 4
core id         : 2
cpu cores       : 4
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf eagerfpu pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority dtherm
bogomips        : 6000.10
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 5
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5472  @ 3.00GHz
stepping        : 6
microcode       : 0x60f
cpu MHz         : 3000.052
cache size      : 6144 KB
physical id     : 1
siblings        : 4
core id         : 2
cpu cores       : 4
apicid          : 6
initial apicid  : 6
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf eagerfpu pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority dtherm
bogomips        : 6000.11
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 6
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5472  @ 3.00GHz
stepping        : 6
microcode       : 0x60f
cpu MHz         : 3000.052
cache size      : 6144 KB
physical id     : 0
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf eagerfpu pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority dtherm
bogomips        : 6000.10
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5472  @ 3.00GHz
stepping        : 6
microcode       : 0x60f
cpu MHz         : 3000.052
cache size      : 6144 KB
physical id     : 1
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 7
initial apicid  : 7
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf eagerfpu pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority dtherm
bogomips        : 6000.11
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

[cookpa]

Yep, I think that's the cause. I think the container is not fully binary compatible with this older CPU. ANTs will have to be compiled from source with different flags to work

https://github.com/ANTsX/ANTs/wiki/Compiling-ANTs-on-Linux-and-Mac-OS#compilation-fails-or-run-time-errors-occur-with-illegal-instruction-errors

[cookpa]

Looks like this has come up before, I had forgotten about it

#764

[walkerped]

Thank you!