Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DS210 CI failing intermittently in confounds workflow #2659

Closed
effigies opened this issue Dec 9, 2021 · 6 comments
Closed

DS210 CI failing intermittently in confounds workflow #2659

effigies opened this issue Dec 9, 2021 · 6 comments
Labels
bug
Milestone
21.0.x (Maintenance)

Comments

@effigies
Copy link
Member

effigies commented Dec 9, 2021

What happened?

DVARS and RenameACompCor nodes are failing. Looking into this.

What command did you use?

RUNNING: docker run --rm -e DOCKER_VERSION_8395080871=20.10.6 -it -e FMRIPREP_DEV=1 -u 1001:1002 -v /tmp/fslicense/license.txt:/opt/freesurfer/license.txt:ro -v /tmp/data/ds210:/data:ro -v /tmp/ds210/derivatives:/out -v /tmp/ds210/smriprep-0.6:/opt/smriprep/subjects -v /tmp/ds210/work:/scratch -v /tmp/ds210/nipype.cfg:/home/fmriprep/.nipype/nipype.cfg:ro nipreps/fmriprep:latest /data /out participant --output-layout legacy --me-output-echos --fs-no-reconall --use-syn-sdc --ignore slicetiming --dummy-scans 1 --sloppy --write-graph --mem_mb 4096 --nthreads 2 -vv --anat-derivatives /opt/smriprep/subjects -w /scratch --output-spaces MNI152NLin2009cAsym

What version of fMRIPrep are you running?

21.0.0rc2+27.g6b110981

How are you running fMRIPrep?

Docker

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

Anatomical derivatives

Please copy and paste any relevant log output.

211209-18:17:53,25 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_02_wf.func_preproc_task_cuedSGT_run_01_echo_1_wf.bold_confounds_wf.dvars" in "/scratch/fmriprep_wf/single_subject_02_wf/func_preproc_task_cuedSGT_run_01_echo_1_wf/bold_confounds_wf/dvars".
211209-18:17:53,35 nipype.workflow INFO:
	 [Node] Running "dvars" ("nipype.algorithms.confounds.ComputeDVARS")
211209-18:17:53,77 nipype.workflow INFO:
	 [MultiProc] Running 2 tasks, and 13 jobs ready. Free memory (GB): 3.99/4.10, Free processors: 0/2.
                     Currently running:
                       * fmriprep_wf.single_subject_02_wf.func_preproc_task_cuedSGT_run_01_echo_1_wf.bold_confounds_wf.dvars
                       * fmriprep_wf.single_subject_02_wf.func_preproc_task_cuedSGT_run_01_echo_1_wf.sdc_report
/opt/conda/lib/python3.8/site-packages/nipype/algorithms/confounds.py:1043: RuntimeWarning: divide by zero encountered in true_divide
  mfunc = (mfunc / np.median(mfunc)) * intensity_normalization
/opt/conda/lib/python3.8/site-packages/nipype/algorithms/confounds.py:1043: RuntimeWarning: invalid value encountered in true_divide
  mfunc = (mfunc / np.median(mfunc)) * intensity_normalization
/opt/conda/lib/python3.8/site-packages/nipype/algorithms/confounds.py:1048: RuntimeWarning: invalid value encountered in subtract
  np.percentile(mfunc, 75, axis=1, interpolation="lower")
/opt/conda/lib/python3.8/site-packages/nipype/algorithms/confounds.py:1229: RuntimeWarning: invalid value encountered in subtract
  regressed_data = data - datahat
211209-18:17:53,108 nipype.workflow WARNING:
	 Storing result file without outputs
211209-18:17:53,110 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_wf.single_subject_02_wf.func_preproc_task_cuedSGT_run_01_echo_1_wf.bold_confounds_wf.dvars" (/scratch/fmriprep_wf/single_subject_02_wf/func_preproc_task_cuedSGT_run_01_echo_1_wf/bold_confounds_wf/dvars)
211209-18:17:53,118 nipype.workflow ERROR:
	 Node dvars failed to run on host c9f66203887e.
211209-18:17:53,135 nipype.workflow ERROR:
	 Saving crash info to /out/fmriprep/sub-02/log/20211209-165718_cc58323e-ea47-433c-b35f-48a560af514c/crash-20211209-181753-UID1001-dvars-cfcb7a8f-3e01-4b6d-91a1-a5476784171b.txt
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 428, in run
    runtime = self._run_interface(runtime)
  File "/opt/conda/lib/python3.8/site-packages/nipype/algorithms/confounds.py", line 166, in _run_interface
    dvars = compute_dvars(
  File "/opt/conda/lib/python3.8/site-packages/nipype/algorithms/confounds.py", line 1057, in compute_dvars
    ar1 = np.apply_along_axis(
  File "<__array_function__ internals>", line 5, in apply_along_axis
  File "/opt/conda/lib/python3.8/site-packages/numpy/lib/shape_base.py", line 379, in apply_along_axis
    res = asanyarray(func1d(inarr_view[ind0], *args, **kwargs))
  File "/opt/conda/lib/python3.8/site-packages/nitime/algorithms/autoregressive.py", line 96, in AR_est_YW
    ak = linalg.solve(Tm, y)
  File "/opt/conda/lib/python3.8/site-packages/scipy/linalg/basic.py", line 137, in solve
    a1 = atleast_2d(_asarray_validated(a, check_finite=check_finite))
  File "/opt/conda/lib/python3.8/site-packages/scipy/_lib/_util.py", line 262, in _asarray_validated
    a = toarray(a)
  File "/opt/conda/lib/python3.8/site-packages/numpy/lib/function_base.py", line 488, in asarray_chkfinite
    raise ValueError(
ValueError: array must not contain infs or NaNs

[...]

211209-18:20:13,934 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_02_wf.func_preproc_task_cuedSGT_run_01_echo_1_wf.bold_confounds_wf.rename_acompcor" in "/scratch/fmriprep_wf/single_subject_02_wf/func_preproc_task_cuedSGT_run_01_echo_1_wf/bold_confounds_wf/rename_acompcor".
211209-18:20:13,942 nipype.workflow INFO:
	 [Node] Running "rename_acompcor" ("fmriprep.interfaces.confounds.RenameACompCor")
211209-18:20:13,951 nipype.workflow INFO:
	 [MultiProc] Running 2 tasks, and 2 jobs ready. Free memory (GB): 3.70/4.10, Free processors: 0/2.
                     Currently running:
                       * fmriprep_wf.single_subject_02_wf.func_preproc_task_cuedSGT_run_01_echo_1_wf.bold_confounds_wf.rename_acompcor
                       * fmriprep_wf.single_subject_02_wf.func_preproc_task_cuedSGT_run_01_echo_1_wf.bold_std_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.unifize
211209-18:20:13,953 nipype.workflow WARNING:
	 Storing result file without outputs
211209-18:20:13,958 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_wf.single_subject_02_wf.func_preproc_task_cuedSGT_run_01_echo_1_wf.bold_confounds_wf.rename_acompcor" (/scratch/fmriprep_wf/single_subject_02_wf/func_preproc_task_cuedSGT_run_01_echo_1_wf/bold_confounds_wf/rename_acompcor)
211209-18:20:13,964 nipype.workflow ERROR:
	 Node rename_acompcor failed to run on host c9f66203887e.
211209-18:20:13,981 nipype.workflow ERROR:
	 Saving crash info to /out/fmriprep/sub-02/log/20211209-165718_cc58323e-ea47-433c-b35f-48a560af514c/crash-20211209-182013-UID1001-rename_acompcor-2585a690-34b3-4566-9247-d2ec1e91405e.txt
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 428, in run
    runtime = self._run_interface(runtime)
  File "/opt/conda/lib/python3.8/site-packages/fmriprep/interfaces/confounds.py", line 139, in _run_interface
    components = pd.read_csv(self.inputs.components_file, sep='\t')
  File "/opt/conda/lib/python3.8/site-packages/pandas/io/parsers.py", line 610, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/opt/conda/lib/python3.8/site-packages/pandas/io/parsers.py", line 462, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/opt/conda/lib/python3.8/site-packages/pandas/io/parsers.py", line 819, in __init__
    self._engine = self._make_engine(self.engine)
  File "/opt/conda/lib/python3.8/site-packages/pandas/io/parsers.py", line 1050, in _make_engine
    return mapping[engine](self.f, **self.options)  # type: ignore[call-arg]
  File "/opt/conda/lib/python3.8/site-packages/pandas/io/parsers.py", line 1873, in __init__
    self._reader = parsers.TextReader(self.handles.handle, **kwds)
  File "pandas/_libs/parsers.pyx", line 521, in pandas._libs.parsers.TextReader.__cinit__
pandas.errors.EmptyDataError: No columns to parse from file

Additional information / screenshots

Recent example: https://app.circleci.com/pipelines/github/nipreps/fmriprep/652/workflows/72ac3c88-f8b2-4bc1-95ec-8eace7933b5d/jobs/3582
@effigies effigies added the bug label Dec 9, 2021
@mgxd
Copy link
Collaborator

mgxd commented Dec 15, 2021

is this happening with the LTS, or just 21.0.x(rc)s?

@effigies
Copy link
Member Author

I believe it was just on master.

@effigies effigies mentioned this issue Jan 20, 2022
Closed
@effigies effigies added this to the 21.0.1 milestone Jan 20, 2022
@effigies
Copy link
Member Author

effigies commented Jan 20, 2022 •
edited
Loading

Okay, I think I understand. In #2610/#2611 we changed the masking for multi-echo. These are used in DVARS and ACompCor. It will take a bit more effort to understand the failure case and come up with the correct fix.

@effigies effigies modified the milestones: 21.0.1, 21.0.x (Maintenance) Jan 21, 2022
@dlevitas dlevitas mentioned this issue Feb 3, 2022
Closed
@effigies
Copy link
Member Author

For future debugging, here is a failure with the master random seed 12187. That may help with reproducing.

@mgxd mgxd mentioned this issue Apr 6, 2022
Merged
@EricDeveaud EricDeveaud mentioned this issue Apr 13, 2022
Closed
@mgxd
Copy link
Collaborator

mgxd commented Apr 13, 2022

Looks like this is not exclusive to multi-echo (see #2761)

@mgxd
Copy link
Collaborator

mgxd commented Aug 30, 2022

I believe this was fixed in #2731

@mgxd mgxd closed this as completed Aug 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
21.0.x (Maintenance)
Development

No branches or pull requests
2 participants
@effigies @mgxd