Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: Merge parcellations fail with extra dimensions #331

Merged
merged 3 commits into from
May 6, 2024
Merged

[BUG]: Merge parcellations fail with extra dimensions #331

merged 3 commits into from
May 6, 2024

Conversation

synchon
Copy link
Member

@synchon synchon commented Apr 15, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Running a simple FC marker with two parcellations fails:

/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/data/parcellations.py:1785: RuntimeWarning: The parcellations have different resolutions!Resampling all parcellations to the first one in the list.
  warn_with_log(
Traceback (most recent call last):
  File "/home/fraimondo/miniconda3/envs/neurodc/bin/junifer", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/api/cli.py", line 223, in run
    api_run(
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/api/functions.py", line 176, in run
    mc.fit(datagrabber_object[t_element])
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/collection.py", line 101, in fit
    m_value = marker.fit_transform(data, storage=self._storage)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/pipeline/pipeline_step_mixin.py", line 229, in fit_transform
    return self._fit_transform(input=input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/base.py", line 201, in _fit_transform
    t_out = self.compute(input=t_input, extra_input=extra_input)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/functional_connectivity/functional_connectivity_base.py", line 138, in compute
    aggregation = self.aggregate(input, extra_input=extra_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/functional_connectivity/functional_connectivity_parcels.py", line 102, in aggregate
    return parcel_aggregation.compute(input, extra_input=extra_input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/parcel_aggregation.py", line 191, in compute
    parcellation_img, labels = get_parcellation(
                               ^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/data/parcellations.py", line 348, in get_parcellation
    resampled_parcellation_img, labels = merge_parcellations(
                                         ^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/data/parcellations.py", line 1805, in merge_parcellations
    parc_data[parc_data == 0] += t_parc_data[parc_data == 0]
                                 ~~~~~~~~~~~^^^^^^^^^^^^^^^^
IndexError: too many indices for array: array is 3-dimensional, but 4 were indexed

Expected Behavior

to run withour error.

Steps To Reproduce

  1. Install latest (--pre) junifer from pip.
  2. run this yaml:
workdir: /tmp

datagrabber:
  kind: DataladAOMICID1000
  types: 
    - BOLD
    - BOLD_confounds
    - BOLD_mask
preprocess:
    kind: fMRIPrepConfoundRemover
    detrend: true
    standardize: true
    strategy:
        motion: full
        wm_csf: full
        global_signal: full
    low_pass: 0.08
    high_pass: 0.01
    masks:
      - inherit
      - compute_epi_mask
      - threshold: 0
markers:
  - name: parccortical-Schaefer100x17-TianxS2x3T
    kind: FunctionalConnectivityParcels
    parcellation: 
        - Schaefer100x17
        - TianxS2x3TxMNInonlinear2009cAsym
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit

storage:
  kind: HDF5FeatureStorage
  uri: "/data/group/riseml/fraimondo/ds003097_FC/ds003097_FC.hdf5"
queue:
  jobname: aomic_fc
  kind: HTCondor
  env:
    kind: conda
    name: neurodc
  mem: 8G
  disk: 5G
  verbose: info
  pre_run: |
    source /data/group/appliedml/tools/afni_23.1.10/afni.sh
    source /data/group/appliedml/tools/ants_2.5.0/ants.sh
    source /data/group/appliedml/tools/fsl_6.0.4-patched2/fsl.sh

Environment

junifer:
  version: 0.0.5.dev27
python:
  version: 3.11.8
  implementation: CPython
dependencies:
  click: 8.1.7
  numpy: 1.26.4
  scipy: 1.11.4
  datalad: 0.19.6
  pandas: 2.1.4
  nibabel: 5.2.1
  nilearn: 0.10.2
  sqlalchemy: 2.0.29
  ruamel.yaml: 0.17.40
  httpx: 0.26.0
  tqdm: 4.66.1
  templateflow: 24.2.0
  looseversion: None
system:
  platform: Linux-6.6.13+bpo-amd64-x86_64-with-glibc2.36
environment:
  LC_CTYPE: en_US.UTF-8
  PATH: 
    /home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games

Relevant log output

Activating neurodc
Sourcing pre_run.sh
Script dir from afni.sh: /data/group/appliedml/tools/afni_23.1.10
Path after sourcing afni.sh: /data/group/appliedml/tools/afni_23.1.10/binaries:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games
Script dir from ants.sh: /data/group/appliedml/tools/ants_2.5.0
Path after sourcing ants.sh: /data/group/appliedml/tools/ants_2.5.0/binaries:/data/group/appliedml/tools/afni_23.1.10/binaries:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games
Script dir from fsl.sh: /data/group/appliedml/tools/fsl_6.0.4-patched2
Path after sourcing fsl.sh: /data/group/appliedml/tools/fsl_6.0.4-patched2/binaries:/data/group/appliedml/tools/ants_2.5.0/binaries:/data/group/appliedml/tools/afni_23.1.10/binaries:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games
Running junifer run /home/fraimondo/dev/scratch/neurodc/junifer_jobs/aomic_fc/config.yaml --verbose info --element sub-0014 in conda environment
2024-04-12 22:25:36,051 - JUNIFER - INFO - ===== Lib Versions =====
2024-04-12 22:25:36,051 - JUNIFER - INFO - numpy: 1.26.4
2024-04-12 22:25:36,051 - JUNIFER - INFO - scipy: 1.11.4
2024-04-12 22:25:36,051 - JUNIFER - INFO - pandas: 2.1.4
2024-04-12 22:25:36,051 - JUNIFER - INFO - nilearn: 0.10.2
2024-04-12 22:25:36,051 - JUNIFER - INFO - nibabel: 5.2.1
2024-04-12 22:25:36,051 - JUNIFER - INFO - junifer: 0.0.5.dev27
2024-04-12 22:25:36,051 - JUNIFER - INFO - ========================
2024-04-12 22:25:36,051 - JUNIFER - INFO - Parsing yaml file: /home/fraimondo/dev/scratch/neurodc/junifer_jobs/aomic_fc/config.yaml
2024-04-12 22:25:36,058 - JUNIFER - INFO - `datadir` is None, creating a temporary directory
2024-04-12 22:25:36,058 - JUNIFER - INFO - `datadir` set to /tmp/tmpgd5w41ln/datadir
2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Marker Collection
2024-04-12 22:25:36,289 - JUNIFER - INFO - DataGrabber output type: ['BOLD', 'BOLD_confounds', 'BOLD_mask']
2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Data Reader:
2024-04-12 22:25:36,289 - JUNIFER - INFO - Data Reader output type: ['BOLD', 'BOLD_confounds', 'BOLD_mask']
2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Preprocessor: fMRIPrepConfoundRemover
2024-04-12 22:25:36,289 - JUNIFER - INFO - Preprocess output type: ['BOLD']
2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Marker: parccortical-Schaefer100x17-TianxS2x3T
2024-04-12 22:25:36,386 - JUNIFER - INFO - Marker output type: ['matrix']
2024-04-12 22:25:36,386 - JUNIFER - INFO - Validating storage for parccortical-Schaefer100x17-TianxS2x3T
2024-04-12 22:27:39,404 - JUNIFER - INFO - Getting element sub-0014
2024-04-12 22:28:45,106 - JUNIFER - INFO - Fitting pipeline
2024-04-12 22:28:45,106 - JUNIFER - INFO - Reading BOLD from /tmp/tmpgd5w41ln/datadir/derivatives/fmriprep/sub-0014/func/sub-0014_task-moviewatching_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz
2024-04-12 22:28:45,106 - JUNIFER - INFO - BOLD is type NIFTI
2024-04-12 22:28:45,107 - JUNIFER - INFO - Reading BOLD_confounds from /tmp/tmpgd5w41ln/datadir/derivatives/fmriprep/sub-0014/func/sub-0014_task-moviewatching_desc-confounds_regressors.tsv
2024-04-12 22:28:45,107 - JUNIFER - INFO - BOLD_confounds is type TSV
2024-04-12 22:28:45,123 - JUNIFER - INFO - Reading BOLD_mask from /tmp/tmpgd5w41ln/datadir/derivatives/fmriprep/sub-0014/func/sub-0014_task-moviewatching_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
2024-04-12 22:28:45,123 - JUNIFER - INFO - BOLD_mask is type NIFTI
2024-04-12 22:28:45,124 - JUNIFER - INFO - Preprocessing data with fMRIPrepConfoundRemover
2024-04-12 22:28:45,124 - JUNIFER - INFO - Preprocessing BOLD
2024-04-12 22:28:47,902 - JUNIFER - INFO - No `t_r` specified, using t_r from NIfTI header
2024-04-12 22:28:47,903 - JUNIFER - INFO - Read t_r from NIfTI header: 2.200000047683716
2024-04-12 22:28:48,266 - JUNIFER - INFO - Cleaning image using nilearn
2024-04-12 22:28:54,670 - JUNIFER - INFO - Fitting marker parccortical-Schaefer100x17-TianxS2x3T
2024-04-12 22:28:54,778 - JUNIFER - INFO - Computing BOLD
2024-04-12 22:28:54,779 - JUNIFER - INFO - Fetching one of Schaefer parcellations.
2024-04-12 22:28:54,779 - JUNIFER - INFO - Parcellation parameters:
2024-04-12 22:28:54,779 - JUNIFER - INFO - 	resolution: 3.0
2024-04-12 22:28:54,779 - JUNIFER - INFO - 	n_rois: 100
2024-04-12 22:28:54,779 - JUNIFER - INFO - 	yeo_networks: 17
2024-04-12 22:28:54,781 - JUNIFER - INFO - Loading parcellation /home/fraimondo/junifer/data/parcellations/schaefer_2018/Schaefer2018_100Parcels_17Networks_order_FSLMNI152_2mm.nii.gz
2024-04-12 22:28:54,816 - JUNIFER - INFO - Found existing xfm file for MNI152NLin6Asym to MNI152NLin2009cAsym at /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5
2024-04-12 22:28:55,773 - JUNIFER - INFO - Downloading template MNI152NLin2009cAsym in resolution 2
2024-04-12 22:28:55,892 - JUNIFER - INFO - antsApplyTransforms command to be executed:
antsApplyTransforms -d 3 -e 3 -n 'GenericLabel[NearestNeighbor]' -i /tmp/tmpol3919xi/parcellationsupbm3qm_/Schaefer100x17.nii.gz -r /tmp/tmpol3919xi/parcellationsupbm3qm_/MNI152NLin2009cAsym_T1w_3.0.nii.gz -t /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 -o /tmp/tmpyk7pw003/parcellationskeha8hp0/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz
2024-04-12 22:29:00,187 - JUNIFER - INFO - antsApplyTransforms command succeeded with the following output:
b'/tmp/tmpol3919xi/parcellationsupbm3qm_/Schaefer100x17.nii.gz is a file\n/tmp/tmpol3919xi/parcellationsupbm3qm_/MNI152NLin2009cAsym_T1w_3.0.nii.gz is a file\n/home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 is a file\n/tmp/tmpyk7pw003/parcellationskeha8hp0/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz is a prefix\nSingularity args: --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_1 --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_2 --bind /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym:/data/mount_3 --bind /tmp/tmpyk7pw003/parcellationskeha8hp0:/data/mount_4\nCorrected args for ANTS: antsApplyTransforms -d 3 -e 3 -n GenericLabel[NearestNeighbor] -i /data/mount_1/Schaefer100x17.nii.gz -r /data/mount_2/MNI152NLin2009cAsym_T1w_3.0.nii.gz -t /data/mount_3/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 -o /data/mount_4/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz\nRunning command: /data/group/appliedml/tools/ants_2.5.0/singularity_cmd exec --cleanenv --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_1 --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_2 --bind /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym:/data/mount_3 --bind /tmp/tmpyk7pw003/parcellationskeha8hp0:/data/mount_4 /data/group/appliedml/tools/ants_2.5.0/ants_v2.5.0.sif antsApplyTransforms -d 3 -e 3 -n GenericLabel[NearestNeighbor] -i /data/mount_1/Schaefer100x17.nii.gz -r /data/mount_2/MNI152NLin2009cAsym_T1w_3.0.nii.gz -t /data/mount_3/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 -o /data/mount_4/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz\n'
2024-04-12 22:29:00,249 - JUNIFER - INFO - Fetching one of Tian parcellations.
2024-04-12 22:29:00,249 - JUNIFER - INFO - Parcellation parameters:
2024-04-12 22:29:00,249 - JUNIFER - INFO - 	resolution: 3.0
2024-04-12 22:29:00,249 - JUNIFER - INFO - 	scale: 2
2024-04-12 22:29:00,249 - JUNIFER - INFO - 	space: MNI152NLin2009cAsym
2024-04-12 22:29:00,249 - JUNIFER - INFO - 	magneticfield: 3T
2024-04-12 22:29:00,251 - JUNIFER - INFO - Loading parcellation /home/fraimondo/junifer/data/parcellations/Tian2020MSA_v1.1/3T/Subcortex-Only/Tian_Subcortex_S2_3T_2009cAsym.nii.gz
2024-04-12 22:29:00,334 - JUNIFER - WARNING - The parcellations have different resolutions!Resampling all parcellations to the first one in the list.

Anything else?

No response

@fraimondo fraimondo added bug Something isn't working triage New issues waiting to be reviewed labels Apr 12, 2024
@synchon synchon self-assigned this Apr 15, 2024
@synchon synchon added parcellation Issues or pull requests related to parcellations (atlases) and removed triage New issues waiting to be reviewed labels Apr 15, 2024
@synchon synchon requested a review from fraimondo April 15, 2024 10:49
@synchon synchon added this to the 0.0.5 (alpha 4) milestone Apr 15, 2024
@synchon
Copy link
Member

synchon commented Apr 15, 2024

@fraimondo Can you check this branch out and see if the issue is solved?

@codecov Codecov
Copy link

codecov bot commented Apr 15, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (a13e587) to head (54e6fb8).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##              main      #331   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            1         1           
  Lines            1         1           
=========================================
  Hits             1         1
Flag Coverage Δ
docs 100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 15, 2024 •
edited
Loading

PR Preview Action v1.4.7
Preview removed because the pull request was closed.
2024-05-06 10:44 UTC

@fraimondo
Copy link
Contributor Author 

workdir: /tmp

datagrabber:
  kind: DataladAOMICID1000
  types: 
    - BOLD
    - BOLD_confounds
    - BOLD_mask
preprocess:
    kind: fMRIPrepConfoundRemover
    detrend: true
    standardize: true
    strategy:
        motion: full
        wm_csf: full
        global_signal: full
    low_pass: 0.08
    high_pass: 0.01
    masks:
      - inherit
      - compute_epi_mask
      - threshold: 0
markers:
  - name: parccortical-Schaefer100x17-TianxS2x3T
    kind: FunctionalConnectivityParcels
    parcellation: 
        - Schaefer100x17
        - TianxS2x3TxMNInonlinear2009cAsym
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit

storage:
  kind: HDF5FeatureStorage
  uri: "/data/group/riseml/fraimondo/ds003097_FC/ds003097_FC.hdf5"
queue:
  jobname: aomic_fc
  kind: HTCondor
  env:
    kind: conda
    name: neurodc
  mem: 8G
  disk: 5G
  verbose: info
  pre_run: |
    source /data/group/appliedml/tools/afni_23.1.10/afni.sh
    source /data/group/appliedml/tools/ants_2.5.0/ants.sh
    source /data/group/appliedml/tools/fsl_6.0.4-patched2/fsl.sh

Yes, it works now.

@synchon synchon merged commit e0cb0f2 into main May 6, 2024
24 of 28 checks passed
@synchon synchon deleted the fix/extra-parc-dim-in-merge branch May 6, 2024 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fraimondo fraimondo fraimondo approved these changes

Assignees

@synchon synchon

Labels
bug Something isn't working parcellation Issues or pull requests related to parcellations (atlases)
Projects
None yet
Milestone
0.0.5 (alpha 4)
Development

Successfully merging this pull request may close these issues.
2 participants
@synchon @fraimondo