[BUG]: Extra dimension added to parcellation after warping to another template space via ANTs

[synchon]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When I apply a mask in FC extraction using a YAML pipeline it fails.

Expected Behavior

I would like it to apply the mask correctly and not fail.

Steps To Reproduce

1. See environment below.
2. See the Yaml here:

workdir: /tmp

datagrabber:
    kind: DataladAOMICID1000


preprocess:
    kind: Smoothing
    using: "nilearn"
    "on": "BOLD"
    smoothing_params:
      fwhm: 10

storage:
  kind: HDF5FeatureStorage
  uri: data/aomic_id1000.hdf5


markers:

  - name: parccortical-Schaefer100x17FSLMNI_marker-empiricalFC
    kind: FunctionalConnectivityParcels
    parcellation: Schaefer100x17
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit

        
queue:
  jobname: smoothing_test
  kind: HTCondor
  collect: on_success_only
  env:
    kind: venv
    name: env
  mem: 20G
  verbose: 8
  pre_run: |
    source /data/group/appliedml/tools/ants_2.5.0/ants.sh

Environment

❱ junifer wtf
junifer:
  version: 0.0.4.dev822
python:
  version: 3.10.10
  implementation: CPython
dependencies:
  click: 8.1.7
  numpy: 1.26.4
  scipy: 1.11.4
  datalad: 0.19.6
  pandas: 2.1.4
  nibabel: 5.2.1
  nilearn: 0.10.2
  sqlalchemy: 2.0.28
  ruamel.yaml: 0.17.21
  httpx: 0.26.0
  tqdm: 4.66.1
  templateflow: 24.2.0
  looseversion: None
system:
  platform: Linux-6.6.13+bpo-amd64-x86_64-with-glibc2.36
environment:
  LC_CTYPE: en_US.UTF-8
  PATH: /home/lsasse/miniconda3/bin:/home/lsasse/miniconda3/condabin:/home/lsasse/bin:/home/lsasse/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games

Relevant log output

It results in the following error:

[DEBUG] Done dropping for Dataset(/tmp/tmplwwl0ock/datadir) 
[DEBUG] Finished dropping datasets at Dataset(/tmp/tmplwwl0ock/datadir) 
Traceback (most recent call last):
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/bin/junifer", line 8, in <module>
    sys.exit(cli())
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/api/cli.py", line 224, in run
    api_run(
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/api/functions.py", line 174, in run
    mc.fit(datagrabber_object[t_element])
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/collection.py", line 101, in fit
    m_value = marker.fit_transform(data, storage=self._storage)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/pipeline/pipeline_step_mixin.py", line 229, in fit_transform
    return self._fit_transform(input=input, **kwargs)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/base.py", line 201, in _fit_transform
    t_out = self.compute(input=t_input, extra_input=extra_input)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/functional_connectivity/functional_connectivity_base.py", line 138, in compute
    aggregation = self.aggregate(input, extra_input=extra_input)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/functional_connectivity/functional_connectivity_parcels.py", line 102, in aggregate
    return parcel_aggregation.compute(input, extra_input=extra_input)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/parcel_aggregation.py", line 191, in compute
    parcellation_bin = math_img(
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/nilearn/image/image.py", line 1059, in math_img
    result = eval(formula, data_dict)
  File "<string>", line 1, in <module>
ValueError: ("Input formula couldn't be processed, you provided 'np.logical_and(img, mask)',", 'operands could not be broadcast together with shapes (60,77,65,1) (60,77,65) ')

### Anything else?

I guess it is missing np.squeeze() somewhere.

[LeSasse]

When I remove

    masks:
      - inherit

from the marker it works.

[LeSasse]

I did not actually define a mask in the preprocessor.

[LeSasse]

I have re-opened it, because the AOMIC ID1000 datagrabber does have BOLD_masks that I should be able to inherit in the marker extraction. But I think the preprocessor for Smoothing from #161 is messing with it.

[synchon]

@LeSasse I believe I've found the bug and it's that when we warp parcellation via ANTs, it somehow adds an extra axis when loading via nibabel. If you try with a parcellation without the need to warp, it'll be ok. So, I've added an explicit np.squeeze in nilearn's math_img. Check this branch out for your run and let me know if it solves it and you get correct results.

[github-actions]

PR Preview Action v1.4.7
Preview removed because the pull request was closed.
2024-04-10 17:09 UTC

[LeSasse]

Works like a charm!

[LeSasse]

This now works, the np.squeeze makes sense to me

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (0fde637) to head (c0c893b).

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #324   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            1         1           
  Lines            1         1           
=========================================
  Hits             1         1           

Flag	Coverage Δ
docs	100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.