Missing dependency #119

araikes · 2024-02-14T22:27:56Z

I was trying to build this into a Singularity container and got... almost there. I get an error because the Science overlay no longer has scikit-image.

Any thoughts on a solution for that?

TheChymera · 2024-02-14T23:00:20Z

Hi @araikes , happy to hear you're interested to use SAMRI.

dev-python/scikit-image is now in ::gentoo, which is to say the main Gentoo package repository, so in a sense it's even more accessible now :)

The issue you have encountered is not caused by it being moved, but rather by it being renamed from sci-libs/scikits_image to dev-python/scikit-image, which I have now fixed in this commit.

If you are looking into containerization, we are currently overhauling the .gentoo specification.
Basically we have a container file model which will generate an image based on a .gentoo directory, used:

We are using SAMRI images here, and we have them for both:

Singularity: https://gin.g-node.org/asmacdo/opfvta-singularity
OCI (Docker/Podman): https://hub.docker.com/r/centerforopenneuroscience/opfvta

The spec is not updated in the SAMRI package, because thus far we didn't need it on its own, but we can prioritize this it would be helpful for you. Or if you want, you could also just submit an analogous PR, and I'd be happy to review and fix it up.

araikes · 2024-02-14T23:16:08Z

@TheChymera,

That's terrific and I'd rather not rebuild the wheel if I don't need to. In essence, I'm looking to test alternative preprocessing pipelines for my mouse data (particularly T2-weighted anatomical images) as masking is inconsistent at best. I assume that the SAMRI images would suit my needs at least for the time being, though I'm interested in the atlas standardization workflow, as the relationship between diffusion-based tractography and ABI data would be interesting.

I'll give the existing images a shot and see if they give me any issues

Thanks

TheChymera · 2024-02-14T23:28:38Z

Let me know. As for the ABI/DTI idea, feel free to reach out via email, I'd also be interested in looking at that. I actually expect there to be significant differences, since DTI is insensitive to low-density projections in areas of low fiber coherence. So you tend to not see neuromodulatory networks, which are the most interesting part in tracing studies. In a sense a nuanced negative finding (i.e. these are the things you're misisng in DTI) might be even more interesting.

araikes · 2024-02-14T23:42:17Z

If I don't have functional images, will SAMRI fail on me?

singularity run $SAMRI SAMRI generic-prep nifti /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects sub-1024 -o $PWD/derivatives/samri
Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.10/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.10/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'type'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 743, in common_select
    functional_scan_types = data_selection.loc[data_selection['type'] == 'func']['acq'].values
  File "/usr/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'type'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'datatype'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.10/SAMRI", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/samri/cli.py", line 13, in main
    argh.dispatch_commands([diagnose, bru2bids, l1, generic, legacy])
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 349, in dispatch_commands
    dispatch(parser, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 179, in dispatch
    for line in lines:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 290, in _execute_command
    for line in result:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 273, in _call
    result = function(*positional, **keywords)
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 392, in generic
    bids_base, out_base, out_dir, template, registration_mask, data_selection, functional_scan_types, structural_scan_types, subjects_sessions, func_ind, struct_ind = common_select(
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 746, in common_select
    functional_scan_types = data_selection.loc[data_selection['datatype'] == 'func']['acquisition'].values
  File "/usr/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'datatype'

TheChymera · 2024-02-15T00:28:23Z

Short answer, no, it should not fail.
The SAMRI data preparation functions can do functional, structural, or both.
The issue is probably that the logic fails to detect it's just structural. Let me look into it. Are you using the opfvta-reexecution-article image?

TheChymera · 2024-02-15T02:02:21Z

Ok, so the code that originally handled the struct-only processing has degraded over time since I don't use it, and we don't yet have a robust test suite.
In any case I fixed it in 1f14439 .
Thanks for reporting this.

One thing to note is that in addition to the issue in SAMRI, there's another issue in your invocation.
The --subjects parameter takes the subject values, not including the sub- key.
A correct invocation would be, e.g.:

(dev) [deco]/tmp ❱ SAMRI generic-prep -m '/usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii' --subjects 6471 -s '{"acquisition":["TurboRARE"]}' -o . /usr/share/opfvta_bidsdata/ '/usr/share/mouse-brain-templates/dsurqec_200micron.nii'

The above command now works on my machine, let me know if it doesn't on yours.
If it fails for you but not on the reference data, please send me a link to your data so I can check whether the problem might be input data formatting.

araikes · 2024-02-15T17:28:21Z

I was running the image up on DockerHub. If I rebuild the image using the Dockerfile for e.g. the article, will it pull the current version with that fix in place?

I also assume that it's going to have some issues. I used brkraw, rather than SAMRI, to convert to BIDS, so my files are named as sub-XXX_ses-XX_T2w.nii.gz rather than having the acq- tag. Based on some of the documentation, it looks like I should be able to use -s '{"modality":["T2w"]}', yes?

TheChymera · 2024-02-15T18:51:04Z

will it pull the current version with that fix in place?

I'm actually not sure, I think it uses the last versioned release from portage. Perhaps I should set up a container here for more flexibility.

Based on some of the documentation, it looks like I should be able to use -s '{"modality":["T2w"]}', yes?

Yes, tested it here with my data as well just to be sure, and that works.

araikes · 2024-02-15T21:59:35Z

Glad that works on your end.

If you come up with a container solution, let me know. I don't know Gentoo enough to modify any of the Dockerfiles to make sure that its grabbing the current state of the Git repo.

Or, alternatively, do you know where the .py files are in the filesystem? I could just bind the new version over the top of that path.

araikes · 2024-02-15T23:45:01Z

Last note... for now....

If I fix the -s parameter to match my modality, the current Docker container throws an error:

singularity run --containall -B $PWD:$PWD /groups/adamraikes/singularity_images/opfvta_2.0.0-alpha.sif SAMRI generic-prep $PWD/nifti /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects 1024 -o $PWD/derivatives/samri -s '{"modality":["T2w"]}'

Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.10/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.10/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
usage: SAMRI generic-prep [-h] [-a] [-d] [--functional-blur-xy] [-f FUNCTIONAL_MATCH] [--functional-registration-method FUNCTIONAL_REGISTRATION_METHOD] [-k] [--n-jobs] [--n-jobs-percentage N_JOBS_PERCENTAGE] [-o OUT_BASE] [--realign REALIGN] [-m REGISTRATION_MASK] [--sessions SESSIONS] [-s STRUCTURAL_MATCH]
                          [--subjects SUBJECTS] [--tr TR] [-w WORKFLOW_NAME] [--params PARAMS] [--phase-dictionary PHASE_DICTIONARY] [--enforce-dummy-scans ENFORCE_DUMMY_SCANS] [--exclude EXCLUDE]
                          bids_base template
SAMRI generic-prep: error: argument -s/--structural-match: invalid loads value: '{modality:[T2w]}'

TheChymera · 2024-02-16T02:32:16Z

do you know where the .py files are in the filesystem? I could just bind the new version over the top of that path.

So the files that were updated are the ones in the last commits, in the image they reside under /usr/lib/python3.11/site-packages/samri.

[...] -s paramater [...]

I have no idea why you might be getting that error, I can't reproduce it here on my end. The analogous command to yours, with my data, is the following:

(dev) [deco]/tmp ❱ SAMRI generic-prep /usr/share/opfvta_bidsdata  /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects 6471 -o $PWD/ -s '{"modality":["T2w"]}'
Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.11/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.11/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
/home/chymera/src/SAMRI/samri/pipelines/utils.py:235: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '['0' '0' '1' '1' '1' '0' '0' '0' '0' '1' '0' '0' '0' '1' '1' '0' '1' '1'
 '0' '0' '1' '0' '0' '0' '1' '1' '1' '0' '1' '0' '1' '1' '0' '0' '0' '1'
 '0' '1' '1' '0' '1' '0' '1' '1' '0' '0' '0' '1' '1' '0' '1' '0' '0' '0'
 '1' '1' '1' '0' '0' '0' '0' '1' '0' '1' '1' '0' '0' '0' '1' '1' '1' '0'
 '0' '1' '0' '1' '1' '0' '0' '0' '0' '0' '0' '0' '0' '0' '0' '1' '1' '0'
 '0' '1' '0' '0' '0' '0' '0' '0' '0' '0' '1' '0' '1' '1' '0' '0' '0' '0'
 '0' '0' '0' '1' '1' '0' '1' '0' '0' '0' '0' '0' '1' '1' '1' '0' '0' '0'
 '0' '0' '0' '0' '1' '1' '0' '0' '1' '0' '0' '0' '0' '0' '0' '0' '1' '0'
 '1' '1' '0' '1' '0' '1' '0' '1' '0' '0' '1' '1' '0' '0' '1' '0' '1' '1'
 '0' '0' '1' '0' '0' '0' '0' '0' '0' '0' '1' '0' '1' '0' '1' '0' '0' '0'
 '1' '1' '1' '0' '0' '0' '1' '0' '0' '1' '0' '1' '0' '0' '0' '1' '0' '0'
 '1' '0' '1' '1' '0' '0' '1' '1' '0' '0' '1' '1' '1' '0' '0' '1' '1' '0'
 '1' '0' '0' '0' '0' '0' '0' '0' '0' '1' '0' '1' '0' '0' '1' '0' '0' '0'
 '0' '0' '0' '0' '0' '0' '0' '0' '0' '0' '0' '0']' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
  df.loc[df['run'].notna(),'run'] = df.loc[df['run'].notnull(),'run'].apply(int).apply(str)
                                                 path subject session task acquisition        acq  run modality  type scan_type
26  /usr/share/opfvta_bidsdata/sub-6471/ses-ofM/an...    6471     ofM  NaN   TurboRARE  TurboRARE  NaN      T2w  anat
27  /usr/share/opfvta_bidsdata/sub-6471/ses-ofMr1/...    6471   ofMr1  NaN   TurboRARE  TurboRARE  NaN      T2w  anat
[['6471', 'ofM'], ['6471', 'ofMr1']]
240215-21:27:28,150 nipype.workflow INFO:
	 Generated workflow graph: /tmp/generic_work/graph.png (graph2use=hierarchical, simple_form=True).
240215-21:27:28,167 nipype.workflow INFO:
	 Workflow generic_work settings: ['check', 'execution', 'logging', 'monitoring']
240215-21:27:28,179 nipype.workflow INFO:
	 Running in parallel.
240215-21:27:28,181 nipype.workflow INFO:
	 [MultiProc] Running 0 tasks, and 2 jobs ready. Free memory (GB): 56.27/56.27, Free processors: 10/10.
240215-21:27:28,249 nipype.workflow INFO:
	 [Job 0] Cached (generic_work.get_s_scan).
240215-21:27:28,260 nipype.workflow INFO:
	 [Job 1] Cached (generic_work.get_s_scan).

It also doesn't sound like it's related to the data since it pops up before any read-in is attempted. The weird thing is that the strings in the variable aren't quoted in the error message '{modality:[T2w]}'... might be an indicator. Perhaps something is wrong with your shell?

I could get a similar error with a slightly modified call.
The difference is that here, instead of '{"modality":["T2w"]}', I use "{"modality":["T2w"]}" (the unquoted output gave me the idea to try this out).

(dev) [deco]/tmp ❱ SAMRI generic-prep /usr/share/opfvta_bidsdata  /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects 6471 -o $PWD/ -s "{"modality":["T2w"]}"
Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.11/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.11/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
usage: SAMRI generic-prep [-h] [-a] [-d] [--functional-blur-xy] [--functional-match FUNCTIONAL_MATCH] [--functional-registration-method FUNCTIONAL_REGISTRATION_METHOD] [-k] [--n-jobs] [--n-jobs-percentage N_JOBS_PERCENTAGE] [-o OUT_BASE] [--realign REALIGN]
                          [--registration-mask REGISTRATION_MASK] [--sessions [SESSIONS ...]] [--structural-match STRUCTURAL_MATCH] [--subjects [SUBJECTS ...]] [-t TR] [-w WORKFLOW_NAME] [--params PARAMS] [--phase-dictionary PHASE_DICTIONARY]
                          [--enforce-dummy-scans ENFORCE_DUMMY_SCANS] [--exclude EXCLUDE]
                          bids-base template
SAMRI generic-prep: error: argument --structural-match/-s: invalid loads value: '{modality:[T2w]}'

Looks like somewhere in between the input and samri, in your case, the single quotes get converted to double quotes. Maybe it's singularity?

araikes · 2024-02-16T17:49:31Z

Found it. It's in python3.10 not python3.11 and I was able to start the process by opening a shell (rather than directly running it). E.g. I changed from:

singularity run --containall -B $PWD:$PWD -B /groups/adamraikes/git_repos/SAMRI:/usr/lib/python3.10/site-packages/samri /groups/adamraikes/singularity_images/opfvta_2.0.0-alpha.sif SAMRI generic-prep $PWD/nifti /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects 1024 -o $PWD/derivatives/samri -s '{"modality":["T2w"]}'

to

singularity shell --containall -B $PWD:$PWD -B /groups/adamraikes/git_repos/SAMRI/samri:/usr/lib/python3.10/site-packages/samri /groups/adamraikes/singularity_images/opfvta_2.0.0-alpha.sif

SAMRI generic-prep $PWD/nifti /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects 1024 -o $PWD/derivatives/samri -s '{"modality":["T2w"]}'

However, I now get the following:

Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.10/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.10/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.10/SAMRI", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/samri/cli.py", line 13, in main
    argh.dispatch_commands([diagnose, bru2bids, l1, generic, legacy])
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 349, in dispatch_commands
    dispatch(parser, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 179, in dispatch
    for line in lines:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 290, in _execute_command
    for line in result:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 273, in _call
    result = function(*positional, **keywords)
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 392, in generic
    bids_base, out_base, out_dir, template, registration_mask, data_selection, functional_scan_types, structural_scan_types, subjects_sessions, func_ind, struct_ind = common_select(
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 769, in common_select
    subjects_sessions = data_selection[["subject","session"]].drop_duplicates().values.tolist()
  File "/usr/lib/python3.10/site-packages/pandas/core/frame.py", line 3813, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['subject', 'session'], dtype='object')] are in the [columns]"

Further, if I try SAMRI diagnose, I get

Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.10/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.10/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.10/SAMRI", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/samri/cli.py", line 13, in main
    argh.dispatch_commands([diagnose, bru2bids, l1, generic, legacy])
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 349, in dispatch_commands
    dispatch(parser, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 179, in dispatch
    for line in lines:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 290, in _execute_command
    for line in result:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 273, in _call
    result = function(*positional, **keywords)
  File "/usr/lib/python3.10/site-packages/samri/pipelines/diagnostics.py", line 84, in diagnose
    datafind = nio.DataFinder()
  File "/usr/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 189, in __init__
    BaseInterface._etelemetry_version_data = check_latest_version() or "n/a"
  File "/usr/lib/python3.10/site-packages/nipype/__init__.py", line 85, in check_latest_version
    import etelemetry
ModuleNotFoundError: No module named 'etelemetry'

As an aside, I pass BIDS validation with ignorable issues related to TSV files and inconsistent session numbers.

araikes · 2024-02-16T17:58:14Z

Manually check: None of the nipype imports work due to missing etelemetry and can confirm that it's not present in /usr/lib/python3.10/site-packages

TheChymera · 2024-02-19T16:30:09Z

It's not, but it should be patched to not give the error, let me check why it's giving you the error.

yarikoptic · 2024-02-27T23:40:12Z

reminiscent of

OCI Image failing con/opfvta-reexecution#68

KeyError: "None of [Index(['subject', 'session'], dtype='object')] are in the [columns]" suggests me to ask to check if data is made available where it is expected and is loaded first... Overall -- might be good for SAMRI to code more defensively -- verify first that there is ['subject', 'session'] whenever it expects it to be there later on.

re etelemetry - looking at https://github.com/nipy/nipype/blob/master/nipype/__init__.py#L75 and https://github.com/nipy/nipype/blob/056b48f23c896bed0fce1c571f33bb01922a1713/nipype/info.py#L135 it seems like etelemetry is not an optional dependency... so in this container -- how was nipype installed? or moreover -- how was it functioning for anyone?

TheChymera · 2024-02-28T21:42:14Z

@yarikoptic nipype has the NIPYPE_NO_ET variable, but it also had/has some imports which will fail even if the variable is set. I patched that code on Gentoo a while ago. Maybe I should upstream the patches, it's just that I did this in a rush since there were other things I needed to get back to. Perhaps I should re-examine the fix.

In any case, with both the patches and the variable it works, the problem was that source /etc/profiles needs to be run, or the shell I start needs to be a login shell.

@araikes so the issue is that the way in which you invoke the command doesn't source the profile. Can you prepend source /etc/profile to the command you run in the container?

araikes · 2024-02-29T16:38:06Z

I did that. Still get the following:

adamraikes@r4u27n1 /xdisk/adamraikes/PCA $ SAMRI generic-prep $PWD/nifti /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects 1024 -o $PWD/derivatives/samri -s '{"modality":["T2w"]}'
Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.10/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.10/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.10/SAMRI", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/samri/cli.py", line 13, in main
    argh.dispatch_commands([diagnose, bru2bids, l1, generic, legacy])
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 349, in dispatch_commands
    dispatch(parser, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 179, in dispatch
    for line in lines:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 290, in _execute_command
    for line in result:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 273, in _call
    result = function(*positional, **keywords)
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 392, in generic
    bids_base, out_base, out_dir, template, registration_mask, data_selection, functional_scan_types, structural_scan_types, subjects_sessions, func_ind, struct_ind = common_select(
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 769, in common_select
    subjects_sessions = data_selection[["subject","session"]].drop_duplicates().values.tolist()
  File "/usr/lib/python3.10/site-packages/pandas/core/frame.py", line 3813, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['subject', 'session'], dtype='object')] are in the [columns]"

araikes · 2024-02-29T16:48:42Z

Interestingly, if I exclude the --subjects flag, I now get:

adamraikes@r4u27n1 /xdisk/adamraikes/PCA $ SAMRI generic-prep $PWD/samri_bids /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii -o $PWD/derivatives/samri -s '{"modality":["T2w"]}'
Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.10/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.10/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
                                                 path subject session modality  type  run bval bvec scans scan_type
0   /xdisk/adamraikes/PCA/samri_bids/sub-1652/ses-...    1652      01      T2w  anat  NaN  NaN  NaN   NaN          
1   /xdisk/adamraikes/PCA/samri_bids/sub-1645/ses-...    1645      01      T2w  anat  NaN  NaN  NaN   NaN          
2   /xdisk/adamraikes/PCA/samri_bids/sub-1643/ses-...    1643      01      T2w  anat  NaN  NaN  NaN   NaN          
3   /xdisk/adamraikes/PCA/samri_bids/sub-1869/ses-...    1869      01      T2w  anat  NaN  NaN  NaN   NaN          
4   /xdisk/adamraikes/PCA/samri_bids/sub-1650/ses-...    1650      01      T2w  anat  NaN  NaN  NaN   NaN          
..                                                ...     ...     ...      ...   ...  ...  ...  ...   ...       ...
73  /xdisk/adamraikes/PCA/samri_bids/sub-2264/ses-...    2264      01      T2w  anat    2  NaN  NaN   NaN          
74  /xdisk/adamraikes/PCA/samri_bids/sub-2264/ses-...    2264      01      T2w  anat    2  NaN  NaN   NaN          
75  /xdisk/adamraikes/PCA/samri_bids/sub-2264/ses-...    2264      01      T2w  anat    2  NaN  NaN   NaN          
76  /xdisk/adamraikes/PCA/samri_bids/sub-2264/ses-...    2264      01      T2w  anat    2  NaN  NaN   NaN          
77  /xdisk/adamraikes/PCA/samri_bids/sub-2264/ses-...    2264      01      T2w  anat    1  NaN  NaN   NaN          

[78 rows x 10 columns]
[['1652', '01'], ['1645', '01'], ['1643', '01'], ['1869', '01'], ['1650', '01'], ['2601', '01'], ['1747', '04'], ['1747', '01'], ['1449', '01'], ['1316', '02'], ['1316', '01'], ['1874', '01'], ['2026', '01'], ['2940', '03'], ['2940', '02'], ['1733', '01'], ['1024', '04'], ['1024', '01'], ['1651', '01'], ['2091', '01'], ['2497', '01'], ['1450', '01'], ['1653', '01'], ['1452', '01'], ['1730', '01'], ['1888', '01'], ['1453', '01'], ['1455', '01'], ['1621', '01'], ['2213', '01'], ['3175', '01'], ['1732', '01'], ['1873', '02'], ['1873', '01'], ['1892', '01'], ['1629', '01'], ['1745', '01'], ['1644', '01'], ['1728', '01'], ['1887', '01'], ['1886', '02'], ['1654', '01'], ['3174', '01'], ['1734', '01'], ['3173', '01'], ['1803', '01'], ['2801', '01'], ['1622', '01'], ['1317', '01'], ['1017', '01'], ['1870', '01'], ['3172', '01'], ['1623', '01'], ['2270', '01'], ['1889', '02'], ['1889', '01'], ['1885', '03'], ['1885', '01'], ['2268', '01'], ['2264', '01']]
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.10/SAMRI", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/samri/cli.py", line 13, in main
    argh.dispatch_commands([diagnose, bru2bids, l1, generic, legacy])
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 349, in dispatch_commands
    dispatch(parser, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 179, in dispatch
    for line in lines:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 290, in _execute_command
    for line in result:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 273, in _call
    result = function(*positional, **keywords)
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 392, in generic
    bids_base, out_base, out_dir, template, registration_mask, data_selection, functional_scan_types, structural_scan_types, subjects_sessions, func_ind, struct_ind = common_select(
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 789, in common_select
    return bids_base, out_base, out_dir, template, registration_mask, data_selection, functional_scan_types, structural_scan_types, subjects_sessions, func_ind, struct_ind
UnboundLocalError: local variable 'functional_scan_types' referenced before assignment

Gonna go out on a limb and say it's because of line 392:

	bids_base, out_base, out_dir, template, registration_mask, data_selection, functional_scan_types, structural_scan_types, subjects_sessions, func_ind, struct_ind = common_select(
			bids_base,
			out_base,
			workflow_name,
			template,
			registration_mask,
			functional_match,
			structural_match,
			subjects,
			sessions,
			exclude,
			)

Looks like the call to common_select is incomplete.

TheChymera mentioned this issue Feb 27, 2024

Interactive container working differently than automatic execution con/opfvta-reexecution#69

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing dependency #119

Missing dependency #119

araikes commented Feb 14, 2024

TheChymera commented Feb 14, 2024 •

edited

Loading

araikes commented Feb 14, 2024

TheChymera commented Feb 14, 2024

araikes commented Feb 14, 2024

TheChymera commented Feb 15, 2024 •

edited

Loading

TheChymera commented Feb 15, 2024 •

edited

Loading

araikes commented Feb 15, 2024

TheChymera commented Feb 15, 2024 •

edited

Loading

araikes commented Feb 15, 2024 •

edited

Loading

araikes commented Feb 15, 2024

TheChymera commented Feb 16, 2024 •

edited

Loading

araikes commented Feb 16, 2024

araikes commented Feb 16, 2024

TheChymera commented Feb 19, 2024 •

edited

Loading

yarikoptic commented Feb 27, 2024

TheChymera commented Feb 28, 2024

araikes commented Feb 29, 2024

araikes commented Feb 29, 2024 •

edited

Loading

Missing dependency #119

Missing dependency #119

Comments

araikes commented Feb 14, 2024

TheChymera commented Feb 14, 2024 • edited Loading

araikes commented Feb 14, 2024

TheChymera commented Feb 14, 2024

araikes commented Feb 14, 2024

TheChymera commented Feb 15, 2024 • edited Loading

TheChymera commented Feb 15, 2024 • edited Loading

araikes commented Feb 15, 2024

TheChymera commented Feb 15, 2024 • edited Loading

araikes commented Feb 15, 2024 • edited Loading

araikes commented Feb 15, 2024

TheChymera commented Feb 16, 2024 • edited Loading

araikes commented Feb 16, 2024

araikes commented Feb 16, 2024

TheChymera commented Feb 19, 2024 • edited Loading

yarikoptic commented Feb 27, 2024

TheChymera commented Feb 28, 2024

araikes commented Feb 29, 2024

araikes commented Feb 29, 2024 • edited Loading

TheChymera commented Feb 14, 2024 •

edited

Loading

TheChymera commented Feb 15, 2024 •

edited

Loading

TheChymera commented Feb 15, 2024 •

edited

Loading

TheChymera commented Feb 15, 2024 •

edited

Loading

araikes commented Feb 15, 2024 •

edited

Loading

TheChymera commented Feb 16, 2024 •

edited

Loading

TheChymera commented Feb 19, 2024 •

edited

Loading

araikes commented Feb 29, 2024 •

edited

Loading