Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Interactive container working differently than automatic execution #69

Closed
TheChymera opened this issue Feb 27, 2024 · 6 comments
Closed

Interactive container working differently than automatic execution #69

TheChymera opened this issue Feb 27, 2024 · 6 comments

Comments

@TheChymera
Copy link
Collaborator

As a short preface, the interactive container is not really needed for this study, it's just a shell entry point with possible utility for debugging.
This issue came to my attention via a user relying on the opfvta-reexecution container as a way to gain containerized access to SAMRI (since there's no SAMRI container yet).

SAMRI uses nipype, and as part of running the preprocessing analysis nipype is imported.
That works fine (at the beginning I quit as soon as the preporcessing started, but it would continue going).
However, if I enter the container and try to import either nipype or analogous submodules of nipype which SAMRI imports, that fails:

[deco]~/docsrc/opfvta-reexecution ❱ make analysis-oci OCI_BINARY=podman
podman run \
	-it \
	--rm \
	-v /home/chymera/docsrc/opfvta-reexecution:/opt \
	-v /home/chymera/docsrc/opfvta-reexecution/inputs/opfvta_bidsdata:/usr/share/opfvta_bidsdata \
	-v /home/chymera/docsrc/opfvta-reexecution/inputs/mouse-brain-templates/mouse-brain-templates:/usr/share/mouse_brain_atlases \
	-v /home/chymera/docsrc/opfvta-reexecution/scratch:/root/.scratch \
	-v /home/chymera/docsrc/opfvta-reexecution/code/empty:/opfvta/ \
	--workdir /opt/code \
	docker.io/centerforopenneuroscience/opfvta:2.0.0-alpha \
	./produce-analysis.sh podman
/opt/code/opfvta /opt/code
rubber --clean article.tex
removing tree pythontex-files-article
removing tree pythontex-files-article
rm _minted-article -rf
rubber --clean pitch.tex
removing tree pythontex-files-pitch
removing tree pythontex-files-pitch
rm _minted-pitch -rf
rubber --clean review.tex
removing tree pythontex-files-review
rm _minted-review -rf
pushd prepare && ./run.sh
/opt/code/opfvta/prepare /opt/code/opfvta
!!! Repository 'gentoo' has sync-uri attribute, but is missing sync-type attribute
Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.10/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:283: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:296: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
                                                   path modality subject session   task acquisition        acq  run  type scan_type
1365  /root/.scratch/opfvta/bids/sub-6257/ses-ofM/fu...      cbv    6257     ofM   CogB         EPI        EPI    0  func
1367  /root/.scratch/opfvta/bids/sub-6257/ses-ofM/an...      T2w    6257     ofM    NaN   TurboRARE  TurboRARE  NaN  anat
225   /root/.scratch/opfvta/bids/sub-6259/ses-ofM/fu...      cbv    6259     ofM   CogB         EPI        EPI    0  func
230   /root/.scratch/opfvta/bids/sub-6259/ses-ofMr1/...      cbv    6259   ofMr1  JPogP         EPI        EPI    0  func
232   /root/.scratch/opfvta/bids/sub-6259/ses-ofMr1/...      cbv    6259   ofMr1   CogB         EPI        EPI    1  func
...                                                 ...      ...     ...     ...    ...         ...        ...  ...   ...       ...
1350  /root/.scratch/opfvta/bids/sub-6654/ses-ofMr1/...      T2w    6654   ofMr1    NaN   TurboRARE  TurboRARE  NaN  anat
265   /root/.scratch/opfvta/bids/sub-6655/ses-ofM/fu...      cbv    6655     ofM   CogB         EPI        EPI    0  func
269   /root/.scratch/opfvta/bids/sub-6655/ses-ofMr1/...      cbv    6655   ofMr1  CogBr         EPI        EPI    0  func
267   /root/.scratch/opfvta/bids/sub-6655/ses-ofM/an...      T2w    6655     ofM    NaN   TurboRARE  TurboRARE  NaN  anat
272   /root/.scratch/opfvta/bids/sub-6655/ses-ofMr1/...      T2w    6655   ofMr1    NaN   TurboRARE  TurboRARE  NaN  anat

[134 rows x 10 columns]
[['6257', 'ofM'], ['6259', 'ofM'], ['6259', 'ofMr1'], ['6260', 'ofMr2'], ['6260', 'ofM'], ['5682', 'ofMr2'], ['5682', 'ofMr1'], ['5682', 'ofMr3'], ['5683', 'ofMr4'], ['5683', 'ofM'], ['5683', 'ofMr1'], ['5683', 'ofMr3'], ['5685', 'ofM'], ['5685', 'ofMr1'], ['6467', 'ofM'], ['6468', 'ofM'], ['6471', 'ofM'], ['6471', 'ofMr1'], ['6560', 'ofM'], ['6560', 'ofMr1'], ['6562', 'ofM'], ['6562', 'ofMr1'], ['6573', 'ofM'], ['6573', 'ofMr1'], ['6574', 'ofM'], ['6574', 'ofMr1'], ['6575', 'ofM'], ['6575', 'ofMr1'], ['6576', 'ofM'], ['6576', 'ofMr1'], ['6578', 'ofM'], ['6578', 'ofMr1'], ['6582', 'ofMr1'], ['6582', 'ofM'], ['6586', 'ofM'], ['6587', 'ofM'], ['6587', 'ofMr1'], ['6588', 'ofM'], ['6588', 'ofMr1'], ['6589', 'ofM'], ['6589', 'ofMr1'], ['6590', 'ofM'], ['6591', 'ofM'], ['6592', 'ofM'], ['6592', 'ofMr1'], ['6593', 'ofM'], ['3839', 'ofM'], ['6639', 'ofM'], ['6643', 'ofM'], ['6642', 'ofM'], ['6654', 'ofM'], ['6654', 'ofMr1'], ['6655', 'ofM'], ['6655', 'ofMr1']]
240227-18:26:48,280 nipype.workflow INFO:
	 Generated workflow graph: /root/.scratch/opfvta/preprocess_work/graph.png (graph2use=hierarchical, simple_form=True).
240227-18:26:48,303 nipype.workflow INFO:
	 Workflow preprocess_work settings: ['check', 'execution', 'logging', 'monitoring']
^CTraceback (most recent call last):
  File "/opt/code/opfvta/prepare/preprocess.py", line 17, in <module>
    generic(bids_base,
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 637, in generic
    workflow.run(plugin="MultiProc", plugin_args={'n_procs' : n_jobs})
  File "/usr/lib/python3.10/site-packages/nipype/pipeline/engine/workflows.py", line 637, in run
    self._write_report_info(self.base_dir, self.name, execgraph)
  File "/usr/lib/python3.10/site-packages/nipype/pipeline/engine/workflows.py", line 699, in _write_report_info
    dict(source=nodes.index(u), target=nodes.index(v), value=1)
KeyboardInterrupt
make: *** [Makefile:55: data] Interrupt
make: *** [Makefile:76: analysis-oci] Error 130
[deco]~/docsrc/opfvta-reexecution ❱ make analysis-oci-interactive
docker run \
	-it \
	--rm \
	-v /home/chymera/docsrc/opfvta-reexecution:/opt \
	-v /home/chymera/docsrc/opfvta-reexecution/inputs/opfvta_bidsdata:/usr/share/opfvta_bidsdata \
	-v /home/chymera/docsrc/opfvta-reexecution/inputs/mouse-brain-templates/mouse-brain-templates:/usr/share/mouse_brain_atlases \
	-v /home/chymera/docsrc/opfvta-reexecution/scratch:/root/.scratch \
	-v /home/chymera/docsrc/opfvta-reexecution/code/empty:/opfvta/ \
	--workdir /opt/code \
	docker.io/centerforopenneuroscience/opfvta:2.0.0-alpha \
	/bin/bash
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.
make: *** [Makefile:89: analysis-oci-interactive] Error 125
[deco]~/docsrc/opfvta-reexecution ❱ make analysis-oci-interactive OCI_BINARY=podman
podman run \
	-it \
	--rm \
	-v /home/chymera/docsrc/opfvta-reexecution:/opt \
	-v /home/chymera/docsrc/opfvta-reexecution/inputs/opfvta_bidsdata:/usr/share/opfvta_bidsdata \
	-v /home/chymera/docsrc/opfvta-reexecution/inputs/mouse-brain-templates/mouse-brain-templates:/usr/share/mouse_brain_atlases \
	-v /home/chymera/docsrc/opfvta-reexecution/scratch:/root/.scratch \
	-v /home/chymera/docsrc/opfvta-reexecution/code/empty:/opfvta/ \
	--workdir /opt/code \
	docker.io/centerforopenneuroscience/opfvta:2.0.0-alpha \
	/bin/bash
f3444f102fb7 /opt/code # python
Python 3.10.10 (main, Mar 26 2023, 17:29:04) [GCC 12.2.1 20230121] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import nipype
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/site-packages/nipype/__init__.py", line 101, in <module>
    BaseInterface._etelemetry_version_data = check_latest_version() or "n/a"
  File "/usr/lib/python3.10/site-packages/nipype/__init__.py", line 85, in check_latest_version
    import etelemetry
ModuleNotFoundError: No module named 'etelemetry'

Any ideas what could be going on @yarikoptic @asmacdo
@asmacdo
Copy link
Member

asmacdo commented Feb 27, 2024

My first guess is this looks like a nipype bug to me, theres some unusual magic happening in nipype with etelemetry

https://github.com/nipy/nipype/blob/master/nipype/__init__.py#L75-L89

Maybe try setting NIPYPE_NO_ET

https://github.com/nipy/nipype/blob/056b48f23c896bed0fce1c571f33bb01922a1713/nipype/__init__.py#L96C47-L96C59

@asmacdo
Copy link
Member

asmacdo commented Feb 27, 2024

Since nipype is doing some unusual load logic, maybe try import nipype.interfaces.io as nio like how samri does it.

https://github.com/IBT-FMI/SAMRI/blob/1f1443973d1bcf123aa81456d80d27863d8b8d71/samri/pipelines/preprocess.py#L12C1-L12C35

@yarikoptic
Copy link
Member

yarikoptic commented Feb 27, 2024 •
edited
Loading

sorry -- too much text.. what is the problem question exactly?

meanwhile followed up on IBT-FMI/SAMRI#119 (comment)

@TheChymera
Copy link
Collaborator Author

@asmacdo oh god. Thank you so much for bringing up that variable, I now remember everything. So yes, nothing is broken, I just forgot some finer points regarding how to use it:

f3444f102fb7 /opt/code # source /etc/profile
f3444f102fb7 /opt/code # python
Python 3.10.10 (main, Mar 26 2023, 17:29:04) [GCC 12.2.1 20230121] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from nipype.interfaces import fsl
>>>
KeyboardInterrupt
>>>
f3444f102fb7 /opt/code # cat /etc/env.d/98nipype
export NIPYPE_NO_ET=1

We need to run source /etc/profile for NIPYPE_NO_ET to be loaded. Our produce_analysis.sh script has that command.

But now that I hit this snag (again), maybe we could do something about it?
On a normal Gentoo system, you only need that if you want to have the environment variables loaded in currently open sessions.
Each new session will auto-load the environment variables on startup via source /etc/profile.
Any idea why OCI images don't auto-do it when you start your shell?

@asmacdo
Copy link
Member

asmacdo commented Feb 28, 2024

According to ChatGPT

Invoking /bin/bash without -l (or --login) starts a non-login shell. Non-login shells do not automatically source /etc/profile or the user's .profile, .bash_profile, etc. They typically source ~/.bashrc.

So I guess you could add -l and see if that does it

@TheChymera TheChymera mentioned this issue Feb 28, 2024
Open
@TheChymera
Copy link
Collaborator Author

@asmacdo thanks, can confirm 61d09b9 fixes it 💛

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yarikoptic @TheChymera @asmacdo