Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update_documentation #214

Merged
merged 3 commits into from
Sep 19, 2022
Merged

Update_documentation #214

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
af80a8e
Update_documentation
mcespedes99 Sep 9, 2022
0bae0a3
added an additional note in the preamble
akhanf Sep 13, 2022
f577c66
made a few more additions
akhanf Sep 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 50 additions & 9 deletions docs/contributing/contributing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ Hippunfold dependencies are managed with Poetry, which you\'ll need
installed on your machine. You can find instructions on the [poetry
website](https://python-poetry.org/docs/master/#installation).

Note: These instructions are only recommended if you are making changes to HippUnfold code to commit back to this repository, or if you are using Snakemake's cluster execution profiles. If not, it is easier to run HippUnfold when it is packaged into a single singularity container (e.g. `docker://khanlab/hippunfold:latest`).

## Set-up your development environment:

Clone the repository and install dependencies and dev dependencies with
Expand Down Expand Up @@ -71,7 +73,7 @@ from the `hippunfold` folder that contains the
## Instructions for Compute Canada

This section provides an example of how to set up a `pip installed` copy
of HippUnfold on CompateCanada\'s `graham` cluster.
of HippUnfold on Compute Canada\'s `graham` cluster.

### Setting up a dev environment on graham:

Expand All @@ -91,10 +93,19 @@ graham to run HippUnfold:
git clone https://github.com/khanlab/hippunfold.git
pip install hippunfold/

3. Run hippunfold:
3. To run Hippunfold on Graham as a member of the Khan lab, please configure the
[neuroglia-helpers](https://github.com/khanlab/neuroglia-helpers) with the khanlab profile.

4. To avoid having to download trained models (see section [below](#deep-learning-nnu-net-model-files)), you can set an environment variable in your bash profile (~/.bash_profile) with the location of the
trained models. For Khan lab's members, the following line must be add to the bash profile file:

export HIPPUNFOLD_CACHE_DIR="/project/6050199/akhanf/opt/hippunfold_trained_models"

Note: make sure to reload your bash profile if needed (`source ~./bash_profile`).

5. For an easier execution in Graham, it's recommended to also install
[cc-slurm](https://github.com/khanlab/cc-slurm) snakemake profile for cluster execution with slurm.

hippunfold ...

Note if you want to run hippunfold with modifications to your cloned
repository, you either need to pip install again, or run hippunfold the following, since
an `editable` pip install is not allowed with pyproject:
Expand All @@ -111,32 +122,62 @@ regularSubmit or regularInteractive wrappers, and the
In an interactive job (for testing):

regularInteractive -n 8
hippunfold PATH_TO_BIDS_DIR PATH_TO_OUTPUT_DIR participant \
--participant_label 001 -j 8
hippunfold <PATH_TO_BIDS_DIR> <PATH_TO_OUTPUT_DIR> participant \
--participant_label 001 -j 8 --modality T1w --use-singularity \
--singularity-prefix $SNAKEMAKE_SINGULARITY_DIR

Here, the last line is used to specify only one subject from a BIDS
Where:
- `--participant_label 001` is used to specify only one subject from a BIDS
directory presumeably containing many subjects.
- `-j 8` specifies the number of cores used
- `--modality T1w` is used to specify that a T1w dataset is being processed
- `--singularity-prefix $SNAKEMAKE_SINGULARITY_DIR` specifies the directory in
which singularity images will be stored. The environment variable is created
when installing neuroglia-helpers.

Submitting a job (for larger cores, more subjects), still single job,
but snakemake will parallelize over the 32 cores:

regularSubmit -j Fat \
hippunfold PATH_TO_BIDS_DIR PATH_TO_OUTPUT_DIR participant -j 32
hippunfold PATH_TO_BIDS_DIR PATH_TO_OUTPUT_DIR participant -j 32 \
--modality T1w --use-singularity --singularity-prefix $SNAKEMAKE_SINGULARITY_DIR

Scaling up to \~hundred subjects (needs cc-slurm snakemake profile
installed), submits 1 16core job per subject:

hippunfold PATH_TO_BIDS_DIR PATH_TO_OUTPUT_DIR participant \
--modality T1w --use-singularity --singularity-prefix $SNAKEMAKE_SINGULARITY_DIR \
--profile cc-slurm

Scaling up to even more subjects (uses group-components to bundle
multiple subjects in each job), 1 32core job for N subjects (e.g. 10):

hippunfold PATH_TO_BIDS_DIR PATH_TO_OUTPUT_DIR participant \
--modality T1w --use-singularity --singularity-prefix $SNAKEMAKE_SINGULARITY_DIR \
--profile cc-slurm --group-components subj=10

### Running hippunfold jobs on the CBS server
1. Clone the repository and install dependencies and dev dependencies with poetry:

git clone http://github.com/khanlab/hippunfold
cd hippunfold
poetry install
If poetry is not installed, please refer to the [installation documentation](https://python-poetry.org/docs/). If the command poetry is not found, add the following line to your bashrc file located in your home directory (considering that the poetry binary is located under `$HOME/.local/bin`:

export PATH=$PATH:$HOME/.local/bin
2. To avoid having to download containers and trained models (see section [below](#deep-learning-nnu-net-model-files)), add the `$SNAKEMAKE_SINGULARITY_DIR` and `$HIPPUNFOLD_CACHE_DIR` environment variables to the bashrc file. For Khan lab's members, add the following lines:

export SNAKEMAKE_SINGULARITY_DIR="/srv/khan/shared/containers/snakemake_containers"
export HIPPUNFOLD_CACHE_DIR="/srv/khan/shared/data/hippunfold_models"
This will work only if the `setup_automount_v2` script was already executed.
3. HippUnfold might be executed using `poetry run hippunfold <arguments>` or through the `poetry shell` method. Refer to previous section for more information in regards to execution options.

4. On the CBS server you should always set your output folder to a path inside `/localscratch`, and not your home folder or a `/srv` or `/cifs` path, and copy the final results out after they have finished computing. Please be aware that the CBS server may not be the most efficient option for running a large number of subjects (since you are limited in processing cores vs a HPC cluster).

5. If you are using input files in your home directory (or in your `graham` mount in your home directory), you may also need to also add the following to your bashrc file (Note: this will become a default system-enabled option soon)

export SINGULARITY_BINDPATH="/home/ROBARTS:/home/ROBARTS"


## Deep learning nnU-net model files

The trained model files we use for hippunfold are large and thus are not
Expand Down