Structural longitudinal pipelines

[mmilch01]

Longitudinal HCP pipeline

General longitudinal pipeline execution scheme is:

1. Run PreFreeSurferPipeline.sh and FreeSurferPipeline.sh on every timepoint (cross-sectional batch files)
2. Run LongitudinalFreeSurferPipeline.sh with timepoints and template ID specified (FreeSurferPipelineBatch-long.sh). See "Execution stages" below.
3. Run PostFreeSurferPipelineLongLauncher.sh (called by PostFreeSurferPipelineBatch-long.sh)

Core scripts

1. Wrapper for the longitudinal Freesurfer (FreeSurfer/LongitudinalFreeSurferPipeline.sh)
   Corresponding example batch script: Examples/Scripts/FreeSurferPipelineBatch-long.sh
2. Longitudinal conversions/nonlinear registration that runs prior to PostFreesurferPipeline (PostFreeSurfer/PostFreeSurferPipelineLongPrep.sh): called by PostFreeSurferPipelineLongLauncher.sh, see "Execution stages" below.
   Refer to docs/diagrams/out/PostFreesurferPipelineLongPrep.GeneralScheme.png for the general execution scheme.
3. PostFreesurferPipeline for longitudinal template and timepoints. Called by PostFreeSurferPipelineLongLauncher.sh.
   Refer to docs/diagrams/out/PostFreesurferPipeline-long.GeneralScheme.png for the general execution scheme.

Execution stages
LongitudinalFreesurferPipeline has two stages:

1. TEMPLATE: Create template from all timepoints
2. TIMEPOINTS: Register timepoints to template and generate surfaces/segmentations per each timepoint in template
   space that can be executed consequtively or separately, in this order.

PostFreesurferPipelineLongLauncher has five stages that can also be executed in one run or split between batches in this order:

1. PREP-TP: PostFreeSurferPipelineLongPrep.sh, process each timepoint
2. PREP-T: PostFreeSurferPipelineLongPrep.sh, create template and finalize timepoints
3. POSTFS-TP1: PostFreeSurferPipeline->FreeSurfer2CaretConvertAndRegisterNonlinear, run for each timepoint prior to
   average surface (e.g.MSMSulc) registration
4. POSTFS-T: PostFreeSurferPipeline->FreeSurfer2CaretConvertAndRegisterNonlinear, run for template and run
   registration (MSMSulc by default), and finish all PostFS scripts for template
5. POSTFS-TP2: PostFreeSurferPipeline->FreeSurfer2CaretConvertAndRegisterNonlinear (finalize using average surface
   registration), and finish all PostFS scripts per each timepoint.

Parallel modes
Two-level parallel execution is supported:

1. Pipeline scripts launched from batch scripts under Examples/Scripts. This is subject level parallelization.
2. Parallel execution launched by pipelines (LongitudinalFreesurferPipeline, PostFreesurferPipelineLongLauncher). This is intended for stage-level parallelization. Parallel functions in parallel.shlib implement support of adding jobs to stage and waiting for stage jobs to complete.

Batch scripts can have no parallelization (NONE), or use FSLSUB.

Pipelines can run the following modes:
NONE: no parallelization.
BUILTIN: parallel mode that uses parallel.shlib for concurrent within-stage execution (i.e. multi-timepoint) on the same node. Supports staged execution.
FSLSUB: parallel model that launches all jobs using fsl_sub at pipeline level. In this case, batch scripts should not use fsl_sub.
This is the most flexible mode that allows to schedule all jobs per subject at once. It also supports staged execution. FSLSUB mode adds more flexibility and will speed up longitudinal processing for multi-timepoint, multi-subject scenarios on clusters. Also we will not be dealing with specific grid engine configuration, leaving that support to fsl_sub developers. In addition, fsl_sub provides more control over job tracking compared to BUILTIN mode.

Key terminology:
Subject: In all cases, subject label. In legacy cross-sectional scenarios, this also corresponded to experiment root (see below).
Session, Visit, Timepoint: Experiment label for each session/timepoint used by longitudinal Freesurfer. 'Session' notation is also used in cross-sectional mode, replacing Subject.
Template, longitudinal template: longitudinal template label that is created by longitudinal Freesurfer. This label defines directory names for both timepoint and template files.
Experiment root: In cross-sectional case, same as 'Session' (former Subject). In longitudinal case, directory name where timepoint or template files are stored. These are generated by Freesurfer using the following logic:
Timepoint folder: <Timepoint>.long.<Template>
Template folder: <Subject>.long.<Template>
Freesurfer folder: in longitudinal case, Freesurfer folder is named following these patterns:
Timepoint: <Timepoint folder>/T1w/<Timepoint folder>
Template: <Template folder>/T1w/<Template>

[demsarjure]

The definition of the start stage parameter says:

opts_AddOptional '--start-stage' 'StartStage' 'stage_id' "Starting stage [PREP-TP]. One of PREP-TP (PostFSPrepLong timepoint processing), PREP-T (PostFSPrepLong build template, skip timepoint processing), POSTFS-TP1 (PostFreeSurfer timepoint stage 1), POSTFS-T (PostFreesurfer template), POSTFS-TP2 (PostFreesurfer timepoint stage 2)" "PREP-T"

One would assume that PREP-TP is the first step based on this list. However, the default value is PREP-T. PREP-TP is also the first step in the code. Is this correct?

[demsarjure]

Long FS seems to be working fine. There are some issues with the Post Long FS though. If I execute it with the default starting stage (PREP-T):

/opt/HCP/HCPpipelines/PostFreeSurfer/PostFreeSurferPipelineLongLauncher.sh \
  --study-folder="/data/jdemsar/studies/long/sessions/HCPA001" \
  --subject="HCPA001" \
  --sessions="HCPA001_V1@HCPA001_V2" \
  --longitudinal-template="base" \
  --t1template="/opt/HCP/HCPpipelines/global/templates/MNI152_T1_0.8mm.nii.gz" \
  --t1templatebrain="/opt/HCP/HCPpipelines/global/templates/MNI152_T1_0.8mm_brain.nii.gz" \
  --t1template2mm="/opt/HCP/HCPpipelines/global/templates/MNI152_T1_2mm.nii.gz" \
  --t2template="/opt/HCP/HCPpipelines/global/templates/MNI152_T2_0.8mm.nii.gz" \
  --t2templatebrain="/opt/HCP/HCPpipelines/global/templates/MNI152_T2_0.8mm_brain.nii.gz" \
  --t2template2mm="/opt/HCP/HCPpipelines/global/templates/MNI152_T2_2mm.nii.gz" \
  --templatemask="/opt/HCP/HCPpipelines/global/templates/MNI152_T1_0.8mm_brain_mask.nii.gz" \
  --template2mmmask="/opt/HCP/HCPpipelines/global/templates/MNI152_T1_2mm_brain_mask_dil.nii.gz" \
  --fnirtconfig="/opt/HCP/HCPpipelines/global/config/T1_2_MNI152_2mm.cnf" \
  --freesurferlabels="/opt/HCP/HCPpipelines/global/config/FreeSurferAllLut.txt" \
  --surfatlasdir="/opt/HCP/HCPpipelines/global/templates/standard_mesh_atlases" \
  --grayordinatesres="2" \
  --grayordinatesdir="/opt/HCP/HCPpipelines/global/templates/91282_Greyordinates" \
  --hiresmesh="164" \
  --lowresmesh="32" \
  --subcortgraylabels="/opt/HCP/HCPpipelines/global/config/FreeSurferSubcorticalLabelTableLut.txt" \
  --refmyelinmaps="/opt/HCP/HCPpipelines/global/templates/standard_mesh_atlases/Conte69.MyelinMap_BC.164k_fs_LR.dscalar.nii" \
  --regname="MSMSulc"

I get the error:

Wed Sep 25 07:08:01 UTC 2024:PostFreeSurferPipelineLongPrep.sh: While running '/opt/HCP/HCPpipelines/PostFreeSurfer/PostFreeSurferPipelineLongPrep.sh --subject=HCPA001 --path=/data/jdemsar/studies/long/sessions/HCPA001 --longitudinal-template=base --sessions=HCPA001_V1@HCPA001_V2 --template_processing=1 --t1template=/opt/HCP/HCPpipelines/global/templates/MNI152_T1_0.8mm.nii.gz --t1templatebrain=/opt/HCP/HCPpipelines/global/templates/MNI152_T1_0.8mm_brain.nii.gz --t1template2mm=/opt/HCP/HCPpipelines/global/templates/MNI152_T1_2mm.nii.gz --t2template=/opt/HCP/HCPpipelines/global/templates/MNI152_T2_0.8mm.nii.gz --t2templatebrain=/opt/HCP/HCPpipelines/global/templates/MNI152_T2_0.8mm_brain.nii.gz --t2template2mm=/opt/HCP/HCPpipelines/global/templates/MNI152_T2_2mm.nii.gz --templatemask=/opt/HCP/HCPpipelines/global/templates/MNI152_T1_0.8mm_brain_mask.nii.gz --template2mmmask=/opt/HCP/HCPpipelines/global/templates/MNI152_T1_2mm_brain_mask_dil.nii.gz --fnirtconfig=/opt/HCP/HCPpipelines/global/config/T1_2_MNI152_2mm.cnf --freesurferlabels=/opt/HCP/HCPpipelines/global/config/FreeSurferAllLut.txt':
Wed Sep 25 07:08:01 UTC 2024:PostFreeSurferPipelineLongPrep.sh: ERROR: More than one timepoint is specified in template mode, please check calling script.

If I set start stage to PREP-TP, I get:

Wed Sep 25 07:06:21 UTC 2024:PostFreeSurferPipelineLongPrep.sh: While running '/opt/HCP/HCPpipelines/PostFreeSurfer/PostFreeSurferPipelineLongPrep.sh --subject=HCPA001 --path=/data/jdemsar/studies/long/sessions/HCPA001 --longitudinal-template=base --sessions=HCPA001_V1 --template_processing=0 --t1template=/opt/HCP/HCPpipelines/global/templates/MNI152_T1_0.8mm.nii.gz --t1templatebrain=/opt/HCP/HCPpipelines/global/templates/MNI152_T1_0.8mm_brain.nii.gz --t1template2mm=/opt/HCP/HCPpipelines/global/templates/MNI152_T1_2mm.nii.gz --t2template=/opt/HCP/HCPpipelines/global/templates/MNI152_T2_0.8mm.nii.gz --t2templatebrain=/opt/HCP/HCPpipelines/global/templates/MNI152_T2_0.8mm_brain.nii.gz --t2template2mm=/opt/HCP/HCPpipelines/global/templates/MNI152_T2_2mm.nii.gz --templatemask=/opt/HCP/HCPpipelines/global/templates/MNI152_T1_0.8mm_brain_mask.nii.gz --template2mmmask=/opt/HCP/HCPpipelines/global/templates/MNI152_T1_2mm_brain_mask_dil.nii.gz --fnirtconfig=/opt/HCP/HCPpipelines/global/config/T1_2_MNI152_2mm.cnf --freesurferlabels=/opt/HCP/HCPpipelines/global/config/FreeSurferAllLut.txt':
Wed Sep 25 07:06:21 UTC 2024:PostFreeSurferPipelineLongPrep.sh: ERROR: At least two timepoints must be specified in timepoint processing mode, please check calling script.

So the same script errors out, once it says it needs more than one timepoint and the other time it says it has too many timepoints.