Possibility to continue interrupted analysis?

[MrImaging]

Dear experts,

Our shared server has crashed (more on that below) while I was preprocessing an extensive dataset (12 minutes of resting state for a large number of subjects with sometimes multiple measurements) using docker. As I am using the fast_commonspace option, I was wondering whether it would be possible to somehow just restart processing from the last successful step and still getting the outputs and diagnosis reports for my whole data set all in one place, instead of having to rerun everything?

Furthermore, I would be grateful for assistance regarding the parallelization option -p MultiProc and --scale_min_memory:

I have been using the following parameters:

Running RABIES - version: 0.5.1
[...]
-> plugin = MultiProc 
-> local_threads = 2 
-> scale_min_memory = 1.0 
-> min_proc = 1 
[...]

The last entry in the log file is

240822-12:08:54,906 nipype.workflow INFO:
	 [Node] Running "InhoCorrection" ("rabies.preprocess_pkg.inho_correction.InhoCorrection")
240822-12:08:56,154 nipype.workflow INFO:
	 [MultiProc] Running 2 tasks, and 1262 jobs ready. Free memory (GB): 1226.07/1227.27, Free processors: 0/2.
                     Currently running:
                       * main_wf.anat_inho_cor_wf.InhoCorrection
                       * main_wf.anat_inho_cor_wf.InhoCorrection

after which it stops.

I have been told the virtual server crashed due to the availble RAM (1,4 TB) being fully used. As we are currently trying to locate/understand the issue: Is it plausible/possible that memory has constantly been newly allocated to the next preprocessing steps and memory from completed steps has not been freed?
Do you have any tips or suggestions regarding these parameters?

Thank you very much!

[gdevenyi]

Is it plausible/possible that memory has constantly been newly allocated to the next preprocessing steps and memory from completed steps has not been freed?

With RAM (14 TB) massively unlikely, unless your dataset is hugely malformed and causing some algorithm to go haywire. Are you sure about that 14TB number? That's a $200k+ machine. Perhaps you mean 14GB?

virtual server crashed

There would be logs associated with this, check with your sysadmins regarding what happened in the kernel messages, "Out of Memory OOM" will appear.

[MrImaging]

My bad, the comma got lost, it's 1.4 TB (1.3 TiB). My BIDS input folder is around 60 GB of compressed (*.nii.gz) structural and functional scans.

[gdevenyi]

I believe RABIES processing is resumab le via the functions provided by nipype.

[MrImaging]

Thank you, I will look into that. Maybe something like "-f --resume" would be an option for a future release?

[Gab-D-G]

Nipype automatically handles resuming an interrupted process (--resume is always there). Basically, all nodes (a node is a single operation, e.g. main_wf.anat_inho_cor_wf.InhoCorrection) which were completed in the previous run should not be run again. However if a run was interrupted without finishing, it will be run again. However, can only reliably work if no output was removed, and no parameters were changed in the command.