error: 'video_plot_3d' task killed

[fhurtak]

When trying to generate 3d videos from volume stacks the processing sometimes fails with the following example log:

2023-02-13 13:41:08.977729: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Will run processing on the following flies:
['/mnt/nas2/FH/230208_G13Ci1_3dpe/Fly3']
Will start task manager with the following tasks:
tif: /mnt/nas2/FH/230208_G13Ci1_3dpe/Fly3
video_plot_3d: /mnt/nas2/FH/230208_G13Ci1_3dpe/Fly3
Mon Feb 13 13:41:12 2023: starting tif task for fly /mnt/nas2/FH/230208_G13Ci1_3dpe/Fly3
Mon Feb 13 13:41:12 2023 converting trial to tif: /mnt/nas2/FH/230208_G13Ci1_3dpe/Fly3/010_vol_cc_t1
/home/hurtak/miniconda3/envs/twoppp37/lib/python3.7/site-packages/utils2p/external/tifffile/tifffile.py:1203: UserWarning: truncating ImageJ file
warnings.warn("truncating ImageJ file")
Mon Feb 13 13:46:13 2023: starting video_plot_3d task for fly /mnt/nas2/FH/230208_G13Ci1_3dpe/Fly3
Could not find green_avg and red_avg files. Will compute from raw tif.
Killed

The processing worked on the previous stacks from the same fly. The only difference is the size of the images.

The _fly_dirs_to_process.txt files contains only the line:

/mnt/nas2/FH/230208_G13Ci1_3dpe/Fly3||010||tif,video_plot_3d

Could it be related to a memory problem? If so do you know how to get around it?

[jonasfbraun]

that sounds a lot like a memory problem.
Both your green and red stack have a size of 21GB each.

You could re-write the twoppp.plot.show3d.tmp_avg_3d_stacks() function such that it sequentially loads green, computes green average, deletes green, loads red, computes red avg.
Currently, it loads both and then computes the average of both.

Feel free to push the changes later. that seems like a useful improvement!

[jonasfbraun]

that's the function:

twoppp/twoppp/plot/show3d.py

Line 11 in c9fb1cf

def tmp_avg_3d_stacks(green, red, green_avg=None, red_avg=None):

[jonasfbraun]

if that also does not reduce the memory load enough, you could encorporate sequential loading and averaging of one stack using the utils2p.load_stack_batches() function:

https://github.com/NeLy-EPFL/utils2p/blob/cdb6626d5c27ec93c54e1fee7fb8c7180beba300/utils2p/main.py#L513

[fhurtak]

Ok I'll do that. I monitored htop and indeed the computer runs out of RAM