Seg fault when using batch processing

[MTclement1]

Hi,
MotionCorr works perfectly on single file (although for compiling I needed the -no-pie option as others), but when using batch the program seg fault after image loading.

The command for my tests is (I aliased motionCorr) : motionCor -InMrc ./data/ -OutMrc ./MotionCor/ -Gpu 0 -Patch 5 5 -Iter 10 -Serial 1

In the output the files are found :

added: ./data/test_fram_004.mrc
added: ./data/test_fram_006.mrc
added: ./data/test_fram_005.mrc
added: ./data/test_fram_009.mrc
added: ./data/test_fram_000.mrc
added: ./data/test_fram_010.mrc
added: ./data/test_fram_003.mrc
added: ./data/test_fram_001.mrc
added: ./data/test_fram_008.mrc
added: ./data/test_fram_007.mrc
added: ./data/test_fram_002.mrc

But after seemingly loading 5 files it segfault. Here is the last bit of the output :

MRC file size mode: 4096 4096 20 1
Rendered size mode: 4096 4096 20 1
MRC file size mode: 4096 4096 20 1
Rendered size mode: 4096 4096 20 1
GPU 0 Allocation time: GPU ( 0.06 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 0 Allocation time: GPU ( 0.25 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 0 Allocation time: GPU ( 0.31 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 0 Allocation time: GPU ( 1.25 GB) 0.01 s, CPU ( 0.00 GB) 0.00 s
Create buffers: total memory allocation 0.17 GB
Create buffers: 0.04 seconds

MRC file size mode: 4096 4096 20 1
Rendered size mode: 4096 4096 20 1
Segmentation fault (core dumped)

Then I reduced the amount of files to 4 and it just finish with no output :

MRC file size mode: 4096 4096 20 1
Rendered size mode: 4096 4096 20 1
MRC file size mode: 4096 4096 20 1
Rendered size mode: 4096 4096 20 1
GPU 0 Allocation time: GPU ( 0.06 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 0 Allocation time: GPU ( 0.25 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 0 Allocation time: GPU ( 0.31 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 0 Allocation time: GPU ( 0.01 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 0 Allocation time: GPU ( 1.25 GB) 0.02 s, CPU ( 0.00 GB) 0.00 s
Create buffers: total memory allocation 0.17 GB
Create buffers: 0.04 seconds

Total time: 0.088412 sec

I use a RTX 3080 that is recongnized by serial EM as having 9987 MB of VRAM left which sounds right.

Anyone has an idea on what I can do ?
Thanks

[leetleyang]

We're seeing a similar issue with batch processing:

MotionCor3 -InMrc /data/stack_folder/ -OutMrc /data/corrected/ -LogDir /data/log/ -Patch 5 5 10 -FmDose 1.1 -Kv 200 -PixSize 0.96 -SumRange 0 0 -InFmMotion 1 -Gpu 0 1 2 3 -Serial 1 -OutStar 1

Seems to adds the input files (~5000 gain-normalized MRC stacks) but then breezes through the GPU allocation messages (e.g. above) without writing any outputs.

Processes a single movie fine, but -OutStar 1 does not lead to a corresponding star file being written.

Something amiss during our compilation, perhaps?

[szhengczii]

@MTclement1: Sorry for the tardy response. I will look into this and let you know.
@leetleyang: I will also look into this and let you know.

[jonathanrd]

I also have this issue, have there been any updates or workarounds?

[Fengyun0101]

I also have this issue when I process .mrc files of tomo datasets with -Serial 1.
end with the info as:

MRC file size mode: 5760 4092 6 0
Rendered size mode: 5760 4092 6 0
MRC file size mode: 5760 4092 6 0
Rendered size mode: 5760 4092 6 0
GPU 2 Allocation time: GPU ( 0.09 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 3 Allocation time: GPU ( 0.09 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 2 Allocation time: GPU ( 0.35 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 3 Allocation time: GPU ( 0.35 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 2 Allocation time: GPU ( 0.07 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 3 Allocation time: GPU ( 0.07 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 2 Allocation time: GPU ( 0.00 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 3 Allocation time: GPU ( 0.00 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 2 Allocation time: GPU ( 0.26 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
GPU 3 Allocation time: GPU ( 0.26 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s
Create buffers: total memory allocation 0.37 GB
Create buffers: 0.07 seconds

MRC file size mode: 5760 4092 6 0
Rendered size mode: 5760 4092 6 0
Segmentation fault (core dumped)

By the way, I succefully processed the .eer files with MotionCor3 1.1.1. Could anyone have the updates? Many thanks.

[szhengczii]

Hi Fengyun, I need two movies for debugging. Is it ok share some with me? Thanks. Best, Shawn

…

On Tue, Mar 12, 2024 at 12:18 PM Fengyun0101 ***@***.***> wrote: I also have this issue when I process .mrc files of tomo datasets with -Serial 1. end with the info as: MRC file size mode: 5760 4092 6 0 Rendered size mode: 5760 4092 6 0 MRC file size mode: 5760 4092 6 0 Rendered size mode: 5760 4092 6 0 GPU 2 Allocation time: GPU ( 0.09 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s GPU 3 Allocation time: GPU ( 0.09 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s GPU 2 Allocation time: GPU ( 0.35 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s GPU 3 Allocation time: GPU ( 0.35 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s GPU 2 Allocation time: GPU ( 0.07 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s GPU 3 Allocation time: GPU ( 0.07 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s GPU 2 Allocation time: GPU ( 0.00 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s GPU 3 Allocation time: GPU ( 0.00 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s GPU 2 Allocation time: GPU ( 0.26 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s GPU 3 Allocation time: GPU ( 0.26 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s Create buffers: total memory allocation 0.37 GB Create buffers: 0.07 seconds MRC file size mode: 5760 4092 6 0 Rendered size mode: 5760 4092 6 0 Segmentation fault (core dumped) By the way, I succefully processed the .eer files with MotionCor3 1.1.1. Could anyone have the updates? Many thanks. — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BBUDUPYIBUWK7WFZ6SGSWJ3YX5IJJAVCNFSM6AAAAAA64EC7NOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJSGM3TCMZTGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[Fengyun0101]

Hi Shawn, Thank you so much for the reply. I uploaded 5 files as attached. The mrc files were acquired from Krios K3 and already with gain normalized. The pixel size is 2.62A. Total dose is 0.45e/A2. Best, Feng mrc_fractions_Feng_0314.zip <https://drive.google.com/file/d/1LUbWyPAdmsa-ceajJ8h_wm-WOrRVIHIz/view?usp=drive_web> On Thu, Mar 14, 2024 at 10:05 AM Shawn Zheng ***@***.***> wrote:

…

Hi Fengyun, I need two movies for debugging. Is it ok share some with me? Thanks. Best, Shawn On Tue, Mar 12, 2024 at 12:18 PM Fengyun0101 ***@***.***> wrote: > I also have this issue when I process .mrc files of tomo datasets with > -Serial 1. > end with the info as: > > MRC file size mode: 5760 4092 6 0 > Rendered size mode: 5760 4092 6 0 > MRC file size mode: 5760 4092 6 0 > Rendered size mode: 5760 4092 6 0 > GPU 2 Allocation time: GPU ( 0.09 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s > GPU 3 Allocation time: GPU ( 0.09 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s > GPU 2 Allocation time: GPU ( 0.35 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s > GPU 3 Allocation time: GPU ( 0.35 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s > GPU 2 Allocation time: GPU ( 0.07 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s > GPU 3 Allocation time: GPU ( 0.07 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s > GPU 2 Allocation time: GPU ( 0.00 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s > GPU 3 Allocation time: GPU ( 0.00 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s > GPU 2 Allocation time: GPU ( 0.26 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s > GPU 3 Allocation time: GPU ( 0.26 GB) 0.00 s, CPU ( 0.00 GB) 0.00 s > Create buffers: total memory allocation 0.37 GB > Create buffers: 0.07 seconds > > MRC file size mode: 5760 4092 6 0 > Rendered size mode: 5760 4092 6 0 > Segmentation fault (core dumped) > > By the way, I succefully processed the .eer files with MotionCor3 1.1.1. > Could anyone have the updates? Many thanks. > > — > Reply to this email directly, view it on GitHub > < #8 (comment)>, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/BBUDUPYIBUWK7WFZ6SGSWJ3YX5IJJAVCNFSM6AAAAAA64EC7NOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJSGM3TCMZTGQ> > . > You are receiving this because you commented.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BG5BMII7YX4SGKSIUOAWZFTYYG4DDAVCNFSM6AAAAAA64EC7NOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJXGY3TCNRQHA> . You are receiving this because you commented.Message ID: ***@***.***>

[Poko18]

I have similar problem. Job gets killed during the batch process:
cmd:
/usr/local/MotionCor3 -InTiff dataset/ -InSuffix .tif -OutMrc 3/sum/corrected -Patch 5 5 -Gain dataset/GLP-1_gain.mrc -Gpu 0 -Kv 300 -PixSize 0.83 -FmDose 0.8 -Serial 1 -OutStar 1 -LogDir 3/logdir

Output:

Gain reference has been loaded.
DarkReference not found.

TIFF file size mode: 5760  4092  75  0
Rendered size mode: 5760  4092  75  0

GPU 0 Allocation time: GPU (  0.26 GB)   0.00 s, CPU (  0.00 GB)   0.00 s
GPU 0 Allocation time: GPU (  0.35 GB)   0.00 s, CPU (  0.00 GB)   0.00 s
GPU 0 Allocation time: GPU (  1.65 GB)   0.00 s, CPU (  0.00 GB)   0.00 s
GPU 0 Allocation time: GPU (  0.07 GB)   0.00 s, CPU (  0.00 GB)   0.00 s
GPU 0 Allocation time: GPU (  6.59 GB)   0.00 s, CPU (  0.00 GB)   0.00 s
Create buffers: total memory allocation 0.28 GB
Create buffers: 0.02 seconds

Killed

Im running on A10 GPU with 23Gb of memory. Does anyone have an idea what is going on?