Segmentation fault

[mgujral]

I am attempting to run bwa-mem2 through a singularity container for human 35X whole genome sequence data set with following command. singularity exec /expanse/lustre/projects/scr104/mgujral/singularityContainer/WGS/version_0.0.2/ubuntuWGS.sif bwa-mem2 mem -K 100000000 -Y -t 64 -v 3 -R '@rg\tID:RES07370\tLB:RES07370\tPL:ILLUMINA\tPG:bwa-mem2-2.2.1\tCN:USU\tSM:RES07370\tPU:HY5VTCCXX.1_8' /mnt/hg38/Homo_sapiens_assembly38.fasta /mnt/RES07370/RES07370_1.fq.gz /mnt/RES07370/RES07370_2.fq.gz -o /mnt/RES07370/RES07370_bwa-mem2_128cores64threadsK100M.sam

Everything runs fine for about 10-20 minutes, and throws segment fault. I am using entire node with 256 GB of RAM, and I have tried to run with 64 threads and all the way to 12 threads. Every time application stops after writing 40 GB of the sam file. Any suggestions will be greatly appreciated.

[vasimuddin]

can you provide the stderr log or point to the dataset to reproduce?

[mgujral]

Unfortunately the error file does not contain any useful information. Here are last 15 lines or so.

[0000][PE] # candidate unique pairs for (FF, FR, RF, RR): (8, 314233, 0, 2)
[0000][PE] skip orientation FF as there are not enough pairs
[0000][PE] analyzing insert size distribution for orientation FR...
[0000][PE] (25, 50, 75) percentile: (345, 400, 466)
[0000][PE] low and high boundaries for computing mean and std.dev: (103, 708)
[0000][PE] mean and std.dev: (407.81, 90.62)
[0000][PE] low and high boundaries for proper pairs: (1, 829)
[0000][PE] skip orientation RF as there are not enough pairs
[0000][PE] skip orientation RR as there are not enough pairs
[0000] 3. Calling kt_for - worker_sam
[0000][ M::mem_process_seqs] Processed 664608 reads in 59.830 CPU sec, 1.545 real sec
[0000] read_chunk: 100000000, work_chunk_size: 100000122, nseq: 664664
[0000][ M::kt_pipeline] read 664664 sequences (100000122 bp)...
[0000] Calling mem_process_seqs.., task: 136
[0000] 1. Calling kt_for - worker_bwt
[0000] read_chunk: 100000000, work_chunk_size: 100000271, nseq: 664620
[0000][ M::kt_pipeline] read 664620 sequences (100000271 bp)...
/cm/local/apps/slurm/var/spool/job23023112/slurm_script: line 34: 2532663 Segmentation fault singularity exec /expanse/lustre/projects/scr104/mgujral/singularityContainer/WGS/version_0.0.2/ubuntuWGS.sif bwa-mem2 mem -K 100000000 -Y -t 64 -v 3 -R '@rg\tID:RES07370\tLB:RES07370\tPL:ILLUMINA\tPG:bwa-mem2-2.2.1\tCN:USU\tSM:RES07370\tPU:HY5VTCCXX.1_8' /mnt/hg38/Homo_sapiens_assembly38.fasta /mnt/RES07370/RES07370_1.fq.gz /mnt/RES07370/RES07370_2.fq.gz -o /mnt/RES07370/RES07370_bwa-mem2_128cores64threadsK100M.sam