samtools collate fails when /tmp runs out of disk space

[benjschiller]

Hi,

We're seeing the convertCRAMtoFASTQ task fail for large CRAMs (> 60 GB). It seems that that samtools collate command is writing intermediate files to /tmp, which is outside the /cromwell_root volume (EDIT: we're running this in Terra using the version from Dockstore) in the container. I noticed this behavior elsewhere, and had to either set TMPDIR or provide an explicit prefix with a path inside the desired volume (as described here https://www.htslib.org/doc/samtools-collate.html)

Here is an example log message before the container is reclaimed (usually repeated several times):

samtools collate: Couldn't write to intermediate file "/tmp/collatef.0004.bam": No such file or directory

Relevant code:

vg_wdl/tasks/bioinfo_utils.wdl

Lines 528 to 533 in 00167b2

	if [ ~{in_paired_reads} == true ]
	then
	samtools collate -@ ~{half_cores} --reference ~{in_ref_file} -Ouf ~{in_cram_file} | samtools fastq -@ ~{half_cores} -1 reads.R1.fastq.gz -2 reads.R2.fastq.gz -0 reads.o.fq.gz -s reads.s.fq.gz -c 1 -N -
	else
	samtools fastq -@ ~{in_cores} -o reads.R1.fastq.gz -c 1 --reference ~{in_ref_file} ~{in_cram_file}
	fi