Singularity rusn out of space in /tmp

[manoloff]

Hello!

We find ourselves in a position to want to use rnaseq pipeline on a cluster, where neither docker or singularity are installed with sudo-privilages. A non sudo singularity is set up for this purpose. However when we run the pipeline

"nextflow run nf-core/rnaseq/ -profile test,singularity"

we get the message

mkdir /tmp/rootfs-527978366/root/dev: no space left on device

during RNASEQ:PREPARE_GENOME:GUNZIP_ADDITIONAL_FASTA. This seems to be a known issue for singularity (https://singularity.lbl.gov/faq#no-space-left-on-device) and is amplified by quite small /tmp/ partition on our cluster. The proposed solutions with exporting a different path in SINGULARITY_CACHEDIR or TMPDIR do not solve the issue, as the pipeline keeps trying to create the file in /tmp and not in the directory speicifed in the abovementioned directories.

We're using nextflow version: nextflow version 21.02.0-edge.5516
and singularity 3.7.0+90-g3082f0bd2

Is there another variable in nextflow that handles this? Do you have an idea why setting SINGULARITY_CACHEDIR still doesn't change where temporary files are managed?

This seems to be a similar issue to #115 but on another part of the pipeline, with the executed command being
"gunzip -f gfp.fa.gz
echo $(gunzip --version 2>&1) | sed 's/^.(gzip) //; s/ Copyright.$//' > gunzip.version.txt"

I would appreciate any help on how to point /scratch/ to be the default /tmp/ directory for the execution of the nextflow pipeline.

[drpatelh]

Hi @manoloff ! Apologies for the late response. Yes, this sort of thing can be quite tricky to get right but once it is it generally works out of the box. Have you looked at the docs here?

You could also maybe try to create a very small config file called custom.config like below:

env {
    TMPDIR=/scratch/
    SINGULARITY_CACHEDIR=/scratch/
}

and pass it to the pipeline with -c custom.config on the command-line. The difference here is that those variables will be exported before any process is executed with Nextflow.

Also worth noting is that SINGULARITY_CACHEDIR shouldn't really be necessary anymore because this pipeline will be pulling Singularity containers directly without the need to convert them from Docker images, and hence bypassing the need to put anything in a cache dir.

If things still aren't working then please free to come and find us on the nf-core Slack workspace on the #rnaseq channel. You will be able to get help much quicker response in real-time!

[manoloff]

Unfortunatelly I couldn't pin down exactly what causes the behaviour. It appears that the needed environmental variable is SINGULARITY_TMPDIR, but exporting it in the bash file, submitted to SLURM, or in a nextflow config file didn't help. As a workaround I could get it to set a different variable by wrapping singularity in a bash file in the bin and exporting it within. By renaming the singularity in my PATH to 'singularity.old' and creating a singularity file with the following lines:

#!/bin/bash
export SINGULARITY_TMPDIR='/scratch'
env
~/.local/bin/singularity.old "$@"

I could see that the temp directory finally changed. This could be due to faulty cluster setting as well, as it couldn't be reproduced by other users, so in case you're having a similar issue, try a wrapper.

[asalhab]

Hi @manoloff
I am facing the same problem as yours (except that I installed singularity 6.3 using conda). I tried your solution by renaming the singulairty excutable in my PATH into 'singularity.old' and created an executable file 'singularity' in the PATH where I added the following lines
#!/bin/bash
export SINGULARITY_TMPDIR='/scratch/users/abdosa/tmp'
env
/home/abdosa/miniconda2/envs/singularity/bin/singularity.old "$@"

unfortunately It didn't solve the issue. I got:
panic: Unrecognised executable: /home/abdosa/miniconda2/envs/singularity/bin/singularity.old

did I miss any part of your solution? did you add any env parameters to a custom config file in addition (-c option)?