Add option to remove scratch dir contents at process exit

[amacbride]

It would be nice to support ephemeral scratch directories, where the scratch dir is deleted after the process exits (and finishes copying its declared output files.)

[pditommaso]

See also #165

[pditommaso]

I've uploaded a new snapshot that automatically cleanup the scratch dir. You may want to give a try adding the following var in your env and launch NF as usual.

NXF_VER=0.22.3-SNAPSHOT

[pditommaso]

Included in version 0.22.3

[amacbride]

FYI, I tried the production 0.22.3 release (with the ram-disk option), and it seems to have left a bunch of directories and their contents behind in /run/shm :

sierra [917] du -sh *
1.3G    nxf.0VmrHd835r
1.3G    nxf.wvN70Wrquv
1.3G    nxf.WYF7Fijmk0
1.3G    nxf.zcon8xcC6u

marin [923] du -sh *
1.3G    nxf.80zCsi7N3d
7.6M    nxf.94481dsN44
3.9M    nxf.d6MCzEEcFI
1.3G    nxf.hI5W9clXkY
1.3G    nxf.la254LYViO
2.9M    nxf.NiFSd1pJeL
5.4M    nxf.P9aCdZEcdp
1.3G    nxf.qOcWr3AnDJ
5.7M    nxf.U3d9M9IsLN
1.3G    nxf.vxkvdjbz4b
4.2M    nxf.wkPik1mNYh
6.2M    nxf.wxpotnlGL2
1.3G    nxf.ysAHgFl46k
1.3G    nxf.zoymao85ss

lake [899] du -sh *
1.3G    nxf.0fQBz5axzP
1.3G    nxf.8ZxECMDmnF
4.4M    nxf.D9vCQF3xQt
5.7M    nxf.Dpc2jR3mPu
3.7M    nxf.gEhwBZVqwQ
4.8M    nxf.h11VFPp4Rg
7.5M    nxf.H2YhluOont
9.4M    nxf.JgLbh0KWG4
3.3M    nxf.KNS9m6Myzz
1.3G    nxf.OixCn3gwdT
1.3G    nxf.QPGUe3emHa
4.5M    nxf.wDHtvV1jpA
1.3G    nxf.wuYlPKH9Hc
4.5M    nxf.WxVuVBUv9W

I'll try /tmp and see if that works differently.

[pditommaso]

Double checked and it works as expected. Are you sure you are using the latest version (0.22.3) and that that dirs were not left by a previous run?

Check the file .command.run generated by NF, you should find a function like this:

on_exit() {
  exit_status=${ret:=$?}
  printf $exit_status > /home/pditommaso/projects/rnatoy/work/ff/836e306cde436de39a0e2e5e4c8afc/.exitcode
  set +u
  [[ "$COUT" ]] && rm -f "$COUT" || true
  [[ "$CERR" ]] && rm -f "$CERR" || true
  (sudo -n true && sudo rm -rf "$NXF_SCRATCH" || rm -rf "$NXF_SCRATCH")&>/dev/null || true
  exit $exit_status
}

the command before the exit is the one supposed to cleanup the scratch folder.

[amacbride]

I'm seeing the same behavior with scratch = "/tmp" -- the nxf.* directories and their contents are left behind. I'll poke around today and see if I can isolate the problem.

I'm running 0.22.3:

N E X T F L O W  ~  version 0.22.3 
Launching `../clia-pipeline/nextflow/modular.nf` [stoic_feynman] - revision: e565aba362

System config is Ubuntu Trusty (14.04), with Docker 1.12.1, and SLURM 2.6.5.

The only thing that jumped out at me was that the scratch volume is exposed in Docker, as is the /tmp directory itself. Would that be a possibility?

-v /tmp/nxf.O2YcWkOUBF:/tmp/nxf.O2YcWkOUBF -w /tmp/nxf.O2YcWkOUBF 
[...]
-v /tmp:/tmp

[pditommaso]

The only thing that jumped out at me was that the scratch volume is exposed in Docker, as is the /tmp directory itself. Would that be a possibility?

That's fine. The container need to work in that dir.

You can debug the problem changing into a task work dir, then running bash -x .command.run. It will dump the command executed.