[BUG] Publish processes are not cached when resuming

[cflerin]

Describe the bug
When running the pipeline for a second time with the -resume option, the publish processes always run and are not cached.

To Reproduce
Steps to reproduce the behavior:

1. Run one of the test workflows:

nextflow pull vib-singlecell-nf/vsn-pipelines -r v0.21.0

nextflow run vib-singlecell-nf/vsn-pipelines -profile scenic,test__scenic,singularity -entry scenic -r v0.21.0

2. Re-run using resume:

nextflow run vib-singlecell-nf/vsn-pipelines -profile scenic,test__scenic,singularity -entry scenic -r v0.21.0 -resume

3. Publish steps are not cached:

$ nextflow run vib-singlecell-nf/vsn-pipelines -profile scenic,test__scenic,singularity -entry scenic -r v0.21.0 -resume
N E X T F L O W  ~  version 20.04.1
Launching `vib-singlecell-nf/vsn-pipelines` [reverent_faggin] - revision: 3cc43ce065 [v0.21.0]
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [b6577d79a5]
WARN: DSL 2 IS AN EXPERIMENTAL FEATURE UNDER DEVELOPMENT -- SYNTAX MAY CHANGE IN FUTURE RELEASE
executor >  local (2)
[ff/3b0342] process > scenic:SCENIC:ARBORETO_WITH_MULTIPROCESSING (1) [100%] 1 of 1, cached: 1 ✔
[3a/5b0e9f] process > scenic:SCENIC:CISTARGET__MOTIF (1)              [100%] 1 of 1, cached: 1 ✔
[58/e8f22b] process > scenic:SCENIC:AUCELL__MOTIF (1)                 [100%] 1 of 1, cached: 1 ✔
[74/45ef7f] process > scenic:SCENIC:VISUALIZE (1)                     [100%] 1 of 1, cached: 1 ✔
[64/f374d2] process > scenic:SCENIC:PUBLISH_LOOM (1)                  [100%] 1 of 1, cached: 1 ✔
[81/6c7b8d] process > scenic:PUBLISH_SCENIC:COMPRESS_HDF5 (1)         [100%] 1 of 1, cached: 1 ✔
[8c/19d3a4] process > scenic:PUBLISH_SCENIC:SC__PUBLISH (1)           [100%] 1 of 1 ✔
[26/c75c86] process > scenic:PUBLISH_SCENIC:SC__PUBLISH_PROXY (1)     [100%] 1 of 1 ✔

Expected behavior
All steps should be cached on resume

Please complete the following information:

• OS: CentOS Linux release 7.8.2003 (Core)
• Nextflow Version: 20.04.1
• vsn-pipelines Version: v0.21.0

Additional context
N/A

[cflerin]

After some digging, this is caused by the getOutputFileName function:

vsn-pipelines/src/utils/processes/utils.nf

Lines 374 to 406 in b5167f5

	def getOutputFileName(params, tag, f, fileOutputSuffix, isParameterExplorationModeOn, stashedParams) {
	// - In parameter exploration mode,
	// - returns
	// - <tag>.<fileOutputSuffix>.<stashedParams>.<f.extension> if fileOutputSuffix is null
	// - <tag>.<stashedParams (joined)>.<f.extension> otherwise
	// - If not in parameter exporation mode AND
	// - If pipelineOutputSuffix exists AND
	// - If it is set to 'none'
	// - returns:
	// - <tag>.<f.extension>
	// - <tag>.<pipelineOutputSuffix>.<f.extension>
	// - If fileOutputSuffix is null
	// - returns:
	// - <f.baseName>.<f.extension>
	// - In all other cases
	// - returns:
	// - <f.baseName>.<fileOutputSuffix>.<f.extension>
	if(isParameterExplorationModeOn && !isParamNull(stashedParams))
	return isParamNull(fileOutputSuffix) ?
	"${tag}.${stashedParams.findAll { it != 'NULL' }.join('_')}.${f.extension}" :
	"${tag}.${fileOutputSuffix}.${stashedParams.findAll { it != 'NULL' }.join('_')}.${f.extension}"
	if(params.utils.containsKey("publish")
	&& params.utils.publish.containsKey("pipelineOutputSuffix")) {
	if(params.utils.publish.pipelineOutputSuffix == 'none')
	return "${tag}.${f.extension}"
	if(params.utils.publish.pipelineOutputSuffix.length() == 0)
	throw new Exception("VSN ERROR: The parameter 'params.utils.publish.outputFileSuffix' cannot be empty. If you don't want to add a suffix to the final output, please set this param to 'none'.")
	return params.utils.publish.pipelineOutputSuffix
	}
	if(isParamNull(fileOutputSuffix))
	return "${f.baseName}.${f.extension}"
	return "${tag}.${fileOutputSuffix}.${f.extension}"
	}

since replacing the output of this function with a fixed string in the publish functions results in proper and consistent caching of these processes on resume. I think the problem could be due to the way getOutputFileName is run inside of the publish process, creating inputs that are dynamic, and forcing the process to be re-executed every time.

As a side note, this is a major issue for disk space in larger (e.g. mapping) projects. I ran into an issue where pbs jobs were failing to be submitted late in the atac_preprocess workflow. Each time I re-ran with -resume all of the upstream published files (fastq, bam files, etc.) were copied again within work/, leaving 100s of GBs of extra data.

[dweemx]

I understand, this is quite annoying!
Calling getOutputFileName seems to me deterministic.
Also if getOutputFileName is the root of the issue, I would also expect scenic:PUBLISH_SCENIC:COMPRESS_HDF5 not to resume (since it is also calling this function) but it does.
Very intriguing this bug

[dweemx]

I just noticed that the NXF processes that do not resume are the ones using the outputFileName variable in their publishDir directive.

[cflerin]

I just noticed that the NXF processes that do not resume are the ones using the outputFileName variable in their publishDir directive.

Nice! That's an interesting point there. I hadn't tested COMPRESS_HDF5.

[cflerin]

COMPRESS_HDF5 still appears to cache properly when changing the publishDir output to use outputFileName.