Function call with channel fails (seemingly) non-deterministically. #2447
Comments
|
I encounter a similar issue: intermittent errors |
|
Having a self-contained example to replicate the issue would help a lot here. |
|
In my case the problem was caused by defining a function with a default argument value. This is probably related to: #1698 @SamThePsychoticLeprechaun I noticed you have some default argument values in your function definition as well, I suggest you try and remove those to see if this resolves the issue for you? I must say that other people working on the same pipeline have not observed this issue at all, maybe because of a different Java runtime? |
|
I would exclude a side-effect of different Java runtime version, never seen this happening before. |
|
Hi, I ran the case where this occurs with high frequency 30 times with the default argument excluded on the map parameter. I got no error on any of the runs and I got the correct outputs. When I added the default argument back in, I immediately got the same error as described earlier. Thanks for the suggestion @tverbeiren. @pditommaso, I'm not super familiar with Java/Groovy but it looks to me this is a common pattern to use a map for named parameters that may not be needed, and I understand in 2020 your statement on #1698 was that default arguments are not supported. Perhaps this could be reconsidered, as it will contribute towards achieving the goal DSL2 has of enabling better software engineering practices in Nextflow. |
|
We have this too when we tried to implement optional parameters into existing shared code to avoid breaking the API. We also see non-deterministic errors: sometimes the code runs fine, other times it complains: @SamThePsychoticLeprechaun you are correct that putting a The Nextflow documentation states:
This is misleading: its clear that Nextflow script isn't a superset of Groovy. The semantics of function calls in Nextflow workflow script are not the same as Groovy. The following illustrates the point: The above leads to the following output: The first position We've found two workarounds to this. Firstly abandon Nextflow script entirely and write a "standard" Groovy class with static methods: main.nf: lib/MakeChannel.groovy This appears to work more robustly. I do not know how you would call a Nextflow process within this method however. The other workaround is rather hilarious, and I wouldn't recommend it. Write functions with a single This allows you to stick with writing simple Nextflow function definitions but is slightly silly! |
|
This is a problem that has been recently identified. In a nutshell, function overloading i.e. two or more functions with the same name and a different signature is not supported by nextflow. When using a default parameter, is essentially the same as creating two versions of the same function with and without that parameter. Recent versions of nextflow, detect this problem and report a warning message c0b522ab |
|
I think @cjw85 approach it's a good way to work around the issue but I think it's better to "separate" the business logic (make function in this case) from the workflow logic (channels, process, etc) For example, we can create a These functions don't need to be static and we can split the logic into several classes/methods because it's a "pure" groovy class the we instantiate a a more elaborate workflow can be: or for example a workflow who create the list from the params in this case we are calling the |
I wouldn't get hung up on the examples I gave that contructed channels within the functions; that was just the first thing I wrote to illustrate the point that was along the same lines as the OP. Furthermore I think its perfectly valid in more complex cases to want to create a channel from within a library function, say after having run a Separating business logic from workflow logic isn't the goal here, its to have a way to emulate the type of function polymorphism afforded by the Groovy As @pditommaso points out, function overloading isn't supported in the Nextflow codebase: there is a soft assumption when trying to lookup a function that there is only one of the given name. @SamStudio8 did the codedive for us a few weeks back, incidentally uncovering why for the OP the failures were non-deterministic: the function chosen to be called is pulled from a data stucture with no stable ordering, sometimes the function picked will have the correct signature to be called with the required arguments, sometimes not. In the end we worked around the issue by implementing a library function to perform Nextflow function argument validation for us along the lines of the technique I suggested previously was a bit silly; it at least allows us to create reusable library functions with default argument values. For example: where
|
|
Is this issue fixed by #3052 ? |
|
I can't see a test in that PR that implements the pattern used by the OP, seems wise. |
|
This was first addressed by this commit, and later allowing the full support for function overloading via #3052 |
Bug report
Note: reproducibility has been possible, but in my experience hard and non-deterministic, if anyone has any ideas on how to improve on that front please let me know!
Expected behavior and actual behavior
The expected behaviour is, in defining a function such as:
that a call such as:
collect_file_tuples( some_process.out.reads, "processed", "fq", file_idx: 1, sort: true ).set{ processed_reads }will see that function executed on the channel passed in as an argument.
What happens, however, is a seemingly non-deterministic error of "wrong number of arguments" (Nextflow log attached). The runs seem to work and not work with no rhyme or reason - the exact same code will sometimes run and sometimes not run even in the case of the
.nextflowcache having been deleted.I was previously running on Nextflow 21.04.3, and get the same result on 21.10.0.
Steps to reproduce the problem
The occasion of the error showing up is proving more frequent than times it does not in my full codebase, the following replicates everything leading up to the error, but the error appears via this code much less frequently (I only got the error twice in ~20 runs, each time the same code, and the nextflow cache deleted):
test_func.nf
test.nf
process.nf
process some_long_process { label 'in_container' input: tuple val(id), file(read), file(paired_read) output: path "${id}.log", emit: log tuple val(id), file("${id}.txt"), emit: answer executor = 'lsf' cpus = 1 queue = '<some queue>' clusterOptions = 'span[hosts=1]' time '10m' """ sleep 3 echo "this is a log" >> ${id}.log echo $read >> ${id}.txt echo $paired_read >> ${id}.txt echo "$id + 3 = 4" >> ${id}.txt """ }test.nfconfig
singularity { enabled = true autoMounts = true } process { withLabel: in_container { container = "file://${projectDir}/a-container-with-bash.simg" } }The only concrete thing I can say is that whenever I supply
NXF_DEBUG=3to my fuller codebase - which fails almost every time otherwise - it will succeed nearly every time.I don't know that the container, or having the process & function each defined in a separate file is necessary, but I have only obtained the error with in this form. The rate on this test case of the error occurring is much lower though, and I tested ~20 runs (fresh cache each time) with and without using the container, I got two failures and both only in the case of running the process in the container.
Program output
stdout
.nextflow.log
Environment
The text was updated successfully, but these errors were encountered: