Is channel output from multiMap guaranteed to be in the same order it comes in?

[mahesh-panchal]

Is output from multiMap guaranteed to be in the same order it comes in?
Specifically for process input, can I rely on multiMap to keep data in the same order but in different channels?
Toy example:

#! /usr/bin/env nextflow

nextflow.enable.dsl = 2

workflow {

    input_ch = Channel.of (
        [ id: 'test1', prop1: 'test1_od', prop2: 'test1_pad' ], 
        [ id: 'test2', prop1: 'test2_od', prop2: 'test2_pad' ]
    )

    def mmc = multiMapCriteria {
        sample: it.id
        attributeone: it.prop1
        attributetwo: it.prop2
    }

    FOO ( input_ch.multiMap( mmc ) ) // Does this guarantee test1 and test2 won't be mixed?
}

process FOO {

    echo true

    input:
    val samp
    val x
    val y

    script:
    """
    printf "%s\\t%s\\t%s\\n" $samp $x $y
    """
}

I want to apply it to a more complex scenario to keep the code more readable rather than having a very large tuple with optional inputs.

[bentsherman]

I wouldn't be surprised if multiMap preserves the order, but I don't recommend structuring the channels this way. Values that should be tied together, like (samp, x, y) in this example, should be managed as a tuple, even if that means that you have long tuples. Even if multiMap happens to preserve order, who knows what other channel logic might come into play and break that ordering.

As for optional inputs, there are some efforts to improve Nextflow support for optionals (#1694, #2893). I think that would be a better direction to pursue.

[bentsherman]

Regarding multiple output channels, I think the best practice here is to include some metadata (e.g. sample id) in each channel so that different channels can be joined back together if needed. I believe the nf-core pipelines follow this pattern. Similarly, if you were to output a single channel with a long tuple, you can always split off pieces of that tuple into new channels, as long as you also keep the metadata.

Regarding multiple input channels, the main use case is to process all combinations using each or value channels. Having multiple queue channels without each doesn't really make sense unless you actually want the channels to be combined randomly, for which I haven't seen an example yet. If you want multiple channels to be matched properly, you have to use join or cross to create a single tuple channel first. So you do need multiple input channels. but it might be good if Nextflow gave a warning when multiple queue channels are declared without each, which is equivalent to using the merge operator.

[mahesh-panchal]

I get the point of including meta to join samples, but the point I'm making is about using existing code in a way that wasn't commonly intended. Let's take the example of read alignment. Often the base use is to take two files of reads, and align them against a reference. So then the common implementation is then to pass the reads as a tuple with meta data, and the reference file as a value channel. This is fine for most use cases. However, then someone comes along and wants to say implement a contamination assessment workflow. Here the reads from multiple samples are intended to be aligned against several references. Now if one were to implement the alignment module, one would typically combine all the references with all the samples, and reference would be part of the same input tuple. However, if I want to make use of an existing module, that implements path as a separate value channel, I either need to make a copy, modify the input, and keep it up to date, or in it's current state, use multiMap (as above) and not maintain any patches. It's about facilitating reuse of existing modules/code.

[bentsherman]

Okay, this is an interesting issue which I have encountered before. I wrote up a minimal example at the bottom, hopefully I understand the use case you're describing.

The fundamental issue I see here is that you can't use each on a tuple, and I couldn't find a way to pass references as a value channel. Therefore you have to do the combine yourself, and then use multiMap if you want to have separate input channels.

I think your original question has been answered, that is, multiMap does preserve order, as do most if not all channel operators (as far as I can tell). However, I see a deeper issue here with the Nextflow language. If references wasn't a tuple then it wouldn't have this problem. It seems like the original paradigm of each and multiple inputs did not account for how much tuples would be used to track metadata.

Ultimately I think we should try to make tuples easier to use in cases like this one.

process analyze {
    // need to either do `each` on references channel ...
    input:
    tuple val(sample_id), val(sample_file)
    tuple val(reference_id), val(reference_file)

    script:
    """
    echo 'sample: ${sample_file}, reference: ${reference_file}'
    """
}

workflow {
    samples = Channel.of(1 .. 10) | map { i -> ["sample-${i}", "sample-${i}.fastq"] }
    references = Channel.of(1 .. 3) | map { i -> ["reference-${i}", "reference-${i}.idx"] }

    // ... or provide references as value channel
    analyze(samples, references)
}

[mahesh-panchal]

How does this example work though? You're doing a merge, or are you suggesting that this syntax should ideally perform a combine? I think this kind of implicit behaviour leads to comprehension issues. Novice users of Nextflow already have this kind of learning issue when it comes to spawning tasks combining a queue channel and value channel (many don't realise there is a difference; and a secondary problem is learners often confuse the channels themselves with the data flowing through them). This is why I think it's better that processes should ideally only take a single data entity and are forced to explicitly combine data with operators so they know what the process is operating on.

But aside from that, the original question, has been answered; that multiMap does preserve order.

[bentsherman]

It doesn't actually work, and the comments show different ways that it could work with the relevant changes to Nextflow.

I agree that the clearest solution (and the one that actually works) is to explicitly combine the two channels into a single input. Input repeaters based on each and value channels are essentially syntax sugar for the combine operator. However I brought up these alternatives because you said that single combined input would prevent a module from being reused if it structured the inputs as multiple channels.

I think this workflow example shows that you wouldn't be able to structure such a module that way anyway, because you need to track metadata on each channel and you can't use each with a tuple, so you're only option is to use combine anyway.