Nextflow: ordering the execution of processes; outputs of process A are inputs of process B

Question

I need to build a pipeline where 2nd process starts after completion of the 1st process and the 3rd after 2nd. As I understand nextflow's processes run parallel. But I need it to run in a order 1st>2nd>3rd processes and so on. Because each process runs on an array and produce multiple outputs which are used by the next. If they run paralleled, the 2nd process may look for input 20 whereas the 1st process produced output 10 and then the process stops, because the command in the 2nd process does not find the input. Is there any way to accomplish this. I tried as follows:

process A{
input:
tuple val(some_name), path(some path) from some_channel
output:
tuple val(some val), path(some path) into A_some_ch
tuple va(some other va), path(some other path) into A_some_other_ch
val true into done_ch
script:
"""
some command out some files
some other command out some other files
"""
}
process B {
input:
val flag from done_ch
val (some val), path(some path) from A_some_ch
val (some other val), path(some other path) from A_some_other_ch
output:
some val and path
script:
"""
some commands out some files
"""
}

I got the idea using of flag from here: https://github.com/nextflow-io/patterns/blob/master/docs/mock-dependency.adoc

But if I use the flag, I got:

Operation 'set' does not allow more than one target name

Does it clarify the issue? Any Help? Best, Zillur

Answer 1

Nextflow uses channels (asynchronous FIFO queues) to communicate between processes. The interaction between these processes, and ultimately the pipeline execution flow itself, is implicitly defined by these input and output declarations.

The issue with your code, I think, is the use of multiple input channels. Consider the following example that uses only a single queue channel. To avoid declaring the inputs again in process A's output declaration, we can output a key to join on as the first element in the output tuple. In the case below, this is simply the 'some_name' variable. We can then join up the two channels using the join operator as follows:

Channel
    .fromFilePairs( './data/*.txt', size:1 )
    .map { group_key, files -> tuple( group_key, *files ) }
    .into { input_ch_A; input_ch_B }


process A {

    input:
    tuple val(some_name), path(some_path) from input_ch_A

    output:
    tuple val(some_name), val("procA_${some_name}"), path("procA_${some_name}") \
        into procA_results

    """
    touch "procA_${some_name}"
    """
}

process B {

    input:
    tuple val(some_name), path(some_path), val(some_other_val), path(some_other_path) \
        from input_ch_B.join(procA_results)

    """
    true
    """
}
```