1
$\begingroup$

Simple question but I am stumped. (I am still learning Nextflow)

I have a process which takes as input a channel of files and processes each file separately.

There is a second process which needs to wait until all files are complete, so it can merge the outputs into a single collated file.

How can I do this? I have tried subscribing to "onComplete" event to no avail as I cannot specify a process as the function to trigger.


files = Channel.from params.the_files_list

process process_files {

    input:
    val file_path from files

    output:
    stdout result

    """
    python ./process_one_file.py $file_path
    """
}

process merge_results {
    
    input:
    ????

    output:
    ????

    script:
    """
    echo "How to trigger this process once, after all individual files are processed?"
    """
        
}

??? Want to do something like this
result.subscribe onComplete: merge_results

```
$\endgroup$
1
  • 1
    $\begingroup$ I've run into the same problem but I'm interested in trying this method. Did it work for you? $\endgroup$
    – daray
    Commented May 30, 2022 at 21:59

1 Answer 1

2
$\begingroup$

So from this page... https://groups.google.com/g/nextflow/c/jt77_-uApMs

It is the "scatter-gather" pattern. I think the answer is to collect() the channel as follows - interested to hear if there is a better way

files = Channel.from params.the_files_list

process process_files {

    input:
    val file_path from files

    output:
    stdout result

    """
    python ./process_one_file.py $file_path
    """
}

process merge_results {
    
    input:
    val go from result.collect()

    script:
    """
    echo "Triggered once after all files complete!"
    """
        
}
$\endgroup$
1
  • 3
    $\begingroup$ .collect() is the correct way to do this $\endgroup$
    – Pallie
    Commented Apr 7, 2021 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.