0
$\begingroup$

Simple question but I am stumped. (I am still learning Nextflow)

I have a process which takes as input a channel of files and processes each file separately.

There is a second process which needs to wait until all files are complete, so it can merge the outputs into a single collated file.

How can I do this? I have tried subscribing to "onComplete" event to no avail as I cannot specify a process as the function to trigger.


files = Channel.from params.the_files_list

process process_files {

    input:
    val file_path from files

    output:
    stdout result

    """
    python ./process_one_file.py $file_path
    """
}

process merge_results {
    
    input:
    ????

    output:
    ????

    script:
    """
    echo "How to trigger this process once, after all individual files are processed?"
    """
        
}

??? Want to do something like this
result.subscribe onComplete: merge_results

```
$\endgroup$
1
$\begingroup$

So from this page... https://groups.google.com/g/nextflow/c/jt77_-uApMs

It is the "scatter-gather" pattern. I think the answer is to collect() the channel as follows - interested to hear if there is a better way

files = Channel.from params.the_files_list

process process_files {

    input:
    val file_path from files

    output:
    stdout result

    """
    python ./process_one_file.py $file_path
    """
}

process merge_results {
    
    input:
    val go from result.collect()

    script:
    """
    echo "Triggered once after all files complete!"
    """
        
}
$\endgroup$
1
  • 2
    $\begingroup$ .collect() is the correct way to do this $\endgroup$
    – Pallie
    Apr 7 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.