Run nextflow pipeline multiple times depending on length of a list

Question

This question was also asked on Stack Overflow

I have a Nextflow pipeline that runs a BUSCO search on input data, returns the output and then creates a report based on the output. One of my parameters is the lineage dataset. I would like to have my pipeline run multiple times (preferably at the same time) based on how long my list of lineage datasets is. For example, this is my nextflow.config file. The goal is that my pipeline would run twice in this case. If there are n lineages specified I would like it to run n times.

params { 
    inputFilePath           = "$PWD/input_data/test_data.fasta"
    lineage_datasets        = [
        "fungi_odb10",
        "microsporidia_odb10"
        ]
    augustus_species        = "fumigatus"
    mode                    = "geno"
    
    threads                 = 16
    help                    = false
    outdir {
        main                = "$PWD/results"
    intermediate        = "$PWD/busco_outputs"
    }
}

The first part of my busco process looks like this:

  publishDir params.outdir.intermediate, mode: 'copy'

  input:
    path inputFile
    string lineage_dataset
  
  output:
    path 'hap1/*hap1.txt'

The idea is that each of the summary.txt files will be collected inside of my outdir.intermediate folder which is where my createReport process will target. For context, this is my createReport process:

process createReport {

  publishDir params.outdir.main, mode: 'copy'

  input: 
    path 'busco_outputs/*'

  output:
    path 'report.html'

  script:
    """
    source "$launchDir/venv/bin/activate"
    report.py > report.html
    """ 
}

I have tried the following in my workflow process but it does not seem to work:

workflow {
  runBuscoChannel = Channel.fromPath(params.inputFilePath).cross(Channel.of(params.lineage_datasets)) 
  | mix 
  | collect 
  | set { outputs }

  createReport(outputs)
}

When I try to run the pipeline it does not run properly

Does anyone have any ideas of how I could change my file to allow for this? Thanks very much!

Answer 1

I think the answer to your question is the each input repeaters. You can also check Nextflow FAQ for the question on how to iterate over a process or a file.

Not in front of a PC now but you could try something like each lineage_dataset.