Nextflow: how to create many-to-many tuple for process

Question

Using Nextflow, I need to submit x * y jobs where x is the number of input .bam files and y is the number of genome intervals (e.g., <chrom>:<start>-<end>). i.e., for every .bam file, submit a single job for every defined interval.

I have tried a few variations with no luck (see below).

Ultimate question: For each .bam file, how do I submit a single job for every defined genome interval using NextFlow?

Attempts

Attempt 1 (submit both as individual Channels):

Only submits one .bam for all intervals.

sample_bams = Channel.fromPath(sample_input_path + "*.bam")
intervals = Channel.from(['1:10000-20000', '5:55555-77777'])
PROC(sample_bams, intervals)

Attempt 2 (submit as tuple of mapped Channels):

Submits only 10 jobs and fails because the interval Channel is passed as DataflowBroadcast around DataflowStream[?] object. The PROC was changed to receive a tuple rather than individual arguments.

sample_interval_tuples = Channel.fromPath(sample_input_path + "*.bam")
                         .map { sample_file -> tuple(sample_file, align_to_ref, DRF_jar, Channel.from(['1:10000-20000', '5:55555-77777'])) }

Attempt 3 (submit as tuple of mapped .bams to non-Channel of intervals):

Submits 10 jobs and fails because the intervals are passed as a single list. The PROC was changed to receive a tuple rather than individual arguments.

sample_interval_tuples = Channel.fromPath(sample_input_path + "*.bam")
                         .map { sample_file -> tuple(sample_file, align_to_ref, DRF_jar, ['1:10000-20000', '5:55555-77777']) }

Really appreciate your help!

related query: for DSL v2, we wouldn't include from combined_inputs in the input, yeah?

Answer 1

The combine operator can be used to produce the Cartesian product:

sample_bams = Channel.fromPath( './path/to/bams/*.bam' )
intervals = Channel.of( '1:10000-20000', '5:55555-77777' )

sample_bams
    .combine(intervals)
    .set { combined_inputs }


process test {

    echo true

    input:
    tuple path(bam), val(interval) from combined_inputs

    """
    echo -n "${interval} ${bam}"
    """
}

Regarding the related query: Correct. The from and into 'bind' declarations can be omitted

Answer 2

I think this cheat sheet might help: https://github.com/danrlu/Nextflow_cheatsheet/blob/main/nextflow_cheatsheet.pdf