1
$\begingroup$

Using Nextflow, I need to submit x * y jobs where x is the number of input .bam files and y is the number of genome intervals (e.g., <chrom>:<start>-<end>). i.e., for every .bam file, submit a single job for every defined interval.

I have tried a few variations with no luck (see below).

Ultimate question: For each .bam file, how do I submit a single job for every defined genome interval using NextFlow?

Attempts

Attempt 1 (submit both as individual Channels):

Only submits one .bam for all intervals.

sample_bams = Channel.fromPath(sample_input_path + "*.bam")
intervals = Channel.from(['1:10000-20000', '5:55555-77777'])
PROC(sample_bams, intervals)

Attempt 2 (submit as tuple of mapped Channels):

Submits only 10 jobs and fails because the interval Channel is passed as DataflowBroadcast around DataflowStream[?] object. The PROC was changed to receive a tuple rather than individual arguments.

sample_interval_tuples = Channel.fromPath(sample_input_path + "*.bam")
                         .map { sample_file -> tuple(sample_file, align_to_ref, DRF_jar, Channel.from(['1:10000-20000', '5:55555-77777'])) }

Attempt 3 (submit as tuple of mapped .bams to non-Channel of intervals):

Submits 10 jobs and fails because the intervals are passed as a single list. The PROC was changed to receive a tuple rather than individual arguments.

sample_interval_tuples = Channel.fromPath(sample_input_path + "*.bam")
                         .map { sample_file -> tuple(sample_file, align_to_ref, DRF_jar, ['1:10000-20000', '5:55555-77777']) }

Really appreciate your help!

related query: for DSL v2, we wouldn't include from combined_inputs in the input, yeah?

$\endgroup$

2 Answers 2

3
$\begingroup$

The combine operator can be used to produce the Cartesian product:

sample_bams = Channel.fromPath( './path/to/bams/*.bam' )
intervals = Channel.of( '1:10000-20000', '5:55555-77777' )

sample_bams
    .combine(intervals)
    .set { combined_inputs }


process test {

    echo true

    input:
    tuple path(bam), val(interval) from combined_inputs

    """
    echo -n "${interval} ${bam}"
    """
}

Regarding the related query: Correct. The from and into 'bind' declarations can be omitted

$\endgroup$
2
  • 4
    $\begingroup$ we should crown Steve as King Of Nextflow Answers $\endgroup$
    – Pallie
    Feb 10 at 10:35
  • $\begingroup$ Thanks @Pallie, but I think there's still a way to go before that. Watch this space I suppose. $\endgroup$
    – Steve
    Feb 10 at 12:19
1
$\begingroup$

I think this cheat sheet might help: https://github.com/danrlu/Nextflow_cheatsheet/blob/main/nextflow_cheatsheet.pdf

$\endgroup$
1
  • $\begingroup$ 😍😍 Love the cheat sheet! $\endgroup$ Feb 10 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.