3
$\begingroup$

I have a process that emits BAM files with a key containing the sample name:

process source{
  output:
    tuple val(sample), path("sample.bam") into ch

  [...]

}

1st question: I have several files per sample, let's assume 2 for now. I want to merge them together with samtools merge. I can regroup that channel with groupTuple(), but how can I recover them in the next process? Considering they all have the same name:

grouped_ch = ch.groupTuple(by: 0, size: 2)

process merge{
  input:
    grouped_ch

  shell:
  '''
  samtool merge sample.bam sample.bam > !{sample}.bam
  '''
}

Is there perhaps a way in the input: block to say that I expect a list, and name the list components?

2nd question: In practice, I have a variable number of "sample.bam" files for each sample value, does that change the way to define the input:? It also appears I will need to use the built-in groupKey(), is there documentation available? I have read this Github issue, but it would help if a general description was available.

$\endgroup$

1 Answer 1

3
$\begingroup$

My preference is to avoid working with files with the same name as much as possible. However, sometimes this might be unavoidable. Fortunately, file name collisions can be avoided when working with multiple input files by using the * and ? wildcards:

When a target file name is defined in the input parameter and a collection of files is received by the process, the file name will be appended by a numerical suffix representing its ordinal position in the list.

The target input file name can contain the * and ? wildcards, that can be used to control the name of staged files.

Note that there is also the stageAs path option (see input of type 'path'), which can also take a name pattern (using the above wildcards) to let you use a variable in your script. For example:

process samtools_merge {

    conda 'samtools=1.15.1'
    debug true
    stageInMode 'rellink'

    input:
    tuple val(sample), path(bam_files, stageAs: 'mysample*.bam')

    output:
    tuple val(sample), path("${sample}.bam")

    """
    samtools merge "${sample}.bam" ${bam_files}
    ls -nl
    """
}

workflow {

    Channel.fromPath( ['./path/to/files/*.bam', './path/to/more_files/*.bam'] ) \
        | map { bam -> tuple( bam.baseName, bam ) } \
        | groupTuple(by: 0, size: 2) \
        | samtools_merge
}

Results:

N E X T F L O W  ~  version 22.04.0
Launching `script.nf` [elated_montalcini] DSL2 - revision: 1427d64a1d
executor >  local (3)
[83/25654c] process > samtools_merge (3) [100%] 3 of 3 ✔
total 184
lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -> ../../../path/to/files/sample1.bam
lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -> ../../../path/to/more_files/sample1.bam
-rw-r--r-- 1 1000 985 187911 May 25 12:18 sample1.bam

total 128
lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -> ../../../path/to/files/sample2.bam
lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -> ../../../path/to/more_files/sample2.bam
-rw-r--r-- 1 1000 985 129947 May 25 12:18 sample2.bam

total 196
lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -> ../../../path/to/files/sample3.bam
lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -> ../../../path/to/more_files/sample3.bam
-rw-r--r-- 1 1000 985 197225 May 25 12:18 sample3.bam

If each group contains a variable number of files, you can keep the same input declaration (as above) but you will want use a special groupKey object so that the collected values can be streamed as soon as possible. It's not well documented, but here's a simple example that sorts the input BAM files prior to merging. Once all of the 'samtools_sort' jobs are finished for a particular sample, that sample can proceed to merging without us having to wait for all the other 'samtools_sort' jobs to be finished (which would otherwise be the case if the 'size' parameter had not been specified):

process samtools_sort {

    tag { bam.baseName }

    conda 'samtools=1.15.1'

    input:
    tuple val(sample), path(bam)

    output:
    tuple val(sample), path("${bam.baseName}.sorted.bam")

    """
    samtools sort -o "${bam.baseName}.sorted.bam" "${bam}"
    """
}

process samtools_merge {

    tag { sample }

    conda 'samtools=1.15.1'

    input:
    tuple val(sample), path(bam_files, stageAs: 'mysample*.bam')

    output:
    tuple val(sample), path("${sample}.bam")

    """
    samtools merge "${sample}.bam" ${bam_files}
    """
}
workflow {

    Channel.fromPath( params.bam_files ) \
        | map { bam -> tuple( bam.baseName, bam ) } \
        | groupTuple() \
        | map { sample, files -> tuple( groupKey(sample, files.size()), files ) } \
        | transpose() \
        | set { bam_files }

    samtools_sort( bam_files ) \
        | groupTuple() \
        | samtools_merge \
        | view()
}

Results:

$ nextflow run script.nf -ansi-log false --bam_files './path/to/*/*.bam'
N E X T F L O W  ~  version 22.04.0
Launching `script.nf` [cheesy_swanson] DSL2 - revision: da1a82a059
[04/643129] Submitted process > samtools_sort (sample3)
[1a/93a939] Submitted process > samtools_sort (sample1)
[93/fde0a0] Submitted process > samtools_sort (sample3)
[db/208bac] Submitted process > samtools_sort (sample3)
[8f/f2a231] Submitted process > samtools_sort (sample2)
[f3/8f0f9f] Submitted process > samtools_sort (sample2)
[bb/cca23e] Submitted process > samtools_merge (sample1)
[sample1, /home/steve/testing/work/bb/cca23e016a641b201b7e42f9db74ef/sample1.bam]
[ff/bd3a2a] Submitted process > samtools_merge (sample2)
[sample2, /home/steve/testing/work/ff/bd3a2ac83a0691d2b80b04ef97b755/sample2.bam]
[94/a4f8eb] Submitted process > samtools_merge (sample3)
[sample3, /home/steve/testing/work/94/a4f8eb483602e9335b28f5d40058c8/sample3.bam]

Note that the val specified as input to 'samtools_sort' is an object of class nextflow.extension.GroupKey. In the example above, the input to 'samtools_merge' is also a groupKey - but it doesn't need to be. If you need a string value for whatever reason, you can just call '.toString()' to get it back:

    ...

    samtools_sort( bam_files ) \
        | groupTuple() \
        | map { sample_key, files -> tuple( sample_key.toString(), files ) } \
        | samtools_merge \
        | view()
$\endgroup$
2
  • 3
    $\begingroup$ It's awesome how much effort you put into these nf answers! $\endgroup$
    – user3051
    Commented May 25, 2022 at 6:51
  • 4
    $\begingroup$ Thank you so much for this very comprehensive answer! I had totally missed that section of the input documentation. And as ATpoint notes, this is the second time you help me on a Nextflow question, this has been extremely helpful! $\endgroup$
    – Alexlok
    Commented May 25, 2022 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.