My preference is to avoid working with files with the same name as much as possible. However, sometimes this might be unavoidable. Fortunately, file name collisions can be avoided when working with multiple input files by using the * and ? wildcards:
When a target file name is defined in the input parameter and a collection of files is received by the process, the file name will be appended by a numerical suffix representing its ordinal position in the list.
The target input file name can contain the * and ? wildcards, that can be used to control the name of staged files.
Note that there is also the stageAs path option (see input of type 'path'), which can also take a name pattern (using the above wildcards) to let you use a variable in your script. For example:
process samtools_merge {
conda 'samtools=1.15.1'
debug true
stageInMode 'rellink'
input:
tuple val(sample), path(bam_files, stageAs: 'mysample*.bam')
output:
tuple val(sample), path("${sample}.bam")
"""
samtools merge "${sample}.bam" ${bam_files}
ls -nl
"""
}
workflow {
Channel.fromPath( ['./path/to/files/*.bam', './path/to/more_files/*.bam'] ) \
| map { bam -> tuple( bam.baseName, bam ) } \
| groupTuple(by: 0, size: 2) \
| samtools_merge
}
Results:
N E X T F L O W ~ version 22.04.0
Launching `script.nf` [elated_montalcini] DSL2 - revision: 1427d64a1d
executor > local (3)
[83/25654c] process > samtools_merge (3) [100%] 3 of 3 ✔
total 184
lrwxrwxrwx 1 1000 985 34 May 25 12:18 mysample1.bam -> ../../../path/to/files/sample1.bam
lrwxrwxrwx 1 1000 985 39 May 25 12:18 mysample2.bam -> ../../../path/to/more_files/sample1.bam
-rw-r--r-- 1 1000 985 187911 May 25 12:18 sample1.bam
total 128
lrwxrwxrwx 1 1000 985 34 May 25 12:18 mysample1.bam -> ../../../path/to/files/sample2.bam
lrwxrwxrwx 1 1000 985 39 May 25 12:18 mysample2.bam -> ../../../path/to/more_files/sample2.bam
-rw-r--r-- 1 1000 985 129947 May 25 12:18 sample2.bam
total 196
lrwxrwxrwx 1 1000 985 34 May 25 12:18 mysample1.bam -> ../../../path/to/files/sample3.bam
lrwxrwxrwx 1 1000 985 39 May 25 12:18 mysample2.bam -> ../../../path/to/more_files/sample3.bam
-rw-r--r-- 1 1000 985 197225 May 25 12:18 sample3.bam
If each group contains a variable number of files, you can keep the same input declaration (as above) but you will want use a special groupKey object so that the collected values can be streamed as soon as possible. It's not well documented, but here's a simple example that sorts the input BAM files prior to merging. Once all of the 'samtools_sort' jobs are finished for a particular sample, that sample can proceed to merging without us having to wait for all the other 'samtools_sort' jobs to be finished (which would otherwise be the case if the 'size' parameter had not been specified):
process samtools_sort {
tag { bam.baseName }
conda 'samtools=1.15.1'
input:
tuple val(sample), path(bam)
output:
tuple val(sample), path("${bam.baseName}.sorted.bam")
"""
samtools sort -o "${bam.baseName}.sorted.bam" "${bam}"
"""
}
process samtools_merge {
tag { sample }
conda 'samtools=1.15.1'
input:
tuple val(sample), path(bam_files, stageAs: 'mysample*.bam')
output:
tuple val(sample), path("${sample}.bam")
"""
samtools merge "${sample}.bam" ${bam_files}
"""
}
workflow {
Channel.fromPath( params.bam_files ) \
| map { bam -> tuple( bam.baseName, bam ) } \
| groupTuple() \
| map { sample, files -> tuple( groupKey(sample, files.size()), files ) } \
| transpose() \
| set { bam_files }
samtools_sort( bam_files ) \
| groupTuple() \
| samtools_merge \
| view()
}
Results:
$ nextflow run script.nf -ansi-log false --bam_files './path/to/*/*.bam'
N E X T F L O W ~ version 22.04.0
Launching `script.nf` [cheesy_swanson] DSL2 - revision: da1a82a059
[04/643129] Submitted process > samtools_sort (sample3)
[1a/93a939] Submitted process > samtools_sort (sample1)
[93/fde0a0] Submitted process > samtools_sort (sample3)
[db/208bac] Submitted process > samtools_sort (sample3)
[8f/f2a231] Submitted process > samtools_sort (sample2)
[f3/8f0f9f] Submitted process > samtools_sort (sample2)
[bb/cca23e] Submitted process > samtools_merge (sample1)
[sample1, /home/steve/testing/work/bb/cca23e016a641b201b7e42f9db74ef/sample1.bam]
[ff/bd3a2a] Submitted process > samtools_merge (sample2)
[sample2, /home/steve/testing/work/ff/bd3a2ac83a0691d2b80b04ef97b755/sample2.bam]
[94/a4f8eb] Submitted process > samtools_merge (sample3)
[sample3, /home/steve/testing/work/94/a4f8eb483602e9335b28f5d40058c8/sample3.bam]
Note that the val specified as input to 'samtools_sort' is an object of class nextflow.extension.GroupKey. In the example above, the input to 'samtools_merge' is also a groupKey - but it doesn't need to be. If you need a string value for whatever reason, you can just call '.toString()' to get it back:
...
samtools_sort( bam_files ) \
| groupTuple() \
| map { sample_key, files -> tuple( sample_key.toString(), files ) } \
| samtools_merge \
| view()
