Nextflow: cannot get file pairs from Channel.fromFilePairs

Question

I'm remarkably still struggling with some basic Nextflow concepts. I'm simply trying to collect all pairs of .cram and .crai files from a directory using Channel.fromFilePairs(...), but I get no output when I try to view() the Channel.

Question: Why does my Channel.fromFilePairs(...) command not return file pairs, but I get a list of files when using Channel.fromPath(...)?

As far as I can tell, I'm following the same pattern from the following examples:

1. Nextflow fromFilePairs
2. .bam/.bai example (except without the extra code)

Code

nextflow.enable.dsl=2

workflow {

    /*
     * Channel.fromPath returns the expected files and prints them to
     * terminal.
     */
    Channel.fromPath('/path/to/adsp/cram/snd10000/A-CUHS-CU000*.cram')
        | view()
 
    /*
     * Channel.fromFilePairs does not print anything to terminal.
     */
    Channel.fromFilePairs('/path/to/adsp/cram/snd10000/A-CUHS-CU000*.{cram,crai}')
        | view()
}

Output (note, no file tuples/pairs)

N E X T F L O W  ~  version 21.04.3
Launching `pairs.nf` [pensive_plateau] - revision: f917a5c41b
/path/to/adsp/cram/snd10000/A-CUHS-CU000563-BL-COL-47451BL1_vcpa1.1.cram.crai
/path/to/adsp/cram/snd10000/A-CUHS-CU000387-BL-COL-38200BL2_vcpa1.1.cram.crai
/path/to/adsp/cram/snd10000/A-CUHS-CU000018-BL-COL-46250BL1_vcpa1.1.cram.crai
/path/to/adsp/cram/snd10000/A-CUHS-CU000169-BL-COL-57413BL1_vcpa1.1.cram
/path/to/adsp/cram/snd10000/A-CUHS-CU000563-BL-COL-47451BL1_vcpa1.1.cram
/path/to/adsp/cram/snd10000/A-CUHS-CU000387-BL-COL-38200BL2_vcpa1.1.cram
/path/to/adsp/cram/snd10000/A-CUHS-CU000194-BL-COL-38470BL1_vcpa1.1.cram
/path/to/adsp/cram/snd10000/A-CUHS-CU000194-BL-COL-38470BL1_vcpa1.1.cram.crai
/path/to/adsp/cram/snd10000/A-CUHS-CU000169-BL-COL-57413BL1_vcpa1.1.cram.crai
/path/to/adsp/cram/snd10000/A-CUHS-CU000018-BL-COL-46250BL1_vcpa1.1.cram

Answer 1

TLDR, use:

nextflow.enable.dsl=2

workflow {

    Channel.fromFilePairs('/path/to/adsp/cram/snd10000/A-CUHS-CU000*.cram{,.crai}')
        | view()
}

Results:

N E X T F L O W  ~  version 20.10.0
Launching `xscript.nf` [special_caravaggio] - revision: a4e53657a0
[A-CUHS-CU000169-BL-COL-57413BL1_vcpa1.1, [/path/to/adsp/cram/snd10000/A-CUHS-CU000169-BL-COL-57413BL1_vcpa1.1.cram, /path/to/adsp/cram/snd10000/A-CUHS-CU000169-BL-COL-57413BL1_vcpa1.1.cram.crai]]
[A-CUHS-CU000194-BL-COL-38470BL1_vcpa1.1, [/path/to/adsp/cram/snd10000/A-CUHS-CU000194-BL-COL-38470BL1_vcpa1.1.cram, /path/to/adsp/cram/snd10000/A-CUHS-CU000194-BL-COL-38470BL1_vcpa1.1.cram.crai]]
[A-CUHS-CU000018-BL-COL-46250BL1_vcpa1.1, [/path/to/adsp/cram/snd10000/A-CUHS-CU000018-BL-COL-46250BL1_vcpa1.1.cram, /path/to/adsp/cram/snd10000/A-CUHS-CU000018-BL-COL-46250BL1_vcpa1.1.cram.crai]]
[A-CUHS-CU000387-BL-COL-38200BL2_vcpa1.1, [/path/to/adsp/cram/snd10000/A-CUHS-CU000387-BL-COL-38200BL2_vcpa1.1.cram, /path/to/adsp/cram/snd10000/A-CUHS-CU000387-BL-COL-38200BL2_vcpa1.1.cram.crai]]
[A-CUHS-CU000563-BL-COL-47451BL1_vcpa1.1, [/path/to/adsp/cram/snd10000/A-CUHS-CU000563-BL-COL-47451BL1_vcpa1.1.cram, /path/to/adsp/cram/snd10000/A-CUHS-CU000563-BL-COL-47451BL1_vcpa1.1.cram.crai]]

---

---

When using the fromFilePairs factory method, the trick is to ensure that the pattern you provide can return a common prefix in order for the files to be grouped as expected. An easy way to check to see if this is working as expected is to directly call the readPrefix helper function. For example:

test_cram = file('/path/to/adsp/cram/snd10000/A-CUHS-CU000169-BL-COL-57413BL1_vcpa1.1.cram')
test_crai = file('/path/to/adsp/cram/snd10000/A-CUHS-CU000169-BL-COL-57413BL1_vcpa1.1.cram.crai')

pattern = '/path/to/adsp/cram/snd10000/A-CUHS-CU000*.{cram,crai}'

println("My CRAM prefix is: ${Channel.readPrefix(test_cram, pattern)}")
println("My CRAI prefix is: ${Channel.readPrefix(test_crai, pattern)}")

You can see that these results do not produce a common prefix:

N E X T F L O W  ~  version 20.10.0
Launching `script.nf` [lonely_jang] - revision: 428df7e55e
My CRAM prefix is: A-CUHS-CU000169-BL-COL-57413BL1_vcpa1.1
My CRAI prefix is: A-CUHS-CU000169-BL-COL-57413BL1_vcpa1.1.cram

But if you change your pattern definition to:

pattern = '/path/to/adsp/cram/snd10000/A-CUHS-CU000*.cram{,.crai}'

We can check that it returns a common prefix:

N E X T F L O W  ~  version 20.10.0
Launching `script.nf` [maniac_volta] - revision: a8acf7b03f
My CRAM prefix is: A-CUHS-CU000169-BL-COL-57413BL1_vcpa1.1
My CRAI prefix is: A-CUHS-CU000169-BL-COL-57413BL1_vcpa1.1