How to include only the trimmed fastq files into a nextflow channel and exclude any secondary log files created?

Question

I have the following simple nextflow script, that takes fastq paired reads and runs them through the atria adapter trimming.

// Declare syntax version
nextflow.enable.dsl=2

/* 
 * pipeline input parameters 
 */


params.reads = "$projectDir/data/*_{1,2}.fastq"
params.transcriptome_file = "$projectDir/data/Glycine_max.cdna.all.fasta"
params.gtf_file = "$projectDir/data/Glycine_max.gtf"
params.outdir = "results"


/*
 * define the `TRIM` process that creates trimmed.reads.fastq files
 * given the reads.fastq file. 
 */
 
process TRIM {

    tag "$pair_id"

    input:
    tuple val(pair_id), path(reads)

    output:
    path(pair_id)

    script:
    """
    atria -r ${reads[0]} -R ${reads[1]} -t 1 -l 12 -q 30 -o $pair_id
    """
}

workflow {

    read_pairs_ch = Channel.fromFilePairs( params.reads, checkIfExists:true )

    trim_ch=TRIM(read_pairs_ch)

}

Next what I want to achieve is input the atria.trimmed.fastq files generated by the trim process into a channel so that I could use it as an input for other process. But atria trimming tool also creates two additional log files along with atria.trimmed.fastq. How do i exclude those additional files and only input the trimmed files into a channel for later use.

Best wishes.

Answer 1

With output: path(pair_id), you're telling Nextflow to expect a file or directory with the value of pair_id in the process working directory when the command exits (successfully). Atria outputs all files to a directory specified using -o PATH¹. This directory is created if it doesn't already exist. Note that the output directory defaults to the current working directory if -o is not specified. If you'd like to write the results to a sub-directory, you could try:

process TRIM {

    tag { pair_id }

    publishDir "${params.outdir}/TRIM", mode: 'copy'

    cpus 1

    input:
    tuple val(pair_id), path(reads)

    output:
    tuple val(pair_id), path(trimmed_reads), emit: trimmed_fastqs
    path "${pair_id}/*.atria.log*", emit: logs

    script:
    def (r1, r2) = reads

    trimmed_reads = reads.collect {
        "${pair_id}/${it.baseName}.atria.fastq"
    }

    """
    atria \\
        -r "${r1}" \\
        -R "${r2}" \\
        -o "${pair_id}" \\
        -t ${task.cpus} \\
        -l 12 \\
        -q 30
    """
}

In the above, we used the emit option in the process output definition to assign a name identifier to each declaration. We can use this feature to reference the desired channel in the workflow block. For example:

workflow {

    read_pairs_ch = Channel.fromFilePairs( params.reads, checkIfExists:true )

    trim_out = TRIM(read_pairs_ch)

    trim_out.trimmed_fastqs.view()
}

Results:

$ nextflow run main.nf 
N E X T F L O W  ~  version 22.10.0
Launching `main.nf` [distraught_dalembert] DSL2 - revision: aea2d37092
executor >  local (3)
[5f/78908b] process > TRIM (A) [100%] 3 of 3 ✔
[B, [/path/to/work/6a/440a50e9a0725c18f9eede80549cb8/B/B_1.atria.fastq, /path/to/work/6a/440a50e9a0725c18f9eede80549cb8/B/B_2.atria.fastq]]
[A, [/path/to/work/5f/78908b96adee8b8d663d9b4eb4faba/A/A_1.atria.fastq, /path/to/work/5f/78908b96adee8b8d663d9b4eb4faba/A/A_2.atria.fastq]]
[C, [/path/to/work/b0/a3bb3024a4c05fffec555cce5097a7/C/C_1.atria.fastq, /path/to/work/b0/a3bb3024a4c05fffec555cce5097a7/C/C_2.atria.fastq]]

---

To input the trimmed reads into a downstream process, just ensure that the input tuple matches the input set cardinality declared by the process. If the input declaration is not the same, you may need to apply one or more operators to your channel to make it fit. For your alignment process, you might not need to make any changes. Also, if your alignment process needs to declare one or more reference files, just make sure these are value channels. Most of the time, what you want is one queue channel and one or more value channels when you require multiple input channels, otherwise you might observe unusual behavior (e.g. see this question). The section on multiple input channels in the docs is well worth taking the time to read in my opinion.

A dummy alignment process might look like:

process ALIGN {

    tag { sample }

    debug true

    input:
    tuple val(sample), path(reads)
    path fasta
    path gtf

    script:
    def (r1, r2) = reads

    """
    ls -g "${r1}"
    ls -g "${r2}"
    ls -g "${fasta}"
    ls -g "${gtf}"
    """
}

And the updated workflow:

workflow {

    read_pairs_ch = Channel.fromFilePairs( params.reads, checkIfExists:true )

    transcriptome = file( params.transcriptome_file )
    gtf = file( params.gtf_file )

    TRIM( read_pairs_ch )

    ALIGN( TRIM.out.trimmed_fastqs, transcriptome, gtf )
}

Results:

$ nextflow run main.nf 
N E X T F L O W  ~  version 22.10.0
Launching `main.nf` [extravagant_lattes] DSL2 - revision: 29b5315739
executor >  local (6)
[51/e4e261] process > TRIM (A)  [100%] 3 of 3 ✔
[cc/72c53d] process > ALIGN (A) [100%] 3 of 3 ✔
lrwxrwxrwx 1 users 57 Dec 14 00:21 C_1.atria.fastq -> ../../c1/9a85e19c601f2ab2c4af9c6fd203f0/C/C_1.atria.fastq
lrwxrwxrwx 1 users 57 Dec 14 00:21 C_2.atria.fastq -> ../../c1/9a85e19c601f2ab2c4af9c6fd203f0/C/C_2.atria.fastq
lrwxrwxrwx 1 users 40 Dec 14 00:21 Glycine_max.cdna.all.fasta -> ../../../data/Glycine_max.cdna.all.fasta
lrwxrwxrwx 1 users 29 Dec 14 00:21 Glycine_max.gtf -> ../../../data/Glycine_max.gtf

lrwxrwxrwx 1 users 57 Dec 14 00:21 B_1.atria.fastq -> ../../3f/16747fbd2c9bc4c0dc945b24873cca/B/B_1.atria.fastq
lrwxrwxrwx 1 users 57 Dec 14 00:21 B_2.atria.fastq -> ../../3f/16747fbd2c9bc4c0dc945b24873cca/B/B_2.atria.fastq
lrwxrwxrwx 1 users 40 Dec 14 00:21 Glycine_max.cdna.all.fasta -> ../../../data/Glycine_max.cdna.all.fasta
lrwxrwxrwx 1 users 29 Dec 14 00:21 Glycine_max.gtf -> ../../../data/Glycine_max.gtf

lrwxrwxrwx 1 users 57 Dec 14 00:21 A_1.atria.fastq -> ../../51/e4e261d019c65377d63826963b19b1/A/A_1.atria.fastq
lrwxrwxrwx 1 users 57 Dec 14 00:21 A_2.atria.fastq -> ../../51/e4e261d019c65377d63826963b19b1/A/A_2.atria.fastq
lrwxrwxrwx 1 users 40 Dec 14 00:21 Glycine_max.cdna.all.fasta -> ../../../data/Glycine_max.cdna.all.fasta
lrwxrwxrwx 1 users 29 Dec 14 00:21 Glycine_max.gtf -> ../../../data/Glycine_max.gtf