Remove files after process is terminated

Question

In my pipeline nextflow, I create a channel with channel.fromSRA and the channel contains lots of heavy files.fastq.gz. Then I have a first process to unzip files and transform them in files.fasta and a second process to count kmers. Is it possible to remove files.gz after first process and remove files.fasta after the second process. Can someone explain me how to do? They are too heavy to keep them in the work directory. Thanks.

In my pipeline I use kmer-ssr to count kmers present on my genome sample.

First I create a reads_ch = Channel.fromSRA

In this channel there are 12 files (. fastq.gz) of 20 Gb each one.

Then the first process unzips them and transforms them in files.fasta.

Now in my work directory there are 12 more files of 80 Gb each one. So the work directory is now 1200 Gb.

The second process counts kmer present in reads and creates 12 files of 100 Gb each one. The work directory will be 2200 Gb.

The work directory is too heavy. I want to know how I can suppress the current file. fastq.gz after first process

Answer 1

Unfortunately, this remains an open issue with Nextflow: https://github.com/nextflow-io/nextflow/issues/452

However, writing large amounts of uncompressed data to disk is usually best avoided. If it can't be (e.g. maybe the file needs to be read multiple times), you'd need to merge the two processes so that the writing of the temporary file and the running of the tool is done in a single process. This would then let you remove the temporary input file as soon as it's no longer required.

Another way would be to pipe the uncompressed and FASTA formatted reads directly into your tool. Not all tools support this functionality obviously, but Kmer-SSR can indeed receive input from STDIN, which completely obviates the need to write any temporary files. Note that by default, Kmer-SSR writes also to STDOUT and this lets you write a compressed output file:

process kmer_ssr {

    cpus 4

    input:
    tuple val(sample), path(fastq)

    output:
    tuple val(sample), path("${sample}.tsv.gz")

    """
    zcat -f "${fastq}" \\
        | sed -n '1~4s/^@/>/p; 2~4p' \\
        | kmer-ssr -t ${task.cpus} \\
        | gzip \\
        > "${sample}.tsv.gz"
    """
}

Since the above uses a pipeline, make sure to add the pipefail to your command. One way is to simply apply it to all processes by adding the following to your nextflow.config:

process {

  shell = [ '/bin/bash', '-euo', 'pipefail' ]
}

From the bash man page:

pipefail

If set, the return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully. This option is disabled by default.