1
$\begingroup$

In my pipeline nextflow, I create a channel with channel.fromSRA and the channel contains lots of heavy files.fastq.gz. Then I have a first process to unzip files and transform them in files.fasta and a second process to count kmers. Is it possible to remove files.gz after first process and remove files.fasta after the second process. Can someone explain me how to do? They are too heavy to keep them in the work directory. Thanks.

In my pipeline I use kmer-ssr to count kmers present on my genome sample.

First I create a reads_ch = Channel.fromSRA

In this channel there are 12 files (. fastq.gz) of 20 Gb each one.

Then the first process unzips them and transforms them in files.fasta.

Now in my work directory there are 12 more files of 80 Gb each one. So the work directory is now 1200 Gb.

The second process counts kmer present in reads and creates 12 files of 100 Gb each one. The work directory will be 2200 Gb.

The work directory is too heavy. I want to know how I can suppress the current file. fastq.gz after first process

$\endgroup$
4
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Feb 11, 2023 at 13:26
  • $\begingroup$ Certainly this will be doable. Could you provide the code? Or else sufficient amounts of code to describe the nested loop? I will provide a link below as typical example of a NextFlow question. $\endgroup$
    – M__
    Feb 11, 2023 at 13:31
  • $\begingroup$ So this is yesterdays question here as an example of a standard NextFlow question on the site. $\endgroup$
    – M__
    Feb 11, 2023 at 13:33
  • $\begingroup$ Look into the "temporary" attribute of the inputs that should help you in the right direction. $\endgroup$
    – raygozag
    Apr 13, 2023 at 0:59

1 Answer 1

0
$\begingroup$

Unfortunately, this remains an open issue with Nextflow: https://github.com/nextflow-io/nextflow/issues/452

However, writing large amounts of uncompressed data to disk is usually best avoided. If it can't be (e.g. maybe the file needs to be read multiple times), you'd need to merge the two processes so that the writing of the temporary file and the running of the tool is done in a single process. This would then let you remove the temporary input file as soon as it's no longer required.

Another way would be to pipe the uncompressed and FASTA formatted reads directly into your tool. Not all tools support this functionality obviously, but Kmer-SSR can indeed receive input from STDIN, which completely obviates the need to write any temporary files. Note that by default, Kmer-SSR writes also to STDOUT and this lets you write a compressed output file:

process kmer_ssr {

    cpus 4

    input:
    tuple val(sample), path(fastq)

    output:
    tuple val(sample), path("${sample}.tsv.gz")

    """
    zcat -f "${fastq}" \\
        | sed -n '1~4s/^@/>/p; 2~4p' \\
        | kmer-ssr -t ${task.cpus} \\
        | gzip \\
        > "${sample}.tsv.gz"
    """
}

Since the above uses a pipeline, make sure to add the pipefail to your command. One way is to simply apply it to all processes by adding the following to your nextflow.config:

process {

  shell = [ '/bin/bash', '-euo', 'pipefail' ]
}

From the bash man page:

pipefail

If set, the return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully. This option is disabled by default.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.