Genome QC + Assembly Pipeline semantics

I’m trying to create a pipeline for genome assembly. How best can I “redirect/pipe” from existing fasta files (or files in general) to other steps of the pipeline?

I was thinking of going from the SRA Download step (prefetch and fasterq-dump) to fastqc, then to trimmomatic or bbduk to remove the adapters, one more fastqc check and then to SPAdes for the assembly.

In my (amateur) mind it would be cool to be like:

“SRRXXXXXX” | prefetch | fasterq-dump | fastqc | bbduk.sh | fastqc (second) | spades.py

Any and all help is greatly appreciate! I know this stuff is pretty basic but I am so new to everything it’s painfully obvious to even me.

Maybe take a look at workflow management systems, like

Admitted, each of those adds their additional learning curve. However, especially snakemake is not so much more complicated than a standard bash workflow. Those systems do help a lot organize and orchestrate your workflows

I understand the desire to pipe everything like that but I don't think it will work, even for such a small workflow. And as your pipe grows it'll make it harder to troubleshoot/debug.

These days everyone uses workflow languages to write pipelines. It's a bit more work to get set up but worth it in the long run. I use WDL. Snakemake is very popular. BASH will of course work for small pipes but not with | for every step.

Directly piping will definitely not work, as Liam McIntyre suggests. However, if you don't feel up to learning a workflow language just to do this, it is perfectly possible to just put the various commands in a bash driver script (or whatever you're comfortable with).

Each of those tools will take parameters to define output filenames, or have systematic ways of writing FASTA files and defining inputs. You will have to look at the command line interface of each of those tools to figure it out (for example, sratools here, bbduk.sh here and spades here). an example would be as follows:

#!/usr/bin/bash
SRA=\$1

prefetch $$SRA # writes to a known path locally fasterq-dump$$SRA --split-3  # looks for path_to_sra_file and outputs it. assumes you are splitting.

bbduk.sh in1=$$SRA''_1.fastq in2=$$SRA''_2.fastq out1=$$SRA''_clean_1.fq out2=$$SRA''_clean_2.fq

# re-QC the files- this step does not feed into the other steps!!
fastqc $$SRA''*.fastq$$SRA''*.fq  # runs fastqc on all fastq files

# run assembly
spades.py -o my_assembly --pe1 $$SRA''_clean_1.fq --pe2$$SRA''_clean_2.fq



and so on.

I suppose that from a formal perspective it's better to use a workflow engine as Liam McIntyre says, but with such a straightforward workflow I'd say start simple and get more elaborate only if you need to; you'd spend 10x as long just figuring out the workflow language. This is ok if it's a goal to learn it or if you already know it, but if you just want some results putting it into a script is fine.

• The other virtue of putting everything in a script is every command line option is documented. – swbarnes2 Feb 17 at 22:15
• All workflow management systems I have ever worked with also document all parameters (and much more) meticulously. – Pallie Feb 24 at 9:51