12
$\begingroup$

I want to understand the difference between pipeline systems and workflow engines.

After reading A Review of Scalable Bioinformatics Pipelines I had a good overview of current bioinformatics pipelines. After some further research I found that there is collection of highly capable workflow engines. My question is then based on what I saw for argo. I would say I can be used as a bioinformatics pipeline as well.

So how do bioinformatics pipelines differ from workflow engines?

$\endgroup$
15
$\begingroup$

Great question! Note that from a prescriptive standpoint, the terms pipeline and workflow don't have any strict or precise definitions. But it's still useful to take a descriptive standpoint and discuss how the terms are commonly used in the bioinformatics community.

But before talking about pipelines and workflows, it's helpful to talk about programs and scripts. A program or script typically implements a single data analysis task (or set of related tasks). Some examples include the following.

  • FastQC, a program that checks NGS reads for common quality issues
  • Trimmomatic, a program for cleaning NGS reads
  • salmon, a program for estimating transcript abundance from NGS reads
  • a custom R script that uses DESeq2 to perform differential expression analysis

A pipeline or a workflow refers to a particular kind of program or script that is intended primarily to combine other independent programs or scripts. For example, I might want to write an RNA-seq workflow that executes Trimmomatic, FastQC, salmon, and the R script using a single command. This is particularly useful if I have to run the same command many times, or if the commands take a long time to run. It's very inconvenient when you have to babysit your computer and wait for step 3 to finish so that you can launch step 4!

So when does a program become a pipeline? Honestly, there are no strict rules. In some cases it's clear: the 10-line Python script I wrote to split Fasta files is definitely NOT a pipeline, but the 200-line Python script I wrote that does nothing but invoke 6 other bioinformatics programs definitely IS a pipeline. There are a lot of tools that fall in the middle: they may require running multiple steps in a certain order, or implement their own processing but also delegate processing to other tools. Usually nobody worries too much about whether it's "correct" to call a particular tool a pipeline.

Finally, a workflow engine is the software used to actually execute your pipeline/workflow. As mentioned above, general-purpose scripting languages like Bash, Python, or Perl can be used to implement workflows. But there are other languages that are designed specifically for managing workflows. Perhaps the earliest and most popular of these is GNU Make, which was originally intended to help engineers coordinate software compilation but can be used for just about any workflow. More recently there has been a proliferation of tools intended to replace GNU Make for numerous languages in a variety of contexts. The most popular in bioinformatics seems to be Snakemake, which provides a nice balance of simplicity (through shell commands), flexibility (through configuration), and power-user support (through Python scripting). Build scripts written for these tools (i.e., a Makefile or Snakefile) are often called pipelines or workflows, and the workflow engine is the software that executes the workflow.

The workflow engines you listed above (such as argo) can certainly be used to coordinate bioinformatics workflows. Honestly though, these are aimed more at the broader tech industry: they involve not just workflow execution but also hardware and infrastructure coordination, and would require a level of engineering expertise/support not commonly available in a bioinformatics setting. This could change, however, as bioinformatics becomes more of a "big data" endeavor.

As a final note, I'll mention a few more relevant technologies that I wasn't able to fit above.

  • Docker: managing a consistent software environment across multiple (potentially dozens or hundreds) of computers; Singularity is Docker's less popular step-sister
  • Common Workflow Language (CWL): a generic language for declaring how each step of a workflow is executed, what inputs it needs, what outputs it creates, and approximately what resources (RAM, storage, CPU threads, etc.) are required to run it; designed to write workflows that can be run on a variety of workflow engines
  • Dockstore: a registry of bioinformatics workflows (heavy emphasis on genomics) that includes a Docker container and a CWL specification for each workflow
  • toil: a production-grade workflow engine used primarily for bioinformatics workflows
$\endgroup$
  • 1
    $\begingroup$ Thanks, Daniel. Your answer outlines very well the details about the bioinformatics workflows. Your argument, that a very high level of software engineering is needed to use my listed workflow engines, is on point . If I understand correctly generally speaking bioinformatics pipelines can be pressed into workflow engines. So thanks again. That helped a lot. $\endgroup$ – A.Dumas Mar 29 '19 at 14:55
  • 6
    $\begingroup$ Nice answer! I'm biased, but I feel like nextflow is also worth a mention :) And also WDL to complete the picture of common workflow tools. There's a nice "awesome-pipeline" repo list of workflow tools here too: github.com/pditommaso/awesome-pipeline - a little more comprehensive than the one in the original post, perhaps with more of a bioinformatics slant. $\endgroup$ – ewels Mar 29 '19 at 15:18
  • 1
    $\begingroup$ Thanks for the comment @tallphil. I too am biased by the tools with which I have experience. :-) Feel free to edit my answer to add a bit about nextflow and WDL. $\endgroup$ – Daniel Standage Apr 1 '19 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.