5
$\begingroup$

I learned that GATK 4 is using Spark for parallelization. I googled around, though I am still not quite sure how spark really works and how to use it in practice. Besides GATK 4, are any other bioinformatics tool using spark? Generally, is spark widely used? Is it a necessary skill to learn? Thanks in advance.

$\endgroup$
8
$\begingroup$

The only bioinformatics tool other than GATK4 that I am aware of that uses spark is Hail (a Spark based replacement for Plink). Hail is also supported by researchers at the Broad.

Most places I have worked at have not switched over to Spark. As such, I don't think it is widely used generally. Therefore, I don't think knowing Spark qualifies as a necessary skill to learn for bioinformatics at this time.

However, I think Spark is a superior paradigm for handling the large amount of data that bioinformaticians routinely deal with, and I think we will soon see the field move towards using it more. Any bioinformatician would do well to acquaint themselves with Spark and to play around with Hail.

$\endgroup$
3
  • $\begingroup$ From reading into that a bit more, it looks like calling Hail a Spark-based replacement for Plink is selling it short. It's mostly true, but a bit like calling a car a wheel-based replacement for a horse. The efficiency of variant storage, and ability to distribute the workload, seems like quite a disruptive change in how variant analysis, filtering and QC is currently done. $\endgroup$
    – gringer
    Jun 4 '17 at 4:36
  • $\begingroup$ I agree with you. On the other hand, I think there is value in thinking of a car as a wheel-based replacement for a horse O,o $\endgroup$
    – winni2k
    Jun 4 '17 at 11:26
  • $\begingroup$ This is my impression as well: few tools are using Spark as yet, but there is a lot of interest in Spark as an idiom for concurrency. It's more suited to biological problems than a many other choices and is less complicated. So we'll probably see a lot more use of it in the future. $\endgroup$
    – agapow
    Jun 5 '17 at 11:57
6
$\begingroup$

ADAM and avocado are Spark-based alignment and variant calling tools under active development by a collaboration (http://bdgenomics.org) that also includes the Broad, but I do not believe they have wide adoption.

$\endgroup$
6
$\begingroup$

I think others have provided fine answers to your question already. I just thought it might be relevant to mention about a tool that allows using Spark to distribute computations that re-use existing commandline bioinformatics software ... so, avoiding the need to re-implement proven algorithms in a language that Spark supports, EasyMapReduce.

This perceived need to re-implement algorithms, has otherwise been one of the biggest road blocks for wider adoption of Spark in the bioinformatics community in my understanding.

EasyMapReduce was presented SparkSummit East this year (video link here, as shown on the repository home page).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.