I learned that GATK 4 is using Spark for parallelization. I googled around, though I am still not quite sure how spark really works and how to use it in practice. Besides GATK 4, are any other bioinformatics tool using spark? Generally, is spark widely used? Is it a necessary skill to learn? Thanks in advance.
The only bioinformatics tool other than GATK4 that I am aware of that uses spark is Hail (a Spark based replacement for Plink). Hail is also supported by researchers at the Broad.
Most places I have worked at have not switched over to Spark. As such, I don't think it is widely used generally. Therefore, I don't think knowing Spark qualifies as a necessary skill to learn for bioinformatics at this time.
However, I think Spark is a superior paradigm for handling the large amount of data that bioinformaticians routinely deal with, and I think we will soon see the field move towards using it more. Any bioinformatician would do well to acquaint themselves with Spark and to play around with Hail.
I think others have provided fine answers to your question already. I just thought it might be relevant to mention about a tool that allows using Spark to distribute computations that re-use existing commandline bioinformatics software ... so, avoiding the need to re-implement proven algorithms in a language that Spark supports, EasyMapReduce.
This perceived need to re-implement algorithms, has otherwise been one of the biggest road blocks for wider adoption of Spark in the bioinformatics community in my understanding.
EasyMapReduce was presented SparkSummit East this year (video link here, as shown on the repository home page).