I recently discovered this Snakemake pipeline for RNASeq that uses STAR's quantMode to quantify gene expression for DESeq2 differential expression analysis.
In the past I've always seen the workflow as:
More recently I've also seen:
Kallisto/Salmon pseudo-alignment quantification
I tried searching online for differences between STAR quantMode vs other quantification algorithms but couldn't find many details. What are the benefits and drawbacks of using STAR quantMode vs RSEM/Kallisto/Salmon?
STAR quantMode (GeneCounts) essentially provides the same output as HTSeq-Count would, ie. number of reads that cover a given gene. This is the most simple measure of expression you could get from RNA-seq data. Kallisto and Salmon utilize pseudo-alignment to determine expression measures of transcripts (as opposed to genes). RSEM uses some algorithm to determine isoform fractions, but still based on alignments. (Note that I don't have any personal experience using RSEM.)
It ultimately depends on the research question which method will be most appropriate for you.
relatively easy to understand concept-wise
can be used to generate many different expression measures (FPKMs, TPM, CPM, etc.) and is, thus, more versatile in down-stream analyses
you can't distinguish isoforms (from the counts themselves)
requires alignment and, thus, more time to obtain
faster (due to lack of alignment step)
quantifies (known) isoforms
expression values are more abstract, less easy to explain
gene-level analyses will require some extra work
Likely lies somewhere in between, still requiring alignments will make it slower. But it does give you expression values for isoforms.
A shorter version...STAR's is convenient, in that it gets done right alongside the alignment, but RSEM and Kallisto are smarter about dealing with ambiguous reads. So I'd say your results from either of those are superior.
Personally, I use STAR for alignment, then give the bam to RSEM. STAR's gene count is good for a reality check to make sure that I got the strandedness right, and RSEM's gene level expected counts are good for DESeq once they are rounded.