STAR quantMode (GeneCounts) essentially provides the same output as HTSeq-Count would, ie. number of reads that cover a given gene. This is the most simple measure of expression you could get from RNA-seq data. Kallisto and Salmon utilize pseudo-alignment to determine expression measures of transcripts (as opposed to genes). RSEM uses some algorithm to determine isoform fractions, but still based on alignments. (Note that I don't have any personal experience using RSEM.)
It ultimately depends on the research question which method will be most appropriate for you.
read counts
pros:
- relatively easy to understand concept-wise
- can be used to generate many different expression measures (FPKMs, TPM, CPM, etc.) and is, thus, more versatile in down-stream analyses
cons:
- you can't distinguish isoforms (from the counts themselves)
- requires alignment and, thus, more time to obtain
salmon/kallisto:
pros:
- faster (due to lack of alignment step)
- quantifies (known) isoforms
cons:
- expression values are more abstract, less easy to explain
- gene-level analyses will require some extra work
RSEM
Likely lies somewhere in between, still requiring alignments will make it slower. But it does give you expression values for isoforms.