Selection of differential expressed genes

I'm working with RNA-seq data. I have 40 tumor samples and 5 Normal samples. Differential analysis with Deseq2 based on Fold change > 1.2 and alpha < 0.05 gave very low number of differentially expressed genes. Only 2 upregulated genes.

res <- results(dds, lfcThreshold = log2(1.2), alpha = 0.05)


To get more number of differential expressed genes I have few questions now

1. Instead of FDR < 0.05 can I use FDR < 0.1 (or) FDR < 0.5. Will there be any low confidence with this?

2. Can I select differential expressed genes only based on FDR < 0.05 without any fold change cutoff?

3. As I get very low number of DEGs with FDR < 0.05 & Foldchange > 1.2, Can I select DEGs based on Foldchange > 1.2 and p.value < 0.01 or 0.05 ?

PCA of the samples looks like this:

• (FDR) ɑ < 0.5 would be completely meaningless. With 40 tumour and 5 normal samples you should get extremely robust results, and I’d suggest that you should use something like ɑ < 0.0001. Given your PCA and your lack of DE, the only valid conclusion is that you really have no evidence whatsoever for differential expression. The samples are indistinguishable by the state of the art. This in itself is a striking finding. Jul 10, 2018 at 15:38

1. You can usually get away with FDR < 0.1, but that's as high as you can go. This all presumes you're doing follow-up experiments of some sort, of course. I guess you could increase the FDR more, but you're then really increasing the odds that your follow-up experiments will fail. Obviously increasing the FDR will decrease the confidence in the results, it's called "false discovery rate" for a reason.

2. Yes, fold-change cut-offs are mostly useful when you need to prioritize hits further. Small changes are almost impossible to validate with less-sensitive methods (qPCR) and less likely to be biologically relevant.

3. You're just going to waste your time if you ignore FDR.

Remember that all thresholds are intended to increase the odds of success in follow-up experiments. If you have infinite time and resources then you can set these to whatever you want, but you have neither so you better prioritize reasonably.

It's likely that you have a large batch effect in your data, effect sizes in cancer tend to be much larger than this.

• Thanks Devon. Yes there will be follow up experiments after selection. So, what would recommend me now? Can I go with FC and FDR < 0.1? Yes, there is batch effect what can I do now? Can I apply subsampling procedure to get more DEGs? Jul 10, 2018 at 13:13
• Yes, FDR<0.1 and ideally some reasonable fold-change (say a 30% change). Have a look at packages like sva for dealing with your batch effect. Jul 10, 2018 at 13:14
• Is it a good idea to apply subsampling procedure for DEA? Jul 10, 2018 at 13:35
• @beginner No, that's not useful, presuming you mean subsampling from the counts. Jul 10, 2018 at 14:21
• No, that's not helpful, you're just approximating the results you would have gotten otherwise. Jul 10, 2018 at 14:30

I think since your PCA shows that your normal samples cluster right alongside the tumor ones, you have to conclude that there is much more variation between tumors than between tumors and controls. DESeq's differentially expressed gene list was generated properly.

Your results are what they are, sorry they are so useless.

In general, messing with FDR or log fold change cutoffs beyond the ones you used gets pretty sketchy. I usually only adjust these values in the more conservative direction, although you may be able to get away with FDR < 0.1 if you have a large effect size (which you don't).

You might consider investigating why you don't seem to get any differentially expressed genes before adjusting your cutoffs: PCA on the samples based on their top 1,000 most variably expressed genes would be a good start. Do the tumor and normal transcriptomes cluster apart from each other?

With so few normal samples, it is possible that you are not fully characterizing the variability of that population and it is affecting your ability to identify meaningful differences between normal and tumor.

• Hi, I made the PCA and it is given in the question. please have a look. Jul 9, 2018 at 22:57
• If you have a large effect size you wouldn’t need to increase the FDR to 0.1: a lower FDR would still capture the effect. Jul 19, 2018 at 13:25