If I’m starting with Deseq2 normalized counts what are some preprocessing steps that I should apply to these data before estimating sample correlation using the cor function in R? For example, would it make sense to quantile normalize the data?
You don't absolutely need to perform any further steps before calculating the correlation. However you would likely benefit from the following sort of workflow:
- Transform your data with vst() or robust log (see the DESeq2 vignette) so genes are on a more or less similar scale.
- Select the top 300-500 most variable genes
- Use this subset with to compute sample-wise correlation.
This also what follows what you would find in the DESeq2 user guide, which I strongly encourage you to read.