# How to select the most representative pathways from a gene enrichment analysis?

I have perform an enrichment analysis to a cluster of genes. The output is a list of pathways and their p-value (the pathways are selected because p-value < 0.05). The list is still quite long, so I want to reduce it. For that purpose I have a calculated the Dice coefficient of the pathways in a matrix $p$x$p$ where $p$ is the number of pathways in the list. I want both the ones that are more different (they overlap less, their Dice coefficient is lower) and the pathways more representative of the most similar pathways (So if a there is a group of 5 pathways that overlap over 0.8 take just one).

How can I select the most representatives pathways?

There is a similar tool for GO but it relays on discarding not significant GO, while here all the initial pathways are already significant.

If I do a clustering of the pathways using the Dice coefficient matrix I don't know where (or how) to cut.

I tried using the height to select the pathways. But I am unsure of the interpretation of height.

Some other tools I have seen use a multidimensional scaling plot, but I am not sure if performing it and cutting at certain point of the first dimension would help.

• Are you using p-value < 0.05 or p-value < 0.05 / (# of terms tested)? You'll want to use the former to avoid false positives due to multiple testing May 26 '17 at 19:14
• I'm guessing you'll have to settle with a "reasonably good" heuristic solution, since there are a few different variables in the problem you're trying to solve (number of pathways in group, desired overlap/distance between pathways, etc.) I can imagine many possible answers. May 26 '17 at 19:16
• @CloudyGloudy correcting for multiple testing is already done, sorry I didn't mention it previously. Yes, I was playing with the idea of selecting an overlap/distance of 0.5 and keep those above that, but that would leave outside the pathways that are completely different from the others, but maybe I could select those above 0.75 and those below 0.25
– llrs
May 27 '17 at 9:22

One way to group similar significant pathways, is to quantify how many genes overlap between the pathways, and then use this in clustering (heatmap). I have made a tool in R which calculates the overlap index between GO terms and subsequently clusters them in a heatmap. Overlap index is the fraction of genes that overlap (number between 0-1). Also Pearson correlation can be used for clustering instead of the overlap index. My package (gogadget) works only with goseq analysis, but you can use goseq also for reactome or kegg data.

I have used this overlap clustering approach for different data sets now, and usually we can reduce 200-300 GO terms into 10-20 functional groups.

• Which overlap index does it use? There are several overlap indices. I would advice against using this approach in GO, there are specific methods to measure how similar two GO terms are. See [GOSemSim](www.bioconductor.org/packages/GOSemSim/).
– llrs
Dec 12 '17 at 13:37
• The overlap index is defined by the number of overlapping genes divided by the number of genes in the smaller of the two gene sets. It is described in Bioconductor Case studies, chapter 13.3. This book is written by big names in bioinformatics (such as R. Gentleman and W. Huber).
– benn
Dec 12 '17 at 14:32
• Thanks for the new overlap index, I didn't knew it. BTW the way these similarity between genes is calculated is using other overlap indexes so this won't help.
– llrs
Dec 12 '17 at 14:39
• Okay, but my point is not about which index to use but the clustering approach (same as gringer is suggesting). If you look in the userguide of my package on page 26-28, you'll see a heatmap. Usually I get good results with Ward.D and Euclidean. The tree can be cut in R, but you first might want to see and evaluate which gene sets are clustered together in which branches before you can find the right cutoff.
– benn
Dec 12 '17 at 15:53
• Sorry, my last comment was thinking in another question. Yes, this approach could work
– llrs
Dec 14 '17 at 7:51

This sounds like something that might be amenable to a clustered heatmap plot, or a correlation matrix plot, or something similar. Have you looked at a correlation matrix of the dice coefficient matrix (or maybe just a heatmap plot of that matrix without the correlation matrix)?

The corrplot package looks like it might be useful, in particular the hclust / drawing rectangles presentation.

I can't vouch for this package though; it's just something I found by a search for "R plot correlation matrix".

• With a heatmap or a correlation plot I could observe the similarities between the pathways, as I currently do with the dendrogram and the MDS plot. However the question is how to select those more representative pathways. Thanks for your corrections and comments btw.
– llrs
May 28 '17 at 10:58

If you're happy with a more confident ranking of the most representative gene sets, rather than necessarily cutting down the list, you might try EGSEA. It uses an ensemble approach to give a ranking of the most relevant gene sets, and also produces an interactive HTML output with statistics, heatmaps, pathway maps, summary plots and GO graphs which allows you to examine the output at varying levels of granularity.