What is the significance of enrichment factor regarding machine learning methods?

Question

I'm fairly new to the field of bioinformatics and ran into a question while reading a paper I found on bioRxiv. The overall setting of the paper is using multi-task deep neural networks for kinase classification. The paper that I'm referring to is Kinome-Wide Activity Classification of Small Molecules by Deep Learning (Allen et al., 2019) for anyone who's curious.

I'll get straight into the specific figure that I'm having trouble understanding.

I know that questions like these are a bit discouraged on SE, but I'm having trouble on how to start regarding the interpretation of this figure.

Based on my understanding, the point seems to be that multitaks DNN's have the highest overall EF/EFmax ratio amounts, but what does this ratio signify?

I did some research and according to Wikipedia an "enrichment factor" in terms of bioinformatics is a concept "to measure the added value of a search tool over another one or over the homogeneous distribution in the genome population." Does this mean that multitask DNN's are "better" than the other models, due to its higher number of EF/EFmax points?

But if we look at the curves on the right, it seems to say that for fewer number of compounds, the ratio for multitask DNN's is lower...

Answer 1

I did some research and according to Wikipedia an "enrichment factor" in terms of bioinformatics is...

Where possible, it's best to use explanations provided in the original source. This is because definitions could be different, depending on who is writing. Here's some descriptive information from the paper that may be more informative:

The enrichment factor (EF) is another metric we use and is very popular for use in evaluation of virtual screening performance. EF denotes the quotient of true actives among a subset of predicted actives and the overall fraction of actives.... To evaluate the maximum achievable enrichment factor (EFmax), the maximum number of actives among the percentage of tested compounds is divided by the fraction of actives in the entire dataset. To more consistently quantify enrichment at 0.1% of all compounds tested, we report the ratio of EF/ EFmax. This normalized metric is useful because EF values are not directly comparable across different datasets due to the maximum possible EF being constrained by the ratio of total to active compounds and the evaluated fraction as shown in the equation above.

So for the purpose of this paper, they're saying that this ratio is an appropriately normalised substitute for a plain enrichment factor. High enrichment factor == good; low enrichment factor == bad.

Given the information you've provided, I'd agree that the curves in this figure appear to suggest that the multitask DNN performs worse with smaller numbers of active compounds (where Naive Bayes or Logistic regression would be a better choice). It's hard to tell from the graph (and legend) what the curves represent, and the clipping at EF/EFmax = 0 doesn't help.

The curves have a strange shape, suggesting the number of active compounds is dependent on the EF/EFmax ratio, rather than the other way round (which would make more sense to me). If I consider the curves to be something like a smoothed mean of the active compound number, then any place where there are fewer dots (i.e. low EF/EFmax for Multitask DNN) will have more associated error. For Multitask DNN, I see a large gap between 0.0 and 0.2, with lots of outliers on the 0.0 line (for all methods), so presumably the 0.0 outliers are skewing the curves.

Given that this is a preprint, the generation of the curves isn't specified, and the other figures (especially Figure 3) seem to tell a different story (i.e. that multi-task DNN performs better at all scales), I'd recommend contacting the authors to ask / clarify what the curves represent, or ignoring the curves entirely.

Answer 2

Ok I'll be brief, given the ms is likely under review.

Background in silico drug discovery has been part of deep learning for a long time 2015(?), and tensorflow was deployed. They worked out how to vectorise a small molecule and thereafter the entire field was.wide open to advanced ML.

Ratio The ratio is an accuracy index of the machine learning algorithms. The denominator is the total number of biological observations, the numerator is the predicted number. This 90% would be an accuracy index, i.e. the algorithm is correct 9 out of 10 times.

Result The plot shows, as the data set size increases the accuracy of the prediction increases most notably for the DNN and that conforms to all the theory. DNN/ANN only become powerful for 'big data', standard ML, i.e. random forest, naive Bayes, LR work well on small data sets, are fast and produce a classification which isn't wrapped up in a 3-4 Gig dump, thereafter performance saturates for routine ML methods.

The multi task DNN (MT-DNN) means that more than one loss function is being used, which is unusual in generic deep learning, but has been applied to natural language and drug discovery previously. Why this is being deployed is a long explanation, so I'll leave it there.