Reduce Overfit by removing insignificant proteins through PSEA analysis

Question

I'm training a ML model for disease prediction using protein composition as input, but overfit is present. While looking to remove proteins and reduce multi-correlation, as remove composition predictors through PCA have worked but not enough, this problem appeared and managed to use Metascape (1) for it.

I'm using the Protein Set Enrichment Analysis (PSEA, GSEA for genes) for significance of function as is a method to identify classes of genes or proteins that are over-represented in a large set of genes or proteins, and may have an association with disease phenotypes (2). So:

• A higher significance coefficient (log10pvalue) between two proteins suggest they have a similar function so one should be removed, or a lower coef shows the need to erase this quasi-independent protein from the analysis?

• Is this analysis good enough to reduce overfitting by remove redundant observations?

References:

(1) https://metascape.org/gp/index.html#/main/step1

(2) https://www.sciencedirect.com/science/article/pii/S2376999819300406

Answer 1

Overfitting is usually resolved by adding more data and not removing data sets. The technical blurb is 'augmenting' your training data set to revolve overfitting.

It is possible there is a very large multi-gene family involved distorting the classification, thus reducing that could reduce the overfit and cause the algorithm to spread the classification over a much broader group of genes. The general advice is to find an alternative genomic source of proteins and add that, not for this specific question but in reducing overfit per se.

There are technical issues such as:

• a parameter split, does the training have this? If not its strongly recommended - depending on the algorithm.
• algorithm, for example decision trees are known for overfitting*, whilst naive Bayes would find difficulty overfitting**.

I personally think multiple measures would resolve this. However, general advice is to look at alternative methods and training approaches before looking at slimming down a data set. I would suggest there needs to be good evidence - which would need to be formally presented. You could be 100% correct (probably are), but its not an orthodox approach.

Clearly there is information here that isn't presented. What you've done is a feature selection based on genes and we don't know what that result was nor the gene group that emerged from that analysis - so it is difficult to dissect this precisely. It could be there is a compelling biological reason for concern of overfitting based on the result, but we'll not know that. What I am certain about is the orthodoxy in resolving overfitting is not leveraged here and any data scientist would be aware of this.

*, this is why random forests are considered strong

**, there are a lot of different approaches to ML and how they are classifying is very different, some are more prone to overfitting than others. Naive Bayesian and 'linear regression' are the least sensitive to overfitting.

---

Please be aware I do ML within the Python framework and its one of the few areas I'm not aware of the ease of cross-over into other platforms. This is simply because its an analysis where Python excels.