Model optimization causes overfitting

Question

I'm working on an ML model to predict colorectal cancer using data from the human microbiome (collected from fecal samples).

My dataset is made up of metadata (containing phenotypic information on the N=156 population on which the study is being carried out, and an OTU table containing a taxonomy of bacteria found and their abundance for each sample from one person (each person has one and only one sample).

I filtered the OTU table by eliminating sparse data (I eliminated OTUs below a certain threshold) after which I calculated the relative abundance for each OTU.
I deleted samples with missing values.
I made the metadata join (Subject ID Sample ID Age (years) Gender BMI (kg/m²) Country of Residence Diagnosis TNM Stage AJCC Stage Localization) and the OTU table.

In the end, I ended up with 114 rows × 221 columns .

I normalized the data and then built an SVM(SVC) model with the rbf kernel with C=0.05 and gamma=0.001 I got

Training Accuracy: 0.967032967032967
Test Accuracy: 0.782608695652174
SVC(AUC=0.64)

When I try to optimize the model using the code below, the values I get for gamma and C give me overfitting (trainset accuracy is 100%).

# import packages
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
# range of values to try for gamma and C
Cs = [0.00001,0.0001,0.001,0.1,1,2,5,10,100,1000]
gammas = [0.00001,0.0001,0.001,0.1,1,10,100,10000] 
# number of segments for cross validation
nfolds = 5 # number of segments for class validation
param_grid = {'C':Cs ,'gamma': gammas}
# search for the best parameters using cross-validation 
grid_search = GridSearchCV(SVC(kernel='rbf'),param_grid,cv=nfolds)
grid_search.fit(X_train,y_train)
grid_search.best_params_

Answer 1

You are missing a validation split triggering data leakage. The error is here:

grid_search.fit(X_train,y_train)

You can't use X_train and y_train here. The correction is

grid_search.fit(X_val,y_val)

Generating X_val and y_val is described below.

The code used to generate

Training Accuracy: 0.967032967032967 Test Accuracy: 0.782608695652174

... included one test_split_train and you need to perform a second test_split_train to obtain X_val, and y_val. The original code gave a pretty good as a result, the training score will always be higher.

The diagnosis of the code presented is there is 100% data leakage. What is missing is a second test_train_split. Again one test_train_split is present in the original code and there would have been a,

import pandas as pd
from sklearn.model_selection import train_test_split

X = # a dataframe containing the features remove the target y (below)
y = # a same dataframe but a single column which is the target
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2,random_state=1)

The above will give a training proportion at 80% and test proportion at 20%. It's called an 80:20 split. That code will have been in the original code for the first calculation, I know that because the results are pretty standard for a routine ML test_train_split operation. Whats happening is the training set is being held absolutely separately to the test set. There is 100% compartmentalisation between the two data sets. Thats critical because if they got confused the calculation falls over, you will be testing the exact data thats just been trained. That confusion would be term "data leakage".

In the code you presented what has happened is the same data used for parameterisation is exactly the same data for training (X_train, y_train). Again it's called data leakage and parameter grids are very sensitive to this.

To correct this when a parameter grid is used it must have its own data set. That is instead of having a X_train,X_test,y_train,y_test the test_train_split is modified to generate X_train,X_test,y_train,y_test, X_val, y_val. The code to generate this is,

import pandas as pd
from sklearn.model_selection import train_test_split
 X_train, X_test, y_train, y_test 
    = train_test_split(X, y, test_size=0.2, random_state=1)

 X_train, X_val, y_train, y_val 
    = train_test_split(X_train, y_train, test_size=0.25, random_state=1)

This approach is singly leveraging test_train_split by splitting the training data set twice. This example creates a 60:20:20 split. Some say that 70:20:10 split is used for a parameter split (10 for the parameter search). Matter of opinion (you just adjust the proportions).

Thus

1. the grid search is strictly for the parameter split (X_val, y_val)
2. and those parameters at then using in training split (X_train, y_train)
3. the testing is then performed (X_test, y_test)

What this process does is ensure no part of X_val, y_val is ever seen in the training set. If that happens again - data leakage. Here only the optimised parameters are seen by the training set.

---

Summary Introducing validation parameter split into the calculation isn't trivial and there is nothing wrong with your original result. Thus you don't necessarily need it. Generally speaking 80% accuracy is fine, yours is 1% lower (can be overlooked).

Parameter tuning will increase the accuracy score but it will be 85% rather than 79% and if it's really gone well then it might be 90%. What I'm saying is it will improve things, but will not absolutely solve the accuracy issue, it's just a general and accepted issue within ML.

---

Just to finally add there are biological reasons in cancer prediction why accuracy is a key parameter, but thats a different question.

---

Comments: causes lower scores

What I'm saying the correct approach. I recognise that Geek for Geeks uses your approach - but they are wrong.

A separate validation split will certainly give result in lower scores than grid_search.fit(X_train,y_train) thats inevitable, but it is correct in stopping overfitting.

If the validation split causes its lower accuracy than 0.8 I'd look at gammas = ... 0.0001,0.001,0.1 ... because you have missed 0.01 and that will affect your model. It shouldn't give a lower score than the default parameters - it's not possible and something is wrong with the code and application of parameters.

Once you've the initial parameters you can refine the parameter search around the initial values - nothing wrong with that and that can even be coded (automated). The only minor issue is reducing the size of the training split (60% is a bit low). That can cause issues in which case the split is 70:20:10, i.e. 10 for validation.