1
$\begingroup$

First of all, I'd like to apologize for any spelling or grammar mistakes.

I'm having a problem using R for a classification problem. My dataset contains ~300.000 genomic data, and the features are DNA-related features (number of dinucleotides, number of trinucleotides, the CG Content, and some more). In conclusion, I have a dataset of 300.000 rows and 84 columns (columns= features). The 84th feature is basically the classification variable (there are two classes: class 1 and class 2).

I split the dataset in train set and test set and fit the train set to create a model using different methods, like Random Forest, XGBoost, Adaboost, etc. and check the accuracy using the test set.

In every model, the accuracy of the prediction on the test set is always around 86-87%. What I noticed is that, in the test set, for EVERY MODEL, almost the same number of rows (data) is predicted right (the model predict its true class), and always almost the same data of rows (data) is predicted wrong (the model predict wrong class. For example its true class is 1 but the model predicts 2).

If I take the set of data that are predicted right for every model I've created (250.000 data from the 300.000), and build a new model out of them (splitting them again in a new train and test set), I have an accuracy of 98%.

That means that the other 50.000 data (from the 300.000), add an error at every model I'm trying to create.

But I can't just ignore them. I thought that it would be a way to create two differents models, but doesn't know how to figure this out.

Then I thought that there could be a way to find which variables from the two subsets have strong differences, but even if I figure this out, I don't know what's the way for example to scale the original dataset to prevent those differences from keeping the accuracy in 87% in every model (using the original dataset of the 300.000 rows).

In conclusion, I have a dataset that pretends like "two datasets". It consists of two subsets: the first one is always predicted right from any model (machine learning algorithm) I'm trying, and the second one is always predicted wrong. How could I build a strong model taking into account the differences?

I also took a look at the box plots of the features. The most important feature for every model is called "CG" (number of dinucleotide of cytosines and guanines in DNA). In the following image you can see the boxplot of CG feature separating the original dataset in two datasets (T = for the dataset that is predicted right in every model) and (F = for the dataset that is predicted wrong in every model). 1 stands for data of class 1 and 2 stands for data of class 2. So, T1 means data from the right predicted dataset of class 1, T2 means data from the right predicted dataset of classes 2, T means data from the right predicted dataset for both class 1 and 2, etc.

box plot CG

As I'm a new beginner in R and in machine learning, I hope you do understand my problem. Thanks sincerely.

$\endgroup$
  • $\begingroup$ It needs some thought. Whats your training/test split, 80:20? Any parameter splits? $\endgroup$ – Michael Apr 12 at 18:30
  • $\begingroup$ The persistent 87-88% accuracy between models is a bit weird, but its an interesting problem $\endgroup$ – Michael Apr 12 at 18:44
  • $\begingroup$ I always used a train set with same amount of class 1 and class 2. for example, a train set of 50.000 data (25.000 of class 1 and 25.000 of class 2). $\endgroup$ – Giannis Lazaridis Apr 12 at 20:04
  • $\begingroup$ Okay thats the classificaiton (I don't know what "Class I" and "Class 2" represent, but I get the idea in ML, whatever "Class" is, is the thing you are looking at. The question was the split with testing, or parameterisation. The reliability of the accuracy score across very different models suggests something is not right, but I honestly don't know what. $\endgroup$ – Michael Apr 12 at 21:31
0
$\begingroup$

You could be seeing the effect of SNPs on haplotype-driven conditions (i.e. where a combination of multiple variants predicts the class, rather than a single variant).

Do you have any method to determine the age of particular variants (e.g. via an LD plot to find inconsistencies between variants and the surrounding block structure), so that you can see if that correlates with the likelihood of error?

| improve this answer | |
$\endgroup$
  • $\begingroup$ No, unforetunately no. $\endgroup$ – Giannis Lazaridis Apr 12 at 11:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.