0
$\begingroup$
  1. I got the data of stomach cancer from TCGA, I found some different expression genes between cancer and normal sample.
  2. I put aside my normal sample, and divide my cancer sample into train(80%) and test(20%) groups.
  3. I did Unicox (with survival package), lasso (with glmnet package) and multicox ( with survival package) on train group, to find out which genes are more related to survival.
  4. I use coefficients of each genes (that I got from multicox), to built my model and calculate the risk score. Like that: (expression of gene1×coefficient gene1 + expression of gene2×coefficient gene2 +...).
  5. I divided my sample in to high risk and low risk groups, to compare survival between them, using Kaplan-Meier.

In training group, Kaplan-Meier shows significant difference between high and low risk group. However when I use my model on my test group, it is not significant. Why this problem occurs? I assumed my model is overfit of my train group, but how can I fix it? Why does this happen even when I use lasso?

$\endgroup$
3
  • $\begingroup$ Do you have sufficient power? Are your hazard rates/ratios very different between train and test? It would be possible to have a model that performs exactly as well as it did in the training data, but that shows up with no significant difference simply due to having a smaller sample size in the test data. $\endgroup$ Jul 25, 2022 at 21:01
  • $\begingroup$ Hi, thank u for your answer, yes at first I thought it is because of small sample size in test group, so I tried diving samples, 70-30, 60-40, and even 50-50. In all of the them pvalue in Kaplan meier was <0.001 in train group, and in non of the it was significant in test group $\endgroup$ Jul 25, 2022 at 21:10
  • $\begingroup$ Hi @FaridehJafari could you kindly format your question to make it clear? $\endgroup$
    – M__
    Jul 25, 2022 at 21:15

1 Answer 1

0
$\begingroup$

There is no overtraining here, it isn't deep learning and there appears no parameterisation. If parameterisation was occurring on the training set that can and does cause over-tightening.

Essentially, you haven't included a data split for parameterisation, i.e. there is no parameterisation. The theory of a parameterisation is it must be distinct from training (and obviously) testing. It allows parameter tuned without overtightening. Lasso regression will require parameterisation.

I don't know about the survival regressions for machine learning, but lasso/ridge regression are standard approaches.


So just to explain, what you do is have a split for parameters. Thus instead of 80:20 split it could be 70:20:10 split where the 10% is used for parameter optimisation. Bad code risks leakage from the parameter split and that can result in over-tightening.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.