My model doesn't work in test dataset

Question

1. I got the data of stomach cancer from TCGA, I found some different expression genes between cancer and normal sample.
2. I put aside my normal sample, and divide my cancer sample into train(80%) and test(20%) groups.
3. I did Unicox (with survival package), lasso (with glmnet package) and multicox ( with survival package) on train group, to find out which genes are more related to survival.
4. I use coefficients of each genes (that I got from multicox), to built my model and calculate the risk score. Like that: (expression of gene1×coefficient gene1 + expression of gene2×coefficient gene2 +...).
5. I divided my sample in to high risk and low risk groups, to compare survival between them, using Kaplan-Meier.

In training group, Kaplan-Meier shows significant difference between high and low risk group. However when I use my model on my test group, it is not significant. Why this problem occurs? I assumed my model is overfit of my train group, but how can I fix it? Why does this happen even when I use lasso?

Answer 1

There is no overtraining here, it isn't deep learning and there appears no parameterisation. If parameterisation was occurring on the training set that can and does cause over-tightening.

Essentially, you haven't included a data split for parameterisation, i.e. there is no parameterisation. The theory of a parameterisation is it must be distinct from training (and obviously) testing. It allows parameter tuned without overtightening. Lasso regression will require parameterisation.

I don't know about the survival regressions for machine learning, but lasso/ridge regression are standard approaches.

---

So just to explain, what you do is have a split for parameters. Thus instead of 80:20 split it could be 70:20:10 split where the 10% is used for parameter optimisation. Bad code risks leakage from the parameter split and that can result in over-tightening.