I'm a novice trying to wrap my head around with fancy jargons. If I search residual, I understand that it represents the difference between the observed value of the dependent variable (y) and the predicted value (ŷ).
I'm unable to follow then, how are these residuals used as new phenotype?
The residual is the level of error in a regression model, the lower the residual the better the model. Residuals cannot equate to a phenotype, it is actually the opposite if your regression model is trying to investigate a given genotype/phenotype(s) model. The residual in this case is the amount of infomation which the genotype/phentype does not account for, i.e. there are other factors resulting in your observations which you have not captured in your model.
What I assume is that the regression model you are referring is investigating whether the residual is lowered by examining different gene combinations. The objective of a regression model is to lower the residual and I suspect that it is comparing the new residual against the old one. Therefore it is not being used as an input into a regression model but an assessment of its robustness.