How would one determine the significance of a variable in a glm model?

If I, for example, have a dataframe like seen below, how would I determine if the origin of the sample has a significant effect on the value? (this is the number of enzymes capable of degrading the substrate f that matters)

Substrate    variable value origin
cellulose       M09    8    free
mannan          M12    2    free
glycogen        M65    2    free
chitin          M87    4    free
cellulose       M90    2    isolate
manan           M78    1    isolate
glycogen        M21    4    isolate
chitin          M21    1    isolate

So far I have tried:

mcomp = glm.nb(value ~ origin, data = my_data)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-0.9625  -0.9047  -0.9047   0.1212   3.5232  

              Estimate Std. Error z value Pr(>|z|)   
(Intercept)   -0.01657    0.06571  -0.252  0.80097   
originisolate -0.21911    0.08180  -2.679  0.00739 **
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for Negative Binomial(0.3418) family taken to be 1)

    Null deviance: 2053.5  on 2679  degrees of freedom
Residual deviance: 2046.3  on 2678  degrees of freedom
AIC: 6517.5

Number of Fisher Scoring iterations: 1

              Theta:  0.3418 
          Std. Err.:  0.0186 

 2 x log-likelihood:  -6511.4590 

So free becomes the intercept and then isolate if significantly different from that. Does this mean Origin has a significant effect on the value?

Would the better approach be to do the following?:

mcomp = glm.nb(value ~ origin + Substrate, data = comb_data) 
              Df Sum Sq Mean Sq F value Pr(>F)    
origin         1     23   22.55   6.612 0.0102 *  
Substrate     44   1445   32.84   9.631 <2e-16 ***
Residuals   2634   8981    3.41                   
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 

This shows me that origin and substrate have an effect on value if I understand correctly?


There is no better method, it's a matter of what you want to test or what is your question.

Using the anova() or aov(), test the terms collectively. For example, in your example with Substrate, the null hypothesis is that the coefficients are all zero, meaning cellulose =0, mannan =0 , ....

If the question is, "do the isolate samples have a higher value than origin samples?", then you can use your first model, where free is set as the reference and you test whether the effect of isolate is non-zero. Likewise you can do this for substrate and set of them as your reference. You can also do other pairwise comparisons using this model.

If the question is, "does origin have a significant effect on value, after controlling for substrate?", then you can use your second model.

  • $\begingroup$ Would I control for substrate by forming the model as such: value ~ origin + substrate ? $\endgroup$
    – RMM
    Jun 29 '20 at 7:20
  • $\begingroup$ yes you do. you are giving every substrate a meaning and on top of that estimating the effect of origin $\endgroup$
    – StupidWolf
    Jun 29 '20 at 7:21
  • $\begingroup$ And does the order in which I give the variable to the model, i.e. origin before substrate, indicate some sort of predefined importance to the model? $\endgroup$
    – RMM
    Jun 29 '20 at 7:24
  • $\begingroup$ it doesn't define the importance, you can put it in any order. However if they are correlated, most of the variance will go to the first. I don't think this will apply in your case $\endgroup$
    – StupidWolf
    Jun 29 '20 at 7:28
  • $\begingroup$ see cran.r-project.org/doc/contrib/Faraway-PRA.pdf blocking design. A good ieda to read this $\endgroup$
    – StupidWolf
    Jun 29 '20 at 7:32

Second viewing of the question from what I can see -0.22 as a coefficient of origin is a strong negative association, so yeah it has a major impact. Its not how I would have done it, but that looks to be the result.

First viewing,

I'm going to throw my hat in here. We don't know what 'origin' is about, anyway just throw everything, i.e. each substrate and the origin into the same regression calculation. Check for a low-residual and preferably do a Q-Q plot, transform your data it this doesn't look good.

The key and the thing you are missing is your regression weights, without that I couldn't say very much. If the regression weight is near zero for 'origin' then it has zero impact. If the regression weight of 'origin' is positively greater than everything else ... I assume there are skewed distributions of 'substrates' between the 'origins'. If the regression weight of 'origin' is negative but still greater than all other regression weights then it is adversely affecting the 'value' you are seeking.

I don't know the experiment, the biological system or really the 'substrate' assays, so I can't comment any further.

The two issues I have are:

  1. Doing an ANOVA on the output of a regression analysis doesn't make much sense to me. It is not something I would do, nor something in ML or GLM I've encountered.
  2. Are you doing pairwise substrate/origin calculations? I presue not, but just in case this not how GLM works.
  • $\begingroup$ oh I was under the impression aov(summary(glm_model)) was a pretty common practice as it nicely summarises down to a variable level? Where do I find the regression weights? $\endgroup$
    – RMM
    Jun 29 '20 at 7:31
  • $\begingroup$ When transforming the data do I have to transform both variables or can I transform just one of them until they fit the same distribution? and given that I am using a negative binomial glm does a uniform distribution between the variables matter as much? I am unsure if there is a skewed distribution of variables between origins, that is what I am looking to find out really. $\endgroup$
    – RMM
    Jun 29 '20 at 7:36
  • $\begingroup$ hi @Michael, very nice points here, yes the term anova is quite misleading sometimes. Normally people associate associate with the classical analysis of variance. In more "modern" uses, it can refer to the breakdown of deviance in a model, stats.stackexchange.com/questions/289226/… $\endgroup$
    – StupidWolf
    Jun 29 '20 at 7:36
  • $\begingroup$ It is like a log likelihood ratio test essentially, the "F" statistic instead of Chi-sq is used. $\endgroup$
    – StupidWolf
    Jun 29 '20 at 7:38
  • $\begingroup$ @StupidWolf So it is fine to aov(summary(glm_model)) to get a nice overview of the model? And by pairwise substrate/origin calculations does you mean doing the model on a subsatrate of the dataframe for each substrate @michael? $\endgroup$
    – RMM
    Jun 29 '20 at 7:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.