0
$\begingroup$

I want to compare two AUCs using a bootstraping method from pROC package (roc.test).

I have done a classification using Logistic Regression and Serum miRNAs as prediction factors. The Controls are in both cases the same, the test group is different.

1. Cancer (high risk) vs. Control = AUC 1
2. Cancer (low risk) vs. Control = AUC 2

If I want to compare this two AUCs - should I use a paired or unpaired test?

I would tend to using a paired test, because of exact the same Control group (Control groups are correlated).

$\endgroup$
4
  • $\begingroup$ Not sure this can be answered. What are ROCAUCs? Bootstrapping is usually non-parametric, please confirm. ROC curves are a special case of data integrity used in machine learning, I'm not sure thats what you mean. Bootstrapping might be used for data augmentation, we'd need to know if that was the reason. As a simple answer paired tests are more powerful - so yes. The only time this is questioned is if it doesn't give the answer you are looking for. $\endgroup$
    – M__
    Oct 22 '20 at 8:02
  • $\begingroup$ @Michael ROCAUC is a AUCs of a ROC Curve - nothing special. You are right with the Bootstrapping method - it is non-parametric. So actually I want to compare two AUCs. Because in both cases I am using the same control group I am not sure if I need a paired or an unpaired test. In both cases the results are not significant and this was what I was looking for. So I do not really care about the concrete p-value - I just want to use the right test for the described case. $\endgroup$
    – Mischa
    Oct 22 '20 at 11:29
  • $\begingroup$ Okay I see. It has to be a paired test without bootstrapping. $\endgroup$
    – M__
    Oct 22 '20 at 12:18
  • $\begingroup$ @Michael Why without bootstrapping? My sample size is quite low with approx. 35/ group. Hence I do not really can assume nomality. This why I wanted to use Bootstrapping. DeLong would be another possibile method. $\endgroup$
    – Mischa
    Oct 22 '20 at 12:25
0
$\begingroup$

Generally its paired. The bootstrapping personally I find concerning. I presume you are using it to augment your data set.

Bootstrapping at best tends to be a controversial statistical operation and could result in the non-significance you are observing. Anyway if its not significant between high and low - thats your answer.

You could of course perform a single ML classifying low, and high cancer risk against control. That would provide a method to understand the associations of miRNA.


Bootstrapping issue, I had a think about this. It depends on how the bootstrapping is done. In a paired sample this (I assume) samples with replacement at each paired position for both ROCs. If this is not the case - the test is junk. However, if this is the case, its actually a good test, because the paired structure of the data is always preserved no matter how many bootstrap samples are performed. So like if a 100 were done and the pairwise statistic (which we don't know what it is - but something like KS test) and its >0.05 for all 100 samples - I don't think there's any doubt thats a robust result.

Again if the two ROCs are bootstrapped independently, in my opinion the test is junk.

Reason The concern of bootstrapping is it causes weird distributions which were not present in the original data. This is only okay if there's lots of experience about interpreting the result. For example, in phylogenetics there's decades of experience interpreting bootstrapping and a consensus about what it means. This often isn't the case for most other areas of statistics.

I note this particular issue is really ramping up the rep score ;0)

$\endgroup$
4
  • 1
    $\begingroup$ I have already done a discrimination between Control vs Tumor (low + high). The AUC is quite good with 0.87. Because my Tumor Group as a whole is strongly shifted towards high risk Cancer, I wanted to also show that there is no difference between the AUC of Cancer (high risk) vs Control and the AUC of Cancer (low risk) vs. Control. This way I can strengthen my results. $\endgroup$
    – Mischa
    Oct 22 '20 at 12:31
  • $\begingroup$ I see, I think that bootstrappng will bias towards a 'no-significant' difference result. See what the referees say I guess. $\endgroup$
    – M__
    Oct 22 '20 at 12:44
  • $\begingroup$ Can you please explain briefly, why you think that a paired test should be used? I just want to understand it. Thank you. $\endgroup$
    – Mischa
    Oct 22 '20 at 16:43
  • $\begingroup$ @Mischa answered above. Bootstrapping is an opinionated area of statistics outside phylogenetics. $\endgroup$
    – M__
    Oct 23 '20 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.