4
$\begingroup$

My question is about ROC curves used in CAFA2 experiment. In this paper they used the ROC analysis for the term-centric evaluation. In order to perform ROC curve analysis we should have a continuous variable and a classifier (categorical) variable. In CAFA2 experiment, which variables did they consider as a categorical and continuous? Thank you.

$\endgroup$
2
$\begingroup$

The paper doesn't seem to have been written for general readers. Unfortunately I don't understand the equations enough to be able to explain what the categorical and continuous variables are. Just in case it helps, here are the equations from the paper showing the precision/recall calculations:

$$\begin{array}{@{}rcl@{}} \text{pr}(\tau) &=& \frac{1}{m(\tau)}\sum\limits_{i=1}^{m(\tau)} \frac{{\sum\nolimits}_{f} \unicode{x1D7D9}\left(f \in P_{i}(\tau) \wedge f \in T_{i}\right)}{\sum_{f} \unicode{x1D7D9}\left(f \in P_{i}(\tau) \right)},\\ \text{rc}(\tau) &=& \frac{1}{n_{e}}\sum\limits_{i=1}^{n_{e}} \frac{{\sum\nolimits}_{f} \unicode{x1D7D9}\left(f \in {P}_{i}(\tau) \wedge f \in T_{i}\right)}{{\sum\nolimits}_{f} \unicode{x1D7D9}\left(f \in T_{i} \right)}, \\ F_{\max} &=& \max_{\tau} \left\{ \frac{2\cdot \text{pr}(\tau)\cdot \text{rc}(\tau)}{\text{pr}(\tau) + \text{rc}(\tau)} \right\}, \end{array} $$

where P i (τ) denotes the set of terms that have predicted scores greater than or equal to τ for a protein sequence i, T i denotes the corresponding ground-truth set of terms for that sequence, m(τ) is the number of sequences with at least one predicted score greater than or equal to τ, $\unicode{x1D7D9}\left (\cdot \right)$ is an indicator function, and n e is the number of targets used in a particular mode of evaluation.

$\endgroup$
2
$\begingroup$

From a quick reading of the paper it looks like they are predicting biological function of proteins. They frame this as a classification problem, where they try to classify a protein as pertaining to a set of terms (Abnormality of the lymph nodes, Abnormality of skin physiology, etc). It looks like they predict for each protein whether it falls into this category and create a ROC curve to plot the false positive and true positive rates of prediction.

In this case, the continuous variable would be your predictions (Abnormality of the lymph nodes: predicted 0.65 probability, Abnormality of skin physiology: predicted 0.25 probability). The categorical variable would then be the ground truth that you are trying to predict (Abnormality of the lymph nodes: known 1, Abnormality of skin physiology: known 0). Based on these two variables you can make your ROC curve.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.