1
$\begingroup$

I have made some architectures of neural networks in order to classify proteins in three categories. I have calculated true positive, true negative, false negative, false positive, Mathews, sensitivity, specificity, accuracy for each category. My question is what is the right way to evaluate the architecture. Except for the c-index, could I calculate a cumulative sensitivity, accuracy etc?or should I evaluate separately for each category? Thank you!

$\endgroup$
0
$\begingroup$

Wow, I would just go for accuarcy as the singular most important measure, but be careful of sampling bias.

If the classification training had an unbiased sample* and you have an accuracy of >90%, in general ANN that wouldn't be great, but in bioinformatcs thats really good. If your training comprises sampling bias that gets tricky and you either need to accomodate that in your accuarcy ROC etc ... or else retrain to eliminate the sample bias.

You can account for all of them of course by wrapping everything into a single analysis.

*, by this I mean that each of the 3 classifications comprises 33% in each training set. If this is badly skewed, this requires consideration and gets complicated. If it is precisely 1/3 each then 'accurary' is the key stat.

$\endgroup$
9
  • $\begingroup$ thank you!how can i calculate accuracy for all the three categories? is it correct to calculate the mean ? (sorry in advance for this question) $\endgroup$ – marilu May 1 '20 at 19:45
  • $\begingroup$ Thanks, firstly this looks very cool work, again... its cool. Did you do 3 independent training sets, each with a single classification OR a single training with 3 classifications? $\endgroup$ – M__ May 1 '20 at 19:51
  • $\begingroup$ a single training with 3 classifications.thank you! $\endgroup$ – marilu May 1 '20 at 20:35
  • $\begingroup$ I get it, sorry. Yes and no. Yes: for your own internal purpose yes and a line or two in the report/paper. You don't want major skews between the classifications (these are different skews from the sampling skews). No: for the overall accuracy score, its one overall number (obviously one which isn't hiding a skew). Essentially the answer is its just one number, but you need to be confident its representative. I hope thats ok. $\endgroup$ – M__ May 1 '20 at 21:23
  • 1
    $\begingroup$ Either 0.856 or 0.86, sounds like what you really have, I think is fine and shows archecture convergence. Please refer to other comparable papers however for accuracy comparision. Obviously 90% is better. Things get very complicated indeed increasing the accuracy. $\endgroup$ – M__ May 2 '20 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.