1
$\begingroup$

I am working on clinical study where I am focusing on binary classification problem.

There are few variables which has outliers.

I am trying to know which would be the right value to replace outliers in a clinical study?

Is it mean, median min,max or mode of a column of interest?

Currently I use min and max for values below and above 1.5*IQR threshold

I use IQR approach to identify outliers but would like to know how it is done usually in clinical studies?

Can anyone help me with this?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

In my experience, it is more common to use all clinical data as-is for clinical studies. And if data is missing, either omit the sample or omit the variable with missing data.

If your classifier can't handle the wide variation commonly seen in clinical studies then you may want to use a classifier which is less impacted by outliers.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the response. Upvoted. $\endgroup$
    – The Great
    Jan 1, 2020 at 12:50
1
$\begingroup$

Removing outliers is common practice in statistical modeling and perfectly acceptable. However, with regards 1.5 IQR I am far from certain about this approach. Normally, if you want to be conservative then 3 standard deviations (SD) denote an outlier, which is more stringent than IQR. Some use 2 SD. If the value is lower than 2 SD from the group mean it isn't an outlier, that's because 1.96 SD is the normal distribution.

Normally you would delete the outlier and it takes no further part in the calculation, assigning it to 'na' should work. The subsequent calculation will recognise an 'na' value, I think in python it is np('na') ... Np = numpy.

In this case you appear to be deleting the value and treating it as missing data. Treatment of missing data is complicated I agree 100%. Mean, median are possibilities for missing data I can't see this working for binary data. One approach is regression and in python there is a command specifically to do this (which I forget), R will have the same command and it will assign a value for the missing data. However, you can't do this for outliers.

The other routes to remove outliers are via transformation, log transformation is the most common. Transformation is a big subject.

Improve this answer
$\endgroup$
1
  • $\begingroup$ thanks upvoted for the response $\endgroup$
    – The Great
    Jan 1, 2020 at 0:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.