# How to fix the Error in Kruskal Wallis rank sum test for comparing the guides in python? Also to check whether using the right test for my data?

df

gene_name   guide_1 guide_1_new Correlation
MMP-1   A   A   1
MMP-1   A   B   0.426115
MMP-1   A   C   0.522499
MMP-1   A   D   0.431587
MMP-1   B   A   0.426115
MMP-1   B   B   1
MMP-1   B   C   0.60113
MMP-1   B   D   0.534858
MMP-1   C   A   0.522499
MMP-1   C   B   0.60113
MMP-1   C   C   1
MMP-1   C   D   0.622206
MMP-1   D   A   0.431587
MMP-1   D   B   0.534858
MMP-1   D   C   0.622206
MMP-1   D   D   1


I used four guides in an experiment (data = df). I calculated correlation between them based on the their log fold changes. I want to check which guide is outperforming or underperforming in comparison to other three guides in my experiment. I tried Kruskal and Wallis test to do this analysis with the script given below. However, I got an error ValueError: Samples must be one-dimensional. Also is this the right test to do this analysis?

My script

from scipy import stats

x = df[['guide_1']]
y = df[['guide_1_new']]
z = df[['Correlation']]
stats.kruskal(x, y, z)

• I am a little confused, you are passing the columns of guide_1 and guide_1_new as samples to scipy.stats.kruskal? Those are non-numeric data. And the correlations, while numeric, suggest to me that you are doing something a little different from what Kruskal-Wallis is implementing. It looks like you might be trying to pass in the categories as vectors of strings, but I don't think it works that way (?). And I don't think applying it to correlation coefficients will be very interpretable. Not an expert on scipy.stats but this doesn't look right. I'd suggest going back to raw data. Commented Jan 18, 2023 at 21:32
• Answered below, if you are stuck I can talk you through the calculation if you perform a df.to_dict() and post the output.
– M__
Commented Jan 18, 2023 at 23:18

Reformatting the data for the KW and doing the test is easy.

Using "guide_1" alone and pivot() to get A = [1, 0.426115 ...

df2 = df.pivot(columns='guide_1', values='Correlation')


This will now present A, B, C, D etc ... into columns...

x = df2[['A']].dropna()
y = df2[['B']].dropna()
z = df2[['C']].dropna()
a = df2[['D']].dropna()

stats.kruskal(x, y, z, a)


That will work ... whether that is a good test is a complicated question because this is log transformed data. Transformation is used in parametric tests. Removing the log transformation is strongly recommended because KW works with raw data.

You can switch the calculation to guide_1_new instead. It will give a different answer ... but I don't know the experiment. The original guide looks cool.

Looking at the data - the test should work because I believe KW is 'unpaired' (you need to check) and looking at the value the distribution looks pretty similar between A, B, C and D. However, just because the KW says 'yes' doesn't mean thats a robust answer (log data). If the untransformed data says 'yes' ... thats a robust answer for KW. Just check the 'unpaired' stuff - thats important for the result.

Personal note, I originally used a third df but was not needed,

df3 = pd.concat([df2[col].sort_values().reset_index(drop=True) for col in df2], axis=1, ignore_index=True)

• I'm a little confused here, it looks to me in the OP like the numeric input is correlation coefficients from comparing A to A, A to B, etc. So for A to A, you get an unsurprising correlation of 1, which is now included in the A group, which doesn't seem like a very informative input. Does OP really care about whether guide_1 group A is more correlated to other groups than those groups are among themselves (which is what this is testing for, effectively)? Doesn't make sense to me, shouldn't OP go back to raw (pre-correlation) data? Probably missing something here. Commented Jan 19, 2023 at 11:37
• @MaximilianPress KW will give a result representing the data. Agreed non-parametrics prefer raw data. The presence of A->A , B->B will of course bias the test, but the gist is A ~ B ~ C ~ D - and will not be 'thrown out of court'. Capturing log-change or non-log change isn't easy. A 2-way ANOVA type (4 columns vs 2 rows) would do this - absolutely, but its complicated and non-parametrics are not good at this calculation (Friedman test, statisticians avoid it). Using 1-way ANOVA on this specific data would be a mistake. Getting it 'spot on' is an investment. Right now 'it's ok'.
– M__
Commented Jan 19, 2023 at 12:37