How to statistically test the strength and weakness of a correlated pair?

Subset of df before groupby looks like

Diabetes_Group  DM-1    DM-2    corr
Group1  56  56  1
Group1  56  58  0.611
Group1  56  61  0.448
Group1  56  63  0.3587
Group1  58  56  0.611
Group1  58  58  1
Group1  58  61  0.60113
Group1  58  63  0.534858
Group1  61  56  0.448
Group1  61  58  0.60113
Group1  61  61  1
Group1  61  63  0.622206
Group1  63  56  0.3587
Group1  63  58  0.534858
Group1  63  61  0.622206
Group1  63  63  1
Group2  78  78  1
Group2  78  85  0.622703
Group2  78  94  0.622244
Group2  78  23  0.687826
Group2  85  78  0.622703
Group2  85  85  1
Group2  85  94  0.697313
Group2  85  23  0.612089
Group2  94  78  0.622244
Group2  94  85  0.697313
Group2  94  94  1
Group2  94  23  0.603834
Group2  23  78  0.687826
Group2  23  85  0.612089
Group2  23  94  0.603834
Group2  23  23  1
Group3  27  27  1
Group3  27  80  0.118955
Group3  27  147 0.32038
Group3  27  9   0.335264
Group3  80  27  0.118955
Group3  80  80  1
Group3  80  147 0.430287
Group3  80  9   0.406426
Group3  147 27  0.32038
Group3  147 80  0.430287
Group3  147 147 1
Group3  147 9   0.546452
Group3  9   27  0.335264
Group3  9   80  0.406426
Group3  9   147 0.546452
Group3  9   9   1

1. What statistical test can be used to do comparative analysis between the different ID's (DM1 or DM2) to check which ID pair has weaker/stronger correlation within each diabetes_group?

2. Is there any statistical test to categorise the data into three categories:

(a) pairs of DM1 AND DM2 which are highly correlated than others.

(b) pairs that are not highly correlated

(c) For the category 1 pairs (highly correlated), is there any particular ID that outperforms than other ID's among different groups?

This comprehensive analytical outline for the stats and all the stats questions could be captured inside 3 analyses. Implementing this will obviously take a bit of time, but there should not be any hidden caveats (transformations etc ...).

Question 3

The statistical test you want is K-means clustering based on the correlation score where the number of predefined groups is 3. The labels ... you will need to combine the group with the

Label          Value
Group1-56-56   1
...


This is an absolute solution to the data and really cool use of K-means.

Question 3c An easy solution is:

• perform a new K-means with e.g. 10 groups and find the group with highest correlation values.

A better solution is:

• hierarchical clustering would identify specific groups and for this sub-question is a good analysis. The input is the correlation matrix, the output is a tree. HOWEVER, interpreting this could be difficult if you're not trained in reading trees.

Question 2

A 2-way ANOVA would be a really good solution to this data for Q2. The test would be the columns were DM1 DM2 and the rows Group1, Group2, Group3. Interpreting it would be fun because you'd also have to grapple with a concept called "interaction" - which could be important here. However, the ability to address a whole raft of questions within one analysis is what 2-way ANOVA would permit. There is a slight issue of unequal group sizes 1, 2 and 3, because it would be better if Group 1,2 and 3 etc... were equal in size. However, the raw data looks to be very amenable to ANOVA. You must not use the correlation index in this test - just the raw values.

K-means clustering is in scipy and scikit-learn

Hierarchical clustering is also in scipy

For ANOVA

import statsmodels.api as sm
from statsmodels.formula.api import ols


There will be easy approaches to 2-way ANOVA e.g. SPSS and I think Excel (needs a plug in). I suspect that Hierarchical clustering will also be in a nice point and click package like SPSS (I don't use them, so I don't precisely know.)

• Is it possible for you to explain this in terms of code or any example code Commented Jan 17, 2023 at 16:49
• Hi @Megha, you have asked an entire statistical project within a single question, so I answered as best I could.What I propose is to investigate the facets (K-means is a good start) and have a go and ask a specific question if you're stuck. I've updated the post with the location of the specific packages. You don't need to stick with Python, point and click packages are perfectly fine because this is end analysis. Beyond this there is no further process in a data pipeline
– M__
Commented Jan 17, 2023 at 17:07