Python list comprehension to calculate the set of categories for each row

Question

My data = data

Model   A1BG    A1CF    A2M A2ML1   A3GALT2 A4GALT  foldchange
IM00499 N   L   L   G   N   N   0.285744
IM00499.1   L   N   D   G   D   D   0.005046
IM00980 N   N   N   N   N   N   -0.068949
IM01529 N   N   L   N   N   N   0.065914
IM01624 A   N   L   N   N   N   0.302100
IM01240 N   D   N   D   N   G   -0.428324
IM00807 N   L   A   N   D   N   0.257222
IM00999 N   N   N   N   N   D   0.056268
IM00983 N   N   N   N   N   N   0.107456
IM01076 N   N   D   A   N   N   0.218301
IM00803 N   D   N   N   N   D   -0.479368
IM00125 N   N   A   N   N   A   0.270428
IM00998 D   N   N   N   G   N   0.142748
IM00997 N   L   G   G   N   G   -0.154540
IM00996 N   L   N   N   N   L   -1.302023

The above data frame is divided into 5 categories = A, N, L, G and D

feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']

Feature_level_partitions are arrays of combination of classification values

feature_level_partitions = ['A', 'N' , 'L', 'G', 'D'], ['D'], ['L', 'G']

I was using the below list comprehension to calculate Hedge's G.

        hedges_g = np.array(
            [
                Hedges_g(data[feature].isin(feature_level_partition), y)
                for feature_level_partition in feature_level_partitions
            ],
            dtype=np.float64,
        )

Hedges_g is defined as follows:

def Hedges_g(x, y):
    difference_of_means = abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
    s_pooled_weighted = math.sqrt((((n1 - 1) * s1 ** 2) + ((n2 - 1) * s2 ** 2)) / (n1 + n2 - 2))
    return difference_of_means / s_pooled_weighted

I am interested in knowing how many models there are within each category. So for example if I divide up the data into ["D","L"] on one side, and the rest on the other, I want to know how many models have either "D" OR "L" and how many models have any of the other categories and then don't calculate Hegde's G if the number on either side is less than 2.

To answer the above query I tried the following code

[data[feature].isin(feature_level_partition).sum()
    for feature_level_partition in feature_level_partitions]

The above code gave me the number of models with the feature_level_partitions However I also want to know the number of models without the feature_level_partitions

I also want to create a python function which calculates both above numbers in one line and fit this code as a if statement in my above list comprehension function

Answer 1

What I'm not clear about is the definition of y - thats crucial to understanding what the code does. Just to note, although you've called the dataframe data, I've called it df (traditional shorthand). In terms of speed there will not be much in it between vectorised pandas and numpy. Explicitly using iterrows or else apply is when it slows down a lot.

Possibilities of the definition of y.

1.

y = df.drop(['Model', 'foldchange'], axis=1)

I think the above is more likely, or ...

2.

y = df['fold change']

What I think is occurring is np.mean is performing a boolean operation on the feature columns. What I didn't understand though was this ...

abs(np.mean(y[x])[0] - np.mean(y[~x])[0])

.. this is because for example model 'IM00980' albeit the mean is 0 this will always be 1 under the above equation. This is because y[~x] will be 1 for all values and thus abs(0 - 1) is 1. Equally if all columns in a row were 1 the output would also be 1 because abs(1-0) is 1. Anyway, what I would do is explicitly perform the operation within pandas rather than coercing np into a boolean operation. This is

import pandas as pd

df = pd.read_csv(r'/pathtofile/megha.csv', delimiter="\t")

feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']
df=df.drop(['foldchange'],axis=1)
feature_level_partitions = ['L', 'G']
filtered = df[feature].isin(feature_level_partitions).eq(True).mul(1)
df2 = df['Model'].to_frame().join(filtered)
df2['mean'] = df2[feature].mean(axis=1)
print(df2)

Here ~x would simply 1 - mean as a new pandas column, i.e. df2['~mean'] = df2['mean'] - 1

The output of that code is:

-	Model	A1BG	A1CF	A2M	A2ML1	A3GALT2	A4GALT	mean.
0	IM00499	0	1.	1.	1.	0.	0.	0.500000
1	IM00499.1	1	0	0	1	0	0	0.333333
2	IM00980	0	0	0	0	0	0	0.000000
3	IM01529	0	0	1	0	0	0	0.166667
4	IM01624	0	0	1	0	0	0	0.166667
5	IM01240	0	0	0	0	0	1	0.166667
6	IM00807	0	1	0	0	0	0	0.166667
7	IM00999	0	0	0	0	0	0	0.000000
8	IM00983	0	0	0	0	0	0	0.000000
9	IM01076	0	0	0	0	0	0	0.000000
10	IM00803	0	0	0	0	0	0	0.000000
11	IM00125	0	0	0	0	0	0	0.000000
12	IM00998	0	0	0	0	1	0	0.166667
13	IM00997	0	1	1	1	0	1	0.666667
14	IM00996	0	1	0	0	0	1	0.333333

---

If this is correct it provides the basis to move forward for all other operations. Basically, what I'm saying is I think is good to break open the numpy into explicit pandas operations, what is happening is there's a mix of pandas and numpy and I would recommend going down one route or the other.