2
$\begingroup$

My data = data

Model   A1BG    A1CF    A2M A2ML1   A3GALT2 A4GALT  foldchange
IM00499 N   L   L   G   N   N   0.285744
IM00499.1   L   N   D   G   D   D   0.005046
IM00980 N   N   N   N   N   N   -0.068949
IM01529 N   N   L   N   N   N   0.065914
IM01624 A   N   L   N   N   N   0.302100
IM01240 N   D   N   D   N   G   -0.428324
IM00807 N   L   A   N   D   N   0.257222
IM00999 N   N   N   N   N   D   0.056268
IM00983 N   N   N   N   N   N   0.107456
IM01076 N   N   D   A   N   N   0.218301
IM00803 N   D   N   N   N   D   -0.479368
IM00125 N   N   A   N   N   A   0.270428
IM00998 D   N   N   N   G   N   0.142748
IM00997 N   L   G   G   N   G   -0.154540
IM00996 N   L   N   N   N   L   -1.302023

The above data frame is divided into 5 categories = A, N, L, G and D

feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']

Feature_level_partitions are arrays of combination of classification values

feature_level_partitions = ['A', 'N' , 'L', 'G', 'D'], ['D'], ['L', 'G']

I was using the below list comprehension to calculate Hedge's G.

        hedges_g = np.array(
            [
                Hedges_g(data[feature].isin(feature_level_partition), y)
                for feature_level_partition in feature_level_partitions
            ],
            dtype=np.float64,
        )

Hedges_g is defined as follows:

def Hedges_g(x, y):
    difference_of_means = abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
    s_pooled_weighted = math.sqrt((((n1 - 1) * s1 ** 2) + ((n2 - 1) * s2 ** 2)) / (n1 + n2 - 2))
    return difference_of_means / s_pooled_weighted

I am interested in knowing how many models there are within each category. So for example if I divide up the data into ["D","L"] on one side, and the rest on the other, I want to know how many models have either "D" OR "L" and how many models have any of the other categories and then don't calculate Hegde's G if the number on either side is less than 2.

To answer the above query I tried the following code

[data[feature].isin(feature_level_partition).sum()
    for feature_level_partition in feature_level_partitions]

The above code gave me the number of models with the feature_level_partitions However I also want to know the number of models without the feature_level_partitions

I also want to create a python function which calculates both above numbers in one line and fit this code as a if statement in my above list comprehension function

$\endgroup$
1
  • $\begingroup$ What you might think about @Megha is defining a new question in the context of both the comprehension and the models by defining a criteria for including 'foldchange' parameter. $\endgroup$
    – M__
    Feb 19 at 19:35

1 Answer 1

1
$\begingroup$

What I'm not clear about is the definition of y - thats crucial to understanding what the code does. Just to note, although you've called the dataframe data, I've called it df (traditional shorthand). In terms of speed there will not be much in it between vectorised pandas and numpy. Explicitly using iterrows or else apply is when it slows down a lot.

Possibilities of the definition of y.

1.

y = df.drop(['Model', 'foldchange'], axis=1)

I think the above is more likely, or ...

2.

y = df['fold change']

What I think is occurring is np.mean is performing a boolean operation on the feature columns. What I didn't understand though was this ...

abs(np.mean(y[x])[0] - np.mean(y[~x])[0])

.. this is because for example model 'IM00980' albeit the mean is 0 this will always be 1 under the above equation. This is because y[~x] will be 1 for all values and thus abs(0 - 1) is 1. Equally if all columns in a row were 1 the output would also be 1 because abs(1-0) is 1. Anyway, what I would do is explicitly perform the operation within pandas rather than coercing np into a boolean operation. This is

import pandas as pd

df = pd.read_csv(r'/pathtofile/megha.csv', delimiter="\t")

feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']
df=df.drop(['foldchange'],axis=1)
feature_level_partitions = ['L', 'G']
filtered = df[feature].isin(feature_level_partitions).eq(True).mul(1)
df2 = df['Model'].to_frame().join(filtered)
df2['mean'] = df2[feature].mean(axis=1)
print(df2)

Here ~x would simply 1 - mean as a new pandas column, i.e. df2['~mean'] = df2['mean'] - 1

The output of that code is:

- Model A1BG A1CF A2M A2ML1 A3GALT2 A4GALT mean.
0 IM00499 0 1. 1. 1. 0. 0. 0.500000
1 IM00499.1 1 0 0 1 0 0 0.333333
2 IM00980 0 0 0 0 0 0 0.000000
3 IM01529 0 0 1 0 0 0 0.166667
4 IM01624 0 0 1 0 0 0 0.166667
5 IM01240 0 0 0 0 0 1 0.166667
6 IM00807 0 1 0 0 0 0 0.166667
7 IM00999 0 0 0 0 0 0 0.000000
8 IM00983 0 0 0 0 0 0 0.000000
9 IM01076 0 0 0 0 0 0 0.000000
10 IM00803 0 0 0 0 0 0 0.000000
11 IM00125 0 0 0 0 0 0 0.000000
12 IM00998 0 0 0 0 1 0 0.166667
13 IM00997 0 1 1 1 0 1 0.666667
14 IM00996 0 1 0 0 0 1 0.333333

If this is correct it provides the basis to move forward for all other operations. Basically, what I'm saying is I think is good to break open the numpy into explicit pandas operations, what is happening is there's a mix of pandas and numpy and I would recommend going down one route or the other.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.