Correlation coefficient and p value for each row within a datafarme

问题

I have a matrix, looks like following,

foo = pd.DataFrame(
        [['ASP1',12.45,12.65,1.54,1.56],
        ['ASP2',4.5,1.4,0.03,1.987],
        ['ASP3',0.12,0.34,0.45,0.9],
        ['ASP4',0.65,0.789,0.01,0.876]],
        columns = ('Sam','C1','C2','B1','B2'))
foo
    Sam C1  C2  B1  B2
0   ASP1    12.45   12.650  1.54    1.560
1   ASP2    4.50    1.400   0.03    1.987
2   ASP3    0.12    0.340   0.45    0.900
3   ASP4    0.65    0.789   0.01    0.876

And I wanted to do correlation test for each row in Sam between the columns C1..C2 and B1..B2. And at the end, I am aiming a resulting matrix as follows,

foo_result = pd.DataFrame(
        [['C',0.76,0.06],
        ['B',0.34,0.10]],
        columns = ('Gen','Correlation_coefficent','P-value'))
foo_result

    Gene    Correlation_coefficent  P-value
0   C   0.76    0.060
1   B   0.34    0.100

Any suggestions or solutions would be great. Thank you

回答1:

This should do it:

from scipy.stats import pearsonr

c_values = [column for column in foo.columns.tolist() if column.startswith('C')]
b_values = [column for column in foo.columns.tolist() if column.startswith('B')]

foo['Correlation_coefficent'], foo['P-value'] = zip(*foo.T.apply(lambda x: pearsonr(x[c_values], x[b_values])))
foo_result = foo[['Sam', 'Correlation_coefficent','P-value']]

Output:

    Sam  Correlation_coefficent  P-value
0  ASP1                     1.0      0.0
1  ASP2                    -1.0      0.0
2  ASP3                     1.0      0.0
3  ASP4                     1.0      0.0

Reason why you have these results is the number of variables. Hope your original has at least 3 values.

回答2:

I would use DataFrame.apply

from scipy.stats import pearsonr
foo[['corr_coef', 'p_value']] = foo.apply(lambda x: pearsonr(x=x[1:3], y=x[3:5]), axis=1).apply(pd.Series)

The output is

    Sam     C1      C2    B1     B2  corr_coef  p_value
0  ASP1  12.45  12.650  1.54  1.560        1.0      0.0
1  ASP2   4.50   1.400  0.03  1.987       -1.0      0.0
2  ASP3   0.12   0.340  0.45  0.900        1.0      0.0
3  ASP4   0.65   0.789  0.01  0.876        1.0      0.0

If you have 112 columns of each C and B, you should use pearsonr(x=x[1:113], y=x[113:125])

来源：https://stackoverflow.com/questions/43844463/correlation-coefficient-and-p-value-for-each-row-within-a-datafarme

标签

python

pandas

numpy

stat