pandas - iterate over rows and calculate - faster

问题

I already have a solution -but it is very slow (13 minutes for 800 rows). here is an example of the dataframe:

import pandas as pd
d = {'col1': [20,23,40,41,48,49,50,50], 'col2': [39,32,42,50,63,68,68,69]}
df = pd.DataFrame(data=d)
df

In a new column, I want to calculate how many of the previous values (for example three)of col2 are greater or equal than row-value of col1. i also continue the first rows.

this is my slow code:

start_at_nr = 3 #variable in which row start to calculate
df["overlap_count"] = "" #create new column

for row in range(len(df)):
    if row <= start_at_nr - 1:
       df["overlap_count"].loc[row] = "x"
    else:
       df["overlap_count"].loc[row] = (
           df["col2"].loc[row - start_at_nr:row - 1] >=
           (df["col1"].loc[row])).sum()

df

i obtain a faster solution - thank you for your time!

this is the result i obtain:

col1    col2    overlap_count
0   20  39  x
1   23  32  x
2   40  42  x
3   41  50  1
4   48  63  1
5   49  68  2
6   50  68  3
7   50  69  3

回答1:

IIUC, you can do:

df['overlap_count'] = 0
for i in range(1,start_at_nr+1):
    df['overlap_count'] += df['col1'].le(df['col2'].shift(i))

# mask the first few rows
df.iloc[:start_at_nr, -1] = np.nan

Output:

   col1  col2  overlap_count
0    20    39            NaN
1    23    32            NaN
2    40    42            NaN
3    41    50            1.0
4    48    63            1.0
5    49    68            2.0
6    50    68            3.0
7    50    69            3.0

Takes about 11ms on for 800 rows and start_at_nr=3.

回答2:

You basically compare the current value of col1 to previous 3 rows of col2 and starting the compare from row 3. You may use shift as follow

n = 3
s = ((pd.concat([df.col2.shift(x) for x in range(1,n+1)], axis=1) >= df.col1.values[:,None])
        .sum(1)[3:])

s = (pd.concat([df.col2.shift(x) for x in range(1,n+1)], axis=1).ge(df.col1,axis=0)
                                                                .sum(1)[3:])


Out[65]:
3    1
4    1
5    2
6    3
7    3
dtype: int64

To get your desired output, assign it back to df and fillna

n = 3
s = (pd.concat([df.col2.shift(x) for x in range(1,n+1)], axis=1).ge(df.col1,axis=0)
                                                                .sum(1)[3:])
df_final = df.assign(overlap_count=s).fillna('x')

Out[68]:
   col1  col2 overlap_count
0    20    39             x
1    23    32             x
2    40    42             x
3    41    50             1
4    48    63             1
5    49    68             2
6    50    68             3
7    50    69             3

回答3:

You could do it with .apply() in a single statement as follows. I have used a convenience function process_row(), which is also included below.

df.assign(OVERLAP_COUNT = (df.reset_index(drop=False).rename(
                                columns={'index': 'ID'})).apply(
                                    lambda x: process_row(x, df, offset=3), axis=1))

For More Speed: In case you need more speed and are processing a lot of rows, you may consider using swifter library. All you have to do is:

install swifter: pip install swifter.

import the library as import swifter.

replace any .apply() with .swifter.apply() in the code-block above.

Solution in Detail

#!pip install -U swifter
#import swifter 
import numpy as np
import pandas as pd

d = {'col1': [20,23,40,41,48,49,50,50], 'col2': [39,32,42,50,63,68,68,69]}
df = pd.DataFrame(data=d)

def process_row(x, df, offset=3):
    value = (df.loc[x.ID - offset:x.ID - 1, 'col2'] >= df.loc[x.ID, 'col1']).sum() if (x.ID >= offset) else 'x'
    return value

# Use df.swifter.apply() for faster processing, instead of df.apply()
df.assign(OVERLAP_COUNT = (df.reset_index(drop=False, inplace=False).rename(
                                columns={'index': 'ID'}, inplace=False)).apply(
                                    lambda x: process_row(x, df, offset=3), axis=1))

Output:

   col1  col2 OVERLAP_COUNT
0    20    39             x
1    23    32             x
2    40    42             x
3    41    50             1
4    48    63             1
5    49    68             2
6    50    68             3
7    50    69             3

来源：https://stackoverflow.com/questions/59294105/pandas-iterate-over-rows-and-calculate-faster

标签

python

pandas

loops

pandas-apply