Filtering out outliers in Pandas dataframe with rolling median

Question

I am trying to filter out some outliers from a scatter plot of GPS elevation displacements with dates

I\'m trying to use df.rolling to compute a median and standard

Answer 1

There might well be a more pandastic way to do this - this is a bit of a hack, relying on a sorta manual way of mapping the original df's index to each rolling window. (I picked size 6). The records up and until row 6 are associated with the first window; row 7 is the second window, and so on.

n = 100
df = pd.DataFrame(np.random.randint(0,n,size=(n,2)), columns = ['a','b'])

## set window size
window=6
std = 1  # I set it at just 1; with real data and larger windows, can be larger

## create df with rolling stats, upper and lower bounds
bounds = pd.DataFrame({'median':df['b'].rolling(window).median(),
'std':df['b'].rolling(window).std()})

bounds['upper']=bounds['median']+bounds['std']*std
bounds['lower']=bounds['median']-bounds['std']*std

## here, we set an identifier for each window which maps to the original df
## the first six rows are the first window; then each additional row is a new window
bounds['window_id']=np.append(np.zeros(window),np.arange(1,n-window+1))

## then we can assign the original 'b' value back to the bounds df
bounds['b']=df['b']

## and finally, keep only rows where b falls within the desired bounds
bounds.loc[bounds.eval("lower