Given the following high-frequency but sparse time series:
#Sparse Timeseries
dti1 = pd.date_range(start=datetime(2015,8,1,9,0,0),periods=10,freq='ms')
dti2 = pd.date_range(start=datetime(2015,8,1,9,0,10),periods=10,freq='ms')
dti = dti1 + dti2
ts = pd.Series(index=dti, data=range(20))
I can compute an exponentially weighted moving average with a halflife of 5ms using a pandas function as follows:
ema = pd.ewma(ts, halflife=5, freq='ms')
However, under the hood, the function is resampling my timeseries with an interval of 1 ms (which is the 'freq' that I supplied). This causes thousands of additional datapoints to be included in the output.
In [118]: len(ts)
Out[118]: 20
In [119]: len(ema)
Out[119]: 10010
This is not scalable, as my real Timeseries contains hundreds of thousands of high-frequency observations that are minutes or hours apart.
Is there a Pandas/numpy way of computing an EMA for a sparse timeseries without resampling? Something similar to this: http://oroboro.com/irregular-ema/
Or, do i have to write my own? Thanks!
You can use reindex to align the ewma result with your original series.
pd.ewma(ts, halflife=5, freq='ms').reindex(ts.index)
2015-08-01 09:00:00.000 0.0000
2015-08-01 09:00:00.001 0.5346
2015-08-01 09:00:00.002 1.0921
2015-08-01 09:00:00.003 1.6724
2015-08-01 09:00:00.004 2.2750
2015-08-01 09:00:00.005 2.8996
2015-08-01 09:00:00.006 3.5458
2015-08-01 09:00:00.007 4.2131
2015-08-01 09:00:00.008 4.9008
2015-08-01 09:00:00.009 5.6083
2015-08-01 09:00:10.000 10.0000
2015-08-01 09:00:10.001 10.5346
2015-08-01 09:00:10.002 11.0921
2015-08-01 09:00:10.003 11.6724
2015-08-01 09:00:10.004 12.2750
2015-08-01 09:00:10.005 12.8996
2015-08-01 09:00:10.006 13.5458
2015-08-01 09:00:10.007 14.2131
2015-08-01 09:00:10.008 14.9008
2015-08-01 09:00:10.009 15.6083
dtype: float64
来源：https://stackoverflow.com/questions/31769047/compute-ewma-over-sparse-irregular-timeseries-in-pandas