Python Numpy or Pandas Linear Interpolation For Datetime related Values

Question

I have data that looks like the following but I also have control of how it is formatted. Basically, I want to use Python with Numpy or Pandas to interpolate the dataset to achieve second by second interpolated data so that it is a much higher resolution.

So I want to linearly interpolate and produce new values between each of the real values I currently have while keeping the original values as well.

How can I accomplish this with Pandas or Numpy?

As an example, I have this type of data:

       TIME               ECI_X            ECI_Y          ECI_Z
 2013-12-07 00:00:00, -7346664.77912, -13323447.6311, 21734849.5263,@
 2013-12-07 00:01:00, -7245621.40363, -13377562.35, 21735850.3527,@
 2013-12-07 00:01:30, -7142326.20854, -13432541.9267, 21736462.4521,@
 2013-12-07 00:02:00, -7038893.48454, -13487262.8599, 21736650.3293,@
 2013-12-07 00:02:30, -6935325.24526, -13541724.0946, 21736413.9937,@
 2013-12-07 00:03:00, -6833738.23865, -13594806.9333, 21735778.2218,@
 2013-12-07 00:03:30, -6729905.37597, -13648746.6281, 21734705.6406,@
 2013-12-07 00:04:00, -6625943.01291, -13702423.5112, 21733208.9233,@
 2013-12-07 00:04:30, -6521853.17291, -13755836.5481, 21731288.1125,@
 2013-12-07 00:05:00, -6419753.85176, -13807871.3011, 21729016.1386,@
 2013-12-07 00:05:30, -6315415.32918, -13860754.6497, 21726259.4135,@
 2013-12-07 00:06:00, -6210955.33186, -13913371.1187, 21723078.7695,@
 ...

And I'd like it to be second by second - i.e.

 2013-12-07 00:00:00, -7346664.77912, -13323447.6311, 21734849.5263,@
 2013-12-07 00:00:01, -7346665.10000, -13323448.1000, 21734850.1000,@
 ...
 2013-12-07 00:00:59, -7346611.10000, -13323461.1000, 21734850.1000,@
 2013-12-07 00:01:00, -7245621.40363, -13377562.3500, 21735850.3527,@

Please show me an example of how I can accomplish this. Thanks!

I've tried this:

#! /usr/bin/python

import datetime
from pandas import *

first = datetime(2013,12,8,0,0,0)
second = datetime(2013,12,8,0,2,0)
dates = [first,second]
x = np.array([617003.390723, 884235.38059])
newRange =  date_range(first, second, freq='S')
ts = Series(x, index=dates)
ts.interpolate()
print ts.head()

#2013-12-08 00:00:00, 617003.390723, -26471116.2566, 3974868.93334,@
#2013-12-08 00:02:00, 884235.38059, -26519366.9219, 3601627.52947,@

How do I use the "newRange" to create linearly interpolated values between the real values in "x"?

Answer 1

Using pandas git master (98e48ca) you can do the following:

In [27]: n = 4

In [28]: df = DataFrame(randn(n, 2), index=date_range('1/1/2001', periods=n, freq='30S'))

In [29]: resampled = df.resample('S')

In [30]: resampled.head()
Out[30]:
                         0      1
2001-01-01 00:00:00 -1.045 -1.067
2001-01-01 00:00:01    NaN    NaN
2001-01-01 00:00:02    NaN    NaN
2001-01-01 00:00:03    NaN    NaN
2001-01-01 00:00:04    NaN    NaN

[5 rows x 2 columns]

In [31]: interp = resampled.interpolate()

In [32]: interp.head()
Out[32]:
                         0      1
2001-01-01 00:00:00 -1.045 -1.067
2001-01-01 00:00:01 -1.014 -1.042
2001-01-01 00:00:02 -0.983 -1.018
2001-01-01 00:00:03 -0.952 -0.993
2001-01-01 00:00:04 -0.921 -0.969

[5 rows x 2 columns]

In [33]: interp.tail()
Out[33]:
                         0      1
2001-01-01 00:01:26  0.393  0.622
2001-01-01 00:01:27  0.337  0.571
2001-01-01 00:01:28  0.281  0.519
2001-01-01 00:01:29  0.225  0.468
2001-01-01 00:01:30  0.169  0.416

[5 rows x 2 columns]

By default Series.interpolate() performs linear interpolation. You can use DataFrame.resample() with irregularly sampled data as well.

Answer 2

Ok, I did this:

first = datetime(2013,12,8,0,0,0)
second = datetime(2013,12,8,0,2,0)
dates = [first,second]
x = np.array([617003.390723, 884235.38059])
newRange =  date_range(first, second, freq='S')
z = np.array([x[0]])
for i in range(1,len(newRange)-1):
    z = np.append(z,np.array([np.nan]))
z = np.append(z,np.array([1]))
print len(z)
print len(newRange)
ts = Series(z, index=newRange)
ts = ts.interpolate()
print ts.head()