I have following situation where i may get 300 columns in csv file and some of them are list parameter (50 columns)and they can uneven size including no values.
First you can create the exploded columns like you want with concat
, str.split
and stack
. Use reset_index
and join
to be able to get the column 'Time' associated. Then you need to change the values in the column 'Time' to create the interpolation. I'm not sure if one can interpolate directly a datetime column, but you can change the type to int64
, mask
the values if same than previous row with shift
and interpolate
. So like this:
l_col = ['COL1', 'COL2']
df_f = pd.concat([df[col].str.split(' ', expand=True) for col in l_col ],
axis=1, keys=l_col)\
.stack()\
.reset_index(level=1, drop=True)\
.join(df[['Time']])\
.reset_index(drop=True)
df_f['Time'] = pd.to_datetime(df_f['Time'].astype('int64')
.mask(df_f.Time.eq(df_f.Time.shift()))
.interpolate(method='linear'))
print (df_f)
COL1 COL2 Time
0 0.0 50.0 2020-03-13 10:43:00.500
1 10.0 60.0 2020-03-13 10:43:00.580
2 20.0 70.0 2020-03-13 10:43:00.660
3 30.0 80.0 2020-03-13 10:43:00.740
4 40.0 90.0 2020-03-13 10:43:00.820
5 10.0 10.0 2020-03-13 10:43:00.900
6 20.0 20.0 2020-03-13 10:43:15.700
7 30.0 None 2020-03-13 10:43:30.500
8 70.0 None 2020-03-13 10:43:45.300
9 10.0 2020-03-13 10:44:00.100
10 20.0 None 2020-03-13 10:44:00.100
11 30.0 None 2020-03-13 10:44:00.100
12 70.0 None 2020-03-13 10:44:00.100
I'm not sure what you want for the missing values in COL2 e.g. so you may need some fillna
to work this out.