Remove jumps like peaks and steps in timeseries

Question

I have quite a few sensors in the field that measure water pressure. In the past the height of these sensors have been changed quite a few times creating jumps in the timese

Answer 1

You have sharp peaks and steps in your data. I guess you want to

• remove the peaks and replace by some averaged values
• remove the steps by cumulative changing the offset of the remaining data values

That's in line with what you said in your last comment. Please note, that this will alter (shift) big parts of your data!

It's important to recognize that the width of both, peaks and steps, is one pixel in your data. Also you can handle both effects pretty much independently.

I suggest to first remove the peaks, then remove the steps.

1. Remove peaks by calculating the absolute difference to the previous and to the next data value, then take the minimum of both, i.e. if your data series is y(i) compute p(i)=min(abs(y(i)-y(i-1)), abs(y(i+1)-y(i))). All values above a threshold are peaks. Take them and replace the data values with the mean of the previous and the next pixel like.

2. Now remove the steps, again by looking for absolute differences of consecutive values (as suggested in the comment by AreTor), s(i)=abs(y(i)-y(i-1)) and look for values above a certain threshold. The positions are the step positions. Create an zero-valued offset array of the same size, then insert the differences of the data points (without the absolute value), then form the cumulative sum and subtract the result from the original data to remove the steps.

Please note that this removes peaks and steps which go up as well as down. If you want to remove only one kind, just don't take the absolute value.

Answer 2

You can try it like this:

import numpy as np
import matplotlib.pyplot as plt
import h5py
%matplotlib inline

# I'm not sure that you need all of this packedges

filepath = 'measurment.hdf5'

with h5py.File(filepath, 'r') as hdf:
    data_y = hdf['y'][:]
    data_x = hdf['x'][:]

data = data_y

delta_max = 1 # maximum difference in y between two points
delta = 0 # running correction value
data_cor = [] # corrected array
data_cor.append(data[0:1]) # we append two first points

for i in range(len(data_x)-2): # two first points are allready appended
    i += 2
    delta_i = data[i] - data[i-1]
    if np.abs(delta_i) > delta_max:
        delta += (delta_i - (data_cor[i-1] - data_cor[i-2]))
        data_cor.append(data[i]-delta)
    else:
        data_cor.append(data[i]-delta)

plt.plot(data_x, data_cor)