Signal fitting models in Python

Question

I have a dataset shared in the following link:

https://drive.google.com/open?id=0B2Iv8dfU4fTUSV8wMmUwVGMyRE0

A simple plot of the estimated_data.csv

Answer 1

A Kalman Filter is useful for input data which has the true mean value with Gaussian noise added. You measure or otherwise know the variance of the random noise and supply that to the algorithm as the measurement noise. Your input data (which you call "estimated signal") is does not have (on average) the value of your actual signal. It appears to have an error which causes it to measure near zero very often, and rarely higher than the actual value. It only appears to exceed the actual value as a warning that there is about to be a sharp drop in the actual value.

In a case like this, a Kalman filter is unlikely to help you, because its assumptions about the input data are strongly violated by this dataset. Your best bet would be to improve your input data (in this case, for example, the error could be due to a flaw in a sensor). If that's not possible, your own intuition about the data (taking a rolling maximum) is more consistent with the behavior than a KF could be.