I need to do auto-correlation of a set of numbers, which as I understand it is just the correlation of the set with itself.
I've tried it using numpy's correlate function, but I don't believe the result, as it almost always gives a vector where the first number is not the largest, as it ought to be.
So, this question is really two questions:
To answer your first question, For your second question, I think autocorrelation(a) = ∑ -∞ aivt+i where 0 But what you got was: autocorrelation(a) = ∑ -∞ aivt+i where -∞ What you need to do is take the last half of your correlation result, and that should be the autocorrelation you are looking for. A simple python function to do that would be: You will, of course, need error checking to make sure that numpy.correlate(a, v, mode) is performing the convolution of a with the reverse of v and giving the results clipped by the specified mode. The definition of convolution, C(t)=∑ -∞ aivt+i where -∞ t where both a and v have some overlap. a or v). a and v completely overlap each other. The documentation for numpy.convolve gives more detail on the modes.numpy.correlate is giving you the autocorrelation, it is just giving you a little more as well. The autocorrelation is used to find how similar a signal, or function, is to itself at a certain time difference. At a time difference of 0, the auto-correlation should be the highest because the signal is identical to itself, so you expected that the first element in the autocorrelation result array would be the greatest. However, the correlation is not starting at a time difference of 0. It starts at a negative time difference, closes to 0, and then goes positive. That is, you were expecting: def autocorr(x): result = numpy.correlate(x, x, mode='full') return result[result.size/2:] x is actually a 1-d array. Also, this explanation probably isn't the most mathematically rigorous. I've been throwing around infinities because the definition of convolution uses them, but that doesn't necessarily apply for autocorrelation. So, the theoretical portion of this explanation may be slightly wonky, but hopefully the practical results are helpful. These pages on autocorrelation are pretty helpful, and can give you a much better theoretical background if you don't mind wading through the notation and heavy concepts.
Using the numpy.corrcoef function instead of numpy.correlate to calculate the statistical correlation for a lag of t:
def autocorr(x, t=1): numpy.corrcoef(numpy.array([x[0:len(x)-t], x[t:len(x)]])) Auto-correlation comes in two versions: statistical and convolution. They both do the same, except for a little detail: The former is normalized to be on the interval [-1,1]. Here is an example of how you do the statistical one:
def acf(x, length=20): return numpy.array([1]+[numpy.corrcoef(x[:-i], x[i:]) \ for i in range(1, length)]) As I just ran into the same problem, I would like to share a few lines of code with you. In fact there are several rather similar posts about autocorrelation in stackoverflow by now. If you define the autocorrelation as a(x, L) = sum(k=0,N-L-1)((xk-xbar)*(x(k+L)-xbar))/sum(k=0,N-1)((xk-xbar)**2) [this is the definition given in IDL's a_correlate function and it agrees with what I see in answer 2 of question #12269834], then the following seems to give the correct results:
import numpy as np import matplotlib.pyplot as plt # generate some data x = np.arange(0.,6.12,0.01) y = np.sin(x) # y = np.random.uniform(size=300) yunbiased = y-np.mean(y) ynorm = np.sum(yunbiased**2) acor = np.correlate(yunbiased, yunbiased, "same")/ynorm # use only second half acor = acor[len(acor)/2:] plt.plot(acor) plt.show() As you see I have tested this with a sin curve and a uniform random distribution, and both results look like I would expect them. Note that I used mode="same" instead of mode="full" as the others did.
Your question 1 has been already extensively discussed in several excellent answers here.
I thought to share with you a few lines of code that allow you to compute the autocorrelation of a signal based only on the mathematical properties of the autocorrelation. That is, the autocorrelation may be computed in the following way:
subtract the mean from the signal and obtain an unbiased signal
compute the Fourier transform of the unbiased signal
compute the power spectral density of the signal, by taking the square norm of each value of the Fourier transform of the unbiased signal
compute the inverse Fourier transform of the power spectral density
normalize the inverse Fourier transform of the power spectral density by the sum of the squares of the unbiased signal, and take only half of the resulting vector
The code to do this is the following:
def autocorrelation (x) : """ Compute the autocorrelation of the signal, based on the properties of the power spectral density of the signal. """ xp = x-np.mean(x) f = np.fft.fft(xp) p = np.array([np.real(v)**2+np.imag(v)**2 for v in f]) pi = np.fft.ifft(p) return np.real(pi)[:x.size/2]/np.sum(xp**2) I use talib.CORREL for autocorrelation like this, I suspect you could do the same with other packages:
def autocorrelate(x, period): # x is a deep indicator array # period of sample and slices of comparison # oldest data (period of input array) may be nan; remove it x = x[-np.count_nonzero(~np.isnan(x)):] # subtract mean to normalize indicator x -= np.mean(x) # isolate the recent sample to be autocorrelated sample = x[-period:] # create slices of indicator data correls = [] for n in range((len(x)-1), period, -1): alpha = period + n slices = (x[-alpha:])[:period] # compare each slice to the recent sample correls.append(ta.CORREL(slices, sample, period)[-1]) # fill in zeros for sample overlap period of recent correlations for n in range(period,0,-1): correls.append(0) # oldest data (autocorrelation period) will be nan; remove it correls = np.array(correls[-np.count_nonzero(~np.isnan(correls)):]) return correls # CORRELATION OF BEST FIT # the highest value correlation max_value = np.max(correls) # index of the best correlation max_index = np.argmax(correls) I think the real answer to the OP's question is succinctly contained in this excerpt from the Numpy.correlate documentation:
mode : {'valid', 'same', 'full'}, optional Refer to the `convolve` docstring. Note that the default is `valid`, unlike `convolve`, which uses `full`. This implies that, when used with no 'mode' definition, the Numpy.correlate function will return a scalar, when given the same vector for its two input arguments (i.e. - when used to perform autocorrelation).