Difference between Tensorflow convolution and numpy convolution

Question

import numpy as np
import tensorflow as tf

X_node    = tf.placeholder('float',[1,10,1])
filter_tf = tf.Variable( tf.truncated_normal([3,1,1],stddev=0.1) )

Xconv_tf_tensor = tf.nn.conv1d(X_node, filter_tf,1,'SAME')

X = np.random.normal(0,1,[1,10,1])
with tf.Session() as sess:
    tf.global_variables_initializer().run()
    feed_dict = {X_node: X}
    filter_np = filter_tf.eval()
    Xconv_tf = sess.run(Xconv_tf_tensor,feed_dict)
    Xconv_np = np.convolve(X[0,:,0],filter_np[:,0,0],'SAME')

I am trying to see the results of convolutions from Tensorflow to check if it is behaving as I intended. When I run the numpy convolution and compare it to the Tensorflow convolution, the answer is different. The above code is how I ran the test. I was hoping the Xconv_tf and Xconv_np would be equal.

My final goal is the run 2D convolution on a matrix with a 1 dimensional filter that runs 1d-convolution on each row with the same filter. In order to make this work (which will be basically a loop of 1d convolution over the rows) I need to figure out why my np.convolve and tf.conv1d give me different answers.

Answer 1

The problem that you see is because TF does not really calculate the convolution. If you will take a look at the explanation of what convolution actually does (check for Visual explanations of convolution), you will see that the second function is flipped:

1. Express each function in terms of a dummy variable
2. Reflect one of the functions (this is the flip)
3. ..... Some other stuff which I will not copy here.

TF does everything except of that flip. So all you need to do is to flip the kernel either in TF or in numpy. Flipping for 1d case is just kernel in a reverse order, for 2d you will need to flip both axis (rotate the kernel 2 times).

import tensorflow as tf
import numpy as np

I = [1, 0, 2, 3, 0, 1, 1]
K = [2, 1, 3]

i = tf.constant(I, dtype=tf.float32, name='i')
k = tf.constant(K, dtype=tf.float32, name='k')

data   = tf.reshape(i, [1, int(i.shape[0]), 1], name='data')
kernel = tf.reshape(k, [int(k.shape[0]), 1, 1], name='kernel')

res = tf.squeeze(tf.nn.conv1d(data, kernel, 1, 'VALID'))
with tf.Session() as sess:
    print sess.run(res)
    print np.convolve(I, K[::-1], 'VALID')

Answer 2

The order of filter is reversed. TensorFlow convolution is actually correlation. Numpy gets notation from math, TF gets notation from machine learning papers and somewhere the order got reversed.

This prints True

filter_np2=filter_np[::-1,0,0]
np.allclose(np.convolve(X[0,:,0],filter_np2,'SAME'),  Xconv_tf.flatten())    np.convolve(X[0,:,0],filter_np2,'SAME')