Image Gradient Vector Field in Python

I am trying to get the Gradient Vector Field of an image using Python (similar to this matlab question).

This is the original image:

Here is my code:

import numpy as np
import matplotlib.pyplot as plt
import Image
from PIL import ImageFilter

I = Image.open('test.png').transpose(Image.FLIP_TOP_BOTTOM)
I = I.filter(ImageFilter.BLUR)
p = np.asarray(I)
w,h = I.size
y, x = np.mgrid[0:h:500j, 0:w:500j]

dy, dx = np.gradient(p)
skip = (slice(None, None, 3), slice(None, None, 3))

fig, ax = plt.subplots()
im = ax.imshow(I, extent=[x.min(), x.max(), y.min(), y.max()])
ax.quiver(x[skip], y[skip], dx[skip], dy[skip])

ax.set(aspect=1, title='Quiver Plot')
plt.show()

This is the result:

The problem is that the vectors seem to be incorrect. This point gets more clear when you zoom in the image:

Why do some of the vectors point to the center as expected, while others do not?

Maybe there is an issue with the result of the call to np.gradient?

I think your strange results are, at least in part, because p is of type uint8. Even numpy diff results in clearly incorrect values for an array of this dtype. If you convert to signed integer by replacing the definition of p with the following: p = np.asarray(I).astype(int8) then the results of diff are correct. The following code gives me what looks like a reasonable field,

import numpy as np
import matplotlib.pyplot as plt
import Image
from PIL import ImageFilter

I = Image.open('./test.png')
I = I.filter(ImageFilter.BLUR)
p = np.asarray(I).astype('int8')
w,h = I.size
x, y = np.mgrid[0:h:500j, 0:w:500j]

dy, dx = np.gradient(p)
skip = (slice(None, None, 3), slice(None, None, 3))

fig, ax = plt.subplots()
im = ax.imshow(I.transpose(Image.FLIP_TOP_BOTTOM), 
               extent=[x.min(), x.max(), y.min(), y.max()])
plt.colorbar(im)
ax.quiver(x[skip], y[skip], dx[skip].T, dy[skip].T)

ax.set(aspect=1, title='Quiver Plot')
plt.show()

This gives the following: