Implement Relu derivative in python numpy

I'm trying to implement a function that computes the Relu derivative for each element in a matrix, and then return the result in a matrix. I'm using Python and Numpy.

Based on other Cross Validation posts, the Relu derivative for x is 1 when x > 0, 0 when x < 0, undefined or 0 when x == 0

Currently, I have the following code so far:

def reluDerivative(self, x):
    return np.array([self.reluDerivativeSingleElement(xi) for xi in x])

def reluDerivativeSingleElement(self, xi):
    if xi > 0:
        return 1
    elif xi <= 0:
        return 0

Unfortunately, xi is an array because x is an matrix. reluDerivativeSingleElement function doesn't work on array. So I'm wondering is there a way to map values in a matrix to another matrix using numpy, like the exp function in numpy?

Thanks a lot in advance.

I guess this is what you are looking for:

>>> def reluDerivative(x):
...     x[x<=0] = 0
...     x[x>0] = 1
...     return x

>>> z = np.random.uniform(-1, 1, (3,3))
>>> z
array([[ 0.41287266, -0.73082379,  0.78215209],
       [ 0.76983443,  0.46052273,  0.4283139 ],
       [-0.18905708,  0.57197116,  0.53226954]])
>>> reluDerivative(z)
array([[ 1.,  0.,  1.],
       [ 1.,  1.,  1.],
       [ 0.,  1.,  1.]])

That's an exercise in vectorization.

This code

if x > 0:
  y = 1
elif xi <= 0:
  y = 0

Can be reformulated into

y = (x > 0) * 1

This is something that will work for numpy arrays, since boolean expressions involving them are turned into arrays of values of these expressions for elements in said array.

Basic function to return derivative of relu could be summarized as follows:

f'(x) = x > 0

So, with numpy that would be:

def relu_derivative(z):
    return np.greater(z, 0).astype(int)

def dRelu(z):
    return np.where(z <= 0, 0, 1)

Here z is a ndarray in my case.

You are on a good track: thinking on vectorized operation. Where we define a function, and we apply this function to a matrix, instead of writing a for loop.

This threads answers your question, where it replace all the elements satisfy the condition. You can modify it into ReLU derivative.

https://stackoverflow.com/questions/19766757/replacing-numpy-elements-if-condition-is-met

In addition, python supports functional programming very well, try to use lambda function.

https://www.python-course.eu/lambda.php

This works:

def dReLU(x):
    return 1. * (x > 0)

As mentioned by Neil in the comments, you can use heaviside function of numpy.

def reluDerivative(self, x):
    return np.heaviside(x, 0)

If you want to use pure Python:

def relu_derivative(x):
    return max(sign(x), 0)

user3503711

def reluDerivative(self, x): 
    return 1 * (x > 0)

When x is larger than 0, the slope is 1. When x is smaller than or equal to 0, the slope is 0.

if (x > 0):
    return 1
if (x <= 0):
    return 0

This can be written more compact:

return 1 * (x > 0)

来源：https://stackoverflow.com/questions/46411180/implement-relu-derivative-in-python-numpy

标签

machine-learning

python

derivative

numpy