Python 如何在Numpy中实现ReLU函数

How to implement the ReLU function in Numpy

提问by Andoni Zubizarreta

I want to make a simple neural network and I wish to use the ReLU function. Can someone give me a clue of how can I implement the function using numpy. Thanks for your time!

我想制作一个简单的神经网络，并希望使用 ReLU 函数。有人能给我一个关于如何使用 numpy.js 实现该功能的线索吗？谢谢你的时间！

采纳答案by Sid

There are a couple of ways.

有几种方法。

>>> x = np.random.random((3, 2)) - 0.5
>>> x
array([[-0.00590765,  0.18932873],
       [-0.32396051,  0.25586596],
       [ 0.22358098,  0.02217555]])
>>> np.maximum(x, 0)
array([[ 0.        ,  0.18932873],
       [ 0.        ,  0.25586596],
       [ 0.22358098,  0.02217555]])
>>> x * (x > 0)
array([[-0.        ,  0.18932873],
       [-0.        ,  0.25586596],
       [ 0.22358098,  0.02217555]])
>>> (abs(x) + x) / 2
array([[ 0.        ,  0.18932873],
       [ 0.        ,  0.25586596],
       [ 0.22358098,  0.02217555]])

If timing the results with the following code:

如果使用以下代码对结果进行计时：

import numpy as np

x = np.random.random((5000, 5000)) - 0.5
print("max method:")
%timeit -n10 np.maximum(x, 0)

print("multiplication method:")
%timeit -n10 x * (x > 0)

print("abs method:")
%timeit -n10 (abs(x) + x) / 2

We get:

我们得到：

max method:
10 loops, best of 3: 239 ms per loop
multiplication method:
10 loops, best of 3: 145 ms per loop
abs method:
10 loops, best of 3: 288 ms per loop

So the multiplication seems to be the fastest.

所以乘法似乎是最快的。

回答by Richard M?hn

If you don't mind xbeing modified, use np.maximum(x, 0, x). This was pointed out by Daniel S. It is much faster and because people might overlook it, I'll repost it as an answer. Here's the comparison:

如果您不介意x被修改，请使用np.maximum(x, 0, x). Daniel S指出了这一点。它要快得多，因为人们可能会忽略它，所以我将其重新发布为答案。这是比较：

max method:
10 loops, best of 3: 238 ms per loop
multiplication method:
10 loops, best of 3: 128 ms per loop
abs method:
10 loops, best of 3: 311 ms per loop
in-place max method:
10 loops, best of 3: 38.4 ms per loop

回答by Tobias

I found a faster method for ReLU with numpy. You can use the fancy index feature of numpy as well.

我用 numpy 找到了一种更快的 ReLU 方法。您也可以使用 numpy 的精美索引功能。

fancy index:

花式指数：

20.3ms ± 272 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

每个循环20.3ms ± 272 μs（7 次运行的平均值 ± 标准偏差，每次 10 次循环）

>>> x = np.random.random((5,5)) - 0.5 
>>> x
array([[-0.21444316, -0.05676216,  0.43956365, -0.30788116, -0.19952038],
       [-0.43062223,  0.12144647, -0.05698369, -0.32187085,  0.24901568],
       [ 0.06785385, -0.43476031, -0.0735933 ,  0.3736868 ,  0.24832288],
       [ 0.47085262, -0.06379623,  0.46904916, -0.29421609, -0.15091168],
       [ 0.08381359, -0.25068492, -0.25733763, -0.1852205 , -0.42816953]])
>>> x[x<0]=0
>>> x
array([[ 0.        ,  0.        ,  0.43956365,  0.        ,  0.        ],
       [ 0.        ,  0.12144647,  0.        ,  0.        ,  0.24901568],
       [ 0.06785385,  0.        ,  0.        ,  0.3736868 ,  0.24832288],
       [ 0.47085262,  0.        ,  0.46904916,  0.        ,  0.        ],
       [ 0.08381359,  0.        ,  0.        ,  0.        ,  0.        ]])

Here is my benchmark:

这是我的基准：

import numpy as np
x = np.random.random((5000, 5000)) - 0.5
print("max method:")
%timeit -n10 np.maximum(x, 0)
print("max inplace method:")
%timeit -n10 np.maximum(x, 0,x)
print("multiplication method:")
%timeit -n10 x * (x > 0)
print("abs method:")
%timeit -n10 (abs(x) + x) / 2
print("fancy index:")
%timeit -n10 x[x<0] =0

max method:
241 ms ± 3.53 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
max inplace method:
38.5 ms ± 4 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
multiplication method:
162 ms ± 3.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
abs method:
181 ms ± 4.18 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
fancy index:
20.3 ms ± 272 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

回答by Shital Shah

You can do it in much easier way:

你可以用更简单的方式做到这一点：

def ReLU(x):
    return x * (x > 0)

def dReLU(x):
    return 1. * (x > 0)

回答by Patel Sunil

This is more precise implementation:

这是更精确的实现：

def ReLU(x):
    return abs(x) * (x > 0)

回答by ivanpp

Richard M?hn's comparisonis not fair.
As Andrea Di Biagio's comment, the in-place method np.maximum(x, 0, x)will modify x at the first loop.
So here is my benchmark:

Richard M?hn 的比较是不公平的。
正如Andrea Di Biagio 的评论，就地方法np.maximum(x, 0, x)将在第一个循环中修改 x。
所以这是我的基准：

import numpy as np

def baseline():
    x = np.random.random((5000, 5000)) - 0.5
    return x

def relu_mul():
    x = np.random.random((5000, 5000)) - 0.5
    out = x * (x > 0)
    return out

def relu_max():
    x = np.random.random((5000, 5000)) - 0.5
    out = np.maximum(x, 0)
    return out

def relu_max_inplace():
    x = np.random.random((5000, 5000)) - 0.5
    np.maximum(x, 0, x)
    return x 

Timing it:

计时：

print("baseline:")
%timeit -n10 baseline()
print("multiplication method:")
%timeit -n10 relu_mul()
print("max method:")
%timeit -n10 relu_max()
print("max inplace method:")
%timeit -n10 relu_max_inplace()

Get the results:

获取结果：

baseline:
10 loops, best of 3: 425 ms per loop
multiplication method:
10 loops, best of 3: 596 ms per loop
max method:
10 loops, best of 3: 682 ms per loop
max inplace method:
10 loops, best of 3: 602 ms per loop

In-place maximum method is only a bit faster than the maximum method, and it may because it omits the variable assignment for 'out'. And it's still slower than the multiplication method.
And since you're implementing the ReLU func. You may have to save the 'x' for backprop through relu. E.g.:

就地最大值方法只比最大值方法快一点，这可能是因为它省略了“out”的变量赋值。而且还是比乘法慢。
并且由于您正在实施 ReLU 函数。您可能必须通过 relu 为反向传播保存“x”。例如：

def relu_backward(dout, cache):
    x = cache
    dx = np.where(x > 0, dout, 0)
    return dx

So i recommend you to use multiplication method.

所以我建议你使用乘法方法。

回答by Boooooooooms

If we have 3 parameters (t0, a0, a1)for Relu, that is we want to implement

如果我们有 3 个(t0, a0, a1)Relu参数，那就是我们要实现

if x > t0:
    x = x * a1
else:
    x = x * a0

We can use the following code:

我们可以使用以下代码：

X = X * (X > t0) * a1 +  X * (X < t0) * a0

Xthere is a matrix.

X有一个矩阵。

回答by Donald Su

numpy didn't have the function of relu, but you define it by yourself as follow:

numpy没有relu的功能，你自己定义如下：

def relu(x):
    return np.maximum(0, x)

for example:

例如：

arr = np.array([[-1,2,3],[1,2,3]])

ret = relu(arr)
print(ret) # print [[0 2 3] [1 2 3]]