官术网_书友最值得收藏!

The rectified linear unit function

The rectified linear unit, better known as ReLU, is the most widely used activation function:

The ReLU function has the advantage of being non linear. Thus, backpropagation is easy and can therefore stack multiple hidden layers activated by the ReLU function, where for x<=0, the function f(x) = 0 and for x>0, f(x)=x.

The main advantage of the ReLU function over other activation functions is that it does not activate all the neurons at the same time. This can be observed from the preceding graph of the ReLU function, where we see that if the input is negative it outputs zero and the neuron does not activate. This results in a sparse network, and fast and easy computation.

Derivative graph of ReLU, shows f'(x) = 0 for x<=0 and f'(x) = 1 for x>0

Looking at the preceding gradients graph of ReLU preceding, we can see the negative side of the graph shows a constant zero. Therefore, activations falling in that region will have zero gradients and therefore, weights will not get updated. This leads to inactivity of the nodes/neurons as they will not learn. To overcome this problem, we have Leaky ReLUs, which modify the function as:

This prevents the gradient from becoming zero in the negative side and the weight training continues, but slowly, owing to the low value of .

主站蜘蛛池模板: 册亨县| 丽水市| 共和县| 绿春县| 明水县| 信丰县| 伊金霍洛旗| 施甸县| 石河子市| 灵台县| 安宁市| 潼关县| 双江| 江口县| 巩义市| 双桥区| 张家界市| 仪征市| 顺义区| 长武县| 盐山县| 郴州市| 南开区| 高青县| 五家渠市| 启东市| 玉环县| 赣州市| 龙口市| 将乐县| 南丰县| 潞城市| 措美县| 清涧县| 洪雅县| 襄垣县| 雷州市| 花莲市| 巧家县| 长治县| 调兵山市|