官术网_书友最值得收藏!

The rectified linear unit function

The rectified linear unit, better known as ReLU, is the most widely used activation function:

The ReLU function has the advantage of being non linear. Thus, backpropagation is easy and can therefore stack multiple hidden layers activated by the ReLU function, where for x<=0, the function f(x) = 0 and for x>0, f(x)=x.

The main advantage of the ReLU function over other activation functions is that it does not activate all the neurons at the same time. This can be observed from the preceding graph of the ReLU function, where we see that if the input is negative it outputs zero and the neuron does not activate. This results in a sparse network, and fast and easy computation.

Derivative graph of ReLU, shows f'(x) = 0 for x<=0 and f'(x) = 1 for x>0

Looking at the preceding gradients graph of ReLU preceding, we can see the negative side of the graph shows a constant zero. Therefore, activations falling in that region will have zero gradients and therefore, weights will not get updated. This leads to inactivity of the nodes/neurons as they will not learn. To overcome this problem, we have Leaky ReLUs, which modify the function as:

This prevents the gradient from becoming zero in the negative side and the weight training continues, but slowly, owing to the low value of .

主站蜘蛛池模板: 安吉县| 成都市| 马尔康县| 孟村| 金山区| 黎平县| 绥滨县| 白水县| 集贤县| 启东市| 河池市| 宿松县| 丰顺县| 册亨县| 关岭| 左云县| 大港区| 鲁山县| 柯坪县| 九龙坡区| 海宁市| 故城县| 体育| 建德市| 枣阳市| 拉孜县| 开封县| 左贡县| 大安市| 新丰县| 老河口市| 永善县| 霍山县| 扎鲁特旗| 太仆寺旗| 皮山县| 满洲里市| 西安市| 迭部县| 大庆市| 鸡泽县|