官术网_书友最值得收藏!

The rectified linear unit function

The rectified linear unit, better known as ReLU, is the most widely used activation function:

The ReLU function has the advantage of being non linear. Thus, backpropagation is easy and can therefore stack multiple hidden layers activated by the ReLU function, where for x<=0, the function f(x) = 0 and for x>0, f(x)=x.

The main advantage of the ReLU function over other activation functions is that it does not activate all the neurons at the same time. This can be observed from the preceding graph of the ReLU function, where we see that if the input is negative it outputs zero and the neuron does not activate. This results in a sparse network, and fast and easy computation.

Derivative graph of ReLU, shows f'(x) = 0 for x<=0 and f'(x) = 1 for x>0

Looking at the preceding gradients graph of ReLU preceding, we can see the negative side of the graph shows a constant zero. Therefore, activations falling in that region will have zero gradients and therefore, weights will not get updated. This leads to inactivity of the nodes/neurons as they will not learn. To overcome this problem, we have Leaky ReLUs, which modify the function as:

This prevents the gradient from becoming zero in the negative side and the weight training continues, but slowly, owing to the low value of .

主站蜘蛛池模板: 金华市| 师宗县| 金寨县| 白城市| 兴业县| 罗甸县| 科技| 庄河市| 大田县| 奈曼旗| 伊通| 岳普湖县| 拉孜县| 汉阴县| 邵阳市| 新安县| 文水县| 越西县| 双柏县| 九台市| 海丰县| 武乡县| 舞钢市| 山丹县| 施甸县| 成安县| 翼城县| 屏边| 栖霞市| 融水| 泗水县| 礼泉县| 成武县| 白朗县| 峨眉山市| 衡南县| 龙江县| 游戏| 弋阳县| 武胜县| 高雄市|