官术网_书友最值得收藏!

  • Deep Learning with R for Beginners
  • Mark Hodnett Joshua F. Wiley Yuxi (Hayden) Liu Pablo Maldonado
  • 319字
  • 2021-06-24 14:30:44

Activation functions

The activation function determines the mapping between input and a hidden layer. It defines the functional form for how a neuron gets activated. For example, a linear activation function could be defined as: f(x) = x, in which case the value for the neuron would be the raw input, x. A linear activation function is shown in the top panel of Figure 4.2. Linear activation functions are rarely used because in practice deep learning models would find it difficult to learn non-linear functional forms using linear activation functions. In previous chapters, we used the hyperbolic tangent as an activation function, namely f(x) = tanh(x). Hyperbolic tangent can work well in some cases, but a potential limitation is that at either low or high values, it saturates, as shown in the middle panel of the figure  4.2.

Perhaps the most popular activation function currently, and a good first choice (Nair, V., and Hinton, G. E. (2010)), is known as a rectifier. There are different kinds of rectifiers, but the most common is defined by the f(x) = max(0, x) function, which is known as relu. The relu activation is flat below zero and linear above zero; an example is shown in Figure 4.2.

The final type of activation function we will discuss is maxout (Goodfellow, Warde--Farley, Mirza, Courville, and Bengio (2013)). A maxout unit takes the maximum value of its input, although as usual, this is after weighting so it is not the case that the input variable with the highest value will always win. Maxout activation functions seem to work particularly well with dropout.

The relu activation is the most commonly-used activation function and it is the default option for the deep learning models in the rest of this book. The following graphs for some of the activation functions we have discussed:

Figure 4.2: Common activation functions
主站蜘蛛池模板: 湖州市| 曲松县| 九龙县| 南川市| 黑龙江省| 定襄县| 浦城县| 定襄县| 莱州市| 个旧市| 呼玛县| 冷水江市| 葫芦岛市| 页游| 静宁县| 荃湾区| 清苑县| 绵阳市| 上思县| 内江市| 万载县| 潮州市| 济宁市| 湘潭县| 桃源县| 合肥市| 汉沽区| 府谷县| 山阴县| 乐至县| 洛阳市| 广水市| 和林格尔县| 平阳县| 柳江县| 台江县| 辉南县| 昭通市| 麻栗坡县| 五河县| 怀仁县|