- Deep Learning with R for Beginners
- Mark Hodnett Joshua F. Wiley Yuxi (Hayden) Liu Pablo Maldonado
- 319字
- 2021-06-24 14:30:44
Activation functions
The activation function determines the mapping between input and a hidden layer. It defines the functional form for how a neuron gets activated. For example, a linear activation function could be defined as: f(x) = x, in which case the value for the neuron would be the raw input, x. A linear activation function is shown in the top panel of Figure 4.2. Linear activation functions are rarely used because in practice deep learning models would find it difficult to learn non-linear functional forms using linear activation functions. In previous chapters, we used the hyperbolic tangent as an activation function, namely f(x) = tanh(x). Hyperbolic tangent can work well in some cases, but a potential limitation is that at either low or high values, it saturates, as shown in the middle panel of the figure 4.2.
Perhaps the most popular activation function currently, and a good first choice (Nair, V., and Hinton, G. E. (2010)), is known as a rectifier. There are different kinds of rectifiers, but the most common is defined by the f(x) = max(0, x) function, which is known as relu. The relu activation is flat below zero and linear above zero; an example is shown in Figure 4.2.
The final type of activation function we will discuss is maxout (Goodfellow, Warde--Farley, Mirza, Courville, and Bengio (2013)). A maxout unit takes the maximum value of its input, although as usual, this is after weighting so it is not the case that the input variable with the highest value will always win. Maxout activation functions seem to work particularly well with dropout.
The relu activation is the most commonly-used activation function and it is the default option for the deep learning models in the rest of this book. The following graphs for some of the activation functions we have discussed:

- GitHub Essentials
- Unity 5.x Game AI Programming Cookbook
- 圖解機(jī)器學(xué)習(xí)算法
- 大數(shù)據(jù)導(dǎo)論
- Live Longer with AI
- Python醫(yī)學(xué)數(shù)據(jù)分析入門(mén)
- 深入淺出 Hyperscan:高性能正則表達(dá)式算法原理與設(shè)計(jì)
- 計(jì)算機(jī)應(yīng)用基礎(chǔ)教程上機(jī)指導(dǎo)與習(xí)題集(微課版)
- Chef Essentials
- 實(shí)用數(shù)據(jù)結(jié)構(gòu)
- Python數(shù)據(jù)分析從小白到專(zhuān)家
- 改進(jìn)的群智能算法及其應(yīng)用
- 數(shù)據(jù)庫(kù)原理與設(shè)計(jì)實(shí)驗(yàn)教程(MySQL版)
- 大數(shù)據(jù)分析:R基礎(chǔ)及應(yīng)用
- SQL Server 2012 數(shù)據(jù)庫(kù)教程(第3版)