官术网_书友最值得收藏!

Speeding up the training process using batch normalization

In the previous section on the scaling dataset, we learned that optimization is slow when the input data is not scaled (that is, it is not between zero and one). 

The hidden layer value could be high in the following scenarios:

  • Input data values are high
  • Weight values are high
  • The multiplication of weight and input are high

Any of these scenarios can result in a large output value on the hidden layer.

Note that the hidden layer is the input layer to output layer. Hence, the phenomenon of high input values resulting in a slow optimization holds true when hidden layer values are large as well.

Batch normalization comes to the rescue in this scenario. We have already learned that, when input values are high, we perform scaling to reduce the input values. Additionally, we have learned that scaling can also be performed using a different method, which is to subtract the mean of the input and divide it by the standard deviation of the input. Batch normalization performs this method of scaling.

Typically, all values are scaled using the following formula:

Notice that γ and β are learned during training, along with the original parameters of the network.

主站蜘蛛池模板: 松阳县| 麻阳| 金阳县| 林口县| 长岭县| 中西区| 保靖县| 广元市| 那曲县| 称多县| 阿拉尔市| 德阳市| 蓝山县| 保康县| 巩留县| 广东省| 奉贤区| 务川| 龙里县| 凤阳县| 将乐县| 万源市| 吴旗县| 静海县| 新平| 嘉兴市| 青冈县| 裕民县| 嫩江县| 库车县| 两当县| 广昌县| 左权县| 正宁县| 平湖市| 明星| 丹寨县| 抚顺市| 施秉县| 湛江市| 黄陵县|