官术网_书友最值得收藏!

The vanishing gradient problem

The vanishing gradient problem is one of the problems associated with the training of artificial neural networks when the neurons present in the early layers are not able to learn because the gradients that train the weights shrink down to zero. This happens due to the greater depth of neural network, along with activation functions with derivatives resulting in low value.

Try the following steps:

  1. Create one hidden layer neural network
  2. Add more hidden layers, one by one

We observe the gradient with regards to all the nodes, and find that the gradient values get relatively smaller when we move from the later layers to the early layers. This condition worsens with the further addition of layers. This shows that the early layer neurons are learning slowly compared to the later layer neurons. This condition is called the vanishing gradient problem.

主站蜘蛛池模板: 彭山县| 荣成市| 罗定市| 青铜峡市| 教育| 桃园县| 蕉岭县| 林甸县| 泗洪县| 吉水县| 栾城县| 永德县| 柳林县| 炎陵县| 宕昌县| 正蓝旗| 夏津县| 五家渠市| 铜陵市| 黔江区| 乡宁县| 卢湾区| 许昌县| 芜湖市| 夏津县| 宁津县| 溧水县| 涞源县| 枣强县| 固阳县| 杂多县| 安西县| 钟山县| 防城港市| 辉县市| 丰城市| 唐山市| 屯昌县| 民乐县| 南城县| 浮山县|