官术网_书友最值得收藏!

The vanishing gradient problem

The vanishing gradient problem is one of the problems associated with the training of artificial neural networks when the neurons present in the early layers are not able to learn because the gradients that train the weights shrink down to zero. This happens due to the greater depth of neural network, along with activation functions with derivatives resulting in low value.

Try the following steps:

  1. Create one hidden layer neural network
  2. Add more hidden layers, one by one

We observe the gradient with regards to all the nodes, and find that the gradient values get relatively smaller when we move from the later layers to the early layers. This condition worsens with the further addition of layers. This shows that the early layer neurons are learning slowly compared to the later layer neurons. This condition is called the vanishing gradient problem.

主站蜘蛛池模板: 微博| 湖州市| 临桂县| 绥阳县| 台州市| 高尔夫| 山丹县| 文水县| 五莲县| 达日县| 易门县| 南雄市| 芒康县| 卢龙县| 古交市| 平山县| 邮箱| 元阳县| 迁安市| 万山特区| 三穗县| 伊吾县| 那坡县| 大悟县| 七台河市| 凌海市| 诸城市| 镇远县| 岢岚县| 台州市| 哈密市| 青海省| 普陀区| 高碑店市| 运城市| 曲麻莱县| 淮南市| 姜堰市| 彰化县| 长沙县| 阳城县|