官术网_书友最值得收藏!

The vanishing gradient problem

The vanishing gradient problem is one of the problems associated with the training of artificial neural networks when the neurons present in the early layers are not able to learn because the gradients that train the weights shrink down to zero. This happens due to the greater depth of neural network, along with activation functions with derivatives resulting in low value.

Try the following steps:

  1. Create one hidden layer neural network
  2. Add more hidden layers, one by one

We observe the gradient with regards to all the nodes, and find that the gradient values get relatively smaller when we move from the later layers to the early layers. This condition worsens with the further addition of layers. This shows that the early layer neurons are learning slowly compared to the later layer neurons. This condition is called the vanishing gradient problem.

主站蜘蛛池模板: 柳林县| 安福县| 榆社县| 威海市| 钦州市| 东源县| 安图县| 昂仁县| 丰镇市| 余庆县| 漾濞| 宜兰市| 新竹县| 昆明市| 灵石县| 县级市| 丹巴县| 云和县| 万年县| 台江县| 襄城县| 蓝山县| 荣昌县| 什邡市| 莱阳市| 赤城县| 云南省| 麻栗坡县| 望谟县| 嘉定区| 呼图壁县| 常德市| 纳雍县| 梨树县| 吴川市| 抚松县| 军事| 天台县| 砀山县| 闽侯县| 合阳县|